Unit: Kawaihuelani Center for Hawaiian Language
Program: Hawaiian (MA)
Degree: Master's
Date: Mon Nov 05, 2018 - 11:20:35 am

1) Program Student Learning Outcomes (SLOs) and Institutional Learning Objectives (ILOs)

1. Reading: Demonstrate comprehension of traditional literary texts.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study.)

2. Listening: Demonstrate understanding and interpretation of various levels of spoken Hawaiian, including but not limited to native speaker dialog

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience.)

3. Speaking: Offer a quality* public presentation in Hawaiian

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience.)

4. Writing: Demonstrate competence in formal writing skills that have practical/contemporary application

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience.)

5. Culture: Demonstrate the ability to apply cultural norms in a range of communicative events

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 7. Interact professionally with others.)

6. Research: Construct a culturally sensitive research project that utilizes/analyzes relevant existing resources and contributes to the overall Hawaiian knowledge base

(2. Demonstrate understanding of research methodology and techniques specific to one’s field of study., 3. Apply research methodology and/or scholarly inquiry techniques specific to one’s field of study., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

2) Your program's SLOs are published as follows. Please update as needed.

Department Website URL: http://manoa.hawaii.edu/hshk/units/kamakakuokalani/degrees/master-of-arts-in-hawaiian-studies/
Student Handbook. URL, if available online: http://manoa.hawaii.edu/nhss/wp-content/uploads/2013/03/2017-KCHL-Grad-Student-Handbook.pdf
Information Sheet, Flyer, or Brochure URL, if available online:
UHM Catalog. Page Number:
Course Syllabi. URL, if available online:
Other: Native Hawaiian Student Services Website: http://manoa.hawaii.edu/nhss/graduate-students/masters-hawaiian/
Other:

3) Please review, add, replace, or delete the existing curriculum map.

Curriculum Map File(s) from 2018:

4) For your program, the percentage of courses that have course SLOs explicitly stated on the syllabus, a website, or other publicly available document is as follows. Please update as needed.

0%
1-50%
51-80%
81-99%
100%

5) Does the program have learning achievement results for its program SLOs? (Example of achievement results: "80% of students met expectations on SLO 1.")(check one):

No
Yes, on some(1-50%) of the program SLOs
Yes, on most(51-99%) of the program SLOs
Yes, on all(100%) of the program SLOs

6) Did your program engage in any program learning assessment activities between June 1, 2015 and October 31, 2018?

Yes
No (skip to question 17)

7) What best describes the program-level learning assessment activities that took place for the period June 1, 2015 to October 31, 2018? (Check all that apply.)

Create/modify/discuss program learning assessment procedures (e.g., SLOs, curriculum map, mechanism to collect student work, rubric, survey)
Collect/evaluate student work/performance to determine SLO achievement
Collect/analyze student self-reports of SLO achievement via surveys, interviews, or focus groups
Use assessment results to make programmatic decisions (e.g., change course content or pedagogy, design new course, hiring)
No (skip to question 17)
Investigate other pressing issue related to student learning achievement for the program (explain in question 7)
Other:

8) Briefly explain the assessment activities that took place.

After four years of successful implementation of our rubric assessment process (from AY2011-2012 to AY 2014-2015) in which faculty-developed rubrics have been used to assess our MA program SLOs through the evaluation of final theses/Plan B papers and oral defenses/presentations of our graduate students, we decided in Spring 2016 that it was time to review, analyze, and synthesize all of our rubric results from these four years. All results were compiled into various graphs in order to view percentages, trends, patterns, etc. These findings were then discussed by all of our graduate faculty in several meetings, which concluded with several decisions about how to use these findings in the improvement of our rubric assessment process, the assessment tools themselves, as well as our graduate program overall.

After going through this analysis, synthesis, and improvement, we implemented our revised rubric assessment process and tools in the evaluation of a new group of students who defended and graduated with their MAs in Hawaiian during the AY 2016-2017.

9) What types of evidence did the program use as part of the assessment activities checked in question 7? (Check all that apply.)

Artistic exhibition/performance
Assignment/exam/paper completed as part of regular coursework and used for program-level assessment
Capstone work product (e.g., written project or non-thesis paper)
Exam created by an external organization (e.g., professional association for licensure)
Exit exam created by the program
IRB approval of research
Oral performance (oral defense, oral presentation, conference presentation)
Portfolio of student work
Publication or grant proposal
Qualifying exam or comprehensive exam for program-level assessment in addition to individual student evaluation (graduate level only)
Supervisor or employer evaluation of student performance outside the classroom (internship, clinical, practicum)
Thesis or dissertation used for program-level assessment in addition to individual student evaluation
Alumni survey that contains self-reports of SLO achievement
Employer meetings/discussions/survey/interview of student SLO achievement
Interviews or focus groups that contain self-reports of SLO achievement
Student reflective writing assignment (essay, journal entry, self-assessment) on their SLO achievement.
Student surveys that contain self-reports of SLO achievement
Assessment-related such as assessment plan, SLOs, curriculum map, etc.
Program or course materials (syllabi, assignments, requirements, etc.)
Other 1:
Other 2:

10) State the number of students (or persons) who submitted evidence that was evaluated. If applicable, please include the sampling technique used.

In terms of our review of rubric results from our first four years of implementation, we analyzed and synthesized results from 14 individual students (13 defenses, 10 theses/Plan B papers).

In terms of our revised rubric assessment of student final theses/Plan B papers and oral defenses/presentations, we collected new evidence from seven (7) MA students who successfully completed and defended her research and then graduated during the AY 2016-2017. Faculty sitting on these students’ committees met to review and discuss both the paper and the defense before agreeing as a committee on levels of performance for each product overall and in terms of each SLO. (NOTE: A few students defended in AY 2017-2018 but they are still completing their final revisions so their rubric results will be included in future reports.)

11) Who interpreted or analyzed the evidence that was collected? (Check all that apply.)

Course instructor(s)
Faculty committee
Ad hoc faculty group
Department chairperson
Persons or organization outside the university
Faculty advisor
Advisors (in student support services)
Students (graduate or undergraduate)
Dean/Director
Other:

12) How did they evaluate, analyze, or interpret the evidence? (Check all that apply.)

Used a rubric or scoring guide
Scored exams/tests/quizzes
Used professional judgment (no rubric or scoring guide used)
Compiled survey results
Used qualitative methods on interview, focus group, open-ended response data
External organization/person analyzed data (e.g., external organization administered and scored the nursing licensing exam)
Other: synthesized and then analyzed rubric data over 4 years using

13) Summarize the results of the assessment activities checked in question 7. For example, report the percentage of students who achieved each SLO.

After analyzing and synthesizing all of our rubric results for students who defended and graduated with MAs in Hawaiian during the AY 2011-2012 to AY 2014-2015, here are our overall findings:

  • The majority of students (90-50%) assessed were rated as COMPETENT by their committees in all SLO categories and overall for their final papers and defenses.                                               

Exemplary              Competent           Developing

Speaking:                                       31%                       69%                    0%

Culture (defense):                           23%                       77%                    0%

Listening:                                       46%                       50%                    4% (respond to questions)

Defense Overall (n=13):                38%                      64%                   0%

 

Exemplary              Competent           Developing

Writing:                                          30%                       70%                    0%

Culture (paper):                              10%                       90%                    0%

Research:                                        50%                       50%                    0%

Reading:                                         50%                       50%                    0%

Paper Overall (n=10):                   30%                      70%                   0%

  • All students were rated as either competent or exemplary overall for their final papers and defenses and for all of the SLO categories EXCEPT in the listening SLO. One student in AY12-13 had a split score between competent (listening to questions) and developing (responding to the questions).
  • The MOST exemplary ratings were given to students in the research and reading SLO categories for their final papers (50%, 50%).
  • The LEAST exemplary ratings were given to students in the culture SLO category for both defenses and final papers (23% for defenses; 10% for papers).
  • The number of students rated at the exemplary level for the culture SLO has increased over the years, but just slightly:
    • 1 of 5 students in AY 11-12 were exemplary (defense) à 2 of 5 students in AY 13-14
    • 0 of 4 students in AY 11-12 were exemplary (papers) à 1 of 5 students in AY 13-14

After going through this exercise, we made changes to our rubric assessment process and to the tools themselves (see questions 14 & 15 below) and then implemented these changes in our assessment of new students (N=7) who defended and graduated with MAs in Hawaiian during the AY 2016-2017. Please see Tables 1 and 2 below for a summary of the results of our assessment of these seven student theses/Plan B final papers and oral defenses.

Table 1: Rubric Results for Hawaiian M.A. Student Plan A Theses or Plan B Non-Thesis (How many students were rated at each level? N=7)

Student Learning Objectives

Exemplary/ Distinguished

Between Exemplary & Competent

Competent/ Proficient

Developing/Not Yet Competent

N/A

Writing: Formal Research Paper

3

 

3

1

 

Culture: Worldview, Cultural Norms

4

 

3

 

 

Research: Culturally sensitive research project

4

2

1

 

 

Reading

5

1

 

 

1

OVERALL

4

2

1

 

 


Table 2: Rubric Results for Hawaiian M.A. Student Plan A Defense or Plan B Public Presentation (How many students were rated at each level? N=7)

Student Learning Objectives

Exemplary/ Distinguished

Between Exemplary & Competent

Competent/ Proficient

Developing/Not Yet Competent

Speaking: Public Presentation

3

 

3

1

Listening Comprehension

5

 

2

 

Culture: Worldview, Cultural Norms

4

1

2

 

OVERALL

3

2

2

 


The vast majority of students (6 of the 7) were defined as competent or exemplary in the eyes of their committee members in all SLO categories: writing, research, speaking, listening, culture, and overall performance. Only one student was defined as developing or not yet competent in the areas of writing and speaking. In addition, the majority of students scored above competent (either exemplary or between exemplary and competent) for most of the SLO categories, with only writing and speaking being the categories in which most students were evenly split between exemplary and competent. As these results confirm, this group of MA graduates in Hawaiian demonstrated a high quality of research and Hawaiian language skill overall. We are encouraged that their contributions will have a positive impact on the understanding of Hawaiian and the development of the teaching and learning of Hawaiian.

When comparing these results to our overall findings from the first four years of our rubric assessment shared above, this new group of students shows a positive upward trend across all SLO categories. For example, in previous years, most students scored “competent” across all SLO categories and overall for their papers and defenses. However, in this new group, the majority of students actually scored above competent across most of the SLO categories. And for the two SLO categories in which most of the students were evenly split between the two (writing and speaking), that is still an improvement from the students assessed in the previous years.

Finally, as we have reported in past years, an area we continue to watch closely is the level of student attainment of our cultural SLO in both their written work as well as their oral defense. In our analysis of all students assessed in the first four years of our rubric process, the least amount of exemplary ratings for both defenses and final papers were in the culture SLO category. However, in looking at this new group of students, we are encouraged to see that this is no longer the case, so again we are proud to see development of cultural sensitivity, awareness, and application on the part of our MA graduates, and attribute this upward trend to some of the changes we have made to our curriculum and requirements as explained in previous Annual Assessment Reports.

14) What best describes how the program used the results? (Check all that apply.)

Assessment procedure changes (SLOs, curriculum map, rubrics, evidence collected, sampling, communications with faculty, etc.)
Course changes (course content, pedagogy, courses offered, new course, pre-requisites, requirements)
Personnel or resource allocation changes
Program policy changes (e.g., admissions requirements, student probation policies, common course evaluation form)
Students' out-of-course experience changes (advising, co-curricular experiences, program website, program handbook, brown-bag lunches, workshops)
Celebration of student success!
Results indicated no action needed because students met expectations
Use is pending (typical reasons: insufficient number of students in population, evidence not evaluated or interpreted yet, faculty discussions continue)
Other:

15) Please briefly describe how the program used the results.

In terms of our review of rubric results from our first four years of implementation, after several meetings of our graduate faculty to discuss our overall findings, here are the different ways we decided to use these results:

  • Besides sharing our rubrics with our graduate students during our new graduate student orientations and in our Graduate Student Handbook, we decided to also introduce the rubrics early on in the program through two of our required courses, HAW 602 and 604, since these courses provide opportunities for students to practices the specific skills assessed in these rubrics—formal, oral presentations and writing their proposals. This way students are aware of our expectations and goals for them early on and they can focus on these skill areas throughout their coursework and research.
  • We revised our rubric score sheet slightly to include an area for professors to share feedback on the rubrics themselves so that we know exact how to improve these tools in the future for better implementation and more accurate results.
  • We revised our listening comprehension SLO so that it was less specific and more inclusive of the various types of listening activities that students are required to do not only during the program but also during their defense/final presentation. This new SLO is now reflected in the Defense/Final Presentation rubric, on our department website, and in our graduate student handbook.
  • We adjusted our overall rubric assessment process slightly to better reflect how we have been running it for the last few years.
  • We agreed that we must focus more on supporting students in reaching exemplary levels of achievement in our culture SLO given that our cumulative results during these first four years showed that not many of our students were reaching this level in either their papers or their defenses. NOTE: It is nice to see that our focus on this area has paid off in looking at the most recent rubric results from AY 2016-2017 (see question 13 above).
  • We agreed that we need to be stricter when it comes to assessing and accepting new students into our program initially, those who are more poised to succeed and reach the learning outcomes we have set for those who will graduate with a MA in Hawaiian.

Beyond these concrete decisions made immediately after our analysis, we also discussed other possible ways to continue to support our students in fulfilling our program SLOs. These ideas continue to be discussed and the hope is that some will be developed and implemented in the coming years.

  • Develop an additional requirement for all graduate students that has them do a more applied project that takes them outside the classroom yet still related to Hawaiian language, such as:
    • Assist in one of our undergraduate courses (teaching, curriculum development, assessment, etc.)
    • Assist at one of our Hawaiian language immersion schools
    • Assist in the development and implementation of one of our co-curricular activities at Kawaihuelani that support and engage both our students and larger community
    • Intern at a community organization that either supports directly or uses Hawaiian language broadly in their work
    • Participate in a Hawaiian language competition in our community
  • Continue to develop and strengthen our undergraduate program, possibly by creating a capstone experience required for majors and/or develop a rubric assessment process for our undergraduate program in order to directly assess our undergraduate program SLOs. 
  • Create a student organization for our graduate students so that they have a space to meet and support one another

 

16) Beyond the results, were there additional conclusions or discoveries? This can include insights about assessment procedures, teaching and learning, and great achievements regarding program assessment in this reporting period.

Revising our rubric assessment process and tools, which have been in use for several years, was an important next step in our overall assessment work for our graduate program. It not only provided valuable insight into the areas of our program that are working and deserve celebration but also into areas that we still need to develop further or focus more of our attention on. It was particularly important for our instructional faculty to see the value of this kind of work in general. At times, assessment can be viewed as an extra task that requires a lot of extra work with little return value. But, after presenting these findings to the graduate faculty, it was clear that they could see the value and more importantly the usefulness of this data in implementing changes to our program that will positively affect our students.

17) If the program did not engage in assessment activities, please justify.