Degree Program Assessment Reports

1. Reading: Demonstrate comprehension of traditional literary texts.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study.)

2. Listening: Demonstrate understanding and interpretation of various levels of spoken Hawaiian, including but not limited to native speaker dialog

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience.)

3. Speaking: Offer a quality* public presentation in Hawaiian

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience.)

4. Writing: Demonstrate competence in formal writing skills that have practical/contemporary application

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience.)

5. Culture: Demonstrate the ability to apply cultural norms in a range of communicative events

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 7. Interact professionally with others.)

6. Research: Construct a culturally sensitive research project that utilizes/analyzes relevant existing resources and contributes to the overall Hawaiian knowledge base

(2. Demonstrate understanding of research methodology and techniques specific to one’s field of study., 3. Apply research methodology and/or scholarly inquiry techniques specific to one’s field of study., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

Curriculum Map File(s) from 2018:

• File (03/16/2020)

No
Yes, on some(1-50%) of the program SLOs
Yes, on most(51-99%) of the program SLOs
Yes, on all(100%) of the program SLOs

Create/modify/discuss program learning assessment procedures (e.g., SLOs, curriculum map, mechanism to collect student work, rubric, survey)
Collect/evaluate student work/performance to determine SLO achievement
Collect/analyze student self-reports of SLO achievement via surveys, interviews, or focus groups
Use assessment results to make programmatic decisions (e.g., change course content or pedagogy, design new course, hiring)
No (skip to question 17)
Investigate other pressing issue related to student learning achievement for the program (explain in question 7)
Other:

Artistic exhibition/performance
Assignment/exam/paper completed as part of regular coursework and used for program-level assessment
Capstone work product (e.g., written project or non-thesis paper)
Exam created by an external organization (e.g., professional association for licensure)
Exit exam created by the program
IRB approval of research
Oral performance (oral defense, oral presentation, conference presentation)
Portfolio of student work
Publication or grant proposal
Qualifying exam or comprehensive exam for program-level assessment in addition to individual student evaluation (graduate level only)
Supervisor or employer evaluation of student performance outside the classroom (internship, clinical, practicum)
Thesis or dissertation used for program-level assessment in addition to individual student evaluation
Alumni survey that contains self-reports of SLO achievement
Employer meetings/discussions/survey/interview of student SLO achievement
Interviews or focus groups that contain self-reports of SLO achievement
Student reflective writing assignment (essay, journal entry, self-assessment) on their SLO achievement.
Student surveys that contain self-reports of SLO achievement
Assessment-related such as assessment plan, SLOs, curriculum map, etc.
Program or course materials (syllabi, assignments, requirements, etc.)
Other 1:
Other 2:

Course instructor(s)
Faculty committee
Ad hoc faculty group
Department chairperson
Persons or organization outside the university
Faculty advisor
Advisors (in student support services)
Students (graduate or undergraduate)
Dean/Director
Other:

After analyzing and synthesizing all of our rubric results for students who defended and graduated with MAs in Hawaiian during the AY 2011-2012 to AY 2014-2015, here are our overall findings:

• The majority of students (90-50%) assessed were rated as COMPETENT by their committees in all SLO categories and overall for their final papers and defenses.                                               

Exemplary              Competent           Developing

Speaking:                                       31%                       69%                    0%

Culture (defense):                           23%                       77%                    0%

Listening:                                       46%                       50%                    4% (respond to questions)

Defense Overall (n=13):                38%                      64%                   0%

 

Exemplary              Competent           Developing

Writing:                                          30%                       70%                    0%

Culture (paper):                              10%                       90%                    0%

Research:                                        50%                       50%                    0%

Reading:                                         50%                       50%                    0%

Paper Overall (n=10):                   30%                      70%                   0%

• All students were rated as either competent or exemplary overall for their final papers and defenses and for all of the SLO categories EXCEPT in the listening SLO. One student in AY12-13 had a split score between competent (listening to questions) and developing (responding to the questions).

• The MOST exemplary ratings were given to students in the research and reading SLO categories for their final papers (50%, 50%).

• The LEAST exemplary ratings were given to students in the culture SLO category for both defenses and final papers (23% for defenses; 10% for papers).

• The number of students rated at the exemplary level for the culture SLO has increased over the years, but just slightly:
  • 1 of 5 students in AY 11-12 were exemplary (defense) à 2 of 5 students in AY 13-14
  • 0 of 4 students in AY 11-12 were exemplary (papers) à 1 of 5 students in AY 13-14

After going through this exercise, we made changes to our rubric assessment process and to the tools themselves (see questions 14 & 15 below) and then implemented these changes in our assessment of new students (N=7) who defended and graduated with MAs in Hawaiian during the AY 2016-2017. Please see Tables 1 and 2 below for a summary of the results of our assessment of these seven student theses/Plan B final papers and oral defenses.

Table 1: Rubric Results for Hawaiian M.A. Student Plan A Theses or Plan B Non-Thesis (How many students were rated at each level? N=7)

Student Learning Objectives	Exemplary/ Distinguished	Between Exemplary & Competent	Competent/ Proficient	Developing/Not Yet Competent	N/A
Writing: Formal Research Paper	3		3	1
Culture: Worldview, Cultural Norms	4		3
Research: Culturally sensitive research project	4	2	1
Reading	5	1			1
OVERALL	4	2	1

Table 2: Rubric Results for Hawaiian M.A. Student Plan A Defense or Plan B Public Presentation (How many students were rated at each level? N=7)

Student Learning Objectives	Exemplary/ Distinguished	Between Exemplary & Competent	Competent/ Proficient	Developing/Not Yet Competent
Speaking: Public Presentation	3		3	1
Listening Comprehension	5		2
Culture: Worldview, Cultural Norms	4	1	2
OVERALL	3	2	2

The vast majority of students (6 of the 7) were defined as competent or exemplary in the eyes of their committee members in all SLO categories: writing, research, speaking, listening, culture, and overall performance. Only one student was defined as developing or not yet competent in the areas of writing and speaking. In addition, the majority of students scored above competent (either exemplary or between exemplary and competent) for most of the SLO categories, with only writing and speaking being the categories in which most students were evenly split between exemplary and competent. As these results confirm, this group of MA graduates in Hawaiian demonstrated a high quality of research and Hawaiian language skill overall. We are encouraged that their contributions will have a positive impact on the understanding of Hawaiian and the development of the teaching and learning of Hawaiian.

When comparing these results to our overall findings from the first four years of our rubric assessment shared above, this new group of students shows a positive upward trend across all SLO categories. For example, in previous years, most students scored “competent” across all SLO categories and overall for their papers and defenses. However, in this new group, the majority of students actually scored above competent across most of the SLO categories. And for the two SLO categories in which most of the students were evenly split between the two (writing and speaking), that is still an improvement from the students assessed in the previous years.

Finally, as we have reported in past years, an area we continue to watch closely is the level of student attainment of our cultural SLO in both their written work as well as their oral defense. In our analysis of all students assessed in the first four years of our rubric process, the least amount of exemplary ratings for both defenses and final papers were in the culture SLO category. However, in looking at this new group of students, we are encouraged to see that this is no longer the case, so again we are proud to see development of cultural sensitivity, awareness, and application on the part of our MA graduates, and attribute this upward trend to some of the changes we have made to our curriculum and requirements as explained in previous Annual Assessment Reports.

In terms of our review of rubric results from our first four years of implementation, after several meetings of our graduate faculty to discuss our overall findings, here are the different ways we decided to use these results:

• Besides sharing our rubrics with our graduate students during our new graduate student orientations and in our Graduate Student Handbook, we decided to also introduce the rubrics early on in the program through two of our required courses, HAW 602 and 604, since these courses provide opportunities for students to practices the specific skills assessed in these rubrics—formal, oral presentations and writing their proposals. This way students are aware of our expectations and goals for them early on and they can focus on these skill areas throughout their coursework and research.
• We revised our rubric score sheet slightly to include an area for professors to share feedback on the rubrics themselves so that we know exact how to improve these tools in the future for better implementation and more accurate results.
• We revised our listening comprehension SLO so that it was less specific and more inclusive of the various types of listening activities that students are required to do not only during the program but also during their defense/final presentation. This new SLO is now reflected in the Defense/Final Presentation rubric, on our department website, and in our graduate student handbook.
• We adjusted our overall rubric assessment process slightly to better reflect how we have been running it for the last few years.
• We agreed that we must focus more on supporting students in reaching exemplary levels of achievement in our culture SLO given that our cumulative results during these first four years showed that not many of our students were reaching this level in either their papers or their defenses. NOTE: It is nice to see that our focus on this area has paid off in looking at the most recent rubric results from AY 2016-2017 (see question 13 above).
• We agreed that we need to be stricter when it comes to assessing and accepting new students into our program initially, those who are more poised to succeed and reach the learning outcomes we have set for those who will graduate with a MA in Hawaiian.

Beyond these concrete decisions made immediately after our analysis, we also discussed other possible ways to continue to support our students in fulfilling our program SLOs. These ideas continue to be discussed and the hope is that some will be developed and implemented in the coming years.

• Develop an additional requirement for all graduate students that has them do a more applied project that takes them outside the classroom yet still related to Hawaiian language, such as:
  • Assist in one of our undergraduate courses (teaching, curriculum development, assessment, etc.)
  • Assist at one of our Hawaiian language immersion schools
  • Assist in the development and implementation of one of our co-curricular activities at Kawaihuelani that support and engage both our students and larger community
  • Intern at a community organization that either supports directly or uses Hawaiian language broadly in their work
  • Participate in a Hawaiian language competition in our community
• Continue to develop and strengthen our undergraduate program, possibly by creating a capstone experience required for majors and/or develop a rubric assessment process for our undergraduate program in order to directly assess our undergraduate program SLOs. 
• Create a student organization for our graduate students so that they have a space to meet and support one another