Program: Native Hawaiian Student Services
Date: Mon Oct 14, 2013 - 1:14:30 am
1) Below are your program's student outcomes (SOs). Please add or update as needed.
The core values, which drive and shape our student support activities stems from our mission and informed by our collective experience and initial assessment work. In Summer 2013, NHSS drafted a new program logic model (see attachment). The document is still in working draft form, as we have not yet finalized our eloquent “long-term” impact and vision statements. Additionally, we feel that there is one more piece missing in this overall logic model that still needs to be represented here (as illustrated in the large gap in the second column). We are confident that this next year will help us fill in that missing piece with the assessment data we collect and our ongoing conversations about our work.
(1) Haumāna develop a sense of place & belonging at UH Mānoa.
(2) Haumāna build networks of mentorship support with peers.
(3) Haumāna develop a critical thinking epistemology using Hawaiian identity and history as the foundation.
(4) Haumāna recognize their place within the lāhui.
(5) Haumāna develop transferrable academic & career skills.
(6) Haumāna develop positive attitudes about their career or graduate school potential.
(7) Lāhui Hawai‘i develop a sense of place & belonging at UH Mānoa.
(8) NHSS develop and maintain a contributing presence in the Hawaiian community.
2) Your program's SOs are published as follows. Please update as needed.
Program's Website. URL: facebook.com/nativehawaiianstudentservices/info
Student Handbook. URL, if available online: Information Sheet, Flyer, or Brochure. URL, if available online: UHM Catalog. Page Number: Other: Posted in our two Student Resource Centers at QL 104 and KAMA 211 Other: Our old SLOs are posted in these locations, and will be updating them with the new SLOs.3) Provide the program's activity map or other graphic that illustrates how program activities/services align with program student outcomes. Please upload it as a PDF.
- File (10/13/2013)
4) Did your program engage in any program assessment activities between June 1, 2012 and September 30, 2013? (e.g., establishing/revising outcomes, aligning activities to outcomes, collecting evidence, interpreting evidence, using results, revising the assessment plan, creating surveys, etc.)
Yes
No (skip to question 14)5) For the period June 1, 2012 to September 30, 2013: State the assessment question(s) and/or assessment goals. Include the student outcomes that were targeted, if applicable.
Our assessment activities in the past have targeted all student learning outcomes. However, because we recently updated our student and organizational learning outcomes, we only have one student and one organizational learning outcome assessment data to report on for this yearʻs report:
(6) Haumāna develop positive attitudes about their career or graduate school potential.
(8) NHSS develop and maintain a contributing presence in the Hawaiian community.
Namely, this report focuses on several key NHSS activities: the Aka Lehulehu Internship program, the Hūlili Professional Development Program, and the inagural NHSS Hūlili Student Services conference. The assessment questions that guided these three assessment activities were:
- How did Aka Lehulehu Internship Program students reflect on their collective experiences in a way that interrogates their future path? In what ways did students consider their career or graduate school potential?
- In what ways did the Hūlili Professional Development Program provide Native Hawaiian students meaningful research experience and development? How did students describe the impact of such development on their future educational, career or life goals?
- To what extent did the NHSS Hūlili Student Services conference provide participants opportunities to network with other Native Hawaiian serving programs across the UH System? To what extent did the gathering allow sharing of best practices as well as provide avenues for future collaboration?
6) State the type(s) of evidence gathered to answer the assessment question and/or meet the assessment goals that were given in Question #5.
There were three key types of evidence that were gathered to answer the assessment questions we had:
- Aka Lehulehu Internship Program presentations. At the end of the Internship Program, the Internship Coordinator hosted a presentation day where all students presented on their overall learning and growth in the 12 week internship program. NHSS created a Internship Program rubric to assess the presentations on several dimensions assessing the depth of their reflections, and the ways in which they articulated their growth over measures such as personal reflection, future goals, and the ways in which students tied their own self discovery to their academic and career goals. 5 NHSS staff members filled out a rubric for each of the Internship program students on the presentation day.
- Hūlili Professional Development exit interviews. When Hūlili Professional Development program students completed their research project or sponsored conference travel, the student meets with two NHSS staff members who conduct a one-on-one exit interview with the student. The exit interview asks students to discuss their overall learning and development in the program.
- NHSS Hūlili Student Services confence survey. NHSS staff designed a conference evaluation survey to be sent out to all 65 attendees at the inaugural NHSS UH System Native Hawaiian student services conference. The conference evaluation attempted to understand attendees satisfaction with the overall conference, the opportunities it provided for sharing of best practices, networking, and collaborative work.
7) State how many persons submitted evidence that was evaluated. If applicable, please include the sampling technique used.
For the purposes of this report, 35 students submitted evidence that was evaluated, namely 23 internship presentations and 12 profesional development interviews. All Summer 2013 Internship Program students (n=23) prepared a final presentation at the end of their program. For the Professional Development Program, there were 22 students who received an award from this program, but not all completed their exit interviews due to varying research end dates. To date, we completed 12 exit interviews, which are the ones used in this report.
Additionally, for our NHSS Hūlili Student Services Conference, we sent out the survey to all 65 participants, and after four reminders, got a total of 51 responses.
8) Who interpreted or analyzed the evidence that was collected? Check all that apply.
Program faculty/staff member(s) Faculty/staff committee Ad hoc faculty/staff group Director or department chairperson Persons or organization outside the university Students (graduate or undergraduate) Dean or Associate Dean Advisory Board Other: 9) How did he/she/they evaluate, analyze, or interpret the evidence? Check all that apply.
Compiled survey results Used quantitative methods on student data (e.g., grades, participation rates) or other numeric data Used qualitative methods on interview, focus group, or other open-ended response data Scored exams/tests/quizzes Used a rubric or scoring guide Used professional judgment (no rubric or scoring guide used) External organization/person analyzed data (e.g., Social Science Research Institute) Other: 10) For the assessment questions/goals stated in Question #5, summarize the actual results.
Hūlili Native Hawaiian Student Services Conference
NHSS staff created a survey that assessed conference attendees satisfaction with the opportunities for networking, sharing best practices, and collaboration brainstorming/planning at the inaugural NHSS conference. The survey included a four-point Likert scale with 4 items, as well as 4 open ended questions. The survey was sent to all 65 conference attendees, of which 51 responded in the one month survey collection period, for a response rate of 78.5%. The survey responses were overwhelmingly positive.
Additionally, the open-ended survey responses were also very positive. Here are some highlights:
- I feel more supported in the work that I do as well as have a larger network of professionals I can call with questions.
- Good to see what others are doing and how we can all work together to assist the lahui.
- I talked with professionals that were from my own campus that I had not gotten a chance to work more closely with, that was one of the most valuable aspects of the conference.
- The conference provided a great opportunity to meet services providers from across the system. Having contacts and really "knowing" who everyone is will be a great help in providing continuous service for our students.
- We have never had a space/time to meet with all of those who service Native Hawaiian students in various capacities.
- This networking opportunity builds bridges for partnerships and sharing of best practices. I believe the opportunity to network was the greatest advantage of my participation in the conference. I was surrounded by an outstanding group of professionals who solidified my belief that there are many great minds working together toward a common goal of Native Hawaiian and overall student success.
Hūlili Professional Development Program
Each of the awardees were interviewed before their research project began, and once again at the end of their project. As of September 30, 2013, NHSS staff conducted 12 exit interviews of the Hūlili Professional Development awardees. Interview data was used to piece together a poetic transcription, or a poem-like composition from the raw interview data. In this poetic transcription, there is a quote from each of the 12 exit interviews conducted to piece together a collective representation of what students shared about their overall learning and growth through this experience.
Expand your intellectual horizons
Further reinforces why I am in school
Personal reinforcement
Connections I wouldn’t have made otherwise
Met inspirational people
Learned about community
Confidence boost
I think I can do this, I think I can compete with them
Confidence that you can do the work
Affirmation of my work
It’s reassuring to see others
Seeing people that care about the same things
Others have same issues
Everyone on the same page, refreshing
We’re not alone
Meeting famous experts
Treating us like colleagues
Foot in the door
Great to feel a part of that
Figured out my skills and niche
I need to network
Feeling of belonging
Feeling of being in the right place
Re-motivates me, reminds me why I’m in the program
They’re doing things that we didn’t think of
Inspire to do something back home
Came back with ideas
See how much we’re doing already
We need to get our people at these levels to change things
Want to work with Native Hawaiians and be an advocate
Deep conversations about nation building
Where we stand, where we want to go
We’re the ones that are going to make the change
Overall, at the exit interviews, students talked about how the professional development opportunity (whether it was supported research or conference travel) provided opportunities to network with others in their field, and helped reinvigorate/remind them why theyʻre in school, and the confidence about their work and overall potential to continue.
Aka Lehulehu Internship Program Presentation Rubric
For the past three years, the NHSS Internship Program sponsored a program-end capstone presentation day where interns, their families, and their internship sites are invited to join NHSS staff to celebrate the internship students overall growth and development in the Internship Program. Because much of the Internship Program is based on helping student identify and develop their own self awareness as the basis for their life, education and career goals. Thus, the capstone presentation is a summation of their own self discovery and how that process helped inform their future goals. NHSS staff created a rubric to assess different elements of the studentsʻ reflection process through a model they called "Looking Back to Look Forward." The rubric was out of a total of 20 points. 5 NHSS staff members completed a rubric for each of the 23 internship students in the Summer 2013 cohort. Each studentʻs score from each of the 5 evaluators were logged to come up with an overall cohort score of 16.435. The range was 12.6 to 20 with a median score of 16.4 and a standard deviation of 2.08. Thus, the large majority of the cohort had a evaluation team score of between 14.32 and 18.48.
NHSS staff have not yet met to debrief the results of this rubric evaluation. Preliminary feedback is that itʻs great the we start to map out rubrics to help assess our studentsʻ work. However, we still need to decide if rubric data like this can help us make decisions on improving programming.
11) What was learned from the results?
- The Hūlili Professional Development Program has done an exceptional job at offering a unique research and conference travel program for Native Hawaiian students at UHM. Students have said that NHSS "made it easy" to apply for funding, where alternative funding sources (GSO and departmental funding) is bureaucratic and defeating. Participants talked about how these research support allowed them to network with people in their field, reinforce why theyʻre in school (the pragmatic, seeing their academics in action), and provided them with confidence to continue in the field or further their education.
- The NHSS Hūlili Student Services conference was well received by attendees, namely for the opportunities to share lessons learned, network and meet other NH serving professionals in the UH System, and brainstorming/planning collaboration efforts. However, others remarked that the conference was too short (one day) to do all that they would like to do. Others talked about other formats, with more opportunities to talk with more people in attendance.
- The Internship Program has consistently done a great job at helping students gain valuable skills, but most importantly learn more about themselves to unveil their academic, career and life goals. Nevertheless, weʻve experimented with different evaluation activities, and are still approaching it in different ways so the evaluation best suits the unique program delivery.
12) State how the program used the results or plans to use the results. Please be specific.
- For the NHSS conference, NHSS staff have already started to use the survey results to inform planning for next year's conference. We've already decided that we will probably do a multi-day conference next time to meet participants' comments about getting more done, with opportunities for more in-depth networking.
- For the Hūlili Professional Development (PD) program, NHSS has already implemented the results from the exit interviews in the latest round of the Hūlili PD program, that accepted applications with a deadline of September 30, 2013. In this new round, we made some adjustments based on what we learned from the original cohort interviews. Because of the huge networking element that came out of the program, NHSS staff has expanded the networking pieces to include working with students on getting them business cards and other marketing tools to help them further develop their connections they build. Additionally, student participants will also be doing a professional development activity for other students in exchange for their research support from NHSS. Also, NHSS has expanded the definitions for research support based on the learning and development that students discussed in the first cohort interviews.
- For the Aka Lehulehu Internship Program, NHSS plans to add the exit interviews back into the evaluation plan for this program. Although the rubric exercise was new and helpful, it does not seem to be comprehensive enough in capturing the student learning and development of the program. NHSS plans to continue to find new ways to capture the role of the program in helping students further define and build their skills and confidence in regard to their academic, life, and career potential and goals.
13) Reflect on the assessment process. Is there anything related to assessment procedures your program would do differently next time? What went well?
In the past, the Research & Evaluation Coordinator has exclusively conducted all student interviews. This past year, though, another team mate conducted the Hūlili Professional Development program interviews, which was a great experience. It was really great to have another staff member conducting the interviews because there was shared ownership over the process, and that second team member learned first hand of what the students appreciated, how they developed, and was able to see ways the program could be improved firsthand. We will definitely find more opportunities in the future where other staff can get directly involved in collecting qualitative student evaluation data like this.
This was the first time we used a rubric to assess student presentations. Although it was a positive experience, especially involving 4 other staff members in completing the rubric (and being intentional about what weʻre "looking for" when watching the presentations), the rubrics still need to be revisited.
What went especially well this past year was revisiting our logic model and student learning outcomes this past summer. In the past, we constructed them all from the ground up together with all staff members present. This time around, the Research & Evaluation Coordinator put together several months of internal evaluation data, especially what students have said about the role of NHSS in supporting their growth, and presented a draft to the team. Then, the team spent half a day reviewing them in pairs and presenting their feedback to the larger group. There will probably be 1-2 more meetings required to plan out the new evaluation plan on the finalized organizational documents.