We happily announce the 2023 winning projects. These six projects were selected for their high potential to improve student learning on a program outcome(s) and an institutional learning objective(s), and their commitment to faculty collaboration. Read about the assessment for learning improvement support fund.

2023 Awardees

Asian Studies
Athletic Training (Kinesiology & Rehabilitation Sciences)
Chemistry
Chinese (East Asian Languages & Literatures)
History
Life Sciences

---

Asian Studies

Assessing and Improving Undergraduate Research and Communication Skills in Asian Studies

Cathryn Clayton, Anna Stirr, and Patricio Abinales

Targeted Institutional Learning Objectives:

1. Inquire and conduct research
2. Communicate and report

In our 2018 assessment, two program student learning outcomes (PLOs) emerged as having the lowest rate of achievement and hence the greatest need for improvement: research skills and communication skills. We describe these outcomes as follows:

In a recent poll, Asian Studies faculty observed that our graduating students are only “approaching expectations” when it comes to doing research and expressing their findings in writing. However, faculty have not been able to pinpoint the specific skills that students need to improve, be it finding and assessing research materials, reading critically, synthesizing primary and secondary sources, or the mechanics of expressing their findings in writing. In this project, we will use a systematic, assessment-based approach to skills development aimed at remedying this persistent learning problem.

This project has a dual aim:

1. To form a systematic understanding of students’ research and communication skills through learning assessment
2. To use assessment results to design the curriculum for a 100-level course: Research Experience in Asian Studies (REAS).

Upon project completion, we plan to have in place a rigorous learning assessment structure to assess students’ research and communication skills, and we plan to design the major content for the new course: Research Experience in Asian Studies (REAS). The Assessment Committee will lead and coordinate collaborative discussions among all faculty members to develop a rubric and the REAS curriculum. 

For baseline student learning evaluation, faculty will use a collaboratively developed rubric to conduct a retrospective assessment of the research and communication skills among the current seniors in the Asian Studies BA program. Using these findings on students’ strengths and weaknesses in conducting and communicating research, we will develop the 100-level REAS course curriculum. At the end of fall 2024, the program’s Assessment Committee will use the rubric to assess the students enrolled in the new REAS course. The Assessment Committee will track the REAS-enrolled students and re-assess them in fall 2027 to compare their performance against the baseline assessment as well as their performance in the first year of the program.

Top of page

---

Athletic Training (Kinesiology and Rehabilitation Science)

Improving Assessment Efficacy and Efficiency Through Standardization of “Competent” Criteria by Using an Online Interactive Simulation Based Learning Tool

Kaori Tamura, Yukiya Oba, Bret Freemyer, and Kyoko Shirahara

Targeted Institutional Learning Objectives for advanced degree programs:

1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest, 
2. Critically analyze, synthesize, and utilize information and data related to one’s field of study

The MS in Athletic Training (AT) program is accredited by the Commission of Accreditation on Athletic Training Education (CAATE), and there are 38 competencies that the MS program has to address. Each competency is assessed didactically by the course instructor and/or clinically at the assigned clinical setting by the assigned preceptor. Students are required to obtain a minimum of “competent” rating for all 38 competencies. While we created a standardized assessment tool using a commercial data management system (Typhon) to address each competency, we recognized that there was a large discrepancy in the level of expectation to be “competent” among raters which affects the quality of assessment data. Furthermore, the competency assessment through in-person scenario-based activities in core classes is very time consuming as it requires one-on-one attention. In clinical settings, the challenge is that the student may not encounter specific injuries or medical conditions for the specific competencies assigned to be demonstrated that semester. The program faculty are then responsible for identifying the learning gaps (i.e., competencies not assessed in the clinic) and address them for the individual students. This is troublesome as faculty have to develop additional scenario-based activities to be implemented in class depending on individual students’ needs. 

From our initial assessment data, 38% of students (3 out of 8) received a “non-competent” rating on at least one competency, ranging from 1-8 competencies marked as “non-competent”. There was a large discrepancy in frequency of giving “non-competent” rating per rater. 62% of students (5 out of 8) received at least one “not rated” rating, ranging from 1-6 competencies marked as “non-rated”. This was mostly due to lack of clinical opportunity of specific medical conditions associated with the competencies (i.e., rare conditions that we don’t typically see in the athletic training setting), and we recognize that these competencies need to be addressed didactically through simulation activities. As for the Board of Certification for Athletic Training exam (the BOC) pass rate, we reported a 75% first-time pass rate and 100% overall pass rate for the 2021-2022 cohort.

The project’s goal is twofold: 

1. improve the learning assessment method, and 
2. address the problem of the clinical internship not exposing and testing certain students on certain competencies. 

These goals can be achieved by using an external, online education program, ATu. The ATu E-Learning tool includes case studies specifically to address rare conditions.

This interactive online program is created by AT faculty and the contents are specifically aligned with the CAATE Competencies. ATu offers e-learning using a simulation-based interactive tool to facilitate problem solving and clinical decision making skills. The e-learning has 8 categories, each consisting of 5-15 different case studies with a focus on addressing rare health conditions. As students unfold a case, they are challenged to select appropriate actions or clinical decisions, and at the end of the scenario, students are given a score based on their performance. Students also receive feedback and a summary of evidence-based clinical reasoning. Each case has a cut-off score that defines the “competent” level and students will be able to re-do the case until they become competent. We believe that this e-learning tool will improve the learning assessment method and increase efficiency in competency assessment.

Four program faculty, who are responsible for teaching core courses and assessing competencies in classroom setting, will collaborate to determine the alignment of e-learning cases, specific competency(ies), and core courses, how to best integrate this tool, and the effectiveness of ATu integration in improving our assessment method and student learning experiences.

The targeted program Student Learning Outcome is “Formulate a care plan for patients based on the International Classification of Functioning, Disability, and Health (ICF) examination,” which subsumes many CAATE competencies. The goal is for all students to obtain a “competent” rating for all competencies prior to graduation. The associated Program Goal is to “Prepare students to be successful on the BOC with expected outcomes of

• Overall BOC pass rate above 90%, and 
• First time BOC pass rate 80% or above. 

In addition to the e-learning tool, ATu offers a wide range of educational content based on the CAATE competencies, and students will be able to enhance their BOC preparation by focusing on their weaker domains, as identified by the competency assessments.

Top of page

---

Chemistry

Measuring the Amount of Energy in a Kukui Nut

Amy Fuller

Targeted Institutional Learning Objectives and Strategic Plan Goal:

1. Broad knowledge of the cultural, social, physical, and natural world
2. Integrate Hawaiian culture and history
3. Inquire and conduct research
4. Think critically and creatively
5. Respect for people and cultures, in particular Hawaiian culture
6. Stewardship of the natural environment
7. UH Mānoa Strategic Plan goal: Becoming a Native Hawaiian Place of Learning 

Chemistry 161L (laboratory class), which enrolls over 1,200 students annually, is often students’ first university-level, hands-on chemistry experience. To contribute to the UH Mānoa’s Strategic Plan goal of Becoming a Native Hawaiian Place of Learning, place-based learning with a focus on plants and substances used by Native Hawaiians is being integrated into the course. This assessment for learning improvement project seeks to help students develop a greater connection with Hawaiian culture through chemistry and foster a respect and understanding for Native Hawaiian practices. The project also aims to improve students’ general education knowledge in the physical sciences and their research skills. 

The kukui nut tree is a source of food, light, medicine for Native Hawaiians and is the Hawai‘i state tree. In ancient Hawai‘i, Hawaiians burned kukui nuts to provide light and a string of these nuts were used to tell time. Hawaiians also extracted the oil from the nut and burned it in a stone oil lamp called a kukui hele po (light, darkness goes) with a wick made of kapa cloth.

Student learning achievement is evaluated using a rubric, and students will complete a survey on the targeted learning objectives and goal. Because the lab experiment is new, it will be piloted in summer; implemented in fall, and the results will be used to adjust the curriculum as needed.

Top of page

---

Chinese (East Asian Languages and Literatures)

Development of SLO-focused Reading Materials and Testing Instruments to Improve Chinese Major and Flagship Student Literacy

Haidan Wang, Song Jiang, and Hsin-Tzu Jen

Targeted Institutional Learning Objectives: 

1. General education knowledge (languages)
2. Think critically and creatively
3. Communicate and report
4. Continuous learning and personal growth
5. Respect for people and cultures, in particular Hawaiian culture
6. Civic participation in their communities

Targeted Chinese BA program SLO: Read and comprehend texts written in Chinese from a variety of genres and contexts (e.g., newspapers, essay collections, novels). 

1. Students will be able to make meaningful inquiries that are beneficial to understanding the structure, stance, and opinion of the text;
2. Students will be able to think critically about what is the stance conveyed by the article by observing the linguistic cues;
3. Students will be able to infer the author’s intention by observing the use of lexical items;
4. Students will be able to summarize the text structure by grasping the main idea of each paragraph.

In literacy development, competent learners use their world knowledge and language skills to decode text structure and summarize the main idea efficiently. The Chinese BA program in the department of East Asian Languages and Literatures has started to develop students’ comprehension of Chinese texts from a variety of genres and contexts embracing Chinese language and culture and their critical thinking skills as its reading SLO. Although nearly 75%-80% students meet the targeted reading SLO, about one-third of the Chinese Language Flagship track students failed their Pre-Capstone reading test to reach Advanced Level proficiency, which led to 38% students forced-dropout of the Flagship track. Therefore, the Chinese Section faculty intend to develop advanced level proficiency-based reading kits to help improve student’s overall performance in reading. 

This project aims at providing learners enhanced training materials, using the assessment results to evidence students’ improvement, then integrating the materials into corresponding curricula. We expect this effort could assist all Chinese major students to meet the expected level of reading proficiency, as described in Chinese BA SLOs.

Specifically, we plan to develop reading materials that guide students to make meaningful interpretation of text structure, article stance, logical transitions, author’s intention, hidden meaning between the lines, implications, and other crucial clues for accurate comprehension instead of simply teaching them linguistic knowledge. The effectiveness of including these text decoding techniques in explicit instruction and the positive washback effects in proficiency building from the assessments targeting the same set of skills have been evidenced in various research (e.g., Katan & Andreas Baarts, 2020; Ness, 2016; Wale & Bishaw, 2020). 

One objective of this project is to first identify the Chinese-specific text decoding schemes, develop instructional materials based on the identified schemes, then implement the materials in teaching and tutoring. The other objective is to develop a set of assessment instruments to measure students’ mastery of the skills after the instructions and link the assessment outcomes to general Chinese reading proficiency. The team will collaborate on the following major activities: 

1. pre-testing students’ proficiency via CATRC; 
2. disaggregating pre-testing scores based on the skills intended to be tested, such as facts, opinions, author’s voice, etc.; 
3. preparing reading materials, developing comprehension activities and testing instruments; 
4. implementing materials in tutoring and reading sessions in related curricula; 
5. retesting students and analyzing pre- and-post testing results, and preparing the final report. 

We hope this project will be a useful pilot for us to continue preparing all pre-capstone students to reach or exceed corresponding SLOs.

Students will be reassessed after the implementation of reading materials. Faculty will refer the post-test scores to the expected proficiency stated in SLO in addition to comparing with pre-test scores. The project will run from 2023 to 2024.

Top of page

---

History

A Curricular Intervention to Build Historiography Skills (skills in identifying and analyzing historical literature)

Shana Brown, Joseph Foukona, Yuma Totani, and Karen Jolly

Targeted Institutional Learning Objectives:

1. General education knowledge: arts and humanities
2. Specialized study in an academic field
3. Think critically and creatively
4. Inquire and conduct research 
5. Communicate and report
6. Continuous learning and personal growth

Targeted History BA program learning outcome:
Students can identify, interpret, and evaluate primary sources and other relevant information (PLO 3).

The ability of students to identify, interpret, and evaluate primary sources and other historical texts is crucial to their ability to conduct historical research, which our undergraduate majors must do in order to complete their required capstone sequence. We have found that some students arrive at the capstone sequence (where they have to write an original research paper) without having built the necessary skills in analyzing important sources and information. In fall 2020, History faculty members assessed 27 students’ capstone papers in two courses: HIST 396 and 496. We found that fewer than half of the students (45%) met faculty expectations in achieving the program learning outcome (PLO) 3: identify, interpret, and evaluate primary sources and other relevant information. Faculty held extensive discussions and planned improvement actions.

This project furthers our assessment and curriculum improvement efforts for the improvement of PLO 3. Specifically, we plan to design a succinct “menu” of activities and assignments that History faculty can customize in different classes and thereby enable us to teach these skills more effectively across our program. The project will involve a small team of faculty volunteers (Brown, Foukona, Jolly, and Totani), but will also progress with regular discussion and feedback with all faculty.

Goals of current project:

1. Assess student learning progress across the curriculum
2. Develop curriculum template and options for faculty to adapt and adopt in their own courses to enhance students’ ability to critically synthesize and use sources

Specifically, the project leads will collaboratively identify foundational skills that students need to be able to critically synthesize sources and use classroom visits and discussions with students to identify learning activities that are most effective and have greatest student buy-in. Based on the findings, we will design a menu of effective instructional activities and assignments to scaffold the development of the identified foundational skills, and allow students to develop SLO #3 systematically and explicitly across the curriculum

In Fall 2024, after faculty have implemented one or more effective instructional activity/assignment, we will (re)assess capstone student work to see what level of improvement in learning has happened. Based on our findings, we will make further recommendations to faculty.

Top of page

---

School of Life Sciences

Enhancing and Assessing Undergraduate Teamwork in Life Sciences Research

Justin Walguarnery, Eliza Garfield, and Amy Moran

Targeted Institutional Learning Objectives:

1. Specialized study in an academic field
2. Think critically and creatively
3. Inquire and conduct research
4. Communicate and report

Targeted Life Sciences SLOs:
Students will…
SLO 3. Be able to apply the scientific method to generate new hypotheses, formulate experimental approaches and outline potential outcomes, applying appropriate logical and quantitative methods.
SLO 4. Work individually and in teams in an ethical manner, and demonstrate respect for diversity of viewpoints.
SLO 5. In oral and written forms, be able to communicate biological information clearly and professionally.

In initial assessment at introductory levels, students had difficulty developing and clearly stating appropriate hypotheses and associated testable predictions as components of the scientific method and research project design. To address this, in fall 2021, we added to BIOL 172 Introduction to Biology II a lesson focused specifically on these components of the scientific method, additional reading, in-class discussion, and highlighted examples of hypotheses and testable predictions incorporated into subsequent lessons throughout the semester. Assessed at the end of fall 2021 and then in fall 2022, 70% of students (21/30) scored 7 or higher on a 10-point scale assessing SLO 3 from a mid-term exam question asking students to provide an appropriate hypothesis and associated testable prediction in the context of a specified scenario. Assessed in a 300-level course in fall 2022, 100% of students (10/10) scored 7 or higher, indicating that our students are now acquiring proficiency in some of the individual prerequisite skills for developing their own collaborative research projects.

In terms of teamwork skills, we only have indirect assessment through students’ self-report on a single survey in several courses in one semester. Across biology courses sampled in this preliminary assessment in fall 2022, greater than 70% of students agreed or strongly agreed with all survey questions addressing our SLO 4 (Student will work individually and in teams in an ethical manner, and demonstrate respect for diversity of viewpoints), indicating that they felt their classmates were respectful of their opinions and viewpoints, they felt their classmates were respectful of others’ opinions and viewpoints, they felt their classmates’ behavior was supportive of course goals and activities, that they acted in an academically honest way, and that they demonstrated good compliance with laboratory and field safety guidelines. These results, however, do not provide us with insight into all aspects of the typical professional collaborative research scenarios for which we aim to prepare our students, so coursework specifically designed to mirror these scenarios and rubrics to assess their effectiveness are required.

This project aims to increase students’ ability to apply scientific logic, improve cooperative research skills, and establish a new assessment method for an SLO related to working in groups. The curriculum will be refined in summer 2023 to produce a course fully based on students working together to conduct original field biology research projects. Because all modern science is conducted in the context of collaboration, it is critical that we provide instruction in effective collaboration and have tools for assessing our success in those efforts. Because student groups will use digital tablet devices to work in novel research reflective of professional requirements in the life sciences, the updated course will serve as a testbed through which we will develop a rubric to assess teamwork in real-world scenarios.

The proposed activities and project goals include the following:

1. Curricular refinement to enhance students’ teamwork skills: Using the intensive six-week summer course BIOL 363 Biological Field Studies as our pilot course, we will review the existing pedagogical activities, which assign students into group work to handle discrete predefined tasks (e.g., collecting specimens), and amend these to require students to work in groups to conduct original field biology research. Students will design and implement research projects as a team from beginning to end, negotiating roles, managing timelines, maintaining peer-to-peer communication, analyzing data, considering alternate interpretations, and reaching collective conclusions.

2. Development and piloting of a rubric for teamwork in research: In BIOL 363, we will collect student and instructor input on building a rubric to assess teamwork. The course instructor will conduct midterm and exit discussions with BIOL 363 students to acquire perspectives on specific tasks, skills, challenges, and solutions encountered in ethical and productive cooperative research. Additional anonymous input will be collected from BIOL 363 students through end-of-semester online course and faculty evaluation surveys. Along with published content on collaboration in science education, these data will then be used by the team to assemble our rubric. Trial testing and alignment of the rubric will be conducted in classroom modules in the subsequent fall semester. Ultimately, the teamwork rubric will be integrated into additional lab courses, from the 100 to 400 level.

Top of page