2014

Access to College Excellence (ACE) at UHM provides a series of first year programs that serve approximately 25% of incoming freshmen a year. A major ACE program is Learning Communities, in which ACE staff helps students to register a collection of courses in a cluster (e.g., general education, pre-business) and an Access to College Community freshman seminar course: CAS 110. All the students who registered for the courses in one cluster form a cohort—a learning community. CAS 110 enhances the community by helping students reaching the following learning outcomes: (1) building effective study and time management skills; (2) setting goals for success; (3) learning about the academic resources available; (4) learning core and major graduation requirements at UHM. This poster introduces the ACE learning communities and how it assesses the outcomes by using a pre- and post-survey method. The poster will present quantitative results from the closed-ended questions and qualitative results from the open-ended questions. The presenter will also present the impact of ACE Learning Communities by comparing the retention rates between the participants and non-participants, and between the less-engaged and more-engaged participants.

Assessment of ACE Learning Communities 2013

Access to College Excellence (ACE) at UHM provides a series of first year programs that serve approximately 25% …

TIM school began to build the program assessment frame in 2009 and have developed the student learning outcome, curriculum map, and experienced a couple of rounds of actual assessment. We would like to share the process and learning of TIM school as we engaged in the assessment process and how the Assessment office helped every step of the way. The poster will be designed to illustrate the steps TIM school went through and types of help provided by the Assessment office as well as the learnings we acquired along the way in regards to the program assessment.

How TIM School Began and Progressed with Program Assessment

TIM school began to build the program assessment frame in 2009 and have developed the student learning outcome, …

In Spring 2013, the Matsunaga Institute for Peace and Conflict Resolution (MIPCR) began a program review process by completing a curriculum map of the Graduate Certificate in Conflict Resolution (GCCR). The 15-credit interdisciplinary certificate is designed to address a set of program-level student learning outcomes (SLO). A curriculum assessment would determine if the GCCR’s Peace and Conflict Education (PACE) courses, as well as GCCR-approved courses from other programs, aligned with these outcomes. It would also reveal whether students had adequate opportunities to achieve the outcomes. This initiative would open dialogue among faculty, administrators, and students to discuss student success and program improvement. Instructors of PACE and GCCR-approved courses received a survey listing each program-level SLO and were asked to indicate the degree to which their course addressed each outcome. Responses to the surveys were recorded in a curriculum map and presented to a volunteer review committee for interpretation and discussion. This committee made recommendations for the program, including clarifying program goals and positioning to guide program-level SLOs, and further clarifying SLOs to ensure collective understanding by faculty members. Committee members also noted that there are limited opportunities for students to focus on professional ethics. It was further recommended that faculty and staff ownership of the review process be fostered through transparency and collaboration. Challenges of meeting SLOs through interdisciplinary programming were also revealed. The MIPCR Curriculum Committee discussed the results and recommendations and concluded that a more complete curriculum map and continued discussion was needed before making any curricular decisions.

Starting the Curricular Conversation through Mapping: A Curriculum Review of the Graduate Certificate in Conflict Resolution

In Spring 2013, the Matsunaga Institute for Peace and Conflict Resolution (MIPCR) began a program review process by …

SLS 150 is an introductory course mainly for first-year (freshman) students with a strong interest in second language learning. A series of innovative and rigorous classroom assessment strategies have been piloted in this class. Lessons learned from this internal exercise will be extended to the full suite of the new SLS BA program and its progress with program assessment as we move forward to the five year provisional status review.

Classroom Assessment – Implications for Program Assessment

SLS 150 (Learning Languages and Communicating in a Globalized World) is an introductory course mainly for first-year (freshman) …

In the last 6 months, the Department of Psychology has initiated a new plan that is expected to provide useful information that will eventually assess our Psychology Undergraduate Program. This new Undergraduate Assessment Plan (UAP) emerged from training in Assessment Leadership Institute at UHM, which highlighted major principles and strategies to assess academic programs. With this training knowledge, an undergraduate curriculum map was developed with faculty input that incorporates the most recent Student Learning Objectives (SLOs) from the American Psychological Association (APA). These SLOs that graduating psychology majors should know include: psychological knowledge of key concepts; scientific inquiry and critical thinking using scientific reasoning to interpret psychological phenomenon; ethical and social responsibility in diverse settings; communication development for effective writing and oral presentation; and professional development for meaningful direction after graduation. The curriculum map is important in describing specific SLOs associated with each Psychology undergraduate course taught at the University of Hawaii at Manoa. Effective Fall 2014, all instructors will include SLOs in their syllabus that reflects their course content. Another feature of the UAP in development is the incorporation of a rubric to assess student writing. To do so, a rubric for writing to meet the APA communication SLO was obtained from the American Association of Schools & Universities and modified to assess the writing occurring in our Psychology W courses. Pilot work is being conducted to determine the feasibility and reliability of a rubric to assess student writing in a small sample of student papers. On the basis of this study, the scoring and assessment method of papers may require modifications before presenting a working rubric to the department faculty for discussion. The UAP is to eventually require a final paper written in Psychology W courses to be assessed by the instructor using a rubric accepted by the Psychology faculty.

Department of Psychology Undergraduate Assessment Plan

In the last 6 months, the Department of Psychology has initiated a new plan that is expected to …

The Political Science faculty modified existing departmental, undergraduate student learning outcomes (SLOs), which informed assessment development and curricular improvement. Using informal appraisal by individual faculty of student performance at the senior capstone level, the undergraduate curriculum committee found that students’ writing skills need to be addressed in a more cohesive and purposeful way in the curriculum. A 200- level writing politics course was designed to address the gap. In short, the discussions around the learning outcomes revision resulted in faculty collaboration in assessmentdriven curricular improvement. These discussions and innovations are part of a two-year plan to improve program assessment in the department.

Learning Outcome Driven Curricular Improvement – An Example from Political Science

The Political Science faculty modified existing departmental, undergraduate student learning outcomes (SLOs), which informed assessment development and curricular …

This study was intrigued by the needs to assess oral performance more systematically and to motivate students to take assessment more seriously. The participants of the study were students and faculty of Korean Language Flagship Center (KLFC) MA program in the department of East Asian Languages and Literatures. There was a brief criterion for oral assessment in the program, but it was not enough to meet the aforementioned needs. Therefore, the researcher with the help of faculty in the program and center developed a detailed rubric, Flagship Rubric. Utilizing assessment evidences such as video recordings and student feedback sheets, development phases were as follows: 1) rough version, 2) revision, 3) pilot test, 4) second revision, and 5) ‘Flagship Rubric.’ The poster session will show this process and the product, and how it will be incorporated in the program with a dissemination plan.

Flagship Rubric for Oral Performance

This study was intrigued by the needs to assess oral performance more systematically and to motivate students to …

As the only institution of its kind in the world, the UHM Korean Language Flagship Center (KLFC) undergraduate program has been funded by the National Security Education Program since Fall 2008. The KLFC’s goal is to prepare American students to be capable of functioning in Korean as professionals in their chosen academic or professional fields. Flagship student performance is monitored and evaluated via various types of tests, which are administered on a regular basis throughout the program. The assessment tools that the KLFC BA program uses were developed externally and internally to measure students’ Korean language proficiency in four language skills. One of the assessment tools is an official ACTFL Oral Proficiency Interview (OPI) for speaking, provided by the College of LLL for all graduate BA language majors. Twenty-two Korean Flagship BA students from Spring 2011 to Fall 2013 have taken the official ACTFL OPI tests, and 21 out of 22 students’ speaking proficiency has improved over the two or three year period. We present these exit OPI results along with the entry OPI results to demonstrate the improvement in the student speaking proficiency. Furthermore, based on the exit scores, the Korean Flagship program has implemented changes for curricular improvement. This poster also displays curricular adjustment we have made, and the changes are expected to result in further improvement of our student speaking proficiency.

Korean Flagship BA Program Assessment Results & Curricular Improvement

As the only institution of its kind in the world, the UHM Korean Language Flagship Center (KLFC) undergraduate …

Swine Production is a senior capstone course in which students integrate concepts learned in courses such as nutrition, genetics and reproduction and apply them to practical swine production. Most animal science students at the University of Hawaii at Manoa are urban students interested in veterinary medicine, and little knowledge of or interest in swine production. The department learning objectives include applying principles to livestock production, but having students engage in learning the course material is a challenge. The extension specialist teaching the course tried a new approach in fall, 2013, integrating instruction and extension. Instead of traditional classroom instruction and written reports, teams of three students were sent out to work with a cooperating farmer. The teams described their farm using a list of required information on general farm description, breeding program, feeding program, housing, waste management, health program, production management and economics. The farms were carefully selected to represent different management approaches including varying use of local food waste and agricultural byproducts in the swine rations and different types of housing and waste management. The teams presented the results of farm visits and farmer interviews in class presentations and written reports, which were revised before being submitted in final form to both the instructor and the farmer. The course was designated as developing both oral and written communication skills. Course evaluations indicated that 78% of the students felt that the farm visits and interactions with the farmers were the most valuable part of the course, and 44% mentioned hands-on laboratories (artificial insemination and baby pig processing), while others mentioned constructive feedback, small class size, having to think, oral presentations, and writing. Student journals indicated a strong rapport with and a very positive view of the farmers and farm practices. The farmers changed some of their practices based on student recommendations, most notably adjusting feeding according to condition score. Student grades were assigned using posted rubrics and were equal to or better than in previous years in a more traditional learning environment. The new structure had some challenges. Since this was the first time the course was taught using student engagement with producers, students were informed that some adjustment to assignments and grading rubrics might be made part way through the class and this in fact needed to be done. One example was development of a tool in which the relative contributions of the team members to the group reports were evaluated by all members of each team. Additionally, students were required to sign liability waivers and confidentiality agreements. Students had to use their own cars and to find times when all team members were available for farm visits. Students indicated that the amount of work justified 4 rather than 3 credits for the course.

Integrating Teaching and Extension: Swine Production

Swine Production is a senior capstone course in which students integrate concepts learned in courses such as nutrition, …

New LCME accreditation standards will require medical schools to monitor curriculum and content and to develop a curriculum inventory over the next 1--2 years. There is little literature on the use of curriculum mapping to evaluate Problem-Based Learning (PBL) curriculum. Equipping medical students with knowledge and skills to care for our rapidly aging population is critical. We describe how developing a curriculum map for geriatric medicine core competencies during preclinical education helped identify areas to target revision of curricula. While curriculum mapping will be necessary in meeting AAMC inventory requirements in the future, the Curriculum Mapping of Geriatric Medicine Core Competencies at JABSOM has already proven to be an effective strategy. During our analysis, we found that we covered many LCME survey topics important for accreditation (e.g. care of the disabled, abuse, end--of--life care, health care systems). Through examination of the preclinical curriculum, we identified areas of deficiency, and also that the sequence was not ideal. Greater emphasis on geriatric physiology should occur earlier in the preclinical years, and coverage of optimal medication management should occur in later courses. Greater discussion regarding “Hospital Care for Elders,” including the important topics of patient safety and discharge planning towards the end of the second year, would also help better prepare students for their clinical years. Revisions to the curriculum and examinations are planned. We also plan to expand our analysis to include interprofessional education and cultural competence.

Curriculum Mapping of Geriatric Medicine Core Competencies in the Preclinical Problem-Based Learning Curriculum at the John A Burns School of Medicine, University of Hawai′i

New LCME accreditation standards will require medical schools to monitor curriculum and content and to develop a curriculum …

This poster is going to present how faculty in the Academy for Creative Media undergraduate program utilized an efficient and collaborative process of rubrics development for program assessment that led to program improvement. The Academy for Creative Media undergraduate program has three tracks: Cinematic Digital Production, Critical Studies, and Animation. After developing program SLOs and the curriculum map, the faculty felt the need to develop rubrics for each track to evaluate student learning products. Accordingly, the faculty broke up into sub-groups using examples of exemplary student work from each track to discuss the reasons these works were excellent. These ideas helped the faculty develop descriptions of excellent work for each track’s rubric. The faculty from the Animation and Critical Studies tracks also developed descriptors of other levels of work (e.g., good, average, poor). The process took three department meetings’ time with one hour each.

Collaborative Rubric Development for Program Improvement

This poster is going to present how faculty in the Academy for Creative Media undergraduate program utilized an …

The Communication Department’s undergraduate curriculum is currently guided by seven SLOs (Student Learning Outcomes), which are demonstrable skills or abilities that students are expected to possess before the conferral of a bachelor's degree. We, with our poster presentation, propose an additional SLO in civic engagement. Civic engagement has not been a formally assessed part of the department's curriculum in the past, but a recent internal evaluation has revealed it to be an important aspect of our faculty's teaching. A department-wide curriculum change to formally include this area would, therefore, pose no undo imposition on faculty independence. The students, more importantly, would benefit from new competencies and gain an awareness of issues of public concern that are essential to sound democratic governance. The SLO framework, by design, promotes effective learning through the application of a cohesive curriculum. Put simply, each class serves to advance a common educational aim or aims. Our proposed SLO—and its accompanying curriculum map—functions no differently. The assessment of student portfolios was directed by a rubric endorsed by the Association of American Colleges and Universities, but other assessment tools may be used in its place. In addition to advocating an official position on the teaching of civic engagement, our poster project is intended to foster faculty discussion on enhancing the Communication Department's curriculum.

Encouraging Civic Engagement Among Undergraduates

The Communication Department’s undergraduate curriculum is currently guided by seven SLOs (Student Learning Outcomes), which are demonstrable skills …

The Communication Department’s mission is to meet the challenges and opportunities of communication in the emerging technological, multicultural, and global context of the twenty-first century. We offer one of two undergraduate programs in the School of Communications (College of Social Sciences), serving approximately 200 active Communication Department undergraduates, with an average of 80 graduates per year. Our program offers three area tracks with senior capstones. In each of these capstones, students create an e-portfolio that is used for program assessment. This poster reports the process and results of the Communication Department’s 2013 BA assessment. A panel of faculty and Advisory Board members assessed all the e-portfolios in the Communication in Communities track (n=40). A rubric with measurable items associated with our seven SLOs was employed. We present a summary report demonstrating the percentage of graduates at each level of the rubric (unacceptable, marginal, proficient, or exemplary) and report the total percentage meeting our benchmark for each SLO. We describe how the results of our annual assessment were shared with Communication faculty, and we outline the changes we have begun to implement that we anticipate will have the greatest impact on curricular improvement

Communication BA Curriculum Assessment Using e-Portfolios

The Communication Department’s mission is to meet the challenges and opportunities of communication in the emerging technological, multicultural, …

College of Education (COE) programs select six to eight key program assessments to systematically measure, collect and analyze data on student learning. Faculty members enter assessment data each semester into our in-house COE Student Information System (SIS). The SIS allows us to summarize and compare data within and across programs at all levels. The data allow us to test our beliefs about our programs and make adjustments based on actual candidate performance and stakeholder feedback. During the 2012-13 academic year, using the data entered into the SIS, we began creating visuals of aggregate student performance on key assessments and posting these data on our college website, under the header “Measuring Our Success.” The use of data visuals provides not only a more engaging means for faculty to analyze student performance, but also allows us to share this information with multiple audiences. Through meeting with multiple stakeholders, we have learned how important it is to make data available to the public, as well as to our own educational community. Examples of activities and changes based on student learning data that have led to continuous improvement of candidate performance in COE programs will also be provided.

Measuring Our Success

College of Education (COE) programs select six to eight key program assessments to systematically measure, collect and analyze …

KAPA is FREE open-source software, developed by UHM College of Education. It requires a web application server and a database server.

KAPA Academic Program Assessment System (Technology Solutions for Program Assessment)

KAPA is FREE open-source software, developed by UHM College of Education. It requires a web application server and …

This poster will present the results of efforts to develop a robust program for assessment of the graduate degree programs in Civil Engineering (CE) at UHM. Prior to the start of this project, CE had outdated program outcomes for MS and PhD programs that were difficult to assess and a very simple and inadequate assessment system. This project involved creation of new program-level student learning outcomes (SLOs) for both MS and PhD using better descriptions and Bloom’s taxonomy verbs and then finalization in an interactive faculty department meeting. This was followed by development of rubrics for the SLOs to be used for assessment of graduate student work. The rubrics were worked on in an interactive CE Assessment Committee meeting followed by presentation/modification in a CE Department meeting. This was followed by development and adoption of a plan for analysis of results, determination of necessary program changes and implementation of changes. The products of this project will be presented as well as lessons learned regarding the process.

MS/PhD Outcome Assessment Program Development

This poster will present the results of efforts to develop a robust program for assessment of the graduate …

This poster will present how the Civil Engineering B.S. program utilized multiple sources of evidence to evaluate the student ability to apply mathematical skills in solving engineering programs and how the program has utilized results to make programmatic improvement. To assess students’ ability to apply math skills to solve engineering programs, our faculty used the results from Fundamental in Engineering license exam and course embedded assessment. In the licensure exam, we found student scores on the Structural analysis section, which involves using math to solve engineering problems, have been lower than average in the past few years. Using embedded course assessment, faculty in the course 361 Transport Engineering and 381 Structure analysis give students engineering tasks that require them to use math skills. These tasks are evaluated using a rubric. The results on the math skills show that the student performance did not meet the target (more than 80% students scoring 3 and 4’s and fewer than 5% students scoring 1) set by the department. As the students assessed were juniors and the target was set for the graduating class, the data indicate either more work needs to be done to bring students up to the desired level of performance, or a scaled scoring system needs to be used to compensate the mismatch between the expectation (target) and the status of the students assessed. The program assessment coordinator and course instructors collect and summarized the assessment results. The program faculty collaboratively interpreted the results and discussed improvement strategies. The program used the assessment results and made multiple curricular improvements, such as changing the Structure analysis course from the electives to required courses, limiting enrollment, increasing frequency of course offerings, modifying expectations on the rubric, and applying appropriate standards to students at different academic levels. We think that our program’s assessment practices can inform skill-based outcome assessment for other programs. We will also share lessons learned.

Assessing Math, Science & Engineering Skills in Civil Engineering

This poster will present how the Civil Engineering B.S. program utilized multiple sources of evidence to evaluate the …

Oral proficiency in Mandarin is one of the core language skills targeted by the Chinese program learning outcomes. This skill is primarily assessed through ACTFL Oral Proficiency Interviews (OPI), a nationally recognized language proficiency test. Given the volunteer nature of the test participation, only BA Chinese major students with more confidence have participated in it. Even though the results have shown that seniors from the Chinese B.A. program is performing above expectation, the program is not satisfied with the interview participation rate. These students contributed the reasons of low level of participation to the lack of confidence and practice. To address this need, the program coordinator collaborated with faculty and re-engineered Chinese 411, a speaking capstone course to prepare students for the OPI test in the Fall 2013. A rubric was developed for students to understand the expectations so that they can prepare with guidance. During this course, students were trained with a series of Advanced Level oral performance tasks: self-introduction, description, narration in different time frames in paragraphs, and report of current event as well as various role-plays. After each module, they produced recordings of performance in simulated interviews similar to OPI. Two assistants outside of the class independently evaluated 6 to 7 oral performance recordings and provided feedback. The result of this curricular change is that students have increased confidence in oral performance and willingness to participate in the OPI.

Improving Oral Proficiency in Chinese Using Simulated OPI Assessment

Oral proficiency in Mandarin is one of the core language skills targeted by the Chinese program learning outcomes. …

Providing standardized workshops for faculty interested in teaching in tech-rich can be challenging because faculty are at different levels of development in using technology in teaching and learning. The Technological, Pedagogical, and Content Knowledge framework is helpful because it identifies seven overlapping skill sets and competencies for teaching with technology. Applying the design based research method to faculty development creates a research platform conducive to cyclical iterations inclusive of collegial feedback. The results of three iterations of research and instructional design for faculty use of innovative classrooms, combined with student feedback will be presented.

Design Based Research Approach for Faculty Development on Innovative Classrooms

Providing standardized workshops for faculty interested in teaching in tech-rich can be challenging because faculty are at different …

After analyzing the university’s needs and researching best professional development practices, the UHM Assessment Office initiated an exciting project to build assessment leaders on campus through its inaugural Assessment Leadership Institute in summer 2013 that had 10 participants. The model features extensive support after the initial intensive 4- day institute. The model has shown great success. A longitudinal study has been planned to monitor its institutional impact.

Building and Sustaining Assessment Leaders: A Successful Model at UHM

After analyzing the university’s needs and researching best professional development practices, the UHM Assessment Office initiated an exciting …