Summative evaluation of oral exam culminating experience by student performance on individual SLOs was used as input to new ePortfolio culminating experience, which was implemented for students entering Fall 2018.

Faculty collaborated on the new entering and exiting student seminars where students identified artifacts and wrote reflective essays demonstrating their mastery of each program SLO.  Works in progress were also discussed at faculty meetings, and every faculty member evaluated every ePortfolio, for performance in each of the six SLOs for the first group of graduating studnets under this new program in Spring 2020.

Additionally, faculty were asked to report SLO assessment criteria in their courses via a simple spreadsheet.  For each major assignment that covered a specific SLO, they reported how many students exceeded, met, approached or did not meet the criteria.

All students submitted evidence in the form of coursework and/ror the required culminating experience, whether the previous oral exam or the current ePortfolio.  The total number of students in the LIS Program between Fall 2018 and Fall 2020 is 98.

As a result of our assessment process, the faculty developed the following statement on assessment philosophy and proocess: 

Assessment

Philosophy and process

unanimously updated and reaffirmed by LIS faculty 28 August 2020

Overview:​​ Not all of our strengths can be captured in numbers.  For program-level assessment, we prefer an approach where many voices can be heard and considered, over a longer period of time than a single semester.  When issues arise with a course, instructor or student, those are addressed individually by the LIS chair.  When issues persist and recur across multiple courses and semesters, then we feel we are justified in claiming an area of strength, or in devoting resources to addressing areas where improvement is needed.

Lived experience:​​  The most organic form of assessment is our lived experience, where we enact our values of aloha, ʻohana and kuleana.  While it would be inappropriate to attempt to quantify these experiences, our graduating student surveys tell us that the sense of community within the LIS Program is by far what our students value most.  As part of our teaching, research, service and informal interactions, we share our experiences throughout the year, documented in meeting minutes and year-end reports.

Critical refusal: While acknowledging the usefulness of assessment metrics for some purposes, our philosophy also encompasses critical refusal of harmful data regimes, as summarized in the Feminist Manifest-No.

Course evaluations: ​​As a formative assessment of student learning and faculty instruction, each student evaluates each course in terms of content, workload, delivery and assessment methods, and instructor effectiveness.  Numerical and anecdotal data are collected by the Program Coordinator and reviewed by the chair each semester.  Instructors whose evaluations are lower than 4/5 meet with the chair to debrief.  Adjuncts who consistently fail to reach the 4/5 standard, or who receive very low ratings, are not invited back to teach.

Course SLO assessments:​​ As a formative assessment of program SLOs, each instructor identifies the number of students who exceeded, met, approached or did not meet the standards associated with the primary SLO of the course via a major assignment, or a set of related assignments.  This is a more direct, formative measure of SLO mastery than an overall course grade.  The chair collects and reviews these assessments at the end of each semester (adjunct faculty) or academic year (full-time faculty), and summarizes compiled student achievement by SLO back to the faculty in the following year’s planning session.

Culminating experience assessments:​​ As a summative assessment of program SLOs, students taking the oral comprehensive exam respond to four scenarios associated with the 6 program SLOs.  The average of two faculty members’ scores must be above 5/7 on every scenario in order to pass. Similarly, students completing the ePortfolio produce an artifact and reflective essay associated with each SLO, which is scored by two faculty members.  In both cases, results are aggregated by the chair and reported back to the faculty as a summative measure of student achievement by SLO.

Graduating student surveys:​​ A summative assessment of the overall student experience, given each semester to graduating students

Alumni surveys:​​ A reflective assessment of student experience and professional skills needed and acquired administered every 5 years.

Employer surveys:​​ A merged assessment of UHM LIS alumni skills, job performance and needed skills within each employer organization, administered every 5 years.

Committee work and event documentation: ​​Each faculty member leads one or more standing committees, task forces or initiatives.  As a formative assessment of student experience, stakeholder engagement and faculty service, each faculty member summarizes their groups’ process, student and stakeholder engagement and accomplishments in a year-end report.

Year-end meeting.​​  To contextualize assessment data and to systematically collect and compile stories, reflections and any other issues, the LIS Associate Chair collects year-end reports and meets with each faculty member, either in person or virtually, at the end of the academic year, to review and elicit their reflections on:

• Their course SLO assessments, to contextualize the numerical data and discuss any opportunities or challenges with their courses.

• Their year-end report of committee work, stakeholder engagement, metrics etc.

• Their assessment of the performance of the LIS Chair, Program Coordinator and staff.

The Associate Chair summarizes faculty discussions and reflections (any faculty comments, critical or otherwise, may be anonymized at the discretion of the faculty member providing them), and discusses the results in the last meeting of the year, with the LIS Chair.  The LIS Chair then integrates those data and discussions with course evaluation and survey data over the summer, to serve as input to next year’s planning meeting.

Each step of our curriculum assessment process is linked with our Student Learning Outcomes (SLOs) and Program Goals. SLOs 1-4 directly link to Program Goal 1 (Provide a curriculum that meets the evolving demands of the job market for librarians and other information professionals), and SLO 5 links with Program Goal 5 (Strengthen the emphasis on Hawaiʻi and the Asia-Pacific region in teaching, research, and service). Every course is aligned with the particular SLOs it covers via a matrix.

For the culminating experience, the ePortfolio and oral exam scenarios are also linked with our SLOs, and students are evaluated along the same dimensions (exceed-meet-approach-do not meet). We use this data to revise both the curriculum and the oral exams themselves.

Thesis students are assessed on specific elements of their research, including their problem statement, literature review, method, results, and the quality of their written thesis and oral defense; the evaluation form can be found on p. 7 of our thesis policy and FAQ.