Quantitative Reasoning: Mānoa Institutional Assessment

Multi-State Collaborative (MSC) to Advance Quality Student Learning

 

Are you teaching a course that emphasizes quantitative reasoning?

Want to be involved with institutional learning assessment at Mānoa?

Interested in using authentic student work for assessment (and not a standardized test)?


The University of Hawai'i at Mānoa Assessment Office, General Education Office, and the Institutional Learning Objectives Implementation Committee embarked on a learning assessment project to benefit Mānoa students and faculty: the Multi-State Collaborative (MSC) to Advance Quality Student Learning.


Purpose
This project supports and is in line with Mānoa’s ongoing institutional learning assessment projects. The purpose of this project is to investigate student learning and demonstrate that colleges and universities can evaluate learning without using standardized exams. It uses locally-developed instruments (i.e., authentic student work in the form of written work, exams completed as part of the curriculum, etc.) for assessment.

This project is in collaboration with thirteen other states for the Multi-State Collaborative to Advance Quality Student Learning (MSC).


Significance of Mānoa Participating in the MSC Project
Participating in the MSC project is an important undertaking for Mānoa as we continue to develop long-term institutional-level assessment strategies. The MSC project allows us to move from collecting evidence of student learning to evaluating assessment data that can be used for institutional improvement and future WASC accreditation reviews.

The process is very similar to what Mānoa faculty are already doing (e.g., it uses the VALUE rubrics that Mānoa faculty use) and promotes faculty members as drivers of the assessment process. It also offers the opportunity to help align our assessment practices with other institutions and have our student work evaluated from nationally trained faculty scorers.

This multi-state approach and effort to understanding student learning and outcomes assessment allows Mānoa faculty to engage in conversations of student performance with faculty across the nation. Additionally, the external mechanism for scoring student work will provide results that are useful in reporting our outcomes data to WASC.

Read more benefits of participating.


Who’s Participating?
Mānoa faculty and instructors who teach courses that emphasize quantitative reasoning skills are invited to participate (see the FAQ for more information regarding course and student selection).

What Do Participants Do?
For the 2017 semester, participating faculty and instructors . . .

  • Submit a completed assignment from their randomly selected students
  • Submit assignment guidelines
  • Submit a coversheet and simple matrix detailing the assignment context (download the quantitative reasoning matrix)
  • Participate in "faculty share" sessions

Where Do Participants Submit Materials?
All material will be submitted to the Assessment Office. Please email airo@hawaii.edu or send via campus mail to Crawford 230.

We will gladly scan and shred/return any hard copies for you!


What Happens with the Submitted Materials?
Trained redactors will remove all personal information (student name, faculty name, course, etc.) and a unique code will be assigned to each piece. The purpose of the redacting process is to keep all student work anonymous so that no individual student and faculty will be evaluated. All information will be stored in a secure format/locked area and electronic documents will be password protected. The material will be uploaded to a secure website. The faculty who review the student writing will do so in summer 2017 and they will not know the student names, faculty names, or campus.

Nationally trained scorers, including Mānoa faculty members, will evaluate the student work using the Quantitative Literacy VALUE rubric that Mānoa faculty already use.

How Will the Results be Used?
Only Mānoa will receive our campus results. The results will be distributed to all faculty members and instructors who participate, and various faculty groups: the General Education Committee and its Boards, the Mānoa Assessment Committee, and the Institutional Learning Objectives Implementation Committee. The results will also be used as evidence for Mānoa’s re-accreditation.

Link to 2016 results.


Benefits

  • It uses authentic student work.
  • It encourages Mānoa faculty to be the primary drivers of institutional learning assessment at Mānoa.
  • Mānoa faculty will use the results to better understand students’ level of achievement in the areas of written communication, quantitative reasoning, and ethical reasoning.
  • Only Mānoa will receive Mānoa students’ results and these results will be reported in the aggregate (that is, many campuses are collaborating but campuses receive only their results) (Read the 2016 Mānoa results.)
  • Mānoa faculty have the opportunity to assist with the scoring and participate in discussions about student performance with faculty colleagues from across the U.S.
  • Mānoa faculty will be offered training in assignment design and using rubrics to assess learning. To learn about about training opportunities and to request training, contact the Assessment Office (airo@hawaii.edu).

Other FAQ

What is “institutional-level assessment” of student learning?

Institutional-level assessment of student learning looks across many students in many majors to form an understanding of undergraduate students’ skills and knowledge. In 2015-16, Mānoa investigated written communication, critical thinking, and quantitative reasoning competencies.

In 2017, Mānoa is investigating quantitative reasoning competencies.

Why the MSC project?

First, to gain an institutional perspective of how well undergraduates as a whole are prepared. After faculty have aggregate results—across many students in many majors—institutional-level changes can be made to improve the undergraduate experience.

Second, Mānoa is required to evaluate undergraduate students achievement in five areas and use the overall results for improvement in order to remain accredited and for Mānoa students to be eligible to receive federal financial aid.

The five areas are critical thinking, information literacy, oral communication, quantitative reasoning, and written communication.

Is this a personal research project?

No. This is an institutional learning assessment project that has been approved by President David Lassner.

Does this project require IRB approval?

No. Learning outcomes assessment is not considered research that involves human subjects according to the Protection of Human Subjects regulation (section 46.102(d)). However, we do follow the regular procedures for privacy and securing of confidential information as is expected of all Mānoa employees. Students' work is redacted so that they cannot be linked back to individual student who submitted the work or the faculty member who taught the course.

How are courses and students selected?
  1. We identify potential courses based on department chair/faculty recommendations of appropriate courses and courses' general education designations. We primarily target 100-level courses and 300- and 400-level because we are looking for assignments that challenge first-year students and seniors.

  2. After identifying target courses, we create a pool of eligible, enrolled students who meet the following requirements:
    • Seeking a first undergraduate degree
    • Either (a) completed 15 or fewer credits or (b) completed 90 or more credits

  3. After identifying the pool of eligible students, we use simple random sampling and proportional random sampling to obtain a representative sample across the curriculum and across other desired characteristics (e.g., gender, ethnicity, pell grant status). That is, we take appropriate steps and measures to ensure that students are randomly sampled, we select no more than 10 students from a single faculty member, we select an appropriate proportion from each targeted course (to capture a variety of student work across the curriculum), and to ensure that only one piece of work is submitted for one student.

Do we need to obtain permission and consent from students?

Prior consent from students is not needed. This is similar to other campus efforts and regular reporting of aggregate information (e.g., students' SAT scores, students' average grade points).

However, students may opt out: see also the Catalog section, "Collection of Student Work for Assessment Purposes."

Will this affect a student’s grade or graduation?

No. Grades will not be affected. Graduation will not be affected. The faculty who will review the student work samples will do so after the course has ended. Faculty scorers will not know the student names.

How can a scorer without expertise on the work that is being scored, effectively evaluate the work?

For this institutional-level project, the MSC scoring is "not about the content of the student work, for which the faculty are indeed the best judges, but rather about the underlying abilities associated with the learning outcome. . . . The scoring is about whether the student can communicate about it in written form, oral form, etc., demonstrate critical thinking skills, ethical considerations, etc." (Terry Rhodes, Vice President, Office of Quality, Curriculum, and Assessment and Executive Director of VALUE, at AAC&U)

The faculty participating in the project state whether their assignments are aligned with the rubric.

See here for more FAQs from the Multi-State Collaborative (MSC).

Are there any risks?

No. Student names and faculty names and all personal identifiers will be removed and personal information will be kept strictly confidential to eliminate the risk of being individually identified. All information will be stored in a secure format/locked area and electronic documents are password protected. The faculty who review the student writing will do so in summer and they will not know the student names.

Can I participate even if I did not receive an invitation?

Yes, if you are teaching a spring 2017 course that emphasizes quantitative reasoning and enrolls either first-time freshmen or seniors contact the Assessment Office (airo@hawaii.edu) if you are interested.


Forms and Rubrics

Coversheet and Simple Matrix:

VALUE Rubrics:


Contact

Please direct all questions and send materials to the Assessment Office:

Email: airo@hawaii.edu
Send hard copy materials via campus mail to Crawford 230.


Mahalo on the behalf of the Assessment Office, General Education Office, and the Institutional Learning Objectives Implementation Committee.

Monica Stitt-Bergh, Ph.D.
Associate Specialist, Assessment Office

bergh@hawaii.edu

Hokulani Aikau, Ph.D.
Director, General Education Office
Associate Professor, Political Science

aikau@hawaii.edu

Amy Schiffner
Chair, Institutional Learning Objectives Implementation Committee
Associate Professor, Theatre & Dance

amyls@hawaii.edu

Updated 12/13/2016