Project Nutshell:
We collected 79 eligible senior students’ oral presentations from 300 and 400 level classes or events that focus on oral communication (OC). We randomly sampled course sections from spring and fall 2017. We also included students oral presentations given at the spring 2017 College of Tropical Agriculture and Human Resources and College of Engineering Research Symposium. We will trained and calibrated raters to use the UH Mānoa adapted VALUE rubric to evaluate video recordings of student presentations. We provided resources and support for OC instruction along the way through training and resources. Assessment Office, General Education Office, and Oral Communication Focus Board discussed the results and acted upon the results for improvement of OC teaching and learning.

Who were invited to participate:
Mānoa faculty and instructors who taught OC focus designated courses or emphasize oral communication in their courses were invited to participate.

What did the participants do?
Participating faculty and instructors . . .

• Video-recorded students’ live and formal oral presentations. Submitted the recordings for the eligible students who met the sampling criteria (i.e., senior-standing and first-time degree seeking). The Assessment Office provided the list of eligible students to the instructors. The Assessment Office also provided video-recording for most of the participating instructors.
• Oral presentation assignments needed to meet the collection criteria, as follows:
  • Length: 5-20 minutes;
  • Purpose: to persuade or inform;
  • Language: English;
  • Specify intended audience; and
  • Includes supporting material (e.g., evidence, primary/secondary sources to support claims, graphs, charts, materials, visual aids to support presentation)
• Participants were encouraged to invite an Assessment Office staff member to attend their classes and record students’ presentations.

Example assignments include research/project presentations; analysis reporting; planning/implementation proposals to intended audience; art, culture, or social/natural phenomena/artifacts informational presentation to intended audience; and so on.

See our Resources for video recording tips (e.g., reserving equipment, filming, exporting and sending video)

Where did the participants submit materials? 
Participants submitted videos using video-recording media (e.g., USB, CDs) to the Assessment and Curriculum Support Center via mail (Crawford 231) or email (airo@hawaii.edu). They could also securely email the media file through UH File Drop service: www.hawaii.edu/filedrop.

What happened with the submitted recordings/Media?
Trained video editors edited the recordings to increase visual and audio quality (e.g., increase/decrease volume, stabilize image, alter lighting, etc.). Assessment Office added a confidentiality statement in the video. Each recording was assigned a unique code.

The materials were uploaded to a secure scoring service site: Aqua. The faculty who evaluated the recordings of student presentations were not given student names or faculty names. As students and faculty are identifiable, all editors and scorers (i.e., anyone with access to the recordings) were asked to read and sign a confidentiality agreement.

FAQs

What OC assessment work has been done?

• 2007: Faculty group created OC learning outcomes and rubric
• 2009: Faculty group gave feedback on the rubric
• 2011: Assessment Office (AO) survey: key finding about OC learning after 1st year:
  • 75% reported learning something important during their first year “The most important thing I learned about oral communication is how much practice is necessary in performing a good speech”—first-year student
• 2013: AO survey: key finding about OC learning after 2nd year:
  • Students who completed an OC Focus class reported learning more about OC than those who did not
• 2015: AO and the General Education Office (Gen Ed) completed a “standard setting” activity with faculty who taught OC Focus classes. The standard is 2.4 on the VALUE rubric.
• 2016: The Gen Ed Director and the OC board developed a signature assignment and had faculty pilot it. AO, Gen Ed, and Institutional Learning Outcomes Implementation Committee (ILOIC) facilitated an assignment design workshop in two sessions with 7 faculty attendees.
• 2017: AO collected 41 usable recordings of student presentations from five subject areas and CTAHR Research Symposium.

What were we collecting? What was the criteria for acceptable presentations?

We collected video-recordings of student oral presentations that meet the following criteria:

• Length: 5-20 minutes;
• Purpose: to persuade or inform;
• Language: English;
• Specify intended audience; and
• Includes supporting material (e.g., evidence, primary/secondary sources to support claims, graphs, charts, materials, visual aids to support presentation)

Example assignments include research/project presentations; analysis reporting; planning/implementation proposals to intended audience; art, culture, or social/natural phenomena/artifacts informational presentation to intended audience; and so on.

Why were you not collecting other oral communication assignments? Why only formal presentations?

We recognized there are many forms of oral communication, such as group discussions, interpersonal conversations, other forms of presentations, etc. They are all very important. However, to make a fair comparison of student oral performance, we selected a format that is most commonly used across disciplines and that is supported by documented/existing research for assessment purposes: formal presentations.

Also,

• UH Mānoa has already accomplished developing an evaluation rubric (adapted from the oral communication VALUE rubric) and setting standards for oral presentation performance.
• Our previous work revealed that faculty struggle evaluating presentations of different formats (e.g., live versus recorded, facing an audience versus the camera, 3 minutes versus 30 minutes). Thus, we are beginning the project with only formal oral presentations that meet specific requirements.

How did this project impact individual faculty or students?

We understood that many institutional factors contribute to student learning: the educational policy, the curriculum, the learning environment, the pedagogy, and so on. It takes a collaborative effort to help students learn. Institutional-level assessment is a critical tool to identify educational system issues at UH Mānoa. It was not be used to evaluate individual courses, faculty, or individual students. It has no impact on individual students’ grades/GPA.

All results were aggregated in our reports. Instructors who participated received both the aggregated results and the results from the students in their own classes.

Who were able to participate in this project?

• All instructors that taught OC focus designated courses
• All instructors that required students to give an oral presentation that meet our criteria. Contact the Assessment Office (956-4283, airo@hawaii.edu)

What did the instructors need to do?

• Checked to see whether their assignment had the characteristics of the assignments that we collected.
• The instructor received a list of eligible students.
• They sent us the video recordings of the eligible students or had an Assessment Office staff member come to the class and do the recording of the eligible students.

Did they need to gain students’ consent to participate in the project?

Prior consent from students was not needed. This was similar to other campus efforts and regular reporting of aggregate information (e.g., students’ SAT scores, students’ average grade points). However, students could have opted out: see also the Catalog section, “Collection of Student Work for Assessment Purposes.“

We encouraged faculty to inform eligible students of the project. They were able to adapt our sample information letter.

Did this project require IRB approval?

No. Learning outcomes assessment is not considered research that involves human subjects according to the Protection of Human Subjects regulation (section 46.102(d)). However, we did follow the regular procedures for privacy and securing of confidential information as is expected of all Mānoa employees. Anyone with access to the recordings (editors, scorers) must read and sign a consent form.

How were courses and students selected?

1. We identified all courses taught in the spring and the fall 2017 semesters that had the OC designation.
2. We used a stratified random clustered sampling technique to randomly select courses within each college with course enrollment proportionally matched to the enrollment of upper division students at the college. For example, the College of Social Sciences enrolls 13% of upper division students. If our target total sample size is 300, we needed 39 (= 300 x 13%) eligible students from Social Sciences. Then we randomly added one course to our selection list at a time until the total number of enrolled eligible students reached our target sample size for the college. After a course was selected, we avoided selecting another section of the same course, or another course taught by the same instructor (unless there was no other choice.)
3. After identifying target courses, we identified a pool of eligible, enrolled students. Eligible students met the following requirements:
   • Degree-seeking
   • Completed 90 or more credits
4. After identifying the pool of eligible students, we used simple random sampling to obtain a representative sample across the curriculum and across other desired characteristics (e.g., gender, ethnicity, pell grant status). That is, we took appropriate steps and measures to ensure that students were randomly sampled and to ensure that only one piece of work was submitted for one student.

Was this a personal research project?

No. This was an institutional learning assessment project for the institution to investigate and improve educational quality and effectiveness.

Were there any risks?

There was a minimum risk — everyone who has access to the video recording material were required to sign a consent form.

Could faculty have participated even if I did not receive an invitation?

Yes, if faculty were teaching a course that emphasized oral communication skills and enrolled seniors.