Departmental Assessment Update - Arts and Humanities Report
Department: Speech
Program: BA
Level: Undergraduate
1. List in detail your undergraduate Student Learning Outcomes (SLOs) for each degree/certificate offered.
The Department of Speech at the University of Hawaii offers an innovative B.A. program emphasizing the central processes and functions of human communication. Current student learning outcomes are: 1) students will develop cross-situational understanding of message processing, 2) students will develop cross-situational understanding of relational communication, 3) students will develop cross-situational understanding of social influence, 4) students will develop understanding and skills in communication research methods, 5) students will develop communication skills (written and oral), and 6) students will develop critical thinking skills.
However, we are in the process of revising the student learning outcomes so that they would be more conducive to future assessment activities. We had a departmental retreat and agreed on a set of revised learning outcomes, and are in the process of finalizing them. Once finalized, they will be properly publicized.
2. Where are these SLOs published (e.g., department web page)?
Our Department web page, course catalog, flyers.
3. Explain how your SLOs map onto your curriculum, i.e., how does your curriculum produce the specific SLOs in your students?
For the content areas, we have basic required courses reflecting the three focus areas: Relational Management (Sp 381), Message Processing (Sp 470 and Sp 370), and Social Influence/Persuasion (Sp 364). We also have elective courses in the three focus areas. For critical thinking and research skills, we have a research methods course required of all majors – Sp 302. For communication skills, we have serveral public speaking/presentation courses (Sp 151, Sp 251, Sp 351), courses designated as oral intensive (Sp 333 and Sp 335), and writing intensive courses offered each semester.
4. What specific methodologies were used to collect data? In developing your response, consider the following questions:
The Department of Speech has undertaken various facets of assessment over the past three semesters. Our largest effort to date has been an assessment of Speech majors’ acquisition of fundamental content in human communication as presented in our required classes: Sp 301, Theories in Speech Communication; Sp 251, Public Speaking; Sp 302, Introduction to Inquiry; Sp 364, Persuasion; Sp 370/470 (students choose between Verbal Communication or Nonverbal Communication); and Sp 381, Interpersonal Communication.
Assessment Overview An instrument was developed that uses a multiple choice format to assess students’ knowledge of the content areas covered by our core courses. The instrument is administered in the first week of each semester to the Sp 151 (Personal and Public Speech) and Sp 251 (Public Speaking) classes. Most of these students have not been exposed to Speech department courses before. Their data serve as the base line for assessment. At the end of each semester, all graduating majors and minors are required to make an appointment to complete a variation of the same instrument administered to the Sp 151 and Sp 251 classes earlier in the semester. These data serve as the “post-experimental manipulation” data, the data that allow us to assess acquisition of course content. Instrument Development Each faculty member who teaches or has taught one of the required courses submitted approximately ten items from their test question pools for midterm and final exams in the core courses. Dr. Amy Hubbard, the previous Director of Undergraduate Studies for the Department of Speech, collected the items and developed a pool of assessment items from which each instrument is constructed. Dr. Hubbard selected approximately equal numbers of items from each content area for each instrument used at the beginning and end of the semesters. Data Analysis We have analyzed data collected through summer session 2006. Results showed significant differences in acquired content between graduating majors/minors and students who have never taken a Speech course before. Significant differences (p < .05) were found across each of the 7 required classes. In all cases, the averaged scores of the graduating Speech majors were higher than the Sp 151 students. Effect sizes ranged from lows of 6% (Sp 301 & 370) and 10% (Sp 302) to highs of 36% (Sp 381) and 34% (Sp 364). The overall effect size comparing Sp 151 students to Speech majors collapsed across classes was 45%. Interpretation Speech majors have learned and retained the materials provided in their Speech classes to a degree that is significantly greater than their counterparts who have taken no or minimal Speech classes. This finding is consistent across all of the classes required for majors. However the range of effect sizes across the scores for the individual classes indicates that the difference between majors and non-majors is greater in some classes than others. This may be due to the unique content (i.e., less redundancy) in some classes such as Sp 301 (Theories), Sp 302 (Research Methods), and Sp 370 (Verbal Communication). Also, Sp 370 may not have been taken by all of the grading majors (since they have the option to take either Sp 370 OR Sp 470 (Nonverbal Communication). Effect sizes may be related to when the classes were taken. Ideally, Sp 301 and Sp 302 are taken prior to the other required classes. Interestingly, the classes with the greatest effect sizes are classes that students typically enjoy the most.
Our second facet of assessment has focused on students’ subjective assessments of their learning and, relatedly, students’ assessment of the Department of Speech. These data are collected in an exit survey developed by Dr. Hubbard. The instrument records various demographic data; the number of Speech classes students have taken; students’ perceptions of faculty knowledge and enthusiasm; students’ beliefs about the importance and usefulness of what they learned in Speech classes; and students’ interactions with professors in Speech. Finally, students also respond to items that assess their perceptions of their learning in the various contents covered by the core courses as well as student perceptions regarding their improvement in writing skills; oral communication skills; critical thinking skills; and information search and retrieval skills. A total of 274 of these exit surveys have now been collected and we analyzed the data. The results show that students are highly satisfied with the department overall (Mean of 4.82 on a 1-5 point scale). Students assessment of various aspects of the department was also highly positive. Students were highly satisfied with various aspects of the faculty and the curriculum, with means ranging well above 4.5 on a 1-5 point scale. Students were also agreed that they have improved writing skills; oral communication skills; critical thinking skills; and information search and retrieval skills, with means ranging between 4.13 and 4.60 on a 1-5 point scale. Open-ended questions about the department also yielded very positive assessment by students.
5. How were the assessment data/results used to inform decisions concerning the curriculum and administration of the program?
The faculty met and discussed the results of the assessment analyses and improvements that could be made. For example, we concluded that the Speech Department may serve its majors better by incorporating more discussion of theory, and particularly research methods into other required courses. Specifically, instructors can offer more analysis and criticism of the research that is discussed in classes in order to reinforce the materials covered in Sp 302.
Since the results to date suggest the Speech department is performing effectively and Speech majors are graduating with a knowledge base on human communication that separates them from their peers, we decided that changes in pedagogy or administrative changes are not needed at this time. Faculty members agreed to try to incorporate more discussion of theory and research methods across courses in the Department to reinforce learning of core content in our discipline.
6. Has the program developed learning outcomes? Please indicate yes or no.
Yes.
7. Has the program published learning outcomes? Please indicate yes or no.
Yes.
8. If so, please indicate how the program has published learning outcomes.
Our Department web page, course catalog, flyers, course syllabi
9. What evidence is used to determine achievement of student learning outcomes?
The Department of Speech has undertaken various facets of assessment over the past three semesters. Our largest effort to date has been an assessment of Speech majors’ acquisition of fundamental content in human communication as presented in our required classes: Sp 301, Theories in Speech Communication; Sp 251, Public Speaking; Sp 302, Introduction to Inquiry; Sp 364, Persuasion; Sp 370/470 (students choose between Verbal Communication or Nonverbal Communication); and Sp 381, Interpersonal Communication.
Assessment Overview An instrument was developed that uses a multiple choice format to assess students’ knowledge of the content areas covered by our core courses. The instrument is administered in the first week of each semester to the Sp 151 (Personal and Public Speech) and Sp 251 (Public Speaking) classes. Most of these students have not been exposed to Speech department courses before. Their data serve as the base line for assessment. At the end of each semester, all graduating majors and minors are required to make an appointment to complete a variation of the same instrument administered to the Sp 151 and Sp 251 classes earlier in the semester. These data serve as the “post-experimental manipulation” data, the data that allow us to assess acquisition of course content. Instrument Development Each faculty member who teaches or has taught one of the required courses submitted approximately ten items from their test question pools for midterm and final exams in the core courses. Dr. Amy Hubbard, the previous Director of Undergraduate Studies for the Department of Speech, collected the items and developed a pool of assessment items from which each instrument is constructed. Dr. Hubbard selected approximately equal numbers of items from each content area for each instrument used at the beginning and end of the semesters. Data Analysis We have analyzed data collected through summer session 2006. Results showed significant differences in acquired content between graduating majors/minors and students who have never taken a Speech course before. Significant differences (p < .05) were found across each of the 7 required classes. In all cases, the averaged scores of the graduating Speech majors were higher than the Sp 151 students. Effect sizes ranged from lows of 6% (Sp 301 & 370) and 10% (Sp 302) to highs of 36% (Sp 381) and 34% (Sp 364). The overall effect size comparing Sp 151 students to Speech majors collapsed across classes was 45%. Interpretation Speech majors have learned and retained the materials provided in their Speech classes to a degree that is significantly greater than their counterparts who have taken no or minimal Speech classes. This finding is consistent across all of the classes required for majors. However the range of effect sizes across the scores for the individual classes indicates that the difference between majors and non-majors is greater in some classes than others. This may be due to the unique content (i.e., less redundancy) in some classes such as Sp 301 (Theories), Sp 302 (Research Methods), and Sp 370 (Verbal Communication). Also, Sp 370 may not have been taken by all of the grading majors (since they have the option to take either Sp 370 OR Sp 470 (Nonverbal Communication). Effect sizes may be related to when the classes were taken. Ideally, Sp 301 and Sp 302 are taken prior to the other required classes. Interestingly, the classes with the greatest effect sizes are classes that students typically enjoy the most.
Our second facet of assessment has focused on students’ subjective assessments of their learning. These data are collected in an exit survey developed by Dr. Hubbard. The instrument records various demographic data; the number of Speech classes students have taken; students’ perceptions of faculty knowledge and enthusiasm; students’ beliefs about the importance and usefulness of what they learned in Speech classes; and students’ interactions with professors in Speech. Finally, students also respond to items that assess their perceptions of their learning in the various contents covered by the core courses as well as student perceptions regarding their improvement in writing skills; oral communication skills; critical thinking skills; and information search and retrieval skills. A total of 274 of these exit surveys have now been collected and we analyzed the data. The results show that students are highly satisfied with the department overall (Mean of 4.82 on a 1-5 point scale). Students assessment of various aspects of the department was also highly positive. Students were highly satisfied with various aspects of the faculty and the curriculum, with means ranging well above 4.5 on a 1-5 point scale. Students were also agreed that they have improved writing skills; oral communication skills; critical thinking skills; and information search and retrieval skills, with means ranging between 4.13 and 4.60 on a 1-5 point scale. Open-ended questions about the department also yielded very positive assessment by students.
10. Who interprets the evidence?
The departmental assessment committee is responsible for implementing, analyzing outcomes, and interpreting the analysis results. Dr. Hye-ryeon Lee heads the assessment committee.
11. What is the process of interpreting the evidence?
The above described assessment components are objective and quantitative. We use Statistical Package for Social Sciences (SPSS) to conduct quantitative analyses of the data with appropriate statistical tests. Then, results are presented to all the faculty for a review. All faculty participates in a discussion on the implications of the results as well as the course of action to be taken in response to the results.
12. Indicate the date of last program review.
Summer 2000