Unit: Social Work
Program: Social Work (MSW)
Degree: Master's
Date: Fri Nov 16, 2018 - 12:45:19 pm

1) Program Student Learning Outcomes (SLOs) and Institutional Learning Objectives (ILOs)

1. Demonstrate ethical and professional behavior.

(5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

2. Engage diversity and difference in practice.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

3. Advance human rights and social, economic, and environmental justice.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

4. Engage in practice-informed research and research-informed practice.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 2. Demonstrate understanding of research methodology and techniques specific to one’s field of study., 3. Apply research methodology and/or scholarly inquiry techniques specific to one’s field of study., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

5. Engage in policy practice.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

6. Engage with individuals, families, groups, organizations, and communities.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

7. Assess individuals, families, groups, organizations, and communities.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 2. Demonstrate understanding of research methodology and techniques specific to one’s field of study., 3. Apply research methodology and/or scholarly inquiry techniques specific to one’s field of study., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

8. Intervene with individuals, families, groups, organizations, and communities.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

9. Evaluate practice with individuals, families, groups, organizations, and communities.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 2. Demonstrate understanding of research methodology and techniques specific to one’s field of study., 3. Apply research methodology and/or scholarly inquiry techniques specific to one’s field of study., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

10. Engage, honor, and respect indigenous culture towards decolonized professional practice.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

2) Your program's SLOs are published as follows. Please update as needed.

Department Website URL: https://www.hawaii.edu/sswork/msw.html
Student Handbook. URL, if available online: https://www.hawaii.edu/sswork/forms/msw/MSW_Student_Handbook.pdf
Information Sheet, Flyer, or Brochure URL, if available online: https://www.hawaii.edu/sswork/bulletin.html
UHM Catalog. Page Number: http://www.catalog.hawaii.edu/schoolscolleges/socialwork/SocialWork.html#grad
Course Syllabi. URL, if available online:
Other:
Other:

3) Please review, add, replace, or delete the existing curriculum map.

Curriculum Map File(s) from 2018:

4) For your program, the percentage of courses that have course SLOs explicitly stated on the syllabus, a website, or other publicly available document is as follows. Please update as needed.

0%
1-50%
51-80%
81-99%
100%

5) Does the program have learning achievement results for its program SLOs? (Example of achievement results: "80% of students met expectations on SLO 1.")(check one):

No
Yes, on some(1-50%) of the program SLOs
Yes, on most(51-99%) of the program SLOs
Yes, on all(100%) of the program SLOs

6) Did your program engage in any program learning assessment activities between June 1, 2015 and October 31, 2018?

Yes
No (skip to question 17)

7) What best describes the program-level learning assessment activities that took place for the period June 1, 2015 to October 31, 2018? (Check all that apply.)

Create/modify/discuss program learning assessment procedures (e.g., SLOs, curriculum map, mechanism to collect student work, rubric, survey)
Collect/evaluate student work/performance to determine SLO achievement
Collect/analyze student self-reports of SLO achievement via surveys, interviews, or focus groups
Use assessment results to make programmatic decisions (e.g., change course content or pedagogy, design new course, hiring)
No (skip to question 17)
Investigate other pressing issue related to student learning achievement for the program (explain in question 7)
Other:

8) Briefly explain the assessment activities that took place.

We have continuted to conduct our assessment activities to measure the degree to which our ten Student Learning Outcomes (as listed above #1) have been met throughout the curriculum over the course of the academic year.  We use both direct evidence (instructor evaluation of student products such as reports, presentations, classroom activities, and skill demonstration as measured through the use of the Instructor Evaluation Sheet (IES) and the Field Supervisory Evaluation form (FSE)) as detailed below, and indirect evidence (student self-report as measured through the use of the Student Self-Assessment Survey Instrument (SSASI)).  

We have a standing Assessment Committee - made up of program chairs, the directors of both student services and distance education as well as the Department Chair who functions as Assessment Committee Chair - that collects, analyzes and evaluates SLO data through the IES, FSE, and SSASI, presents results for program decision-making and pedagogical/course content change.  Our assessment results have directly impacted program implementation and performance.  

The current MSW assessment includes direct measures from two data sources utilized by both program options (Campus-based & Distance Education).  The following details the description of two measures used to assess two dimensions of each of the 10 SLOs using a real or simulated practice situation.  The plan also lays out the assessment procedures for each of the measures.

We established both a 5-point Likert scale (fully detailed below) as well as an acquisition benchmark for all SLOs.  Taken in aggregate, the expectation is that 80% of all students surveyed in the MSW Program will score a 4 or higher on a 5-point Likert scale.  As detailed below, a score of 4 on our 5-point Likert scale represents near consistency of competency demonstration.  We recognize that students have varying skills upon entry to the Program and throughout their tenure, and will continue to learn and grow long after their formal educational experience is complete.  The near consistent demonstration of social work competencies is critical if our graduates hope to be legitimate helpers in the community.  Ideally, all of our graduates would score 5 on the 5-point Likert scale and in the estimation of both our Field and classroom instructors, most of our students do in fact demonstrate consistency.  

We also decided that each competency will assess acquisition of the skill dimension for each competency using the FSE (Measure 1).  The IES (Measure 2) would assess one of the other dimensions (knowledge, value, cognitive & affective processing) for each competency as determined by Sequence (HBSE, Generalist Practice, Policy, Research) and Specialization (Behavioral Mental Health, Child & Family, Gerontology, Health) Chairs. 

Assessment Measurement

The attainment of each SLO through specific articulated dimensions is assessed using the following two measures:

(1) Field Supervisory Evaluation (FSE): Field Instructor ratings of the skill dimension for each SLO through demonstration of behaviors upon the student’s completion of foundation/specialization field experience and;

(2) Instructor Evaluation Sheets (IES): Classroom instructor ratings of another dimension (knowledge, values, cognitive & affective processing) of each SLO based on a signature assignment. 

Measure 1: Field Supervisory Evaluation (FSE)

The Field Supervisory Evaluation (an online survey format) was selected as the instrument to assess the skill dimension of each SLO and related behaviors. Field education, the signature pedagogy in social work education, provides students the opportunity to draw upon theories and practice skills they learn in the classroom and apply them to the real practice situations.  Field Instructors assess their respective students on attainment of the skill dimension of each SLO through demonstration of related behaviors. 

Rating Scale

Using the FSE online survey, field instructors rate students on their demonstration of the skill dimension of each SLO and its related behaviors.  The following 5-point Likert scale was developed and approved by the Assessment Committee: 

(1) Not Yet Emerging (This skill of the behavior has not yet emerged in the student’s repertoire.)

(2) Emerging (This skill of the behavior is beginning to emerge in the student’s repertoire.)

(3) Developing (The student is developing this skill of the behavior.)

(4) Approaching Consistency (The student is approaching consistent demonstration of/engagement in this skill of the behavior; consistency is anticipated in the near future.)

(5) Consistently Demonstrates (The student consistently demonstrates/engages in this skill of the behavior.) 

Procedures

  1. The FSE was developed into an online survey using Surveyshare, an online survey application. The FSE lists all skills of the behaviors under each SW competency.
  2. Field Instructors receive an email from the Assessment Chair with instructors on how to complete the online survey.  A hyperlink to the survey is provided in the e-mail invitation.  Field Instructors (FIs) receive training on the instrument by the Field Education Office and the Assessment Committee Chair.  
  3. Field Instructors rate their respective student(s) on each behavior upon the completion of her/his MSW foundation/MSW specialization.  The process is repeated for Field Instructors who have more than one student.
  4. Completed FSE is due at the same time as the University’s grade submission deadline (the end of the generalist or the end of the specialization field education).  A reminder e-mail is sent to Field Instructors who have not completed the survey by the due date. 
  5. The FSE ratings are stored in the online survey application and used for analysis and reporting.

 

Measure 2: Classroom Instructor Evaluation Sheet (IES)

Instructor Evaluation Sheet was designed to assess student’s demonstration of one of three dimensions (knowledge, values, cognitive & affective processing) of each SLO by the classroom instructor.  Each instructor completes the appropriate form for their respective class(es) according to the directions provided by the Assessment Chair.

In each IES form, the top row is used to identify students by number.  The instructor rates each student, on a 5-point Likert scale, on the degree to which the student has achieved a specific SLO through a selected dimension (see below for a detailed explanation of the Likert scale).

Rating Scale

Using the IES form, classroom instructors rate students on their level of proficiency in demonstrating a dimension of a SWC using a 5-point Likert scale developed and approved by the Assessment Committee: 

(1) Not Yet Emerging (This dimension of the competency has not yet emerged in the student’s repertoire.)

(2) Emerging (This dimension of the competency is beginning to emerge in the student’s repertoire.)

(3) Developing (The student is developing this dimension of the competency.)

(4) Approaching Consistency (The student is approaching consistent demonstration of/engagement in this dimension of the competency; consistency is anticipated in the near future.)

(5) Consistently Demonstrates (The student consistently demonstrates/engages in this dimension of the competency.) 

Procedures

  1. The IES forms are stored on Laulima (i.e., the University of Hawai‘i’s online communication system which allows to distribute/receive materials). 
  2. All instructors are informed of the location and access to IES forms that are specific to their courses and their use at MSW Program meetings and adjunct faculty training sessions.  Instructors are also informed on the importance of being cognizant of their ratings on the IES for each student, which may not match her/his class grade on the signature assignment.  This is because the grade for an assignment includes factors such as organization of paper, grammar, met the deadline for the paper or project.  While these factors are important inclusions in grading an assignment, they may not be necessary factors to consider when assessing demonstration of a dimension related to a SLO measured in the respective course(s). The evaluation of the SLOs through respective dimensions was designed to be independent from the grading process.  
  3. Instructors receive email reminders from the Assessment Chair with detailed information on how to access the Instructor Evaluation Sheet (IES).
  4. Instructors were instructed to download their respective IES to their computers/laptops.
  5. Instructors use the IES to assess their students’ achievement on SLOs through respective dimensions relevant to their course(s).
  6. Instructors complete the IES and submit to the Assessment Chair at the same time grades are due at the end of each semester.

 

9) What types of evidence did the program use as part of the assessment activities checked in question 7? (Check all that apply.)

Artistic exhibition/performance
Assignment/exam/paper completed as part of regular coursework and used for program-level assessment
Capstone work product (e.g., written project or non-thesis paper)
Exam created by an external organization (e.g., professional association for licensure)
Exit exam created by the program
IRB approval of research
Oral performance (oral defense, oral presentation, conference presentation)
Portfolio of student work
Publication or grant proposal
Qualifying exam or comprehensive exam for program-level assessment in addition to individual student evaluation (graduate level only)
Supervisor or employer evaluation of student performance outside the classroom (internship, clinical, practicum)
Thesis or dissertation used for program-level assessment in addition to individual student evaluation
Alumni survey that contains self-reports of SLO achievement
Employer meetings/discussions/survey/interview of student SLO achievement
Interviews or focus groups that contain self-reports of SLO achievement
Student reflective writing assignment (essay, journal entry, self-assessment) on their SLO achievement.
Student surveys that contain self-reports of SLO achievement
Assessment-related such as assessment plan, SLOs, curriculum map, etc.
Program or course materials (syllabi, assignments, requirements, etc.)
Other 1:
Other 2:

10) State the number of students (or persons) who submitted evidence that was evaluated. If applicable, please include the sampling technique used.

Students are assessed for skill acquisition in both the classroom and field (2 points) as explained in this assessment report (question #8).  Additonally, we assess across at least two of four dimensions including knowledge, values, cognitive/affective processes and skills.  For our accreditation, skill assessment is mandatory and must be assessed in practice (field) with real clients and not be simulated or evalauated by proxy.  100% of the 213 students in the program were assessed in the classroom and the field.  

213/213 students assessed

11) Who interpreted or analyzed the evidence that was collected? (Check all that apply.)

Course instructor(s)
Faculty committee
Ad hoc faculty group
Department chairperson
Persons or organization outside the university
Faculty advisor
Advisors (in student support services)
Students (graduate or undergraduate)
Dean/Director
Other:

12) How did they evaluate, analyze, or interpret the evidence? (Check all that apply.)

Used a rubric or scoring guide
Scored exams/tests/quizzes
Used professional judgment (no rubric or scoring guide used)
Compiled survey results
Used qualitative methods on interview, focus group, open-ended response data
External organization/person analyzed data (e.g., external organization administered and scored the nursing licensing exam)
Other:

13) Summarize the results of the assessment activities checked in question 7. For example, report the percentage of students who achieved each SLO.

All SLOs (i.e., SW competencies) met the 80% benchmark except for SLO 1in the generalist year for the Campus-based option.  In our Specializations, the Behavioral Mental Health (BMH) data for the campus option did not meet the benchmark for SLOs 3, 9, and 10.  SLOs 3 and 10 were approaching the benchmark. However, demonstration of knowledge of SLO 9 for the instructor assessment and skill for the Field Instructor evaluation were both noticeably low.  It’s an area that needs attention for the BMH specialization. For Child & Family, combined option data showed that the benchmark was met in all competencies and their dimensions.  However, demonstration of knowledge on SLO 1 for the Campus-based fell a little short of the benchmark.  The data on the Gerontology specialization showed 8 out of 10 competencies met the benchmark.  Areas that approached the benchmark included SLOs 3 and 7.  The Health specialization data also showed that 8 out of the 10 benchmarks were met.  SLOs 3 and 4 fell short of the benchmark. For three of the specializations (BMH, Gerontology, Health), SLO 3 did not reach the benchmark and thus warrants further attention. 

For full results please see: https://www.hawaii.edu/sswork/wp-content/uploads/AS4-Forms-MSW-Program-Assessment-of-Student-Learning-Outcomes_2018.pdf

14) What best describes how the program used the results? (Check all that apply.)

Assessment procedure changes (SLOs, curriculum map, rubrics, evidence collected, sampling, communications with faculty, etc.)
Course changes (course content, pedagogy, courses offered, new course, pre-requisites, requirements)
Personnel or resource allocation changes
Program policy changes (e.g., admissions requirements, student probation policies, common course evaluation form)
Students' out-of-course experience changes (advising, co-curricular experiences, program website, program handbook, brown-bag lunches, workshops)
Celebration of student success!
Results indicated no action needed because students met expectations
Use is pending (typical reasons: insufficient number of students in population, evidence not evaluated or interpreted yet, faculty discussions continue)
Other:

15) Please briefly describe how the program used the results.

After the assessment data were gathered and analyzed in aggregate at the program level, the results were presented at the MSW program committee meeting and at the adjunct faculty training sessions at the beginning of the following school year. The MSW Program committee includes the MSW and BSW Program Chairs, Sequence Chairs (HBSE, Policy, Research, Practice), Specialization Chairs (BMH, C&F, Gerontology, Health), Director of DE, MSW Field Coordinator, MSW student representative, and the Director of Student Services.  Other faculty and staff of the department are also welcomed to attend the MSW program committee meetings.  

The presentation of the data of the previous year was shared in aggregate form indicating the percentage who have attained the benchmark.  A further examination into the data included, but not limited to (1) reviewing and comparing results from each measure to check for inconsistencies and consistencies, and (2) reviewing different sections of each course on discrepancies in rating. The presentation was followed by a discussion of the findings with the goal of developing recommendations and actions (i.e., how we use the assessment results).  

Based on the assessment results from AY 2015-2016, several significant changes were made especially regarding the assessment procedures and the instructions for the IES and FSE implementation.  Please see below for the details.  

One issue to overcome was an inconsistent return rate by adjunct faculty for the IES. Low return rate is a problem many researchers grapple with on a daily basis and we were no exception (it should be noted that through the persistent efforts of the Assessment Committee, the MSW Program has always been able to complete the assessment process in total). Many adjunct faculty either failed to complete the form or completed it, but failed to submit it.  The Assessment Committee surmised that adjunct faculty failed to recognize the email notifications because they were sent by the Assessment Chair with whom they had limited contact.  The Assessment Chair instead created a separate email account with title “MBT Department of Social Work Assessment (swassess@hawaii.edu) to increase Department and Program recognition and serve as a cue for attention.  Additionally, the Program Chair were used to follow up with all teaching faculty, again utilizing name recognition as a cue for attention. Finally, the Assessment office got the IES out earlier each semester so that faculty could begin assessing students on demonstration of the dimension of a SLO that was linked to the respective course and assignment.

Another issue was on the consistent administration of the IES and FSE on the part of the faculty.  Early in the developmental stages of the two instruments, discussion revolved around the measurement itself, in particular, the Likert scale.  Were faculty evaluating students on a fixed scale with 1 being a complete neophyte with little to no capacity and 5 being a seasoned veteran, much like those teaching their courses or leading them in the field? As one might imagine the results would vary dramatically based on the assumptions made about the measure itself. As one faculty member noted “No one in my class scored higher than a 3.  How could they, then there would be no room to grow.”  While another faculty stated, “Most of my students are doing quite well considering their year in the program.  I rated most at 4, with many 5s!”  These two responses illustrate the problem.  If the school sets a benchmark of 80% of students scoring 4 or more on the survey tool, then there is no way they can be rated against a standard other than their place in the program in any given year.  

To confront the issue, MSW and BSW Program Chairs provided detailed instruction at each program committee meetings and adjunct faculty training sessions.  As a continued effort to resolve the issue, the department produced a video on assessing student competencies (i.e., SLOs) and effective competency measurement in the spring of 2017, and it was shown as part of the biannual adjunct faculty orientation.  The aim of the video is to: 1) clarify the IES as an assessment tool, 2) decouple competency/SLO assessment from grading, and 3) address some of the common pitfalls in the assessment process.  We have found over the years that faculty will sometimes assess students against the understandable yet unrealistic standard of the seasoned social worker.  Against such a standard, it is nearly impossible for a student to be anything but substandard.  Instead, we want faculty to assess social work competencies based on where the student is at in the academic and professional development.  The question is not “How does this student stack up against a real or imagined seasoned social worker?” but rather, “How is this student performing relative to their position as a generalist year (or specialist year) student?”  This distinction is critical and the single biggest pitfall we face each semester. 

Here is a link to the video that the MSW and BSW programs created to help course instructors use the IES and appropriately score student performance.  It can be seen here: https://www.youtube.com/watch?v=RCNhGtJ1quQ&feature=youtu.be

Additionally, the MSW Field Office met with Field Instructors to go over evaluation procedures.  Taken together, these efforts greatly reduced the number of faculty who were rating students on a fixed scale and against a near unachievable standard.

In addition to assessing competency development, the program also assesses key program functions including student services and the field experience.  Results from student services assessment revealed a want on the part of our students for improved professional advising (in addition to academic advising already being received).  The field assessment process is changing as a result of our current efforts.  In the spring of 2015 the assessment committee met with field office faculty and together created a three-dimensional (so called 360o) assessment tool across student, field instructor/site, and field office, through which the student assesses both the field instructor/site and field office; the field office assesses both the student and the field instructor/site; and the field instructor in turn assesses the student and the field office.  

The School Indigenous Affairs Committee created an additional SLO/Core Competency with four associated practice behaviors several years ago and in the summer of 2015, the program chairs of both the BSW and MSW programs added the SLO/core competency to the curriculum.  This came on the heels of a much larger program initiative, the transition to the 2015 Educational Policy and Accreditation Standards (EPAS).

In the Summer of 2018, the MSW Program at the Myron B. Thompson School of Social Work was notified by the Council on Social Work Education (CSWE) Commission on Accreditation (COA) that it was reaccredited for the maximum allowable eight (8) years without condition or contingency.    

Compared to assessment results from previous years, the AY 2016-2017 assessment results require the program to examine the curriculum in light of the data given the variation between the years.  Here are some of the questions for the program committee to discuss when reviewing the results and planning actions on the use of the results: (1) what areas have consistent results and areas that do not?; (2) is there a possibility of a cohort effect?; (3) is it a pattern based on instructors or the type of instructor (i.e., in-house faculty vs. adjunct faculty)?; (4) are their discrepancies of results by measures?; (5) If so, what may have contributed to the differences in ratings?; (6) how does the number of students impact the results as the number of students in a specialization can range from 5 to 51 in any given year?; and (7) do the ratings reflect the learning process given the time allotted to demonstrate a learning outcome? For example, is there enough time and opportunity in a course (classroom and/or field) for students to demonstrate consistency? MSW program committee will continue to have discussions on the results and plan out a course of action on how to use the results. 

16) Beyond the results, were there additional conclusions or discoveries? This can include insights about assessment procedures, teaching and learning, and great achievements regarding program assessment in this reporting period.

Please see #15.  We have included our additional insights related to the program assessment efforts and activities in the respoonses. 

17) If the program did not engage in assessment activities, please justify.

N/A