Degree Program Assessment Reports : View Reports

Unit: Social Work
Program: Social Work (MSW)
Degree: Master's
Date: Fri Nov 20, 2020 - 5:25:44 pm

1) Program Student Learning Outcomes (SLOs) and Institutional Learning Objectives (ILOs)

1. Demonstrate ethical and professional behavior.

(5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

2. Engage diversity and difference in practice.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

3. Advance human rights and social, economic, and environmental justice.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

4. Engage in practice-informed research and research-informed practice.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 2. Demonstrate understanding of research methodology and techniques specific to one’s field of study., 3. Apply research methodology and/or scholarly inquiry techniques specific to one’s field of study., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study., 5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

5. Engage in policy practice.

6. Engage with individuals, families, groups, organizations, and communities.

7. Assess individuals, families, groups, organizations, and communities.

8. Intervene with individuals, families, groups, organizations, and communities.

9. Evaluate practice with individuals, families, groups, organizations, and communities.

10. Engage, honor, and respect indigenous culture towards decolonized professional practice.

2) Your program's SLOs are published as follows. Please update as needed.

Department Website URL: https://www.hawaii.edu/sswork/msw.html

Student Handbook. URL, if available online: https://www.hawaii.edu/sswork/wp-content/uploads/MSW-Handbook-2019_adjusted.pdf

Information Sheet, Flyer, or Brochure URL, if available online: https://www.hawaii.edu/sswork/bulletin.html & https://www.hawaii.edu/sswork/wp-content/uploads/uh_mbt_ssw_e-version_final_optimized.pdf

UHM Catalog. Page Number: http://www.catalog.hawaii.edu/schoolscolleges/socialwork/SocialWork.html#grad

Course Syllabi. URL, if available online:

Other: New Student Orientation Materials

3) Please review, add, replace, or delete the existing curriculum map.

Curriculum Map File(s) from 2020:

File (11/17/2020)

4) For your program, the percentage of courses that have course SLOs explicitly stated on the syllabus, a website, or other publicly available document is as follows. Please update as needed.

1-50%

51-80%

81-99%

100%

5) Does the program have learning achievement results for its program SLOs? (Example of achievement results: "80% of students met expectations on SLO 1.")(check one):

Yes, on some(1-50%) of the program SLOs

Yes, on most(51-99%) of the program SLOs

Yes, on all(100%) of the program SLOs

6) Did your program engage in any program learning assessment activities between November 1, 2018 and October 31, 2020?

Yes

No (skip to question 17)

7) What best describes the program-level learning assessment activities that took place for the period November 1, 2018 and October 31, 2020? (Check all that apply.)

Create/modify/discuss program learning assessment procedures (e.g., SLOs, curriculum map, mechanism to collect student work, rubric, survey)

Collect/evaluate student work/performance to determine SLO achievement

Collect/analyze student self-reports of SLO achievement via surveys, interviews, or focus groups

Use assessment results to make programmatic decisions (e.g., change course content or pedagogy, design new course, hiring)

Investigate other pressing issue related to student learning achievement for the program (explain in question 8)

Other:

8) Briefly explain the assessment activities that took place since November 2018.

We have continuted to conduct our assessment activities to measure the degree to which our ten Student Learning Outcomes (as listed above #1) have been met throughout the curriculum over the course of the academic year. We use direct evidence (instructor evaluation of student products such as reports, presentations, classroom activities, and skill demonstration as measured through the use of the Instructor Evaluation Sheet (IES) and the Field Supervisory Evaluation form (FSE)) as detailed below.

We have a standing Assessment Committee - made up of program chairs, the directors of both student services and distance education as well as the Department Chair who functions as Assessment Committee Chair - that collects, analyzes and evaluates SLO data through the IES, FSE, and SSASI, and presents results for program decision-making and pedagogical/course content change. Our assessment results have directly impacted program implementation and performance.

The current MSW assessment includes direct measures from two data sources utilized by both program options (Campus-based & Distance Education). The following details the description of two measures used to assess two dimensions of each of the 10 SLOs using a real or simulated practice situation. The plan also lays out the assessment procedures for each of the measures.

We established both a 5-point Likert scale (fully detailed below) as well as an acquisition benchmark for all SLOs. Taken in aggregate, the expectation is that 80% of all students surveyed in the MSW Program will score a 4 or higher on a 5-point Likert scale. As detailed below, a score of 4 on our 5-point Likert scale represents near consistency of competency demonstration. We recognize that students have varying skills upon entry to the Program and throughout their tenure, and will continue to learn and grow long after their formal educational experience is complete. The near consistent demonstration of social work competencies is critical if our graduates hope to be legitimate helpers in the community.

We also decided that each competency will assess acquisition of the skill dimension for each competency using the FSE (Measure 1). The IES (Measure 2) would assess one of the other dimensions (knowledge, value, cognitive & affective processing) for each competency as determined by Sequence (HBSE, Generalist Practice, Policy, Research) and Specialization (Behavioral Mental Health, Child & Family, Gerontology, Health) Chairs.

Assessment Measurement

The attainment of each SLO through specific articulated dimensions is assessed using the following two measures:

(1) Field Supervisory Evaluation (FSE): Field Instructor ratings of the skill dimension for each SLO through demonstration of behaviors upon the student’s completion of foundation/specialization field experience and;

(2) Instructor Evaluation Sheets (IES): Classroom instructor ratings of another dimension (knowledge, values, cognitive & affective processing) of each SLO based on a signature assignment.

Measure 1: Field Supervisory Evaluation (FSE)

The Field Supervisory Evaluation (an online survey format) was selected as the instrument to assess the skill dimension of each SLO and related behaviors. Field education, the signature pedagogy in social work education, provides students the opportunity to draw upon theories and practice skills they learn in the classroom and apply them to the real practice situations. Field Instructors assess their respective students on attainment of the skill dimension of each SLO through demonstration of related behaviors.

Procedures

The FSE was developed into an online survey using Surveyshare, an online survey application. The FSE lists all skills of the behaviors under each SW competency.
Field Instructors receive an email from the Assessment Chair with instructors on how to complete the online survey. A hyperlink to the survey is provided in the e-mail invitation. Field Instructors (FIs) receive training on the instrument by the Field Education Office and the Assessment Committee Chair.
Field Instructors rate their respective student(s) on each behavior upon the completion of her/his MSW foundation/MSW specialization. The process is repeated for Field Instructors who have more than one student.
Completed FSE is due at the same time as the University’s grade submission deadline (the end of the generalist or the end of the specialization field education). A reminder e-mail is sent to Field Instructors who have not completed the survey by the due date.
The FSE ratings are stored in the online survey application and used for analysis and reporting.

Measure 2: Classroom Instructor Evaluation Sheet (IES)

Instructor Evaluation Sheet was designed to assess student’s demonstration of one of three dimensions (knowledge, values, cognitive & affective processing) of each SLO by the classroom instructor. Each instructor completes the appropriate form for their respective class(es) according to the directions provided by the Assessment Chair.

In each IES form, the top row is used to identify students by number. The instructor rates each student, on a 5-point Likert scale, on the degree to which the student has achieved a specific SLO through a selected dimension (see below for a detailed explanation of the Likert scale).

Procedures

The IES forms are stored on Laulima (i.e., the University of Hawai‘i’s online communication system which allows to distribute/receive materials).
All instructors are informed of the location and access to IES forms that are specific to their courses and their use at MSW Program meetings and adjunct faculty training sessions. Instructors are also informed on the importance of being cognizant of their ratings on the IES for each student, which may not match her/his class grade on the signature assignment. This is because the grade for an assignment includes factors such as organization of paper, grammar, met the deadline for the paper or project. While these factors are important inclusions in grading an assignment, they may not be necessary factors to consider when assessing demonstration of a dimension related to a SLO measured in the respective course(s). The evaluation of the SLOs through respective dimensions was designed to be independent from the grading process.
Instructors receive email reminders from the Assessment Chair with detailed information on how to access the Instructor Evaluation Sheet (IES).
Instructors were instructed to download their respective IES to their computers/laptops.
Instructors use the IES to assess their students’ achievement on SLOs through respective dimensions relevant to their course(s).
Instructors complete the IES and submit to the Assessment Chair at the same time grades are due at the end of each semester.

The MSW (and BSW) Program continues to hold “teachering” sessions to address curriculum questions and common classroom issues. Topics have included curriculum rollout, vertical and horizontal sequencing, assignment design and implementation as well as classroom management. Due to the COVID-19 pandemic, we have also held training sessions on utilization of Zoom, creating engaging classroom experiences while online, and sent out notifications for trainings being held in UH Manoa community as well as the community at large. Also covered are SLOs and course assignment alignment. It is critical the both instructors and adjunct faculty (lecturers) recognize the need to connect assignments to stated objectives to measure competence. Because of this, the “teachering” sessions serve as both training site and site for program culture development that is competency based.

In the spring of 2017 the department produced and continues to use a video on assessing student competency and effective competency measurement that is shown as part of the biannual faculty orientation. The aim of the video is to: 1) clarify the IES as an assessment tool, 2) decouple competency assessment from grading, and 3) address some of the common pitfalls in the assessment process. We have found over the years that faculty will sometimes assess students against the understandable yet unrealistic standard of the seasoned social worker. Against such a standard, it is nearly impossible for a student to be anything but substandard. Instead, we want faculty to assess social work competencies based on where the student is at in their academic and professional development. The question is not “How does this student stack up against a real or imagined seasoned social worker?” but rather, “How is this student performing relative to their position as a junior (or senior) level student?” This distinction is critical and while still an "issue' has improved over the course of the last several years.

In addition to assessing competency development, the program also assesses key program functions including student services and the field experience. In the past we modified student advising in response to student feedback. Students were clear that they wanted more professional advising as they prepared for life after school.

9) What types of evidence did the program use as part of the assessment activities checked in question 7? (Check all that apply.)

Artistic exhibition/performance

Assignment/exam/paper completed as part of regular coursework and used for program-level assessment

Capstone work product (e.g., written project or non-thesis paper)

Exam created by an external organization (e.g., professional association for licensure)

Exit exam created by the program

IRB approval of research

Oral performance (oral defense, oral presentation, conference presentation)

Portfolio of student work

Publication or grant proposal

Qualifying exam or comprehensive exam for program-level assessment in addition to individual student evaluation (graduate level only)

Supervisor or employer evaluation of student performance outside the classroom (internship, clinical, practicum)

Thesis or dissertation used for program-level assessment in addition to individual student evaluation

Alumni survey that contains self-reports of SLO achievement

Employer meetings/discussions/survey/interview of student SLO achievement

Interviews or focus groups that contain self-reports of SLO achievement

Student reflective writing assignment (essay, journal entry, self-assessment) on their SLO achievement.

Student surveys that contain self-reports of SLO achievement

Assessment-related such as assessment plan, SLOs, curriculum map, etc.

Program or course materials (syllabi, assignments, requirements, etc.)

Other 1:

Other 2:

10) State the number of students (or persons) who submitted evidence that was evaluated. If applicable, please include the sampling technique used.

Students are assessed for skill acquisition in both the classroom and field (2 points) as explained in this assessment report (question #8). Additonally, we assess across at least two of four dimensions including knowledge, values, cognitive/affective processes and skills. For our accreditation, skill assessment is mandatory and must be assessed in practice (field) with real clients or through simulation. 100% of the 135 students in the program were assessed in the classroom and the field for AY 2019-2020.

The total number of students assessed was: 135

11) Who interpreted or analyzed the evidence that was collected? (Check all that apply.)

Course instructor(s)

Faculty committee

Ad hoc faculty group

Department chairperson

Persons or organization outside the university

Faculty advisor

Advisors (in student support services)

Students (graduate or undergraduate)

Dean/Director

Other:

12) How did they evaluate, analyze, or interpret the evidence? (Check all that apply.)

Used a rubric or scoring guide

Scored exams/tests/quizzes

Used professional judgment (no rubric or scoring guide used)

Compiled survey results

Used qualitative methods on interview, focus group, open-ended response data

External organization/person analyzed data (e.g., external organization administered and scored the nursing licensing exam)

Other:

13) Summarize the results from the evaluation, analysis, interpretation of evidence (checked in question 12). For example, report the percentage of students who achieved each SLO.

Our assessment goal continues to measure the degree to which our ten Program Learning Outcomes (competencies) have been met throughout the curriculum over the course of the academic year. We use both direct evidence (instructor evaluation of student products such as reports, presentations, classroom activities, and skill demonstration as measured through the use of the Instructor Evaluation Sheet (IES) and the Field Supervisory Evaluation form (FSE)) and indirect evidence (student self-report as measured through the use of the Student Self-Assessment Survey Instrument (SSASI)). The Ten Program SLOs are:

1. Demonstrate ethical and professional behavior.

2. Engage diversity and difference in practice.

3. Advance human rights and social, economic, and environmental justice.

4. Engage in practice-informed research and research-informed practice.

5. Engage in policy practice.

6. Engage with individuals, families, groups, organizations, and communities.

7. Assess individuals, families, groups, organizations, and communities

8. Intervene with individuals, families, groups, organizations, and communities.

9. Evaluate practice with individuals, families, groups, organizations, and communities.

10. Engage, honor, and respect indigenous culture towards decolonized professional practice

Benchmark and Rationale

The MSW Program, working with the Department Assessment Committee, established both a 5-point Likert scale as well as an acquisition benchmark for all social work competencies. Taken in aggregate, the expectation is that 80% of all students surveyed in the MSW Program will score a 4 or higher on a 5-point Likert scale. A score of 4 on our 5-point Likert scale represents near consistency of competency demonstration. The MSW Program recognizes that students have varying skills upon entry to the Program and throughout their tenure, and will continue to learn and grow long after their formal educational experience is complete. The near consistent demonstration of social work competencies is critical if our graduates hope to be legitimate helpers in the community.

CSWE identifies several dimensions critical to each competency, they are Knowledge, Values, Skills, and Cognitive & Affective Processes. The MSW program determined that the Skill dimension would be measured on the Field Supervisory Evaluation (FSE), while one of the other remaining dimensions would be measured on the Instructor Evaluation Sheet (IES) as determined by the MSW Faculty.

Two Measures for each Competency

The attainment of SW competencies through specific dimensions as outlined by CSWE is assessed using the following two measures:

Field Supervisory Evaluation (FSE). The FSE measures the acquisition of each of the nine social work competencies outlined by CSWE as well as the tenth competency designed by the DSW and implemented by the MSW Program through the Skill dimension. Full details of the implementation process are detailed below.
Instructor Evaluation Sheets (IES): the IES measures the acquisition of each of the nine social work competencies outlined by CSWE as well as the tenth competency designed by the DSW and implemented by the BSW Program through one of the remaining three dimensions (Knowledge, Values, Cognitive and Affective Processes).

Rating Scale

Using the IES or FSE form, classroom and field instructors rate students on their level of proficiency for each SW competency through specific and particular dimensions that are measured in their courses using 5-point scale developed and approved by the Assessment Committee:

Not Yet Emerging (This dimension of the competency has not yet emerged in the student’s repertoire.)
Emerging (This dimension of the competency is beginning to emerge in the student’s repertoire.)
Developing (The student is developing this dimension of the competency.)
Approaching Consistency (The student is approaching consistent demonstration of/engagement in this dimension of the competency; consistency is anticipated in the near future.)
Consistently Demonstrates (The student consistently demonstrates/engages in this dimension of the competency.)

Results

As stated earlier, the MSW program set a benchmark of 80% of all students scoring 4 or higher on the 5-point Likert-type scale for each of the 10 SLOs/Competencies.

In our Generalist curriculum, seven SLOs (i.e., SW competencies) met the 80% benchmark except SLOs 1 and 7 (that are approaching the benchmark, 78% & 79%), and SLO 9 did not meet the benchmark (76%). Looking into FSE data and IES data separately, evaluations from class instructors were noticeably low, whereas evaluations from field instructors showed that all SLOs met the benchmark.

In our Specializations, the two specializations, Child & Family and Gerontology, met the benchmark for all SLOs. The Behavioral Mental Health (BMH) specialization data showed that 9 out of the 10 benchmarks were met: SLO 8 did not meet the benchmark. For the Health specialization, 8 out of 10 benchmarks were met: SLOs 1 and 2 did not meet the benchmark. The DSW assessment committee will conduct more in-depth analysis in this AY 2020-2021, working with MSW program committee. We will look into scoring errors and discrepancies between instructors because we noted discrepancies in the scores between difference sections of the same course. We will continue to ensure that instructors are fully aware of our scorning systems by sending out its instructional video each semester before instructors fill out their respective IES as a reminder. In addition, we noted that scores from IES vs. FSE showed significant discrepancies which the DSW assessment committee will examine further working with the MSW committee.

14) What best describes how the program used the results? (Check all that apply.)

Assessment procedure changes (SLOs, curriculum map, rubrics, evidence collected, sampling, communications with faculty, etc.)

Course changes (course content, pedagogy, courses offered, new course, pre-requisites, requirements)

Personnel or resource allocation changes

Program policy changes (e.g., admissions requirements, student probation policies, common course evaluation form)

Students' out-of-course experience changes (advising, co-curricular experiences, program website, program handbook, brown-bag lunches, workshops)

Celebration of student success!

Results indicated no action needed because students met expectations

Use is pending (typical reasons: insufficient number of students in population, evidence not evaluated or interpreted yet, faculty discussions continue)

Other:

15) Please briefly describe how the program used its findings/results.

In the past results were used to modify both the implicit and explicit curriculum as well as the rubric used to assess collected data. Current results from assessment activities for 2019-2020 are pending a larger discussion with the MSW program faculty after the DSW assessment committee conducts further analyses.

16) Beyond the results, were there additional conclusions or discoveries? This can include insights about assessment procedures, teaching and learning, and great achievements regarding program assessment in this reporting period.

Please see #13. We have included our insights abour assessment procedures in the responses. We are proud of our assessment systems we have developed and implemented (e.g., IES & FSE) and have been satisfied with whole assessment procedures and great achievements (e.g., fully reaffirmed by our accredited body, Council on Social Work Education).

17) If the program did not engage in assessment activities, please justify.

N/A