Unit: Education (multiple departments)
Program: Education (PhD)
Degree: Doctorate
Date: Thu Oct 22, 2020 - 4:25:01 pm

1) Program Student Learning Outcomes (SLOs) and Institutional Learning Objectives (ILOs)

1. Knowledge: Students will become knowledgeable in the broad issues of education and in the skills, knowledge and problems within their specialization area. Students will develop an inquiring and critical approach to issues and possible solutions to problems in education. The body of knowledge and specific objectives are developed on an individual basis in collaboration among the student, his/her advisor, and committee members. The doctoral program is unique in that students are expected to develop their knowledge of the field to the degree that they can understand the issues and problems in the field of education, and their specialization, in order to develop an inquiry project that will generate new knowledge in education.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 2. Demonstrate understanding of research methodology and techniques specific to one’s field of study.)

2. Research: Students will develop competencies in the broad issues of conducting and evaluating research in education, and develop the skills needed to develop a research problem and questions, design a study about a significant issue, collect appropriate data, analyze results and write a dissertation. Students' dissertation research must be on an original problem and contribute to the knowledge base in education. The results of students' research will be disseminated internationally, nationally and locally, where appropriate, so that it may be utilized for the improvement of education.

(3. Apply research methodology and/or scholarly inquiry techniques specific to one’s field of study., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study.)

3. Caring and Professional Ethics: Students will develop a deep respect for the public trust that is invested in them as future intellectual and social leaders in the field of education. Research that they conduct, or to which they refer in making recommendations, will be carefully vetted for accuracy, fairness, and beneficence regarding the clients, recipients, participants and the broader public good. The program supports an attitude or disposition of caring or consideration for all people, and especially for students at every level and from various walks of life.

(5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

2) Your program's SLOs are published as follows. Please update as needed.

Department Website URL: https://coe.hawaii.edu/node/4932
Student Handbook. URL, if available online: tinyurl.com/phdeduchandbook
Information Sheet, Flyer, or Brochure URL, if available online:
UHM Catalog. Page Number:
Course Syllabi. URL, if available online:
Other:

3) Please review, add, replace, or delete the existing curriculum map.

Curriculum Map File(s) from 2020:

4) For your program, the percentage of courses that have course SLOs explicitly stated on the syllabus, a website, or other publicly available document is as follows. Please update as needed.

0%
1-50%
51-80%
81-99%
100%

5) Does the program have learning achievement results for its program SLOs? (Example of achievement results: "80% of students met expectations on SLO 1.")(check one):

No
Yes, on some(1-50%) of the program SLOs
Yes, on most(51-99%) of the program SLOs
Yes, on all(100%) of the program SLOs

6) Did your program engage in any program learning assessment activities between November 1, 2018 and October 31, 2020?

Yes
No (skip to question 17)

7) What best describes the program-level learning assessment activities that took place for the period November 1, 2018 and October 31, 2020? (Check all that apply.)

Create/modify/discuss program learning assessment procedures (e.g., SLOs, curriculum map, mechanism to collect student work, rubric, survey)
Collect/evaluate student work/performance to determine SLO achievement
Collect/analyze student self-reports of SLO achievement via surveys, interviews, or focus groups
Use assessment results to make programmatic decisions (e.g., change course content or pedagogy, design new course, hiring)
Investigate other pressing issue related to student learning achievement for the program (explain in question 8)
Other:

8) Briefly explain the assessment activities that took place since November 2018.

1. Developed and implemented a dissertation rating rubric

In the 2018-2019 academic year, the Program Committee developed a rubric to assess the quality of components of the dissertation related to the three program SLOs. We began using this assessment in Spring 2019. This provides an end of program assessment that is similar to the proposal assessment rubric used earlier in the program, allowing the Program Committee to evaluate the program and student outcomes at two different time points.

2. Program Completer Survey (Recent Graduates)

The College of Education administers program completer surveys to all graduates. The Program Committee reviews the results to gain an understanding of graduates’ perceptions of program components.

3. Alumni, Employer, and Advisory Board Focus Group

The Program Committee conducted focus groups of various stakeholders as part of a program self study conducted in 2019-2020. The committee discussed and summarized the data for the self study report.

4. Candidate Monitoring, Case Study and Internal Audit

The PhD in Education program conducts an annual check with advisors on the status of each doctoral student. Advisors note whether the students are making Satisfactory or Unsatisfactory progress and provide additional information on those who are making unsatisfactory progress. In Spring 2020, we conducted additional case studies on selected students who were not making adequate progress, by following up with advisors and committee members on their progress. Based on the case studies, the Program Committee conducted an internal audit of program processes and wrote a report for the self study.

9) What types of evidence did the program use as part of the assessment activities checked in question 7? (Check all that apply.)

Artistic exhibition/performance
Assignment/exam/paper completed as part of regular coursework and used for program-level assessment
Capstone work product (e.g., written project or non-thesis paper)
Exam created by an external organization (e.g., professional association for licensure)
Exit exam created by the program
IRB approval of research
Oral performance (oral defense, oral presentation, conference presentation)
Portfolio of student work
Publication or grant proposal
Qualifying exam or comprehensive exam for program-level assessment in addition to individual student evaluation (graduate level only)
Supervisor or employer evaluation of student performance outside the classroom (internship, clinical, practicum)
Thesis or dissertation used for program-level assessment in addition to individual student evaluation
Alumni survey that contains self-reports of SLO achievement
Employer meetings/discussions/survey/interview of student SLO achievement
Interviews or focus groups that contain self-reports of SLO achievement
Student reflective writing assignment (essay, journal entry, self-assessment) on their SLO achievement.
Student surveys that contain self-reports of SLO achievement
Assessment-related such as assessment plan, SLOs, curriculum map, etc.
Program or course materials (syllabi, assignments, requirements, etc.)
Other 1:
Other 2:

10) State the number of students (or persons) who submitted evidence that was evaluated. If applicable, please include the sampling technique used.

Proposal Rating Rubric (n=26)

Sampling Technique: Rubric used for all students who defended a proposal between Dec 2018-Oct 2020

Dissertation Rating Rubric (n=14)

Sampling Technique: Rubric used for all students who defended a proposal between Jan 2019-Jan 2020

Alumni Focus Group (n=8)

Sampling Technique: Focus Group Interview Questions, March 2020

Employer and Advisory Board Focus Group (n=7)

Sampling Technique: Focus Group Interview Questions, Sent in March 2020

Program Completer Survey completers (n=9)

Sampling Technique: Survey sent to recent graduates each semester, 2018-2019

11) Who interpreted or analyzed the evidence that was collected? (Check all that apply.)

Course instructor(s)
Faculty committee
Ad hoc faculty group
Department chairperson
Persons or organization outside the university
Faculty advisor
Advisors (in student support services)
Students (graduate or undergraduate)
Dean/Director
Other: Graduate Chair

12) How did they evaluate, analyze, or interpret the evidence? (Check all that apply.)

Used a rubric or scoring guide
Scored exams/tests/quizzes
Used professional judgment (no rubric or scoring guide used)
Compiled survey results
Used qualitative methods on interview, focus group, open-ended response data
External organization/person analyzed data (e.g., external organization administered and scored the nursing licensing exam)
Other:

13) Summarize the results from the evaluation, analysis, interpretation of evidence (checked in question 12). For example, report the percentage of students who achieved each SLO.

Student Performance (Proposal and Dissertation Rubrics) in relation to SLOs

Tables 1 and 2 below present data from PhD in Education program assessments checked in #7 (Evaluation of Student Work/performance). Each student's doctoral committee completes these rubrics based on their written product and oral defense at the proposal and dissertation stage. Below, we provide an overview of these assessment results in relation to program SLOs and include a summary of the PhD in Education Program Committee's discussion and reflection on each item.

Table 1: Form IIA: Proposal Rating Form

Data collected from December 2018-Oct 2020 (n=26)

ILO

SLO

Rubric Items

Rating Scale

N

MEAN SCORE

MEDIAN SCORE

1

2

3

4

5

ILO #1 & 2

SLO #1

1. Justification/Contribution to the Field

0

0

1

7

18

26

4.63

5

2. Review of the Literature: Appropriateness/Relevance

0

0

1

11

14

26

4.50

5

3. Review of the Literature: Synthesis/Analysis

0

1

1

6

18

26

4.25

5

4. Theoretical Framework Basis

0

1

3

9

13

26

4.0

4

ILO #3 & 4

SLO#2

5. Purpose Statement/Research Questions

0

1

1

9

14

25

4.44

5

6. Proposed Methodology

0

0

3

12

11

26

4.5

4

7. Proposed Analysis

0

0

4

9

13

26

4.5

4.5

ILO

5, 6, 7

SLO #3

8. Professional Dispositions and Ethics

0

0

0

3

20

23

4.67

5

 

Form IIIA: Dissertation Rating Form

Data collected from January 2019-January 2020 (n=14)

ILO

SLO

Rubric Items

Rating Scale

N

MEAN SCORE

MEDIAN SCORE

1

2

3

4

5

ILO #1 & 2

SLO #1

1. Justification/Contribution to the Field

0

0

0

5

9

14

4.64

5

2. Review of the Literature: Appropriateness/Relevance

0

0

1

3

10

14

4.64

5

3. Review of the Literature: Synthesis/Analysis

0

0

3

3

8

14

4.36

5

4. Theoretical Framework Basis

0

0

3

3

8

14

4.43

5

ILO #3 & 4

SLO#2

5. Purpose Statement/Research Questions

0

0

0

4

10

14

4.71

5

6. Methodology

0

0

1

4

9

14

4.57

5

7. Analysis & Discussion

0

0

1

5

8

14

4.50

5

ILO 5, 6, 7

SLO 3

8. Oral Defense Professional Dispositions and Ethics

0

0

0

1

13

14

4.93

5

Analysis:

SLO #1: Knowledge (ILO #1 & 2):

Based on means for items #1-4, the Program Committee felt that core courses were adequately preparing students to demonstrate knowledge of issues in education within their specialization areas and an understanding of how to frame a problem and develop an inquiry for a dissertation study. Item #4 (theoretical frameworks) had a lower mean relative to the other items. The Program Committee discussed ways to integrate more information on theoretical frameworks in initial coursework and standardize how this information is taught across core courses.

SLO #2: Research (ILO #3 & 4)

As evidenced by the means on Items #5-7, students are prepared to successfully write the Methods, Results and Discussion chapters of their proposals and dissertations. The Program Committee concurred that the three core research courses required by the doctoral program adequately prepared students to design, implement, and write up their research studies. The Committee also discussed the fact that students get a great deal of individual mentoring in addition to the knowledge they gain in courses. By the time students write their proposals and dissertation, they are expected to have mastery of skills in relation to their research topics. The strong scores on these items demonstrate that students develop their skills through program experiences and advising provided by their dissertation chair and committee members. For ratings of 3 and below, students were given feedback by committee members and expected to revise and strengthen sections as directed.

SLO #3: Caring and Professional Ethics (ILO #5, 6, 7)

Based on means for Item #8, PhD in Education candidates demonstrate excellent professional dispositions and ethics. Candidates have various opportunities to gain and demonstrate these skills, including collaborative projects with faculty, participation in conference presentations, and the mentorship of faculty during college teaching experiences and internships.

Advisory Board and Employer Focus group (March 2020) 

The Advisory Board and Employer Focus included stakeholders from various sectors (public and private schools, universities, and non-profit organizations). Employers of PhD in Education program graduates noted that recent hires have been well-prepared and knowledgeable about the educational needs of Hawaii and the Pacific, have developed relationships with service providers and research organizations, and are passionate about their work and the communities they serve. Graduates have been hired in high-level responsible positions in Hawaii, nationally, and internationally and have a COE network of faculty and peers to rely upon. One respondent noted that "colleagues from the COE, who have received their PhDs within the last five or so years have been innovative, creative, and problem-solvers." 

The program Advisory Board commented on three areas of strength: (a) preparation of doctoral graduates to be nationally and internationally competitive researchers and educational leaders, (b) flexibility of the program regarding topics and focus areas, and (c) strong mentoring by faculty. Respondents also noted that COE doctoral programs integrate training relevant for the Pacific region served by the University of Hawaii. Regarding the quality of doctoral graduates, one respondent noted,“COE doctoral students are exceptionally well prepared academically and socially and are competitive in national searches for faculty candidates." Another respondent stated that "the ability of graduates to compete for leadership positions in diverse venues successfully speaks to the high quality of their doctoral preparation."

Alumni Focus Group (March 2020)

Eight alumni who have graduated from the program within the last three years participated in the focus group. Their responses indicated that graduates were well prepared for roles as teacher educators and university faculty members. In response to the question ”In what areas did the doctoral program prepare you most effectively for your professional practice?" alumni noted that “the program prepared me to be a productive faculty member. I not only knew how to create a syllabus, assignments, exams and to effectively teach students, I understood the administrative tasks that surrounded setting up courses and schedules. I was prepared to write grants, set-up research projects and a research agenda, write an IRB and present to my profession."  The program also prepared students to be aware of issues of diversity and inclusion in relation to teaching.

The Program Committee discussed the focus group data and noted that program strengths include our graduates' ability to work in the unique contexts of Hawaii and the Pacific, with an understanding of the cultural norms and values. PhD in Education graduates are knowledgeable about the needs of stakeholders in Hawaii and the Pacific and well-prepared to develop ongoing relationships with service providers and research organizations. They are passionate about the communities they serve and respectful of issues of diversity and inclusion in local and global contexts.

14) What best describes how the program used the results? (Check all that apply.)

Assessment procedure changes (SLOs, curriculum map, rubrics, evidence collected, sampling, communications with faculty, etc.)
Course changes (course content, pedagogy, courses offered, new course, pre-requisites, requirements)
Personnel or resource allocation changes
Program policy changes (e.g., admissions requirements, student probation policies, common course evaluation form)
Students' out-of-course experience changes (advising, co-curricular experiences, program website, program handbook, brown-bag lunches, workshops)
Celebration of student success!
Results indicated no action needed because students met expectations
Use is pending (typical reasons: insufficient number of students in population, evidence not evaluated or interpreted yet, faculty discussions continue)
Other:

15) Please briefly describe how the program used its findings/results.

Findings and Results from Proposal and Dissertation Assessment Rubrics:

In our Program Committee discussion, the lower scores on Item #4 (theoretical frameworks) were a concern. Faculty discussed ways to include ongoing discussions about theoretical frameworks in coursework and in post-coursework experiences. We will examine course syllabi to ensure that theoretical frameworks are addressed in core research courses. We will also consider ways to address theoretical frameworks at various stages of the doctoral program to ensure that students have adequate knowledge of this topic prior to writing their proposals.

The Program Committee noted that students should receive feedback from their committees on specific areas to strengthen during their proposal defenses and that these changes should be reflected in their dissertations. In the future, the Program Committee will monitor the ratings of this item between the two points (proposal assessment and dissertation assessment) in order to evaluate whether there are changes in individual student scores on Item #4 (theoretical framework).

Findings and Results of Self Study:

When reviewing all assessment data for a self-study, the Program Committee discussed how we met standards for our professional accreditation with the Association for Advancing Quality Educator Preparation (AAQEP).  The PhD in Education program aligned with AAQEP Standard 1 in the following ways:

PhD in Education graduates work as educational researchers, scholars, and leaders. Our graduates conduct research studies contextualized within local, global, and international contexts and disseminate their research in varied national and international outlets, via publications and conference presentations. Our graduates are able to utilize their skills as researchers and advanced practitioners in varied professional roles appropriate for doctoral graduates (e.g., tenure track positions, school administrators, education non-profit leaders). Program faculty provide ongoing mentorship and support to graduates to enable them to advance in their careers (e.g., publishing, conferences, etc)

The assessment of program data for our self-study helped us identify areas we can strengthen in regard to collection of evidence. For example, we will more closely track information on specific aspects (e.g., collect information on students' knowledge of culturally responsive practices) and examine content of elective courses in the program. To provide additional student support, the program is in the process of implementing Individualized Development Plans (IDPs). IDPs are a tool for professional and scholarly growth, self-reflection, goal-setting and communication with advisors and mentors. IDPs can be one mechanism to provide ongoing support for students during the program.

16) Beyond the results, were there additional conclusions or discoveries? This can include insights about assessment procedures, teaching and learning, and great achievements regarding program assessment in this reporting period.

In 2019-2020, the PhD in Education program conducted a self-study as part of the process of professional accreditation with AAQEP. As a part of the self-study process, the Program Committee closely reviewed data from assessments added since our last Manoa Assessment Report in 2018. The committee concurred that have we have a better overview of program quality across the stages of the program (admission, during program, program completion, and post-completion) with the new assessments we have added.

The PhD in Education program is currently adding a new process to facilitate student persistence and completion and to support scholarly and professional development for our students. As a result of assessment discussions and student feedback, the program explored the possibility of integrating Individualized Development Plans (IDPs) in 2019-2020. IDPs are used by doctoral trainees and post-doctoral scholars to set goals and receive mentorship on scholarly, research, and professional goals in conjunction with their development as researchers and educational scholars (National Institutes of Health, 2019). In AY 2019-2020, we began a systematic process to explore how IDPs can be customized for and integrated into our PhD program. The graduate chair, along with a few other faculty members, reviewed the literature on IDPs in various fields, spoke with other faculty at UH whose programs use IDPs, and conducted faculty and student focus groups to get stakeholders’ perspectives on IDPs.  As a result of these efforts, the program has developed a customized IDP plan for PhD in Education students, called the “PhD in Education Personalized Plan”. The program is piloting the “PhD Personalized Plan” with a group of doctoral students in Fall 2020 and will continue to improve and integrate this support based on feedback from students and faculty.

17) If the program did not engage in assessment activities, please justify.