Degree Program Assessment Reports

1. Knowledge: Students will become knowledgeable in the broad issues of education and in the skills, knowledge and problems within their specialization area. Students will develop an inquiring and critical approach to issues and possible solutions to problems in education. The body of knowledge and specific objectives are developed on an individual basis in collaboration among the student, his/her advisor, and committee members. The doctoral program is unique in that students are expected to develop their knowledge of the field to the degree that they can understand the issues and problems in the field of education, and their specialization, in order to develop an inquiry project that will generate new knowledge in education.

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 2. Demonstrate understanding of research methodology and techniques specific to one’s field of study.)

2. Research: Students will develop competencies in the broad issues of conducting and evaluating research in education, and develop the skills needed to develop a research problem and questions, design a study about a significant issue, collect appropriate data, analyze results and write a dissertation. Students' dissertation research must be on an original problem and contribute to the knowledge base in education. The results of students' research will be disseminated internationally, nationally and locally, where appropriate, so that it may be utilized for the improvement of education.

(3. Apply research methodology and/or scholarly inquiry techniques specific to one’s field of study., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study.)

3. Caring and Professional Ethics: Students will develop a deep respect for the public trust that is invested in them as future intellectual and social leaders in the field of education. Research that they conduct, or to which they refer in making recommendations, will be carefully vetted for accuracy, fairness, and beneficence regarding the clients, recipients, participants and the broader public good. The program supports an attitude or disposition of caring or consideration for all people, and especially for students at every level and from various walks of life.

(5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

Curriculum Map File(s) from 2020:

• File (10/14/2020)

No
Yes, on some(1-50%) of the program SLOs
Yes, on most(51-99%) of the program SLOs
Yes, on all(100%) of the program SLOs

Create/modify/discuss program learning assessment procedures (e.g., SLOs, curriculum map, mechanism to collect student work, rubric, survey)
Collect/evaluate student work/performance to determine SLO achievement
Collect/analyze student self-reports of SLO achievement via surveys, interviews, or focus groups
Use assessment results to make programmatic decisions (e.g., change course content or pedagogy, design new course, hiring)
Investigate other pressing issue related to student learning achievement for the program (explain in question 8)
Other:

Artistic exhibition/performance
Assignment/exam/paper completed as part of regular coursework and used for program-level assessment
Capstone work product (e.g., written project or non-thesis paper)
Exam created by an external organization (e.g., professional association for licensure)
Exit exam created by the program
IRB approval of research
Oral performance (oral defense, oral presentation, conference presentation)
Portfolio of student work
Publication or grant proposal
Qualifying exam or comprehensive exam for program-level assessment in addition to individual student evaluation (graduate level only)
Supervisor or employer evaluation of student performance outside the classroom (internship, clinical, practicum)
Thesis or dissertation used for program-level assessment in addition to individual student evaluation
Alumni survey that contains self-reports of SLO achievement
Employer meetings/discussions/survey/interview of student SLO achievement
Interviews or focus groups that contain self-reports of SLO achievement
Student reflective writing assignment (essay, journal entry, self-assessment) on their SLO achievement.
Student surveys that contain self-reports of SLO achievement
Assessment-related such as assessment plan, SLOs, curriculum map, etc.
Program or course materials (syllabi, assignments, requirements, etc.)
Other 1:
Other 2:

Course instructor(s)
Faculty committee
Ad hoc faculty group
Department chairperson
Persons or organization outside the university
Faculty advisor
Advisors (in student support services)
Students (graduate or undergraduate)
Dean/Director
Other: Graduate Chair

Student Performance (Proposal and Dissertation Rubrics) in relation to SLOs

Tables 1 and 2 below present data from PhD in Education program assessments checked in #7 (Evaluation of Student Work/performance). Each student's doctoral committee completes these rubrics based on their written product and oral defense at the proposal and dissertation stage. Below, we provide an overview of these assessment results in relation to program SLOs and include a summary of the PhD in Education Program Committee's discussion and reflection on each item.

Table 1: Form IIA: Proposal Rating Form

Data collected from December 2018-Oct 2020 (n=26)

ILO	SLO	Rubric Items	Rating Scale					N	MEAN SCORE	MEDIAN SCORE
			1	2	3	4	5
ILO #1 & 2	SLO #1	1. Justification/Contribution to the Field	0	0	1	7	18	26	4.63	5
		2. Review of the Literature: Appropriateness/Relevance	0	0	1	11	14	26	4.50	5
		3. Review of the Literature: Synthesis/Analysis	0	1	1	6	18	26	4.25	5
		4. Theoretical Framework Basis	0	1	3	9	13	26	4.0	4
ILO #3 & 4	SLO#2	5. Purpose Statement/Research Questions	0	1	1	9	14	25	4.44	5
		6. Proposed Methodology	0	0	3	12	11	26	4.5	4
		7. Proposed Analysis	0	0	4	9	13	26	4.5	4.5
ILO 5, 6, 7	SLO #3	8. Professional Dispositions and Ethics	0	0	0	3	20	23	4.67	5

 

Form IIIA: Dissertation Rating Form

Data collected from January 2019-January 2020 (n=14)

ILO	SLO	Rubric Items	Rating Scale					N	MEAN SCORE	MEDIAN SCORE
			1	2	3	4	5
ILO #1 & 2	SLO #1	1. Justification/Contribution to the Field	0	0	0	5	9	14	4.64	5
		2. Review of the Literature: Appropriateness/Relevance	0	0	1	3	10	14	4.64	5
		3. Review of the Literature: Synthesis/Analysis	0	0	3	3	8	14	4.36	5
		4. Theoretical Framework Basis	0	0	3	3	8	14	4.43	5
ILO #3 & 4	SLO#2	5. Purpose Statement/Research Questions	0	0	0	4	10	14	4.71	5
		6. Methodology	0	0	1	4	9	14	4.57	5
		7. Analysis & Discussion	0	0	1	5	8	14	4.50	5
ILO 5, 6, 7	SLO 3	8. Oral Defense Professional Dispositions and Ethics	0	0	0	1	13	14	4.93	5

Analysis:

SLO #1: Knowledge (ILO #1 & 2):

Based on means for items #1-4, the Program Committee felt that core courses were adequately preparing students to demonstrate knowledge of issues in education within their specialization areas and an understanding of how to frame a problem and develop an inquiry for a dissertation study. Item #4 (theoretical frameworks) had a lower mean relative to the other items. The Program Committee discussed ways to integrate more information on theoretical frameworks in initial coursework and standardize how this information is taught across core courses.

SLO #2: Research (ILO #3 & 4)

As evidenced by the means on Items #5-7, students are prepared to successfully write the Methods, Results and Discussion chapters of their proposals and dissertations. The Program Committee concurred that the three core research courses required by the doctoral program adequately prepared students to design, implement, and write up their research studies. The Committee also discussed the fact that students get a great deal of individual mentoring in addition to the knowledge they gain in courses. By the time students write their proposals and dissertation, they are expected to have mastery of skills in relation to their research topics. The strong scores on these items demonstrate that students develop their skills through program experiences and advising provided by their dissertation chair and committee members. For ratings of 3 and below, students were given feedback by committee members and expected to revise and strengthen sections as directed.

SLO #3: Caring and Professional Ethics (ILO #5, 6, 7)

Based on means for Item #8, PhD in Education candidates demonstrate excellent professional dispositions and ethics. Candidates have various opportunities to gain and demonstrate these skills, including collaborative projects with faculty, participation in conference presentations, and the mentorship of faculty during college teaching experiences and internships.

Advisory Board and Employer Focus group (March 2020) 

The Advisory Board and Employer Focus included stakeholders from various sectors (public and private schools, universities, and non-profit organizations). Employers of PhD in Education program graduates noted that recent hires have been well-prepared and knowledgeable about the educational needs of Hawaii and the Pacific, have developed relationships with service providers and research organizations, and are passionate about their work and the communities they serve. Graduates have been hired in high-level responsible positions in Hawaii, nationally, and internationally and have a COE network of faculty and peers to rely upon. One respondent noted that "colleagues from the COE, who have received their PhDs within the last five or so years have been innovative, creative, and problem-solvers." 

The program Advisory Board commented on three areas of strength: (a) preparation of doctoral graduates to be nationally and internationally competitive researchers and educational leaders, (b) flexibility of the program regarding topics and focus areas, and (c) strong mentoring by faculty. Respondents also noted that COE doctoral programs integrate training relevant for the Pacific region served by the University of Hawaii. Regarding the quality of doctoral graduates, one respondent noted,“COE doctoral students are exceptionally well prepared academically and socially and are competitive in national searches for faculty candidates." Another respondent stated that "the ability of graduates to compete for leadership positions in diverse venues successfully speaks to the high quality of their doctoral preparation."

Alumni Focus Group (March 2020)

Eight alumni who have graduated from the program within the last three years participated in the focus group. Their responses indicated that graduates were well prepared for roles as teacher educators and university faculty members. In response to the question ”In what areas did the doctoral program prepare you most effectively for your professional practice?" alumni noted that “the program prepared me to be a productive faculty member. I not only knew how to create a syllabus, assignments, exams and to effectively teach students, I understood the administrative tasks that surrounded setting up courses and schedules. I was prepared to write grants, set-up research projects and a research agenda, write an IRB and present to my profession."  The program also prepared students to be aware of issues of diversity and inclusion in relation to teaching.

The Program Committee discussed the focus group data and noted that program strengths include our graduates' ability to work in the unique contexts of Hawaii and the Pacific, with an understanding of the cultural norms and values. PhD in Education graduates are knowledgeable about the needs of stakeholders in Hawaii and the Pacific and well-prepared to develop ongoing relationships with service providers and research organizations. They are passionate about the communities they serve and respectful of issues of diversity and inclusion in local and global contexts.

Findings and Results from Proposal and Dissertation Assessment Rubrics:

In our Program Committee discussion, the lower scores on Item #4 (theoretical frameworks) were a concern. Faculty discussed ways to include ongoing discussions about theoretical frameworks in coursework and in post-coursework experiences. We will examine course syllabi to ensure that theoretical frameworks are addressed in core research courses. We will also consider ways to address theoretical frameworks at various stages of the doctoral program to ensure that students have adequate knowledge of this topic prior to writing their proposals.

The Program Committee noted that students should receive feedback from their committees on specific areas to strengthen during their proposal defenses and that these changes should be reflected in their dissertations. In the future, the Program Committee will monitor the ratings of this item between the two points (proposal assessment and dissertation assessment) in order to evaluate whether there are changes in individual student scores on Item #4 (theoretical framework).

Findings and Results of Self Study:

When reviewing all assessment data for a self-study, the Program Committee discussed how we met standards for our professional accreditation with the Association for Advancing Quality Educator Preparation (AAQEP).  The PhD in Education program aligned with AAQEP Standard 1 in the following ways:

PhD in Education graduates work as educational researchers, scholars, and leaders. Our graduates conduct research studies contextualized within local, global, and international contexts and disseminate their research in varied national and international outlets, via publications and conference presentations. Our graduates are able to utilize their skills as researchers and advanced practitioners in varied professional roles appropriate for doctoral graduates (e.g., tenure track positions, school administrators, education non-profit leaders). Program faculty provide ongoing mentorship and support to graduates to enable them to advance in their careers (e.g., publishing, conferences, etc)

The assessment of program data for our self-study helped us identify areas we can strengthen in regard to collection of evidence. For example, we will more closely track information on specific aspects (e.g., collect information on students' knowledge of culturally responsive practices) and examine content of elective courses in the program. To provide additional student support, the program is in the process of implementing Individualized Development Plans (IDPs). IDPs are a tool for professional and scholarly growth, self-reflection, goal-setting and communication with advisors and mentors. IDPs can be one mechanism to provide ongoing support for students during the program.