Degree Program Assessment Reports

1. Knowledge.� Students will become knowledgeable in the broad issues of education and in the skills, knowledge and problems within their specialization area.� Students will develop an inquiring and critical approach to issues and possible solutions to problems in education.� The body of knowledge and specific objectives are developed on an individual basis in collaboration among the student, his/her advisor, and committee members. The doctoral program is unique in that students are expected to develop their knowledge of the field to the degree that they can understand the issues and problems in the field of education, and their specialization, in order to develop an inquiry project that will generate new knowledge in education.�

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 2. Demonstrate understanding of research methodology and techniques specific to one’s field of study.)

2. Research. Students will develop competencies in the broad issues of conducting and evaluating research in education, and develop the skills needed to develop a research problem and questions, design a study about a significant issue, collect appropriate data, analyze results and write a dissertation.� Students� dissertation research must be on an original problem and contribute to the knowledge base in education.� The results of students� research will be disseminated internationally, nationally and locally, where appropriate, so that it may be utilized for the improvement of education.

(3. Apply research methodology and/or scholarly inquiry techniques specific to one’s field of study., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study.)

3. Caring and Professional Ethics.� Students will develop a deep respect for the public trust that is invested in them as future intellectual and social leaders in the field of education.� Research that they conduct, or to which they refer in making recommendations, will be carefully vetted for accuracy, fairness, and beneficence regarding the clients, recipients, participants and the broader public good.� The program supports an attitude or disposition of caring or consideration for all people, and especially for students at every level and from various walks of life.

(5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

Curriculum Map File(s) from 2018:

• File (03/16/2020)

No
Yes, on some(1-50%) of the program SLOs
Yes, on most(51-99%) of the program SLOs
Yes, on all(100%) of the program SLOs

Create/modify/discuss program learning assessment procedures (e.g., SLOs, curriculum map, mechanism to collect student work, rubric, survey)
Collect/evaluate student work/performance to determine SLO achievement
Collect/analyze student self-reports of SLO achievement via surveys, interviews, or focus groups
Use assessment results to make programmatic decisions (e.g., change course content or pedagogy, design new course, hiring)
No (skip to question 17)
Investigate other pressing issue related to student learning achievement for the program (explain in question 7)
Other:

Artistic exhibition/performance
Assignment/exam/paper completed as part of regular coursework and used for program-level assessment
Capstone work product (e.g., written project or non-thesis paper)
Exam created by an external organization (e.g., professional association for licensure)
Exit exam created by the program
IRB approval of research
Oral performance (oral defense, oral presentation, conference presentation)
Portfolio of student work
Publication or grant proposal
Qualifying exam or comprehensive exam for program-level assessment in addition to individual student evaluation (graduate level only)
Supervisor or employer evaluation of student performance outside the classroom (internship, clinical, practicum)
Thesis or dissertation used for program-level assessment in addition to individual student evaluation
Alumni survey that contains self-reports of SLO achievement
Employer meetings/discussions/survey/interview of student SLO achievement
Interviews or focus groups that contain self-reports of SLO achievement
Student reflective writing assignment (essay, journal entry, self-assessment) on their SLO achievement.
Student surveys that contain self-reports of SLO achievement
Assessment-related such as assessment plan, SLOs, curriculum map, etc.
Program or course materials (syllabi, assignments, requirements, etc.)
Other 1:
Other 2:

Course instructor(s)
Faculty committee
Ad hoc faculty group
Department chairperson
Persons or organization outside the university
Faculty advisor
Advisors (in student support services)
Students (graduate or undergraduate)
Dean/Director
Other: Graduate Chair of Program

Tables 1 and 2 below present date from PhD in Education program assessments checked in #6 (Evaluation of Student Work/performance). Between Fall 2016-Fall 2018 (the period when this program assessment was in use), twenty-three students completed the product (proposal) that was assessed. Evaluation data was submitted for 22 students. Below, we provide information how SLO achievement in respect to each assessment item, including a summary of the PhD in Education program administrative committee's discussion and reflection on each item in relation to program assessment.

SLO #1: Knowledge (ILO #1 & 2)

The means of the items that assessed SLO#1: Knowledge ranged from 4.44 to 4.63 (on a 5 point scale).

The program administrative committee reviewed the data including open-ended comments for each item.  Faculty concurred that core courses were adequately preparing students with the knowledge required of a doctoral student, including a broad knowledge of issues in education and within specialization areas, and an understanding of how to frame a problem and develop an inquiry for a dissertation study.

SLO #2: Research (ILO #3 & 4)

The means of the three items that assessed SLO#2: Research ranged from 4.5 to 4.63 (on a 5 point scale).

The program administrative committee concurred that from the data it was clear that by the time students reached the proposal phase (written and oral), they had been given appropriate advising and feedback to succeed with this major doctoral program requirement. The students successfully integrated knowledge and skills gained in the three core courses required by the doctoral program and completed their proposals under the guidance of their advisor and committee.  Based on the open-ended comments, faculty noted that students were given the opportunity and training to apply a variety of frameworks and theories related to their specialization areas, for example indigenous methodologies.  Our college-wide program includes tracks with various disciplinary foci, and the program committee agreed that it was a strength of the program to provide core inquiry methods and allow for individualized and flexible

SLO #3: Caring and Professional Ethics (ILO #5, 6, 7)

To assess Caring and Professional Ethics, the program administrative committee reviewed open-ended comments written by faculty about students’ overall performance with the written and oral portions of the dissertation proposal.  Overall, the data showed that students were prepared and aware of ethical and professional considerations. The program committee agreed that a more targeted measure is required to collect data on SLO #3.  The committee will add an assessment to further evaluate SLO#3 in the future.

 

Table 1

ILO	SLO	Rubric Items	Rating Scale					N	MEAN SCORE
			1	2	3	4	5
ILO #1 & 2	SLO #1	1. Justification/Contribution to the Field	0	0	2	8	12	22	4.63
		2. Review of the Literature: Appropriateness/Relevance	0	0	1	11	10	22	4.44
		3. Review of the Literature: Synthesis/Analysis	0	0	1	10	11	22	4.5
		4. Theoretical Framework Basis	0	0	1	10	11	22	4.44
ILO #3 & 4	SLO#2	5. Purpose Statement/Research Questions	0	0	1	6	15	22	4.63
		6. Proposed Methodology	0	0	1	5	16	22	4.5
		7. Proposed Analysis	0	0	1	10	11	22	4.5

 

Table 2: Open-ended comments for each item

ILOs/SLOs	Sample of open-ended comments related to SLOs
ILO #1 & 2 SLO #1	Has the potential to provide a unique and valuable contribution to the field. Proposed research addresses an important and timely issues in science education. Finding from the proposed study have the potential to inform an expanded understanding of sustainability minded education in traditional science content areas such as Chemistry and Biology. The theories of Dewey and Greene were used and they were solid scholarly choices Strong case for study was made based on theory. Very thorough This is an important topic. No empirical study on it yet exists. Research adds to the literature regarding Chamorro sexual health for their youth.
ILO #3 & 4 SLO#2	Methodology is appropriate for the study and clearly articulated in the proposal. We appreciate that he had several options noted for how to gather participants as that is likely to be challenging. Narrative, Mo'olelo selected. This a very appropriate approach to her work. In the defense Methodology was a big part of the defense discussion. The original written methodology was weak and was a space for major revisions which has been addressed by the candidate.     [Student's] use of Hawaiian mele as a tool for analysis will make a unique and promising contribution to the growing field of Indigenous/Hawaiian research methodologies. The purpose and need for this research was perceived as strong and the results from the work will be a good addition to the field. After the defense the committee selected to require the candidate to engage in major revisions to the written text but did not require a new defense. Thus, the candidate passed her defense but spent considerable time revising the written proposal. The proposal was then re-reviewed by all committee members and approved by members as their individual and collective concerns were met. The analyses were appropriate to the methodology and should allow him to gain an array of data that should mesh well with the theoretical frame.
ILO #5, 6, 7 SLO #3	{Student's} committee was extremely impressed with his depth of knowledge and analysis on his proposal and oral comps. They have even encouraged him at this point to start publishing his work. [Student] was able to put very careful thought into her proposal. She listened to the advise of her committee and formulated a dissertation proposal that will hopefully yield very interesting results and contribute to the literature surrounding transformative teaching and place-based Hawaiian education.

 

 

a) Proposal Assessment Rubric: In our program committee discussion, we felt that our existing rubric does not fully capture whether students are meeting SLO#3. To better align the items to SLO#3, we will add an item to the rubric to collect evidence related to ethics/professional dispositions. Our committee will also continue discussions about adding an assessment product at the end of the program (dissertation assessment).

b) Based on the results of surveys of recent graduates and current students, we have added program supports for students who have completed coursework and are in the phase of working on research and writing independently.  We have implemented weekend “open writing” sessions, where students are invited to come and work on their writing at the College of Education. Program faculty are available at these sessions to provide guidance as needed.  In addition, the doctoral student organization has continued to expand on these opportunities with weekday writing sessions for students with their peers.