Unit: Education (multiple departments)
Program: Education (PhD)
Degree: Doctorate
Date: Fri Nov 02, 2018 - 3:25:03 pm

1) Program Student Learning Outcomes (SLOs) and Institutional Learning Objectives (ILOs)

1. Knowledge.� Students will become knowledgeable in the broad issues of education and in the skills, knowledge and problems within their specialization area.� Students will develop an inquiring and critical approach to issues and possible solutions to problems in education.� The body of knowledge and specific objectives are developed on an individual basis in collaboration among the student, his/her advisor, and committee members. The doctoral program is unique in that students are expected to develop their knowledge of the field to the degree that they can understand the issues and problems in the field of education, and their specialization, in order to develop an inquiry project that will generate new knowledge in education.�

(1. Demonstrate comprehensive knowledge in one or more general subject areas related to, but not confined to, a specific area of interest., 2. Demonstrate understanding of research methodology and techniques specific to one’s field of study.)

2. Research. Students will develop competencies in the broad issues of conducting and evaluating research in education, and develop the skills needed to develop a research problem and questions, design a study about a significant issue, collect appropriate data, analyze results and write a dissertation.� Students� dissertation research must be on an original problem and contribute to the knowledge base in education.� The results of students� research will be disseminated internationally, nationally and locally, where appropriate, so that it may be utilized for the improvement of education.

(3. Apply research methodology and/or scholarly inquiry techniques specific to one’s field of study., 4. Critically analyze, synthesize, and utilize information and data related to one’s field of study.)

3. Caring and Professional Ethics.� Students will develop a deep respect for the public trust that is invested in them as future intellectual and social leaders in the field of education.� Research that they conduct, or to which they refer in making recommendations, will be carefully vetted for accuracy, fairness, and beneficence regarding the clients, recipients, participants and the broader public good.� The program supports an attitude or disposition of caring or consideration for all people, and especially for students at every level and from various walks of life.

(5. Proficiently communicate and disseminate information in a manner relevant to the field and intended audience., 6. Conduct research or projects as a responsible and ethical professional, including consideration of and respect for other cultural perspectives., 7. Interact professionally with others.)

2) Your program's SLOs are published as follows. Please update as needed.

Department Website URL: https://coe.hawaii.edu/node/4932
Student Handbook. URL, if available online:
Information Sheet, Flyer, or Brochure URL, if available online:
UHM Catalog. Page Number:
Course Syllabi. URL, if available online:
Other:
Other:

3) Please review, add, replace, or delete the existing curriculum map.

Curriculum Map File(s) from 2018:

4) For your program, the percentage of courses that have course SLOs explicitly stated on the syllabus, a website, or other publicly available document is as follows. Please update as needed.

0%
1-50%
51-80%
81-99%
100%

5) Does the program have learning achievement results for its program SLOs? (Example of achievement results: "80% of students met expectations on SLO 1.")(check one):

No
Yes, on some(1-50%) of the program SLOs
Yes, on most(51-99%) of the program SLOs
Yes, on all(100%) of the program SLOs

6) Did your program engage in any program learning assessment activities between June 1, 2015 and October 31, 2018?

Yes
No (skip to question 17)

7) What best describes the program-level learning assessment activities that took place for the period June 1, 2015 to October 31, 2018? (Check all that apply.)

Create/modify/discuss program learning assessment procedures (e.g., SLOs, curriculum map, mechanism to collect student work, rubric, survey)
Collect/evaluate student work/performance to determine SLO achievement
Collect/analyze student self-reports of SLO achievement via surveys, interviews, or focus groups
Use assessment results to make programmatic decisions (e.g., change course content or pedagogy, design new course, hiring)
No (skip to question 17)
Investigate other pressing issue related to student learning achievement for the program (explain in question 7)
Other:

8) Briefly explain the assessment activities that took place.

1. Developed and implemented a rubric; program committee reviewed results

  • In the 2015-2016 academic year, the program administrative committee developed a rubric to assess the quality of components of the dissertation proposal (written proposal and oral defense) related to the three program SLOs.

  • In Fall 2016, the dissertation proposal assessment rubric was implemented and faculty were asked to use this assessment tool to evaluate their advisees' written proposals and oral proposal defenses.

  • In Fall 2018, the program administrative committee reviewed data collected from the 2016-2018 program assessments (proposal assessment rubric) and discussed areas of strengths and needs for improvement based on the results.

2. Program Completer Survey (Recent Graduates) and Post-Coursework Survey (Current Students)

  • The College of Education administers program completer surveys to all graduates. Our PhD in Education program administrative committee reviews the results to gain an understanding of graduates’ perceptions of program components.

  • In Fall 2017, based on our review of program completer surveys from prior years, we decided to we need to further examine the supports needed by students after they finish coursework. In their comments, students noted the need for support during the independent phase of developing their proposals, implementing their research studies. and completing their dissertations, activities related to SLOs #2 and #3 As a result, we developed and implemented a survey to administer to current students to gain a better understanding of student needs.

9) What types of evidence did the program use as part of the assessment activities checked in question 7? (Check all that apply.)

Artistic exhibition/performance
Assignment/exam/paper completed as part of regular coursework and used for program-level assessment
Capstone work product (e.g., written project or non-thesis paper)
Exam created by an external organization (e.g., professional association for licensure)
Exit exam created by the program
IRB approval of research
Oral performance (oral defense, oral presentation, conference presentation)
Portfolio of student work
Publication or grant proposal
Qualifying exam or comprehensive exam for program-level assessment in addition to individual student evaluation (graduate level only)
Supervisor or employer evaluation of student performance outside the classroom (internship, clinical, practicum)
Thesis or dissertation used for program-level assessment in addition to individual student evaluation
Alumni survey that contains self-reports of SLO achievement
Employer meetings/discussions/survey/interview of student SLO achievement
Interviews or focus groups that contain self-reports of SLO achievement
Student reflective writing assignment (essay, journal entry, self-assessment) on their SLO achievement.
Student surveys that contain self-reports of SLO achievement
Assessment-related such as assessment plan, SLOs, curriculum map, etc.
Program or course materials (syllabi, assignments, requirements, etc.)
Other 1:
Other 2:

10) State the number of students (or persons) who submitted evidence that was evaluated. If applicable, please include the sampling technique used.

a) Dissertation Proposal Assessment (n=22)

Sampling Technique: Rubric used for all students who defended a proposal between 2016-2018

b) Program Completer Survey completers (n=40)

Sampling Technique: Survey sent to recent graduates each semester, 2015-2018

c) Post-Coursework Survey (n=47)

Sampling Technique: Survey sent to all currently enrolled PhD students, Fall 2017

 

11) Who interpreted or analyzed the evidence that was collected? (Check all that apply.)

Course instructor(s)
Faculty committee
Ad hoc faculty group
Department chairperson
Persons or organization outside the university
Faculty advisor
Advisors (in student support services)
Students (graduate or undergraduate)
Dean/Director
Other: Graduate Chair of Program

12) How did they evaluate, analyze, or interpret the evidence? (Check all that apply.)

Used a rubric or scoring guide
Scored exams/tests/quizzes
Used professional judgment (no rubric or scoring guide used)
Compiled survey results
Used qualitative methods on interview, focus group, open-ended response data
External organization/person analyzed data (e.g., external organization administered and scored the nursing licensing exam)
Other:

13) Summarize the results of the assessment activities checked in question 7. For example, report the percentage of students who achieved each SLO.

Tables 1 and 2 below present date from PhD in Education program assessments checked in #6 (Evaluation of Student Work/performance). Between Fall 2016-Fall 2018 (the period when this program assessment was in use), twenty-three students completed the product (proposal) that was assessed. Evaluation data was submitted for 22 students. Below, we provide information how SLO achievement in respect to each assessment item, including a summary of the PhD in Education program administrative committee's discussion and reflection on each item in relation to program assessment.

SLO #1: Knowledge (ILO #1 & 2)

The means of the items that assessed SLO#1: Knowledge ranged from 4.44 to 4.63 (on a 5 point scale).

The program administrative committee reviewed the data including open-ended comments for each item.  Faculty concurred that core courses were adequately preparing students with the knowledge required of a doctoral student, including a broad knowledge of issues in education and within specialization areas, and an understanding of how to frame a problem and develop an inquiry for a dissertation study.

SLO #2: Research (ILO #3 & 4)

The means of the three items that assessed SLO#2: Research ranged from 4.5 to 4.63 (on a 5 point scale).

The program administrative committee concurred that from the data it was clear that by the time students reached the proposal phase (written and oral), they had been given appropriate advising and feedback to succeed with this major doctoral program requirement. The students successfully integrated knowledge and skills gained in the three core courses required by the doctoral program and completed their proposals under the guidance of their advisor and committee.  Based on the open-ended comments, faculty noted that students were given the opportunity and training to apply a variety of frameworks and theories related to their specialization areas, for example indigenous methodologies.  Our college-wide program includes tracks with various disciplinary foci, and the program committee agreed that it was a strength of the program to provide core inquiry methods and allow for individualized and flexible

SLO #3: Caring and Professional Ethics (ILO #5, 6, 7)

To assess Caring and Professional Ethics, the program administrative committee reviewed open-ended comments written by faculty about students’ overall performance with the written and oral portions of the dissertation proposal.  Overall, the data showed that students were prepared and aware of ethical and professional considerations. The program committee agreed that a more targeted measure is required to collect data on SLO #3.  The committee will add an assessment to further evaluate SLO#3 in the future.

 

Table 1

ILO

SLO

Rubric Items

Rating Scale

N

MEAN SCORE

1

2

3

4

5

ILO #1

& 2

SLO #1

1. Justification/Contribution to the Field

0

0

2

8

12

22

4.63

2. Review of the Literature: Appropriateness/Relevance

0

0

1

11

10

22

4.44

3. Review of the Literature: Synthesis/Analysis

0

0

1

10

11

22

4.5

4. Theoretical Framework Basis

0

0

1

10

11

22

4.44

ILO #3

& 4

SLO#2

5. Purpose Statement/Research Questions

0

0

1

6

15

22

4.63

6. Proposed Methodology

0

0

1

5

16

22

4.5

7. Proposed Analysis

0

0

1

10

11

22

4.5

 

Table 2: Open-ended comments for each item

ILOs/SLOs

Sample of open-ended comments related to SLOs

ILO #1 & 2

SLO #1

  • Has the potential to provide a unique and valuable contribution to the field.
  • Proposed research addresses an important and timely issues in science education. Finding from the proposed study have the potential to inform an expanded understanding of sustainability minded education in traditional science content areas such as Chemistry and Biology.
  • The theories of Dewey and Greene were used and they were solid scholarly choices
  • Strong case for study was made based on theory. Very thorough
  • This is an important topic. No empirical study on it yet exists.
  • Research adds to the literature regarding Chamorro sexual health for their youth.

ILO #3 & 4

SLO#2

  • Methodology is appropriate for the study and clearly articulated in the proposal. We appreciate that he had several options noted for how to gather participants as that is likely to be challenging.
  • Narrative, Mo'olelo selected. This a very appropriate approach to her work. In the defense Methodology was a big part of the defense discussion. The original written methodology was weak and was a space for major revisions which has been addressed by the candidate.
  •     [Student's] use of Hawaiian mele as a tool for analysis will make a unique and promising contribution to the growing field of Indigenous/Hawaiian research methodologies.
  • The purpose and need for this research was perceived as strong and the results from the work will be a good addition to the field. After the defense the committee selected to require the candidate to engage in major revisions to the written text but did not require a new defense. Thus, the candidate passed her defense but spent considerable time revising the written proposal. The proposal was then re-reviewed by all committee members and approved by members as their individual and collective concerns were met.
  • The analyses were appropriate to the methodology and should allow him to gain an array of data that should mesh well with the theoretical frame.

ILO #5, 6, 7

SLO #3

  • {Student's} committee was extremely impressed with his depth of knowledge and analysis on his proposal and oral comps. They have even encouraged him at this point to start publishing his work.
  • [Student] was able to put very careful thought into her proposal. She listened to the advise of her committee and formulated a dissertation proposal that will hopefully yield very interesting results and contribute to the literature surrounding transformative teaching and place-based Hawaiian education.

 

 

14) What best describes how the program used the results? (Check all that apply.)

Assessment procedure changes (SLOs, curriculum map, rubrics, evidence collected, sampling, communications with faculty, etc.)
Course changes (course content, pedagogy, courses offered, new course, pre-requisites, requirements)
Personnel or resource allocation changes
Program policy changes (e.g., admissions requirements, student probation policies, common course evaluation form)
Students' out-of-course experience changes (advising, co-curricular experiences, program website, program handbook, brown-bag lunches, workshops)
Celebration of student success!
Results indicated no action needed because students met expectations
Use is pending (typical reasons: insufficient number of students in population, evidence not evaluated or interpreted yet, faculty discussions continue)
Other:

15) Please briefly describe how the program used the results.

a) Proposal Assessment Rubric: In our program committee discussion, we felt that our existing rubric does not fully capture whether students are meeting SLO#3. To better align the items to SLO#3, we will add an item to the rubric to collect evidence related to ethics/professional dispositions. Our committee will also continue discussions about adding an assessment product at the end of the program (dissertation assessment).

b) Based on the results of surveys of recent graduates and current students, we have added program supports for students who have completed coursework and are in the phase of working on research and writing independently.  We have implemented weekend “open writing” sessions, where students are invited to come and work on their writing at the College of Education. Program faculty are available at these sessions to provide guidance as needed.  In addition, the doctoral student organization has continued to expand on these opportunities with weekday writing sessions for students with their peers.

16) Beyond the results, were there additional conclusions or discoveries? This can include insights about assessment procedures, teaching and learning, and great achievements regarding program assessment in this reporting period.

In the past two years, we gained insights on how to more effectively collect assessment data in a timely manner. Our program is unique because it is college-wide and participating faculty are from various across the college. To make it easier for faculty to submit assessment data, we modified a paper-based rubric and created an online form for submission of data. Faculty can submit this in a more timely fashion and our program office is better able to monitor the regular collection of data.

We are currently in the process of considering adding another program assessment at the dissertation phase to evaluate students’ mastery of SLOs at the end of their program. The program administrative committee will be discussing this in Spring 2019.

 

17) If the program did not engage in assessment activities, please justify.