Program: Professional Educational Practice (EdD)
Degree: Doctorate
Date: Sat Sep 27, 2014 - 1:13:51 pm
1) Below are your program's student learning outcomes (SLOs). Please update as needed.
- Outcome One: Leaders in professional educational practice work collaboratively to solve problems and implement plans of action
- Outcome Two: Leaders in professional educational practice are able to apply research skills to bring about improvements in practice.
- Outcome Three: Leaders in professional educational practice can reflect critically and ethically on matters of educational importance.
- Outcome Four: Leaders in professional educational practice are able to take a broad, interdisciplinary perspective on a wide variety of educational issues
2) Your program's SLOs are published as follows. Please update as needed.
Department Website URL: https://coe.hawaii.edu/academics/educational-foundations/edd Student Handbook. URL, if available online: Link to Handbook at https://coe.hawaii.edu/academics/educational-foundations/edd Information Sheet, Flyer, or Brochure URL, if available online: Link to Brochure UHM Catalog. Page Number: Catalogue directs students to Course Syllabi. URL, if available online: Course syllabi are available on Laulima and in the student Handbook Other: Laulima. See MAN EdD Cohort 2 at https://laulima.hawaii.edu/portal/site/2da80ada-3b85-41c4-b19f-7b6efb078f94 Other: Program FAQ at https://coe.hawaii.edu/academics/educational-foundations/edd3) Select one option:
- File (03/16/2020)
4) For your program, the percentage of courses that have course SLOs explicitly stated on the syllabus, a website, or other publicly available document is as follows. Please update as needed.
0% 1-50% 51-80%
81-99% 100%5) Did your program engage in any program assessment activities between June 1, 2013 and September 30, 2014? (e.g., establishing/revising outcomes, aligning the curriculum to outcomes, collecting evidence, interpreting evidence, using results, revising the assessment plan, creating surveys or tests, etc.)
Yes No (skip to question 14)6) For the period between June 1, 2013 and September 30, 2014: State the assessment question(s) and/or assessment goals. Include the SLOs that were targeted, if applicable.
Doctorate In Professional Educational Practice
Assessment 1: Group Consultancy Project
2012-13 Update
A. Data Table: Summary of Final Assessment of Group Consultancy Project and Report—Spring 2012
|
Requirement |
|
Minimal |
Acceptable |
Target |
|---|---|---|---|---|
|
Objective One: Work collaboratively to solve problems and implement plans of action |
||||
|
Collaboration and individual contribution to project |
|
0 |
2 |
26 |
|
Objective Two: Application of research to bring about improvements in practice |
||||
|
Application of research methods to project |
|
0 |
5 |
23 |
|
Review of literature |
|
0 |
3 |
25 |
|
Data Collection and Analysis |
|
0 |
3 |
25 |
|
Objective Three: Reflect critically and ethically on matters of educational importance |
||||
|
Critical and ethical reflection on research |
|
0 |
0 |
28 |
|
Objective Four: Broad interdisciplinary perspective on project |
||||
|
Presentation of findings and executive report |
|
0 |
0 |
28 |
B. Brief Analysis of Findings and Next Steps:
The consultancy project in the EdD is a collaborative assignment in which a team of five doctoral students works together on an evaluation project or do research on a matter of educational interest or concern at a particular school or complex. Each group works with and for a client under the supervision of UH faculty advisors and field mentors. Group members consult with the school or school complex administration and faculty to study a problem of practice, in depth. The project is advisory. The task of the group in taking on an issue is to examine the nature of the program or project; explore the wider issues that may inform the project; gather and analyze data; and make a report of conclusions and recommendations based on the evidence.
As indicated in the client assessments that were requested at the completion of these projects, the students in Cohort I produced reports that were considered of value in providing recommendations that would inform future decision making. These assessments are reflected in the above chart. Students were highly collegial in their work and worked closely with the clients to complete their assessments.
One area for improvement is to ensure a more equal contribution from each of the group members in future cohorts.
The high ratings in this assessment have a lot to do with the excellence of the students in the cohort and their years of experience as educators. The group of 28 were selected from over 70 EdD applicants.
Program Name
Assessment 2: Midterm Assessment of Dissertations in Practice
2012-13 Update
A. Data Table:
Midterm Assessment of Dissertation in Practice
EdD—Cohort I
Standard |
Minimal0 |
Accept-able1 |
Target2 |
|---|---|---|---|
|
Objective One Work Collaboratively to solve problems and implement plans of action |
1 |
8 |
19 |
|
Objective Two Application of research to bring about improvements in practice |
1 |
12 |
15 |
|
Objective Three Reflect critically and ethically on matters of educational importance |
1 |
6 |
21 |
|
Objective Four Take a broad interdisciplinary perspective on the project |
1 |
9 |
18 |
B. Brief Analysis of Findings and Next Steps:
The above Chart represents progress towards completion of the dissertation in practice according to the four measures that represent the program objectives. The score do reflect some variability among the raters (6 faculty advisors and 8 field mentors), and although we did discuss the criteria beforehand we should expect some variability as this is the first time that we have used this rubric.
In most cases the students are well on their way to completing the program on schedule. However, there may be a few (5 students) who will need an extra semester to finish their work. One student has fallen behind a bit more because of a problem with his project that was outside his control.
Assessment 3: Dissertations in Practice and Program Conferce
Completed August, 2014
Assessment Summary
|
Requirement
|
Minimal |
Acceptable |
Target |
|
Quality of writing |
The report is poorly written, unorganized and contains spelling, punctuation, and grammatical errors. |
The quality of writing is acceptable. The report is coherent and contains only a few spelling, punctuation, and grammatical errors. |
The report is well written. It conforms to APA style throughout. As a whole it is well organized, shows logical consistency, and is free of spelling, punctuation and grammatical errors. |
|
Assessment |
|
1 |
22 |
|
Research Questions |
Research question is poorly defined and inappropriately in relation to the action project. Poorly aligned with literature review. |
Research questions are clearly articulated, but the relationship between questions, data collection, and analysis are unclear. |
The project was clearly articulated and the research questions concisely stated. |
|
Assessment |
|
|
23 |
|
Action Plan |
Action plan was not well formulated and poorly implemented. Lack of data. |
Action plan was carried out and actions taken were sufficient to answer research questions and support conclusions. |
The action plan was well thought out, thoroughly organized, and effectively implemented with attention to research questions |
|
Assessment |
|
6 |
17 |
|
Data Collection |
Data sources are limited, insufficient information to support aims of project. Inadequate information about data collection procedures. |
Data sources are appropriate and data collection methods sufficiently described |
Appropriate methods were followed and the data collected provided valuable project information.
|
|
Assessment |
|
7 |
16 |
|
Data Analysis |
Methods of analysis are not explained. Failure to include all data collected. Poor interpretation of data |
Methods of analysis are appropriate, though other methods may have been employed to generate conclusions. |
Analysis of data was insightful and provided useful implications for practice. |
|
Assessment |
|
3 |
20 |
|
Findings |
Findings are presented in an unorganized fashion. Little interpretation of data, and/or conclusions presented that are unrelated to data. |
Findings are presented that connect with the data, but are incomplete. |
Findings are well organized and consistent with research questions and data. |
|
Assessment |
|
3 |
20 |
This assessment focused on two of the program SLOs: application of research skills to bring about improvements in practice. and critical and ethical reflection on matters of educational importance, but by targeting specific aspects of the research such as data collection and analysis, and the application of research to practical educational probems.
7) State the type(s) of evidence gathered to answer the assessment question and/or meet the assessment goals that were given in Question #6.
Assessment 1: Individual assessments of the group consultancy reports and client presentations. Faculty assessments were made in collaboration with field mentors. They involved progress monitoring (eg. IRB application, timely submission of proposal and drafts, executive report and report to clients) and answered the questions—Did the student work collaboratively in their group? Did student contribute fully to the project? Did student review the appropriate literature? Did student participate in collection and analysis of data? Did they engage in critical and ethical reflection on the project? Did they contribute to the executive report and presentation to clients?
Assessment 2. Individual assessment of student's progress towards their dissertation in practice according to the four program SLOs. At this stage each student was expected to have completed a dissertation proposal, to have obtained IRB approval for their research project, other approvals (DOE Data Governance approval, charter or private school approval). In addition, students made a presentation of their research project progress during fall semester 2013 attended by UH faculty and program mentors. Assessment was directed to determining progress on the four program SLOs. Did student work collaboratively to solve problems and implement plans of action? Did student employ research methods to bring about improvements in practice? Did student reflect critically and ethically on issues raised by their research? Did student take a broad interdisciplinary perspective on the project?
Assessment 3. This assessment focused on two of the program SLOs: application of research skills to bring about improvements in practice. and critical and ethical reflection on matters of educational importance, but by targeting specific aspects of the research such as data collection and analysis, and the application of research to practical educational probems. It answered the question that the students were applying appropriate research methods in for their respective problems of practice, that they applied data analysis to thesepropblems and that they were reflecting critcally on their work to arrive at specific findings.
8) State how many persons submitted evidence that was evaluated. If applicable, please include the sampling technique used.
All twenty-eight students in the cohort were assessed according to the first 2 assessments (rubrics). All assessments involved collaboration between faculty advisors and program mentors.
Assessment 1 involved the submission of a group report, an executive report, and a client presentation.
Assessment 2 involved the submission of dissertation proposal, IRB approval, and progress report to faculty at a formal presentation in Fall 2013
Twenty-three students were assessed using the final assessment (Five have elected to finish their disssertations in Fall 2014).
Assessment 3 involved the submission of a completed dissertation in practice to Graduate Division during the summer of 2014 and a conference presentation that was attended by the faculty, fellow students, and public and held at the EW Center on July 15,16, and 17 2014.
9) Who interpreted or analyzed the evidence that was collected? (Check all that apply.)
Course instructor(s) Faculty committee Ad hoc faculty group Department chairperson Persons or organization outside the university Faculty advisor Advisors (in student support services) Students (graduate or undergraduate) Dean/Director Other: Program field mentors. Each student was assigned to one of 9 field mentors who each worked with one of the five faculty advisors throughout the three years of the program.10) How did they evaluate, analyze, or interpret the evidence? (Check all that apply.)
Used a rubric or scoring guide Scored exams/tests/quizzes Used professional judgment (no rubric or scoring guide used) Compiled survey results Used qualitative methods on interview, focus group, open-ended response data External organization/person analyzed data (e.g., external organization administered and scored the nursing licensing exam) Other: 11) For the assessment question(s) and/or assessment goal(s) stated in Question #6:
Summarize the actual results.
2012-13 Update
A. Data Table: Summary of Final Assessment of Group Consultancy Project and Report—Spring 2012
|
Requirement |
|
Minimal |
Acceptable |
Target |
|---|---|---|---|---|
|
Objective One: Work collaboratively to solve problems and implement plans of action |
||||
|
Collaboration and individual contribution to project |
|
0 |
2 |
26 |
|
Objective Two: Application of research to bring about improvements in practice |
||||
|
Application of research methods to project |
|
0 |
5 |
23 |
|
Review of literature |
|
0 |
3 |
25 |
|
Data Collection and Analysis |
|
0 |
3 |
25 |
|
Objective Three: Reflect critically and ethically on matters of educational importance |
||||
|
Critical and ethical reflection on research |
|
0 |
0 |
28 |
|
Objective Four: Broad interdisciplinary perspective on project |
||||
|
Presentation of findings and executive report |
|
0 |
0 |
28 |
Midterm Assessment of Dissertation in Practice
EdD—Cohort I
Standard |
Minimal0 |
Accept-able1 |
Target2 |
|---|---|---|---|
|
Objective One Work Collaboratively to solve problems and implement plans of action |
1 |
8 |
19 |
|
Objective Two Application of research to bring about improvements in practice |
1 |
12 |
15 |
|
Objective Three Reflect critically and ethically on matters of educational importance |
1 |
6 |
21 |
|
Objective Four Take a broad interdisciplinary perspective on the project |
1 |
9 |
18 |
Assessment 3—Dissertation in Practice and Conference Presentation
Assessment Summary
|
Requirement
|
Minimal |
Acceptable |
Target |
|
Quality of writing
|
|
1 |
22 |
|
Research Questions
|
|
|
23 |
|
Action Plan
|
|
6 |
17 |
|
Data Collection
|
|
7 |
16 |
|
Data Analysis
|
|
3 |
20 |
|
Findings
|
|
3 |
20 |
12) State how the program used the results or plans to use the results. Please be specific.
Assessment results have been used to make program changes for the second cohort of students. For example, we have lengthened the period of time devoted to the group consultancy project to reflect experience in obtaining human studies approval. Some student experienced difficulties with obtaining DOE approval so we have arranged to work with the DOE so that the consultancy projects arre covered by a Memorandum of Understanding between the COE and DOE. In addition, the sequence of courses has been changed so that greater emphasis is made to qualitative research methods earlier in the program.
13) Beyond the results, were there additional conclusions or discoveries?
This can include insights about assessment procedures, teaching and learning, program aspects and so on.
The EdD program are assigned to two large problems of practice projects over a three year period. As a result, assessments are not simply derived from faculty evaluations at critical points in the program. The consultancy project, for example, involves working with clients who also provide feedback on the students' work. In addition, the dissertation in practice is a participant inquiry project involving the sharing of results at a conference, so feedback and developmental evaluation processes are built into the program and go well beyond mere summative appraisals (M.Q. Patton, 2011). So additional forms of assessment provide important data for program changes and innovation. For example, clients were polled to provide additional feedback on the consultancy projects.
The were asked: What was the value of the project to your organization? What are the short-term and potential long-term benefits of the consultancy report for your organization? What actions have you taken to meet the recommendations of the report? If you were to repeat the experience with the consultancy group what would you do differently to improve the process (proposal submission, meetings, report presentations, etc.).
The main recommendation gleaned from the client assessments relates to timing and the need for a longer period of time for preparation of the report. This recommendation has been addressed in Cohort II of the program.
Assessment is not always about making program changes; it can also provide useful information about what is going right in the program. The cleint assessments provide useful feedback on the practical value of the consultancy project. For example, in one project that studied the financial sustainability of small independent schools, HAIS responded that they were putting plans in place to discuss the benchmarks and assessment process identified in the consultancy report with the HAIS Accreditation Committee for inclusion in the accreditation process. Another client reporting on the Mililani Character Education Program consultancy project wrote that the schools in the complex were working on building stronger family connections with the program in the schools.
An important source of feeback has been the regular meeting of the program advisory council. The council discuss issues as they arise. This developmental evalution model facilitates program innovation. For example, council discussions of faculty advisors, mentors, and students has helped to claify the respective roles of mentors and advisors and has promoted collaboration and communication between groups. The council has also brought issues about content to the fore and we have been able to remedy these in the Saturday meetings. Students requested additional instruction in qualitative research methods and this was provided in a timely way. Also, we arranged for presenters to demonstrate the use of various online tools and software for the collection and analysis of data.