Degree Program Assessment Reports

In preparing for re-accreditation, we undertook the following assessment activities:

1. Revised the MURP curriculum map and aligned it with an assessment plan. 

2. Conducted three surveys -- current students, recent graduates (last 2-5 years), and employers -- to gather data about the MURP program and its alignment with industry needs. In April 2019, the Department surveyed 121 students who graduated from the MURP degree program between August 2012 and December 2018, inclusive of those two points in time. E-mail addresses were available for 114 graduates. Of those graduates, 50 responded with valid responses to the survey. The 50 respondents could be considered a representative sample because they included MURP graduates from each year of the review period. Of the 50 respondents, 22 reported that they had graduated between 2 and 5 years ago. The survey provided data on: #1 Graduate Satisfaction; #2 Graduate Service to Community and Profession; and #3 Graduate Employment. The DURP ‘Ohana (alumni association) Accreditation Committee administered an employer survey in June 2019. The survey included 48 organizations (public, private, non-profit) and received a 33% response rate. Its main purpose was to assess the skills and attributes of MURP graduates in the last seven years to examine how the master's degree is meeting the needs of the firms/organizations that employ them. 

3. Developed assessment tools (e.g. rubric, client evaluation form) for core courses in the MURP program. 

SLO 1: Explain major planning paradigms and their applications

Faculty members, who are on student capstone committees, scored two criteria from the capstone assessment rubric for SLO 1: "The student demonstrates working knowledge of the stated planning sub-field," and "the capstone document presents a clear analysis and cohesive argument/logic." Average scores for these two criteria were calculated for all students who submitted capstones between Fall 2015 and Fall 2018. The maximum score for each criterion was 5. Students received an average of 3.3 for “demonstrating working knowledge of the stated planning sub-field,” and an average of 3.0 for “presenting a clear analysis and cohesive argument/logic.” Thus, for all students who submitted capstones during this review period, combining the two criteria for assessing SLO 1 yielded an average score of 3.2 (out of the maximum of 5). 

SLO 2: Articulate processes leading to urbanization and rationales for planned interventions

Using several questions on a mid-term exam, 47 students enrolled in PLAN 600: Public Policy and Planning Theory were evaluated in Fall 2017 (23) and Fall 2018 (24) to assess the attainment of SLO 2. Four of these exam questions related to the urbanization process (numbers 1a, 1b, 1c, and 3) and five related to planning history (2, 4, 5, 6 and 7). The maximum possible score in each set of questions was 4.5. In Fall 2017, we found that students correctly answered 64% of the questions in the set pertaining to understanding urbanization processes, for an average total score of 2.7; likewise, with 80% correct answers, the average total score on questions to test the understanding of planning history was 3.8. Combining both scores yielded an average score of 3.3 for SLO 2. In Fall 2018, 62% correct answers on understanding urbanization processes produced an average total score of 2.8, and, for understanding planning history, the respective figures were 78% and 3.7. Combining the two total scores indicated that the Fall 2018 average score for SLO 2 was 3.3. 

SLO 3: Apply planning methods to organize, analyze, interpret and present information

To assess the achievement of SLO 3, in Fall 2017 and Fall 2018, the instructor analyzed responses to Question 25 on the mid-term exam in PLAN 601: Planning Methods: “Big Island residents were polled regarding their attitudes toward tourism expansion on their island. The data were broken down by location of residence. The purpose is to determine if there is an association between location of residence and attitude towards tourism expansion at the significance level α = 0.05.” Of the 11 students in Fall 2018, the performance of 73% demonstrated achievement of SLO 3 because, of a maximum possible score of 12 points, the average score was 9.4. This was a marked improvement over Fall 2017 when SLO 3’s fulfillment was met by only 39% of the 18 students, and the average score was 8.2 points. The difference in the results could be due to changes in the course design that introduced small group discussion to assist students in understanding key concepts in this course.

A rubric in PLAN 678: Site Planning also contributed to assessing SLO 3. A team of external jurors evaluated student teams for the following:

Concept: articulation of concept for developing the site and its link to the existing context.

Site plan: translation of the concept into a program of land uses.

Quality: transition from site analysis and synthesis of site conditions, market assumptions, and stakeholder involvement to the conceptual site plan; graphic and oral communication.

External jurors, four in Fall 2016 and five in Fall 2017, gave 11 teams (34 students in all) the following average scores: 4.5 and 4.6 (out of a maximum of 5) for concept; 14 and 15.5 (out of 20) for site plan; 7.4 and 7.8 (out of 10) for graphic communication; and 4.6 and 4.6 (out of 5) for oral communication. The scores were supported by qualitative evaluation of the strengths and weaknesses of students’ final assignments. Overall, the results revealed that students did well in articulating a concept and presenting it verbally. Design development and graphic communication, however, had scope for improvement. Although we typically provide an intensive graphic communication workshop featuring Adobe Photoshop, InDesign, and Illustrator in the course curriculum, efforts are ongoing to develop and offer a 1-credit course on graphic communication for planners during the summer session. 

SLO 4: Critically and creatively develop planning inquiries or processes to foster solutions-oriented decision-making 

A client assessment form for PLAN 751: Planning Practicum was piloted in Fall 2018 and administered again in Fall 2019. The results indicate that clients were very satisfied with student performance, their professionalism, and the quality of project deliverables. They agreed with students' recommendations and found them useful.  

SLO 5: Effectively collaborate as a planning team to work with a client and/or stakeholders to assess and address a relevant planning problem to create a plan or professional report

In Fall 2018, PLAN 751: Planning Practicum introduced a questionnaire that allowed students to self-assess their ability to address a planning problem. Each student team was asked to first discuss and then describe two aspects of the project they deemed successful, and two that they felt they could have approached or executed better. They were also asked to describe what they would do differently if they were to do the project again. The results highlighted aspects of the project (process and outcomes) students found challenging such as preparing appropriate outreach materials and enhancing stakeholder participation.   

SLO 6: Effectively present oral and written work (as a plan, professional report, or research paper) in a coherent, persuasive and professional manner

Two criteria from the capstone assessment rubric inform SLO 6: “Overall, the Capstone document is of professional quality” and “Overall, the presentation is of professional quality.” For all 239 students who submitted their capstones between Fall 2015 and Fall 2018, calculating the average score for each criterion (maximum 5 points) and then computing the average for both criteria produced the score for SLO 6. On average, students scored 3.2 for presenting a professionally written document and 3.3 for making a professional oral presentation, which gave an average score of 3.2 for SLO 6. For the academic year 2017-2018, specifically, the aggregate score for SLO 6 was 4.3; 4.2 for presenting a professionally written document and 4.4 for making a professional oral presentation.

SLO 7: Reflect upon the ethical implications of the choices planners make as professionals

Question 21b of the mid-term exam in PLAN 601 helped in testing the achievement of SLO 7. The assessment evaluated the grades of all 11 students enrolled in the course during Fall 2017 and 18 students enrolled in Fall 2018. A maximum of 4 points could be earned for a correct answer to Question 21b, which asked: “Who is the researcher similar to in hypothesis testing: the defense attorney or the prosecuting attorney? Why? [Please limit your answers to 150 words].” With an average score of 3.6 out of a maximum 4, the sensitivity sought by SLO 7 was demonstrated by 73% of the students in Fall 2017; in Fall 2018, the average score was 3.1 and 67% of the students attained SLO 7. Fall 2018 also saw assessment measures for SLO 7 pilot tested in PLAN 751: Planning Practicum. Upon completing the project, students submitted one to two-page reflection statements, which the instructor qualitatively analyzed using a rubric that categorized them as suggesting: non-reflection, understanding, reflection, and critical reflection. Of the 14 students evaluated, 57% demonstrated reflection and critical reflection. Their reflection ranged from description and recording of experiences (low degree) to questions about the fundamental nature of planning (high degree). The results led to introducing more readings and inviting professional planners to speak on professional ethics to emphasize its importance in practice.

We used the findings to prepare our re-accreditation report. The Planning Accreditation Board site visit team stated "The Student Learning Outcomes Assessment was particularly rigorous with attention being given to course revisions necessary to improve outcomes. The quality of this work clearly requires a significant commitment of resources from the Department. As being carried out, it could serve as a model for other programs" (April, 2020).