Degree Program Assessment Reports

• File (03/16/2020)

The Department of Natural Resources and Environmental Management (NREM) gathered evidence to address the three assessment questions through four primary activities: (i) curriculum review that centered on the needs of the students to meet professional expectations in priority areas; (ii) an inclusive exercise with the Manoa Assessment Office to evaluate and rewrite the SLOs for the three graduate degree programs in NREM (Ph.D., M.S. Plan A, and M.S. Plan B); (iii) creation of a new curriculum map for each graduate degree program based on the revised SLOs; and (iv) departmental development, lead by the NREM Curriculum Committee, of a revised, interdisciplinary, team-taught, project-based core course to be required of all graduate students (M.S. and Ph.D.) during their first year in NREM.

1) Are NREM graduate degree program SLOs appropriate?

(a) In Spring of 2012, NREM received funding support from the College of Tropical Agriculature and Human resources (CTAHR) to undertake an overall curriculum review with the following objectives:

· Understand the academic and curricular needs of current and prospective NREm graduate students

· Assess how the NREM graduate program is addressing the needs and expectations of current and prospective students

· Ensure that NREM graduates have the knowledge and skills needed, as determined by NREM faculty, to be effective managers, successful professionals, and respected researchers

· Map out priority areas for both NREM faculty and students

To achieve these objectives, NREM undertook the following activities:

· Review of other NREM-related programs in other states (program goals/objectives, degree requirements – number of required core courses and electives, required core courses)

· Compilation and review of syllabi of NREM-based courses as well as popular electives from other departments taken by NREM graduate students

· Curriculum review survey of current students

(b) In August of 2013 the NREM faculty met for a half-day faculty retreat. During ~2 hours of the retreat, the NREM Curriculum Committee asked the Assessment Office (Monica Stitt-Bergh and Yao Zhang Hill) to lead an inclusive exercise to evaluate and rewrite the SLOs for the three graduate degree programs in NREM (Ph.D., M.S. Plan A, and M.S. Plan B). To our knowledge, this was the first time that the graduate degree program SLOs had been evaluated since the inception of the NREM department in the early 2000s. This was accomplished by projecting the existing SLOs onto a large screen, breaking the faculty in groups of 3-4, assigning them an existing SLO, and asking them to discuss and rewrite the SLO to better capture the skills and knowledge that NREM expects all students to have upon graduating. These modified SLOs were then placed on the screen under the original SLOs, and discussed and modified amongst the entire faculty. The NREM Curriculum Committee was then tasked with refining the final language for the SLOs, which was presented to the NREM faculty at a NREM Departmental Meeting IN early Fall 2013, where they were voted on and accepted (unanimous support). These new SLOs were used in last year’s Annual Assessment Report, and are presented again in this year’s report. The NREM faculty as a whole feels that the new SLOs are a large improvement in that they better articulate expectations of students graduating from our degree programs and, for the first time, better differentiate expectations across the three degree programs.

2) Do current departmental graduate courses adequately address the new graduate degree program SLOs developed in August 2013? In particular, do the required core courses adequately address SLOs not covered by elective courses?

(a) Upon completion of the new graduate degree program SLOs, the NREM Curriculum Committee undertook an exercise throughout the Fall 2013 semester to align course content with degree program SLOs, with the goal of creating a new curriculum map for each graduate degree program (Ph.D., M.S. Plan A, and M.S. Plan B).  The first step taken was for the Curriculum Committee to create new curriculum maps for each degree program based on our understanding of the content of the elective and core courses in the department. These documents were then presented at a NREM Departmental Meeting, and were made available to all NREM faculty via online google documents. Faculty were encouraged (via email reminders) to update the curriculum maps on the online documents for the classes they teach over the following 4-6 weeks, after which they were presented again at a NREM Departmental Meeting, and finalized. The new curriculum maps for the three graduate degree programs in NREM are presented for the first time in this Annual Assessment Report. The primary outcome from this exercise was the identification that a primary SLO (#1) is not currently being adequately covered in core or elective coursework in NREM. As a result, the NREM Curriculum Committee began the process of creating a proposal for a completely modified core course package for the department (see next section).

3)  If current core and elective courses do not adequately cover the existing SLOs, how can the required core courses be modified to better address SLOs?

After conducting the above two analysis processes, it became apparent that current core and elective courses do not adequately cover the revised SLOs, particularly SLO #1 requiring knowledge of both social and ecological issues and interdisciplinary aspects of NREM issues.  In order to address this gap, the department began in Spring 2014 to engage all NREM faculty in development of a new interdisciplinary, team-taught, and project-based core course to be required of all graduate students (M.S. and Ph.D.) during their first year in NREM.  Led by NREM’s curriculum committee, beginning in Spring 2014 NREM engaged in an extensive process of feedback and course development:  

(i) Review and discussion of need for a revamped core course

Activities and results of the assessments described above (student survey, along with development of new SLOs and Curriculum Map) were reviewed and discussed at seven curriculum committee meetings during spring semester, one graduate student organization meeting, as well as three departmental faculty meetings.  These latter discussions included asking faculty members and graduate students to describe pros and cons of the course, and to discuss questions, concerns and suggestions.  Two faculty meetings also allowed faculty to provide feedback and final approval of eight focal areas for the course, as well as a conceptual diagram developed by the curriculum committee to organize these focal areas and conceptualize the overall course.  Minutes of all of these meetings were transcribed and analyzed to create a more in-depth feedback and course development process.

(ii) Development of course focal areas and generation of critical topics in each focal area

Topics within each focal area for the course were developed through three different approaches. 

o   Small group meetings: Held in spring 2014 (including 4/21, 4/23, and 4/29) with individual faculty and graduate students, these meetings were attended by a total of 20 individuals.  At these informal meetings focal areas were identified, and topics within focal areas were brainstormed.  Participants were also asked to raise and discuss opportunities and challenges of such a course, as well as some key themes.  Two curriculum committee members attended and facilitated each session and met individually with some faculty members who couldn’t attend group sessions.    

o   Google Document:  The curriculum committee created an interactive google document with separate tabs for each focal area.  This allowed individual faculty to suggest and comment on topics on their own time without attending additional meetings.  Faculty were encouraged to suggest topics in every focal area, and not just those within their core area of expertise.  Participants were also asked to suggest cross cutting themes for the course on a separate tab.    

o   Core Curriculum Revamp Workshop (May 8, 2014): A half-day workshop was organized and run by the NREM Curriculum Committee to get more intensive feedback and faculty engagement in honing the core course (objectives and the meeting agenda are available upon request).  The meeting began with a short PowerPoint to review the assessment activities leading up to that point, and set out key elements of course structure (number of credits, teachers etc.) based on prior input and the need to make the course more concrete.  Focal areas and the conceptual diagram were also reviewed.  Faculty then worked in disciplinary groups on two – three focal areas of their choice.  Their task was to organize, group and refine all of the topics that had been proposed within the focal area.  The curriculum committee took each group’s feedback and reorganized topics and further refined and organized them as needed based on each group’s directions and notes.  The last part of the meeting focused on addressing some of the course challenges, and misgivings raised by faculty and students in earlier stages of gathering feedback. Four – five key challenges were written on chart paper and posted around the room and participants were given markers and asked to write solutions for each.  Finally, each participant completed an exit survey suggesting themes, case study projects, their envisioned contribution/role in the new course, and their feedback on the meeting and overall process.  Faculty members’ offered contributions ranging from teaching the entire course, to leading a small group project, to guest lecturing, to presenting & developing case studies, to helping to create instructional materials.

(iii) Departmental Retreat (August 21, 2014)

A final workshop was held as part of the NREM Departmental retreat on August 21, 2014.  The goal of this workshop was to prioritize topics within each reorganized focal area.  This goal emerged from past feedback raising the concern that trying to cover too much in the course would compromise depth and student’s ability to learn from the class.  Honed topics for each focal area emerging from the prior workshop (May 8, 2014) were printed in large font, and taped to poster papers posted around the meeting room.  Each faculty member and department staff were given sticker dots and asked to vote for the most important 3-5 topics (varied depending on the number of choices) within each focal area.  Cross cutting themes also received prioritization votes. Participants were not allowed to use all of their dots in one focal area or on one topic, so that participants couldn’t weight the voting for their personal area of research or specialty, and so that people had to make decisions about what they felt was important in disciplines outside their own.  Votes were recorded and topics within each focal area were ranked.

The meeting ended with a short exercise in which faculty members and staff formed teams to work on an assigned course theme applied to a project of their choice.  They were asked to brainstorm project sites, potential stakeholders and partners, project outcomes, and final products while also considering which topics could be covered within different focal areas through the projects. 

Combined, the activities described above to develop a new, required core course in NREM which meets departmental SLOs have succeeded in engaging NREM faculty, students, and staff, raising excitement for and awareness of the new course.  These activities have also taken an abstract and slightly overwhelming concept, and transformed it into something far more concrete and ready for a team of professors (to be identified by the NREM Department Chair in the very near future) to guide to implementation.  

Link to figures: NREM_Assessment Report_Figures_2014_09-30-14.pdf

1) Are NREM graduate degree program SLOs appropriate?

The student survey was constructed using NREM program materials drawn from the catalog and website. Thirty eight graduate students responded to the survey, or ~50% of the NREM graduate student body at the time (10 M.S. Plan A, 14 M.S. Plan B, and 14 Ph.D.). Nearly half of respondents (41%) entered the program in 2010 and the majority of the remaining respondents entered in 2009 or 2011 (Fig. 1; All Figures available as an appendix).  Less than half of students (42%) agreed or strongly agreed with the statement that an NREM education/current curriculum adequately prepared them to solve contemporary resource use and environmental problems to assist in sound policy development and implementation. M.S. Plan A students were evenly split (50% agreed, 30% neutral and 20% disagreed), but M.S. Plan B students were quite negative (75% either neutral or disagree), and Ph.D. students were bi-modal (36% disagree, 43% agree). Students felt particularly strongly that the core courses were not useful (Fig. 2).

M.S. program: When asked specifically about the content, rigor, and relevance of the individual required core courses NREM 600 (Evaluation of NREM), 601 (Economic Analysis of NREM), 605 (Research Skills), and 701 (Research Seminar), M.S. students had mixed opinions (Fig. 3). Many students felt that NREM 600 was relevant, but that the content and rigor was below par. There was a difference between the two programs (Plan A and B) – 35% of M.S. Plan B students thought 600 (Evaluation of Natural Resource Management) had very poor content, very low rigor, and no relevance. Turning to 601 (Economic Analysis) and 605 (Research Methods), student ratings of the content improved, although M.S. Plan A students rated 601 lower than Plan B students did in all three categories, and in both courses, as well as NREM 701 (Seminar), students thought the rigor was low. The capstone experience was very important for most M.S. Plan B students. Nearly everyone thought it was at least somewhat relevant for their preparation as NREM professionals, and the vast majority thought it was very relevant.

Ph.D. program: When asked specifically about the content, rigor, and relevance of the individual required core courses NREM 605, 701, 611 (Resource and Environmental Policy), and 612 (Predicting and Controlling Degradation in Human-Dominated Landscapes), Ph.D. students gave mixed reviews of NREM 611 and 701, but 612 rated highly across all categories (Fig 4). In fact, one student commented that she thought NREM 612, which was “well structured and organized” should be the core course for M.S. students rather than NREM 600, which was “a hodgepodge of information” irrelevant to most incoming graduate students.

Based on reviews of other programs, we asked students about the number of course/credit requirements. The vast majority (75%) agreed that the quantity of core courses was “about right”, and this opinion was consistent across degrees. A majority (56%) of students thought the number of elective courses were “about right”. M.S. Plan A students were split evenly (1/3 too little, 1/3 about right, 1/3 too much), M.S. Plan B students were content (80% about right, 10% each too much and too little), and Ph.D. students were generally positive (54% about right, 15% too much, 31% too little). A large majority of students across degrees (63%) felt that the quantity of NREM-based elective courses was too sparse, and courses were desired in a number of key fields (Table 1). Many students felt the ability to take courses outside of NREM, according to their individual interests, was a critical positive feature of NREM. Identified course shortcomings surrounded professional skills for M.S. Plan B students in particular (management, planning, environmental laws and regulations), and interdisciplinary and social science, policy development and implementation, and marine and coastal issues. 

NREM strives to prepare students with technical knowledge and skills, as well as professional skills. Using language from NREM materials, we formulated questions to gage student perception about their attainment. Results show that students feel the NREM program is stronger in some technical areas (e.g., biological and physical sciences and economics) than others (e.g., institutions, social science, policy) (Fig. 5), and some professional skills particularly important for NREM careers could be strengthened (e.g., systems integration, interdisciplinary methods, statistics and ethics) (Table 2). In addition, some students called for more field skills, project management skills, and contact with NREM professionals during the seminar to talk about career paths.

In terms of the outcomes of an NREM education, we asked students the degree to which NREM prepared them to undertake a number of things they will be required to do as professionals (Fig. 6). It appears that the NREM curriculum is better at helping students identify problems than solve them and/or translate their science to managers, policy makers, and planners. Most students felt that NREM helped them refine their career goals (53%), while only 40% felt the NREM program was meeting expectations (34% strongly disagreed or disagreed) and only 42% felt NREM’s education/current curriculum adequately prepared them to solve contemporary resource use and environmental problems to assist in sound policy development and implementation.

We cannot interpret this survey, done in 2012 with a fairly low response rate (~50%), as representative of all graduate students, nor should we base program revisions on opinions of current graduate students who are buried in courses and may not yet have the professional perspective to judge their relevance. However, these results clearly revealed some issues that NREM is currently trying to address through core course modifications and expansion of elective course offerings.

2) Do current departmental graduate courses adequately address the new graduate degree program SLOs developed in August 2013? In particular, do the required core courses adequately address SLOs not covered by elective courses?

No. The primary outcome from the development of a new Curriculum Map based on the newly developed degree program SLOs was the identification that a primary SLO (#1) is not currently being adequately covered in core or elective coursework in NREM. As a result, the NREM Curriculum Committee began the process of creating a proposal for a completely modified core course package for the department (see other sections).

3) If current core and elective courses do not adequately cover the existing SLOs, how can the required core courses be modified to better address SLOs?

The NREM Curriculum Committee generated a proposal for a new, required core course including course format, faculty composition, number of credits, organizing interdisciplinary themes, focal areas to be covered, prioritized topics and skills within each focal area, and potential project / case study ideas, along with preferred contributions of each faculty member in the department to the new course (See question 12).  

Being on the NREM Curriculum Committee is a lot of work.