Assessment of Advising Learning Outcomes: Decision-making in Educational Planning

Colleges of Arts and Sciences – Student Academic Services University of Hawai‘i at Manoa

Background

The focus of this assessment project is the development of methods to better assess one of CASSAS main departmental goals for students: the identification, development, and implementation of “suitable” educational plans. Through the administration of a survey to CASSAS advisers on the advising goals outlined by the Council of Advancement of Standards, this goal was identified as being of highest priority. While it is recognized that others in the campus community contribute to the development of student educational planning, the information obtained from this project will provide the foundation on which CASSAS can strengthen its activities that contribute towards meeting this goal.

Learning Outcomes

We identified three learning outcomes, which were derived from CASSAS program objectives:

  • Self-reflection/Self-regulation (based on student knowledge and understanding of personal values; achievement motivation goals in response to intrinsic or extrinsic factors; and ongoing awareness of goal progress)
  • Information seeking strategies (based on student knowledge and understanding of higher education rationale; an awareness of and ability to interpret information at both a “global” and campus level, i.e., national trends, UHM curricular/co-curricular options, policies, procedures)
  • Decision-making skills in educational planning (based on using self-reflection/-regulation skills, motivation, and information-seeking strategies).

Because the third outcome — decision-making in educational planning — includes the sub-learning objectives of self-reflection/-regulation and use of appropriate information seeking strategies, we identified this third outcome as the primary learning outcome to be assessed for this project.

When we know the level of educational planning at which students are actually performing, advisers can continue or enhance their various teaching and advising practices, that increase student ability to: reflect on their interest and abilities, question their motivation and assumptions, research curricular and co- curricular fields of interest, and make decisions based on this self-reflection and critical analysis of information. The information gained with also contribute to the development of specific advising activities and services developed by individual academic advisors or by CASSAS in general.

Development of Assessment Measures

We conducted a thorough review of current assessment methods and instruments (journals, web, conferences, workshops, other institutional practices), both in the academic advising field as well as within the broader higher education arena. We found that most assessment studies/practices utilize institutional statistics (e.g., drop or graduation rates) or involve the use of instruments that primarily measure student perception of or satisfaction with a variety campus activities or services (e.g., CSEQ, NSSE, items that ask level of satisfaction with a course or instructor). Only one instrument, the Survey of Academic Advising (ACT, Inc. 1997), has been specifically designed to assess academic advising. However, this instrument measures student perceptions of academic advising services or adviser qualities and not student learning outcomes as a result of adviser interaction. In addition, this instrument is based on the narrow assumption that students have a single adviser with whom they work throughout their undergraduate careers. Of those instruments measuring actual student learning, general critical thinking instruments (CTI) were identified as being measurement tools that would be the most effective in studying decision-making learning outcomes. However, CTIs, while purporting to be content neutral,

Assessment Update 1 – UH Manoa CASSAS 1 3/06 L. Higa & M. Kirk-Kuwaye
seem to measure the area of symbolic logic rather than discipline based critical thinking. We concluded that the application of these types of instruments to the broad area of student educational planning was inappropriate.

We did find, however, the use of rubrics as having the potential to be an authentic and accurate method
of assessing decision-making activities in the development of educational plans. A rubric is a set of
criteria used to judge complex thinking processes, such as the criteria that expert raters use to judge the level of analysis students show in answering a literature question in an English Advance Placement exam question. This method of assessment appeared promising for measuring the advising goal we were investigating because it provides both an evaluation of actual student thinking and a way to systematically differentiate between responses (e.g., excellent, above average).

Developing and Refining Question Prompts

We conducted a preliminary test of different questions or prompts that would elicit responses that allow us to identify how students were progressing through the educational planning cycle. These prompts were tested on incoming freshmen as well as on a convenience sample of students on the Mall and at the Campus Center.

Once we had a framework of the educational planning cycle and how students engage in it, we refined the question prompt that would elicit students to demonstrate their level of planning. We approached students at our group advising sessions and those who were peer mentors in our advising centers. In this phase of the project, we not only asked students to respond to the prompt on educational planning, which they responded in writing, but also conducted focus group discussions after they completed their writing. The students indicated that the prompt should be concise, but with an option for more detailed information; that we should specify an “audience” to whom the student is writing; and that it would be interesting to do. Based on these comments, we refined the prompt in preparation for a larger field test.

Collecting Planning Statements for Rubric Development

We mailed out invitations to a randomly selected group of Arts and Sciences students to participate in a study on writing educational planning statements. To date, about 60 students responded and have been turning in their planning statements. In “debriefing” sessions conducted with students after they had turned in their statements, the students indicated that they found writing the statements useful. While they were not sure whether their responses were “right” (in spite of us telling them at the outset that there were no right or wrong answers), they felt that the process of writing about their educational plans made the planning process clearer in their minds. They were able to reaffirm where they were effective and see how they could improve, or in the case of graduating seniors, reflect back on their undergraduate experience. A positive response to the writing of the planning statement is critical because students will put more effort in writing, which will then be more revealing of their actual level of competency in planning.

Development of a Rubric on Student Decision-Making in Educational Planning

An “inductive approach” to rubric development was utilized. Members of the ad hoc CASSAS Advising Assessment Committee individually rated the statements by broad rating categories – good, fair, poor — and then met to discuss why they rated a statement in a certain way. From this discussion emerged patterns of competencies desired and levels of quality, and possibly the expansion of the categories to a five-point scale. We will determine specific guidelines for each category level, e.g., “In a ‘good’ statement, a student must address four of the six self factors.” The committee has finalized the performance criteria and levels of the rubric and is in the process of defining descriptors to demonstrate expected competency at the different levels.

Assessment Update- UH Manoa CASSAS 2 3/06 L. Higa & M. Kirk-Kuwaye