Unit: Languages, Linguistics, & Literature, College of
Program: Hawai'i English Language Program
Date: Fri Oct 10, 2014 - 7:17:42 pm

1) Below are your program's student outcomes (SOs). Please add or update as needed.

Program Learning Outcomes (PLOs)

While the HELP Mission Statement and program goals list several stakeholders, the primary focus is on the second language learner, many of whom are interested in future university studies while others are interested only in personal development. For both types of students, we provide the following Program Learning Outcomes:

  1. Effective Writing Skills

Students demonstrate the ability to write well-organized, well-developed, and well-supported paragraphs and essays of varying lengths and rhetorical styles at each level of the program.

 2. Effective Reading Skills

Students demonstrate the application of reading strategies to extract meaning and ideas from a variety of texts in different genres and at different levels of complexity as experienced at each level of the program.

 3. Effective Listening and Speaking Skills

Students demonstrate the ability to apply listening strategies to comprehend, interpret, and respond to discourse that is either non-participative or participative as well as produce spoken language to express themselves and their relationship with others in a variety of social and academic situations in developing stages at each level of the program.

 4. Grammar Competency 

Students demonstrate a solid foundation in the form, meaning, and use of grammatical structures in speaking, reading, and writing in increasing complexity at each level of the program.

 5. Personal and Social Responsibility     

Students demonstrate personal development in becoming autonomous lifelong learners who engage in self-assessment, critical thinking, ethical reflection, cooperative and collaborative teamwork, and participate in activities with the larger community both on and off campus.

2) Your program's SOs are published as follows. Please update as needed.

Program's Website. URL: http://manoa.hawaii.edu/eslhelp/wordpress/?page_id=28#OurMission
Student Handbook. URL, if available online:
Information Sheet, Flyer, or Brochure. URL, if available online:
UHM Catalog. Page Number:
Other: Teacher's Handbook
Other:

3) Provide the program's activity map or other graphic that illustrates how program activities/services align with program student outcomes. Please upload it as a PDF.

Activity Map File(s) from 2014:

4) Did your program engage in any program assessment activities between June 1, 2013 and September 30, 2014? (e.g., establishing/revising outcomes, aligning activities to outcomes, collecting evidence, interpreting evidence, using results, revising the assessment plan, creating surveys, etc.)

Yes
No (skip to question 14)

5) For the period between June 1, 2013 and September 30, 2014: State the assessment question(s) and/or assessment goals. Include the student outcomes that were targeted, if applicable.

1.  Do the Program Learning Outcomes (PLOs) truly reflect the aims of the program?

     Reviewing the last reported curriculum map, it was clear that we did not and, in fact, could not offer enough distinct        and separate courses in listening and speaking to justify maintaining separate learning outcomes. The question            was whether the PLOs for Active Listening Skills and Effective Oral Communications Skills needed to be merged or      whether new courses for listening needed to be added to the curriculum.

2.  Do the level outcomes use specific, measurable, and action-based verbs following Bloom’s Taxonomy?

     Last year’s feedback pointed out the need for more specificity in the wording of the level outcomes. The major                question is the degree of specificity in comparison with course outcomes.

3.  How can PLO 5: Personal and Social Responsibility be added to the level and course outcomes?

     While the values expressed in the outcome are highly desirable, they are difficult to instill in international students,        especially from Asian countries, for whom the concepts are largely foreign to them culturally and socially. A way to        introduce these values had to be developed.

4.  How can Service Learning, the vehicle for fulfilling “activities with the larger community both on and off campus” from PLO 5, be assessed?

     As the majority of the community activities provided to the students were one-time volunteer events, more than log        sheets of the time spent on an activity were needed to elevate the activity from simple volunteerism to service                learning. A better system of assessment is needed.

5.  Is the rating of the students’ achievement of outcomes consistent across levels and skills?

                   Since inaugurating the system of providing students with feedback from all of their teachers, there has not been an
                   analysis of the data.

6) State the type(s) of evidence gathered to answer the assessment question and/or meet the assessment goals that were given in Question #5.

The evidence collected was the final student assessment rating averages for all classes from Fall 1, 2012 to Summer 2, 2014.

7) State how many persons submitted evidence that was evaluated. If applicable, please include the sampling technique used.

27 teachers completed Student Final Assessment Reports between October 2012 and August 2014.

8) Who interpreted or analyzed the evidence that was collected? Check all that apply.

Program faculty/staff member(s)
Faculty/staff committee
Ad hoc faculty/staff group
Director or department chairperson
Persons or organization outside the university
Students (graduate or undergraduate)
Dean or Associate Dean
Advisory Board
Other: Assistant Director

9) How did he/she/they evaluate, analyze, or interpret the evidence? Check all that apply.

Compiled survey results
Used quantitative methods on student data (e.g., grades, participation rates) or other numeric data
Used qualitative methods on interview, focus group, or other open-ended response data
Scored exams/tests/quizzes
Used a rubric or scoring guide
Used professional judgment (no rubric or scoring guide used)
External organization/person analyzed data (e.g., Social Science Research Institute)
Other:

10) For the assessment questions/goals stated in Question #5, summarize the actual results.

1.  It was decided to merge the PLOs for Active Listening Skills and Effective Oral Communications Skills into Effective Listening and Speaking Skills. As much as we would like to offer separate courses for the two skills each term, it is not possible given the already heavy course load for full-time students. At present, full-time students take four classes per day with the following level requirements:

                  100-level: Four core courses – reading, writing, grammar, listening/speaking
                  200-level: Three core courses – reading, writing, grammar and one elective
                  300-level: Two core courses – reading, writing and two electives
                  400-level: (Spring & Summer) Two core courses – reading, writing and two electives
                                     (Fall) Three core courses – reading, writing, academic listening/speaking and one elective

For the electives, offerings are varied to meet the interests of both the academically and non-academically-bound. Although the course name may not include the words listening and speaking, it is our contention that all courses should be providing practice in both skills.

2.  In January 2014, a new set of Level Outcomes was drafted. The wording was left broad with the intention of providing much more specificity in the course learning outcomes.

                   400 LEVEL
                   When completing the 400 LEVEL (ready to begin study in the University of Hawai‘i ELI level), students should                      be able to:

1.  Write well-organized, well-developed, and well-supported academic writing, including the research paper, with attention to audience, the writing process as well as appropriate style, discourse conventions, and documentation. (PLO 1)

2. Effectively read, comprehend, summarize, and synthesize complex content from a variety of academic text and literary fiction as well as outline and efficiently annotate text. (PLO 2)

3. Effectively use strategies for comprehending advanced academic lectures and discussions; critically evaluate speakers’ perspectives, techniques, and arguments; and organize, synthesize and use information from notes for overall studies. In addition, demonstrate the ability to communicate fluently and spontaneously in informal and formal conversations, discussions, and presentations. (PLO 3)

4. Demonstrate the form, meaning, and use of grammatical structures by correctly and appropriately using complex grammatical patterns with increasing fluency in oral and written communication. (PLO 4)

5. Display academic values and readiness for content-area courses in an American post-secondary system by taking responsibility for one’s own learning, by engaging in cooperative work with classmates, and by demonstrating competence with information technology. (PLO 5)

                   300 LEVEL
                   When completing the 300 LEVEL (ready to enter the 400 Level), students should be able to:

1.  Compose multi-paragraph essays in a variety of rhetorical styles with proper organization and formatting as well as the use of a limited set of documentation/citations. (PLO 1)

2. Effectively read and extract meaning from multi-paragraph content in authentic text; identify the organizational nature of content and its main and supporting points; use schemata, grammatical and word clues to decode meaning; and take and use notes on content as well as efficiently annotate text.  (PLO 2)

3. Effectively use listening strategies for comprehending speech of moderate complexity and use signal words, symbols, and abbreviations in note-taking. In addition, appropriately and effectively use features of American English pronunciation as well as use appropriate organization, expressions, rhetorical structures, and non-verbal language to make presentations, handle questions, discuss issues, and communicate in varied situations. (PLO 3)

4. Demonstrate the form, meaning and use of verb tenses and other grammatical structures, including parts of speech, in writing, speaking, and reading with greater syntactic fluency and appropriateness. (PLO 4)

5. Use independent learning strategies, critical thinking, resources and information technology as well as display cooperative and collaborative teamwork with classmates and demonstrate tolerance and the ability to resolve conflicts. (PLO 5)

                   200 LEVEL
                   When completing the 200 LEVEL (ready to enter the 300 Level), students should be able to:

1.  Write well-formed and organized paragraphs and short essays in a variety of rhetorical styles to explore and express views of oneself, others, and the world. (PLO 1)

2. Adequately construct meaning from simplified and authentic print material on primarily familiar topics and increase reading speed with extensive reading activities. (PLO 2)

3. Effectively use listening strategies for comprehending speech on familiar topics and begin to use symbols and abbreviations in note-taking. In addition, effectively use features of American English intonation, stress, and rhythm as well as use appropriate organization, expressions, and non-verbal language to make presentations, discuss issues, and communicate in varied situations. (PLO 3)

4  Demonstrate the form, meaning and use of verb tenses and other basic grammatical structures, including parts of speech, in writing, speaking, and reading. (PLO 4)

5.  Use independent learning strategies, critical thinking, resources, and information technology as well as display cooperative and collaborative teamwork with classmates and demonstrate tolerance and the ability to resolve conflicts. (PLO 5)

                   100 LEVEL
                   When completing the 100 LEVEL (ready to enter the 200 Level), students should be able to:

1.  Produce well-formed simple and compound sentences to form paragraphs on familiar topics. (PLO 1)

2. Adequately construct meaning from simplified print material on primarily familiar topics and increase reading speed with extensive reading activities. (PLO 2)

3. Demonstrate comprehension of simplified spoken English passages and simple conversations as well as respond appropriately to simple instructions and requests. In addition, demonstrate accurate and comprehensible communication to describe basic needs and relate common activities, using clarification strategies when appropriate (PLO 3)

4. Demonstrate the form, meaning and use of simple and continuous verb tenses and other basic grammatical structures, including parts of speech, in writing, speaking, and reading. (PLO 4)

5. Begin to use some independent learning strategies, resources, and information technology as well as display cooperative and collaborative teamwork with classmates.  (PLO 5)

3.  PLO 5: Personal and Social Responsibility has not been consistently added whole or in part to all course learning outcomes because (1) some teachers continue to believe that all students are generally cooperative and collaborative in class and (2) because there are no materials in the course books to help in teaching critical thinking and autonomy, even though the words appear as exercise titles. The last component of the PLO on engaging with the community both on and off campus is addressed on the program level with mandatory Service Learning for all students.

      During the 2014 Spring Terms, graduate students/HELP instructors undertook research studies on aspects of critical thinking and autonomous learning, the results of which would inform future teachers on techniques to deal with these issues. Then, at the beginning of the current Fall Term, three graduate assistants held mandatory workshops, primarily for 300 and 400-level students, on critical thinking, plagiarism and paraphrasing, and presentation skills. These introductory sessions were designed to impress upon students the importance HELP places on autonomous learning and critical thinking. Follow-up work on the skills presented is being addressed, primarily in writing and listening/speaking classes.

     Work is still needed to determine the best techniques to lead students toward a better understanding of how to learn and how to engage with what they read.

4.  In order to make students reflect on the service projects or events they participate in, each student was required to give a short presentation on one service performed at a lunchtime Brown Bag Session. A number of students complained about the format as it was stressful and prevented them from enjoying their lunch. In the summer of 2014, the Brown Bag was changed from presenting to a large roomful of people to small group presentations, where several students stationed around a room could present to small groups of people. Students much preferred this format as they could spend five minutes listening to one presentation rather than 40 minutes for four or five.

          The current Fall Term 1 with 125 students and no long open period makes Brown Bag sessions impossible. Posting blog reflections was suggested as an alternative, but it will not get underway until Fall Term 2. An experiment that has started is curricular service learning, in which a course has service learning as a graded component of the course. A 200-level listening and speaking class is working with the Hawaii Foodbank as it corresponds with a unit on food and health.  This type of service learning is encouraged and the service learning coordinator will be working with teachers to see if one or two more classes can adjust their curriculum to accommodate a service project. Although it will mean more work for the teacher, curricular service learning should be more meaningful for the students, more easily assessed, and a wonderful extension to course themes or topics.

5.  Along with final grade reports, students receive Student Final Assessment Reports  (SFAR) from each of their teachers, which provide ratings of their achievement of course learning outcomes along with written feedback on strengths and weaknesses.  Rating averages place students at one of three levels of proficiency:

                      Developing 1 and 2 = performing below the level of the outcomes
                      Competent 3 and 4 = performing at the level of the outcomes
                      Exemplary 5 and 6 = performing above the level of the outcomes

HELP divides each level further into two grades, primarily for level promotion purposes. It is the goal of the teacher to ensure that students reach a competent level of performance. Competent students with a rating average of 3 are doing well but would definitely benefit from an additional term at the same level, whereas those with an average of 4 may be considered for promotion to the next level. There will often be one or two students still at the developing stage due to the lack of a foundational base for the current proficiency level or a lack of evidence of learning submitted by the student. Students at the exemplary level, who are also few in number, excel at everything they do and display evidence of learning that is beyond the current level. These students are definitely ready for promotion to the next proficiency level.

      All SFARs from Fall II 2012 to Summer II 2014 have been analyzed.  As predicted, for each term there is usually a bell curve with the majority of the students rated at the competent level for each skill course. For a few terms, there is a larger than expected number of students in the developing and exemplary stages.  It has been determined that three teachers rated 75-100% of their students at the exemplary stage in reading, writing, and grammar, while one teacher rated 83-100% of the students at the developing stage.  There has obviously been miscommunication about the rating criteria, which must be more clearly explained.

RATINGS BY PLO            
  100 Level 200 Level 300 Level 400 Level TOTAL STUDENTS % of Total
Effective Writing Skills            
Student Totals 51 217 206 52 526  
          Developing 24 26 39 6 95 0.180608365
          Competent 23 138 134 34 329 0.653846154
          Exemplary 4 53 31 12 100 0.230769231
             
Effective Reading Skills            
Student Totals 59 223 227 57 566  
         Developing 7 28 11 5 51 0.090106007
         Competent 47 183 183 28 441 0.779151943
         Exemplary 5 21 44 24 94 0.166077739
             
Effective Listening & Speaking Skills            
Student Totals 62 153 184 16 415  
         Developing 9 38 33 0 80 0.192771084
         Competent 42 92 101 13 248 0.597590361
         Exemplary 30 20 50 8 108 0.260240964
             
Grammar Competency            
Student Totals 46 217 128 25 416  
         Developing 25 36 10 0 71 0.170673077
         Competent 11 127 96 11 245 0.588942308
         Exemplary 10 54 21 14 99 0.237980769
             
Personal & Social Responsibility            
Student Totals 37 47 94 32 210  
        Developing 2 1 5 3 11 0.052380952
        Competent 12 21 54 21 108 0.514285714
        Exemplary 23 25 35 5 88 0.261363636

11) What was learned from the results?

Several points stand out from the activities done over the past year:

1.   The program needs not only to find a place in the curriculum for academic skills deemed necessary for success in higher education but also to find a way to convince students of their importance.  These skills and concepts include:

·     Plagiarism
·     Critical thinking
·     Citation and information literacy
·     Autonomous learning

2.     We need to better articulate our requirements of students in a clear but forceful manner

3.     We need to better inform our teachers about types of assessment and how such assessment, especially with rubrics, can aid in better assessing their students at the end of the term

12) State how the program used the results or plans to use the results. Please be specific.

Our plans for each question raised in #5 has been stated in the summary provided in #10.  It is imperative that the collection and review of evidence of learning be done in order to better help teachers with their assessments of students and their courses.

13) Reflect on the assessment process. Is there anything related to assessment procedures your program would do differently next time? What went well?

First and foremost we recognize the need to establish a regular schedule for the analysis of any data.  With 6- and 8-week terms throughout the year, there is no lack of assessment data.  What is lacking is in-depth analysis of this data with an eye towards implementing changes in the overall curricular map or student/program outcomes.  The major hindrance to so doing is the lack of personnel.  

14) If the program did not engage in assessment activities, please explain.
Or, if the program did engage in assessment activities, please add any other important information here.

As it is not apparent in either the PLOs or Curriculum Map, it should be noted that HELP is a non-credit, S-Funded ESL Program. Students are placed into one of the four HELP levels according to the results of placement tests – one a standardized placement test developed at the University of Michigan and two created in house. It is vanishingly rare for a student to enter at the 100 level and exit at the 400 level. As such, it is often the case that a student enters the program lacking mastery of all the learning outcomes at the level one below placement. For these reasons, we are unable to label any of our courses for mastery.