Unit: Languages, Linguistics, & Literature, College of
Program: English Language Institute
Date: Fri Oct 14, 2011 - 4:09:17 pm

1) List your program's student outcomes.

Student Learning Outcomes

ELI Listening & Speaking Curriculum Area

After successful completion of ELI Listening & Speaking courses, students will be able to:

  • demonstrate effective use of strategies for: comprehending advanced academic lectures in English; critically evaluating speakers’ perspectives, techniques, and arguments; and incorporating information from academic lectures into their overall studies
  • make academic presentations (individually or in group or panel contexts) with a high degree of formal accuracy and cultural and stylistic appropriacy
  • autonomously lead academic discussions using academic English, and demonstrate effective use of advanced strategies for participation in academic discussions with expert users of English
  • self-assess their strengths in terms of listening/speaking abilities, identify areas for continued development, and state a range of strategies to address those areas

Student Learning Outcomes

ELI Reading Curriculum Area

After successful completion of ELI Reading courses, students will be able to:

  • select reading and note-taking strategies appropriately for a range of different academic reading tasks, in accordance with courses they are enrolled in as well as their own purposes for reading advanced academic English texts
  • evaluate authors’ messages, perspectives, techniques, and arguments
  • evaluate print and web-based sources
  • self-assess their strengths in terms of reading abilities, identify areas for continued development, and state a range of strategies to address those areas

Student Learning Outcomes

ELI Writing Curriculum Area – Undergraduate Level

After successful completion of ELI Writing courses, undergraduate students will be able to:

  • fluently generate sufficient written text, at the brainstorming and drafting stages of the writing process, in response to a writing assignment
  • Compose college writing that achieves a specific purpose and responds adeptly to an identifiable audience.
  • Provide evidence of effective strategies for generating, revising, editing, and proofreading a text in order to produce finished prose.
  • Compose an argument that makes use of source material that is relevant and credible and that is integrated in accordance with an appropriate style guide.

Working definitions for SLO #2 (by committee representing ELI and the English Department – the two programs offering FW):

  • Revising: making global changes (e.g., adding, deleting, or moving content; rewriting for a different audience; rewriting in a different tone)

  • Editing: making changes at the sentence level, including changes in sentence style, syntax, phrasing

  • Proofreading: correcting grammar, punctuation and mechanics, spelling, formatting

Student Learning Outcomes

ELI Writing Curriculum Area – Graduate Level

After successful completion of ELI Writing courses, graduate students will be able to:

  • fluently generate sufficient written text, at the brainstorming and drafting stages of the writing process, in response to a writing assignment
  • analyze discipline and genre-specific academic English writing conventions and effectively apply that knowledge to graduate level writing tasks
  • self-assess their strengths as academic writers, as well as areas for continued development, incorporating personal reflection and feedback from others
  • appropriately incorporate a variety of reliable information sources that are relevant for doing graduate-level research in their academic writing

2) Where are your program's student outcomes published? Mark all that apply and include URLs when appropriate.

Program's Website. URL: http://www.hawaii.edu/eli/students/SLO.html
Student Handbook. URL, if available online:
Information Sheet, Flyer, or Brochure. URL, if available online:
UHM Catalog. Page Number:
Other:
Other:

3) Provide the program's activity map or other graphic that illustrates how program activities/services align with program student outcomes. Please upload it as a PDF.

Activity Map File(s) from 2011:

4) For the period June 1, 2010 to September 30, 2011: State the assessment question(s) and/or assessment goals. Include the student outcomes that were targeted, if applicable.

Three assessments were carried out during the period in question, labeled A, B, & C hereafter.

A)    Ethnographic study of two content courses commonly taken by ELI undergraduate students.

  • What content courses at UHM are commonly taken by undergraduate students in the ELI?
  • What classroom and assessment practices are followed in these courses?
  • What recommendations can be made related to the ELI’s curriculum?

B)    Follow-up assessment of SLO #3 for FW, conducted by the Assessment Office, involving student writing from ELI 100, ENG 100, ENG 100A, and ENG 190 (all the courses that fulfill the FW requirement at UH Manoa).

Addresses SLO #3: Students will be able to compose an argument that makes use of source material that is relevant and credible and that is integrated in accordance with an appropriate style guide.

C)   Evaluation of the ELI listening/speaking curriculum via a survey of ELI “alumni”.

  • What do ELI alumni (students who have completed all ELI requirements) see as the listening/speaking tasks and requirements in their UHM content courses?
  • Do any of these tasks or requirements identify potential student needs that are not included in the SLOs for the listening/speaking curriculum area?
  • How effective were ELI courses in meeting the SLOs and preparing students for handling the tasks and requirements they identified?

5) State the type(s) of evidence gathered to answer the assessment question and/or meet the assessment goals that were given in Question 4.

A)    Ethnographic study of two content courses commonly taken by ELI undergraduate students.

  • Computational analysis of total UHM registration of 200 ELI undergraduate students (via STAR database) to determine the most commonly taken non-ELI courses.

  • Observation of two courses (ECON 130 and HWST 107), with documentation of classroom and assessment practices.

  • Interviews with the instructors of the two courses observed, to supplement and confirm findings from observations.

B)    Assessment of SLO #3 for FW.  The Assessment Office provided the ELI with names of students, randomly selected, for each section of ELI 100, and the ELI provided the Assessment Office with papers written by the three students. Readers from ENG and ELI were invited to be raters, and used a rubric developed by the Assessment Office (and approved by ENG and ELI) to score the papers on four factors derived from the SLO (“makes use of source material”, “relevancy of sources”, “credibility of sources”, and “style integration”).  Raters were trained by faculty from the Assessment Office.

C)   Evaluation of the ELI listening/speaking curriculum via a survey of ELI “alumni”.

  • Interviews with ELI program administrators about specific areas of focus for the study.

  • Focus-group meetings with teachers of ELI listening/speaking courses to discern how they implement the SLOs for ELI 70 and 80.

  • Students who had successfully completed ELI 70 and ELI 80 between Fall 2008 and Summer 2010 were invited to complete an online questionnaire addressing the types of academic listening and speaking tasks and requirements they experienced since completion of ELI 70 and/or 80, their impressions of the effectiveness of ELI listening/speaking courses, a self-assessment of their abilities in terms of ELI listening/speaking learning outcomes, and an opportunity for the alumni to make additional comments.

[The interviews with administrators and focus-group sessions were used to design the survey taken by ELI alumni.]

6) State how many persons submitted evidence that was evaluated. If applicable, please include the sampling technique used.

A)    Ethnographic study of two content courses commonly taken by ELI undergraduate students.  Two instructors allowed the researcher to interview them and observe their courses.

B)    Follow-up assessment of SLO #3 for FW.  Due to unavailability of a sufficient number of reader/raters, this assessment has not yet been completed (further reading/rating will be done in November 2011).  Data for this assessment will be included in the 2012 ELI Assessment Report.

C)   Evaluation of the ELI listening/speaking curriculum via a survey of ELI “alumni”. 62 respondents at both the undergraduate and graduate level, representing 14 native languages other than English.  Email invitations were sent to 150 ELI alumni, so the response rate was 41.33%.

7) Who interpreted or analyzed the evidence that was collected. Check all that apply.

Program faculty/staff
Faculty/staff committee
Ad hoc faculty/staff group
Director or department chairperson
Persons or organization outside the university
Students (graduate or undergraduate)
Dean
Other (Different people analyzed data for A, B & C studies.)

8) How did he/she/they evaluate, analyze, or interpret the evidence? Check all that apply.

Compiled survey results
Used qualitative methods on interview, focus group, open-ended response data
Scored exams/tests/quizzes
Used a rubric or scoring guide
Used professional judgment (no rubric or scoring guide used)
External organization/person analyzed data (e.g., Social Science Research Institute)
Other: Ethnographic interpretation of observation and interview data. (Different methods for A, B & C studies.)

9) For the assessment questions/goals stated in Question 4, summarize the actual results.

A)    Ethnographic study of two content courses commonly taken by ELI undergraduate students.

  • The most commonly taken courses by undergraduates who were also in the ELI during the academic year studied were ECON 130, HWST 107, REL 150, CHEM 161, ECON 131, OCN 201 and MATH 100.

  • In the two courses observed (with supplementary interviews of instructors), there was a great deal of variation in instructional techniques and materials employed, ranging from a class of over 100 students that primarily involved lectures that incorporated PowerPoint slides and use of the board to illustrate examples in one course, to another course of 35 students that involved a range of audio/visual support and prompts, such as PowerPoint slides, pictures, movies, songs, maps, cartoons, newspaper articles and scholarly articles with a mix of fast-paced lectures and discussion.

  • A number of recommendations were provided to the ELI related to adjustments to existing activities, projects, and materials, or creation of new ones.  There were different recommendations that applied to each curriculum areas in the ELI.

B)    Follow-up assessment of SLO #3 for FW.  Due to unavailability of a sufficient number of reader/raters, this assessment has not yet been completed (further reading/rating will be done in November 2011).  A summary for this assessment will be included in the 2012 ELI Assessment Report.

C)   Evaluation of the ELI listening/speaking curriculum via a survey of ELI “alumni”.

  • The highest ranked listening and speaking requirements in non-ELI courses were (a) understanding the structure and organization of lectures, (b) oral interaction with native English speakers, (c) giving formal speeches/presentations, (d) taking notes, and (e) participating in small group discussions.

  • There is a high correlation between the listening/speaking requirements identified by ELI alumni and the SLOs of ELI 70 and 80.  While the ELI courses do not specific address “oral interaction with native English speakers”, the courses do emphasize discussion skills (given the nature of ELI courses -- being for non-native speakers of English -- it’s not feasible to provide in-class opportunities for all students to have individual interaction with native speakers on a regular basis, although occasional visits by expert lecturers, etc., are possible).

  • The five highest rated activities from ELI courses were (a) oral interaction with other international students, (b) participating in small group discussions, (c) leading small group discussion, (d) giving formal speeches/presentation, and (e) understanding the structure and organization of lectures. Three of these items exactly match the recognized needs (understanding the structure and organization of lectures, giving formal speeches/presentations, and participating in small group discussions); one item was closely related (the need for oral interaction with native English speakers vs. the provision of oral interaction with other students who were second-language speakers of English). The only highly ranked need that was not highly rated in terms of coverage in ELI courses was “taking notes”.

10) What was learned from the results?

A)    Ethnographic study of two content courses commonly taken by ELI undergraduate students.  While ELI courses raise student awareness about English for academic purposes, provide a variety of helpful academic strategies, and help students become more familiar with English as it is used at UHM, the recommendations from this project point to three important areas where the ELI should revisit its curriculum:

  • Techniques and materials that capitalize on various technologies.  Instructors in UHM courses are incorporating a variety of technologies in their courses. ELI courses could provide tasks and strategies designed to help the students extract what’s relevant from these different sources in ways that are appropriate and effective for each source.

  • Common genres for tasks and projects. There are a number of genres used in undergraduate courses that may not be getting enough exposure in ELI courses (e.g., reaction papers for non-text sources such as songs or video, or PowerPoint handouts as a common reading genre). Introducing these genres and strategies for successfully working with them could be an important addition to the ELI curriculum.

  • Tasks and projects that help students make better connections between their ELI courses and their content courses.  ELI courses could devote blocks of time to activities that allow students to act as ethnographers of their own content courses (i.e., by bringing in recordings, lecture notes, handouts of PowerPoint slides, etc. to the ELI course, where the class could analyze them and discuss strategies for maximizing learning.

B)    Follow-up assessment of SLO #3 for FW.  Due to unavailability of a sufficient number of reader/raters, this assessment has not yet been completed (further reading/rating will be done in November 2011).  Results for this assessment will be included in the 2012 ELI Assessment Report.

C)   Evaluation of the ELI listening/speaking curriculum via a survey of ELI “alumni”.

  • There are many ways that the current ELI listening/speaking curriculum matches well with the needs students have in their other UHM courses (including the focus on discussion skills, academic interaction, presentation skills, and strategies for lecture listening).

  • More emphasis could be placed on note taking. Considering the results of the concurrently researched ethnographic study (Item A in this Report), perhaps curriculum development could be done on extracting important information into various forms of notes, depending on the genre in which they are presented.

  • By the time they finish ELI 80, students have become quite comfortable in leading and participating in discussions where their interlocutors are all non-native speakers of English who are cooperative and patient when it comes to negotiating mutual understanding, but much less comfortable in discussion and debate with native speakers or with interlocutors who are perhaps more competitive than cooperative in their approach to the interaction.  More could be done in curriculum development related to strategies students can use to gain the floor and hold the floor in academic interactions (e.g., skills for politely interrupting, and skills to ensure the students can hold the floor when someone else has interrupted them).

  • The ELI may want to explore ways of working with students to find more opportunities for the students to engage in interaction with a variety of native speakers and other users of English on campus and in the community.

11) State how the program used the results or plans to use the results. Please be specific.

A)    Ethnographic study of two content courses commonly taken by ELI undergraduate students.  As a first step, the ELI Director and Associate Director will review the report with the student-researcher, to discuss in more detail about the recommendations section.  Ideas can then be taken to each curriculum area for discussion and action (e.g., curriculum development or materials development) in curriculum-area meetings.

B)    Follow-up assessment of SLO #3 for FW.  Once the assessment is completed, the ELI hopes to join the English Department in a presentation of data and discussion of results. Additionally, the ELI’s writing curriculum area will set aside time for further discussion of the implications of the results.

C)   Evaluation of the ELI listening/speaking curriculum via a survey of ELI “alumni”. The ELI Director and Associate Director met with the two student-researchers of this study to discuss implications of the results.  One major implication related to students feeling less comfortable in interactions with native speakers.  One question that arose, given the diversity of the student (and faculty) population at UHM, is whether it was really “native speakers” that posed difficulties or perhaps whether the interlocutors were taking a competitive rather than cooperative approach to discussion and other interactions.  To gain further data about this, one of the student-researchers is currently involved in a follow-up study (Fall 2011) that involves recording ELI students’ interactions with other students (native speakers or otherwise) in non-ELI courses; we hope to have results to report on for this study in next year’s assessment report.  Additionally, an graduate-assistant (GA) instructor in the ELI listening/speaking curriculum area was given a course release in Spring 2011 to develop new materials for raising students’ awareness about discussions and other academic interactions.  The ELI had hoped the GA would have time to also develop strategy-focused materials related to gaining and holding the floor (as mentioned in Item 10 above), but the GA did not have enough time to tackle this part of the project.  Should the opportunity to grant a course release come up again in the near future, we hope to assign that task to an accomplished GA.

D)   Discussion in Writing Curriculum Area of Previous Results.  Summaries of two studies from the 2010 ELI Assessment Report were presented by the ELI Director to the ELI’s writing curriculum area for discussion and potential action.  The two studies were a “Survey of ex-ELI 83 students” and “Assessment of SLO #3 for FW”.  Highlights of both studies showed that the ELI had been doing well in terms of meeting the SLOs in question, but also raised some questions for further discussion and potential action.  The ELI hopes to have follow up with the writing curriculum area to explore how teachers may have made adjustments to address some of the issues raised (particularly the need to help students understand the values underlying citation of sources, and more awareness building and practice with paraphrasing, summarizing, quoting and citing in students own writing).

12) Reflect on the assessment process. Is there anything related to assessment procedures your program would do differently next time? What went well?

As mentioned in previous reports, the ELI would like to learn more about “best practices” for using the results of assessments, such as effective ways of sharing the data with relevant constituents (teachers, students, administrative staff), discussing results and what can be learned, and applying those ideas. We would also like to learn more about ways of making assessment an important part of our standard practices, given the constraints we face (e.g., limited human resources, occasional availability of testing/assessment specialists among our graduate students).

13) Other important information

Unlike curricular programs (from which students graduate with a degree or certificate), the ELI is a co-curricular, non-degree bearing program. Therefore, there is not a fixed set of courses that must be successfully completed by all students. Students’ course requirements are determined by a battery of placement tests taken at the time they enter UH Manoa, and placements range from some students being completely exempt (no ELI courses required) to others needing six courses across all three curriculum areas (with the majority of students falling in between zero and six courses required). Thus, it is important to note that exempting out of a curriculum area based on the placement test does not guarantee that a student has mastered all the outcomes of that area.

The ELI is one of two programs that are housed within the Department of Second Language Studies.  The other is the Hawai`i English Language Program (HELP).  The ELI is for students who have matriculated at UH Manoa, whereas HELP is an intensive English program for students whose English is not yet high enough for admission to university.  Many students who complete HELP are admitted to UH Manoa and place into ELI courses.  Because of our efforts to coordinate the SLOs of our two programs, the transition from HELP to ELI is usually as seamless as possible, and assessment is an important part of that process.  Thus, our assessment efforts are designed not only to serve within the ELI, but across our two programs.

While we sometimes have GAs who specialize in assessment and evaluation, we also have “gap semesters” where no GA is available for these purposes.  Currently, program- and department-level resources are not adequate to engage in sustained comprehensive assessment, and no college-level resources have been allocated to this system-mandated initiative. Once additional support is consistently available, more will be done in a consistent and sustained way.  In the meantime, the ELI must rely primarily on assessment projects done as projects in graduate level SLS courses.  The ELI is particularly interested in learning more about “best practices” for how programs integrate assessment into their standard practices in a sustainable way, especially programs with very limited human and other resources.