Unit: Languages, Linguistics, & Literature, College of
Program: English Language Institute
Date: Wed Oct 31, 2012 - 4:00:21 pm

1) Below are your program's student outcomes (SOs). Please add or update as needed.

Student Learning Outcomes

ELI Listening & Speaking Curriculum Area

After successful completion of ELI Listening & Speaking courses, students will be able to:

  • demonstrate effective use of strategies for: comprehending advanced academic lectures in English; critically evaluating speakers’ perspectives, techniques, and arguments; and incorporating information from academic lectures into their overall studies
  • make academic presentations (individually or in group or panel contexts) with a high degree of formal accuracy and cultural and stylistic appropriacy
  • autonomously lead academic discussions using academic English, and demonstrate effective use of advanced strategies for participation in academic discussions with expert users of English
  • self-assess their strengths in terms of listening/speaking abilities, identify areas for continued development, and state a range of strategies to address those areas

Student Learning Outcomes

ELI Reading Curriculum Area

After successful completion of ELI Reading courses, students will be able to:

  • select reading and note-taking strategies appropriately for a range of different academic reading tasks, in accordance with courses they are enrolled in as well as their own purposes for reading advanced academic English texts
  • evaluate authors’ messages, perspectives, techniques, and arguments
  • evaluate print and web-based sources
  • self-assess their strengths in terms of reading abilities, identify areas for continued development, and state a range of strategies to address those areas

Student Learning Outcomes

ELI Writing Curriculum Area – Undergraduate Level

After successful completion of ELI Writing courses, undergraduate students will be able to:

  • fluently generate sufficient written text, at the brainstorming and drafting stages of the writing process, in response to a writing assignment
  • Compose college writing that achieves a specific purpose and responds adeptly to an identifiable audience.
  • Provide evidence of effective strategies for generating, revising, editing, and proofreading a text in order to produce finished prose.
  • Compose an argument that makes use of source material that is relevant and credible and that is integrated in accordance with an appropriate style guide.

Working definitions for SLO #2 (by committee representing ELI and the English Department – the two programs offering FW):

  • Revising: making global changes (e.g., adding, deleting, or moving content; rewriting for a different audience; rewriting in a different tone)

  • Editing: making changes at the sentence level, including changes in sentence style, syntax, phrasing

  • Proofreading: correcting grammar, punctuation and mechanics, spelling, formatting

Student Learning Outcomes

ELI Writing Curriculum Area – Graduate Level

After successful completion of ELI Writing courses, graduate students will be able to:

  • fluently generate sufficient written text, at the brainstorming and drafting stages of the writing process, in response to a writing assignment
  • analyze discipline and genre-specific academic English writing conventions and effectively apply that knowledge to graduate level writing tasks
  • self-assess their strengths as academic writers, as well as areas for continued development, incorporating personal reflection and feedback from others
  • appropriately incorporate a variety of reliable information sources that are relevant for doing graduate-level research in their academic writing

2) Your program's SOs are published as follows. Please update as needed.

Program's Website. URL: http://www.hawaii.edu/eli/students/SLO.html
Student Handbook. URL, if available online:
Information Sheet, Flyer, or Brochure. URL, if available online:
UHM Catalog. Page Number:
Other:
Other:

3) Provide the program's activity map or other graphic that illustrates how program activities/services align with program student outcomes. Please upload it as a PDF.

Activity Map File(s) from 2012:

4) Did your program engage in any program assessment activities between June 1, 2011 and September 30, 2012? (e.g., establishing/revising outcomes, aligning activities to outcomes, collecting evidence, interpreting evidence, using results, revising the assessment plan, creating surveys, etc.)

Yes
No (skip to question 14)

5) For the period June 1, 2011 to September 30, 2012: State the assessment question(s) and/or assessment goals. Include the student outcomes that were targeted, if applicable.

Three assessments were carried out during the period in question, labeled A, B, & C hereafter.

A)   Internal evaluation of the ELI’s “Discussion Circle” project in ELI 80.

  • Does the Discussion Circle project meet the ELI academic discussion SLOs? (“autonomously lead academic discussions using academic English, and demonstrate effective use of advanced strategies for participation in academic discussions with expert users of English”
  • What are students’ perceptions about their acquisition of discussion skills?
  • How is the Discussion Circle project implemented in online and face-to-face classes?
  • How do online and face-to-face students’ perceptions of the Discussion Circle project compare?
  • What aspects of the Discussion Circle project’s instructional design can be enhanced?

B)    Construct validation of the listening subtests in the ELI Placement Test (ELIPT).

  • How does the ELI define the construct of “academic listening comprehension”, and thus, the construct for validity of academic listening comprehension for its placement test?
  • How well do the two listening subtests in the ELIPT measure this construct?
  • How differently do test-takers perform on the two listening subtests according to groups based on similar native languages?

C)    Analysis of item types in the Academic Listening subtest in the ELIPT.

As a follow-up from Study B above (construct validation of the listening subtests of the ELIPT), the ELI sought a more detailed understanding of the reliability and validity of its multiple-choice listening subtest (the “Academic Listening Test”, hereafter “ALT”).  In addition, the ELI hoped to gain insight about how the test might be revised (if deemed necessary) to optimize reliability and validity. Specific questions to be addressed were:

  • How well is the ALT currently working in terms of matching the population of test-takers, item reliability, item-to-model fit, and item discrimination?
  • What similarities and differences can be found with previous analyses of the ALT, and what the implications of these comparisons?
  • What effect do different item types (based on levels of detail and attention) have on item difficulty, reliability, interaction, and item discrimination?

6) State the type(s) of evidence gathered to answer the assessment question and/or meet the assessment goals that were given in Question #5.

A)   Internal evaluation of the ELI’s “Discussion Circle” project in ELI 80.

  • Document analysis and interviews of ELI administrative and instructional staff to inform the development of questionnaires.
  • Questionnaire re: needs analysis
  • Questionnaire re: effectiveness of the Discussion Circle project.
  • Interviews of ELI 80 instructors re: the effectiveness of the Discussion Circle project
  • Observation of one face-to-face class session involving Discussion Circles and review of the transcript of the online class’ Discussion Circle.

B)    Construct validation of the listening subtests in the ELI Placement Test (ELIPT).

  • Test score analysis.
  • Test item analysis.
  • A survey of students about their impression of test difficulty and their test results, given immediately after the test was taken (before actual results were available).
  • Document analysis related to the listening/speaking curriculum area.
  • Interviews with ELI administrators and instructors of listening/speaking courses.

C)    Analysis of item types in the Academic Listening subtest in the ELIPT.

  • Rasch analysis
  • G-theory analysis

7) State how many persons submitted evidence that was evaluated. If applicable, please include the sampling technique used.

A)   Internal evaluation of the ELI’s “Discussion Circle” project in ELI 80.

  • Interviews to inform the development of questionnaires:  2 ELI administrators, 2 lead teachers (online and listening/speaking curriculum area lead teachers), and five teachers of ELI 80 were interviewed.
  • Questionnaire re: needs analysis:  37 students from ELI 80, from both face-to-face and online course sections (out of 59 solicited; response rate of 63%)
  • Questionnaire re: effectiveness of the Discussion Circle project:  34 students from ELI 80, from both face-to-face and online course sections (out of 59 solicited; response rate of 58%)
  • Interviews of ELI 80 instructors re: the effectiveness of the Discussion Circle project:  5 instructors were interviewed.
  • Observation of one face-to-face class session involving Discussion Circles and review of the transcript of the online class’ Discussion Circle:  1 face-to-face class session was observed, and transcripts from one online class discussion circle was reviewed.

B)    Construct validation of the listening subtests in the ELI Placement Test (ELIPT).

  • Document analysis:  Provided by the ELI administrators and the listening/speaking lead teacher.
  • Test score analysis:  304 test takers
  • Test item analysis:  304 test takers
  • Survey of students:  304 test takers
  • Interviews:  two administrators in the ELI

C)    Analysis of item types in the Academic Listening subtest in the ELIPT.

  • ALT results from 502 test-takers

8) Who interpreted or analyzed the evidence that was collected? Check all that apply.

Program faculty/staff member(s)
Faculty/staff committee
Ad hoc faculty/staff group
Director or department chairperson
Persons or organization outside the university
Students (graduate or undergraduate)
Dean or Associate Dean
Advisory Board
Other: (note that this is the same for all three studies)

9) How did he/she/they evaluate, analyze, or interpret the evidence? Check all that apply.

Compiled survey results
Used quantitative methods on student data (e.g., grades, participation rates) or other numeric data
Used qualitative methods on interview, focus group, or other open-ended response data
Scored exams/tests/quizzes
Used a rubric or scoring guide
Used professional judgment (no rubric or scoring guide used)
External organization/person analyzed data (e.g., Social Science Research Institute)
Other: (Note that there are differences in methods of evaluating used in the 3 different studies)

10) For the assessment questions/goals stated in Question #5, summarize the actual results.

A)   Internal evaluation of the ELI’s “Discussion Circle” project in ELI 80.

  • Although both students and teachers perceive learning of strategies for effective leadership and participation in discussions, it was noted that it is very difficult to clearly assess whether the SLO has been met. The researchers recommended that, in the future, a standardized assessment procedure be created, involving rubrics and clear performance indicators.
  • The Discussion Circle project is implemented similarly, although not exactly in the same way, in both online and face-to-face sections of ELI 80. The researchers noted that there is a need for flexibility for variability in teaching styles and format (even across face-to-face sections there is a great deal of variability in how a teacher implements the project).
  • In open-ended questions, some students stated improvement in terms of awareness of academic discussion structures, the need for skills and strategies for effectively leading discussions, and a variety of ways of participating effectively.  However, some highlighted that, although they recognized the value of these strategies and had improved their abilities in using them, they did not yet feel comfortable in using them effectively in non-ELI class settings.

B)    Construct validation of the listening subtests in the ELI Placement Test (ELIPT).

  • Based on curriculum document review, the researcher found that the construct of “academic listening comprehension” comprised three main listening skills: listening comprehension, critical listening, and interactive listening (she noted that “interactive listening” was the only skill that involved two-way communication between a speaker and listener; listening comprehension and critical listening involved a one-directional transaction of information from speaker to listener; in addition, critical listening involved application, analysis, synthesis, and evaluation of the content by the listener).
  • There are two listening subtests in the ELIPT: a dictation and a multiple-choice test. The researcher performed item analysis of the multiple-choice test and found that items could be classified as having a focus on either global or local listening comprehension. Out of 35 test items, 7 were global and 28 were local. Missing were items that had a focus on critical listening, and it appeared that there was an imbalance, with far more local than global items.
  • Review of teacher confidence about the placements being correct (in one semester done by the ELI institution) suggests that very few, if any, of the students were wrongly placed. In general, teachers felt that students needed the assistance that was provided by the course where they were placed. Further evidence of the validity of the test for accurate placement was found via teacher and administrator interviews.
  • Both subtests had respectable reliability, with the dictation having a much higher reliability coefficient than the multiple-choice test.
  • Item discrimination for the multiple-choice test showed that five items did not discriminate well between those test-takers who scored high and those who scored low.
  • There was a greater correlation between general language proficiency and ELIPT scores than there was based on language group.

C)    Analysis of item types in the Academic Listening subtest in the ELIPT.

  • Results from Rasch analysis indicate the most items fit the model and item difficulty relatively closely with the range of examinee ability.  However, two of the 35 test items were found to be misfitting the model and may need to be either revised or replaced.
  • Additionally, the Rasch analysis and G-theory analysis suggests that there may be an abundance of easier items, and suggests the potential need for replacing some of these easier items, particiularly with items that involve more complex listening or critical listening skills.
  • As with a previous study (Study B above), this study found that there were far more items focusing on local comprehension, and fewer focusing on global comprehension or critical listening.  Of the 7 items focusing on global comprehension, 4 items have low item-discrimination values.  The researcher pointed out, however, that “global comprehension” may involve require less attention in general and less attention to detail than either local comprehension or critical listening items.  The researcher analyzed one item that appeared to have a very high item-discrimination value, and found that item to involve deeper levels of comprehension.

11) What was learned from the results?

A)   Internal evaluation of the ELI’s “Discussion Circle” project in ELI 80.

The Discussion Circle project is well organized, and focuses on helping students develop strategies to be more successful and confident in participating in or leading academic discussions (that is, not just on discussion for its own sake). This class project tends to be highly successful in terms of raising students’ awareness about the genre of academic discussions, and about a variety of strategies for leading or participating effectively in discussions.  However, further work needs to be done to help students more fully develop the ability to use the strategies confidently.

B)    Construct validation of the listening subtests in the ELI Placement Test (ELIPT).

Although the ELI’s listening/speaking curriculum area clearly focuses on three main areas of developing students’ academic listening skills and strategies, it was very helpful to have definitions tied to the construct.

Further, it was helpful to learn that the subtests of the ELIPT do not have items focusing on critical listening, and have an imbalance between local and global comprehension test items.

C)    Analysis of item types in the Academic Listening subtest in the ELIPT.

More research should be done into what underlies the more complex types of listening common to performance in academic settings, and how these levels of complexity can be incorporated into question types for the ALT.  New test items, covering a range of item types, should be developed and piloted to see if reliability can be optimized.

12) State how the program used the results or plans to use the results. Please be specific.

A)   Internal evaluation of the ELI’s “Discussion Circle” project in ELI 80.

  • The researchers shared their results with program administrators and discussed possible future directions.  The administrators asked that the researchers also share their results and have discussion with teachers and the lead teacher in the listening/speaking curriculum area, which they did.  The results and suggestions were well received by teachers.
  • The ELI has added two related items to its research agenda related to the results of this study.  One is for research of ELI students’ participation in discussion in classes outside of the ELI.  The other is for development of additional lessons and activities that help students gain more practice with and confidence in the use of discussion strategies.
  • Further, the ELI will encourage future assessments of the same SLO, with the development of a rubric as recommended by the researchers.

B)    Construct validation of the listening subtests in the ELI Placement Test (ELIPT).

  • The ELI added to its research agenda a project for test-item development and piloting for the multiple-choice test, which would include adding more global test items and developing critical listening test items.  This agenda item was immediately taken up by a graduate student who is one of the ELI’s listening/speaking instructors.  The same graduate student also conducted a further study of the multiple-choice listening test (Item C in this report).

C)    Analysis of item types in the Academic Listening subtest in the ELIPT.

  • The graduate student who conducted this research project went on to work on item development for the ALT.  In Summer 2012,  he developed 10 new items, which he expects (based on this study) to have higher item difficulty (and thus, discriminate better than some of the low-discrimination-value items), and in the Fall 2012 ELIPT, these 10 items were piloted.  Results are pending.

13) Reflect on the assessment process. Is there anything related to assessment procedures your program would do differently next time? What went well?

As mentioned in previous reports, the ELI would like to learn more about “best practices” for using the results of assessments, such as effective ways of sharing the data with relevant constituents (teachers, students, administrative staff), discussing results and what can be learned, and applying those ideas. We would also like to learn more about ways of making assessment an important part of our standard practices, given the constraints we face (e.g., limited human resources, occasional availability of testing/assessment specialists among our graduate students).

14) If the program did not engage in assessment activities, please explain.
Or, if the program did engage in assessment activities, please add any other important information here.

Unlike curricular programs (from which students graduate with a degree or certificate), the ELI is a co-curricular, non-degree bearing program. Therefore, there is not a fixed set of courses that must be successfully completed by all students. Students’ course requirements are determined by a battery of placement tests taken at the time they enter UH Manoa, and placements range from some students being completely exempt (no ELI courses required) to others needing six courses across all three curriculum areas (with the majority of students falling in between zero and six courses required). Thus, it is important to note that exempting out of a curriculum area based on the placement test does not guarantee that a student has mastered all the outcomes of that area.

The ELI is one of two ESL programs that are housed within the Department of Second Language Studies.  The other is the Hawai`i English Language Program (HELP).  The ELI is for students who have matriculated at UH Manoa, whereas HELP is an intensive English program for students whose English is not yet high enough for admission to university.  Many students who complete HELP are admitted to UH Manoa and place into ELI courses.  Because of our efforts to coordinate the SLOs of our two programs, the transition from HELP to ELI is usually as seamless as possible, and assessment is an important part of that process.  Thus, our assessment efforts are designed not only to serve within the ELI, but across our two programs.

While we sometimes have GAs who specialize in assessment and evaluation, we also have “gap semesters” where no GA is available for these purposes.  Currently, program- and department-level resources are not adequate to engage in sustained comprehensive assessment, and no college-level resources have been allocated to this system-mandated initiative. Once additional support is consistently available, more will be done in a consistent and sustained way.  In the meantime, the ELI must rely primarily on assessment projects done as projects in graduate level SLS courses.  The ELI is particularly interested in learning more about “best practices” for how programs integrate assessment into their standard practices in a sustainable way, especially programs with very limited human and other resources.