Unit: Languages, Linguistics, & Literature, College of
Program: English Language Institute
Date: Thu Oct 10, 2013 - 10:01:54 am

1) Below are your program's student outcomes (SOs). Please add or update as needed.

Student Learning Outcomes

ELI Listening & Speaking Curriculum Area

After successful completion of ELI Listening & Speaking courses, students will be able to:

  • demonstrate effective use of strategies for: comprehending advanced academic lectures in English; critically evaluating speakers’ perspectives, techniques, and arguments; and incorporating information from academic lectures into their overall studies
  • make academic presentations (individually or in group or panel contexts) with a high degree of formal accuracy and cultural and stylistic appropriacy
  • autonomously lead academic discussions using academic English, and demonstrate effective use of advanced strategies for participation in academic discussions with expert users of English
  • self-assess their strengths in terms of listening/speaking abilities, identify areas for continued development, and state a range of strategies to address those areas

Student Learning Outcomes

ELI Reading Curriculum Area

After successful completion of ELI Reading courses, students will be able to:

  • select reading and note-taking strategies appropriately for a range of different academic reading tasks, in accordance with courses they are enrolled in as well as their own purposes for reading advanced academic English texts
  • evaluate authors’ messages, perspectives, techniques, and arguments
  • evaluate print and web-based sources
  • self-assess their strengths in terms of reading abilities, identify areas for continued development, and state a range of strategies to address those areas

Student Learning Outcomes

ELI Writing Curriculum Area – Undergraduate Level

After successful completion of ELI Writing courses, undergraduate students will be able to:

  • fluently generate sufficient written text, at the brainstorming and drafting stages of the writing process, in response to a writing assignment
  • Compose college writing that achieves a specific purpose and responds adeptly to an identifiable audience.
  • Provide evidence of effective strategies for generating, revising, editing, and proofreading a text in order to produce finished prose.
  • Compose an argument that makes use of source material that is relevant and credible and that is integrated in accordance with an appropriate style guide.

Working definitions for SLO #2 (by committee representing ELI and the English Department – the two programs offering FW):

  • Revising: making global changes (e.g., adding, deleting, or moving content; rewriting for a different audience; rewriting in a different tone)

  • Editing: making changes at the sentence level, including changes in sentence style, syntax, phrasing

  • Proofreading: correcting grammar, punctuation and mechanics, spelling, formatting

Student Learning Outcomes

ELI Writing Curriculum Area – Graduate Level

After successful completion of ELI Writing courses, graduate students will be able to:

  • fluently generate sufficient written text, at the brainstorming and drafting stages of the writing process, in response to a writing assignment
  • analyze discipline and genre-specific academic English writing conventions and effectively apply that knowledge to graduate level writing tasks
  • self-assess their strengths as academic writers, as well as areas for continued development, incorporating personal reflection and feedback from others
  • appropriately incorporate a variety of reliable information sources that are relevant for doing graduate-level research in their academic writing

2) Your program's SOs are published as follows. Please update as needed.

Program's Website. URL: http://www.hawaii.edu/eli/students/SLO.html
Student Handbook. URL, if available online:
Information Sheet, Flyer, or Brochure. URL, if available online:
UHM Catalog. Page Number:
Other:
Other:

3) Provide the program's activity map or other graphic that illustrates how program activities/services align with program student outcomes. Please upload it as a PDF.

Activity Map File(s) from 2013:

4) Did your program engage in any program assessment activities between June 1, 2012 and September 30, 2013? (e.g., establishing/revising outcomes, aligning activities to outcomes, collecting evidence, interpreting evidence, using results, revising the assessment plan, creating surveys, etc.)

Yes
No (skip to question 14)

5) For the period June 1, 2012 to September 30, 2013: State the assessment question(s) and/or assessment goals. Include the student outcomes that were targeted, if applicable.

Three assessments were carried out during the period in question, labeled A, B, & C hereafter.

  1. Analysis of a Task-Based “Critical Listening” Construct in the ELI Placement Test

Based on two studies from the previous academic year (Items B and C from the ELI’s 2012 assessment report), it was found that the ELI Placement Test (ELIPT) did not include any items that focused on the ability to listen critically in an academic setting, even though SLOs for the listening/speaking curriculum area include “critically evaluating speakers’ perspectives, techniques, and arguments”).  The subtest within the ELIPT where changes seemed most possible was the “Academic Listening Test” (ALT).  This study looked at the following questions:

  1. How does the current form of the ALT perform in terms of matching the ability and difficulty of the population of examinees, reliability of the scores on the test, and item and person fit?
  2. What constructs are represented in the current form of the ALT?
  3. How do items based on the skills of organization, connecting ideas, and determining the importance of ideas perform on the ALT in terms of matching the population of examinees, reliability of scores on the test, and item and person fit?
  4. How do the piloted items compare to the items they replaced?
  1. Curriculum Development for ELI 72 involving critical pedagogy and learner autonomy

One of the ELI’s primary purposes is to help students who have English as a second language develop skills and strategies in using English for academic purposes (EAP).  One very important part of this is helping students learn to effectively read academic texts.  This project focused on a curriculum and materials development for ELI 72 (the intermediate level course in academic reading), incorporating critical pedagogy, critical literacy, and extensive reading, with aims of helping students develop more academic autonomy.  Reading-Area SLOs that are related to learner autonomy include students’ ability to:

  • select reading and note-taking strategies appropriately for a range of different academic reading tasks, in accordance with courses they are enrolled in as well as their own purposes for reading advanced academic English texts
  • self-assess their strengths in terms of reading abilities, identify areas for continued development, and state a range of strategies to address those areas
  1. Materials Development for Improving Students’ Self-Editing Skills in ELI 100

ELI writing courses emphasize a process approach to writing (combined with awareness about the importance of the final product, as well).  One important part of the process is editing (which can include both self-editing and editing with feedback provided by others).  While the editing part of the process is included in every ELI writing course, it has not been studied, in and of itself, within the ELI’s writing program.  Thus, one item in the ELI’s research agenda is a review of how ELI courses teach and promote self-editing, how this meshes with “best practices” and current research on the topic, plus development of materials and resources that ELI teachers can draw upon for to teach about editing and also promote greater learner autonomy in self-editing.  The related SLO from ELI 100 (which is also an SLO agreed upon for all courses meeting the “Foundations in Written Communication (FW) requirement) is that students will be able to:

  • provide evidence of effective strategies for generating, revising, editing, and proofreading a text in order to produce finished prose.

6) State the type(s) of evidence gathered to answer the assessment question and/or meet the assessment goals that were given in Question #5.

  1. Analysis of a Task-Based “Critical Listening” Construct in the ELI Placement Test

Data came from ALT results collected in two stages, once for the existing form of the ALT and once for revised versions of the ALT.  For the existing form of the ALT, results were analyzed from four semesters of administrations of the ELIPT, covering a period from Fall 2010 through Spring 2012 semesters. The revised versions of the ALT involved two different forms of the ALT (Form A and Form B), which were administered as part of the ELIPT in Fall 2012.  Each revised form included 5 pilot items, which were designed as potential replacements for “low performing” items in the existing version of the ALT.

  1. Curriculum Development for ELI 72 involving critical pedagogy and learner autonomy
  • Review of literature on critical reading in the ELI and other programs; previous research done on ELI 72, ELI 82, and the ELI reading curriculum area in general; and research related to materials development, textbook selection, and sourcebook development for EAP reading.
  • Evaluation of reading textbooks and other materials.
  • Review of goals and objectives (and later SLOs) from ELI 72 syllabi from 1983 through the present.
  • Interviews with ELI administrators, reading lead teacher, and current and former teachers of ELI 72.
  • Follow-up interviews with ELI administrators, reading lead teacher, and other reading teachers for feedback and input about materials developed.
  • Feedback from students on piloted materials.
  • Informal discussions with reading experts, at conferences and via email exchanges.
  1. Materials Development for Improving Students’ Self-Editing Skills in ELI 100
  • Review of literature on editing and self-editing of second-language writers of English, including review of recent research done in the ELI writing curriculum area.
  • Interviews with ELI administrators, the writing lead teacher, and current teachers of ELI writing courses.
  • Observations of ELI 73 and ELI 100 classes.
  • Survey of ELI teachers to find out usefulness of proposed ideas for an ELI writing teachers’ manual for teaching self-editing.
  • Survey of students in ELI 100 related to their attitudes toward self-editing.
  • Presentation of draft materials in an ELI writing curriculum-area meeting, to gain further feedback for final revisions.

7) State how many persons submitted evidence that was evaluated. If applicable, please include the sampling technique used.

  1. Analysis of a Task-Based “Critical Listening” Construct in the ELI Placement Test
  • Existing ALT (Fall 2010-Spring 2012 ELIPT).  501 test takers.
  • Revised ALT (Fall 2012).  141 test takers (78 for Form A, and 63 for Form B)
  1. Curriculum Development for ELI 72 involving critical pedagogy and learner autonomy

N/A

  1. Materials Development for Improving Students’ Self-Editing Skills in ELI 100
  • Student survey.  12 students from ELI 100.
  • Teacher survey.  6 writing teachers.

8) Who interpreted or analyzed the evidence that was collected? Check all that apply.

Program faculty/staff member(s)
Faculty/staff committee
Ad hoc faculty/staff group
Director or department chairperson
Persons or organization outside the university
Students (graduate or undergraduate)
Dean or Associate Dean
Advisory Board
Other: For all three assessments, while it was graduate students who analyzed data and interpreted results, it was always under the supervision of, and with support from, a graduate-faculty with expertise in the area of study.

9) How did he/she/they evaluate, analyze, or interpret the evidence? Check all that apply.

Compiled survey results
Used quantitative methods on student data (e.g., grades, participation rates) or other numeric data
Used qualitative methods on interview, focus group, or other open-ended response data
Scored exams/tests/quizzes
Used a rubric or scoring guide
Used professional judgment (no rubric or scoring guide used)
External organization/person analyzed data (e.g., Social Science Research Institute)
Other: Review and evaluation of a variety of existing reading materials. [Note that there were three assessment studies, and each involved different combinations of the above.]

10) For the assessment questions/goals stated in Question #5, summarize the actual results.

  1. Analysis of a Task-Based “Critical Listening” Construct in the ELI Placement Test

First, Rasch analysis of the current ALT was run with data from the existing ALT (Fall 2010 through Spring 2012 administrations). Underperforming and misfitting items were identified and five items were selected for removal. 10 pilot items were developed using a theorized task-based construct of critical listening based on the current listening passages for the ALT (5 pilot items for Form A, and 5 for Form B). The two revised versions of the ALT were administered during the Fall 2012 ELIPT. Lastly, the pilot tests and items were analyzed again using Rasch. Pilot item performance was compared against the population, other items on the test, and the five removed items.  The results found that:

  • Revised forms (with pilot items) had higher reliability (.79 for Form A and .76 for Form B, compared to .73 for the existing ALT).
  • 9 out of 10 pilot items fell within the current range of difficulties on the ALT. Further, the pilot items difficulty levels were in a range that was more likely to have an effect on placement decisions.
  • Item fit was high (with only one “misfitting” item among the pilot items).

Second, factor analysis was done on both the existing ALT and the revised forms of the ALT.  The results found that:

  • Revised forms (with pilot items) mirrored the existing ALT in terms of a clearly identified “listening” construct.
  • There were no indications of multiple constructs outside a single factor of “listening comprehension”, which means that no construct for “critical listening” could be validated.
  1. Curriculum Development for ELI 72 involving critical pedagogy and learner autonomy
  • At the ELI 72 level, one important need of students was development of reading fluency and confidence.  Inclusion of extensive reading, involving graded readers, was seen by the researcher as a vital part of the ELI 72 curriculum.
  • A “sourcebook” (incorporating texts from a variety of sources) would be far more effective than a single textbook that focused on developing reading skills and strategies.  The researcher argued that an essential component to a sourcebook would be a variety of readings that deal critically with issues that were directly relevant to the lives of ELI students (including texts from a book published by the University of Michigan Press in which second-language students are the main characters, in the settings that involve universities or university classes) (a further recommendation was made to encourage ELI writing teachers to have students write their own stories, that could later be used as texts for ELI reading courses).
  • A sourcebook was created, along with a semester-long curriculum that sought to achieve the goals of this researcher’s study, along with addressing other SLOs and identified needs of students at the ELI 72 level.
  1. Materials Development for Improving Students’ Self-Editing Skills in ELI 100
  • Interviews with the ELI’s administrators, writing lead teacher, and writing teachers all indicated strong recognition of the importance of editing in the writing process, and the importance of helping students develop greater autonomy as self-editors.  While all ELI writing courses include awareness of, and practice with, editing, the ELI’s materials bank did not include an extensive set of easily adaptable, ready-to-go materials on this topic.
  • Students surveyed clearly indicated familiarity with editing and self-editing, and recognition of the value and importance of self-editing.  Eight students believed themselves to be capable of self-editing, while four felt that they could not effectively self-edit.  Students also confirmed the value of feedback from others in improving their writing. (Note that this was somewhat contradictory of teacher’s views of students’ ability to self-edit or to effectively incorporate feedback into subsequent drafts.)
  • The researchers developed an ELI writing teachers’ manual for teaching self-editing, with feedback on drafts from the ELI’s writing teachers and lead teacher.  Components of their manual included:
    • An overview of editing and self-editing, including reference to relevant literature.
    • A discussion of written corrective feedback (hereafter WCF), including choices between selective or comprehensive feedback, direct or indirect feedback, the value of a “key” for WCF, awareness of individual differences in terms of learning styles of students, and ideas for encouraging students to take a more autonomous role in self-editing.
    • Information about sources of positive input (including corpora, such as the Michigan Corpus of Upper-Level Student Papers) and how to use them in classes or help students to use them independently.
    • A commonly used, somewhat comprehensive key for WCF (including abbreviations).
    • A template “Error Log” for students to track their own errors.
    • Sample lessons and lesson plan template for mini lessons on grammar and mechanics.
    • A sample “Self-Editing Checklist”, which could easily be adapted for common needs of a class, or by individual students based on their own common errors.
    • A sample survey of student attitudes toward editing and self-editing.

11) What was learned from the results?

  1. Analysis of a Task-Based “Critical Listening” Construct in the ELI Placement Test
  • Even though the pilot items did not result in a clearly identified construct for “critical listening”, they were clearly found to be stronger items than the “misfitting” items that they replaced (as indicated by both the higher reliability of the revised forms, and the strong item fit).  This suggests that there is value in designing items like the pilot items (that operationalized skills of listening for organization of ideas, connections between ideas, and importance of ideas).
  • Further research is still needed to determine what skills comprise a construct of “critical listening” and how those skills can be operationalized in a listening placement test.
  1. Curriculum Development for ELI 72 involving critical pedagogy and learner autonomy
  • The teacher’s role in setting up an extensive reading component to the course, and ensuring the relevance of discussion of what was read was essential to sustaining student motivation for extensive reading.
  • Ongoing development of an evolving “sourcebook” could be a very effective way to deal with student engagement and motivation in an EAP reading course.
  1. Materials Development for Improving Students’ Self-Editing Skills in ELI 100
  • While it is clear that all ELI teachers and program administrators recognized the importance of editing and self-editing as part of the writing process, and as something that is included in the process of writing papers within ELI writing courses, the ELI did not have materials and clearly defined lessons focusing on how students can develop self-editing skills (individual teachers may have used a variety of materials and lessons, but the ELI did not have a set of materials that addressed the topic systematically and thoroughly).
  • Even though the n-size for the student survey was too small to safely generalize about, it was very encouraging to note that, by the end of ELI 100, students recognized the value of editing, self-editing, and feedback as part of the writing process.

12) State how the program used the results or plans to use the results. Please be specific.

  1. Analysis of a Task-Based “Critical Listening” Construct in the ELI Placement Test

A new “existing version” of the ALT was created from incorporating the best-performing pilot items from the two revised versions.

  1. Curriculum Development for ELI 72 involving critical pedagogy and learner autonomy

The researcher, who is a graduate-assistant instructor in the ELI, is piloting the materials in Fall 2013, and will do a follow-up research project, as well as further revisions to the materials.  The “sourcebook” idea is being pilot-tested in AY 2013-14, as well.

  1. Materials Development for Improving Students’ Self-Editing Skills in ELI 100

As recommended by the researchers, the ELI will include the manual for teaching self-editing in its teacher-resources website.  Having the manual (including the WCF key, sample “Error Log”, etc.) in MS-Word format allows individual teachers to download and tweak the materials in ways that fit their classes or individual students.

13) Reflect on the assessment process. Is there anything related to assessment procedures your program would do differently next time? What went well?

As mentioned in previous reports, the ELI would like to learn more about “best practices” for using the results of assessments, such as effective ways of sharing the data with relevant constituents (teachers, students, administrative staff), discussing results and what can be learned, and applying those ideas.

Perhaps even more important, the ELI would also like to learn more about realistic ways to integrate SLO assessment and other assessment practices into the standard practices of our academic year.  However, given current constraints (e.g., very limited human resources, compounded by only occasional availability of testing/assessment specialists among our graduate students), we are, regrettably, currently unable to accomplish this.  ELI administrators have been talking with people in other programs to get ideas of how we might make positive steps toward systematizing SLO assessment.

14) If the program did not engage in assessment activities, please explain.
Or, if the program did engage in assessment activities, please add any other important information here.

  • Unlike curricular programs (from which students graduate with a degree or certificate), the ELI is a co-curricular, non-degree bearing program. Therefore, there is not a fixed set of courses that must be successfully completed by all students. Students’ course requirements are determined by a battery of placement tests taken at the time they enter UH Manoa, and placements range from some students being completely exempt (no ELI courses required) to others needing six courses across all three curriculum areas (with the majority of students falling in between zero and six courses required). Thus, it is important to note that exempting out of a curriculum area based on the placement test does not guarantee that a student has mastered all the outcomes of that area.
  • The ELI is one of two ESL programs that are housed within the Department of Second Language Studies.  The other is the Hawai`i English Language Program (HELP).  The ELI is for students who have matriculated at UH Manoa, whereas HELP is an intensive English program for students whose English is not yet high enough for admission to university.  Many students who complete HELP are admitted to UH Manoa and place into ELI courses.  Because of our efforts to coordinate the SLOs of our two programs, the transition from HELP to ELI is usually as seamless as possible, and we aim, when resources allow, to make assessment an important part of that process.  Thus, our assessment efforts are designed not only to serve within the ELI, but across our two programs.
  • While we sometimes have GAs who specialize in assessment and evaluation, we also have “gap semesters” where no GA is available for these purposes.  Currently, program- and department-level resources are not adequate to engage in sustained comprehensive assessment, and no college-level resources have been allocated to this system-mandated initiative. Once additional support is consistently available, more will be done in a consistent and sustained way.  In the meantime, the ELI must rely primarily on assessment projects done as projects in graduate-level SLS courses.  The ELI is particularly interested in learning more about “best practices” for how programs integrate assessment into their standard practices in a sustainable way, especially programs with very limited human and other resources.