Degree Program Assessment Reports

The medical school uses a number of processes to interpret, evaluate and analyze assessment data.

Results of USMLE and other national exams

USMLE. The USMLE Step 1 and Step 2 exam results are sent to the Office of Student Affairs (OSA) by the NBME. The results are reviewed by OSA, the curriculum committee, subcommittees, department chairs, the medical school Executive Committee, and the faculty senate. Unsatisfactory performance is reviewed by the Evaluation Review and Remediation Committee (ERRC) and by the Student Standing and Promotion Committee as needed. 

Comprehensive Basic Science Exam. To help students gauge their progress in learning the basic sciences and becoming familiar with the depth and breadth of questions on a national standardized exam, the NBME Comprehensive Basic Science Examination is administered to all students at the beginning and end of their second year of medical school. The exam, first given in the 2003 – 2004 academic year, is optional at the beginning of the year and required at the end of the year. Results are reviewed by OSA and the curriculum committee. Unsatisfactory performance on the required end-of-year exam also is reviewed by the ERRC. Students whose performance indicates a need for improvement are counseled by an OSA learning specialist.

NBME Clinical Discipline Shelf Exams. The relevant NBME Clinical Discipline Shelf Exams are administered to students at the end of the internal medicine, ob/gyn, psychiatry, and surgery clerkships. Exam results are reviewed by the respective clerkship directors and faculty as well as by OSA. The ERRC also reviews unsatisfactory performances and recommends plans for remediation.

Results of internally-developed examinations

Student scores on internally developed examinations. Written knowledge exams, consisting of essay and/or multiple choice questions, are prepared, administered, and graded by course and clerkship directors and their faculties. Similarly, preclinical laboratory exams are prepared, administered, collected, and graded by basic science instructors. Relevant course directors, faculty, and the Office of Student Affairs review all results. Student achievement of course objectives is formally assessed in the Curriculum Committee’s Curriculum Review Process.  The ERRC reviews the exams of students with unsatisfactory performance and recommends plans for remediation.

Performance-based assessment of clinical skills (e.g., OSCEs).  The OSCE. Students are required to pass a six-station multidisciplinary OSCE before starting the third-year clerkships and a ten-station multidisciplinary OSCE (BIOM 531: Comprehensive OSCE) at the end of the clerkship year. OSCEs are administered by the Center for Clinical Skills (CCS), a part of the JABSOM Office of Medical Education. CCS collects and grades the exams. The results are reviewed by the clinical skills committee, OSA, curriculum committee, and the clerkship directors committee.

Clinical Skills Exams. In addition, standardized patient clinical skills exams are included in the evaluation of student performance in the first- and second-year clinical skills course, the third-year family medicine, internal medicine, and pediatric clerkships, and the fourth-year geriatric course. CCS also administers and grades these exams, which then are reviewed at the relevant course or departmental level.  

Oral Exams. The pediatric and psychiatry clerkships design and administer oral exams to students as part of their evaluations. These exams assess clinical knowledge, clinical reasoning, and problem-solving. The clerkship directors and faculty collect and review the results, which inform the clerkships about each student’s ability to integrate knowledge and skills. 

Triple Jump. The Triple Jump, an oral and written assessment of students’ knowledge, self-directed learning, critical appraisal, and problem-solving, is administered individually to all first-year students. Students are rated on both their knowledge and skills. Ratings then are collected and reviewed by the course director. 

Other measures of curriculum effectiveness

Student responses on the AAMC Medical School Graduation Questionnaire.  JABSOM students complete the AAMC Medical School Graduation Questionnaire online. A paper copy of the comparison between JABSOM student and national responses is sent to OSA. The results are reviewed annually by OSA and the curriculum committee. 

Student evaluation of courses and clerkships.  As part if JABSOM’s larger program evaluation process, the Office of Medical Education (OME) designs a set of standardized surveys with core items based on the school’s graduation objectives. Within this set are surveys that are administered to students at the end of every course and clerkship. OME collects the surveys, analyzes the data, and forwards written reports of the findings to respective course and clerkship directors, the chair of the curriculum committee, and the director of the Office of Medical Education. A summary of the findings is reported to the curriculum committee. Relevant course and clerkship directors are invited to attend the curriculum committee meetings at which their survey findings are presented and discussed. The curriculum committee makes recommendations to the course and clerkship directors as needed. The directors, in turn, are responsible for addressing the recommendations and reporting the results of actions taken to the curriculum committee. 

Student advancement and graduation rates.  OSA monitors students’ academic progress and program completion rates. This information is reported to the curriculum committee.

NRMP results.  NRMP results are sent to OSA. Both OSA and the curriculum committee review these results. Information about the students’ ranking of the residency program with which they matched was made available through 2001. In 2001, more than half of JABSOM students matched with their first choice and 92 percent matched with one of their first three choices.

Specialty choice of graduates.  As part of JABSOM’s curriculum evaluation process, OME sends surveys to alumni six years after graduation. Once every four years, the Alumni Survey includes demographic items such as licensure, specialty choice, specialty certification, practice location and practice type. The findings, which are included in the written report of the Alumni Survey, inform the chair of the curriculum committee, the director of OME, and the chairs of the preclinical education and clerkship directors committees about JABSOM graduates’ practice selections. 

Assessment of residency performance of graduates.  JABSOM’s curriculum evaluation process also includes a residency program director survey, sent at the end of graduates’ first year of residency to their program directors.  This survey asks program directors to rate the performance of JABSOM graduates relative to their peers.  A written report of the survey findings is given to the chair of the curriculum committee, the director of OME, and the chairs of the preclinical education and clerkship directors committees. A written and oral summary of the report is given to the curriculum committee. 

Licensure and specialty certification rates of graduates.  The Alumni Survey, sent by OME to JABSOM alumni six years after graduation, includes an item about licensure and an item about specialty certification once every four years. The licensure and specialty certification rates are included in the written report given to the chair of the curriculum committee, the director of OME, and the chairs of the preclinical education and clerkship directors committees. They also are included in the summary report given to the curriculum committee, where they are reviewed and discussed.

Practice location and practice type of graduates.  Every fourth year, the Alumni Survey also includes items about practice location and practice type. Graduates’ responses are included in the written report given to the chair of the curriculum committee, the director of OME, and the chairs of the preclinical education and clerkship directors committees. They also are included in the summary report given to the curriculum committee, where they are reviewed and discussed.

Graduate evaluation of preparation for internship.  JABSOM sends an Intern Survey to graduates near the end of their first year of residency. The survey focuses on graduates’ perceptions of how well JABSOM prepared them for residency. Together, the findings of this survey and the Residency Program Director Survey provide information about the effectiveness of JABSOM’s curriculum from two perspectives, that of the postgraduate learner and that of the postgraduate teacher. A written report of the findings is sent to the chair of the curriculum committee, the director of OME, and the chairs of the preclinical education and clerkship directors committees. A written and oral summary of the report is given to the curriculum committee for review and discussion. 

Graduate evaluation of preparation for practice.  JABSOM sends an Alumni Survey to graduates six years after graduation when most are thought to have completed their postgraduate training. This survey focuses on graduates’ perceptions of their preparation for medical practice, thereby providing a long view of the effectiveness of the JABSOM curriculum. A written report of the findings is sent to the chair of the Curriculum Committee, the director of OME, and the chairs of the preclinical education and clerkship directors committees. A written and oral summary of the report is given to the curriculum committee for review and discussion.  

Student evaluation of teachers.  Preclinical students use a standardized form to evaluate their tutors and lecturers at mid-course and again at the end of the course. The evaluations consist of both ratings and narrative responses. At the end of every clerkship, third-year students complete an evaluation form for all faculty members with whom they have had learning experiences. Fourth-year students in electives, geriatrics, and emergency medicine evaluate their preceptors at the end of the course.  

Faculty peer review of courses and clerkships.  Each course and clerkship is systematically reviewed every three years by a set of two faculty members designated by the curriculum committee. The review process is outlined in a handbook which is distributed to the directors of the course or clerkship and to the faculty reviewers. Directors and reviewers are encouraged to work collaboratively during the review process in pursuit of innovative curriculum improvement. Faculty reviewers prepare a report of the findings, which they present to the curriculum committee with the relevant course or clerkship directors in attendance. Following a discussion, recommendations are given to the directors as needed. Course directors are required to respond to the recommendations with a written and oral report to the curriculum committee, thereby completing the communication loop.

How the information is used for curriculum review and change

The student and program evaluation data are used to evaluate student acquisition of the knowledge, skills, behaviors, and attitudes described in JABSOM’s educational objectives. The multi-trait, multi-method data collected from students, faculty, graduates, and residency program directors inform educational leaders about the effectiveness of the overall curriculum as well as each individual course and clerkship. The information guides decision-making at all curricular levels and allows the school to monitor the effects of changes that are made to the curriculum on the basis of these decisions. The data collected is presented and reviewed by the JABSOM Curriculum Committee and other major stakeholders in the educational program.