Degree Program Assessment Reports

• File (03/16/2020)

The program employs an assessment system specific to its professional field. There are multiple sources of knowledge,

skills and disposition data with multiple collection points (entry, middle, exit). The data are analyzed, shared and discussed

among department faculty to make decisions about students and the program. An annual retreat allows faculty to reassess

the vision, mission, goals and objectives of the program. The Educational Technology master’s program common

assessment element with the unit is the conceptual framework. The program prepares professionals who are

KNOWLEDGEABLE, EFFECTIVE, and CARING. The LTEC Masters uses six discrete assessments that are

evaluated using detailed rubrics aligned to AECT standards.

(1) The Practicum Project/Paper is a culminating, year-long research or design project and paper. They may choose one

of three types of projects: instructional design, action research, or other approved topic. LTEC 687 supports proposal

writing. LTEC 690, during final semester, supports completion of the project and the culminating public presentation at an

international online conference.

(2) The Electronic Portfolio is a collection of work in the program that clearly presents a student’s knowledge, skills,

attitudes, and professionalism. The portfolio is initiated in LTEC 602. It is completed and formally presented during the

LTEC 690. In it, students (a) provide a resume, (b) showcase their best work, and (d) provide reflections of their

growth including how they achieved the AECT standards.

(3) The Front-end Analysis is conducted in the first semester of a core course. It includes: (1) the characteristics and

interactions of a system, (2) a dissemination and diffusion plan with an analysis of a system with regard to change

characteristics and an adoption plan, (3) a needs assessment plan and needs statement, (4) alternate solutions with

justifications, (5) a professional mediated presentation.

(4) Practicum Project – The required practicum course 687 provides a field/clinical experience to put instructional

design theories and research skills into practice in a guided setting. Students take this course during the first semester of

their final year. They are responsible for choosing a topic, deciding on a schedule through a learning contract, and writing

and presenting a proposal for their practicum project. Approval of their practicum project plan is a requisite for enrolling in

690, the capstone course where students will implement their project and publish the results.

(5) The Disposition assessment is embedded in the Instructional Design project for LTEC 613. The purpose of the course, Instructional Design and Development, is to prepare candidates to follow a systematic approach for designing instruction. Students work in teams to develop and evaluate an instructional module and are assessed for teamwork and professionalism. The Disposition assessment is conducted during the Instructional Design project ETEC 613. Since there is not a Disposition standard in the AECT standards for this report, this sub-standard was used from the New AECT standards:

Professional Knowledge & Skills: Ethics – demonstrate ethical behavior within the applicable cultural context during all aspects of their work and with respect for the diversity of learners in each setting.

(6) The Instructional Design Project is conducted in the second semester of a core course. The student must achieve

project milestones within the time allotted. It includes the following steps: (1) Identify instructional goals. (2) Conduct an

audience analysis. (3) Conduct an instructional analysis. (4) Write performance objectives. (5) Develop criterion referenced test items. (6) Develop an instructional strategy. (7) Develop an instructional module. (8) Design andconduct a formative evaluation. (9) Recommend and make revisions. (10) Make a professional mediated presentation of the project.

(7) The Technology Project is conducted in the first semester of a core course. Its purpose is to prepare a product that

promotes a particular instructional technology or application. The target audience is educators involved in teaching or

training who need to stay current with technology trends. The product may be a website, video, animation, slide

presentation, brochure or any other medium that allows for independent viewing. It includes the following steps: (1) Create

a content outline. (2) Create a storyboard or flowchart. (3) Decide on media format (4) Identify software hardware needed.

(5) Identify skill needs and locate or arrange for training. (6) Produce the promotional product. (7) Make a professional

mediated presentation.

The assessment data shows that our master’s candidates have performed well in all three major areas: content

knowledge; professional and pedagogical knowledge, skills and dispositions; and student learning. It has also been shown

that the assessment supports the fulfillment of all five AECT standards for master’s preparation: design, development,

utilization, management and evaluation.

Content Knowledge

Content knowledge is measured through two assessments in our program: the Practicum Project and Paper (Assessment 1) and the

Portfolio (Assessment 2). The Practicum Project and Paper assessment items are primarily aligned with the Design and Management standards of AECT, while the Portfolio assessment items are primarily aligned with the Development standards. However, all 5 AECT standards are addressed between both assessments. While our candidates' content knowledge is consistently

assessed in all of the courses in the program, and especially in the 6 required courses, the master's project and paper and

culminating electronic portfolio are considered the best measures for overall content knowledge in Educational Technology.

Candidates have done remarkably well in meeting all of the standards for both assessments with a 100% success rate.

Much of the success may be due to the long process involved in completing both products, where ample opportunity for

feedback to the candidate is provided.

Professional and Pedagogical Knowledge, Skills and Dispositions

The assessments for professional and pedagogical knowledge, skills and dispositions are determined from rubric scores

from the Front-end Analysis (Assessment 3) conducted in LTEC 600 - Theory & Practice in Ed Tech and LTEC 687- Instructional

Development Practicum (Assessment 4). Between the two courses, all of five AECT Standards are covered: Design, Development,

Management and Evaluation. The total scores for each course fall well into the “acceptable” and “target” range indicating

that students in general are meeting standards in professional and pedagogical knowledge, skills and dispositions. We

plan to look more closely and be more systematic about ensuring that both the on-campus and online cohorts are well

aligned, especially cases where two instructors teach the same course. Currently, candidates receive feedback from the

course instructors and proceed to improve accordingly. Between the initial semester assessment, the Front-end Analysis and the mid-point Practicum assessment, students gain a measure of sophistication in their knowledge, skills, and

dispositions. It is clear that by the time they serve clients, such as College of Education faculty, they are well-received. The

additional feedback they receive from such client groups adds authentic assessment of their capabilities.

Effects on Student/Client Learning in a Supportive Environment

The Instructional Development Project from LTEC 613 (Assessment 6) and the LTEC 602 Technology Promotion Project (Assessment 7) are used to assess our master’s candidate’s effects on providing supportive learning environments for student or client learning. Between the two project outcomes, four of five AECT Standards are primarily covered: Design, Development, Utilization and Evaluation, although all five Standards were addressed in the assessment rubrics. Candidates’ acceptable and target

rating on both assessments indicate that they are doing well in meeting the standards for student/client learning in a supportive environment.

The Instructional Development Project and Technology Promotion Project  are based on candidates' ability to apply a systems approach to designing and delivering and evaluating instruction or training. Candidates receive both peer and instructor feedback on both projects and revisions are an ongoing process. The ID Project emphasizes the small group testing of instructional modules for formative evaluation. This experience helps to guide the improvement of student learning using actual data.

While our assessment indicates that candidates are quite capable of meeting the standards aligned with both projects, the

department would like to improve its approach to collecting data. Currently, both projects are completed by teams of 2-3

candidates. Under these conditions, it is sometimes difficult to determine a true score for an individual. Minor adjustments were made to the assessment items and course activities to accommodate the assessment of individual performance. Peer evaluations are now factored in to group assessment scores.

We look forward to growing as a program as we continue to learn from our assessments and continue to improve

our assessment system. We have been highly successful based on the assessments we have done but now face revising

our program and assessments as new standards have been adopted by AECT. We expect to take the results from our

existing design and apply what we have learned in going forward with the revisions. We expect to implement our revised standards and assessments beginning Fall 2015.