Unit: Learning Design and Technology
Program: Educational Tech (MEd)
Degree: Master's
Date: Fri Oct 11, 2013 - 8:29:56 am

1) Below are your program's student learning outcomes (SLOs). Please update as needed.

The Department of Educational Technology (ETEC) delivers courses and programs at multiple levels including: undergraduate, masters-level graduate, and a doctoral emphasis. At the master’s level, our Masters in Educational Technology is designed for candidates in many learning environments including K-12 and higher education, government, business, industry, and health occupations, whether they are teachers, trainers, developers, administrators, or support personnel. The program places emphasis on applications of technology in educational settings rather than just technical skills. Individuals from diverse backgrounds immediately apply what they learn to their particular contexts. Upon graduation, these new professionals will have a clearer vision of how to prepare learners for the future.

The University of Hawaii and the College of Education support the use of national standards to guide programs; therefore, ETEC has chosen to incorporate Association for Educational Communications and Technology (AECT) standards into the objectives for our program. Recently, the Hawaii Department of Education has incorporated technology standards within content standards so no longer includes educational technology as a discrete category. Since ETEC students come from a broad variety of professions, AECT standards were determined as the best fit for our program. Students are provided the following information in the UH catalog, on the website, and in advising documents.

The department has set the following student learning outcomes for its graduate students based on national standards for accreditation from the Association for Educational Communications and Technology (AECT):

1.    Design – Proficiency in instructional design, the systematic approach to designing educational/instructional systems, materials, and processes, including analyzing, designing, developing, implementing, and evaluating.

2.    Development - Demonstration of major instructional models and their technological applications to develop instructional materials and experiences using print, multimedia, computer-based, and integrated technologies.

3.    Utilization - Application of principles and theories of media utilization, diffusion, implementation, and policymaking, as well as, the attitudes, ethics, and, interpersonal and communication skills required for active involvement in appropriate professional organizations and community services.

4.    Management - Ability to plan, organize, coordinate, and supervise instructional technology by applying principles of project, resource, delivery system, and information management.

5   Evaluation – Capability of planning and executing research using knowledge of the existing body of research in the field, and, ability to evaluate the adequacy of instruction and learning by applying principles of problem analysis, criterion-referenced measurement, formative and summative evaluation, and long-range planning.

2) Your program's SLOs are published as follows. Please update as needed.

Department Website URL: http://etec.hawaii.edu
Student Handbook. URL, if available online: http://etec.hawaii.edu/programsmed.html
Information Sheet, Flyer, or Brochure URL, if available online: NA
UHM Catalog. Page Number:
Course Syllabi. URL, if available online: NA
Other: Advising Documents
Other:

3) Select one option:

Curriculum Map File(s) from 2013:

4) For your program, the percentage of courses that have course SLOs explicitly stated on the syllabus, a website, or other publicly available document is as follows. Please update as needed.

0%
1-50%
51-80%
81-99%
100%

5) Did your program engage in any program assessment activities between June 1, 2012 and September 30, 2013? (e.g., establishing/revising outcomes, aligning the curriculum to outcomes, collecting evidence, interpreting evidence, using results, revising the assessment plan, creating surveys or tests, etc.)

Yes
No (skip to question 14)

6) For the period June 1, 2012 to September 30, 2013: State the assessment question(s) and/or assessment goals. Include the SLOs that were targeted, if applicable.

We collected data through rubrics on each of the 5 standard in our field, AECT. Data are recorded in the College of Education Student Information System (SIS).

1. Design – Proficiency in instructional design, the systematic approach to designing educational/instructional systems, materials, and processes, including analyzing, designing, developing, implementing, and evaluating.

2. Development - Demonstration of major instructional models and their technological applications to develop instructional materials and experiences using print, multimedia, computer-based, and integrated technologies.

3. Utilization - Application of principles and theories of media utilization, diffusion, implementation, and policymaking, as well as, the attitudes, ethics, and, interpersonal and communication skills required for active involvement in appropriate professional organizations and community services.

4. Management - Ability to plan, organize, coordinate, and supervise instructional technology by applying principles of project, resource, delivery system, and information management.

5 Evaluation – Capability of planning and executing research using knowledge of the existing body of research in the field, and, ability to evaluate the adequacy of instruction and learning by applying principles of problem analysis, criterion referencedmmeasurement, formative and summative evaluation, and long-range planning.

7) State the type(s) of evidence gathered to answer the assessment question and/or meet the assessment goals that were given in Question #6.

The program employs an assessment system specific to its professional field. There are multiple sources of knowledge, skills and disposition data with multiple collection points (entry, middle, exit). The data are analyzed, shared and discussed among department faculty to make decisions about students and the program. An annual retreat allows faculty to reassess the vision, mission, goals and objectives of the program. The Educational Technology master’s program common assessment element with the unit is the conceptual framework. The program prepares professionals who are KNOWLEDGEABLE, EFFECTIVE, and CARING. The ETEC Masters uses six discrete assessments that are evaluated using detailed rubrics aligned to AECT standards.

(1) The Practicum Project/Paper is a culminating, year-long research or design project and paper. They may choose one of three types of projects: instructional design, action research, or other approved topic. ETEC 687 supports proposal writing. ETEC 690, during final semester, supports completion of the project and the culminating public presentation at an international online conference.

(2) The Electronic Portfolio is a collection of work in the program that clearly presents a student’s knowledge, skills, attitudes, and professionalism. The portfolio is initiated in ETEC 602. It is completed and formally presented during the ETEC 690. In it, students (a) provide a resume, (b) showcase their best work, and (d) provide reflections of their growth including how they achieved the AECT standards.

(3) The Front-end Analysis is conducted in the first semester of a core course. It includes: (1) the characteristics and interactions of a system, (2) a dissemination and diffusion plan with an analysis of a system with regard to change characteristics and an adoption plan, (3) a needs assessment plan and needs statement, (4) alternate solutions with justifications, (5) a professional mediated presentation.

(4) Practicum Project – The required practicum course 687 provides a field/clinical experience to put instructional design theories and research skills into practice in a guided setting. Students take this course during the first semester of their final year. They are responsible for choosing a topic, deciding on a schedule through a learning contract, and writing and presenting a proposal for their practicum project. Approval of their practicum project plan is a requisite for enrolling in 690, the capstone course where students will implement their project and publish the results.

(5) The Instructional Design Project is conducted in the second semester of a core course. The student must achieve project milestones within the time allotted. It includes the following steps: (1) Identify instructional goals. (2) Conduct an audience analysis. (3) Conduct an instructional analysis. (4) Write performance objectives. (5) Develop criterion referenced test items. (6) Develop an instructional strategy. (7) Develop an instructional module. (8) Design and conduct a formative evaluation. (9) Recommend and make revisions. (10) Make a professional mediated presentation of the project.

(6) The Technology Project is conducted in the first semester of a core course. Its purpose is to prepare a product that promotes a particular instructional technology or application. The target audience is educators involved in teaching or training who need to stay current with technology trends. The product may be a website, video, animation, slide presentation, brochure or any other medium that allows for independent viewing. It includes the following steps: (1) Create a content outline. (2) Create a storyboard or flowchart. (3) Decide on media format (4) Identify software hardware needed. (5) Identify skill needs and locate or arrange for training. (6) Produce the promotional product. (7) Make a professional mediated presentation.

8) State how many persons submitted evidence that was evaluated. If applicable, please include the sampling technique used.

A total of 15 rubric assessments were submitted, one by each course instructor. Six different instructors were involved. The sample included all students enrolled in the 15 courses/sections involved.

9) Who interpreted or analyzed the evidence that was collected? (Check all that apply.)

Course instructor(s)
Faculty committee
Ad hoc faculty group
Department chairperson
Persons or organization outside the university
Faculty advisor
Advisors (in student support services)
Students (graduate or undergraduate)
Dean/Director
Other:

10) How did they evaluate, analyze, or interpret the evidence? (Check all that apply.)

Used a rubric or scoring guide
Scored exams/tests/quizzes
Used professional judgment (no rubric or scoring guide used)
Compiled survey results
Used qualitative methods on interview, focus group, open-ended response data
External organization/person analyzed data (e.g., external organization administered and scored the nursing licensing exam)
Other:

11) For the assessment question(s) and/or assessment goal(s) stated in Question #6:
Summarize the actual results.

The assessment data shows that our master’s candidates have performed well in all three major areas: content knowledge; professional and pedagogical knowledge, skills and dispositions; and student learning. It has also been shown that the assessment supports the fulfillment of all five AECT standards for master’s preparation: design, development, utilization, management and evaluation.

Content Knowledge

Content knowledge is measured through two assessments in our program: the Practicum Project and Paper and the Portfolio. The Practicum Project and Paper assessment items are primarily aligned with the Design and Management standards of AECT, while the Portfolio assessment items are primarily aligned with the Development standards. However, all 5 AECT standards are addressed between both assessments. While our candidates' content knowledge is consistently assessed in all of the courses in the program, and especially in the 6 required courses, the master's project and paper and culminating electronic portfolio are considered the best measures for overall content knowledge in Educational Technology.

Candidates have done remarkably well in meeting all of the standards for both assessments with a 100% success rate. Much of the success may be due to the long process involved in completing both products, where ample opportunity for feedback to the candidate is provided.

Professional and Pedagogical Knowledge, Skills and Dispositions

The assessments for professional and pedagogical knowledge, skills and dispositions are determined from rubric scores from the Front-end Analysis conducted in ETEC 600 - Theory & Practice in Ed Tech and ETEC 687- Instructional Development Practicum. Between the two courses, all of five AECT Standards are covered: Design, Development, Management and Evaluation. The total scores for each course fall well into the “acceptable” and “target” range indicating that students in general are meeting standards in professional and pedagogical knowledge, skills and dispositions. We plan to look more closely and be more systematic about ensuring that both the on-campus and online cohorts are well aligned, especially cases where two instructors teach the same course. Currently, candidates receive feedback from the course instructors and proceed to improve accordingly. Between the initial semester assessment, the Front-end Analysis and the mid-point Practicum assessment, students gain a measure of sophistication in their knowledge, skills, and dispositions. It is clear that by the time they serve clients, such as College of Education faculty, they are well-received. The additional feedback they receive from such client groups adds authentic assessment of their capabilities.

Effects on Student/Client Learning in a Supportive Environment

The Instructional Development Project from ETEC 613 and the ETEC 602 Technology Promotion Project are used to assess our master’s candidate’s effects on providing supportive learning environments for student or client learning. Between the two project outcomes, four of five AECT Standards are primarily covered: Design, Development, Utilization and Evaluation, although all five Standards were addressed in the assessment rubrics. Candidates’ acceptable and target rating on both assessments indicate that they are doing well in meeting the standards for student/client learning in a supportive environment.

The Instructional Design Project (Assessment 5) and Technology Project (Assessment 6) are based on candidates' ability to apply a systems approach to designing and delivering and evaluating instruction or training. Candidates receive both peer and instructor feedback on both projects and revisions are an ongoing process. The ID Project emphasizes the small group testing of instructional modules for formative evaluation. This experience helps to guide the improvement of student learning using actual data.

While our assessment indicates that candidates are quite capable of meeting the standards aligned with both projects, the department would like to improve its approach to collecting data. Currently, both projects are completed by teams of 2-3 candidates. Under these conditions, it is sometimes difficult to determine a true score for an individual. Assessment items and course activities will need to be reviewed and revised to accommodate the assessment of individual performance.

12) State how the program used the results or plans to use the results. Please be specific.

Overall, faculty feel positive about the results of our 6 assessments for meeting our program goals. Students appear to be achieving the goals with no major problems. The faculty have taken steps to make changes when needed to improve student and program performances. While regular faculty meetings and annual faculty retreats allow for formal discussions about improvements, the department is small and frequent informal discussions about students and the program are conducted as needed. Ideas generated from these discussions are brought to the entire faculty for decisions.

13) Beyond the results, were there additional conclusions or discoveries?
This can include insights about assessment procedures, teaching and learning, program aspects and so on.

We look forward to growing as a program as we continue to learn from our assessments and continue to improve our assessment system. We have been highly successful based on the assessments we have done but now face revising our program and assessments as new standards have been adopted by AECT. We expect to take the results from our existing design and apply what we have learned in going forward with the revisions.

14) If the program did not engage in assessment activities, please explain.
Or, if the program did engage in assessment activities, please add any other important information here.

N/A