Departmental Assessment Update - Education Report

Department: Educational Technology
Program: MEd
Level: Graduate

1. List in detail your graduate Student Learning Outcomes (SLOs) for each degree/certificate offered.

This nationally accredited program prepares students to create resources for teaching and learning through diverse media as well as integrate technology into educational environments. The Educational Technology (ETEC) Masters Program:

·     Provides theoretical knowledge and scientific principles that can be applied to problems that arise in a social context.

·     Prepares individuals to devise effective messages, teams, materials, devices, techniques, and settings.

·     Involves the study of theory and practice of design, development, utilization, management, and evaluation of processes and resources for learning.

Practitioners in educational technology, whether they are teachers, trainers, developers, administrators, or support personnel, seek innovative and effective ways of organizing the teaching and learning process through the best possible application of technological developments.

The program places emphasis on applications of technology in educational settings rather than simple technical skills. Individuals from diverse backgrounds can immediately apply what they learn to their particular context. Upon graduation, these new professionals will have a clearer vision of how they can prepare learners for the future. ETEC graduates are found in many learning environments including K-12 and higher education, government, business, industry, and health occupations.

The Masters in Education in Educational Technology program requires a minimum of 36 semester credit hours, with 8 required and 4 elective ETEC courses. The selection of electives is based on students’ orientation and career plans within the broad field of Educational Technology. Full-time students typically complete the program in two academic years, while part-time students may take longer. Initial courses are offered in a cohort system where students take their first-year required classes together to build support and professional community. The program culminates with student presentations of their electronic portfolio and final ETEC Master's project.

The department has set the following objectives for its graduate students based on national standards for accreditation:

1.   Design – Proficiency in instructional design, the systematic approach to designing educational/instructional systems, materials, and processes, including analyzing, designing, developing, implementing, and evaluating.

2.   Development - Demonstration of major instructional models and their technological applications to develop instructional materials and experiences using print, multimedia, computer-based, and integrated technologies.

3.   Utilization - Application of principles and theories of media utilization, diffusion, implementation, and policymaking, as well as, the attitudes, ethics, and, interpersonal and communication skills required for active involvement in appropriate professional organizations and community services.

4.   Management - Ability to plan, organize, coordinate, and supervise instructional technology by applying principles of project, resource, delivery system, and information management.

5.   Evaluation – Capability of planning and executing research using knowledge of the existing body of research in the field, and, ability to evaluate the adequacy of instruction and learning by applying principles of problem analysis, criterion-referenced measurement, formative and summative evaluation, and long-range planning.

2. Where are these SLOs published (e.g., departmental web page)?

Department webpage: http://etec.hawaii.edu/masters.html?p=obj

UH Catalog and advising documents


3. Explain how your SLOs map onto your curriculum, i.e., how does your program of graduate studies produce the specific SLOs in your students?

Most courses cover some aspects of the SLOs. However, to ensure the students’ success in achieving the objectives, there is at least one core course for each objective. The students’ practicum courses, final Masters project, and electronic portfolios provide a complete venue for students to display their newly acquired knowledge and skills. The following shows how each objective is covered in the curriculum.

1.   Design – Proficiency in instructional design, the systematic approach to designing educational/instructional systems, materials, and processes, including analyzing, designing, developing, implementing, and evaluating. 

There are 2 core courses that cover this objective cover theory and application ETEC 600 Theory & Practice in Educational Technology and ETEC 603 Instructional Design & Development. In addition, students take their choices of 4 electives that provide development skills in specific context, such as, ETEC 620 Visual Design, ETEC 630 Video/TV Design, ETEC 645 Design Web-based Instruction, ETEC 647 Hypermedia Design, etc.

2.   Development - Demonstration of major instructional models and their technological applications to develop instructional materials and experiences using print, multimedia, computer-based, and integrated technologies.

ETEC 602 Teaching/Training Technologies and ETEC 603 Instructional Design & Development are the core course that addressed this objective. In addition, students take their choices of 4 electives that provide development skills in specific context, such as, ETEC 644 Programming Concepts/Structures, ETEC 649 Development of Online Courseware, etc. The required ETEC 687 Instructional Development Practicum allows students to practice their knowledge and skills in an authentic setting.

3.   Utilization - Application of principles and theories of media utilization, diffusion, implementation, and policymaking, as well as, the attitudes, ethics, and, interpersonal and communication skills required for active involvement in appropriate professional organizations and community services.

The core and elective courses cover utilization particular to the technology covered. In addition, specific courses go into utilization in depth. For example ETEC 661 Distance Education Technology, ETEC 662 Computer Networks in Education, ETEC 663 Planning for Technology and Resources, ETEC 664 Technology and Instructional Applications, ETEC 686 Information Literacy and Learning Resources, etc.

4.   Management - Ability to plan, organize, coordinate, and supervise instructional technology by applying principles of project, resource, delivery system, and information management.

The required ETEC 650 Instructional Technology Services Management is the primary course in this area. In addition, other courses, such as, ETEC 600 Theory & Practice in Educational Technology, ETEC 663 Planning for Technology and Resources, ETEC 664 Technology and Instructional Applications, cover some aspects of management.

5.   Evaluation – Capability of planning and executing research using knowledge of the existing body of research in the field, and, ability to evaluate the adequacy of instruction and learning by applying principles of problem analysis, criterion-referenced measurement, formative and summative evaluation, and long-range planning.

There are 4 core courses that cover this objective. ETEC 603 Instructional Design & Development and ETEC 601 ET Research Review cover theory and application. ETEC 688 Educational Technology Practicum, ETEC 750 (Alpha) Seminar in Educational Technology Issues are used to support students in conducting a literature review, writing their research Masters projects, and in analyzing and reporting their data.

4. What population(s) is covered by your assessment(s)?

There are two levels of assessment - program assessment and student assessment.

Program Assessment

The department uses a self-report survey instrument was aligns directly with the national standards. The instrument asks students to assess their progress in the 5 objectives or student learning outcomes. Data is collected from students as they enter the program, as they advance to candidacy, and as they complete the program.

Data is also collected from graduates of the program at annual networking events. Students and graduates are asked individually and in focus groups to provide specific feedback with regard to program improvement.

Faculty meet monthly to discuss current departmental issues including the program and its assessment. Additionally, the faculty hold a day-long annual retreat to discuss the data collected over the past year and their observations regarding the program. Short and long term goals are set and revisions are made to the program.

Students

Students are assessed throughout the program on skills, knowledge, and dispositions, in a number of ways.  Courses provide rigorous assessment throughout the program.  Students begin an electronic portfolio in their first core course and collect artifacts throughout the program. Students also complete a rigorous Masters paper and project throughout the last year of the program. They show both the portfolio and project in their culminating oral presentation.

In addition, a self-report instrument is provided to them regarding their dispositions for success in the program. They complete this as they enter the program, advance to candidacy and exit the program. The faculty are currently working on a way to better incorporate this data into the assessment system.

The following is an example of the results of the survey.


ETEC SELF-ASSESSMENT SUMMARY

DOMAINS OF AN EDUCATIONAL TECHNOLOGIST

 

1=Low    2=Somewhat Low    3=Average    4=Somewhat High    5=High

 

                                              Before entering                      Since entering

Status in Program

Entry

Middle

Exit

Post

 

Entry

Middle

Exit

Post

Number of students

14

45

33

15

 

14

45

33

15

 

 

 

 

 

 

 

 

 

 

DESIGN

 

 

 

 

 

 

 

 

 

  • Confident

2.46

2.28

1.97

1.77

 

3.93

4.00

4.27

4.42

  • Knowledgeable

2.39

2.25

1.83

1.75

 

4.06

4.01

4.22

4.44

  • Skillful

2.40

2.21

1.87

1.72

 

4.05

3.85

4.17

4.41

TOTAL

2.41

2.25

1.89

1.75

 

4.01

3.95

4.22

4.42

 

 

 

 

 

 

 

 

 

 

DEVELOP

 

 

 

 

 

 

 

 

 

  • Confident

2.77

2.67

2.06

1.98

 

3.84

3.80

4.34

4.44

  • Knowledgeable

2.88

2.60

1.97

2.05

 

3.80

3.81

4.25

4.47

  • Skillful

2.88

2.59

1.99

2.05

 

3.75

3.71

4.25

4.47

TOTAL

2.84

2.62

2.01

2.03

 

3.80

3.77

4.28

4.46

 

 

 

 

 

 

 

 

 

 

UTILIZE

 

 

 

 

 

 

 

 

 

  • Confident

2.51

2.33

1.88

1.67

 

3.67

3.71

4.15

4.27

  • Knowledgeable

2.46

2.25

1.75

1.70

 

3.78

3.68

4.09

4.32

  • Skillful

2.49

2.28

1.73

1.68

 

3.72

3.62

4.17

4.25

TOTAL

2.49

2.19

1.79

1.68

 

3.72

3.67

4.15

4.28

 

 

 

 

 

 

 

 

 

 

MANAGE

 

 

 

 

 

 

 

 

 

  • Confident

2.67

2.53

2.01

1.70

 

3.77

3.74

4.17

4.52

  • Knowledgeable

2.51

2.52

2.06

1.72

 

3.76

3.80

4.14

4.52

  • Skillful

2.49

2.54

2.03

1.68

 

3.70

3.75

4.08

4.45

TOTAL

2.56

2.53

2.03

1.70

 

3.74

3.76

4.13

4.49

 

 

 

 

 

 

 

 

 

 

EVALUATE

 

 

 

 

 

 

 

 

 

  • Confident

2.43

2.30

1.99

1.57

 

3.88

3.76

4.14

4.07

  • Knowledgeable

2.39

2.31

1.97

1.55

 

3.98

3.80

4.17

4.07

  • Skillful

2.34

2.32

1.96

1.59

 

3.88

3.83

4.06

4.04

TOTAL

2.39

2.31

1.97

1.57

 

3.91

3.80

4.13

4.06 



5. Please list/describe all the assessment events and devices used to monitor graduate student progress through the program. Consider the following questions:

Courses

The majority of the courses require students to produce authentic semester-long projects that tests  knowledge and aptitude with the course content. The ETEC program has a reputation of being a very rigorous program. This is partially due to the use of carefully designed checklists and rubrics used to evaluate the projects. As students are shown past projects, they compete each year to do better and better projects. The majority of the courses require culminating oral presentations that are enhanced with technology to showcase their course project.

Many of the courses require that students work in groups to produce course projects for the purpose of improving their dispositions in a field that is very collaboratively based. To counter-balance this, however, the faculty are continually working to ensure that students are also evaluated on an individual basis as well. Most often, this is in the form of take-home essay exams, individual research papers, and projects.

General Departmental Examination

At the end of the Spring semester in which the student completes all of the Educational Technology core courses, a faculty committee will administer the General Departmental Examination (oral) to determine whether advancement to candidacy should be recommended to the Graduate Division.

The departmental examination, conducted as an interview, is designed to ascertain whether the prospective candidate's level of academic work, personal characteristics, poise, attitude and communication skills are indicative of the student's likelihood of success as an Educational Technology professional by giving the student an opportunity to assure the faculty that:

a) A feasible plan for the final project has been or is being developed (for that purpose, an "idea" paper explaining the project in 2-3 pages is expected from each eligible student two weeks prior to the scheduled examination).

b) Satisfactory academic progress has been made and will continue in future semesters.

c) The student is therefore prepared professionally to continue the degree program as a formal degree candidate.

Students who have elected to take more than two years to complete their ETEC M.Ed. may choose to complete and present their idea paper in the year before they plan to graduate but will be reviewed by the graduate faculty for continuation in the spring of each year prior to admission to candidacy.

The Final Project is all-important, as it represents the culmination of the student's work in the program (see below). After advancement to candidacy, when the topic of the final project or research study has been clarified, an advisor is assigned to the candidate. Solid evidence must be shown to the faculty ensuring that the project is meaningful to the student and that it is already reasonably well defined, or  advancement to candidacy will be postponed. Detailed guidelines toward orderly progress are issued to all (as document "Graduation Pre-Conditions") in preparation for the "idea" paper interview. Students are also assisted with project definition in ETEC 601, Educational Technology Research Review.

Instructional Development Practicum

The required ID Practicum course is available only during the Fall semester of the second or third year. Each student has an opportunity to apply what he or she has learned, and to add additional experiences and new perspectives to his or her professional knowledge and skill set. The practicum is conducted as a group enterprise, with emphasis on real-world situations. Most often, the experience consists of designing and delivering workshops to a defined target audience. Regular class meetings are a part of the 3-credit ETEC 687 "Instructional Development Practicum" course. Minimally, students will have completed at least two (2) electives prior to enrolling in the Practicum course or must have instructor approval.

Electronic Portfolio and Final Project

An electronic portfolio that clearly presents the student's knowledge, skills, attitudes, and professionalism in the field of Educational Technology is to be completed by the conclusion of studies in the program. Development of the portfolio is initiated in ETEC 602 Teaching/Training Technologies; continued independently by the student with advice by his or her advisor; completed during the ETEC 688 Educational Technology Practicum; and formally presented in that practicum. As part of the electronic portfolio, students develop and write the results of their final project and also include a resume, showcase of best work, and reflections of their growth in the program and on artifacts demonstrating how they have met AECT standards. Students are required to post their final electronic portfolios on the ETEC web site.

In any good graduate program, the student ends his/her program with a presentation of a major research or professional study. Often, such a study is called a "thesis", though at the University of Hawaii a thesis is a study specifically done under the "thesis plan" (Plan A), while a study done under the "non-thesis plan" (Plan B) is usually referred to as a Seminar Report or Final Project. Students may choose one of three types of projects under Plan B, the option selected by most students: an instructional design/development project; an action research project; or other Department approved project, such as, a needs assessment, task analysis, etc. Students may choose Plan A (thesis) when the project they are conducting better suits these more research focused requirements and/or they potentially plan to continue in a doctoral program. Topic selection for the final project is part of ETEC 601, "Educational Technology Research Review," and support for completion of the project and the culminating presentation is provided in ETEC 688, ET Practicum, during the student's final semester.

Final Oral Examination

ETEC majors, whether under Plan A or Plan B, complete their program officially with a final oral examination, which culminates their program of study. The examination is given only in the spring semester as part of the ETEC 688 course, after the final project paper is read by the appropriate faculty. The examination is intended to be a comprehensive one, though the description, explanation and showcasing of the final study and electronic portfolio form the major part of it. The examination is open to other students as well as guests.

6. Please list/describe how your graduate students contribute to your discipline/academic area? Consider the following questions:

Beginning with one of their first core courses, ETEC 602 Teaching/Training Technologies, students are introduced to our professional organizations and encouraged to submit articles for publications and presentations at conferences. In several courses, students are required to write and present papers for international and on-line conferences. One course, ETEC 750 - Seminar in Educational Technology Issues, beginning this fall, will require that students write articles and publish a new online journal for our field. In addition, students involved in research and grant projects often have the opportunity to co-author with faculty in journal publications, as well as, conference presentations.

7. What attempts are made to monitor student post-graduate professional activities?

Graduates of the program are expected to assume leadership roles in a multitude of areas, including, but not limited to, those who work in K-12 schools, State Departments of Education, higher education institutions, as well as, government, health, business, and industry settings. Their future careers may involve teaching, training, instructional design, curriculum development, online course development, technology utilization, management, or research. Many of our graduates are employed in the UH system. We are extremely proud of our graduates and they are well sought after.

ETEC for many years has made it a priority to create an ETEC ohana (family). Students are introduced into the program through an orientation that includes presentations from graduates of the program. They are encouraged to become involved in the ETEC Graduate Student Organization called the ETEC Hui. These students throughout the program are involved in organizing various events that include students and graduates. We try to systematically involve students and graduates in evaluating the program.

The program keeps the ETEC ohana together through a number of email lists addressed to current students, graduates and friends of the program. Frequent communications keep the graduates updated on what is happening in the program.

 

8. How were the assessment data/results used to inform decisions concerning the curriculum and administration of the program?

Once a year the faculty hold a day-long retreat to discuss the data collected over the past year and their observations regarding the program. Goals are set for the year and revisions are made to the program. There are changes in all areas: pedagogy, administration, advising, degree requirements, and courses.

The department has created both major changes and minor changes. Two major initiatives have been the addition of a fully online program that began fall 2006. Including the campus-based program, our previous enrollment is now doubled. We also now have a Doctoral Specialization in Educational Technology as part of the College of Education PhD program. This will allow students to continue their studies beyond the Masters level.  It begins this fall 2007.

The following is an example of the results of our last retreat of approved program changes based on the data collected.

Summary of Proposed Changes for the ETEC Masters program for 2007-8

1.    Eliminate ETEC 414 as a program prerequisite and redesign ETEC 602 to be a graduate version of 414 that will contain its important features. Focus this redesign on aspects that will help students succeed in the program. ETEC 602 will also continue to contain the development of ePortfolios. Move 602 to the Fall semester.

Justification:  Having 414 as a prerequisite for the program means the total number of credit hours for the program is 39 credit hours. This is not competitive with other programs around the country, or locally. In addition, having to take 414 as a true prerequisite means that some students are not ready to begin the program, and then, must wait another year to begin. Potentially, this means losing students to other programs. Or, what is currently happening is that students take the course within the first year at best, or sometimes later. Essentially, then, the course is not a prerequisite. 

The problem of 414 is further complicated in the online Master’s program. Many students are residents of other islands or abroad, and the reason they are in the online program is because they cannot come to Oahu. ETEC 414 is currently not an online course, as such, creates complications for online students. ETEC 602 is already similar to a Master’s level 414 and is online. Therefore, a review of the content and 602’s alignment with the important features of the current prerequisite should solve the problem.

2.    Require students to take a common set of core courses for the first year. ETEC 600 and 602 in the Fall, and ETEC 601 and 603 in the Spring.

Justification:  Currently, students although guided by the website and graduate chair to take courses in a particular order, they sometimes drop a course or change the order because of their particular schedule. Since they are not assigned an official advisor until they have been advanced to candidacy, they often make these decisions at the last minute without advising, creating problems for themselves and advisors as they continue the program. This eliminates the issue of them choosing incorrectly. There is a best order for courses and a reasonable commitment that students need to have to complete the program successfully. Also, students having a common set of courses, means that the instructor can have certain expectations about the prerequisite skills and knowledge students enter their courses with. Students are also better prepared to learn the course content and complete projects.

3.    Students must take a minimum of 2 UH ETEC graduate level elective courses (6 credits) after their first semester in the program. They may take a maximum of 6 credits of undergraduate courses. They may take outside electives only with departmental approval prior to taking the course.

Justification:  Currently some students are relying too heavily on undergraduate courses for their elective courses. To increase the rigor of the program, a new minimum of 6 credits of graduate level courses would be required. This would still allow students to transfer 2 courses they took in the Technology Activators Project (TAP) or Technology Intensive Enhancement Series (TIES) so that they continue to serve as good recruiting pathways into the Master’s Program.

There has also been a concern about too many requests for outside electives, especially those transferred into the program. Again, the requirement for 2 UH ETEC graduate courses reduces this problem, yet still allows students to transfer some courses from a similar ETEC program outside of UH, or, to take electives that truly enhance their Master’s project.

4.    Require an ETEC 750 seminar to meet requirement from graduate division. Students may take the seminar after they have completed the core requirements. 

Justification:  Graduate division requires that students in Master’s program take a seminar course. Currently, students are guided to take a seminar course, but do not always do so. This creates problems when they go through their graduation check. Having a clearly stated requirement of one of the ETEC 750 courses will solve this problem.

5.    Require publication of a showcase electronic portfolio on the ETEC website that includes: a resume; showcase of best works; their final project; and, reflections of their growth in the program, and on AECT standards and how they achieved them.

Justification:  Currently students are required to produce an electronic portfolio, but are not required to publicly share them on the website. Many of these products are excellent and would benefit future students through their public display. As with other programs around the country, having these displayed on the website would serve as an excellent recruiting tool. In addition, much of the data that could be captured for NCATE and other reviews are lost because there is not a consistent way of storing these portfolios. The department plans to develop a rubric to help students meet targets in each portfolio area, and to evaluate the overall success in the program for accreditation purposes. This requirement would help with these issues.

6.    Change the requirements for the final Masters Plan B project to enhance the products they produce. Students may choose one of three types of projects: an instructional design/development project; OR an action research project, OR other Department approved project, such as, a needs assessment, task analysis, etc. Encourage students to choose Plan A when the project they are conducting better suits these requirements, and potential plans to continue into a doctoral program.

Justification:  Currently most students choose Plan B as their option. Over time the Plan B requirements and resulting projects have come to resemble a Plan A thesis. This change would enhance both Plan A & Plan B projects by having them be distinctly and appropriately different. Plan A projects would have the requisite “research paper” components, such as, a problem statement, literature review, methodology, analysis, and results. Whereas Plan B projects would be practical in nature with components appropriate to the chosen project. This would particularly enhance design projects where currently the “research” components leave little time for advanced design and media production. This change should have an overall positive effect on the quality of work produced.

Summary of the Proposed Changes to the ETEC Masters Program

Eight core, four electives = 36 credits

Four electives over the course of the program, all students must complete the first year core as shown. Where electives are taken is suggested, but optional. Electives are offered every semester. Must take 688 in final semester. PLEASE NOTE: All of the required courses are only offered in the semester they are shown.

2-year program – Must follow sequence for required courses

Year 1

Fall

Spring

Summer

ETEC 600 (1st semester)

ETEC 603 (2nd semester)

*Elective (graduate level)

ETEC 602 (1st semester)

ETEC 601 (2nd semester)

*Elective

 

*Elective

 

Year 2

Fall

Spring

Summer

ETEC 687 (after core)

ETEC 688 (final semester)

 

ETEC 750 (after core)

ETEC 650 (after core)

 

*Elective (graduate level)

 

 

* Sequence of electives is optional, but 2 graduate level electives must be taken after the 1st semester


3+year program – 1st year sequence required

Year 1

Fall

Spring

Summer

ETEC 600 (1st semester)

ETEC 603 (2nd semester)

 

ETEC 602 (1st semester)

ETEC 601 (2nd semester)

 

Year 2

Fall

Spring

Summer

ETEC 687 (after core)

ETEC 650 (after core)

 

*Elective (graduate level)

*Elective (graduate level)

Year 3

Fall

Spring

Summer

ETEC 750 (after core)

ETEC 688 (final semester)

 

*Elective

*Elective

* Sequence of electives is optional, but 2 graduate-level electives must be taken after the 1st semester

9. Has the program developed learning outcomes? Please indicate yes or no.

Yes

10. Has the program published learning outcomes? Please indicate yes or no.

Yes

11. If so, please indicate how the program has published learning outcomes.

On the department website: http://etec.hawaii.edu/masters.html?p=obj

And in the UH catalogue, in advising materials, and in the ETEC survey.

12. What evidence is used to determine achievement of student learning outcomes?

As stated above in questions 4 and 5 (for more detail, see those responses):

Students are assessed throughout the program on skills, knowledge, and dispositions, in a number of ways. Courses provide rigorous assessment throughout the program. Students begin an electronic portfolio in their first core course and collect artifacts throughout the program. Students also complete a rigorous Masters paper and project throughout the last year of the program. They show both the portfolio and project in their culminating presentation.

In addition, a self-report instrument is provided to them regarding their dispositions for success in the program. They complete this as they enter the program, advance to candidacy and exit the program. The faculty is currently working on a way to better incorporate this data into the assessment system.

The majority of the courses require students to produce authentic semester-long projects that test the students’ knowledge and aptitude with the course content. The ETEC program has a reputation of being a very rigorous program. This is partially due to the use of carefully designed checklists and rubrics used to evaluate the projects. As students are shown past projects, they compete each year to do better and better projects.

The majority of the courses require culminating oral presentations that are enhanced with technology to showcase their course project.

Individually, all students are required to (1) pass a general department examination after they have advanced to candidacy, (2) successfully complete a practicum course that showcases their knowledge, (3) complete an electronic portfolio that is a collection of work accomplished in the program, (4) complete a year-long research/design project and accompanying research paper, (5) make an oral public presentation of their research and portfolio. 

13. Who interprets the evidence?

ETEC Faculty

14. What is the process of interpreting the evidence?

Once a year the ETEC faculty hold a day-long retreat to discuss the data collected over the past year, and their observations regarding the program. Goals are set for the year and revisions are made to the program. There are changes in all areas: pedagogy, administration, advising, degree requirements, and courses.

15. Indicate the date of last program review.

The College of Education has just completed our national accreditation process that involved our department during the 2006-7 academic year. ETEC conducts an internal program review annually.