Evaluation of Awards & Projects
Engage collaborates with campus partners and faculty awardees to evaluate the effectiveness of instructional technologies. The aim of Engage evaluation is:
- To measure the effectiveness of instructional technologies in enhancing student learning and/or student’s perception of learning across a variety of disciplines and learning activities.
- To measure how the effective use of technology for teaching and learning aligns with various pedagogical frameworks and campus initiatives.
- To share methods of good practice for effectively implementing technology into teaching.
- To measure the impact of Engage Award Programs on teaching and learning (with and without technology) on our campus and beyond.
2007-2009: Simulations and Games, Round 1
Evaluation of the Simulations and Games projects includes a variety of methodologies from focus groups to experimental studies. The purpose of evaluation is to gather evidence of the student experience to inform future simulation and game design and measure the effectiveness of instructional games and simulations on student learning. View the evaluations from Round 1 here.
Engaging to Learn: Simulations & Games 2007-2012
Digital Media Assignments 2009-2012
Technology-Enhanced Collaborative Group Work 2007-2010
2010 Follow-up Survey 1-year out (.pdf)
Evaluation Report (.pdf)
Data Party - Consultant Meeting (Oct. 8, 2009) (.pdf)
Data Party - Engage Faculty Advisory Group Meeting (Jan. 12, 2010) (.pdf)
View the one-pager here (.pdf)
Evaluation and Assessment Presentation (.pdf)
Podcasting & Podcasting Plus 2005-2007
Technology-Enhanced Collaborative Group Work Follow-up Survey 2010
2011 Follow-up Survey: 2 years out 1-pager
2-year Follow-up Survey Results
The 2008-2009 Engage Adaptation Award on Technology-Enhanced Collaborative Group Work (TECGW) supported campus initiatives including the Strategic Framework, the IT Strategic Plan, LEAP, the Wisconsin Idea and the Wisconsin Experience. As part of the program evaluation a survey was sent to 41 instructors who were involved in the Engage TECGW Adaptation Award. The purpose of the survey was to find out: 1.) Current use of collaborative technologies to support group work, 2.) How, if at all, the award program had an impact on teaching and research, 3.) If the award helped bring in more funding for teaching with technology, 4.) If technology-enhanced group work results are being disseminated to a larger audience. Thirty-five responses were collected for a 85.4% response rate.
- 65% are still using a collaborative technology for group work
- 79% indicated the award changed the way they teach
- 85% have explored other instructional technologies as a result of the award
- 45% have presented on their experience with Technology-Enhanced Group Work
Podcasting Awardee Follow-up Survey 2010
In November 2010, a brief 3-year follow-up survey was sent to 92 people who were involved in the Podcasting or Podcasting Plus Engage Adaptation Awards. The purpose of the survey was to find out if the past awardees were still using podcasting in their teaching and if they were sharing their experiences with campus and beyond. Fifty-one people responded, for a response rate of 55%.
- 71% of respondents are still using podcasting in their teaching
- 40% of respondents have published or presented on their experience using podcasting in their teaching
- Presentations have been done locally, nationally, and internationally
Technology Enhanced Collaborative Group Work 2008-2009
Data Party - Consultant Meeting (Oct. 8, 2009)
Data Party - Engage Faculty Advisory Group Meeting (Jan. 12, 2010)
The 2008 Engage Technology Enhanced Collaborative Group Work (TECGW) Adaptation Award provided 40 instructors with technical and pedagogical support to implement technology enhanced group projects in their courses. The support included a literature review of research related to group work, pedagogical and instructional technology consultation, evaluation consultation, support for technology tools and services, and hosting community events to share challenges and solutions in facilitating group projects.
An evaluation of the program was conducted to gather evidence of methods of good practice in facilitating technology enhanced group projects. The recommendations below, resulting from the instructor, student, and consultant surveys, provide methods of good practice for instructors planning to implement technology enhanced group projects. The results of our award program, in conjunction with the literature on group work, provide evidence of how task design, group process facilitation, and technology choice influence the success of the group project. The recommendations follow.
Be sure the task is worthy of collaborative group work
- Consider the pedagogical challenge. Will a group add value to the task?
- Specify how the group project aligns with the course learning objectives.
Design the task
- Identify the types of interactions desired (student-student, student-instructor, student-content).
- Identify the tasks students will need to complete.
- Identify challenges students might face. Where have students had problems in the past?
- Choose a technology tool that aligns with the interactions, tasks, and challenges.
- Develop an assessment plan for the process as well as the assigned product.
Prepare students for collaboration
- Communicate the rationale for group work, including why the challenge is worthy of collaborative group work and how the group project relates to course objectives.
- Help your students understand what makes good collaboration.
Do regular assessments of process and product
- Use milestones or check-ins to ensure groups are on the right track.
- Provide students the opportunity to evaluate their group members’ contributions to the group project.
Podcasting Awardee Follow-up 2008
Participants of the Podcasting Adaptation Award completed a follow-up survey in fall 2008. The purpose of the follow-up survey was to find out 1.) Current use of podcasting for teaching and learning, 2.) How, if at all, the award program had an impact on teaching and research, 3.) If the award helped bring in more funding for teaching with technology, 4.) If podcasting results are being disseminated to a larger audience. Fifty-nine responses were collected for a 59.6% response rate. A brief summary of the results:
- 74% of the respondents indicated they are still using podcasting in their teaching.
- 75% indicated they have explored other instructional technologies as a results of the Podcasting Adaptation Award.
- 79% of respondents indicated the Podcasting Award changed they way the teach.
- One awardee used the podcasting experience to secure a 5-year, $40,000 grant to develop a new undergraduate course.
- 29% of respondents have presented on podcasting at various professional conferences, nationally and internationally.
- 9% of respondents have published articles related to podcasting in teaching.
The Podcasting Adaptation Award evaluation included pre- and post-surveys for awardees and students. A list of suggestions from the findings was collected from the surveys to provide methods of good practice to be shared with other instructors wishing to experiment with podcasting. The recommendations are the following:
- Consult with other instructors and department support staff about their experiences using podcasting.
- Visit the Engage website for examples of effective podcasts and instructions.
- Start early, and allow plenty of time to create the podcast episodes.
- Have a clear idea of why you are incorporating podcasts into your course.
- Communicate the rationale for using podcasting to your students.
- Provide students with instructions for accessing the podcasts- don’t assume they already know how to do it.
- Make listening to the podcasts a required part of the course.
- Commit to a schedule for developing podcast episodes.
- Keep the podcasts short.
- Enjoy it.