Do Online Courses Work?

Promising Evaluation Results for a Distance Learning Program for Middle School Drug and Violence Prevention Coordinators

(February, 2003) Trainers wondering whether online courses are effective and feasible—particularly in a time of budget cutbacks that make travel

The evaluation, conducted by Social Science Research and Evaluation, Inc. SSRE), examined the immediate and long-term effects of a series of web-based courses offered by HHD’s National Training and Technical Assistance Center for Middle School Coordinators (National Center).

The series was part of a continuing education program for a nationwide group of middle school drug and violence prevention coordinators (MSCs), who are funded by the federal Department of Education’s Safe and Drug-Free Schools Program to help schools identify and implement effective drug and violence prevention strategies. From April 2001 to September 2002, nine online courses were offered, each lasting four or five days. An average of 60 MSCs participated in each course, which covered topics such as conducting a needs assessment, sustaining a program, and responding to a crisis (see chart for a list of all nine events).

According to lead evaluator Scott Formica, the evaluation provided a unique opportunity to learn about the feasibility of offering professional development courses using web-based learning. “[The trainers] wanted to find out whether online courses were an effective medium for changing both knowledge and behavior,” said Formica. “Would people like taking such a course, as much as a face-to-face training? Would they find the content relevant to their work? Would they be able to apply the knowledge they gain to their practice in the real world? In all of these areas, the evaluation found encouraging results.”

Were Participants Satisfied

with the Training Experience?

Online learning is relatively new territory for trainers and participants alike, so one important measure was whether participants enjoyed online events and felt satisfied with the learning experience. Satisfaction would indicate that trainees would be likely to take another online course, and that they would consider web-based learning as a valuable alternative or supplement to face-to-face trainings.

or not they would recommend the event to other Middle School Coordinators, 95% indicated they would either “Strongly Recommend” (53%) or “Recommend” (42%) the event to others. Participants themselves were also likely to come back for more: 75% participated in more than one event.

Did They Learn the Course Content?

One of the main objectives of the evaluation was to determine whether participants learned and retained knowledge they gleaned from the course. For six of the nine events, evaluators compared results of a pre-test and a post-test to detect whether the participants learned the course content. The results were promising: for each event, MSCs made gains on between 8 and 10 items out of the 10-question test. Many of these gains were statistically significant. Even more striking was that the knowledge gains appeared to last—follow-up tests given three or six months after the course showed no significant declines in knowledge.

Did They Apply What They Learned?

Another promising result—particularly encouraging to educators who know that motivating people to act is even more important than increasing their knowledge—was that participants apparently were able to apply what they learned to their real-world situations. In follow-up questionnaires given three or six months after the events, every single respondent was able to give at least one example of how they had used what they learned. For example:

• “I use the information every time I talk to a politician, principal, or teacher to convince them of the benefit of drug prevention programming.”
  
• "My school was looking for a bullying prevention program, and we were able to look at the list of research-based programs and discuss which one would best fit the school climate.”
  
• “After the event, I met with the curriculum director to find out how to get our school to adopt a policy stating that we are an “asset-building” school district. She was open to the idea and said that we should make this addition to the policy of our school district.”
  
• “I used tactics from the events to enlist the support of several key teachers and together we worked out a school-wide prevention program that should get stronger as the years go by.”

Moving Forward

The findings from this evaluation will have important implications for HHD’s future work, and potentially on the distance learning field at large.

“The findings have been incredibly useful for shaping our new online events, such as a series of web-based courses on substance abuse prevention,” says Melanie Adler, senior writer for the MSC project and now the distance learning manager for Northeast CAPT, a substance abuse prevention project serving the northeast states. “It was critical in helping us determine what works in the world of online learning, and it confirmed that distance learning is a useful adjunct to face-to-face training, and that people will choose to participate. At this time of funding cuts, distance learning is not only a nice addition but may become a mainstay of professional development courses, so learning about its effectiveness encourages us to continue using this method.”

Trainers nationwide are also interested. The course developers have presented the evaluation results nationally, including at the annual meetings of the American Public Health Association, the Association for Educational Communications and Technology, the Association for the Advancement of Computing in Education, and at the Centers for Disease Control and Prevention’s 2002 National Leadership Conference to Strengthen HIV/AIDS Education and Coordinated School Health Programs. They have published in trade magazines focused on distance learning and technology and are also seeking publication in peer-reviewed journals.