Guides & Toolkits

17 • Institute for Work & Health (IWH) A key outcome variable would be the trainees' scores on tests of their learning taken after the training. Depending on the availability of equipment, such tests could include both a written test of knowledge and observation of trainees performing tasks using equipment that they would use at the workplace. You would then perform a statistical analysis of whether the scores are higher for those who took the modified program, while controlling for other variables (as explained above). If resources do not permit a formal evaluation using a comparison group, plan to interview instructors with experience in delivering the standard curriculum about their experiences with the modified curriculum, as a way of getting a sense of whether the curriculum changes were helpful. (This is recommended also as a supplement to quantitative data analysis if you conduct a formal evaluation.) Quick Tip In our pilot project, we collected data at the first session of each training intake on age, first language, languages spoken at home, educational attainment, experience in the industry, and experience in the hoisting and rigging occupation. These variables were used as "controls" in the analysis of the outcome data. We also used the score on the essential skills test as a control variable.* Going Beyond If resources permit and it is logistically feasible, it may be useful to conduct interviews with trainees and instructional staff to better understand their experiences and the effectiveness of the training. You may want to conduct worksite observations of trainees after they complete the training program to assess the extent to which workplace practices integrate what was taught in the classroom. It may also be useful to follow up with a sample of training graduates to learn about their experiences applying what they learned on the job. Appendix B provides a starting point for worksite observation considerations, and also includes sample interview questions for training graduates and instructional staff. *It is important to keep trainee data confidential and to de-identify the data prior to data analysis. This was done in our pilot project, although it was deemed by the University of Toronto Research Ethics Board to be exempt from research ethics review on the grounds that it "constitutes program evaluation, quality assurance (QA) or quality improvement (QI)."

• Cover