The Data Collection Challenge.
This column begins a series describing the challenges, issues and concerns that may arise when measuring the ROI of your programs and projects. This first article focuses on data collection, specifically the use of questionnaires. Data collection has always been a problem, which is sometimes exacerbated when the ROI Methodology is applied because gathering credible data for ROI analysis is imperative. While there are a variety of data collection instruments that can be used, more often than not a low-cost option is pursued. For this reason, questionnaires are the more popular data collection instrument because they can be used to collect a variety of data for minimum cost.
We will start the column with a brief review of the ROI Methodology, then move right into the issues involved in collecting data with questionnaires.
ROI Methodology.
The ROI Methodology captures six types of data categorized into five levels:
- Reaction and Planned Action (Level 1)
- Learning (Level 2)
- Application and Implementation (Level 3)
- Business Impact (Level 4)
- ROI (Level 5)
- Intangible Benefits
These datasets represent a balanced viewpoint of success of a particular program or project. The data represents qualitative and quantitative data, as well as financial and non-financial data. Data is captured through a series of systematic steps as shown in Figure 1.
Figure 1: ROI Process Model
During the evaluation planning phase, program objectives are set and the evaluation plans are completed. Upon rolling out the program or project, the data collection process begins.
Data Collection Methods.
Four types of data are collected during the evaluation process. Two types, reaction (Level 1) and learning (Level 2) data, are collected during project implementation. Two other types of data, application (Level 3) and business impact (Level 4), are collected sometime after the program when new knowledge, skills, behaviors and/or attitudes are applied on a routine basis. Figure 2 shows the different methods of data collection typically used to capture these first four levels of data.
Figure 2: Data Collection During and After Program.
| Method | Level | 1 | 2 | 3 | 4 |
| Surveys | x | x | x | ||
| Questionnaires | x | x | x | x | |
| Observation | x | x | x | ||
| Interviews | x | x | x | ||
| Focus Groups | x | x | x | ||
| Tests/Quizzes | x | ||||
| Demonstrations | x | ||||
| Simulations | x | ||||
| Action planning/Improvement plans | x | ||||
| Performance contracting | x | x | |||
| Performance monitoring | x | x |
Capturing reaction and learning data during project implementation is usually easy because the participants are often captive. They are involved in a learning session, a formal meeting or working sessions around a particular human resources program or project. The challenge comes when collecting application data (Level 3) and business impact data (Level 4). This is often done weeks or months after a project has been implemented.
A fundamental principal in selecting a data collection method is the balance between cost of data collection and accuracy of the results. The more accurate the data collection, the more expensive it will be. Concerns about resource availability often cause individuals conducting studies to default to low-cost methods, such as a questionnaire. In our most recent benchmarking, the questionnaire is the number one method used to capture Levels 3 and 4 data, which are then used in the ROI analysis. This is a dilemma. One of the weakest methods, the questionnaire, is the most used method to evaluate at the ROI level.
Use of Questionnaires.
The questionnaire is used because of the resources, cost and convenience. But it has it challenges, including what to ask. After all, if the questionnaire is the principal source for data collection for ROI analysis; it must capture data that represent business impact, as well as set the stage for the ROI calculation. Figure 3 shows a checklist of the topics covered in questionnaires designed to collect Level 3 and Level 4 data.
Figure 3: Follow-Up Questionnaire Content Issues Checklist.
| Content Issues | Optional Content Issues |
| Progress with objectives | Improvements and accomplishments |
| Action plan implementation | Improvement linked with program |
| Relevance of Program | Monetary impact |
| Perceived value | Confiednce level |
| Use of materials | |
| Knowledge/skill enhancement | |
| Skills Used | |
| Changes with work | |
| Linkage with output measures | |
| Other benefits | |
| Barriers | |
| Enablers | |
| Management support | |
| Other solutions | |
| Recommendations for target audience | |
| Suggestions for Improvement | |
| Other comments |
These are topics that reflect on reaction of participants and learning, but more importantly, they ask respondents to reflect on how knowledge, skills and information have been applied and how that application has influenced business measures. Questions are often asked to help in the step to isolate the effects of a program, as well as to convert data to monetary value. These optional questions link the program to results and convert business measures to money, allowing the respondent to do much of the work for the evaluator. Figure 4 shows a series of specific questions targeted toward impact (Level 4) and ROI (Level 5) data.
Figure 4: Key Questions Important to ROI Analysis.
| Key Questions |
| 1. How did you use the material from this project or program? 2. What influence did it have in your work? Team? 3. What specific measure was influenced? Define it. 4. What is the unit value of the measure? (Profit or Cost) 5. What is the basis of this value? 6. How much did the measure change since the project was implemented? 7. What is the frequency of the measure? Daily, weekly, monthly, etc. 8. What is the total annual value of the improvement? 9. List the other factors that could have caused this total improvement? 10. What percent of the total improvement can be attributed to this project? 11. What is your confidence estimate, expressed as a percent, for the above data? 0 percent = no confidence, 100 percent = certainty |
The Challenge with Questionnaires.
While questionnaires allow us to ask a variety of questions that can lead us to ROI, the challenge is how to ensure the sources of data respond to the questionnaire. Between the types of questions being asked and the potential length of the questionnaire, given the number of questions asked, success with response is at risk. To ensure credible analysis, a response rate of 60-90 percent is the typical target for most users of the ROI Methodology. So, how do we overcome this challenge?
It is all in the planning.
Planning a strategy for successful response to questionnaire is critical. This requires more than just offering an incentive for response (although, that is one of the strategies). Multiple techniques are used to ensure a high response rate, as well as good responses. Figure 5 lists a variety of techniques used by organizations to generate an appropriate response rate to questionnaires.
Figure 5: Techniques to Increase Questionnaire Response Rates.
| Techniques |
|
Although most items on the list are self-explanatory, the following four are the most powerful techniques:
- Provide advance communication about the data collection. Advanced communication lets the participants involved know from the beginning that they must provide data in a questionnaire as part of the project. This shifts responsibility to the participants not only to achieve success and drive business impact, but report the results of their efforts.
- Review the questionnaire at the end of the formal session. Although data will not be collected for weeks or months after the particular project is implemented, the questionnaire is reviewed explaining question by question to ensure participants understand the questions and the importance of each one.
- Have the introduction letter signed by a top executive. Engaging a top executive in administering the questionnaire often puts a new perspective on the activity of responding.
- Provide the summary of results to the participants and show them what improvements are made. This is a two-part process. Provide a summary of what all the participants have said, often the participants want to see what others have provided. Then, a few weeks later, show what changed as a result of their input. This indicates to participants that their input is driving change.
[inlinead-1]
While these four are the most effective, all of the techniques listed above should be considered. A simple planning process is to work with the evaluation team to decide upfront what you will do before the questionnaire is administered, during the evaluation period and after the evaluation period to ensure a positive response. Figure 6 presents an example of a simple plan to ensure positive, credible responses.
Figure 6: Plan for Ensuring Positive Response Rates.
| Program: Leadership Challenge Duration: Five Days Number of Participants: 150 Response Rate Target: 75 percent Questionnaire to be Administered: Six months after program (give participant three weeks to respond) |
Prior to Administering Questionnaire
|
During Evaluation Period
|
After Evaluation Period
|
A fundamental standard supporting the ROI Methodology is involved here—no data, no improvement. Essentially, for each non-response, it is assumed no value has been gained by investing in that person’s participation in the program. This conservative rule ensures that we do not assume benefits even though there may have some. This is one of several standards that help us remove subjectivity from the less than robust methods of data collection. This is also your incentive to go after the data!
For more detail on how to obtain a high level of response, visit us at www.roiinstitute.net and download the article, "Return to Sender" published in Performance Improvement Journal. Also, for an example of using a questionnaire for an ROI study on a leadership development program, contact us at info@roiinstitute.net and ask for the study "International Car Rental (IRC)."
Our next column will address additional challenges of applying the ROI Methodology.