Here's How to Measure ROI in the Real World

There are times when I feel as if I’m all alone in the world when it comes to the viability of trying to demonstrate quantitative ROI on learning. For me, the bottom line is always – either you believe it’s worth it or you don’t.

For example, My husband and I used to take it in turns to read to our children every night when they were little. Recently Donald Clark published a post that produced evidence that there were no educational benefits to this. Ah. No return on investment, then. Really? What about the fact that we have a wonderfully close relationship with our boys? That they come and ask us about any- and everything under the sun? What about their love for a wide range of subjects? What about their fond memories of their childhood? What about my own misty-eyed memories of a clean-smelling boy falling asleep curled up in my arms? Or of finding my son awake and ‘reading’ the pictures of the book my husband has fallen asleep over?

There is absolutely no monetary value to be demonstrated, nor can I demonstrate in any way that my kids do any better academically than they would have done if we had not had a nightly story. But I defy anyone to tell me that it bore no returns!

I believe that much about our relationship with my boys, much about their own characters can be imputed to those final moments of sleepy intimacy at the end of each and every day of their early lives. I believe it was worth it. I believe it so firmly that I would do it again, even though I can’t prove the academic benefits.

Yes, I’ve gone all subjective here, but I hope that I have made my point that many returns on investment cannot be calculated with a spreadsheet.

Employee confidence and morale cannot be quantified, but I contend that they are invaluable in the long term.

Wow! Karyn’s comment says it all.

I changed ROI for ROE a long time ago. ROE = Return on Expectations. Following Tom’s Customer Satisfaction = Success makes it easy.

II love Karen’s answer, but unfortunately people that approve projects and positions often don’t know if training is ‘worth it’ or not. They want numbers and spreadsheets, because that’s what the accountants need to justify the cost of the program to the stockholders.

So here are some easy ways to produce a stockholder-worthy ROI when you can’t put a dollar value on it (I’ve never yet worked for a company that could tell me what a task ‘costs’ before training – they all had no idea or only a vague concept about how much time a task takes to accomplish).

Easy ROI 1 – count questions asked to the help desk/SMEs on the content matter, and then calculate the percentage of those askers that successfully completed the course.

Easy ROI 2 – compare the time needed to take the course to the predicted time saved by the skill and present it as a percentage (qualify this ‘when the recommended process is used …).

Easy ROI 3 – compare the time needed to develop the course to the time spent presenting the course (good for content developers/instructional designers; also to justify e-learning production).

Hope that helps make your accountants happy – it’s served me well in contracting jobs.

Tom

Where did you ever find Dr. Werner? Absolutely love him (tips presentation). I found myself really engaged in the message and in his simplistic approach to the burning question “How to measure…”. Also liked the way you reused a previous posting “production strategy…” to bring the thought home. Thanks.

I think everyone’s comments are excellent. One thing I have not noticed in the ROI discussion is the advantage that online training can bring in recruiting. I work in the Health Care industry in Canada and the market place for qualified RN & RPN’s is very tight. With the new nursing population graduating they are looking for the best work environment and the company with the most “perks”. Us offering training or online learning has been a great advantage. Employees want skills and want education that applies to them, especially in this economic time. While this is hard to measure, it certainly is an important factor for us. I guess it comes back to Tom’s point about identifying expectations and addressing the ones you can.

Great posting,

Linda

Right on Karyn! I love your example.

However, I believe that the most important part of this article is “Align the Course Outcome to a Measurable Goal”. I see too many e-learning courses with vague objectives using words like “understand” or “review”. How can you possibly determine if learning has occured with objectives like that? Without clear, measurable objectives, there is no hope for calculating any type of ROI and there is no point in designing the course in the first place.

Nice information! I just found myself getting interested in this topic over the past couple weeks so I’m happy to read it here.

ROI is a discounted cash flow representing either a cost savings or revenue gain often associated with Hurdle Rates. Hurdle rates are like the return on a CD. Did you or did you not make >15% on what you invested in? Many excel spreadsheet examples abound.

Karyn’s example is interesting and touching. But, one, a family isn’t a business; the family doesn’t need to optimize ROI. The family is thankfully free to pursue many different undefined agendas. Two, she’s sort of made the indirect case. The investment in intangibles produce value, only it’s hard to quantify the value.

But more importantly, the philosophy reflected is a retreating strategy. HR/training/learning types need to earn a seat at the table by understanding the language of business and connecting their contribution to the company’s overall goals. The retreat into “this is subjective” or even “this is intangle, indirect, and long-term in payofff” will always be doomed because it can justify any expenditure.

Businesses exist to produce some sort of ROI. Period. (metaphorically, i mean, ROI is imprecise and has no precise financial definition. In many organizations it would be a confusing goal owing.). Only efforts that support the goal can be justified. Sometimes the issue is short-term/long-term; i.e., we make hard investments today in return for intangible and long-term returns. Okay, that makes it more difficult, but it’s no excuse for not trying to tie an investment to something that is eventually concrete. Otherwise, the business is actually forsaking its fiduciary to make such investments. ROI ultimately means, it was worth doing; lack of it ultimately means, not worth doing or we don’t sufficiently understand our investment.

David

Great article and follow up comments.
Having worked both sides of the table I can see how it is sometimes difficult to promote the value of a learning intervention. Jack is correct about using ROE rather than ROI when it comes to meeting expectations. However, sitting down with your client (internal/external) and getting a good brief will ultimately guide you in whether they are looking for a financial (direct/indirect) return on the project. I thinks will see less of the “nice” to have and more of the “need” to have learning programs in the future as every organization watches it’s dollar. I found asking the question “why” is a real good start as I often like to start with the end in mind.

[…] » Here’s How to Measure ROI in the Real World The Rapid eLearning Blog (tags: roi training performance consulting) […]

Karyn nailed the idea of an intangible value. David also points out that in business unfortunately we don’t get to the “table” without some kind of measurable value.

In Tom’s first topic – Customer Satisfaction = Success you cannot go wrong.

An observation: We focus so much on results that often times the quality of training is second to the compliance of who completed it. We know results are driven by performance; performance is driven by behaviors; behaviors are driven by thoughts. Yet, the attention on ‘results’ gets more traction.

I looked at this some time ago and tested this theory by flipping it the other way around – Think, Behave, Perform, Result. Given the learner the tools and resources to ‘think’ about their task and how to do it better, then suggest a change in those thoughts to change their ‘behavior.’ The goal is they would walk away from the training and ‘perform’ beter. Thus, the ‘results’ will take care of itself.

Back to ROI and the Customer. Various surveys sent to customers that focused on particular outcomes on how the learner was to ‘behave’ and ‘perform’ proved successful. Happy customers spend more money!

Measuring the business effect of training is a job security task in this economic climate. If you can’t tell how training is affecting business outcomes, then you need to figure out how to measure it. Are sales up? Support calls down? Support calls shorter? Is it cheaper to do something?
ROI is actually an abbreviation with multiple meanings. For a front line manager, the objective of the training (reducing call times, for example) is what ROI means. For a manager of managers, dollars saved or headcount reduced become more important. The key thing is to answer the question with the asker in mind.
Because every training course must have a business need, the results can be measured by studying how well the course met the business need. If there is no business need for a course, then there is no reason to do the course.
Can we say in absolute certainty that training caused the ROI? Of course not. Too many things affect employee performance simultaneously, including factors over which the business has no control. We can, however, infer the effect of training. If sales go up 50% immediately after a training course, the likelihood is that some, if not all, the increase is due to the course. Businesses do not operate on certainty. They operate on probability without having all the facts to make a completely reasoned decision. As people engaged in the business of training, we need to enter the reality of our customers.

[…] to show that your elearning adds value? Tom Kuhlmann recently suggested these […]

When it comes to ROI, I have found my saving grace is the more frequent use/availability of contact management systems, and referral/sales tracking systems. I create, and evaluate, sales and product knowledge e-learning programs. Using the staff who did not take the course (there are always those who “don’t get around to taking the e-learning”) as a control group, many correlations can be made when comparing the two groups. The fore-mentioned systems provides a lot of data to compare and use of a control group eliminates many of the extraneous factors, such as seasonality, economics, marketing campaigns, etc.

I feel it is important to note that prior to attempting ROI, one MUST first progress through all 4 levels of evaluation (reaction, learning, behavior, results). For example, you will save yourself a lot of aggravation if you identify if they are even learning anything prior to seeing if it is affecting behavior.

[…] should be when you put it online. Tom Kuhlmann, an online training expert talks about this in his blog on measuring the ROI for online learning. He explains that you need to align the course or presentation outcome to measurable goals. I […]

[…] consuming whether it’s online or not.  With elearning, each time the course is accessed your return on investment improves because you are dividing the fixed production costs by number of uses.  You also have […]

[…] courses, there are different ways to measure success. In a previous post, we looked at measuring return on investment.  Today we’ll review a few reasons why some are building courses and look at how they can be […]

[…] ci sono diversi modi per misurarne il successo. In un precedente post abbiamo visto come misurare il “Return on Investment” (letteralmente ritorno sull’investimento, comunemente conosciuto come ROI, ndr). Oggi vedremo […]

[…] to deal with, one of my readers has been asked about it for his new project. I have recommended an old favourite source in the Rapid E-Learning Blog as well as the the JISC project report […]

[…] and realistic e-learning goals. That will also help you keep spending in check and calculate the return on your e-learning investment. Training programs are expensive, so you need to be able to show how well your new e-learning […]