The Rapid Elearning Blog

It never fails.  Fill a room with instructional designers and you’ll always have someone ask, ‘How do you measure your return on investment."  To which I respond, "What are you? An accountant?"  As the great elearning pioneer, Dr. Werner Oppelbaumer says, "If you want better ROI, hire smarter people who don’t need training." 

Here’s the deal with return on investment (ROI).  It’s tricky.  I find that the concern is really less about ROI and more about how instructional designers can promote their value to the organization.  And considering our economic climate, that’s not a bad idea.

Here are three easy ways to help you determine your ROI and promote the value you bring to your organization.

Customer Satisfaction = Success 

I hired a neighborhood boy to mow my lawn.  I didn’t hear the mower running, so I looked outside.  He was sitting on my porch with his laptop entering data into a spreadsheet.  I asked why he wasn’t mowing my lawn.  He told me that he was trying to figure out the ROI.

"Look! I know the ROI.  I am paying you $20 to mow my lawn.  If you mow the lawn to my satisfaction, you’ll prove your value.  I don’t need you to create a bunch of charts and spreadsheets."

The same goes for elearning. Assume that the customers know what they want.  If you do what you agree to do and the customer’s happy, you have your ROI.

ACTION ITEMS

  • Make a detailed agreement on what you’ll deliver.  A common problem is that we use the same words but we don’t always mean the same thing.  When working with your clients, set clear expectations and measures of success. 
  • Establish a timeline that is realistic.  From my experience, this is an area that trips up a lot of projects.  The timeline is fuzzy and with that so are expectations.  Fuzzy expectations lead to a less than stellar end result, which ultimately leads to an unhappy customer.  Assume that when all is said and done, you’re to blame.  So it’s in your best interest to set a clear timeline and work to meet it or beat it.
  • Do a post project debrief with your client to measure their level of satisfaction.  If you wait until the end of the year, their enthusiasm will wane and they’ll be bombarded by other people looking for end-of-year feedback.  I make it a habit to follow up right after the project is complete.  I want to know what went well and where I can make improvements for the next project.

Align the Course Outcome to a Measurable Goal 

The challenge for a lot of courses is that they aren’t tied to real performance goals.  They might have value to the organization, but it’s not always clear how.  So you end up with a lot of information-dumping and no focused performance improvements.  Your job is to get the course focused on performance.

How do you measure the ROI of a new hire training course?  If it’s an information dump, that’s hard.  However, if you tie the information to a performance goal then you have a metric to hang your hat on.  For example, new employees need more hand holding for initial IT support.  Suppose your course teaches them how to self-serve their IT issues. You can measure a drop in call frequency or IT help requests.  That is a measurable improvement and shows the impact of your course.

Ideally your course is built around a real performance goal.  If you’re not quite sure what that is, help the customer sort through their expectations.  The key question here is, "How will you know you’re successful?"  You’re looking for some metric that they use as proof of success.  Then you want to build your course to change that number.

ACTION ITEMS

  • Find a metric you can use.  There’s always something.  Even if it’s taking a 30 minute course down to 15 minutes. That’s a 50% decrease in time.  Multiply that over 10 people and you’ve saved 150 minutes. :) 
  • Make sure the metric is meaningful.  As you can see by the example above, percentages can be deceiving.  No one cares if you save 150 minutes.  Besides, you have to save a lot more than that if you want your CEO to get his bonus.
  • Learn to be a performance consultant so you’re better able to help your clients focus on the right things.  Robinson’s book, Performance Consulting, is a good place to start.

Lower Your Production Costs 

Sometimes you have no access to the metrics that measure success.  And even if you did, trying to prove your ROI becomes too costly.  Instead of focusing on what you can’t control, focus on what you can, such as your production time and the resources needed to create a course.

I’ll benchmark my courses.  For example, a simple elearning course can cost anywhere from $5,000 to $15,000 depending on what you do.  I just split the difference and set my baseline cost at $8,000.  My goal is to bring my simple courses in under that cost.   It’s not a completely accurate number but it is a good place to start.  Your ultimate goal is to cost less than your competition.  In these tough economic times, you want to demonstrate that you are a cost-effective solution to meet the organization’s needs.

ACTION ITEMS

  • Find out what your courses cost.  You don’t need to be an accountant.  Ball park the figures.  If someone challenges your numbers, let them do the math for you.  The main goal is to know what you cost the organization.  Without that number you won’t be able to prove your value.
  • Benchmark your courses.  Low-end courses are usually less than $10,000.  Many of the high-end courses can costs hundreds of thousands.  It makes sense to figure out where you fit in comparison.  In tough times, training is usually an early casualty.  You want to demonstrate that you are a cost-effective solution.
  • Cut production costs.  Make one of your performance goals to cut your production costs by 20%.  You’ll save money and probably find ways to become more efficient as a team.  In an earlier post, I outlined my production strategy.  I start with rapid elearning software and move up from there.  It saves time and money and frees up my more expensive multimedia programmers to do what the software can’t. 

When it comes to return on investment, you have two goals.  You want to bring value to the organization and you want show that YOU bring value to the organization.  You can do this by meeting your customer’s expectations, aligning your projects to real measurable objectives, and controlling your production costs.

I look forward to your feedback and comments.  Feel free to share them by clicking on the comments sections.


 

Upcoming E-Learning Events

  • August 3 & 4 (Seattle, WA). Connect with your peers in the Pacific Northwest and learn all sorts of tips & tricks in this fun community-based workshop. SOLD OUT.
  • September 20-21 (Dallas, TX). Connect with your peers in Texas and learn all sorts of tips & tricks in this fun community-based workshop. Register here.
  • October 12-13 (Vancouver, BC). Connect with your peers in British Columbia and learn all sorts of tips & tricks in this fun community-based workshop. Register here.
  • November 1 & 2 (London). Articulate Roadshow. Details coming soon.
  • November 4 (Dublin). Articulate E-Learning Workshop. Details coming soon.

 

Free E-Learning Resources

Want to learn more? Check out these articles and free resources in the community.

Here’s a great job board for elearning, instructional design, and training jobs

Participate in the weekly elearning challenges to sharpen your skills

Get your free PowerPoint templates and free graphics & stock images.

Lots of cool elearning examples to check out and find inspiration.

Getting Started? This elearning 101 series and the free e-books will help.



23 responses to “Here's How to Measure ROI in the Real World”

There are times when I feel as if I’m all alone in the world when it comes to the viability of trying to demonstrate quantitative ROI on learning. For me, the bottom line is always – either you believe it’s worth it or you don’t.

For example, My husband and I used to take it in turns to read to our children every night when they were little. Recently Donald Clark published a post that produced evidence that there were no educational benefits to this. Ah. No return on investment, then. Really? What about the fact that we have a wonderfully close relationship with our boys? That they come and ask us about any- and everything under the sun? What about their love for a wide range of subjects? What about their fond memories of their childhood? What about my own misty-eyed memories of a clean-smelling boy falling asleep curled up in my arms? Or of finding my son awake and ‘reading’ the pictures of the book my husband has fallen asleep over?

There is absolutely no monetary value to be demonstrated, nor can I demonstrate in any way that my kids do any better academically than they would have done if we had not had a nightly story. But I defy anyone to tell me that it bore no returns!

I believe that much about our relationship with my boys, much about their own characters can be imputed to those final moments of sleepy intimacy at the end of each and every day of their early lives. I believe it was worth it. I believe it so firmly that I would do it again, even though I can’t prove the academic benefits.

Yes, I’ve gone all subjective here, but I hope that I have made my point that many returns on investment cannot be calculated with a spreadsheet.

Employee confidence and morale cannot be quantified, but I contend that they are invaluable in the long term.

Wow! Karyn’s comment says it all.

October 21st, 2008

I changed ROI for ROE a long time ago. ROE = Return on Expectations. Following Tom’s Customer Satisfaction = Success makes it easy.

II love Karen’s answer, but unfortunately people that approve projects and positions often don’t know if training is ‘worth it’ or not. They want numbers and spreadsheets, because that’s what the accountants need to justify the cost of the program to the stockholders.

So here are some easy ways to produce a stockholder-worthy ROI when you can’t put a dollar value on it (I’ve never yet worked for a company that could tell me what a task ‘costs’ before training – they all had no idea or only a vague concept about how much time a task takes to accomplish).

Easy ROI 1 – count questions asked to the help desk/SMEs on the content matter, and then calculate the percentage of those askers that successfully completed the course.

Easy ROI 2 – compare the time needed to take the course to the predicted time saved by the skill and present it as a percentage (qualify this ‘when the recommended process is used …).

Easy ROI 3 – compare the time needed to develop the course to the time spent presenting the course (good for content developers/instructional designers; also to justify e-learning production).

Hope that helps make your accountants happy – it’s served me well in contracting jobs.

Tom

Where did you ever find Dr. Werner? Absolutely love him (tips presentation). I found myself really engaged in the message and in his simplistic approach to the burning question “How to measure…”. Also liked the way you reused a previous posting “production strategy…” to bring the thought home. Thanks.

I think everyone’s comments are excellent. One thing I have not noticed in the ROI discussion is the advantage that online training can bring in recruiting. I work in the Health Care industry in Canada and the market place for qualified RN & RPN’s is very tight. With the new nursing population graduating they are looking for the best work environment and the company with the most “perks”. Us offering training or online learning has been a great advantage. Employees want skills and want education that applies to them, especially in this economic time. While this is hard to measure, it certainly is an important factor for us. I guess it comes back to Tom’s point about identifying expectations and addressing the ones you can.

Great posting,

Linda

Right on Karyn! I love your example.

However, I believe that the most important part of this article is “Align the Course Outcome to a Measurable Goal”. I see too many e-learning courses with vague objectives using words like “understand” or “review”. How can you possibly determine if learning has occured with objectives like that? Without clear, measurable objectives, there is no hope for calculating any type of ROI and there is no point in designing the course in the first place.

Nice information! I just found myself getting interested in this topic over the past couple weeks so I’m happy to read it here.

ROI is a discounted cash flow representing either a cost savings or revenue gain often associated with Hurdle Rates. Hurdle rates are like the return on a CD. Did you or did you not make >15% on what you invested in? Many excel spreadsheet examples abound.

Karyn’s example is interesting and touching. But, one, a family isn’t a business; the family doesn’t need to optimize ROI. The family is thankfully free to pursue many different undefined agendas. Two, she’s sort of made the indirect case. The investment in intangibles produce value, only it’s hard to quantify the value.

But more importantly, the philosophy reflected is a retreating strategy. HR/training/learning types need to earn a seat at the table by understanding the language of business and connecting their contribution to the company’s overall goals. The retreat into “this is subjective” or even “this is intangle, indirect, and long-term in payofff” will always be doomed because it can justify any expenditure.

Businesses exist to produce some sort of ROI. Period. (metaphorically, i mean, ROI is imprecise and has no precise financial definition. In many organizations it would be a confusing goal owing.). Only efforts that support the goal can be justified. Sometimes the issue is short-term/long-term; i.e., we make hard investments today in return for intangible and long-term returns. Okay, that makes it more difficult, but it’s no excuse for not trying to tie an investment to something that is eventually concrete. Otherwise, the business is actually forsaking its fiduciary to make such investments. ROI ultimately means, it was worth doing; lack of it ultimately means, not worth doing or we don’t sufficiently understand our investment.

David

October 21st, 2008

Great article and follow up comments.
Having worked both sides of the table I can see how it is sometimes difficult to promote the value of a learning intervention. Jack is correct about using ROE rather than ROI when it comes to meeting expectations. However, sitting down with your client (internal/external) and getting a good brief will ultimately guide you in whether they are looking for a financial (direct/indirect) return on the project. I thinks will see less of the “nice” to have and more of the “need” to have learning programs in the future as every organization watches it’s dollar. I found asking the question “why” is a real good start as I often like to start with the end in mind.

[…] » Here’s How to Measure ROI in the Real World The Rapid eLearning Blog (tags: roi training performance consulting) […]

Karyn nailed the idea of an intangible value. David also points out that in business unfortunately we don’t get to the “table” without some kind of measurable value.

In Tom’s first topic – Customer Satisfaction = Success you cannot go wrong.

An observation: We focus so much on results that often times the quality of training is second to the compliance of who completed it. We know results are driven by performance; performance is driven by behaviors; behaviors are driven by thoughts. Yet, the attention on ‘results’ gets more traction.

I looked at this some time ago and tested this theory by flipping it the other way around – Think, Behave, Perform, Result. Given the learner the tools and resources to ‘think’ about their task and how to do it better, then suggest a change in those thoughts to change their ‘behavior.’ The goal is they would walk away from the training and ‘perform’ beter. Thus, the ‘results’ will take care of itself.

Back to ROI and the Customer. Various surveys sent to customers that focused on particular outcomes on how the learner was to ‘behave’ and ‘perform’ proved successful. Happy customers spend more money!

October 23rd, 2008

Measuring the business effect of training is a job security task in this economic climate. If you can’t tell how training is affecting business outcomes, then you need to figure out how to measure it. Are sales up? Support calls down? Support calls shorter? Is it cheaper to do something?
ROI is actually an abbreviation with multiple meanings. For a front line manager, the objective of the training (reducing call times, for example) is what ROI means. For a manager of managers, dollars saved or headcount reduced become more important. The key thing is to answer the question with the asker in mind.
Because every training course must have a business need, the results can be measured by studying how well the course met the business need. If there is no business need for a course, then there is no reason to do the course.
Can we say in absolute certainty that training caused the ROI? Of course not. Too many things affect employee performance simultaneously, including factors over which the business has no control. We can, however, infer the effect of training. If sales go up 50% immediately after a training course, the likelihood is that some, if not all, the increase is due to the course. Businesses do not operate on certainty. They operate on probability without having all the facts to make a completely reasoned decision. As people engaged in the business of training, we need to enter the reality of our customers.

[…] to show that your elearning adds value? Tom Kuhlmann recently suggested these […]

When it comes to ROI, I have found my saving grace is the more frequent use/availability of contact management systems, and referral/sales tracking systems. I create, and evaluate, sales and product knowledge e-learning programs. Using the staff who did not take the course (there are always those who “don’t get around to taking the e-learning”) as a control group, many correlations can be made when comparing the two groups. The fore-mentioned systems provides a lot of data to compare and use of a control group eliminates many of the extraneous factors, such as seasonality, economics, marketing campaigns, etc.

I feel it is important to note that prior to attempting ROI, one MUST first progress through all 4 levels of evaluation (reaction, learning, behavior, results). For example, you will save yourself a lot of aggravation if you identify if they are even learning anything prior to seeing if it is affecting behavior.

I think there are some interesting insights in here. I appreciate what’s been added. I typically lean more on the practical business side of things. However Karyn’s post reminds me of a book I once read.

The essence of the book was that sometimes you do stuff not so much because it’s practical or measurable, but you do it for the perception of value. I had a friend who worked for a company that I thought was quite progressive in how they trained. They set him up with all of the resources. Put him in a lab with experienced SME. Gave him a mentor. Etc. It was a really well designed training process and he was up to speed very quickly.

However, he complained that they never “trained” him. I asked what he expected and he basically wanted to go into a room and have a more formal training program with lectures and elearning courses.

[…] should be when you put it online. Tom Kuhlmann, an online training expert talks about this in his blog on measuring the ROI for online learning. He explains that you need to align the course or presentation outcome to measurable goals. I […]

[…] consuming whether it’s online or not.  With elearning, each time the course is accessed your return on investment improves because you are dividing the fixed production costs by number of uses.  You also have […]

[…] courses, there are different ways to measure success. In a previous post, we looked at measuring return on investment.  Today we’ll review a few reasons why some are building courses and look at how they can be […]

[…] ci sono diversi modi per misurarne il successo. In un precedente post abbiamo visto come misurare il “Return on Investment” (letteralmente ritorno sull’investimento, comunemente conosciuto come ROI, ndr). Oggi vedremo […]

[…] to deal with, one of my readers has been asked about it for his new project. I have recommended an old favourite source in the Rapid E-Learning Blog as well as the the JISC project report […]

[…] and realistic e-learning goals. That will also help you keep spending in check and calculate the return on your e-learning investment. Training programs are expensive, so you need to be able to show how well your new e-learning […]