kirkpatrick - Blogs - DPG Community2024-03-29T00:17:25Zhttps://community.dpgplc.co.uk/blog/feed/tag/kirkpatrickThe role of data and analytics in evaluationhttps://community.dpgplc.co.uk/blog/the-role-of-data-and-analytics-in-evaluation2019-06-20T10:44:39.000Z2019-06-20T10:44:39.000ZAdy Howeshttps://community.dpgplc.co.uk/members/AdyHowes<div><img src="https://storage.ning.com/topology/rest/1.0/file/get/3036011792?profile=RESIZE_400x&width=400"></div><div><p><strong>What is the sort of data we should be gathering and what questions should we ask?</strong></p>
<p>Last week, I saw a session at <em>The Festival of Work</em> about <strong>Learning Data and Analytics</strong> at capacity.</p>
<p>No more space, not one seat, packed, full, people turned away (I myself had to watch from the side lines from the standing room only section).</p>
<p>Data and analytics, it seems is a hot topic right now and as ever you can find this topic explored already in some detail here on the DPG Community. There are articles <a href="https://community.dpgplc.co.uk/blog/i-m-not-a-data-scientist-but">here</a> and <a href="https://community.dpgplc.co.uk/blog/are-l-d-slow-to-change">here</a> that are well worth reading that link to data in L&D.</p>
<p>So why now?</p>
<p>I think the focus on data has always been there but one of the issues is accessibility and the type of data & information we as an industry have focussed on. I also think data is being looked upon to understand relationships and the way in which we draw conclusions and make links to how and why things are as they are.</p>
<p>There is a seismic shift across all industries to use the abundance of data & insight now available. Using this data to help and better understand customer behaviour, customer lifecycles, problems we’re looking to solve, product / service satisfaction, more effective product development, credibility, buying habits and trends and positive and developmental feedback (reviews). We live in a digital world and this has changed our relationship with data, but the principle of data analysis and product development can apply to all aspects of L&D. For me data helps us ask fundamentally better questions or helps us know the right questions to ask.</p>
<p>If L&D were a start-up business, developing a suite of products, this is also exactly the type of data we should be looking to gather or questions to ask in terms of:</p>
<ul><li>Clearly identifying the problem / challenge / pain faced by potential customer</li>
<li>Credibility – consultative conversations with customer to ensure deep understanding and right approach is used with valid recommendations provided that meet that need.</li>
<li>Customer lifecycles – where and at what point is support / training / solutions required to address any problems</li>
<li>Customer behaviour – how do customers interact with a product to address problem, where do they access it or buy it and crucially how does the product change / modify behaviour in a positive way and ‘solve’ problem.</li>
<li>Product satisfaction (feedback on learning solutions / performance support tools)</li>
<li>Product development / continuous improvement based on feedback</li>
</ul><p>Maybe one of the previous challenges for L&D is the ‘product’ has been predominantly a face to face classroom session or a face to face workshop and we just haven’t thought about what we do and how we do it in these terms before. The questions and approach above can be applied to any L&D product but I just don’t think we think as commercially as we should when it comes to demonstrating value and measuring success. As a cost centre L&D need to be able to demonstrate the value they provide, and more is being demanded of us than ever before.</p>
<p>To respond to this demand, more and more L&D teams are working using agile principles and methodology. Rapidly prototyping and developing minimal viable products quickly and far more collaboratively with end users to ensure whatever product is developed meets the need of the end user. Pilots, experiments and beta products are all things you will find in other industries, but we don’t tend to embrace these approaches as much as we perhaps should. One of the things I’ve been thinking about is how we link data & insight to asking and answering the questions above to traditional L&D evaluation.</p>
<p>Evaluation in L&D needs to evolve. I’m not suggesting that the likes of the <a href="https://www.dpgplc.co.uk/courses-qualifications/learning-and-development/kirkpatrick-four-levels-evaluation-certification-programme/">Kirkpatrick Evaluation Model</a> (Link) are obsolete. As a <a href="https://www.dpgplc.co.uk/courses-qualifications/learning-and-development/kirkpatrick-four-levels-evaluation-certification-programme/">Kirkpatrick</a> Partner we are big believers in the KP model. The framework that the KP model sets out helps still provides a structure to follow that can be adapted to each of the elements listed above. It’s a move from an old training mindset to that of a data driven and evidence-based department who focus less on individual training events to that of overall engagement and customer or employee lifecycles and product / performance improvement.</p>
<p>Rather than taking the 4 levels as a linear model and moving through them 1 by 1, analysing the key elements of each and applying them in a way that meets the need of the customer. For example:</p>
<p><strong>Results (level 4)</strong></p>
<p>Start with the end in mind, analyse challenges and pain points – what does success look like and how will it be measured. Consultative approach working collaboratively with customer.</p>
<p><strong>Learning (level 2)</strong></p>
<p>Make a recommendation and begin prototyping with the customer to build an MVP. Get it in front of people to test and use. Pilot, pilot, pilot.</p>
<p><strong>Reaction (Level 1)</strong></p>
<p>Get effective user feedback, is it effective, easy to use, easy to understand, what works what doesn’t, how can it be improved, collaborate and demand feedback.</p>
<p><strong>Behaviour/Transfer (Level 3)</strong></p>
<p>Does it have the desired impact, does it change behaviour, provide new skills, can this be observed, how does this link to the desired success and measures identified.</p>
<p>If yes – release and get back to & track ‘Results’</p>
<p>If no – back to Learning and better product development</p>
<p>Repeat.</p>
<p>OK, so I appreciate this is a crude example but the point I’m making is that we have always looked at evaluation as something that we do to other people or something we apply to the end of the training process as opposed to something that we apply to our process and continually analyse the review the feedback and data we have.</p>
<p>To do this in a much more agile way and to work much closer with our customers to provide better products and service. Working with pilot groups and working in sprints can speed up the development process and make things so much quicker. Today, we don’t need to wait 3-6 months to analyse a change in behaviour and we also don’t have to rely on outdated modes of gathering feedback. Apps, pulse surveys and polls can get us closer to the data we need much quicker if we’re prepared to experiment and explore.</p>
<p>In this age of data, we need to look at our own processes and approaches to stay relevant and keep up with the demands of our customers. Not a complete overhaul of everything we do but looking at what we do and how we do it through other lenses and borrowing ideas and concepts from other industries to make us more resilient and responsive.</p>
<p>Perhaps learning analytics while the latest buzz word is just another way for L&D to evolve its’ practices and that’s why there is such a focus on it. For me though it isn’t about disregarding everything that we’ve done before but using that we can see and experience to make better and more informed decisions. Yes, there is an element of shiny new technology, analytics dashboards and ‘evidence based’ approaches but this needs to blend together with ‘evaluation’ of sorts and we can learn a lot from new forms of data but also our existing data and use it in a more considered and effective way.</p>
<p><em>What do you think?</em></p>
<p><em>How are you using data & evidence-based practice to improve your product development?</em></p>
<p><em>Is the way you do evaluation changing?</em></p>
<p>If you are interested in finding more about how to use a data and evidence based practice to your Learning and Development activities, check out our <a href="https://www.dpgplc.co.uk/courses-qualifications/learning-and-development/kirkpatrick-four-levels-evaluation-certification-programme/">Kirkpatrick Four Levels Certificate</a></p>
<p> </p>
<p> </p></div>How to evaluate learninghttps://community.dpgplc.co.uk/blog/how-to-evaluate-learning2018-11-27T12:22:53.000Z2018-11-27T12:22:53.000ZAdy Howeshttps://community.dpgplc.co.uk/members/AdyHowes<div><img src="https://storage.ning.com/topology/rest/1.0/file/get/139392503?profile=RESIZE_400x&width=400"></div><div><p>'Around the world, training and development is in a state of crisis'. That's <a href="https://www.td.org/newsletters/atd-links/why-evaluate-learning" target="_blank" rel="noopener">according to the Association for Talent and Development</a>. They go on to say, 'Training professionals who only think abot training events and not what happens before and particularly after them will become extinct'.</p>
<p>It's a scary picture when you look at it like that, but you can't deny what's happening here. As the article rightly points out, training budgets are among the first to be cut when economic times get tough. Is L&D demonstrating enough of a positive impact on the performance of the orgnanisation?</p>
<p>If there's ever been a time to demonstrate the return that learning iniatives bring now has to be it. Part of the problem is that many L&D functions don't consider the desired results of learning interventions from the outset and there are often now well alligned to the desired performance requiremets of the organisation. Failing to create stong links between learning and performance could bring about situations where L&D is placed in the nice-to-have category.</p>
<p>Here at DPG, we're the UK's sole partner for the most recognised model of evaluation, Kirkpatrick. You can <a href="https://www.dpgplc.co.uk/courses-qualifications/learning-and-development/kirkpatrick-four-levels-evaluation-certification-programme/" target="_blank" rel="noopener">find out more details about that on our website</a>.</p>
<p>But is the Kirkpatrick model of evalution still relevant and how can it work in today's modern organisations?</p>
<p>We got chance to catch up with Phil Willcox, Kirkpatrick practitioner and facilitator of the <a href="https://www.dpgplc.co.uk/courses-qualifications/learning-and-development/kirkpatrick-four-levels-evaluation-certification-programme/" target="_blank" rel="noopener">DPG Kirkpatrick Programmes</a>. Check out this recording, under ten minutes long, of why he thinks it is....</p>
<p> </p>
<p><iframe src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/536192190&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_teaser=true&visual=true" width="100%" height="300" frameborder="no" scrolling="no"></iframe></p>
</div>#TrainingCrimes: Episode 3 - Evaluationhttps://community.dpgplc.co.uk/blog/trainingcrimes-episode-3-evaluation2018-07-11T14:10:00.000Z2018-07-11T14:10:00.000ZAdy Howeshttps://community.dpgplc.co.uk/members/AdyHowes<div><img src="https://storage.ning.com/topology/rest/1.0/file/get/42332915?profile=RESIZE_400x&width=400"></div><div><p></p>
<div class="video-container"><iframe width="1280" height="720" src="https://www.youtube.com/embed/DPMlB68NzUE?rel=0" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen="allowfullscreen"></iframe></div>
<p></p>
<p> </p>
<p><strong>Training Crimes: Evaluation</strong></p>
<p>In the <a href="https://community.dpgplc.co.uk/blog/trainingcrimes-episode-1-training-needs-analysis" target="_blank" rel="noopener">first #TrainingCrimes episode</a> we created in association with <a href="https://www.myfirsttrainers.com/">My First Trainers</a> we looked at Training Needs Analysis while the <a href="https://community.dpgplc.co.uk/blog/trainingcrimes-episode-2-design-and-delivery" target="_blank" rel="noopener">second episode focussed on design and delivery</a>.</p>
<p>In this concluding episode, we consider the topic of evaluation. DPG are the UK partner for the <a href="https://www.dpgplc.co.uk/courses-qualifications/learning-and-development/kirkpatrick-four-levels-evaluation-certification-programme/" target="_blank" rel="noopener">Kirkpatrick Programme </a>so this is a topic which is close to our hearts.</p>
<p> </p>
<p><strong>No Alignment to the Business</strong></p>
<p>This is one sure-fire way of making sure a training initiative falls flat on its face. You’d assume that aligning a solution to the business should be done at the beginning of the process and you’d be right in making that assumption.  Starting with the end in mind from the offset is a good thing to do. Consider what the initiative is setting out to achieve and what measures could be used. It’s those that will help you demonstrate the value of the work that’s been done and the progress that’s been made to stakeholders.</p>
<p> </p>
<p><strong>No measurement of the impact of training</strong></p>
<p>Time and money is precious. How do you know the investment has been worthwhile? How do you know that it’s been a success? Our <a href="https://www.dpgplc.co.uk/courses-qualifications/learning-and-development/kirkpatrick-four-levels-evaluation-certification-programme/" target="_blank" rel="noopener">Kirkpatrick Programme</a> here at DPG is designed to help people measure the true impacts of the work done by L&D on many levels. You can find out more about that here.</p>
<p> </p>
<p><strong>Bribing the delegates</strong></p>
<p>Feedback is crucial. If you truly care about making approaches to learning the best they can be, honest feedback is essential. It’s not about getting the best scores on the happy sheet! Take time to gather and assess the feedback you get from those making use of the learning on offer. These are your customers who will give you clues on how to make what you do even better for the future.</p>
<p> </p>
<p>To get better at evaluating learning initiatives, you can find out more about our <a href="https://www.dpgplc.co.uk/courses-qualifications/learning-and-development/kirkpatrick-four-levels-evaluation-certification-programme/" target="_blank" rel="noopener">Kirkpatrick programme here</a>.</p>
<p>What do you think?  Have you come across these crimes before? What other crimes happen when it comes to evaluation can you think of? How can we avoid them?</p>
<p>I’m interested to hear in the comments below….</p>
</div>The most likely used model to demonstrate the value of learning is...https://community.dpgplc.co.uk/blog/kirkpatrick2018-07-06T08:32:00.000Z2018-07-06T08:32:00.000ZAdy Howeshttps://community.dpgplc.co.uk/members/AdyHowes<div><img src="https://storage.ning.com/topology/rest/1.0/file/get/49941880?profile=RESIZE_400x&width=400"></div><div><p>With many different types of evaluation models available to use, its sometimes quite hard to decipher which one is best for organisations or project’s. As L&D professionals, the importance of demonstrating the business value of learning & development initiatives must be known. According to the CIPD, “By evaluating learning and development programmes, employers are better able to ensure that these initiatives are aligned with their business objectives and overall learning and development strategy.”</p>
<p>A survey of our DPG Community members revealed that <a href="https://www.dpgplc.co.uk/courses-qualifications/learning-and-development/kirkpatrick-four-levels-evaluation-certification-programme/" target="_blank" rel="noopener">Kirkpatrick</a>is the most commonly used method.</p>
<p>It’s over 50 years since Donald Kirkpatrick published his training evaluation model. It quickly became adopted within L&D and became the most widely used evaluation model in modern times. The <a href="https://www.dpgplc.co.uk/courses-qualifications/learning-and-development/kirkpatrick-four-levels-evaluation-certification-programme/" target="_blank" rel="noopener">Kirkpatrick Evaluation Model</a> has recently been adapted to The New World Kirkpatrick Model, to meet the needs of 21st century Learning and Development teams. The New World Kirkpatrick Model encourages learning professionals to measure and track success on four levels. Rather than start with the learner and ‘happy sheets’ the model flips itself to ensure that the business objectives and measurements are agreed with key stakeholders and leading indicators are identified to gauge success.</p>
<p>According to Lesley Blower, Learning Resource Manager at utility giant Npower who completed a <a href="https://www.dpgplc.co.uk/courses-qualifications/learning-and-development/kirkpatrick-four-levels-evaluation-certification-programme/" target="_blank" rel="noopener">Kirkpatrick programme</a> with us, <strong>"….the light bulb moment was the realisation that you start at level 4, identify your desired results and leading indicators to measure if you’re on track, and design your training to reflect that outcome."</strong></p>
<p>The levels in the model are:</p>
<ul>
<li><strong>Level 4 - Results</strong> – The degree to which targeted outcomes occur as a result of the training and the support and accountability package</li>
<li><strong>Level 3 - Behaviour</strong> – The degree to which participants apply what they learned during training when they are back on the job</li>
<li><strong>Level 2 - Learning</strong> – The degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment based on their participation in the training</li>
<li><strong>Level 1 - Reactio</strong>n – The degree to which participants find the training favourable, engaging and relevant to their jobs</li>
</ul>
<p>Find out more about joining thousands of other L&D professionals who have gained the internationally recognised Kirkpatrick certification by clicking here.</p>
<p>How do you evaluate the learning initiatives for your organisation? How do you demonstrate value?</p>
</div>Develop your evaluation skills!https://community.dpgplc.co.uk/blog/develop-your-evaluation-skills2017-06-26T16:08:34.000Z2017-06-26T16:08:34.000ZDebora Figueiredohttps://community.dpgplc.co.uk/members/DeboraFigueiredo<div><img src="https://storage.ning.com/topology/rest/1.0/file/get/2217331?profile=RESIZE_400x&width=400"></div><div><p>Hi everyone,</p>
<p>Here's a great opportunity for you to network with other L&D professionals and line managers and gain the internationally-recognised Kirkpatrick Evaluation Certification.</p>
<p>In this 2-day open programme, you will:</p>
<ul>
<li>learn how to evaluate the effectiveness of training in your business </li>
<li>apply your knowledge back in your organisation to demonstrate the effectiveness of your learning interventions</li>
<li>understand how to effectively bridge the gap between learning and organisational development</li>
</ul>
<p>The Kirkpatrick Model teaches you the four levels of training evaluation starting with the desired outcome: <strong>Results</strong>, <strong>Behaviour</strong>, <strong>Learning</strong> and <strong>Reaction</strong>.</p>
<p>Hope to see you in London on the 11th & 12th September.</p>
</div>Work-based performance modelhttps://community.dpgplc.co.uk/blog/work-based-performance-model2013-09-06T08:00:00.000Z2013-09-06T08:00:00.000ZHenrik Svenssonhttps://community.dpgplc.co.uk/members/HenrikSvensson<div><p>A couple of months ago I published the post <a rel="nofollow" title="Work-based learning" href="http://ennoble.se/work-based-learning/">Work-based learning</a> (based on work by <a rel="nofollow" href="http://www.palorial.com">Niklas Angmyr</a> and <a rel="nofollow" href="http://www.duntroon.com/">Charles Jennings</a>' "<a rel="nofollow" title="Re-thinking workplace learning" href="http://charles-jennings.blogspot.com.au/2013/02/re-thinking-workplace-learning.html">Re-thinking workplace learning</a>") and the time has now come to update the model presented and take it to the next level. In short the change can be attributed to:</p><blockquote>Beneficial knowledge is nice to have but<br/>beneficial behavior is what gets you results</blockquote><p><br/><a rel="nofollow" href="http://ennoble.se/wp-content/uploads/2013/08/competence-behavior.png"><img class="aligncenter size-large wp-image-553" alt="competence-behavior" src="http://ennoble.se/wp-content/uploads/2013/08/competence-behavior-1024x654.png" width="600" height="383"/></a></p><p>Let’s start at the top left. The classic L&D sector is the blue box of <b>competence building activities</b>. Here I’d suggest one would place activities described in the post <a rel="nofollow" title="Multi-layer learning" href="http://ennoble.se/multi-layer-learning/">Multi-Layer Learning</a>, i.e. not only formal ones like classroom training but also informal stuff like communities, role-modeling and databases. And yes, by saying that the top blue box is the classic L&D sector I also say that L&D often doesn’t do anything else of what’s shown in the model above. This model is more applicable for a department working as a <a rel="nofollow" title="From Training to Performance partners" href="http://ennoble.se/from-training-to-performance-partners/">Knowledge and Performance Partner</a> than that of a Learning and Development department.</p><p>Competence building activities <i>could</i> be utilized before work is initiated by the employee – but it isn’t a must. Some people like to <b>interpret their task</b> or work activity by themselves and in most cases this is completely fine since nothing really bad can happen. I’m not talking about airplane pilots here as you might’ve figured…</p><p>While the activity or task is interpreted and later <b>performed</b> it’s important that the employee get as much <b>performance support</b> and usable tools as possible, just in time, as well as getting an overall support by the organization. The latter is where the <a rel="nofollow" href="http://www.kirkpatrickpartners.com">Kirkpatrick’s organizational</a> <b>Drivers</b> come to play. Instead of demanding that all employees must take a certain mandatory course the organization could choose to provide with guidance, support, encouragement and tools to interpret and perform the work activities correctly.</p><p>In these stages of doing (interpreting + actually performing) it is also important that you <b>evaluate the work</b> and the results themselves. Was the activity correctly interpreted? Could it be performed correctly and within the timeframe? Was the end-result what we wanted? All of these are stated within - and compared to - the <b>Leading Indicators</b> (also a Kirkpatrick term by the way).</p><p>When the performed work and its result are evaluated and measured this forms the basis of the <b>reflection stage</b>, which is where <i>learning is extracted from work</i> (Charles Jennings words). The learning from this is shared with colleagues and perhaps with those in charge of education. The reflection also needs the analysis of how the performance support tools worked (was the database complete?) and if the organizational drivers helped at all (did your manager support you?). The learning from the latter is shared with the Knowledge & Performance team to create and/or update new tools and support.</p><p>This isn’t rocket science by any means at all. It’s all about the fact that we shouldn’t add learning to work but more of extracting learning <i>from</i> work. However, we must lift our gazes from the training/learning swamp and start getting with the program of actually getting things <i>done</i>. It doesn’t matter if all of the learning is formal, informal or abnormal.</p><blockquote>If the critical behavior isn’t present – no result will ever come of the performance campaign.</blockquote></div>How L&D became masters of doing the wrong thinghttps://community.dpgplc.co.uk/blog/how-l-d-became-masters-of-doing-the-wrong-thing2013-07-05T09:31:31.000Z2013-07-05T09:31:31.000ZHenrik Svenssonhttps://community.dpgplc.co.uk/members/HenrikSvensson<div><p>In a comment to Niklas Angmyr’s <a rel="nofollow" href="http://www.palorial.com/two-dimensions-of-learning/" target="_blank">post on the subject of learning, corporate learning</a> and the article <a rel="nofollow" href="http://www.europeanbusinessreview.com/?p=8427" target="_blank">Making Corporate Learning work</a> by Shlomo Ben-Hur and Nik Kenley, Niklas get one of the authors to <a rel="nofollow" href="http://www.palorial.com/two-dimensions-of-learning/#comment-1115" target="_blank">comment</a>, namely Nik Kenley, and a very interesting point is made:</p><p>Corporate learning has <i>behavioral change</i> as its main goal – not learning. The clinch is that the L&D departments across the globe DO have learning as its main goal – not behavioral change.</p><p>Why does this matter? It’s similar to leaving your car at the mechanic’s expecting a tune-up and getting it washed, because the mechanic didn’t have tune-up as her main goal but to get your car as good looking as possible. Not only will the car malfunction as before, but you’d probably still get charged for the washing you didn’t ask for.</p><p>Arguing about how to best make sure people learn is therefore all beside the point. Is it more effective to <i>extract</i> learning from work than to <i>add</i> learning to it? This is a totally meaningless debate. What the discussion should be about is of course how we get people to change. The thing is, this also makes the 70/20/10 model moot as it’s about <i>how people learn</i>. Even if it is correct in describing the ways of learning to the very last percentage, it doesn’t really matter if it’s the 10, 20 or 70 if we’d be better off not looking at learning at all.</p><p>Learning should be seen as <a rel="nofollow" href="http://ennoble.se/aligning-inputs-and-outputs-of-a-change-campaign/" target="_blank">one of many possible inputs to a change process</a>. The outputs are the motivation, commitment, behavioral change and result.</p><p>I attended a three day certification course/workshop on the New World <a rel="nofollow" href="http://ennoble.se/planting-a-tree-with-the-4-levels-of-kirkpatrick/" target="_blank">Four Levels of Kirkpatrick</a>some months ago and one of the biggest revelations for me was that the model wasn’t at all that focused on learning and its measuring as I’d previously thought. <a rel="nofollow" href="http://performancexdesign.wordpress.com/2011/02/24/evaluating-with-the-success-case-method/" target="_blank">There’s belief to the contrary</a> but I suspect that this depends on them not researching the newest version of the model but just basing their opinions by assuming that it’s the same as it ever was when it was created 50+ years ago.</p><p>Stating this revelation out loud in class didn’t go that well though. Here’s what I said: “If we’d make sure we get an optimally stated end-result and also get the proper leading indicators, their critical behavior and their supporting organizational drivers down… Then we could produce bad training, or even skip it completely, and still get that end-result!” Jim Kirkpatrick smiled and nodded in silent agreement when outcries where heard from around us in the room.</p><p>“You can’t eliminate training! That’s what L&D do! That’s what our companies want!” with mutters of agreement going around the room…</p><p>Jim answered this training assuredness onslaught with an anecdote about his father, Donald Kirkpatrick, saying the same thing to him. I don’t think that anyone else got the message but me. Jim agreed but we let the subject rest right there and then.</p><p>The new world Kirkpatrick model isn’t about training or learning. It’s all about making sure the changes we want are identified, analyzed, measured, supported and met. (And “supported” does not mean Learning as the only option by far.)</p><p>Nik Kenley continues the comment of Niklas Angmyr’s post by saying:</p><blockquote><p>“The methods and mechanisms of how to change behavior are different from how to help people learn, and therefore, by seeing Corporate Learning as being about learning rather than behavior change, Corporate Learning Functions have been approaching the task ineffectively.”</p></blockquote><p>So, should we really bask L&D for not getting any real organizational result? No, because that would be like asking the car washer to tune-up your car.</p><p>*******************</p><p>More blog posts on the subject can be found on my blog <a rel="nofollow" href="http://ennoble.se/">http://ennoble.se</a></p></div>Collaborate to Evaluatehttps://community.dpgplc.co.uk/blog/collaborate-to-evaluate2013-06-26T15:35:44.000Z2013-06-26T15:35:44.000ZLucy Malleyhttps://community.dpgplc.co.uk/members/LucyMalley<div><p><a target="_self" href="http://storage.ning.com/topology/rest/1.0/file/get/1357312?profile=original"><img class="align-center" src="http://storage.ning.com/topology/rest/1.0/file/get/1357312?profile=original" width="499"></a>In my last blog, I talked about my personal introduction to the world of evaluation with the New World Kirkpatrick Model. Having learnt all about the Four Levels Model and how to effectively apply this theory to display return on expectations, I was really eager to learn more about the other evaluation methodologies out there.</p>
<p> The second programme I attended took a much more quantitative view of evaluation than I had experienced with the Kirkpatrick Certification, although the Phillips Return on Investment (RoI) programme had a similar 2-day, learning-then-doing format. I’d never thought about learning and development from a purely financial view before; for me, it’s very much about soft skills acquisition, building a team and growing as an organisation, so it was interesting to see it all broken down into numbers and financial returns.</p>
<p><a target="_self" href="http://storage.ning.com/topology/rest/1.0/file/get/1357329?profile=original"><img class="align-left" src="http://storage.ning.com/topology/rest/1.0/file/get/1357329?profile=original" width="300"></a>For some areas and in certain industries, RoI works really well and can be a great way to justify that you are getting back the money you’re spending on development. One problem I found with it though; RoI can’t really go into the inherent value of customer service, branding, and all those other things that are not as easily quantifiable as ‘time saved’. It’s these intangibles that, for me, are the basis of Learning and Development- that essential foundation, not particularly quantifiable but nonetheless vital to success, to up-skilling - and ultimately to the bottom line.</p>
<p></p>
<p>An interesting way to look at successful RoI is through the classic call centre training environment- think more efficient phone calls, time saved, all that good stuff. In a situation like this, there are plenty of things that can be measured; and these things all cost money, so you can do an effective RoI on how much money the training has cost vs. the amount of money saved or the increase in revenue (or hopefully, both). Okay, so let’s think about call centres for a minute. Living in an era of technology, getting to actually speak to a real person is increasingly difficult a task. And, by the time you do get through… you might be a little less than cheerful. So, imagine then the relief when you’re put through to someone who is just incredibly good at their job. Friendly, <a target="_self" href="http://storage.ning.com/topology/rest/1.0/file/get/1357352?profile=original"><img class="align-right" src="http://storage.ning.com/topology/rest/1.0/file/get/1357352?profile=original" width="300"></a>helpful, knowledgeable, going the extra mile so that you leave that call with an overwhelmingly positive image of the organisation. You’d mention that to people, wouldn’t you? They might be looking for a new service provider. They remember your conversation, and decide to go with the same company as you because they can be assured that if something goes wrong, it will be resolved. They have a positive experience themselves. They mention it to people in their own circles. And, just like that, it snowballs into a fantastic reputation and great corporate success.</p>
<p> Now, that organisation may well have spent massive amounts of money on training its call centre staff. And it is true that part of the return on this is financially quantifiable efficiency; each employee can now answer more calls, save more time, make the company more money and provide a return on investment for the training. The effect of that training on overall organisational branding, however – this isn’t quantifiable. But it is nevertheless massively important. And it can make you a lot of money.</p>
<p>This was really my main niggle with the Phillips RoI programme, and the gaps that the Kirkpatrick programme filled in for me; it was great learning how to mathematically demonstrate RoI in certain circumstances, but sometimes you need to justify your learning and development programmes by stepping back and taking a look at the bigger picture. Training and subsequent evaluation should be a consultative process; there should be a collaborative approach to ensuring the right solution is selected and delivered so that there is a genuine return on expectations. For me, interesting as it is to add a mathematical element to evaluation, I find that in using RoI exclusively, the roles of consultation and collaboration in effective evaluation are massively underplayed.</p>
<p>Do you evaluate your training?</p>
<p>Which evaluation methodologies deliver for you - and why?</p>
<p>Get in touch, I'd love to hear more about your own experiences of evaluation and evaluation training programmes!</p></div>Evaluation - Between the Happy Sheetshttps://community.dpgplc.co.uk/blog/evaluation-between-the-happy-sheets2013-06-14T10:37:17.000Z2013-06-14T10:37:17.000ZLucy Malleyhttps://community.dpgplc.co.uk/members/LucyMalley<div><p><a target="_self" href="http://storage.ning.com/topology/rest/1.0/file/get/1357469?profile=original"><img class="align-center" src="http://storage.ning.com/topology/rest/1.0/file/get/1357469?profile=RESIZE_480x480" width="444"></a>I consider myself, above all things, a learner. A recent graduate, I’ve spent the majority of my adult life in an educational environment, to the point where I feel, rightly or wrongly, somewhat of an expert in learning as a process on the participatory level. And, above all – I love to learn. Which is why I jumped at the opportunity to take my classroom learning as a benchmark, and from it, learn about (and attempt to become an expert in) something completely foreign to me.</p>
<p>As a newcomer to any industry, there is always a lot to learn; and the L&D world is absolutely no exception. Evaluation is something I’d only ever experienced as a learner, a happy-sheet-ticker if you will. It had never really crossed my mind that an incredible amount of thought goes into designing those tick-boxes, and indeed that a lot can (and should!) be done to evaluate past this point. I’d considered that the comments I’d written on programme evaluations might help to build and positively affect future programmes, but I never looked beyond that at evaluation from the wider viewpoint of view of a university, or any organisation- and how to measure not only the reaction of a participant but the subsequent learning, behaviour change and impact that learning can have.</p>
<p><a target="_self" href="http://storage.ning.com/topology/rest/1.0/file/get/1357483?profile=original"><img class="align-right" src="http://storage.ning.com/topology/rest/1.0/file/get/1357483?profile=original" width="268"></a>I can honestly say that while I may have experienced materials, educators and even programmes that I have not enjoyed, or that haven’t been right for me - I have never had an overall negative learning experience. However, I’ve also never really considered any impact further than knowledge acquisition, perhaps as behaviour change and results are far more transient concepts at University than in a business environment. Thinking about this and attending programmes with learning and development professionals really brought to light for me the relative fragility of L&D- and with this, the absolute necessity to justify training programmes and prove the validity and necessity of development from a business perspective. After a couple of months working in an L&D environment, I became increasingly aware that I’d only experienced one side of evaluation; so when I was offered the opportunity to gain a more in-depth knowledge of the topic, I was excited to explore the value it could provide- not just for the participants, but all the way down to real business impact and positively affecting the bottom line.</p>
<p>My official introduction to evaluation, and the first programme I attended, was the New World Four Levels Kirkpatrick Model. I’d read about the model- and how it been updated from the original 1950s model to apply directly to a contemporary learning environment- but experiencing the programme first-hand really brought to life to me the absolute essential that is effective evaluation at every level. The 2-day programme was split into a day of theory with activities and exercises, and a second thoroughly interactive day. There was a lot to learn in day 1, but having a good group size and more than just one facilitator meant that is was all very engaging and genuinely interesting. Having learned all about taking evaluation from reaction, through learning, and all the way to behaviour change and results, day 2 of the programme was a fantastic opportunity to spend some time in small groups and work one-on-one with the facilitators. Each attendee put together and presented a business case for their respective organisations, working through the leading indicators to show training effectiveness and demonstrating the value of their training programmes, from reaction to results. For me this was essential because it really focused on the practical application of the methodology – using it to justify the value of and need for learning and development programmes within any organisation.</p>
<p><a target="_self" href="http://storage.ning.com/topology/rest/1.0/file/get/1357488?profile=original"><img class="align-left" src="http://storage.ning.com/topology/rest/1.0/file/get/1357488?profile=original" width="400"></a>As well as leaving the 2-day programme with a tangible and effective business case, I found that I was also armed with a bank of very transferable knowledge and skills. I really appreciated how the Kirkpatrick Four Levels model, rather than focusing solely on Return on Investment (which can lead to a side-lining of the Learning and Development department, and an implied loss in its intrinsic value) really emphasised L&D in the context of a business as a collaborative entity. Learning and development in isolation is not an effective way to implement change or a cost-effective way to deliver training that works. The collaborative approach of the Kirkpatrick model showed me how much effective training can occur when individual department/s work together for the greater good of an organisation. </p>
<p>Essentially, the Kirkpatrick Four Levels model is more than just a ‘learn to evaluate’ methodology; it is a consultative tool to help learning and development add maximum value, and it carries through on this promise with each attendee leaving the programme with a bespoke business case proposal to show the real impact of training. To me, that is valuable in ways that discussions on numerical returns alone can’t always be; evaluation of effective training is fundamentally about working together, starting with the end in mind and working collaboratively. The Kirkpatrick model comes through on all these promises and more, and has got me really excited to delve deeper into the world of evaluation- in my next blog I’ll be talking about my experiences on the Phillips ROI programme, and if you have any thoughts or things you’d like to share in the meantime I’d love to hear from you!</p></div>Aligning inputs and outputs of a change campaignhttps://community.dpgplc.co.uk/blog/aligning-inputs-and-outputs-of-a-change-campaign2013-05-23T14:00:00.000Z2013-05-23T14:00:00.000ZHenrik Svenssonhttps://community.dpgplc.co.uk/members/HenrikSvensson<div><p class="last">PS: If you enjoy this post, find more at <a href="http://ennoble.se/" rel="nofollow">ennoble.se</a> ! DS</p><p>Inspired by the <a rel="nofollow" href="http://en.wikipedia.org/wiki/Donald_Kirkpatrick">four levels of Kirkpatrick</a> and the article <a rel="nofollow" href="http://www.europeanbusinessreview.com/?p=8427">Making Corporate Learning Work</a> by Shlomo Ben-Hur & Nik Kinley I came up with a new model this morning. From the latter the inspiration was the following quote (amongst others):</p><blockquote>academic learning is primarily focused on inputs, what is taught and what is learned; but corporate learning should be primarily interested in outputs, how the things we learn are used, and how they can be of value to individuals and organisations.</blockquote><p><br/>From the “Four levels of Kirkpatrick” I’m using the concept of aligning all results with critical behaviors and, if it’s needed, what knowledge is required to be learned to pull that new behavior off.</p><p>Let's combine these two concepts and look at how input/output often is misaligned:</p><p><a rel="nofollow" href="http://ennoble.se/wp-content/uploads/2013/05/a-b-with-inputs-outputs.png"><img class="aligncenter size-large wp-image-383" alt="a-b-with-inputs-outputs" src="http://ennoble.se/wp-content/uploads/2013/05/a-b-with-inputs-outputs-1024x626.png" width="600" height="366"/></a></p><p>All change campaigns aims to get from today, point A, to a desired goal, point B. To get there we need to create a series of Inputs that pushes us in the direction of the desired goal in point B. What really drives the change however is the outputs, the changes in behavior and their results.</p><p>And keep in mind that:</p><ul><li>The inputs are actions that always cost money or other resources</li><li>The outputs are results that make money if they occur.</li></ul><p><br/>In a classic L&D department the inputs are things like webinars, e-learning and workshops. For a marketing department they might be brochures, documentation and exhibitions. These are things that are the “go-to-solution”. We start with creating these content bearers before really knowing what outputs we want.</p><p>But, wait a minute, why don’t the inputs drive the change? If we look at learning as an input (via e-learning or whatever) it’s not the knowledge itself that drives change but how it helps you towards a more productive behavior. One could argue that it’s better to learn as little as possible as long as you actually change to a beneficial behavior… as much as possible.</p><p>Classic outputs are for example: shorten sales process, increase quality and fewer customer complaints. What’s interesting is when we look at how each output is supported, i.e. is there a specific input or range of inputs for each and every output? Sadly it often it looks like this:</p><p style="text-align: center;"><a rel="nofollow" href="http://ennoble.se/wp-content/uploads/2013/05/misalignment.png"><img class="aligncenter wp-image-384" alt="misalignment" src="http://ennoble.se/wp-content/uploads/2013/05/misalignment.png" width="505" height="337"/></a></p><p>There’s a gap between the input and output as they don’t really align. They’re misaligned since no-one has really mentioned that we should be looking at results for our brochures. We can easily calculate how much each and every brochure costs to produce, translate and print but we cannot show you what they do to shorten our sales process. Why? Well, we didn’t produce the brochure for that reason… We must have a brochure, don’t we? So… we produced one…</p><blockquote>We too often put the highest focus on our standard inputs without aligning them with the money making outputs we need to reach our campaign goal in point B.</blockquote><p><br/>Take a look at the complete model below and feel free to come back with any comments.</p><p><a rel="nofollow" href="http://ennoble.se/wp-content/uploads/2013/05/input-and-output-misalignment-model.png"><img class="aligncenter size-large wp-image-390" alt="input-and-output-misalignment-model" src="http://ennoble.se/wp-content/uploads/2013/05/input-and-output-misalignment-model-1024x626.png" width="600" height="366"/></a></p><p>More information regarding the Four levels of Kirkpatrick can either be found at the previously linked wikipedia article or by reading the <a rel="nofollow" title="Planting a tree with the 4 levels of Kirkpatrick" href="http://ennoble.se/planting-a-tree-with-the-4-levels-of-kirkpatrick/">tree planting example</a>.</p></div>Evaluation - What's In It For Me?https://community.dpgplc.co.uk/blog/evaluation-what-s-in-it-for-me2013-01-18T11:00:00.000Z2013-01-18T11:00:00.000ZHoward Rosehttps://community.dpgplc.co.uk/members/HowardRose<div><h1><span class="font-size-2"><a target="_self" href="http://storage.ning.com/topology/rest/1.0/file/get/1357125?profile=original"><img class="align-left" src="http://storage.ning.com/topology/rest/1.0/file/get/1357125?profile=RESIZE_320x320" width="200"></a>It's a cold and wet Monday morning and I'm on my way to meet <a rel="nofollow" title="Dr. Jim Kirkpatrick" href="http://www.linkedin.com/in/kirkpatrickfourlevelevaluation" target="_blank">Dr. Jim Kirkpatrick</a> at a hotel near Heathrow. As I climbed in to my car my thoughts reflected back to a conversation I had with Jim during the recent Kirkpatrick Partners Bronze level programme that I attended..</span></h1>
<p class="float">I was thinking about how, when I worked for a worldwide organisation back in the 80s they used to send us to the 'training school' every three or four months to attend a course of some sort. "So what's the problem" I hear you say, "what a great employer", "they looked after you", "developed you", "spent money on you" etc, and yes of course I would agree with almost all of those points except what a great employer! Why? Well they were wasting their money in the worst way possible, in the belief that I would become a better employee with all these new skills. I learnt to solder well enough to assemble a satellite so it would not fall apart on take-off! I learnt how to compile a business report and a whole host of new skills that I did not need!! </p>
<p class="float"><a target="_self" href="http://storage.ning.com/topology/rest/1.0/file/get/1357137?profile=original"><img class="align-left" src="http://storage.ning.com/topology/rest/1.0/file/get/1357137?profile=original" width="275"></a>The training school were fantastic, the courses received great reactions from the trainees and we always passed the end of course assessments, but somehow things were not changing back in the business units, so often we were sent on other courses or refreshers!! Unfortunately some years ago this organisation closed its training school as they just could not see the value in it and weren't getting results. Was sending us to a training school away from the job really the right approach when most learning is done on the job anyway, is it time for new thinking and new approaches?</p>
<p>As I arrived at the hotel my thoughts turned to the day ahead. I was here to learn more about how Jim delivers his two day Kirkpatrick Partners Bronze programme, as I have the great honor of co-delivering with him in April 2013. We spent the day looking at why our profession, learning and development or as Jim prefers learning and performance, have come to rely on and in many cases mis-use the original Kirkpatrick evaluation model. And how the 'New World' kirkpatrick approach puts things straight.</p>
<p>For many learning professionals the Kirkpatrick four levels is a tool that is used towards the end of training to demonstrate the value of the event or programme, some never even get past level two. In a survey carried out by the ASTD "<a rel="nofollow" href="http://www.astd.org/Publications/Research-Reports/2009/2009-Value-of-Evaluation-Making" target="_blank">The value of evaluation: Making Training evaluation more effective (2009)</a> their research suggests that 91% of respondents use level 1 and 80% go as far as level 2, for levels 3 and 4 it drops significantly to 54% and 37% respectively. This is in line with Jim's comments that the training profession can often see evaluation as a way of patting its self on the back or in other words "Look we did a great job, the delegates loved us and answered all our questions correctly, the organisation is going to be so happy with us, don't you think"? And there-in lies the problem, there can be a big disconnect between the training department and the rest of the business! Does the rest of the business really have much more than a passing interest in how well we deliver the training or how good lunch was or even whether our delegates got 100% in a test? No, the rest of the business wants results, and improved performance, they want things to change as a result of the training. Today, stakeholders are asking "So What?" and more importantly, "What's in it for me?"</p>
<p><a target="_self" href="http://storage.ning.com/topology/rest/1.0/file/get/1357159?profile=original"><img class="align-center" src="http://storage.ning.com/topology/rest/1.0/file/get/1357159?profile=original" width="480"></a>This is where the "New World" Kirkpatrick model comes into its own, no longer should it be used towards the end of the training it should be used as a tool to help shape business thinking, to design learning solutions that matter most and along the way provide opportunities to generate evaluation data. The 'New World' approach is much more than a bolt on after training, it is a fresh up front approach to robust consultancy and training design. For mission critical programmes it allows the learning professional, people like you and me to cross the metaphorical bridge to the business. Starting the conversations off with questions that echo Dr. Steven Covey's "Start with the end in mind" such as "what are the business results you are looking for"? and "what behavioural change will give you this"? These are the starting points for business focused conversations. Conversations that will help you and your organisation answer their what's in it for me question and provide data to demonstrate impact and results.</p>
<p>So the starting point is loud and clear from Jim it is Level Four, what results do you want to see, then we can work back through the levels identifying the behaviours and importantly what the business needs to do to support and encourage them, finally we can think about levels one and two.</p>
<p>On the way home I returned to thinking about that organisation I worked for and the training school, if only they had said to the business "so what results do you want to see from training" and reacted by helping the business define clear goals that could then be used to define some smaller measures of success and the behavours needed to get there, they may even be there today. Some questions to leave you with:</p>
<p><strong><em>How are you currently looking at evaluation in your business?</em></strong></p>
<p><em><strong>What conversations do you have with your stakeholders to identify what the desired change and end result is going to be?</strong></em></p>
<p><em><strong>When your stakeholder's ask you "what's in it for me? How do you respond?</strong></em></p>
<p><em>If you are looking to get a fresh perspective on evaluation and take your approach to the next level, join Howard Rose, Phil Wilcox and Dr Jim Kirkpatrick at the next UK Kirkpatrick Certification Programme in London on April 22nd/23rd 2013. For more information on this exclusive event then please email <strong><a rel="nofollow" href="mailto:service@dpgplc.co.uk">service@dpgplc.co.uk</a></strong>or alternatively all the details can be found here</em></p>
<p><em><strong><a rel="nofollow" href="http://kirkpatrick2013.eventbrite.co.uk/" target="_blank">Kirkpatrick Four Levels Programme London 22nd/23rd April</a></strong></em></p></div>