ELearning: The “Testing Effect”

“2010 New York Times article reported on a scientific Study
that showed that students who read a passage of text and then
 took a test asking them to recall what they had read retained
 an astonishing 50% more of the information a week later than
 students who had not been tested.”

The “Testing Effect” describes the power of retrieval. Its most common application can be seen in testing in school exams; to help measure learning and assign grades. In the corporate world, this process is often shunned as a facile means of overcoming employee resistance or circumventing policy & procedure issues.

Testing as a whole has come under fire in recent years. Various elementary, secondary and post-secondary learners and academic bodies have cited that testing in the traditional manor is not an effective means of determining comprehension and this notion has carried over into corporate learning to some extent. Testing has gotten a “bad rap” and it is often related to memorization.  

In their book “Make It Stick”, authors Peter C. Brown, Henry L Roediger III and Mark A. McDaniel cited that a “2010 New York Times article reported on a scientific Study that showed that students who read a passage of text and then took a test asking them to recall what they had read retained an astonishing 50% more of the information a week later than students who had not been tested.”

 The book, Make It Stick,  suggests that the New York Times article would seem like good news, but here’s how it was greeted in many online comments:

“Once again, another author confuses learning with recalling information.”

“I personally would like to avoid as many tests as possible, especially with my grades on the line,. Trying to learn in a stressful environment is no way to help retain information.”

“Nobody should care whether memorization is enhanced by practice testing or not. Our children cannot do much of anything anymore.” 

  eLearning can be an important component of learning new skills, enhancing product knowledge, reinforcing policies and procedures or changing behaviour, but only when a well-defined testing process is associated with any given topic or group of topics.  This process enhances retrieval of information and can have an important influence on work practices; but only when content is coupled with a well thought out testing process that is extended over time. 

I suppose the next question is how? How does the process of retrieval of information enhance memory?  The basic concept is this - the act of retrieving information from memory makes it easier to retrieve later. We retrieve information from memory every moment of every day; how to tie a knot, how to start your car, how to program your remote (well maybe not this one), how to add and subtract, etc. These may seem common and mundane but they are made thus through the repetition of retrieving information repeatedly, thereby embedding knowledge more permanently.

I think many of us can bare witness to the effects of retrieval strength in our daily lives, for example, if we go away for an extended trip, our minds would be far away from the normal processes and routines we follow daily. As a result, when we return, we have lost some information, perhaps:  passwords, log in procedures, actions that we may undertake less frequently but typically would recall when retrieved periodically. This can be described as the ebbing of retrieval strength. We are not reminded of the information and gradually we forget.

What does it all mean for eLearning? I think we have to re-consider how we perceive testing, its frequency and the veracity of the testing process. In designing learning programs within corporate environments, we often view learning as a one-time event or a one-time window. We tend to design testing around that event or window but do not plan repeated opportunities to review content, refresh knowledge and enhance retrieval strength through a more rigorous testing process over the long term.

The process need not be onerous. We recommend offering shorter and more persistent testing on information. Make sure the information is prioritized, meaning focus on topics of greater importance to your organization’s mission. For example you may evaluate your learning calendar for a given year and consider reintroduction of key learning principles along with repeated testing opportunities. This can be coupled with the introduction of new content each calendar year. 

Making the process fun and “not boring” will have an important effect on learning and embedding information making it easier to retrieve, however and perhaps more importantly, designing an ongoing process of presenting the prioritized information through multiple channels while reinforcing the learning process with testing is essential to effecting change through learning in your organization.

White Space: Why do we need it?

“But wait Mr. Advertiser, how can I use -
white space  to create more effective ads?”

I guess we need to start at the beginning; what is white space? … and how is it used? The concept of white space in advertising and communications revolves around the idea of creating space around important ideas …either images or text, resulting in an island of focus. These areas can be found in virtually any medium we are informed or entertained by; mobile devices, desk top screens, newspapers, magazines, tablets, billboards, etc.

The idea of white space evolved from early magazine and newsprint layouts, where advertisers sought to highlight ideas and cater to emotional responses by creating negative space with no content or design in it. But today the concept of “white space” applies to any medium since most use white as the default background. While “white” is the baseline, the concept applies to any solid background colour that will serve to hold visual content.

The idea of white space has been around in advertising since its inception and in the early days; using ads designs which made great use of white space was considered elegant. We often revisit ad layouts from the early years and marvel at their unique and interesting lines and use of colour.

As time has progressed and the medium(s) and the message(s) have become more complex and audiences more fragmented, owners and executives of businesses large and small have decided to get more involved in deciding what goes into advertising content and layout without a clear understanding of design and layout principles. With this increase in the number of people involved, more advertising messages in general, the perceived need to get more information into messages and a lack of understanding about how messages are absorbed by audiences, has resulted in advertising becoming a more crowded and less effective medium for some. 

Now I am sure you are saying to yourself, right this minute “But wait Mr. Advertiser, how can I use white space create more effective ads?”  There are a few ways to improve your communications by using white space more effectively. 

1. Consider more carefully the recommendations of your designer or agency – they have your company’s or product’s interest at heart, and they have experience and study data to support their recommendations
2. When you develop copy for an ad – edit it  to a bare minimum and then cut your copy down by 50%
3. Try and have “one” focus for your ad – one message – people scanning mediums, do so quickly and will not stop to consider crowded and confusing content
4. Be consistent with your message across all mediums – its noisy out there and you need to be communicating the same message across all mediums to have impact
5. Create real white space in your design – collages, and multiple images take white space away by confusing the “audience eye”
6. If you are considering adding a paragraph – make it a  short sentence
7. If you are considering adding a sentenced make it a short title
8. If you want to include 10 bullets highlighting features- use 3 bullets instead
9. Try and limit contact information – depending on your products and sales methods a location may not be necessary
10.  Use white space in your work environment as well- keep a clear desk o help promote the application of white space in your thinking
11. Keep you communication short and to the point– business correspondence such as e-mails are a form of advertising- your clients infer a lot about you in your correspondence

While white space may seem like an oxymoron- white space with no content helps focus visual and mental attention on the content that is there!

Learning: Not A One-Time Event

The upshot of all this is that spaced out intervals of 
study reinforces learning and learned practices.

Learning in general, and eLearning in particular are largely regarded as a onetime events in most organizational settings today. The ever increasing pace of change, the avalanche of information, the increased activity within any given job responsibility have created a need to develop an ever-green strategy for learning within an organization.

The evergreen approach to learning should result in a lifelong process of learning within an organization that periodically reinforces existing information, along with new concepts and skills that relate to a particular topic. Yes…it does demand additional resources and funding to dedicate learning as a core concept within an organization. This has not been particularly high on the list of budget items for most organizations. 

Why do I make a case for an evergreen approach? - well because through my experience in developing eLearning for various organizations over the past 25 years I have found that most if not all organizations I have worked with have tended to look at developing learning and eLearning on a project by project basis? While costs have not increased dramatically, the pace of change, the scope of content has – which leaves me wondering why new and more strategic approach to learning have not been implemented.

Common sense tells us we are not doing a very good at creating effective learning environments. When I refer to this I am referring to learning outside the traditional school system. Research tells us that spaced learning produces superior test scores. The question becomes for developing any learning strategy, “What is the typical interval of spaced learning that produces good storage and retrieval strength” (see my article on How We Learn). There is no real handbook or definitive data on this but there is some research that provides a pretty reliable guide to how we can space eLearning and learning in general to improve retention, comprehension and application.

In 2008, a research team led by Melony Wiseheart a psychologist at York University, Toronto and Harold Pashler, a psychologist at the University of California, San Diego, conducted a large study that provided the first good answer to this question. The book “How We Learn” explains that the team enrolled 1354 people from all ages from across the US. The team had them study 32 obscure facts. The participants studied the facts at different intervals –some only 10 minutes apart and others as much as 6 months apart. In total there were 26 different study intervals. The researchers also varied the timing of the final exam.

The study produced an optimal interval chart to help us better understand the relationship between spaced learning and remembering what you have learned.  

Time To Test	First Study Interval
Week	1-2 days
1 Month	1 week
3 Months	2 weeks
6 Months	3 weeks
1 Year	1 month

The further away the exam the more the time you have to prepare - the larger the optimal between sessions one and two. This study included a test to validate the learning process. In most organizational learning (as opposed to traditional school learning) and even eLearning where testing is relatively easy, testing is not always a part of the process.

The upshot of all this is that spaced out intervals of study reinforces learning and learned practices in eLearning. ELearning is usually implemented as a one off and there is rarely follow up through eLearning or other learning methods. This results in poor adoption of learned skills or practices.

In a well-designed “evergreen learning approach” the reintroduction of concepts will help under-write a spaced out study process creating a continuum of learning. This will result in the better application of learned skills and processes, while exposing the learner to new information. In a planned approach to learning this might include spaced out eLearning programs where existing concepts are reexamined and new or revised content is added to create depth of knowledge. Testing is also an important part of the process to ensure that learners take the process as seriously as those who have planned out the learning program.

Finally, based on the previous article “How We Learn” - using different delivery methods of knowledge will help improve “storage and retrieval strength.” For example you may utilize eLearning as a core component, offering  versions of the same eLearning modules with new questions, which can then reinforce with scheduled events, webinars, activities, and print based elements throughout a one year learning cycle. Often, these are not costly; they just demand planning an evergreen approach to learning.

Signal to Noise: What Cambridge Analytica Taught Us

When a company can be brought down through one indiscriminate
 act it becomes increasingly important to embed ethical behavior

Ultimately, it is the individual (an employee at any level) who is responsible for assessing knowledge and how that may or may not affect business prospects, company credibility, industry disruption and ever changing customer preferences in any company. As businesses continue to flatten their hierarchy and employees are becoming more educated they have put decision making in more and varied hands within their organizations.  

Gone are the days that senior management sat at the top of a company hierarchy and made decisions about the company’s future (of course - there are still a few dinosaurs out there that manage from the top down.)Today, resilient forward thinking companies empower all levels of employees to consider and evaluate potential agents of change. Employees are more educated, they tend to enter fields of work that drive their interests and as a result, they are able to research, identify and consider new and alternative strategies more readily.

Take for example, the recent Facebook data leak engineered by Cambridge Analytica where Christopher Wylie claims, the company he worked for, improperly harvested Facebook data from some 50 million users in order to help seal victories for the Trump campaign. While Wylie was simply named as a contractor by Cambridge Analytica , he was still able to influence the direction and fortunes of this company without being integrated into the management hierarchy of the company. His research and unique viewpoint created a new means of utilizing data.

This can be cautionary tale, really a tale of two possibilities depending on how you evaluate knowledge and the process you have in place. The first possibility, we have already covered but let’s look at another opportunity that presented itself to Mr. Wylie. He pitched his services to the Liberal caucus research office and the Liberals signed a contract with Wylie in 2016 and he launched a pilot project. After seeing what Wylie had to offer, the party chose not to proceed further with the project. We are assuming that the techniques are similar in both instances, with Cambridge Analytica and with the Liberal Party Caucus.

I can’t provide a lot of insight into Cambridge Analytica’s process for evaluating knowledge, but one can guess that the evaluation parameters are far less onerous and opportunities are largely evaluated based on profitability rather than credibility, as was not the case with the Liberal Party Caucus.

Most tool sets and strategies available to business in this relatively new category of assessing and evaluating new ideas and strategies use tools such as the Balanced Scoreboard which analyses profit, customer experience, goals and innovation - or alternatively, the Success Case Method that primarily promotes the use of a field trail to evaluate effectiveness and outcomes. Regardless of the process, most of these options are designed for companies with larger infrastructure. Small business has a much more difficult time of it given the lack of resources and the need to innovate.

Embedding ethical culture in companies is becoming an increasingly important component of an organizations’ s life-cycle. When a company can be brought down through one indiscriminate act it becomes increasingly important to embed ethical behavior to ensure that all levels of management understand, promote and communicate ethical tenants and benefits when evaluating opportunities.

An article in Financial Management Magazine provides us with some interesting insight into the process about how effective embedding ethical behavior is- in today's decisions making. This study was conducted in 2017 and it explores the importance of upholding ethical standards. The study indicates that there are differences in demographics in how fraud, corruptions and unethical behavior is viewed by differences age categories.

While Christopher Wylie had a pang of quilt in the aftermath as he surveyed the damage his initiative wrought, clearly the management of Cambridge Analytica did not –even when they got caught with their hands in the cookie jar. If your company is built to succeed now and greed (the gentrified term of course is profit) is the defining architect of the company vision - then embedding ethical standards all levels is not your concern. But if you are building for long term growth you need to consider developing strategies that raise awareness among employees and their role in preventing financial, reputation, or regulatory damage to the organization when considering new and transformative business initiatives.

How We Learn: Storage & Retrieval Strength

" Learning is rooted very much in the mind’s ability  to 

extrapolate a broad range information linked to external stimulus..."

A common thread that affects learning and our comprehension is “How We Learn”. Funny, isn’t it - that this is perhaps the single most important factor in developing learning programs –yet it is the least understood and often least considered aspect of skill development and behavioural change.

 Learning programs, and in particular eLearning programs demand that an individual focus on a set of learning objectives we have set out for that learner. We seek to entrench comprehension and change behaviour through such learning programs. Most learning strategies within organizations and corporations are based on cost and availability of resources with little concern for long term implementation strategies and a need to validate learning.  Effective eLearning involves understanding how our brain works to store, recall and apply knowledge. If we don’t understand how that takes place and how we can best create an environment that will allow this to happen, how can we hope people will apply what they have learned.

Traditional classrooms and their learning strategies are built on historical experiences in organized learning regimens and do not create a connection between learning and understanding. The foundational learning strategies we currently use in most of our business learning environments date back to the 19th century. Experiential learning or the classroom based learning in a one to many environment is the primary means of teaching/learning.

This strategy is entrenched in teachers and learners who focus on a learning formula that suggests: creating a link between memorization, repetition and application equals comprehension and learned skills. It is my contention, based on the existing data sets resulting from many learning studies, that there is significant evidence that this is not the most effective way to teach or learn, however, it may be the cheapest. I wonder of cheaper is better if nothing is learned?

Learning is rooted very much in the mind’s ability to extrapolate a broad range information linked to external stimulus, thus creating links in our ability to store, recall and apply knowledge based on how that knowledge gets embedded.  Creating unique environments, delivering a message in a memorable way and allowing the learner to learn in a way that suites their abilities and capacity to paramount to learning. Check out the Godden & Baddeley study on the effects of context cues on recall.

Robert Bjork of UCLA and his wife, Elizabeth Ligon Bjork also of UCLA can be said to have developed the theory of “Forget To Learn” in the 80’s. The principle theory is this “Any memory has two strengths, a storage strength and a retrieval strength.”  

“How We Learn”, author, Benedict Carey, tells us that storage strength is just that, a measure of how well learned something is. It builds up steadily with studying and more sharply with use.”   He goes on to say, “According to Bjorks theory, storage strength can increase but never decrease”…and, “The brain holds onto only what is relevant, useful or interesting - or maybe so in the future.”   Casey tells us that “Retrieval strength…is a measure of how easily a nugget of information comes to mind. It too, increases with studying and with use.”

Retrieval strength can be a matter of how quickly we can bring things to mind, while storage strength is a matter of how familiar something might be. ELearning is affected by this learning philosophy. If an eLearning program, for example has 6 modules, and each is about 45 minutes long which we take only once, than the storage strength and retrieval strength are both low.

How do we improve on this? Each module can be followed up with questions. This serves as repetition of content and improves retrieval. Adding case studies and interactive scenarios helps improve storage strength as we apply and use the knowledge. We can further improve storage and retrieval with follow up meetings and one-on-one sessions that extend our learning of the concept, by creating a series of demands on our memory that recall and apply the principles learned.

Spreading out the learning process over a longer period of time, changing the delivery medium and varying the time and place in which we learn are all hall marks of eLearning. This creates an environment where the knowledge can become more embedded in our memory, improving our storage and retrieval strength and allowing us to recall information more readily. This is especially true if we continue to follow up and extend the learning process through a variety of additional learning opportunities. These complementary learning opportunities that complement eLearning can be simple and low cost including; group meetings, one-on-one simulations, learner surveys, reading assignments, learner feedback, follow up evaluation of implementation strategies, etc.  

ELearning is an ideal medium for knowledge transfer, but on its own it’s is far less effective. Learning strategies, that use eLearning as the primary learning medium, delivered over longer periods while including multiple learning opportunities and multiple mediums create the best outcomes.

Not For Profit: A lesson Learned

He looked at me and said: “No, while those objectives
 are all admirable – your one most important objective
 is to – find your replacement.” 

I have served as senior decision maker (president) for two “Not for Profit” organizations for a total of more than 10 years. For me, it was a way of giving back and contributing in my community.  While the rewards of contributing, working with like-minded altruistic professionals and helping change people lives was great – I learned a very important lesson that I would like to pass along should you be considering participating in a not for profit organization in any decision making capacity.

The situation unfolded some 8 years ago when I chose to take on the position of president for a not for profit organization.  I really had no experience but I wanted to find out more about what I was capable of and what I might be able to contribute.  After being elected to the position one of Board of Directors approached me privately and asked “What do you think your primary objective should be?” After a bit of thought I regurgitated the usual objectives, including some “boiler plate” mission statement objectives that the organization was committed to.

My initial thoughts on the question, suggested that perhaps this senior Board Member, who had a great deal of experience on many prestigious Boards of Directors was not confident in my abilities and was perhaps testing me in some way. After my response (and I am paraphrasing here) he said quietly “No, while those objectives are all admirable – your one most important objective is to – find your replacement.”  Now I was certain ….this person did not have a lot of confidence in my leadership abilities.  At that point he turned and walked away and he did not say another word…nor did I have any chance to ask about what the comment meant. I felt terrible…my first day as president of this Not For Profit…and I was being asked to seek my replacement as quickly as possible.

At this point I decided to make the best of it and keep a stiff upper lip and continue.  Months passed and I attended a number of in-person and teleconference meetings in my new position and did not have any occasion to find out more about the comment. A year passed and I was asked to take on a second Not for Profit senior decision making position – and in spite of my misgivings I accepted this position; and now I was President of two organizations.

 In the handling my responsibilities of this second more demanding positon I gradually began to understand what the comment had meant. As I continued in my responsibilities I oversaw many changes to both organizations and years passed I began thinking about how I could relieve myself of at least one if not both of these demanding positions.  I was working full time as an entrepreneur, our family was growing and changing, responsibilities for each organization was becoming more intense and I needed an exit strategy. 

Now that comment began to make sense - and I decided to put the question directly to the person who had originally posed it to me, “What did you mean when you told me I had to focus on planning for my replacement?” He smiled and he started by fist telling me what a great job I was doing and then went on to tell me, on the many boards he had served on he had learned one thing, that senior board positions are difficult to fill and if I wanted someone to carry on with the strategic process I had started I needed to find and put in place several people who I felt might one day be able to step into my shoes.

Suddenly it all made sense, for several years I had been second guessing myself and I finally realized that  this has just been a piece of advice from someone who understand the burden of leadership and what it takes to carry on the important work of not for profits.  I understood in that moment it takes time and effort to create a succession plan that would allow for seamless transition in leadership. 

I have since left both organizations and passed on the mantle of leadership to capable leaders who have worked to improve the reality for each of these organizations.  I have not forgotten this important lesson and I often see it play out in those I connect with in my everyday business experience – leaders doing great work who have not thought ahead to the day when they want to reduce their role and have a capable replacement  to carry on the important work of their organization.

E Learning Buy-in: Getting your staff Engaged

Check out the 11 steps you can take to improve the
potential  success of your eLearning implementation

E-Learning like most training and knowledge transfer initiatives demands planning and careful implementation. Often, the audience is the last consideration in the process and few if any audience marketing, achievement reward programs, audience feedback or engagement tracking is implemented as part of the plan.

I have been witness to several implementation programs within small and large organizations that have failed miserably. In one case we developed a program that was launched to reach 10,000 employees. There are few statistics (since organizations rarely share such information) and developers are not asked to participate or offer any meaningful input into the distribution process of eLearning within an organization.

Herein lies the problem; organizations form management and administrative bodies across disciplines, departments and regional divisions etc. The penchant for administrative protocol creates a lack of communication so that these bodies focus more attention on process rather than on results.

Since eLearning has been designed to by accessed anywhere, anytime, understanding and catering to personal and professional lives plays an important role in determining how we engage learners. Management and administrative bodies do not consider this and assume (wrongly) that if we create program learners will be forced to participate since they are employees of the company or organization. .  What they fail to understand is that learning is not a forced exercise that automatically engages learners and creates retention and the willingness to apply that learning in their work experience. It is often seen as many company initiatives are a well-intentioned exercise that is not really designed for “me”, the very person we are trying to reach.

Creating and incentive and an understanding of what the eLearning program hopes to achieve and how each employee (or better yet “person”), if they play their part can contribute to the success and rewarding that success both personal and organizationally are what can make the difference between success and failure.

Alright…so we have created a case for incentivizing eLearning for employee buy in; so how do we go about doing it? This is a complex topic that demands an understanding of the work environment, the learner’s demographic profile, personal worker preferences and the organization’s ability to commit to a longer term and a consistent implementation process that seeks to respond to learner needs. 

Having said this, I can provide a basic bullet point list of issues to consider and steps to take, in no particular order:

1. Think about implementation in terms of advertising. Get the word out in a creative and engaging way
2. Involve the learners in the process of advertising; coming up with creative ideas for its implementation gives them ownership and they become invested in the process
3. Get third party input. This could take the form of the eLearning developer, internal communication resources, advertising agency, etc.
4. Repeat the “getting the word out” process throughout the implementation and distribution process. Before, during and after
5. Promote the reward process and how and how individuals and the organization have benefited
6. Tell personal stories and provide statistical evidence on the program’s success or “lack thereof”
7. Track and share engagement statistics.
8. Define and beginning and an end to the implementation process. Leaving it open ended suggests that it is not important
9. When as organization meets organized resistance, i.e., unions,  go back to more communications with employees  because you probably have not created enough incentive for them to fully participate or understand its value
10.  Use learner surveys, feedback meetings, webinars, etc., and follow up opportunities to  get feedback on  what worked and what didn’t
11. Last but not least, act on the feedback provided. You don’t have to act on every suggestion but you do want to  let your learners know you are listening and genuinely want this to work