This article is from 2009. For a fresher take on employee engagement, see:
Defining employee engagement: What does it mean to the experts?
Fostering a culture of belonging and engagement in a hybrid environment
Engagement surveys failing to provide the full picture
Peter Hutton (pictured) is not a happy soul. After more than 30 years in research, including a stint as the deputy MD of pollster MORI, Hutton is convinced many employee engagement surveys are not worth the paper they are written on. To prove the point, he has just written a book, What Are Your Staff Trying To Tell You and he does not pull his punches.
Speaking to HR magazine, he says: "My starting point is this: no one has decided what engagement is and surveys don't ask what bosses want their staff to be engaged with. It's so woolly."
Of course, a bit of controversy is always useful when you have a book to promote but Hutton does the raise the question of just how sophisticated models seeking to measure engagement are.
Singled out for criticism are the criteria used by both Gallup and Best Companies. "The problem I see with Gallup," he says "is that it is entirely based on agree/disagree questions - the so-called Likert scale."
He calls these "ambiguous measures passed off as engagement" because, from a professional researcher's point of view, agree/disagree questions are the weakest type. "There is no other research that just uses agree/disagree questions and then makes such large claims from it," he contends. "It's a joke."
He adds: "Most things you want to ask about a company do not fit this way of asking, and the questions don't link. One response doesn't explain why someone answers another statement."
Hutton claims the correlation between the Gallup Q12 questions and business performance is "extremely small" - ranging from a low of 0.057 to a high of 0.191 (a 100% correlation would be 1). He adds: "No statistician would put any credence on this. Although Gallup does not claim there is a direct engagement:performance correlation, I believe it implies it. But correlation does not mean causation."
So are engagement surveys really so flawed? Jane Owen Jones, who has run two-year, cross-cultural staff engagement projects at consultancy Lloydmasters for the likes of BP oil workers, says: "HR has allowed itself to get into trouble with engagement surveys because they've let them fit into the Ulrich value model - that is, they've become a methodology to show HR provides value to the business, rather than anything to do with the needs of the organisation."
The second company to incur Hutton's ire is Best Companies, which last month published its Sunday Times Best Companies to Work For list, 2009. Its methodology is based on eight concepts.
"It uses factor analysis on 65 or so statements, from which it correlates eight statements," says Hutton. "To me it's just a series of unrelated statements, and they are not all equal as might be implied. It's mono-dimensional. A model should be a scaled-down version of how a business works, but neither Gallup nor Best Companies has this. They offer the wrong questions in the wrong way."
Debate around the right type of questions to ask is a moot point. Bob Ferneyhough, HR director of homecare and DIY brand Henkel, says he makes sure his engagement surveys ask questions "that are pretty close to the bone".
In his survey, for example, one statement reads: 'Ignoring core values gets you into trouble' (and rate your response on a scale of 1 to 5). Maybe HR directors simply need to work with providers on what they want.
"This is difficult, though," confesses Carol Mote, HRD of HR management consultancy Verdant Futures. With previous HR experience at Birds Eye and Gate Gourmet, she says the reality of asking what you want is always more difficult than it sounds: "At Birds Eye we never really got what we wanted, because we couldn't ask the questions we wanted.
Questions like 'What would be the top three things you would like changed in the next quarter?' would be diluted to something like 'How could we improve productivity.' Where I agree with Hutton, and where he isn't far off the mark, is that many engagement surveys don't often get you to where you want to be, and in certain circumstances - such as when I was at Gate Gourmet - if you asked the questions you really wanted, the unions would have used the responses for their own purposes."
Mote believes engagement surveys themselves are not to blame, it is what HRDs are prepared to ask of them. But she still argues they are only a useful starting-point and not an answer to organisational health in themselves.
"I used various suppliers in the past, and the more 'ethereal' ones want you to use their consultancy services to tell you what the surveys mean. In a best-case scenario, engagement surveys help you understand people, their direction, or their empowerment, so they succeed depending on how well you communicate why you are doing them, as that's the only reason people give answers."
One argument is that even if engagement models are not 'perfect', they at least offer consistent questions that provide some barometer of organisational health. Polly Barnes, HR director of marketing agency BEcause - winner of the Growing Business People and Performance Award - takes this further.
She says: "We use the same survey - a Best Companies-styled survey that asks 25 questions - but then interview about 50-60 of our 850 staff to get more detailed responses. On their own, staff surveys aren't useful, they don't supply the answer to problems. A yes/no response won't get to the root of them."
But while the idea of using consistent surveys is an appealing notion, not even this seems to satisfy an unrepentant Hutton. "I can see the thinking," he concedes. "If things are going up, it probably means you're improving and, at least with Best Companies, respondents are obliged to look at what's going on in their own company. But when you take an off-the-shelf product, it's like admitting you've booted out your own company visions."
Although Gallup and Best Companies receive strong criticism from Hutton, both companies feel confident their criteria stand up to scrutiny (see page 26). But it seems all the major players will be kept on their toes for some time to come as new experts emerge professing to address existing models' inadequacies.
Pollster Harris is keen to use its market research roots to find a model to satisfy Hutton. It is called 'Engagement Plus' and seeks to work out how much staff will 'talk-up' their company outside the workplace. "Nearly all engagement models are internally-focused," says employee research director Ian Barrow. "We want to measure how much staff positively or negatively tell others about their job and company."
Whether this will win approval, time will tell. But both Gallup and Best Companies say they welcome the debate.
HOW GALLUP AND BEST COMPANIES RESPOND
Jim Harter, chief scientist for workplace management, Gallup says: "We're very open to critique and peer review but we feel Hutton's analysis is not the full picture. The meta-analysis from which Q12 is derived now includes results from 23,910 businesses.
"The test of any model is its robustness across numerous repeat studies and Q12 has been subjected to six iterations of meta-analysis. Gallup considered many scales prior to selecting the five-point agreement scale.
"As far as its usefulness in the feedback process is concerned, Likert has several advantages, including measuring the direction and intensity of feelings about particular issues and quantification for numerical comparison of progress over time. We also see proof in results with clients.
"They typically double the ratio of engaged to actively disengaged employees in just two intervention periods from the baseline. We have done research that directly links staff engagement levels with bottom-line business benefits. We are happy to make our resources available to those interested."
Jonathan Austin managing director, Best Companies says: "Peter's not the first person to look at our methodology, but it has been independently tested seven times. So we dispute our 'eight measures' of engagement are standalone.
"They were developed from a six-month research project in conjunction with the University of Plymouth that identified 132 engagement items. These were put through a process of exploratory and confirmatory factor analysis that retained around 65 statements - the best measures of employee engagement.
"We know that our 'giving something back' measure influences all of the seven other factors bar none. We also know 'leadership' has a 0.96 correlation with engagement. What we're not saying is that if you focus, say, on the 'giving something back' metric that's the place to start with engagement.
"That will only kick in when you deal with what we've found to be the three lead factors: 'leadership', 'my manager' and 'personal growth'. The art of the methodology is in its interpretation, but for Peter to suggest that the eight factors do not relate to each other is completely wrong."
A NEW APPROACH TO ASKING ABOUT ENGAGEMENT
"For too long, engagement has been an entirely opinion and attitude-based discipline," laments Tony Graham, former HRD of Scottish & Newcastle (S&N). "Little research about what the fundamental issues are for employees has been incorporated into these models and yet the modern workforce has very different needs."
According to Graham, his epiphany occurred four years ago when an S&N European-wide engagement survey by his supplier "appeared to show the company had high buy-in from staff, but I knew staff were massively disengaged and extremely unhappy", he says.
Graham's answer has been to partner with Duncan Fergusan, director of research at brand/marketing research agency Insitas, to produce a model "for the 21st century". H
e explains: "The 21st century worker is different; the psychological contract has shifted. We've looked at how the ground has shifted and how the best companies have tried to accommodate it. Our standpoint is that if the purpose is to understand people, you need to make sure you're asking the right questions in the right way."
According to Graham, not only are most engagement models passive (asking, for example, what managers did for the employee six months ago) most capture what people 'think', which is meaningless because it does not correlate with what they 'do'. "Because Insitas has a marketing background, we look at this from the viewpoint of people's behaviour, not attitude. Engagement is all about what people do, not think."
Insitas asks bespoke questions in five areas - advocacy, motivation, alignment, pride and commitment - and attempts to link these questions to business outcomes. "It's not about labelling people as 'engaged' or 'not engaged' but more about the areas they can improve on that's good for the business," says Graham.
He believes it is vital this realistic picture of engagement is produced. "Once you label people as 'engaged', the motivation to change disappears. Apparently 12% of the population are fully engaged, but this hasn't changed for years. Does this mean you can't/shouldn't do anything with these people? No. Engagement is too complex for single numbers."
WHAT BOB FERNEYHOUGH, HR DIRECTOR, HENKEL, THINKS
"The most widely used survey at Henkel is that by US provider Denison Consulting. The methodology is based on 66 statements in four areas - our mission; business consistency (around our core values); involvement; and adaptability. If you imagine these four as segments of a circle, the survey allows us to see a highly graphical 'wheel' around how we are doing in these areas, how certain businesses compare with an average Henkel score, and how these scores change over time.
"We have done three global surveys using it, the last of which was in 2007, and the time before that was 2004, where we had an 80% response rate. We really believe in the methodology though. Our 2004 survey revealed a lot of variation throughout the organisation, including which leaders were working better than others.
"It also provided us with 600 new improvement projects that different areas of the business had to promise to implement. Of course, engagement surveys are only an indicator of a business's health, and it is not a substitute for good management. In fact we call ours the 'Culture Indicator', but I still believe you need to have both.
"I feel strongly there is no need to run a survey every year, and that not all are ready to use. We modified our questions to include topics such as diversity and corporate identity because that was our focus at the time. We used to use Gallup, but because its questions were yes/no responses, we didn't feel it got to the heart of an issue.
"In our new survey, we ask people to rate responses on a scale from 1 to 5 (from strongly disagree to strongly agree), which we feel gives more shades of opinion. Most of the Gallup questions are also indirect, but I feel you need to come at questions from lots of different angles."
ADDITIONAL RESPONSE FROM BEST COMPANIES
"Best Companies has nearly 10 years experience in the field of workplace engagement and in this time has surveyed over 900,000 UK employees from thousands of organisations - the largest engagement data set in the world. Our methodology, which we openly publish on our website, explains our survey and employee engagement model which was co-developed with the Psychology department at the University of Plymouth back in 2002.
"Our work is designed to help organisations understand and measure levels of engagement within their workplace, to enable them to initiate conversations and implement action plans for improvement. It also allows organisations to benchmark themselves against other companies within their sector, against companies of comparable size and against the best workplaces in the UK.
"We appreciate the opportunity to respond to the comments made by Peter Hutton in his book ‘What are your staff trying to tell you'.
1. 'It uses factor analysis on 65 or so statements, from which it correlates 8 statements from this'
"A more accurate way of describing this is that we measure 66 statements/questions, 54 of which are used to score and measure engagement levels. Each one of the 54 loads onto one of our 8 workplace factors.
"These 8 factors were initially developed in a 6 month research project, conducted in conjunction with the University of Plymouth. We started by undertaking a literature search and from this work 132 employee engagement items were identified.
"These items were then put through a process of exploratory and confirmatory factor analysis which retained around 65 statements - the best measures of employee engagement. This is a widely recognised and standard statistical approach which has allowed us to determine the underlying factor structure of a series of statements/questions.
"Since the initial research project we have continuously developed and tested many more potential questions using a battery of statistical tests. In addition, our 8 factor model of employee engagement has now been independently re- tested on 7 separate occasions (each one using a minimum of 50,000 responses). It still proves statistically to be the best model available for measuring levels of employee engagement.
2. 'The claim is that this is a model of engagement, but to me, it's just a collection of un-related statements. These are not all equal either as might be implied.'
"The process of factor analysis allows us to take what might appear to be a collection of disparate statements and groups them into the meaningful buckets we call our workplace factors. By virtue of the process of factor analysis, each one of these buckets is discrete i.e. each one is measuring a specific element of engagement.
"However, each factor is made up of varying numbers of questions (statements), depending on the inherent complexities of each. Some factors, like Leadership, include sub factors (e.g. the leader, the senior management team, organisational values/principles) but others like Fair Deal are a more straightforward concept to measure. For these reasons, we are well aware that the statements are 'not all equal'.
3.'It's mono-dimensional. Any engagement model should be a scaled down version of how a business works, but neither Gallup nor Best Companies have this.'
"At Best Companies we measure output - how engaged the individuals are within the organisation. As our model represents only employee engagement, we do not, and nor should anyone else, expect it to be "a scaled down version of how a business works". A scaled down version of how a business works, would in fact be a model of business processes, not employee engagement, which if anything is a scaled down model of what engages employees.
"With the benefit of researching this area for nearly 10 years we do also recognise the input and strategic elements required to create high levels of engagement. We believe that this is about the following:
- Providing a clear Primary Purpose - what is it the organisation is here to do? It needs to be about more than simply making money
- Identifying some Core Principles - how are people expected to behave within the workplace
- Setting an Outrageous Ambition - this is about providing a vision for the future and allowing everyone within the organisation to understand how their role contributes to achieving this
- Communicating the Ingenious Plan - which essentially aligns all these elements and ensures that the organisation can deliver on them.
Ian Dennis, senior lecturer in psychology, University of Plymouth, adds: "Hutton's book makes some sound points- it is vital to be clear about the reasons for conducting an employee survey, different types of items have different applications, assessing knowledge is a very different enterprise from determining attitudes and feelings. However there are also some serious misunderstandings running through the book.
"The misunderstandings in the book begin with an unsatisfactory account of the work of Rensis Likert on attitude measurement. Attitudes and feelings, including those about the workplace, are not micro-analytic but have a broad focus. You do not love somebody's nose and hate their mouth, rather you love or hate the whole person.
"This is one of several reasons why, contrary to what Hutton would have you believe, you cannot know what determines the response to a survey item solely by inspecting the item. Likert understood all this and realised that the measurement of attitudes requires scales made up of multiple items. Hutton wrongly labels Likert items as Likert scales. Likert scales are made up of multiple items and the heart of Likert's contribution was not the type of item he invented, but the procedures he developed for selecting items and putting them together to make sound and reliable scales.
"Generations of psychologists and social scientists, including some of the dominant figures of twentieth century psychology have accepted and built on the foundation that Likert provided. Hutton makes no mention of the fact that his critique of the approach of some of his commercial competitors is also implicitly an attack on a set of ideas and methods that have been almost universally adopted in the measurement of attitudes and related constructs.
"Rejecting conventional wisdom is not necessarily a bad thing. Not every view that is widely held is correct. But the authors of a book that does this owe it to their readers to give a clear account of the standard position, and of the weight of opinion behind it, before setting out their reasons for dissent. Hutton singularly fails to discharge this responsibility.
"Rather he seeks to overturn the use of multi-item scales for measuring constructs and factors not by explaining the rationale and then de-bunking it, but simply by writing as if no such rationale exists.
"Nobody with knowledge and understanding of psychological and social research methods is likely to be persuaded by this aspect of Hutton's book. Those who lack that background and want to know more about the use of surveys in assessing employee's attitudes to work and the workplace would be well advised to look elsewhere for guidance."
