What Employee-Engagement Surveys Really Tell Us
A recent journal article has harsh words for the way employee engagement surveys are commonly conducted, claiming they can do more harm than good when done improperly. Nevertheless, the author and others still see value in administering engagement surveys, so long as organizations and HR leaders are able to craft them in ways that elicit honest, useful feedback.
By Mark McGraw
In a recent article in the American Society for Quality's Journal for Quality and Participation, author, consultant and statistician Robert Gerst notes that American companies are spending $720 million annually on improving employee engagement, according to Deloitte by Bersin research.
Much of that money and effort, he writes, is thrown into creating and administering employee-engagement surveys, a practice Gerst says yields very little -- if any -- useful information for the organization.
"The dirty little secret of employee engagement surveys," he writes, "is that they're largely junk science -- placing the marketing objective of telling and selling a good story above the practical and ethical objective of telling the truth."
Gerst, partner in charge of operational excellence and statistical methods at Calgary, Alberta-based Converge Consulting Group Inc., goes on to outline what he sees as the flawed methods many organizations rely on to craft engagement surveys.
For example, he says, the use of statistical significance to identify "important findings" really only uncovers "trivial differences of no practical importance."
In a recent interview with HRE, Gerst elaborated on his theory that conducting the surveys can actually hamper engagement, and create employee issues where none previously existed.
Say, for example, an engagement survey reveals a 5 percent difference in engagement-level scores between two given departments within an organization. Such a differential may be noticeable, but not necessarily significant, he says.
"When we try to sell the notion that [these numbers] mean there's an important difference between those two departments, bad things happen. Let's say a department gets branded as having a low engagement score, but in fact isn't lower. All of a sudden, morale [takes a hit]. People look around and wonder, 'What are we doing wrong? We thought everything was great, but now we're being told we're somehow deficient.' "
In his article, Gerst also claims the common practice of building regression models of employee engagement creates "statistical fairy tales ... composed of cherry-picked engagement drivers, ensuring employees are held accountable for the failings of executives."
"It gets even worse," he continues, noting that organizations wind up basing compensation and other critical decisions on misleading data gleaned from engagement surveys.
"We've seen quite a bit of the practice in which managers' or executives' bonuses or promotions are being based on [their teams'] engagement scores -- dependent on some kind of positive score that is in fact based on a fallacy; a simple scientific miscalculation. So I wouldn't blame managers for being upset over [not getting a bonus] because of a low engagement score."
Add to this "the belief that complex concepts such as employee engagement can be reduced to a single number," he says, "and the result eliminates any hope of understanding or improving the employer-employee relationship."
Adam Zuckerman, global practice leader for employee surveys with New York-based Towers Watson, takes a different view.
"This article is essentially saying [employee engagement] is too complex to measure," says Zuckerman. "And that's patently absurd. It is complex, but that doesn't mean that it's impossible to measure or meaningless.
"Employee engagement is nothing more than an attitude an employee holds about his or her relationship with the employer," he continues. "Since at least the 1950s, the psychology of attitudes -- including their formation, change and link to behavior -- has been the subject of comprehensive academic scholarship. These are social scientists using the scientific method -- empirical, systematic observation, description, and measurement, as well as experimentation and statistical analysis -- to inform the development and testing of falsifiable hypotheses."
Researchers regularly document and share methodologies and results, he says, making them available for scrutiny by other scientists, "who can verify or refute their findings," says Zuckerman. "The field of psychology has clearly demonstrated that attitudes can be reliably measured over time, and that we can identify the factors that influence their development and their connection to behavior."
Engagement is "no doubt a complex subject," says Mark Royal, senior principal at Philadelphia-based Hay Group Insight. "But," he asks, "is it any more complex than emotional intelligence, leadership effectiveness or personality?"
Researchers commonly develop and use surveys and tests in an effort to understand those concepts and the dynamics associated with them, he says, adding that "employee engagement surveys reflect a similar attempt to apply social science techniques to a topic widely seen as critical to business success."
A common concern for employers is that engagement surveys may motivate employees to respond negatively in an effort to further their own interests. For example, employees angling for a raise may indicate feeling they're not adequately compensated. As such, a survey's success largely hinges on employers' and HR leaders' ability to craft surveys that elicit honest and useful employee feedback, says Royal.
"The emphasis on careful communication describing survey objectives, how data will and will not be used, and the protections in place to ensure anonymity and confidentiality in responses is all driven by a worry that -- absent these reassurances -- employees may not be willing to be fully candid in sharing unfavorable opinions of the organization and their work environments."
Royal does note that employee responses to compensation and rewards are "generally among the least favorable," saying it's "quite likely" there's an element of self-interest behind many employees' tendency to indicate through surveys they feel underpaid.
"In any social science, you have to think about the mindset of respondents," adds Zuckerman. "But that's what makes benchmarking data extremely valuable, because you can essentially take into account or control for that exact influence."
Savvy organizations and HR leaders make use of both internal and external benchmarks to put results in an appropriate context, allowing leaders to understand the significance of concerns about compensation and rewards and respond accordingly, says Royal.
"Given that [HR] teams generally have access to adequate information about the true competitiveness of the pay, benefits and perquisites they offer, unfavorable responses in these areas are often seen as a signal that communications need to be enhanced related to company practices and offerings."
Organizations are increasingly turning to surveys as strategic tools, he continues, leveraging employee input in order to drive organizational performance.
This strategy "reflects a growing recognition of the critical link between people and strategy, and the extent to which human capital provides the most sustainable source of competitive differentiation for organizations," says Royal. "Accordingly, organizations are using surveys to gauge clarity regarding organizational and individual objectives, the quality of collaboration within and across teams in highly interdependent environments, perceived support for innovation and risk taking, and the effectiveness of work structures."
Despite seeing flaws in how engagement surveys are often created and administered, Gerst agrees they can be valuable in measuring employee engagement; provided the surveys are used as just one piece of the puzzle.
"I think employee engagement surveys are an absolutely critical tool to get feedback, in conjunction with other methods," says Gerst. "Your people are taking time to give feedback. If we're going to take in that information, we deserve to treat that feedback with a level of respect, and to analyze it properly."
"Employee surveys are never the only source of information leaders have about employee engagement levels," adds Royal.
"There are a variety of employee touch points -- performance discussions, team meetings, focus group discussions -- that can inform an understanding of motivation and commitment. The employee survey, however, offers leaders a unique opportunity to supplement anecdotal information with a representative look across the whole organization."