Who's Really Moving the Big-Data Needle for HR?
While HR struggles to capitalize on the potential of big data, other departments such as IT are already finding business value within the numbers – and conveying that information directly to the powers that be.
By Peter Cappelli
My colleagues Cade Massey and Adam Grant put on an interesting conference at Wharton this month with the title of "People Analytics." That title referred to the use of sophisticated analytic techniques to solve human resource problems, aka Big Data Meets HR.
Here's what I thought was interesting about it: They had a huge turnout, not just among MBA students, who did most of the organizing, but also among employers. For people who work around the topic of human resources, that's a big surprise. It's hard to get business students interested in workforce topics. It's even harder to get companies interested in showing up for events on these topics, let alone donating money -- as they did -- to support this event.
Here's the second surprise, or maybe shock, depending on who you are. Most of those people at the conference, both students and employers, weren't interested in human resources. Most of them, frankly, didn't seem to know a lot about it. Who were they? The best label might be engineers, people with a background in industrial engineering and sophisticated applied statistics. The business function where most of these people were located was probably information technology, because that's where most of the data they were using was either based or accessed.
In short, this was not a meeting of HR people who were using sophisticated techniques to answer their questions. This was a meeting of people who know sophisticated techniques who were moving into HR.
A few days later, I was at a presentation by an IT colleague here at Wharton who had done a sophisticated analysis of email traffic for a company, where they were able to determine the type of communication that made employees more productive. The general content of the emails also predicted who was likely to be laid off. The study led to a new arrangement for employees to use in interacting with each other. All very interesting.
Someone in the audience asked where this project was housed in the company. The answer was the CIO's office. What was HR's involvement? None. HR was seen as an obstacle, likely just to throw up legal concerns.
There is a fair amount of carping among HR experts about big-data researchers, specifically about the things that we already know about HR outcomes that the big data people don't know. This seems to be particularly so for psychologists who have been in the business of studying selection and predicting good hires. Many of the big-data people are indeed trying to predict things that have been studied for some time, including how to predict good hires, improve retention and so forth.
Here's the thing, though. The big-data people aren't reinventing the wheel. They've already found things that traditional HR researchers never knew and, frankly speaking, never thought to ask. One reason is because they have better data. While HR researchers have been kicking around small and simple sets of data, much of it collected decades ago, the big-data people have fresh information on hundreds of thousands of people -- in some cases, millions of people -- and the information includes all kinds of performance measures, attributes of the individual employers, their experience and so forth. There are a lot of new things to look at.
A second reason they are making progress is precisely because of what they don't know. They aren't constrained by having been taught what should predict a good hire and what shouldn't. They are completely agnostic about what predicts these outcomes, and as a result, they are willing to look at all kinds of factors, which allow them to find new relationships. The fact that commuting distance has been found to be a strong predictor of job performance -- a finding from big-data studies -- is pretty important for hiring managers, is easy for employers to check and is not something we knew before.
Finally, the reason the big-data people are going to make progress is because they are far better at analytics than are current HR researchers and certainly better than HR departments. The "People Analytics" conference had a student competition in which all the participants got data on thousands of Teach for America applicants to analyze. The winning presentation from Wharton MBAs identified how the three rounds of interviews applicants went through had their criteria organized inefficiently: They weren't screening out enough people in the second round, and the recommendations for change included how to reorganize the questions across interview rounds to get more accurate predictions at lower cost.
Could your HR department do that?
My bet is that the CIO offices in most big companies will soon start using all the data they have (which is virtually everything) to build models of different aspects of employee performance, because that's where the costs are in companies and it's also the unexamined turf in business. The big-data people have no trouble fashioning arguments in terms of return on assets -- in other words, speaking to the CEO -- and they don't have much interest in explaining what they are doing to people who can't follow it.
The HR people, meanwhile, will stand on the sidelines without studies of their own and without really understanding the big-data studies, but probably complaining about things the big-data people don't understand about HR. The CEOs will listen to the big-data studies, which offer practical advice on how to make more money and ignore the HR complaints. Within a year, most of the topics of those studies will fall under the CIO's umbrella.
Anyone want to take that bet?
Peter Cappelli is the George W. Taylor Professor of Management and director of the Center for Human Resources at The Wharton School. His latest book is Why Good People Can't Get Jobs: The Skills Gap and What Companies Can Do About It.