An edited version of this article was originally published in theHRDIRECTOR in June 2014

The buzz is about Big Data, but does that mean ‘Small Data’ is brushed under the carpet?

Big Data is a fashionable phrase describing the vast amounts of digital information now available – from stratospheric growth in online traffic and reductions in storage costs – and the clever analytics used to find patterns in that data. But, with Big Data grabbing so many headlines, what is ‘small data’ and why is it so important?

Far from being trivial, small data on a human scale can herald early warning signs of change in organisations. That’s why it is so important for HR Directors and other business leaders to ensure it is not simply brushed under the carpet.

‘Small data’ refers to the kind of information that we can all pick up, every day, in the normal course of our work. It might take the form of an unexpected comment which suddenly galvanises people into action; a growing disconnect between two groups; a new story that starts circulating virally; or a dip in energy within a team. As the name suggests, small data may be indistinct, so it can be tricky to define and articulate, even when we sense it. What makes it still harder to grasp is that small data is distributed around organisations. We each hold a piece of the puzzle, yet no-one can stand aside from the constant creation and flow of data in organisations to see the whole picture. Nevertheless, small data is human scale data, often qualitative, that we can all notice if we know what to look for.

Small data, on a human scale, can herald early warning signs of change in organisations.

And small data can be powerful in challenging what we think we know. You only need to notice one black swan in order to prove that swans can be black. Even more importantly, it is the best data we have about how an organisation is changing, until after the fact. The power of small data is that it can give us advance warning of potential problems or completely new opportunities. It can signal important twists and turns in change, heralding the kind of surprises and unintended consequences that you just cannot plan for. And, the earlier that leaders and managers become aware of these signals, the sooner they can choose how to respond. When the patterns of change become really obvious, what we have is hindsight, and it can be too late to do anything about it. So, what we need is foresight – the ability to anticipate and make sense of dynamic patterns as they emerge in the here and now.

These issues are becoming more acute since many of us now work in multi-change environments. In HR, for example, we might simultaneously be working on a big re-organisation, introducing new staff engagement initiatives, and be upgrading our HR systems. These changes can impact on one another in ways that cannot be predicted. Add to that Marketing’s new product launch, IT’s Bring-Your-Own-Device rollout, and Finance’s introduction of new payment authority levels, and there’s every chance that a perfect (change) storm could be brewing without our knowledge.

But, the good news is there may be vital clues in the small data we pick up every day about surprising or unintended patterns. When market researchers for Febreze noticed consumers were using their product after cleaning, rather than to eliminate smells, they refocused their marketing, and sales doubled. Within organisations, a ripple of excitement in a room, or on social media, may indicate an opportunity to enhance staff engagement. Yet localised resentment about new car parking charges, for example, may signify more than an inevitable, hopefully short-lived resistance to change. It could be a clue in an emerging story that ‘they’ don’t care about ‘us’, which aggravates existing divisions.

By noticing small data, and then asking ‘what’s going on here’, leaders and managers can choose their responses, rather than acting automatically. And the good news is that human beings are better than computers at noticing and interpreting small data, which is qualitative, indistinct and very human. Utilising small data to detect potential problems and opportunities before they become big issues is a strategic leadership competence; one which could help avoid missed opportunities and nasty surprises.

Yet, the risk remains that the beguiling allure of Big Data will blind us to small data. So we might miss those vitally important weak signals – until it’s too late.