What is big data? Googling “big data” gives you more than 15 million results—that gives you a pretty good starting point about the buzz.

With advancements in technology, it seems like we have nearly infinite amounts of data and information. The much mentioned statistic that “90% of today’s data was created within the past two years” shows just how fast data are accumulating.

Data are being collected with every click and move we make, and additional powers of analysis have come with all of these extra data.

In their book Freakonomics the authors show the power of “more data and analysis” through the example of sumo wrestling.

Allegations of corruption in the sport had surfaced over the years, but there was never any smoking gun. Economist Steven Levitt looked at 66 000 sumo matches over 11 years; what had not been seen through random small samplings, but was evident from looking at the bigger picture, was a series of seemingly insignificant matches in which one of the competitors was “gifted” a win when there would be no harm to the standing of the other competitor. Without the wider view, the match fixing was essentially invisible.

But big data can’t do everything. When it was launched in 2008, Google Flu Trends gathered all search queries about influenza and added other data, such as on air travel—this eventually showed a near real-time spread of the disease across the United States of America. However, recently Scientific American reported on its limitations, when Google Flu Trends overestimated the prevalence of flu during two seasons by more than 50%.

So why should we care? The authors of the book Big data: a revolution that will transform how we live, work, and think cited this example and note that there is a treasure hunt under way for insights to be extracted from data.

And if there is a treasure hunt, how do we find the treasure and, at the same time, protect it? We know data affect us on many levels—from the kind of adverts we see, to improvements in the products we buy. And some of us worry about confidentiality of data, for example confidentiality of medical records.

“It’s about quality and trust,” says one of the coauthors of Big data, Viktor Mayer-Schönberger, the Professor of Internet Governance and Regulation at the Oxford Internet Institute at the University of Oxford. Mr Mayer-Schönberger argues that it’s not possible anymore to teach everyone how to analyse data, so we need impartial intermediaries to help us. This all means that we may need a new framework on how to simplify and harness big data while protecting our privacy. 

This week Daily Development will hear in an interview with Colin Koopman, of the University of Oregon, a cautionary tale about the use of big data, and Stew Langille, the Chief Executive Officer and cofounder of Visually, gives his thoughts about the new visual narrative about data.