Learning Journeys >
This learning journey has been curated by AmbITion lead, Hannah Rudman.
‘Big data’ is one of those phrases currently being overused but little understood! Big data refers to a combination of technologies that can search and analyse massive amounts of information nearly instantly no matter what format they are in: tweets, posts, e-mails, documents, audio, video, etc. Big data is changing things for three reasons – it:
- can handle massive amounts of information in all sorts of formats.
- works fast — practically instantly.
- is affordable because it uses ordinary, low-cost hardware.
This learning journey introduces and explores the opportunities of the increases in the volume of data being generated by ever-growing computer power. Big Data will be assisted by the emergence of the “quantum computer”: the hard maths and huge amounts of data crunching won’t overcome the computers, which look to be going quantum. Explains The Observer’s John Naughton:
“At its heart is the idea of a quantum bit or qubit. The bits that conventional computers use are implemented by transistors that can either be on (1) or off (0). Qubits, in contrast, can be both on and off at the same time, which implies that they could be used to carry out two or more calculations simultaneously. In principle, therefore, quantum computers should run much faster than conventional, silicon-based ones, at least in calculations where parallel processing is helpful.”
Four technologies make up Big Data:
- Analytics — analyzing big amounts of data to come up with answers
- In-memory databases — processing information super-fast
- SQL databases — cloud-based computing allowing infinitely scaleable data collection and analysis
- Toolkits and dashboards — to immediately reflect to the end user trends and results
Google’s BigQuery is an example of the technologies blended to provide big data analysis capability.
Big Data is considered by most academic business and management schools as a new resource for better understanding (and therefore responding to) customer behaviour. Digital businesses such as Amazon, Google, LinkedIn and Facebook have been built on insights gleaned from big data. However, arts, cultural and heritage organisations and practices seem to be behind the curve of embracing big data. It could be an opportunity for the cultural sector to reframe the terms of the current debate about its impact, by better capturing the ways the sector creates social capital and cultural value. But it also has the potential to be of great value within our organisations and businesses.
1 Background Reading: on big data and society
If you’re an organisation or business focusing on the big data question and how it might revolutionise your marketing, I advise you to take stock of your objectives. Is there a clear connection between your strategy and the data? Are you only planning to collect the data you really need (regardless of size)? Do you have a clear business case for why you’re collecting the data and a roadmap or implementation plan to steer the course?
Some of the bigger implications are considered here in these reflections by Kenneth Cukier of The Economist, which were published originally in The Guardian Media Professionals Network by Adam Davidi on Friday 12 April 2013.
“Can you tell us a little bit about your role as data editor for The Economist?
The position is a new one, but it stems from the idea that the new wealth of data and new tools to process and visualise it means that we as journalists can tell stories in new ways. Instead of basing stories on a string of anecdotes with a single statistic dropped in, we can invert the form and make the data the story, while just using a judicious anecdote to illustrate the information. In this respect, The Economist can be said to have been practising data journalism for 170 years; we’re known for our data-driven content like the Big Mac index to compare currencies.
How do you characterise big data?
There is no concrete definition and that is probably a good thing since to define is also to limit. But it’s not woolly either. We can understand big data by its features, and the central one is this: we can do things with a huge corpus of data that we are unable to do with smaller amounts, to extract new insights and create new sources of value. This encompasses things like machine learning, in which we have self-driving cars and decent language translation. This is not because we have faster chips or cleverer algorithms, but because we have more data (and the tools to process it at a vast and affordable scale).
In what ways is big data transforming the world?
It is on track to touch all aspects of society. We will go from a world that we understand by experiencing it on an individual level, to one we comprehend on a more universal level. By that, I mean that we tend to base decisions on small amounts of data that are usually just a simulacrum of the complex reality we are trying to deal with, and tailored to our cognitive limitations to make sense of it. Tomorrow, we will use big data to surpass our faith in our individual powers and instead place trust in the data (though not blind trust).
Take medicine. Today, doctors make diagnoses based on their judgement. Sounds reasonable? In time, this will probably be considered as barbaric as bloodletting. Why not use big data? We could enshrine the experience of all doctors, and of hundreds of millions of patients over decades, to identify the best treatments to achieve the best outcomes and spot hidden adverse drug side effects. After all, the sum of all medical knowledge isn’t in the possession of any single physician. But if we aggregate vast troves of healthcare information, we may learn what works best, just as Amazon recommends books not based on the inklings of a literary critic but from correlating sales data. This will mark a revolution in how society uses information.
Concerns have been raised regarding the privacy implications of big data. What is your view of this?
Privacy is a big problem today and it will be a bigger problem tomorrow. We need to improve the legal regime to govern privacy to move beyond the system of notice-and-consent (that is, companies inform users what data they collect and how it’s used, and people give their okay). In reality it means that people tick a box agreeing to 60 pages of legal jargon with nary a glance. Instead, we need to consider the use and misuse of the data, not just the collection. We need to focus on the area of harm and not just on the inert, potential harm.
Are there any other challenges associated with big data?
While privacy is a problem, a newer issue is ‘propensity’. This refers to the idea that algorithms may be making predictions about what we are likely to do, and we may find that we’re penalised before we’ve actually committed the infraction. So big data may assign a 95% likelihood that a certain person will shoplift, or default on a loan, or fail to survive a surgical operation. We’ll need to sanctify human agency and freewill. At the same time, we’ll need a new class of professional the “algorithmists” to review big data analyses and provide society the same transparency and accountability that we have today, to ensure that big data is not a black box that obviates the public interests.“
These reflections by Kenneth Cukier of The Economist, were published in The Guardian Media Professionals Network by Adam Davidi on Friday 12 April 2013.
2 Background Reading: The Rise of the Datavores - an introduction to the landscape
This Nesta paper introduces the new landscape of big data – what are businesses in the UK doing with the amount of data available to them? How are they using it to grow or innovate?
This November 2012 Nesta report by Hasan Bakhshi and Juan Mateos-Garcia introduces the idea that the internet economy is growing, and so is the amount of data that it generates. Are UK businesses making the most of this data to innovate and grow? Or are they missing out on the opportunities of online data?
A business survey by Nesta shows how a small but significant group of UK companies – the datavores – are making sophisticated use of data to drive business decisions. This paper illustrates both the opportunities and challenges that online data presents for UK businesses.
3 What can big data do for the cultural, arts and heritage sector?
This Nesta paper explores the potential of big data specifically for the culture, arts and heritage sector.
The current approach to the use of data in the cultural sector is out-of-date and inadequate, suggests this February 2013 Nesta report by Anthony Lilley and Professor Paul Moore.
“The sector as a whole and the policy and regulatory bodies which oversee it are already failing to make the most of the considerable financial and operational benefits which could arise from better use of data. In addition, a significant opportunity to better understand and possibly increase the cultural and social impact of public expenditure is going begging.
It is high time for a step-change in the approach of arts and cultural bodies to data and for them to take up and build on the management of so-called “big data” in other sectors.
This report aims to set the issues in a wide strategic context. The overall objective is to help senior cultural decision-makers to understand the importance and urgency of the need to think differently about the potential of big data and to encourage them to set in train changes to the environment, the metrics and the skills to make the most of big data which are needed to harness its potential.”
4 Tracking, evaluating and measuring digital engagement - tools from Culture24
The brilliant Culture 24 project which works with UK Galleries, Libraries, Archives and Museums (GLAMs!) worked with 22 organisations to develop effective ways to understand and measure digital engagement, using a number of different tracking and measuring tools to pool social media and web data (together = big data), culminating with this 2013 report.
And here’s links to the other useful resources:
Social media metrics toolkit – A framework suggesting ways to make use of your social media metrics.
Social media tools comparison – A comparison of the tools identified to track diffident different social media channels.
Thanks to the brilliant Culture24 for sharing the outcomes and insights and tools of their latest 2013 action research project. Galleries, Libraries, Archives and Museums (GLAMs!) worked together to share experiences and exercises in understanding and measuring levels of digital engagement.