Big data

• 1. Introduction

  ›

  Big data is not just a lot of information. It is a phenomenon that has outgrown the tools humans built to contain it. Every day, an estimated 2.5 exabytes of new data are generated across the world. To put that in perspective, the world's entire capacity to exchange information through telecommunications was only 281 petabytes in 1986. Within two decades, that figure had grown to 65 exabytes annually. Within another decade, it dwarfed even that.

  The term itself has been in use since the 1990s, with John Mashey credited with bringing it into wider circulation. But the idea it describes is older than the label. In 1984, a company called Teradata marketed a parallel-processing database system. By 1992, that system had stored and analyzed a full terabyte of data. At the time, a standard hard disk drive held only 2.5 gigabytes. The gap between what data could do and what tools could handle it has been widening ever since.

  What raises the real questions is not sheer volume. A quote captured in the source material frames the tension precisely: "There is little doubt that the quantities of data now available are indeed large, but that's not the most relevant characteristic of this new data ecosystem." So if size is not the point, what is? And if big data promises to spot business trends, prevent diseases, and combat crime, why do critics call it a fad, a buzzword, and part of a mythology? The rest of this documentary follows those questions.

• 2. Defining the Indefinable

  ›

  A 2018 definition of big data cuts straight to the machinery: it is data where parallel computing tools are needed to handle it. That framing matters because it marks a shift in the underlying computer science, away from the relational model that dominated databases for decades and toward parallel programming, with some of the guarantees of that older model left behind.

  Traditional relational database management systems were built for structured data, the kind that fits neatly into rows and columns. Big data resists that neatness. It encompasses text, images, audio, video, sensor readings, log files, and social media posts. The original shorthand for its qualities came in three words beginning with the same letter: volume, variety, and velocity. Volume refers to the sheer quantity of stored data. Variety names the expanding range of data types. Velocity captures how fast the data arrives and must be processed.

  A fourth quality was added when researchers recognized that raw scale without reliability was dangerous: veracity, meaning the truthfulness of the data. Without investment in ensuring veracity, an organization's volume and variety of data can generate costs and risks that outpace any value it creates. Later frameworks added a fifth quality, value, which measures the worth of the insights that processing actually produces, and a sixth, variability, which accounts for the fact that data formats and sources shift over time.

  Researchers Robin Kitchin and Gavin McArdle found, in a comparative study, that none of these commonly accepted characteristics appear consistently across all real-world cases of big data. That inconsistency led other researchers to propose a different definition: what matters is not the intrinsic qualities of the data itself but the way data is collected, stored, made available, and analyzed. The label keeps moving even as the phenomenon it names keeps growing.

• 3. The Architecture Behind the Scale

  ›

  Teradata installed the first petabyte-class relational database system in 2007. For many years, WinterCorp published annual reports cataloguing the world's largest databases. Teradata's systems, rooted in that 1984 parallel-processing architecture, eventually grew to include installations exceeding 50 petabytes.

  In 2000, a company called Seisint Inc. built a C++-based distributed platform for data processing called the HPCC Systems platform. The system could automatically partition, distribute, store, and deliver structured and unstructured data across multiple commodity servers. Users wrote data pipelines in a declarative programming language called ECL, and analysts did not need to define data schemas in advance. LexisNexis acquired Seisint in 2004 and used the platform to integrate the data systems of Choicepoint Inc. when they acquired that company in 2008. In 2011, HPCC was open-sourced under the Apache v2.0 License.

  In 2004, Google published a paper describing a process called MapReduce. The model splits queries across parallel nodes in a map step, then gathers and delivers results in a reduce step. The architecture proved influential enough that the Apache open-source project replicated it in a framework called Hadoop. Apache Spark followed in 2012, developed specifically to address MapReduce's limitations, adding in-memory processing and the ability to chain many operations rather than only map followed by reduce.

  Studies in 2012 confirmed that multi-layer distributed architectures were the practical solution to big data's processing demands. Practitioners in this world tend to avoid shared storage architectures, preferring direct-attached storage in forms ranging from solid-state drives to high-capacity disks buried inside parallel processing nodes. The reason is latency: real- or near-real-time information delivery is one of the defining traits of big data analytics, and shared storage adds delays the field cannot afford. DARPA's Topological Data Analysis program, launched publicly in 2008 with a company called Ayasdi, took a different angle, seeking the fundamental structure of massive datasets rather than just faster throughput.

• 4. Science at the Frontier

  ›

  The Large Hadron Collider at CERN runs about 150 million sensors that deliver data 40 million times per second. Nearly 600 million particle collisions occur every second. After filtering out more than 99.99995 percent of those sensor streams, the LHC is still left with roughly 1,000 collisions of interest per second. The four main LHC experiments together produce about 25 petabytes of data annually before any replication. After replication, that rises to nearly 200 petabytes. If every sensor reading were recorded without filtering, the flow would exceed 500 exabytes per day, roughly 200 times more than every other data source in the world combined.

  The Sloan Digital Sky Survey began collecting astronomical data in 2000. In its first few weeks it accumulated more information than all previous astronomical observations combined. Running at a pace of roughly 200 gigabytes per night, SDSS amassed more than 140 terabytes over its operational life. Its planned successor telescope was expected, when it came online in 2020, to collect the same volume of data every five days.

  In genomics, the transformation is equally stark. Decoding the human genome originally required ten years of processing. The same work can now be completed in under a day. DNA sequencers have reduced the cost of sequencing by a factor of 10,000 in ten years, a pace 100 times faster than Moore's law predicted. The 23andMe database, by the time the source was written, held genetic information from over one million people, and a single study linking 15 genome sites to depression generated nearly 20 requests to access the depression data in the two weeks after publication.

  The Square Kilometre Array, a radio telescope built from thousands of antennas expected to become operational by 2024, was projected to gather 14 exabytes and store one petabyte of data per day. The NASA Center for Climate Simulation stores 32 petabytes of climate observations and simulations on its Discover supercomputing cluster. CERN and other physics experiments have been generating these scales of data for decades, usually processed on specialized high-performance computing clusters rather than the commodity cloud infrastructure associated with commercial big data.

• 5. Government, Surveillance, and Society

  ›

  China's Integrated Joint Operations Platform, known by the acronym IJOP, monitors the population and has been used to track Uyghurs specifically. Biometrics including DNA samples are gathered through a program framed as providing free physical examinations. China also began piloting a Social Credit System in a number of cities, intending to assign all citizens a personal score based on behavior, with a target date of 2020 for nationwide implementation.

  The United States government is a major operator of big data infrastructure. The federal government owns four of the ten most powerful supercomputers in the world. The NSA's Utah Data Center was built to handle a large portion of internet-collected information, with estimates placing its storage capacity on the order of a few exabytes. In March 2012, the Obama administration announced a Big Data Research and Development Initiative spread across six federal departments, committing more than $200 million to big data research. The initiative included a ten-million-dollar National Science Foundation grant to the AMPLab at the University of California, Berkeley, funded over five years, and a twenty-five-million-dollar commitment from the Department of Energy over five years to establish the Scalable Data Management, Analysis and Visualization Institute, led by Lawrence Berkeley National Laboratory. Big data analysis also played a documented role in the 2012 Obama re-election campaign.

  In the United Kingdom, Channel 4 has been identified as a leader in the field of big data and data analysis among broadcasters. The British government announced in March 2014 the founding of the Alan Turing Institute, named after the code-breaker and computing pioneer, with a mandate to develop new ways to collect and analyze large datasets. Health insurance providers in the sector have been collecting data on social determinants of health, including food and television consumption, marital status, clothing size, and purchasing habits, then using that data to make predictions about health costs. Whether those predictions are used for pricing is contested.

  Sarah Brayne's research identifies three ways big data policing can reproduce existing social inequalities: by using the authority of mathematical algorithms to justify placing people under increased surveillance; by widening the scope of law enforcement tracking in ways that exacerbate existing racial overrepresentation in the criminal justice system; and by discouraging people from interactions with institutions that would create a digital record, creating barriers to social inclusion.

• 6. Critiques and Limits

  ›

  Researchers Ulf-Dietrich Reips and Uwe Matzat wrote in 2014 that big data had become a fad in scientific research. Researcher danah boyd raised concerns that working with enormous datasets had become its own priority, crowding out the basic principle of selecting a representative sample. The article "Critical Questions for Big Data" described big data as part of a mythology, arguing that large datasets carry "the aura of truth, objectivity, and accuracy" while users are often "lost in the sheer volume of numbers."

  Google Flu Trends, one of the most cited showcases for big data prediction, overstated flu outbreaks by a factor of two. Predictions about Academy Awards and election outcomes based solely on Twitter data were, as the source puts it, more often off than on target. Big data's attempt to predict the results of the 2016 U.S. presidential election produced varying degrees of success.

  Catherine Tucker wrote directly that "by itself, big data is unlikely to be valuable," arguing that processing skills matter more than raw data in creating value. Even among companies investing eight- and nine-figure sums to derive insight from supplier and customer data, fewer than 40 percent of employees have sufficiently mature processes and skills to act on it. An article in the Harvard Business Review concluded that big data, however comprehensive, must be complemented by what it called "big judgment."

  Large datasets are also prone to spurious correlations, arising from non-causal coincidences, the statistical properties of large random numbers as described by Ramsey theory, or the presence of factors not included in the model. Epidemiologist John Ioannidis argued that most published research findings are false, in part because when many research teams each run many experiments and only positive results are published, the probability that any given significant result is actually false grows quickly. Multiple comparisons tested simultaneously across a large dataset amplify this problem significantly. The Johns Hopkins Turbulence Databases, which store over 350 terabytes of spatiotemporal fields from direct numerical simulations of turbulent flows, represent the opposite tendency: data shared through virtual sensors and used in over 150 scientific publications, with the value coming from careful, purposeful analysis rather than scale alone.

• 7. The Human Stakes

  ›

  In 2011, McKinsey and Company estimated that if the U.S. healthcare system used big data creatively and effectively, it could generate more than $300 billion in value every year. In developed European economies, government administrators could potentially save more than 100 billion euros in operational efficiency alone. Personal-location data, the same report suggested, could generate $600 billion in consumer surplus for users of location-based services.

  By 2021, according to IDC estimates, global spending on big data and business analytics solutions was expected to reach $215.7 billion. Statista projected that the global big data market would reach $103 billion by 2027. Major companies including Software AG, Oracle, IBM, Microsoft, SAP, EMC, HP, and Dell collectively spent more than $15 billion acquiring software firms specializing in data management and analytics. In 2010, this industry was worth more than $100 billion and was growing at nearly 10 percent annually, about twice the growth rate of the software industry overall.

  In healthcare, epilepsy monitoring generates five to ten gigabytes of data per day. A single uncompressed breast tomosynthesis image averages 450 megabytes. The sector faces a particular challenge captured in the source: "Big data very often means dirty data, and the fraction of data inaccuracies increases with data volume growth." Human review at big data scale is impossible, yet the consequences of inaccuracy in clinical settings are severe.

  In 2015, researchers Blumenstock and colleagues used mobile phone metadata to estimate poverty and wealth at a population level. The following year, Jean and colleagues combined satellite imagery and machine learning to make similar predictions. A McKinsey Global Institute study found a shortage of 1.5 million highly trained data professionals and managers, a gap that prompted universities including the University of Tennessee and UC Berkeley to launch dedicated master's programs, while private programs like The Data Incubator and General Assembly developed to meet the same demand. The founding members of the BigSurv initiative, which brought together big data and survey science across conferences in 2018, 2020, and 2023, received the Warren J. Mitofsky Innovators Award from the American Association for Public Opinion Research in 2021, a signal that the two fields have begun to recognize what they can do together that neither can do alone.