Big Data: A Revolution That Will Transform How We Live, Work, and Think

1. 1Shift your data mindset from static to renewable: treat every dataset not as spent once its primary purpose is served (the plane lands, the query is processed) but as raw material with latent option value for future, undefined uses.
2. 2Collect at maximum granularity and comprehensiveness (N=all where feasible), because full or near-full datasets preserve the freedom to re-focus later—like a Lytro camera capturing the whole light field so you decide the focal point after the shot rather than before.
3. 3Datafy the world around your process: attach sensors, capture accelerometer readings, log every interaction, so that phenomena previously invisible become reusable data (GreenGoose sensors on dental floss; iTrem's use of the phone's built-in accelerometer for tremor monitoring).
4. 4Identify the primary use, then deliberately catalog candidate secondary uses—ask what other question this exhaust could answer, since data's value now shifts from its primary use to its potential future uses.
5. 5Design dual-purpose capture systems that perform a task while harvesting reusable data as a byproduct (ReCaptcha authenticates a human AND disambiguates scanned words in one motion).
6. 6Merge datasets to create value neither holds alone: recombine your data with credit reports, consumer-marketing data, or transaction records to build proxies (Aviva's lifestyle-data model standing in for blood and urine tests).
7. 7Build extensibility in from the start—store data in forms that let you test new hypotheses at many levels of granularity later, rather than pre-filtering to a single question.
8. 8Feed accumulating temporal data back into your models continuously; systems improve as more data flows in, so keep gathering over time rather than freezing a one-off sample.

Oren Etzioni built Farecast by reusing airfare data whose primary purpose was completed the moment a ticket sold or a plane landed. He first trained predictions on a 12,000-point sample, which performed well. But as he added more data, prediction quality improved, and eventually Farecast used the domestic flight records for most routes for an entire year. "This is temporal data—you just keep gathering it over time, and as you do, you get more and more insight into the patterns," Etzioni says. The recombination of comprehensive historical fare records let the system tell travelers when to buy without any theory of why airfares move—predicting savings of, on average, $50 per ticket. The point is that stale transaction data, kept and recombined at scale, became the good itself.

• If the phenomenon's interesting cases are outliers or anomalies (fraud, match fixing), then use all the data, not a sample—anomalies only surface against the full mass of normal transactions, as with Xoom spotting the New Jersey Discover Card pattern.
• If a task requires a human action anyway, then design it to capture reusable data as a byproduct (ReCaptcha), rather than paying separately for that data later.
• If you can answer the operational question with correlation ('what'), then don't wait to establish causality ('why')—knowing the best time to buy a ticket is good enough without understanding airfare madness.
• If a cheaper data-driven proxy predicts the target outcome reliably, then substitute it for the expensive direct measurement (Aviva's ~$5 data model versus ~$125 lab tests), accepting some messiness.
• If storage and processing are affordable, then keep the full granular dataset instead of pre-aggregating, because you cannot re-focus on detail you discarded at collection.

• Treating data as spent after its primary use—the tell: you delete or archive transaction data once the deal closes, foreclosing all secondary and recombination value.
• Over-sampling out of habit when N=all is now feasible; the tell is you report only a normal distribution and miss the subgroups and outliers where the real signal lives (flu in a city, not just a nation).
• Pre-filtering data to a single hypothesis at collection time, destroying extensibility—the tell: when a new question arises you must re-collect from scratch because the granularity is gone.
• Demanding micro-level exactitude when the goal is macro insight; the tell is you reject messy or lower-quality feeds that would still give you a reliable general direction (a phone accelerometer only slightly less accurate than medical equipment).
• Ignoring data exhaust—failing to datafy the actions and objects around your process, so the reusable byproduct never gets captured (no sensors, no interaction logs), and the latent option value is never released.

• ▸Primary use: state the original purpose this data was collected to serve.
• ▸Completion trigger: name the moment the primary purpose ends (e.g., transaction closes, plane lands).
• ▸Granularity held: record the finest level of detail you currently store and whether it's N=all or a sample.
• ▸Candidate secondary uses: list new questions this data could answer beyond its original purpose.
• ▸Recombination partners: name external datasets that, merged with yours, create new proxies or insight.
• ▸Data exhaust sources: identify byproduct signals (interactions, sensor readings) not yet captured.
• ▸Dual-purpose redesign: describe how one existing task could harvest reusable data as a byproduct.
• ▸Exactitude tolerance: state where messy/good-enough data is acceptable to gain scale or lower cost.
• ▸Option value estimate: note the potential future value or cost saved (e.g., $5 vs $125 per test) that justifies retention.

1. 1Frame the decision as a prediction, not an explanation: ask only "what is likely to happen" (will this price rise, is this email spam) rather than "why"—Etzioni concluded he didn't need to decrypt the reasons behind airline price differences, only predict direction of change.
2. 2Assemble the largest available dataset for the phenomenon; volume is what makes the model perform. Etzioni started with 12,000 scraped price observations, then obtained an industry reservation database to crunch nearly 200 billion flight-price records covering every seat on most U.S. routes over a year.
3. 3Search broadly for correlations rather than pre-selecting variables you believe matter. Google tested 450 million mathematical models against 50 million search terms to find the 45-term combination that tracked CDC flu data.
4. 4Validate predictions against a known ground-truth series before deploying. Google compared model predictions against actual CDC flu cases from 2007 and 2008; Etzioni tested his model on simulated passengers to confirm it produced savings.
5. 5Score and expose your confidence level so the human decider knows how much weight to give the prediction—Farecast presented its own confidence rating to users alongside the buy/wait recommendation.
6. 6Build the system to improve itself as more data arrives, keeping tabs on which signals and patterns work best, the way spam filters and autocorrect adapt to new inputs.
7. 7Position the prediction against the incumbent judgment method and win on timeliness or scale: Google's near-real-time flu signal beat CDC statistics that lagged a week or two, proving decisive during the 2009 H1N1 crisis.

Oren Etzioni wanted to tell travelers whether a displayed airfare was a good deal. Instead of modeling the airlines' hidden pricing logic—unsold seats, seasonality, Saturday-night-stay rules—he reframed the problem as pure prediction: will this price go up or down? He scraped 12,000 price observations from a travel site over 41 days and built a model that analyzed prices paid relative to days before departure: if the average trended down, wait; if up, buy now. The model "had no understanding of why, only what," yet it handed simulated passengers real savings. He then scaled it by acquiring a flight reservation database, growing to nearly 200 billion records, and launched it as Farecast—which even scored its own confidence and showed users that number. The correlation-only approach beat the intuition of booking early, and saved consumers a bundle.

• If the prediction reliably tells you what will happen, then act on it even when you cannot explain why—correlation without causation is sufficient for the decision.
• If you have a validated ground-truth series (CDC cases, historical sales) to test against, then trust the model's out-of-sample accuracy over expert gut instinct.
• If the model's timeliness beats the incumbent method's reporting lag, then switch the operational decision to the model even if the underlying accuracy is comparable.
• If confidence in a given prediction is low, then surface that uncertainty to the human decider rather than presenting a single recommendation as fact.
• If a subject-matter expert's judgment conflicts with a robust big-data analysis, then the expert must contend with what the data says—as baseball scouts had to yield to statisticians in Moneyball.

• Insisting on causal explanation before acting—waiting to "decrypt the rhyme or reason" delays or blocks a decision the correlation already supports. Tell: repeated demands for "but why does this work?" before deployment.
• Under-feeding the model. Others tried predicting flu from search terms and failed because they lacked Google's data, processing power, and statistical know-how. Tell: predictions perform in demo but degrade in production.
• Hand-picking variables you assume matter instead of letting the search find the signal—Google didn't assume which of 50 million terms mattered; it tested them all. Tell: a short, intuition-chosen feature list.
• Skipping validation against real outcomes and shipping a model that fits noise. Tell: no held-out comparison to CDC-style ground truth before launch.
• Treating the deferral to a "black box" as harmless—when a doctor cannot justify an intervention or a loan is denied by algorithm, the shift from privacy to probability raises free-will and accountability problems. Tell: no path to explain or contest a prediction-driven decision affecting an individual.

• ▸Decision being made: state the operational or managerial call in one line.
• ▸Prediction target: the specific 'what will happen' outcome, not the 'why'.
• ▸Ground-truth source: the real series you will validate predictions against.
• ▸Data assembled: source, size, and how obtained (e.g., scraped, licensed database).
• ▸Model search scope: how many signals/models tested and how the winning combination was selected.
• ▸Validation result: measured accuracy or savings on out-of-sample data.
• ▸Confidence score: how you rate and display uncertainty per prediction.
• ▸Timeliness edge: how much faster this beats the incumbent judgment method.
• ▸Human override rule: when individual volition or expert judgment may trump the model.

1. 1Identify data you already generate for one purpose but discard afterward — the exhaust that has a latent secondary value, as von Ahn saw in millions of daily Captcha keystrokes.
2. 2Reframe the data as raw material, not a byproduct: assume its usefulness does not end when the original purpose is served (the plane lands, the query is processed, the user proves they are human).
3. 3Confirm you can operate at the scale where new value appears — big data lets you do at large scale what is impossible at a small one; Farecast and Google Flu both needed hundreds of billions of data points.
4. 4Abandon the search for causes and hunt for correlations instead: predict what will happen, not why. Etzioni did not decrypt airline pricing logic; he predicted whether a price would rise or fall.
5. 5Build a model that learns as more data feeds in — Google's system found 45 search terms correlating with flu; a spam filter adapts as junk changes rather than being hand-coded.
6. 6Design a secondary use into the primary transaction so value compounds: ReCaptcha proves humanity AND digitizes text in the same keystroke.
7. 7Measure the value you deliver in concrete units — dollars saved per ticket, real-time lead over official statistics — so the economic gain is demonstrable, not asserted.

Oren Etzioni bought an early plane ticket believing early meant cheaper, then discovered seatmates who booked later had paid less. Rather than decrypt why airlines price the way they do, he reframed the problem as prediction: forecast whether a given price will rise or fall. He gathered flight-price data and built a model that, by 2012 running inside Bing, made the correct call 75 percent of the time and saved travelers on average $50 per ticket. The airline data existed for the airlines' own purposes; Etzioni reused it to create new value and shift economic power to consumers. Farecast could not have been built five or ten years earlier — the required computing power and storage were too expensive — but the deeper change was in mindset: treating data as a vital economic input rather than something stale once its first purpose was met.

• If the data still exists after its first purpose is served, then treat it as a reusable asset and look for a secondary application rather than discarding it.
• If you only need to know what will happen and not why, then optimize for correlation and prediction — decrypting causation wastes effort Etzioni deliberately skipped.
• If the insight only emerges above a certain data volume, then invest to reach that scale, because water-strider-versus-gravity: the operative law changes with size.
• If you can bolt a secondary value onto the primary interaction at no extra user cost, then do it — that is how ReCaptcha turned wasted keystrokes into digitized books.
• If reduced sampling error lets you tolerate more measurement error, then accept messier data in exchange for using all of it and seeing granular submarkets.

• Treating data as static and disposable — the tell is that your data is deleted or archived unused the moment its first job is done, foreclosing reuse value.
• Chasing causation when correlation suffices — the tell is a stalled project trying to explain 'why the price varies' instead of shipping a working prediction.
• Operating below the scale threshold — the tell is a model that fails to find stable patterns; Farecast and Flu Trends only worked on hundreds of billions of data points.
• Over-trusting a model that overfits transient signals — the tell is Google Flu Trends overstating 2012 cases, likely inflated by media attention rather than real infections.
• Believing technology alone creates the value — the tell is buying processing tools (MapReduce, Hadoop) without the mindset shift that data is an economic input, leaving datasets collected but never re-listened to.

• ▸Primary purpose: what the data was originally collected to do.
• ▸Discard point: the moment its usefulness currently 'ends' (plane lands, query processed).
• ▸Secondary use candidate: a new good, service, or prediction the same data could power.
• ▸Prediction target: the specific 'what' you will forecast, not the 'why.'
• ▸Scale required: the data volume needed before the pattern becomes reliable.
• ▸Correlation signal: the terms, attributes, or features that track your target (e.g. 45 flu search terms).
• ▸Value unit: the concrete gain delivered ($ saved, hours ahead of official data).
• ▸Bolt-on mechanism: how a secondary value can ride inside the primary interaction at no extra user cost.
• ▸Accuracy check: how you confirm the model stays right as new data feeds in (e.g. same fuzzy word shown to five people).

1. 1Datafy the phenomenon first: take the thing you want to predict—engine health, flu spread, ticket price, pregnancy stage—and convert its observable signals into quantified data (heat, vibration, stress, sound, search terms, purchase patterns). You cannot correlate what you have not measured.
2. 2Collect at scale rather than sampling. Get the full dataset if you can—Etzioni moved from 12,000 scraped observations to an industry reservation database covering every seat on every flight for a year. More comprehensive data lets you break into subgroups without the erroneous-prediction risk that plagues random samples.
3. 3Choose a target variable and gather a labeled ground-truth set: actual CDC flu cases, actual airfare movements, actual breakdowns. Correlation is validated against known outcomes, not asserted.
4. 4Run large-scale correlational search to find which variables (or combination) track the target. Google tested search terms until 45 of them, used together, correlated strongly with CDC figures. Do not pre-limit yourself to a few hypotheses—let the data surface the proxy.
5. 5Identify a good proxy and watch it. The aim is to find the telltale pattern that reliably precedes the event—the whirring motor before failure, the search-term spike before flu—then monitor only that.
6. 6Accept 'what' over 'why.' Etzioni's model knew nothing of seat inventory or Saturday-night stays; it predicted on probabilities from other flights. Do not stall waiting for a causal explanation.
7. 7Score and expose your own confidence. Farecast presented users a confidence rating on each prediction; publish the uncertainty so decisions can be weighted.
8. 8Compare cost against the exact method. If the proxy is materially cheaper (Aviva's data approach at ~$5 vs $125 lab tests; Equifax proxies under $1), the proxy wins on cost-effectiveness even at lower precision.

Google Flu Trends is the cleanest case in the passages. The team compared search-query data against actual CDC flu cases from 2007 and 2008, searching for terms that tracked the official figures. Their software found a combination of 45 search terms that, used together in a mathematical model, correlated strongly with the CDC's nationwide numbers. The advantage was timing: the CDC reported a week or two after the fact, while Google could indicate flu spread in near real time. When H1N1 hit in 2009, Google's system beat government statistics precisely because it dodged their reporting lag. Notably, the method used no mouth swabs and no physician contact—only correlation between existing search data and known outcomes, a pure 'what, not why' prediction.

• If the exact measurement is expensive or unpleasant and a cheaper proxy predicts the target reliably, use the proxy—Aviva swaps $125 blood/urine tests for ~$5 lifestyle data.
• If the average price/signal trends up as the event nears, act now; if it trends down, wait—Etzioni's rule for buy-now vs buy-later on airfare.
• If you need to break predictions into small subgroups, do not rely on random sampling—its accuracy degrades as subgroups shrink; collect the fuller dataset instead.
• If knowing the outcome saves money or lives, treat the missing causal explanation as acceptable—the aspirin-and-orange-juice remission case: that they lived matters more than why.
• If timing beats a lagging authoritative source, the near-real-time proxy is more useful even when slightly less exact—Google beat the CDC on H1N1.

• Assuming a dataset has one fixed purpose. The tell: analysts 'steeped in the analog paradigm' who tie a dataset's value to its original use and never mine its latent, dormant value.
• Demanding causal understanding before acting. The tell: paralysis while you hunt for why prices move, when a correlational 'what' would already save money.
• Chasing micro-level exactitude on a big dataset. The tell: obsessing over penny/inch precision when the payoff is macro-level insight—you lose the general direction you actually need.
• Trusting a biased sample. The tell: systematic collection bias like landline-only polling that undercounts younger, more liberal cell-phone users—producing wrong election predictions off by 1–3 points in the 2008 Obama/McCain race.
• Ignoring the chilling and 'creepy' side effect of proxy prediction. The tell: people alter behavior (avoiding extreme-sports sites, certain shows) once they sense their clicks feed premiums—your proxy degrades as subjects react to it, as with Target's pregnancy-prediction backlash.

• ▸Target event: the outcome you want to forecast (e.g., breakdown, flu, price move)
• ▸Datafication source: what observable signal you will quantify (heat, vibration, search terms, purchases)
• ▸Ground-truth set: the known-outcome data you validate against (e.g., CDC cases, actual failures)
• ▸Candidate proxies: variables/patterns you will test for correlation
• ▸Scale of data: full dataset or sample—note subgroup needs before choosing
• ▸Selected proxy + pattern: the telltale sign you will monitor going forward
• ▸Confidence score method: how you rate and display prediction certainty
• ▸Cost comparison: proxy cost vs exact-measurement cost per prediction
• ▸Behavioral/bias risk: how subjects might react or how collection might skew results

1. 1Map the full data lifecycle for the personal data you hold: what you collect, what secondary uses you reserve, and how long you retain it — because scale and reuse, not just collection, drive erosion. Treat 'store forever' as the default failure to correct.
2. 2Audit for recombination risk: identify every dataset that, when commingled with another, could re-identify individuals. The passages show dormant value is 'unleashed by combining one dataset with another' — the same mechanism re-identifies people.
3. 3Stop relying on notice-and-consent and opt-out as your privacy defense. The passages call existing privacy law a 'largely useless Maginot Line' because people willingly share data as a core feature; treat consent as necessary but not sufficient.
4. 4Distinguish privacy harm (surveillance, re-identification) from propensity harm (penalties based on predictions). Flag any use where a big-data prediction judges or punishes a person before they act — heart-attack likelihood raising insurance, mortgage default denying a loan, crime prediction triggering pre-emptive scrutiny.
5. 5Build accountability into the algorithm itself, using the three concrete safeguards named in the passages: transparency (make the data and algorithm inspectable), certification (have a third-party expert certify sensitive uses as sound and valid), and disprovability (give people concrete ways to disprove a prediction about themselves).
6. 6Groom and cull data actively rather than hoarding it. Because most data loses utility over time and old data can even destroy the value of fresh data, set a rule that data is used only as long as it stays productive — which also shrinks the surveillance surface.
7. 7Guard human agency and individual responsibility explicitly: require that decisions about people be tied to actual behavior, not to risk categories, so predictions cannot become punishment without confrontation.

Telefonica illustrates the tension precisely. As a mobile operator it already holds records of where and when phones connect to base stations and at what signal strength — data first collected purely to fine-tune network performance. That same banal operational data can be recombined to reveal where people gather on a Friday night, real estate values, or billboard prices. In 2012 Telefonica moved to monetize it by spinning up a separate company, Telefonica Digital Insights, to sell subscriber-location data to retailers. Their stated safeguard was to sell it only 'anonymous and aggregated' — but the passages elsewhere flag that anonymization fails at scale, so aggregation is a claim to test, not a guarantee. The erosion here is not the original collection; it is the secondary reuse of location traces that can, when combined with other datasets, resolve back to individuals.

• If a dataset was collected for one purpose and you now propose a secondary use, then re-check re-identification risk against all other data you hold — do not assume the original consent or anonymization still covers it.
• If you claim data is 'anonymous and aggregated,' then require someone to attempt re-identification by recombination before you treat the claim as true.
• If an algorithm's output will judge or penalize a person (insurance price, loan denial, added scrutiny), then require transparency, third-party certification, and a disprovability path before deploying it.
• If a piece of data has stopped improving predictions (e.g., a decade-old purchase no longer reflects the person), then cull it — retaining it both erodes accuracy and needlessly extends surveillance.
• If a decision is being framed as 'objective risk management' rather than a response to actual behavior, then stop: you are drifting toward penalties on propensity that negate fairness and free will.

• Treating consent as sufficient. Tell: your privacy program leans on notice-and-opt-out while users 'willingly share' as a feature — the Maginot Line that big-data reuse walks around.
• Confusing collection limits with use limits. Tell: criticism and controls focus on 'what information is capable of being collected' rather than what is actually done with it, the way Facebook's critics fixated on capability.
• Hoarding data because storage is cheap. Tell: you retain records 'as long as economically possible' and never cull, degrading both accuracy and privacy — Amazon's decade-old book purchase that poisons fresh recommendations.
• Trusting 'anonymous and aggregated' as a final state. Tell: no one on the team has attempted re-identification by commingling the dataset with another before release.
• Sliding from prediction to pre-emptive punishment. Tell: individuals are subjected to special treatment (monthly social-worker visits, extra scrutiny) for acts that have not happened and feel stigmatized without ever being held responsible for real behavior.

• ▸Dataset & original purpose: name the data and why it was first collected.
• ▸Proposed secondary use: state the new purpose and who benefits.
• ▸Recombination risk: list other datasets that, if combined, could re-identify individuals.
• ▸Anonymization test result: record who attempted re-identification and whether it held.
• ▸Prediction/penalty check: does output judge or penalize a person before they act? (yes/no + description)
• ▸Safeguards in place: transparency, certification, disprovability — mark which are implemented.
• ▸Retention limit: state the date or condition at which this data is culled as no longer productive.
• ▸Human-agency guard: describe how the affected person can contest or disprove the outcome.

1. 1Separate the prediction from the person: before any predictive score touches a decision, ask whether it describes what someone did or what a model expects them to do. Only completed acts qualify for judgment.
2. 2Restrict predictive analytics to the investigative front-end, not the verdict. Use big-data correlation to flag where to look (which of nine suspected firms may be colluding), then require independent evidence of actual conduct before assigning responsibility.
3. 3Refuse solitary reliance on big data for state judgments. When the state judges past actions, prohibit any decision resting solely on statistical analysis; require corroborating, non-predictive proof.
4. 4Preserve the presumption of innocence as a hard floor: treat a high-risk propensity score as a research signal, never as a finding of culpability, because the model knows 'what,' not 'why.'
5. 5Build accountability into the data user, not just consent from the data subject. Assign a named party responsible for how predictions are applied to individuals, so someone answers for punishing propensity.
6. 6Document that the model has no understanding of causation. Record explicitly that a prediction is a correlation over past data points and cannot establish intent, choice, or agency.

Regulators suspect nine companies of price fixing and run a big-data analysis across their pricing behavior. The correct use: the analysis identifies possible collusion so regulators know where to investigate. It is entirely acceptable to point the inquiry using these correlations. What is not acceptable is convicting a firm because the model predicts it would collude, or because its data pattern resembles that of colluders. The state must gather evidence of the actual price-fixing acts and judge those acts. Even then, it cannot rely solely on the big-data output; the prediction narrows the search but the punishment attaches to demonstrated conduct. This mirrors how a court holds people responsible for what they have done, not for their propensities.

• If the input is a prediction of future behavior, then it may guide investigation but must never ground punishment.
• If the subject has actually performed an act, then judge the act with corroborating evidence, not the correlation that flagged it.
• If a decision rests solely on big-data analysis, then reject it; require independent, non-statistical proof before the state judges past actions.
• If a model can report 'what' but not 'why,' then treat its output as a signal of association, never as proof of intent or choice.
• If a score would deny someone opportunity based on the company they keep (a social-graph credit score), then recognize it punishes propensity by association and demand actual behavioral grounds.

• Treating an accurate prediction as a finding of guilt. The tell: the case file cites a risk score or correlation as the reason for the sanction, with no completed act described.
• Letting propensity-by-association stand in for conduct — e.g., a Facebook social-graph credit score inferring you are profligate because your friends are. The tell: the adverse decision references who someone is connected to rather than what they did.
• Confusing correlation with causation and asserting intent. The tell: an analyst explains 'why' a person will offend when the model only produced probabilities from other cases.
• Collecting everything first, judging later — the surveillance-state reflex of storing 20 trillion transactions because it's cheap. The tell: predictions are applied to individuals who were never suspected of any act.
• Eliminating human review because the model is confident. The tell: an automated propensity score triggers punishment with no accountable person answering for its application to that individual.

• ▸Subject and decision at stake: name the individual and the concrete consequence being considered.
• ▸Basis type: mark whether the input is a completed ACT or a PREDICTED future behavior.
• ▸Completed-act evidence: list the specific past actions and their non-statistical proof.
• ▸Sole-reliance check: state what corroborating evidence exists beyond the big-data output.
• ▸Causation honesty: note that the model shows 'what' not 'why' — record it cannot establish intent.
• ▸Association test: confirm the decision is not driven by whom the subject is connected to.
• ▸Investigation-only boundary: confirm predictive scores are used to direct inquiry, not to render the verdict.
• ▸Accountable owner: name the person responsible for how this prediction is applied.
• ▸Presumption-of-innocence sign-off: certify the subject is not being punished for a propensity.

1. 1Name the metric you rely on and ask directly: what does this number claim to capture, and does it actually capture that? McNamara's body count claimed to measure war progress but captured neither enemy strength nor political will.
2. 2Separate what the data tells you (correlation, the 'what') from why it happens (causality, the 'why'). McGregor's preemie system flags infection risk 24 hours early via correlation—so treat its output as an alert to act, never as an explanation to defend as proven cause.
3. 3Audit the underlying data for four defects before trusting output: is it poor quality, biased, mis-analyzed/used misleadingly, or does it fail to capture what it purports to quantify? Reject the reflex to fix a problem by 'getting more data' until you clear these.
4. 4Check whether the number invites you to abandon judgment. If you find yourself deferring to output despite reasonable grounds to suspect something is amiss, you are being governed by the data rather than using it.
5. 5When a prediction affects a person, attach three safeguards: openness (make the data and algorithm available), certification (have an expert third party validate soundness), and disprovability (specify concrete ways the person can disprove the prediction about themselves).
6. 6Preserve human agency and individual responsibility as an override: individual volition must be able to trump the statistics, so decisions rest on actual behavior rather than assessed propensities.
7. 7Distinguish measurement from prediction-as-verdict: never let a predicted likelihood substitute for confronting and holding someone accountable for what they actually did—that is the slide toward penalties based on propensities.

Robert McNamara ran the Vietnam War as a numbers guy, trained at Harvard Business School and Statistical Control, believing Truth could be found in data and a complex situation tamed by quantification. Among the figures reported back to him was the body count, which he and the men around him relied on and fetishized—he felt he could comprehend the ground only by staring at spreadsheet rows and columns. But the underlying data was corrupted: subordinates told the higher-ups what they wanted to hear, so the figures failed to capture what they purported to quantify. The number looked like progress, but the metric governed the decision-makers rather than informing them. The correct move was to interrogate whether body count actually measured winning, and to weight field intuition against a suspect figure. Instead, mindless deference to the output turned a measurement into a mandate—the pattern any big-data era must heed.

• If the output tells you 'what' but not 'why,' then act on it as a triggered alert (like earlier, lighter intervention for preemies), but do not present it as proven cause or use it to assign culpability.
• If you suspect on reasonable grounds that something is amiss yet keep deferring to the number, then stop—you are under a dictatorship of data, not using data.
• If your first instinct to a hard problem is 'collect more data,' then first test whether the existing data even captures the thing you care about (as standardized tests may not capture actual student ability).
• If a prediction would judge or penalize a person before they act, then require openness, certification, and disprovability, and let individual volition override the statistic.
• If you cannot explain a decision except by asking someone to defer to a black box, then the burden shifts to you to justify why that deference is warranted, not to them to accept it.

• Fetishizing the output: treating figures as one standard deviation closer to God. Tell: you defend the number even when frontline reality contradicts it.
• Attributing undeserved truth to data. Tell: 'In God we trust—all others bring data' becomes an excuse to end the inquiry rather than begin it.
• Collecting data for its own sake. Tell: watch lists, no-fly lists, or standardized tests multiply without anyone asking whether they deliver the protection or measurement promised.
• Converting correlation into causal verdict. Tell: a predictive flag ('likely to commit a felony') triggers stigmatizing intervention, punishing an action that has not happened.
• Trusting biased or gamed source data. Tell: reporters feed you what you want to hear (McNamara's subordinates), and constant-looking vital signs read as 'safe' when they are the calm before the storm.

• ▸Metric in use: name the single number you are relying on.
• ▸Claimed capture: what does this metric purport to quantify?
• ▸Actual capture gap: where might it fail to capture that, be biased, or be gamed?
• ▸What vs. why: is this correlation (what) or established cause (why)?
• ▸Contradiction check: what field/intuitive evidence suggests something is amiss?
• ▸Person impact: does this output judge or penalize an individual?
• ▸Safeguards attached: openness / certification / disprovability — yes or no for each.
• ▸Override clause: how can individual volition or actual behavior trump this figure?
• ▸More-data reflex: are you adding data instead of testing whether current data is valid?

1. 1Replace notice-and-consent with a use-assessment gate: before any secondary use of personal data, run a formal assessment of the impact on the individuals whose information is processed, sized to the risk of the use.
2. 2Sort planned uses into risk tiers—permissible without safeguards, permissible with limited standardized safeguards, and riskier initiatives requiring regulator-set ground rules for how the data user assesses dangers and mitigates harm.
3. 3Give accountability teeth: attach legal liability, mandates, fines, and possibly criminal prosecution to sloppy assessments or poor implementation, so accurate assessment becomes the cheaper path.
4. 4Where numbers must be published, fuzz outputs deliberately—report approximate figures (differential privacy) so results can't be narrowed to any individual.
5. 5Enshrine human agency: judge people only on past actions, never on statistical predictions of future ones; forbid the state from convicting on big-data probability alone, using it only to open investigations pursued by traditional means.
6. 6For significant private-sector decisions driven by prediction (hire/fire, mortgage, credit), require three safeguards: openness (disclose the data and algorithm behind the prediction affecting the individual), certification (an expert third party validates the algorithm as sound for sensitive uses), and further recourse mechanisms.
7. 7Create algorithmists—a new caste of big-data auditors—to inspect and interpret analytics so complex predictions do not become an unaccountable black box.
8. 8Apply antitrust-style regulation: identify overarching principles, then muscular, technology-neutral rules that protect competition without favoring one technology, and require governments to release their own data publicly.

Take the car antitheft service that datafies drivers' sitting posture as a unique identifier. The firm later reanalyzes that posture data to predict drivers' attention states—drowsy, tipsy, angry—and sends alerts to nearby drivers to prevent accidents. Under notice-and-consent, the firm believes it must run a fresh round of notification and consent for this unimagined secondary use, which is impractical. Under the accountability framework, the firm instead runs a formal big-data use assessment weighing the impact on the drivers whose posture is processed and implements standardized safeguards for the risk tier. If the assessment is done correctly and the safeguards implemented accurately, the firm may proceed without returning to each individual for consent. If it does the assessment sloppily, it faces liability, fines, and regulatory mandates—accountability with teeth.

• If a use was unimagined at collection and its value lies in secondary reuse, then abandon consent and require a use-assessment holding the data user accountable for impact.
• If a planned use is riskier, then apply regulator-set ground rules for danger assessment and mitigation; if low-risk, then permit it with limited or no standardized safeguards.
• If a decision would judge a person on predicted future behavior rather than actual past acts, then reject it—human volition is inviolable and guilt requires real actions.
• If big data flags probable wrongdoing (e.g., price fixing by suspect firms), then use it only to trigger investigation and build a case by traditional means—never as the sole basis for a guilty finding.
• If a business bases a significant decision (hire, fire, mortgage, credit) mostly on prediction, then require openness of data and algorithm plus third-party certification before it stands.

• Refreshing old rules instead of replacing them—the tell: you're still asking for consent at collection while the real value sits in unforeseen reuse.
• Accountability without enforcement—the tell: assessments exist on paper but there are no fines, mandates, or liability, so data users skip rigor.
• Drifting from acts to propensities—the tell: interventions framed as 'reducing risk' start targeting people for what they might do, devaluing individual responsibility (the predictive state as nanny state).
• Deploying opaque prediction models with no one able to trace the basis of a decision—the tell: like Google Translate versus the 1954 rule-based translator, no human can explain why the algorithm decided as it did, and no algorithmist reviews it.
• Over-fuzzing or under-fuzzing published data—the tell: either the reported approximation destroys the insight, or it's precise enough to narrow down to a specific person (e.g., pinning an exact count on Asian women in Atlanta doing Ashtanga yoga).

• ▸Planned use: describe the secondary use and whether it was foreseen at collection.
• ▸Individuals affected: identify whose personal information is processed and how.
• ▸Risk tier: classify as no-safeguard / limited-standardized / riskier requiring regulator ground rules.
• ▸Impact analysis: state potential harms to individuals and how each is avoided or mitigated.
• ▸Agency check: confirm the use judges past actions, not predicted future behavior.
• ▸Openness: record what data and algorithm can be disclosed to affected individuals.
• ▸Certification: name the expert third party validating the algorithm for sensitive uses.
• ▸Fuzzing plan: specify how outputs are approximated so no individual is identifiable.
• ▸Liability owner: name the accountable party exposed to fines/mandates if implementation fails.

1. 1Define the phenomenon narrowly enough that a comprehensive dataset is attainable — e.g. 'ticket prices for a given route relative to days before departure,' not 'airline pricing in general.'
2. 2Ask whether you need the 'why' or only the 'what.' If prediction suffices, drop the search for causal variables and commit to using every available observation instead of a curated sample.
3. 3Acquire the fullest dataset you can: start by scraping or pulling from public sources, then pursue the authoritative source database once the concept is proven (Etzioni scraped a travel site first, then obtained an industry reservation database).
4. 4Accept messiness as the price of scale: with more data you trade micro-level exactitude for macro-level insight, so do not stall trying to clean or perfect every record.
5. 5Look deliberately at the granular — subcategories, submarkets, and outliers — because these are exactly what a sample cannot assess and where N=all pays off.
6. 6Score and expose your own confidence. Compute a probability or direction for each prediction and present the confidence level alongside it, as Farecast did.
7. 7Feed performance back: as more comprehensive data arrives (the full reservation database), retrain the model to improve accuracy rather than freezing on the initial sample.

Oren Etzioni wanted to tell travelers whether an airfare shown online would rise or fall. Rather than decrypt the airlines' hidden pricing logic, he reframed the problem as pure prediction: analyze all ticket sales for a given route against the number of days before departure, then recommend 'buy now' if prices tend to rise and 'wait' if they tend to fall. He started with a sample of 12,000 price observations scraped from a travel site over 41 days and built a model that saved simulated passengers money — a model that knew what, never why, ignorant of unsold-seat counts, seasonality, or Saturday-night-stay rules. Named Hamlet ('to buy or not to buy'), it became the startup Farecast, which also scored and displayed its own confidence in each prediction. To push performance further, Etzioni moved from the scraped sample toward one of the industry's full flight reservation databases. The progression — sample to near-comprehensive dataset — is the operational shape of N=all.

• If you only need to predict what will happen, then abandon the hunt for causal variables and let correlations in the full dataset carry the decision.
• If your insight depends on subgroups, submarkets, or outliers, then use all the data — a sample structurally cannot reveal them.
• If a comprehensive dataset is available, then accept its messiness rather than shrinking to a small, exact, curated set; gain at the macro level outweighs micro error.
• If a decision is binary and consequential at the record level (enough money to clear a check), then keep exactitude — N=all does not license sloppiness where precision is the whole point.
• If you can score your prediction's confidence, then publish that score so users can weigh it, upholding transparency against itself.

• Defaulting to a sample out of habit — the tell is designing the study around 'the smallest amount of data to confirm the finding,' a statistical reflex from the era of scarcity that ignores now-available comprehensive data.
• Demanding causal explanation before acting — the tell is stalling to 'decrypt the rhyme or reason' when a what-only correlation would already answer the business question.
• Over-cleaning to preserve exactitude — the tell is treating messy, distributed, uneven-quality data as unusable rather than accepting general-direction accuracy.
• Averaging away the granular — the tell is reporting only aggregate figures and never inspecting subcategories or outliers, forfeiting the one advantage N=all offers over a sample.
• Hiding uncertainty — the tell is presenting predictions as flat facts with no confidence score, so users can't tell a strong call from a weak one.
• Letting comprehensive data lie fallow — the tell is holding rich data (Amazon's Kindle annotation data) but only exploiting a narrow primary use, never mining the fuller picture.

• ▸Phenomenon: state the narrow question in one line (e.g. 'will this route's fare rise before departure?').
• ▸Need type: circle 'why' (causal) or 'what' (predictive) — predictive favors N=all.
• ▸Data source (start): the scrapeable/public source and its size (e.g. 12,000 obs, 41 days).
• ▸Data source (full): the authoritative comprehensive dataset to pursue next.
• ▸Granular targets: the subgroups/outliers you expect only N=all to reveal.
• ▸Messiness tolerance: what micro-level error you'll accept for macro insight.
• ▸Signal variable: the correlation you'll act on (e.g. price vs. days-to-departure).
• ▸Confidence output: how you'll score and display prediction certainty.
• ▸Feedback loop: how new comprehensive data will retrain and improve the model.

1. 1State the tradeoff explicitly before collecting: what accuracy are you sacrificing, and what breadth or frequency do you gain in return? If you can't name the return, don't accept the messiness.
2. 2Decide whether you're sacrificing accuracy for breadth (more data points, e.g. one sensor per vine) or exactitude for frequency (more readings per unit time). The two trades introduce different errors—wrong readings versus out-of-sequence records.
3. 3Substitute many cheap instruments for one precise one, but check that cheap instruments introduce no systematic bias—random error washes out in the aggregate, systematic bias does not.
4. 4Combine sources that don't align perfectly (e.g. voice-recognition on call complaints against operator handling time) and accept the imperfect snapshot rather than waiting for clean, matched data.
5. 5Use tagging and free-form entry instead of predefined taxonomies; let anyone add labels by typing rather than forcing content into fixed rows, fields, and predefined indexes.
6. 6Set an explicit 'good enough' threshold—Forrester's '2 plus 2 can equal 3.9'—so you know the general trend without demanding the exact figure. Confirm the data isn't completely incorrect, only imprecise.
7. 7Run a cost check: painstaking accuracy often fails the economic test because greater data volume delivers more value than the error prevented. Spend on resources to fix errors only where sequence or precision genuinely matters (as in 30,000 NYSE trades/second).
8. 8Display and report results probabilistically at scale—show '4K' not the exact count, '2 hours ago' not the timestamp—so precision effort tracks the point where exactness stops adding value.

The U.S. Bureau of Labor Statistics produces the CPI by employing hundreds of staff to call, fax, and visit stores in 90 cities, collecting about 80,000 prices monthly at a cost near $250 million—neat, clean, orderly data that arrives weeks stale. In 2008 that lag was dangerous: decision-makers, including the Federal Reserve setting interest rates, needed inflation numbers fast and couldn't get them from a method that prizes precision and sampling. MIT economists Alberto Cavallo and Roberto Rigobon built the Billion Prices Project as a big-data alternative by steering a much messier course, using software to gather prices off the web. The data is less curated and noisier than 80,000 hand-checked prices, but it arrives in near real time and in vastly greater volume. The trade is explicit: they gave up the sterile exactitude of the official index for speed and scale. The result is a messier dataset that answers a time-sensitive question the precise method structurally cannot.

• If the value of more data volume exceeds the cost of preventing errors, then tolerate the error rather than fix it.
• If you can add many more data points at the cost of per-reading accuracy, then take breadth—the aggregate gives a more comprehensive picture even when individual readings are wrong.
• If cheap instruments introduce only random error, then use many of them; if they introduce systematic bias, then don't—messiness that skews in one direction corrupts the aggregate.
• If the figures are large or changing fast, then display an approximation, because an exact number is out of date the moment it appears.
• If the questions you'll ask aren't clear at the outset and the data is varied in type and quality, then reject the rigid relational schema and use flexible storage that need not sit in tidy rows.

• Pursuing 'a single version of the truth': treating consistency across teams as achievable and worth the fight, when its pursuit is itself the distraction from harnessing data at scale.
• Prizing per-point precision and missing the bigger picture—measuring minutiae to the nth degree while the macro trend goes unseen. The tell: you can defend every data point but can't state the overall direction.
• Mistaking random error for acceptable when it's actually systematic bias. The tell: the aggregate drifts consistently in one direction rather than noise canceling out.
• Forcing varied, unpredictable data into predefined fields, lengths, and indexes—so a seven-digit field can't hold 10 million and 'not available' can't be entered. The tell: you keep altering the schema to accommodate real entries.
• Throwing resources at exactitude where it doesn't pay—spending like the CPI's $250 million for clean data that arrives too late to act on. The tell: your data is pristine but stale by the time decisions must be made.

• ▸Question to answer: state the decision the data must inform and its time sensitivity.
• ▸Trade type: mark breadth-for-accuracy or frequency-for-exactitude.
• ▸What you gain: the specific detail, scale, or speed the messier approach delivers.
• ▸What you sacrifice: the exact accuracy or sequence you're giving up.
• ▸Bias check: confirm errors are random, not systematic (note any directional skew).
• ▸Good-enough threshold: the tolerance where the general trend is reliable (your '3.9').
• ▸Economic check: cost to prevent errors vs. value of the added data volume.
• ▸Storage/format fit: does data need predefined fields, or flexible tags/schema-free storage?
• ▸Display rule: where you'll show approximations instead of exact figures.

1. 1Frame the question as a prediction, not an explanation: state the outcome you want to forecast (flu spread, ticket price, stock move) and treat 'why' as out of scope. Google explicitly designed a system that 'didn't care' why people searched.
2. 2Assemble the largest available dataset already tied to the outcome rather than a hand-picked sample. Google took the 50 million most common U.S. search terms and matched them against CDC flu data for 2003–2008.
3. 3Do not pre-specify hypotheses. Before big data you tested a small number of hypotheses defined before collecting data; here you let the data speak and let candidate proxies emerge from the full set.
4. 4Search the association space by brute force: test many term combinations against actual outcomes. Google processed 450 million mathematical models to find which searches tracked flu.
5. 5Select the combination with the strongest fit to known ground truth. Google's software found 45 search terms that, used together, correlated strongly with official CDC figures nationwide.
6. 6Validate the correlation against held-out real cases before trusting it: Google compared predictions against actual flu cases from 2007 and 2008.
7. 7Deploy for what the correlation delivers that causation can't — speed and timeliness. Google could report flu location in near real time versus the CDC's one-to-two-week reporting lag.
8. 8Cap the use to prediction and alerting. Use the signal to know that something is happening and where to intervene, not to make causal claims about any individual.

Google Flu Trends is the operative case. Facing the 2009 H1N1 outbreak with no vaccine and only reporting lags to work with, Google mined its three-billion-plus daily search queries instead of swabs or physician calls. It compared 50 million common search terms against 2003–2008 CDC seasonal-flu data, ran 450 million models, and isolated 45 terms whose combined frequency correlated strongly with official flu figures. Validated against 2007–2008 actual cases, the model predicted where flu had spread in near real time. When H1N1 struck, it proved more timely than government statistics. Notably, the correlation may exist not because searchers were ill but because healthy co-workers heard sneezes and searched how to protect themselves — the model still worked as a location signal even though the causal story was uncertain.

• If you need speed and scale (real-time hotspots, price timing) and the mechanism is hard or expensive to nail down, then take the correlation and skip the 'why.'
• If you have enough data to test candidate proxies against real outcomes, then let the data surface the predictors instead of pre-defining hypotheses.
• If the correlation will be used to act on or punish an individual (quarantine, deny a loan), then stop — correlation does not imply causation and you must test the individual, not the propensity.
• If a subject-matter expert's intuition contradicts a well-validated big-data signal, then make the expert contend with what the analysis says rather than defaulting to experience.
• If micro-level exactitude adds cost but not decision-relevant insight, then accept messiness and act on the general direction.

• Treating correlation as causation for individual action: the passage's quarantine scenario rounds up searchers as 'prisoners of a prediction' who may not even have the flu — the tell is any move to punish propensities rather than tested actions.
• Overconfidence and obsession with the tool — applying it where it doesn't fit because it 'can do so much.' The tell is the McNamara curse: faith in the numbers overriding judgment.
• Skipping validation against ground truth. The tell is a model chosen for fit to training data with no comparison to held-out real cases (Google validated against 2007–2008).
• Working from a small pre-chosen hypothesis set instead of the full data, so novel connections never appear. The tell is defining your variables before collecting data.
• Chasing 'a single version of the truth' and micro-level precision at scale. The tell is a team fighting to eliminate messiness rather than accepting it as par for the course.

• ▸Outcome to predict: name the single thing you want to forecast, not explain
• ▸Ground-truth source: the existing real-record dataset your outcome is measured against (e.g., CDC figures)
• ▸Candidate proxy pool: the largest set of already-collected signals you can test (e.g., 50M search terms)
• ▸Search method: how you'll test combinations at scale (number of models / brute-force plan)
• ▸Selected proxy set: the winning combination and its correlation strength to ground truth
• ▸Validation window: the held-out real cases used to confirm the fit before deployment
• ▸Speed/scale payoff: what this delivers that causal methods can't (e.g., real-time vs. reporting lag)
• ▸Individual-action guardrail: explicit statement that this signal will not be used to judge any single person without testing
• ▸Messiness tolerance: the level of inexactitude you accept for macro-level insight

1. 1Identify a phenomenon currently treated as non-information—a person's location, posture, mood, engine vibration, or the stress on a bridge—and ask what latent value it holds if made quantifiable.
2. 2Locate an existing corpus of raw material that captures traces of that phenomenon, even material 'no one thought held any value'—old logbooks, archived price data, saved search queries, or stored tweets.
3. 3Define the physical or observable signal you can measure and specify a quantification scheme: fixed measurement points and a numeric scale (Koshimizu measures pressure at 360 seat points, each indexed 0–256).
4. 4Tabulate the readings into a structured, analyzable format so each observation becomes a comparable record—Maury extracted and tabulated the Navy's musty logbooks with pencil and paper.
5. 5Separate digitization from datafication: converting analog to computer-readable is not enough; you must render the aspect into quantified, tabulated form that supports new analysis.
6. 6Look for correlations between your datafied signal and an outcome you care about, testing many models rather than presuming why (Google tested 450 million models against CDC flu figures).
7. 7Capture the metadata, not just the primary content—geo-location, follower networks, language, timestamps—because 'information about information' often carries the richer signal (a tweet has 33 discrete metadata items).
8. 8Reuse the datafied record for purposes beyond its original collection, treating data as a raw material and standing input rather than something stale once its first job is done.

Matthew Fontaine Maury, sidelined from sea by a crippling stagecoach injury and placed at the Depot of Charts and Instruments, recognized that the Navy's old logbooks were not clutter but 'data' that could be extracted and tabulated. Sailors zigzagged across oceans because they treated the sea as unpredictable, steering only familiar courses; Maury had seen from his own voyages that winds and currents ran in patterns, like the Valparaiso gale that stopped at sundown 'as if someone had turned off a tap.' He converted information generated for one purpose—individual voyage records—into charts of winds and currents built from earlier sea voyages, the same logic UPS later used to optimize routes from data on previous deliveries. This was datafication done with pencil and paper, predating digitization entirely. The value was real: four countries knighted him, eight awarded gold medals, and U.S. Navy pilot charts still bore his name at the dawn of the twenty-first century. His case shows the core move—unearthing analyzable data from material no one thought valuable, and reusing it in ways its originators never intended.

• If the aspect can be observed but is not yet numeric or tabulated, then it is a datafication candidate—not yet done just because a computer can read it.
• If you can specify fixed measurement points and a numeric scale, then quantify directly; if not, harvest metadata as a proxy signal (moods datafied via 509 million tweets' patterns).
• If you have far more data than rivals for the same question, then favor correlation-hunting over causal theory—Google's edge was scale, processing power, and statistical know-how, not knowing why people searched.
• If data was collected for one purpose, then treat it as reusable raw material, not spent stock—Farecast reused old airline price data into a $110 million business.
• If you want to predict behavior rather than describe aggregate state, then datafy connections and networks (who follows whom), as Salathé did to detect unvaccinated subgroups.

• Conflating digitization with datafication: you make analog readable by computers but leave it as unstructured content, never rendering it quantified and tabulated. Tell: you can store and search the material but cannot run analysis across records.
• Assuming a dataset has a single purpose tied to its collection—the analog prejudice. Tell: you discard or archive data once its first job is done, as if it were stale after the plane landed.
• Insisting on knowing why before acting, so you never test enough correlations. Tell: you reject a strong predictive signal because the mechanism is unexplained (Google's system deliberately didn't care why).
• Ignoring metadata and measuring only the obvious primary content. Tell: you analyze tweet text but skip the 33 metadata items—geo-location, follower networks—that hold the richer patterns.
• Believing the tools alone create the value, crediting fast chips and software. Tell: you invest in technology but not in rendering more aspects of reality into data form, which is the deeper driver.

• ▸Target phenomenon: name the aspect of the world you want to render (location, posture, mood, behavior, vibration).
• ▸Latent value hypothesis: state the new use or prediction you expect once it is quantified.
• ▸Raw material source: identify the existing corpus of traces, even if currently considered valueless.
• ▸Measurement scheme: define the observation points and numeric scale (e.g., 360 points, 0–256).
• ▸Tabulation format: specify the record structure that makes observations comparable and analyzable.
• ▸Metadata to capture: list the 'information about information' worth harvesting (geo-location, network, timestamp, language).
• ▸Correlation target: name the outcome you will test the datafied signal against.
• ▸Reuse plan: describe purposes beyond the original collection that turn this data into a standing input.

1. 1Treat every dataset you already hold as unspent: after its primary purpose is done, ask what secondary uses remain instead of erasing it. Con Edison's century-old cable and maintenance records had latent value nobody at the firm imagined.
2. 2Estimate option value, not primary-use value: sum the choices — the possible future ways the data can be employed — rather than only counting what you gained the first time.
3. 3Move from sampling toward N=all wherever tools allow. Start with a sample if you must (Farecast's initial 12,000 points worked), then keep adding data because granularity improves and predictions sharpen down to the subgroup level.
4. 4Trade exactitude for volume when the situation permits: accept messiness (error, inconsistent formats, merged sources) in return for far more data, because sometimes 'more trumps better.'
5. 5Hunt for correlations rather than chasing causality; use the correlations to disprove intuitive causal stories, and stop demanding a causal 'why' before acting.
6. 6Pursue three reuse moves explicitly: basic reuse (search terms repurposed), merging datasets (combining sources), and finding 'twofers' (one collection serving two ends).
7. 7Decide extraction strategy: use the data yourself, license it to third parties for a percentage of value extracted rather than a flat fee, or — for government holders — release it as standardized, machine-readable open data so outsiders can find value you can't.
8. 8Recruit or become the creative outsider. Value the fresh statistician and new tooling over incumbent domain expertise, since the people who spot secondary uses are rarely the original collectors.

Oren Etzioni built Farecast on a mindset shift, not new physics: he saw that airfare data, useless to the airline once the plane landed, held dormant predictive value. He began with a sample of 12,000 data points, which performed well, then kept adding data — eventually the full domestic flight record — and prediction quality rose with volume. By 2012 the integrated Bing system called airfare moves correctly 75 percent of the time, saving travelers an average of $50 per ticket, and Microsoft bought Farecast for around $110 million. Etzioni says building it five or ten years earlier would have been impossible: storage and computing were too costly. But the deeper change was mindset — data reused as a raw material of business rather than treated as stale after its first purpose. His planned extension shows the pattern-recognition: apply the same method to hotel rooms, concert tickets, used cars — anything with little product differentiation, high price variation, and tons of data.

• If a dataset has finished its primary purpose, then do not delete it — store it and log candidate secondary uses; its option value persists like a compressed spring.
• If more data is available and tools can handle it, then prefer N=all over a sample, because the interesting findings hide in the subgroups a normal distribution smooths over.
• If getting more data requires accepting errors or messy formats, then take the trade — as long as the added volume improves the answer more than the noise degrades it.
• If your fast-thinking brain hands you a causal story, then treat it as a hypothesis to test against correlations, not as insight; intuited causality is often a shortcut that leaves you in the dark.
• If you can't extract all a dataset's value yourself, then license it for a percentage of extracted value so every party is incentivized to maximize reuse.

• Treating data as static and single-purpose — the tell is a data-deletion or archiving policy triggered the moment the primary use completes, discarding the option value.
• Defaulting to sampling out of habit — the tell is a report proud of its 3 percent margin of error that can't answer questions about specific subgroups or cities.
• Demanding causal proof before acting — the tell is projects stalled waiting to explain 'why,' when there is no clean mathematical way to prove causation anyway.
• Assuming domain incumbents will spot the value — the tell is the people who collected the data (like Con Edison staff) never imagining a secondary use, while outsiders with new tools do.
• Publishing 'open' data that isn't machine-readable — the tell is data that is public in name only, unstandardized and effectively unusable by the private sector or society.

• ▸Dataset name: what you already hold or can collect easily.
• ▸Primary use (spent): what it was originally collected for.
• ▸Latent uses imagined: list every secondary use you can picture (basic reuse / merge / twofer).
• ▸Merge candidates: other datasets it could combine with, noting alignment friction.
• ▸N=all feasibility: can you use everything, or must you sample — and what granularity do you lose?
• ▸Acceptable messiness: what error or format inconsistency you'll tolerate to gain more volume.
• ▸Correlation targets: relationships worth testing without needing a causal explanation.
• ▸Extraction strategy: own use / license for percentage / open release — pick and justify.
• ▸Option value note: rough sum of the choices above, treated as an asset even if off the balance sheet.