Big Data: A Very Short Introduction (Very Short Introductions)

1. 1Split labelled historical data into a training set (to build the model) and a test set (to check it generalizes) before you build anything; never evaluate on data you trained on.
2. 2Select predictors algorithmically rather than by hand to avoid bias, but cap the number: accuracy rises with enough terms and then falls as you add too many. Google used 45 search terms, not one term like 'flu' and not all 50 million.
3. 3Screen every strong correlate for a plausible mechanism before admitting it. Google found high-school basketball in the top 100 correlates and excluded it because basketball and flu merely peak in the same season.
4. 4Guard against over-fitting: when too many predictors are used, the model captures small random cases in the training data, fits training data very well, but predicts future data poorly. Watch the gap between training fit and test/real-world performance.
5. 5Interrogate the data source for selection bias before trusting output: Google's search-query data excluded everyone not using a computer and everyone using other search engines, so results were not scientifically sound.
6. 6Check for double-counting in the input signal: a person searching 'flu symptoms' explores several flu sites and gets counted several times, inflating the signal.
7. 7Update the model for changing conditions rather than freezing it. Real-time internet data was meant to deliver results within a day versus CDC's two-week-old collated data, but a static model degrades as behaviour shifts.
8. 8Set an explicit confidence threshold and only act when it is reached, as Watson does with confidence-scoring algorithms — the equivalent of a contestant hitting the buzzer only when sure.

Google Flu Trends aimed to forecast flu spread faster than the CDC, whose physician-reported data arrives about two weeks old. Google mined five years of search queries (2003–2008), used IP addresses to group searches by State, and integrated this with CDC data collected across ten regions (e.g. Region 9: Arizona, California, Hawaii, Nevada), relying on the known high correlation between flu-related searches and doctor visits. The team selected 45 search terms algorithmically and excluded high-school basketball — a top-100 correlate — because it peaked at the same time of year as flu without being relevant. The model still failed: with fifty million distinct search terms, many correlate strongly with CDC data by chance, so over-fitting crept in and the model fit training data well but predicted poorly. Compounding this, searchers on 'flu symptoms' visited multiple sites and were counted several times, and many flu-like searches ended in a common-cold diagnosis rather than flu. The selective source also excluded non-computer-users and users of rival search engines, producing biased, unsound results.

• If a variable correlates strongly with the outcome but has no plausible causal mechanism (like basketball and flu sharing a seasonal peak), then exclude it as spurious.
• If the model fits training data extremely well but predicts test/future data poorly, then you have over-fitting — reduce the number of predictors.
• If adding predictors stops improving test accuracy and starts degrading it, then you have passed the optimal count; stop and prune.
• If the data source systematically omits a population (non-computer-users, users of other search engines), then treat the output as biased and do not present it as scientifically sound.
• If confidence in a prediction has not reached the set threshold, then withhold the answer rather than acting on a weak signal.

• Confusing correlation with relevance — the tell: your top predictors include variables with no mechanism linking them to the outcome (basketball predicting flu).
• Over-fitting by piling in predictors — the tell: near-perfect training accuracy paired with weak real-world forecasts.
• Double-counting the signal — the tell: single users generate multiple hits (one 'flu symptoms' searcher browsing many sites), inflating counts.
• Trusting a biased sample as a population — the tell: your source silently excludes whole groups (non-computer-users, rival-search-engine users), yet you report results as definitive.
• Freezing the model against changing conditions — the tell: accuracy that was strong at launch drifts down over time because behaviour shifted and the model was never updated.

• ▸Outcome being forecast: state the exact target and its real-world source of truth.
• ▸Predictors and count: list them; note whether accuracy still improves at this count.
• ▸Training/test split: confirm the model was built on training data and scored on held-out test data.
• ▸Mechanism check: for each strong correlate, record the plausible causal link or mark 'spurious — exclude'.
• ▸Over-fitting gap: record training accuracy minus test accuracy; flag if large.
• ▸Source bias audit: list populations excluded by the data source.
• ▸Double-counting check: note whether one subject can generate multiple counted signals.
• ▸Confidence threshold: state the minimum confidence required before acting.
• ▸Update cadence: state when and how the model is retrained for changing conditions.

1. 1Classify the data you hold before protecting it: separate personal health data from sensors (blood pressure, pulse, sleep tracking), corporate sensitive material, and customer transaction data (address, credit card details), because each attracts different controls and privacy obligations.
2. 2Encrypt sensitive data in storage and transit — Chapter 7 pairs security with 'the importance of encryption' as the primary defence against data theft in the age of massive data leaks.
3. 3Anonymize any dataset that identifies individuals before analysis or sharing: strip identifiers so calls, movements, or records cannot be traced back, following the Flowminder model of constructing population maps from anonymized historic mobile data.
4. 4Require consent before tracking: confirm each subject has agreed to collection, since the Ebola case failed this test — individuals who had not given consent could still be identified from call trails and tower locations.
5. 5For employee monitoring, define and disclose the scope of tracking (websites visited, keystroke logging, private-use checks on company devices) and weigh it against the privacy pressure it creates before deployment.
6. 6Guard against re-identification even in anonymized sets: an approximate location plus caller/callee numbers can still pinpoint a person, so test whether combined fields defeat the anonymization.
7. 7Treat data as an owned asset, not a commodity: resolve ownership and privacy questions up front because, unlike oil, data 'appears not to be a finite resource' and its value depends on lawful, protected use.

During the 2014 West African Ebola outbreak, a distinguished international team wanted to use mobile phone data to track population movements and predict where the disease would next appear. The raw data was basic — caller and callee phone numbers plus an approximate location derived from the tower used for each call — but that was enough to identify individuals who had not consented to being tracked, creating a genuine privacy problem. To proceed responsibly, the Flowminder Foundation used anonymized historic data to construct movement maps rather than tracking named individuals in real time. This preserved analytical value (showing where people tend to travel in an emergency) while removing direct identifiers. The compromise was that anonymized historic data was of limited use because behaviour changes during epidemics. The security lesson holds regardless: the protective measure — anonymization plus reliance on historic rather than live consented data — let public health work continue without disclosing who called whom from where.

• If a dataset contains identifiers that could name a non-consenting individual, then anonymize before any analysis or sharing.
• If data is sensitive corporate or personal material at rest or moving between systems, then encrypt it as the default control against theft and leaks.
• If you plan to track employees on company devices, then disclose the scope first and weigh the privacy pressure against the theft risk you are reducing.
• If individuals have not given consent (as in the mobile-phone Ebola case), then do not use live identifiable data — substitute anonymized or historic data.
• If combining several anonymized fields (location + numbers) could re-identify a person, then treat the set as still-identifying and add further controls.

• Assuming anonymization is permanent: approximate location plus caller/callee numbers can re-identify people — the tell is that you can still trace a single record back to one person.
• Collecting sensor or fitness data via incentives without addressing privacy — the tell is employees feeling 'unwelcome pressure' to opt in, signalling consent is coerced not genuine.
• Treating data like oil once collected — the tell is proceeding as if you have a marketable commodity while ownership and privacy questions remain unresolved.
• Relying on historic data as if conditions are static — the tell is predictions that ignore behaviour change (Ebola figures ran over 50% higher than actual cases).
• Monitoring emails and websites as the only defence — the tell is confidence that theft is handled while stored data itself is left unencrypted.

• ▸Data category: label as health-sensor / corporate-sensitive / customer-transaction / research
• ▸Identifiers present: list fields that could name an individual
• ▸Consent status: recorded / not obtained / coerced-risk
• ▸Anonymization applied: yes/no + which identifiers stripped
• ▸Re-identification test: fields that combined could still identify (e.g. location + numbers)
• ▸Encryption: at rest yes/no; in transit yes/no
• ▸Monitoring scope (if employees): what is tracked and disclosed
• ▸Ownership resolved: who owns and controls this dataset

1. 1Define the specific decision you want big data to improve before touching the data — e.g., 'flag fraudulent transactions', 'estimate 30-year earthquake risk', 'reduce healthcare cost via remote monitoring'. Without the right data tied to a question, you produce nothing of value.
2. 2Pre-process the raw data to remedy consistency, repetition, and reliability problems, since 'there is considerable variability in the quality of the data collected electronically' and it must be cleaned before useful analysis.
3. 3Choose the learning method by whether you hold labelled examples: use supervised classification (decision tree) when you have transactions already labelled genuine/fraudulent; use unsupervised clustering when no target exists and you must discover hidden patterns.
4. 4Build the model on a training set, then validate it on a held-back test set to confirm the classifier generalizes before deploying it to score new observations as they arrive.
5. 5Translate model output into an operational action tied to the decision — a triggered phone call to the cardholder, a resource-allocation probability for buildings, a remote alert on blood pressure — so the analysis actually changes what happens.
6. 6Present results with interactive, regularly-updated visualization (like the GPS journey graphic based on satellite data) rather than static charts, because big data is constantly being added to.
7. 7Guard veracity: use volume to average out imprecise inputs (the Plataean brick-counting logic), but stay cautious because sufficient data will surface spurious correlations that mislead the decision.

A financial institution wants to decide, in real time, whether a card transaction is fraudulent. Because the transaction volume is too large for humans to check, it turns to classification. Historical transactions already labelled genuine or fraudulent (Figure 2) are split into a training set to build a decision tree (Figure 3) and a test set to verify it. A new transaction on Mr Smith's account enters the tree: first, has the card been reported lost or stolen? If yes, it is deemed fraudulent immediately. If no, the system checks whether an unusual item or unusual amount was purchased; if nothing unusual, it is labelled genuine. If the item is unusual, a phone call to Mr Smith is triggered — if he confirms the purchase it is genuine, if not, fraudulent. The organizational value is a fraud decision made at scale without a human reviewing every transaction, while a competing unsupervised approach would instead cluster the account's spending and flag a purchase that lands in distant Cluster A as unlikely to be legitimate.

• If you already have correctly labelled example outcomes, then use classification (supervised); if you have no target and want hidden patterns, then use clustering (unsupervised).
• If a new observation lands very distant from a customer's usual cluster, then treat it as unlikely to be legitimate and escalate.
• If the data source is inherently imprecise (e.g., social networks), then lean on volume to overcome noise — but stop trusting correlations that lack a plausible mechanism, since large data manufactures spurious ones.
• If the decision depends on real-time response (autonomous car, remote patient monitoring), then require that data be transmitted and analysed fast enough to send instructions back in a timely fashion.
• If the value case depends only on 'data is the new oil' marketing, then reject it — value exists only when you have the right data for the specific question, unlike oil which is marketable once struck.

• Collecting data with no experimental design or concept of what question matters, then expecting information to emerge — the tell is an analyst asking 'what can we do with this data?' after it's already stored.
• Trusting a model without a test-set check, so an over-fitted classifier looks accurate on historical data but misclassifies new transactions — the tell is validation performance never being reported separately from training.
• Mistaking a spurious correlation from massive volume for a real signal — the tell is a striking pattern nobody can explain by mechanism, discovered by scanning rather than hypothesizing.
• Ignoring privacy and ownership when the value comes from personal data (fitness trackers shared with employers) — the tell is employees feeling pressured to opt in and no consideration of the coercion or leak risk.
• Presenting fast-changing big data with static pie charts or infographics — the tell is a dashboard that is out of date the moment it's produced instead of an interactive, regularly-updated view.

• ▸Decision to improve: state the single business/care/risk decision this data must change
• ▸Right-data check: does the data you hold actually bear on this decision? (yes/no + gap)
• ▸Labelled outcomes available?: yes → classification / no → clustering
• ▸Pre-processing done: consistency, repetition, reliability issues fixed (list)
• ▸Train/test split: model built on training set, validated on test set (record test result)
• ▸Veracity note: source reliability and whether volume mitigates or manufactures noise
• ▸Operational action: the concrete trigger the model output causes (call, alert, resource shift)
• ▸Visualization: interactive and updated? how often is it refreshed
• ▸Privacy/ownership flag: whose data, consent basis, and leak/coercion risk

1. 1Inventory every data stream you collect (fitness-tracker output, keystroke logs, EHR fields, mobile-tower location trails) and label each as identifying or non-identifying — location trails and DNA sequence data are identifying even without a name attached.
2. 2Establish a consent basis before collection, not after: for the employer wearable-incentive scheme, document what data is shared, with whom, and confirm the opt-in is genuinely voluntary rather than pressured by the incentive.
3. 3Anonymize before pooling or sharing for research — the way EHR data 'may be anonymized and pooled for research purposes' — but test whether anonymization survives linkage against other datasets, since location and genetic data re-identify easily.
4. 4Encrypt data at rest and in transit; Chapter 7 treats encryption as the core defence against data theft, so apply it to the 600+ Tb an average hospital stores and to any corporate store of sensitive material.
5. 5Separate legitimate monitoring from surveillance creep: decide in advance which employee activities (email, websites, keystrokes) you monitor to prevent theft of sensitive material, and disclose that scope rather than logging everything silently.
6. 6Match retention to purpose — historic mobile data was useful to Flowminder only for movement patterns, not real-time behaviour, so keep only what the stated purpose requires and delete the rest.
7. 7Reassess as data volume compounds: with IBM estimating medical data doubling every 73 days by 2020, re-run your consent, anonymization, and encryption checks on a fixed schedule rather than once at project start.

During the 2014 West African Ebola outbreak, public health teams wanted mobile phone records to track population movement and predict where the disease would next appear. The digital trail was basic — caller and recipient numbers plus an approximate location derived from the cell tower used — yet privacy was a genuine concern because individuals who had never consented to being tracked could be identified from that trail. The workaround was to use historic, anonymized data rather than live identifiable records: Flowminder, a Swedish non-profit already experienced from a 2008 WHO malaria initiative, built maps of typical population movements from aggregated past data. This sidestepped consent problems but bought limited accuracy, because behaviour changes during an epidemic and mobile density was uneven — only just over half of Liberian and Sierra Leonean households had a phone in 2013, skewed away from poor rural areas. The case shows the core tension: the same data that saves lives by predicting spread also exposes non-consenting people, and anonymizing to protect them degrades the very signal you collected it for.

• If data can identify a person who never consented — as with mobile-tower location trails — then treat it as identifying and require anonymization or explicit consent before use.
• If an incentive is attached to sharing personal data (wearable-for-fitness-target schemes), then assume the opt-in may be coerced and document that pressure as a privacy issue, not a solved one.
• If you monitor employee activity to reduce theft of sensitive material, then define and disclose the exact scope (emails, websites, keystrokes) rather than defaulting to total logging.
• If the data is highly re-identifiable — genomic sequences, real-time location — then anonymization alone is insufficient; add encryption and access restriction.
• If the analytic purpose only needs aggregate patterns, then use historic or anonymized data even at a cost to accuracy rather than live identifiable records.

• Treating anonymization as permanent — the tell: pooled EHR or location data is re-identified when linked to another dataset, revealing you stripped names but not identifying structure.
• Assuming an opt-in incentive equals free consent — the tell: employees report feeling pressured into the wearable scheme, exposing the 'voluntary' framing as coercive.
• Over-collecting because storage is cheap and volume compounds — the tell: you hold data far beyond the stated purpose (real-time trails kept when only movement patterns were needed), enlarging the breach surface.
• Confusing security monitoring with surveillance — the tell: keystroke and website logging expands silently past its anti-theft justification into blanket tracking of private use.
• Believing 'data is the new oil' means raw data is inherently valuable and therefore worth hoarding — the tell: you retain masses of unprocessed personal data that carries breach risk but, unlike oil, no realized value until the right data is processed.

• ▸Data stream: name the source (e.g. fitness tracker, cell-tower trail, EHR field).
• ▸Identifying? mark yes/no — location and genetic data count as yes.
• ▸Consent basis: state opt-in, contract, or none, and whether any incentive applies.
• ▸Coercion check: note if an incentive could pressure opt-in.
• ▸Purpose: one line on the specific use justifying collection.
• ▸Anonymization + re-identification test: method used and whether it survives linkage.
• ▸Encryption: at rest / in transit status.
• ▸Access + monitoring scope: who can see it; if employee monitoring, what exactly is logged.
• ▸Retention & review date: delete-by date and next reassessment.

1. 1Classify every source you plan to ingest as structured, semi-structured, or unstructured before choosing tools: structured fits rows/columns (order forms, name/address/age); semi-structured carries identifying tags (tagged images, hashtagged posts, emails); unstructured resists tabulation (photos, videos, tweets, word docs). Expect roughly 80% of your data to fall in the unstructured bucket.
2. 2Estimate volume in real byte units and check it against the traditional-methods threshold: if you cannot collect, store, and analyse the dataset with conventional computing and statistical methods, it meets the volume criterion. Use the byte scale (1 Tb ≈ one good laptop hard drive; 1 Pb = a million Tb; 1 Eb = 10^18 bytes) to size storage.
3. 3Separate the two senses of velocity when you design the pipeline: rate of generation (how much arrives per second) and speed of processing required (how fast a response must be returned). Sensor and autonomous-vehicle data forces near-instant processing; social virality forces high ingest throughput.
4. 4Account for variability — the changing flow rate — by provisioning for peak-time surges, because systems are most prone to failure exactly when data flow spikes.
5. 5For unstructured or high-velocity data, reject the relational database (RDBMS): its fixed schema is hard to change and rows/columns cannot hold unstructured content. Distribute data across servers with multiple identical copies (e.g. Hadoop DFS) since failure is treated as inevitable.
6. 6Match analytic technique to the data type: apply traditional statistical analysis only to structured data; use sentiment analysis for tweets, and streaming analytics (which bypasses traditional statistical/data-processing methods) for real-time sensor feeds.
7. 7Add veracity as a fourth check — assess whether the collected data is accurate and reliable before mining, since accuracy has been the hallmark of statistical analysis.

Take the autonomous car as a variety-plus-velocity problem. Each autonomous car is estimated to generate on average 30 Tb of data daily from its sensors — that is the volume dimension, and it is unstructured/semi-structured sensor data, the variety dimension. The velocity dimension is dual: the data is generated in real-time and must be sent wirelessly to a central location, analysed, and returned as driving instructions in a timely fashion or the car fails. Traditional statistical and data-processing methods cannot meet this, so streaming analytics is the emerging research area meant to handle it. Tesla's own figure — one fatality in 130 million autopilot miles versus one per 94 million non-automated US miles — shows the stakes of processing that velocity correctly. Contrast this with the Large Hadron Collider, where even 1% of generated data leaves 25 Pb annually: same 'big' label, but a volume/storage problem rather than a real-time velocity problem.

• If the dataset cannot be collected, stored, and analysed with traditional computing and statistical methods, then classify it as meeting the volume criterion — do not fix a numeric size threshold, because yesterday's 'big' is today's ordinary.
• If data is unstructured or high-velocity with real-time processing needs, then use a distributed file system with replicated copies, not an RDBMS with a fixed schema.
• If a response must be returned in real-time (sensor, GPS, autonomous vehicle), then prioritize processing speed and consider streaming analytics over batch statistical methods.
• If the data carries identifying features like hashtags, word tags, or email header fields, then treat it as semi-structured and exploit those tags to make it searchable rather than discarding it as unstructured.
• If flow rate varies by peak times, then provision for the surge, because computer systems fail most at high-flow moments.

• Defining big data by size alone. The tell: you set a fixed Tb/Pb cutoff — but the IBM/Oxford survey of 1,144 professionals showed only 10% chose 'large volumes' as the defining trait while 18% chose 'a greater scope of information', and a third didn't know.
• Assuming your data is analysable with standard statistics. The tell: ~80% of the world's data is unstructured text, photos, and images, so traditional structured methods silently fail to touch most of it.
• Forcing unstructured data into an RDBMS. The tell: you cannot change the schema after building it, and the content won't fit rows and columns.
• Treating velocity as one thing. The tell: you engineer for ingest throughput but miss the required processing-response deadline (or vice versa), so a real-time system like the autonomous car returns instructions too late.
• Ignoring variability and veracity. The tell: your system crashes during peak-flow times, or you mine data whose accuracy and reliability were never checked, corrupting the extracted 'value'.

• ▸Source name: which engine, sensor, or platform generates it (e.g. Google, tweets, VLT telescope)
• ▸Structure type: structured / semi-structured / unstructured — one label per source
• ▸Volume estimate: expected size in real byte units (Tb / Pb / Eb) per day or per run
• ▸Velocity — generation rate: how fast data arrives (e.g. real-time, 500M tweets/day)
• ▸Velocity — processing deadline: how quickly a response is required (real-time / batch)
• ▸Variability note: expected peak-time surge and failure risk window
• ▸Veracity check: is the data accurate and reliable? source of quality assurance
• ▸Storage decision: RDBMS or distributed file system with replicated copies
• ▸Analytic method: statistical / sentiment analysis / streaming analytics

1. 1Before analysis, run every dataset through a pre-processing pass that checks explicitly for three defects: consistency, repetition, and reliability. Flag records that contradict each other, remove duplicate entries, and mark sources of unknown provenance.
2. 2Deduplicate first: with big datasets, unwanted repetitions both corrupt counts and waste finite storage. Screen for the same entity counted multiple times (e.g. one user visiting several related pages in a single session, inflating the tally).
3. 3Assess the collection method for structural bias before trusting the sample. Ask who is systematically excluded — for search-query data, that means everyone who does not use a computer and everyone using a different search engine — and document that exclusion as a known limitation.
4. 4Use volume as a corrective, not a cure: engage the largest possible number of independent observations (the Plataean brick-counting logic) to get closer to the true value, but cap it, because sufficient data guarantees spurious correlations will appear.
5. 5Do variable/predictor selection algorithmically rather than by hand to avoid analyst bias, but guard against over-fitting: too many predictors model random noise in the training data and predict poorly out of sample.
6. 6Test any strong correlation for a plausible mechanism before including it. Coincidental co-timing (two things peaking in the same season) is a reason to exclude, not include.
7. 7For applications where veracity is fundamental — medical data especially — take deliberate steps to eliminate inaccurate data and protect it from tampering, since data that can be altered has no veracity to trust.

Google Flu Trends is the cautionary case. The team correlated flu-related search queries against CDC data, aiming to predict flu spread within a day instead of the CDC's two-week lag. From fifty million distinct search terms they algorithmically selected forty-five predictors. High-school basketball ranked in the top 100 correlations but was excluded because both it and flu peak at the same time of year with no causal link. Despite this, veracity problems sank the predictions: with fifty million candidate terms, other spurious correlations inevitably slipped in; the model over-fit the training data; searches were biased toward computer users on Google alone; a single person exploring several flu websites was counted multiple times; and many 'flu-like' searches were actually the common cold, not flu. The lesson is that raw search data is imprecise, biased, and repetitive, and volume alone amplified rather than fixed these flaws.

• If a predictor correlates strongly but has no plausible mechanism and merely peaks at the same time (basketball vs. flu), then exclude it.
• If adding more predictors improves fit on training data but not on new data, then you are over-fitting — reduce the predictor count.
• If the collection method systematically omits a population (non-computer users, other search engines), then treat results as biased and state the limitation rather than generalizing.
• If a single actor can be counted multiple times through repeated interactions, then deduplicate before aggregating counts.
• If the application is medical or safety-critical, then veracity outranks speed and volume — spend the effort to eliminate inaccurate data and secure it against alteration.

• Trusting volume to solve quality: given enough data you will find any number of spurious correlations. Tell: your model has hundreds of significant predictors with no explanation for most.
• Over-fitting: the model matches historical data almost perfectly but degrades on new periods. Tell: excellent training accuracy, deteriorating live predictions (Google Flu Trends).
• Double-counting through repetition: one user's multiple related searches or page visits inflate the tally. Tell: counts rise faster than the underlying population could support.
• Ignoring collection bias: sampling only from one channel and generalizing to everyone. Tell: your source silently excludes whole groups (non-computer users) yet you claim population-level findings.
• Leaving data alterable: assuming stored data is trustworthy when it can be changed. Tell: as in the fitness-tracker study, seventeen devices tested and none was secure enough to stop data being changed, only four attempted to preserve veracity and all were bypassed.

• ▸Data source & collection method: name the source and how records were captured
• ▸Consistency check: describe contradictions found and how resolved
• ▸Repetition/deduplication: state duplicate/double-count rule applied
• ▸Exclusion bias: list who or what is systematically left out of the sample
• ▸Reliability/provenance: rate trust level of source and note tampering safeguards
• ▸Predictor count & selection method: how many, chosen algorithmically or manually
• ▸Over-fitting test: training accuracy vs. hold-out accuracy result
• ▸Mechanism check: for each strong correlation, plausible causal link (Y/N)
• ▸Veracity verdict: fit for analysis, fit with caveats, or reject

1. 1Decide storage model by data structure: keep structured, row/column data in an RDBMS; route unstructured and semi-structured data to a distributed file system like Hadoop DFS where it stores cheaply and readily.
2. 2Distribute the dataset across many nodes (Hadoop clusters run across tens of thousands of nodes in data centres worldwide) rather than scaling a single machine vertically, which loses function.
3. 3Set up the master-slave topology: one NameNode to handle client requests, allocate storage space, track availability and data location, and manage file operations; many DataNodes to physically store the data.
4. 4Enable block-level replication so each block sits on multiple DataNodes — e.g. Block A on both DataNode 1 and DataNode 2 — treating system failure as inevitable and building recovery into storage.
5. 5Configure the Heartbeat mechanism: each DataNode sends the NameNode a message every three seconds; treat any missing Heartbeat as a failed node and promote a replica-holding node to working status.
6. 6For continually growing, non-relational workloads, add a NoSQL layer that scales horizontally to preserve scalability, availability, and performance as new data arrives.
7. 7Apply the CAP Theorem when designing the distributed database: accept that only two of consistency, availability, and partition tolerance can be met, and choose the pairing that fits your workload.
8. 8Pair storage with a distributed processing method like MapReduce, which spreads calculations or queries over many computers to process the data held in the Hadoop DFS.

Consider a single Hadoop DFS cluster with one NameNode and DataNodes 1 and 2. A client request arrives at the NameNode, which distributes storage space, records where data lives, and controls the client's access. Block A is written to DataNode 1 and replicated to DataNode 2, so two identical copies exist. Every three seconds each DataNode sends a Heartbeat to the NameNode confirming it still functions. If DataNode 1 stops sending its Heartbeat, the NameNode assumes it has ceased to function, and DataNode 2 — already holding a copy of Block A — becomes the working node and continues processing tasks without loss of data. This is why organizations like Facebook, Twitter, eBay, Amazon, and Google can run Hadoop invisibly in the background while users interact with their services.

• If data is unstructured or semi-structured and the schema may change, then use a DFS or NoSQL, not an RDBMS, because a relational schema is difficult to change once built.
• If you need to keep adding new data continuously and maintain performance, then scale horizontally with NoSQL rather than scaling an RDBMS vertically, which loses function.
• If the system is genuinely distributed and the network can partition, then you must sacrifice one of consistency, availability, or partition tolerance — pick the two that match your priorities.
• If a DataNode misses its three-second Heartbeat, then treat it as failed and switch processing to a node holding a replica of its blocks.
• If you require both reliable storage and analysis of the same big data, then store in Hadoop DFS and process with MapReduce, since MapReduce is core Hadoop functionality.

• Forcing unstructured data into an RDBMS: the tell is a rigid schema you cannot alter and data that will not fit conveniently into rows and columns.
• Scaling vertically to cope with growth: the tell is degraded function and lost performance as the single system is pushed beyond its limits.
• Storing only one copy of each block: the tell is that a single DataNode failure halts processing and loses data, because no replica exists to take over.
• Ignoring the Heartbeat/replication design and assuming hardware won't fail: the tell is that failure is treated as exceptional rather than inevitable, so no recovery is built in.
• Demanding all three of consistency, availability, and partition tolerance from a distributed system: the tell is a design that silently breaks under a network partition because CAP allows only two.

• ▸Data type: mark structured / semi-structured / unstructured to pick RDBMS vs DFS/NoSQL
• ▸Growth pattern: state whether new data is added continuously (favours horizontal NoSQL scaling)
• ▸Node count and location: estimate number of DataNodes and data centre distribution
• ▸Replication factor: number of identical block copies per DataNode (e.g. Block A on nodes 1 and 2)
• ▸NameNode responsibilities: list request handling, space allocation, location tracking, file operations
• ▸Heartbeat interval: confirm three-second check and failover node for each block
• ▸CAP choice: circle the two of consistency / availability / partition tolerance you will guarantee
• ▸Failover plan: name which replica-holding node takes over when a DataNode fails
• ▸Processing method: specify the analysis layer (e.g. MapReduce) running over the stored data

1. 1Classify your data first: separate structured (spreadsheet tables with defined fields like name, address, age), semi-structured (emails with structured metadata headers plus unstructured message bodies; social posts with hashtags), and unstructured (photos, videos, tweets, documents) — roughly 80 per cent of the world's data is unstructured and not amenable to traditional statistical methods.
2. 2Add structure to unstructured data before analysis by applying metadata tags and descriptive references — tag an image with a word to make it searchable, use hashtags to group topic-related messages — so existing analytic techniques can operate on data that appeared to have no structure.
3. 3Match the technique to the response-time constraint: for datasets processed after storage use batch pattern-search algorithms; for GPS, autonomous vehicles, or sources generating data continuously (an autonomous car averages 30 Tb/day), apply streaming analytics, which bypasses traditional statistical and data-processing methods to handle near-instant processing.
4. 4Distribute the computation across clusters of computers rather than a single machine, because as data volume rises the performance of traditional relational databases decreases — the core problem is scalability.
5. 5Run the pattern-discovery step: apply algorithms that meld statistics and computer science to search large datasets for patterns (as in Web tracking logs recording query term, IP address, submission time, dwell time, and visit order).
6. 6Filter for importance: determining which patterns matter is the deciding factor for success — discard patterns that carry no useful information rather than reporting everything the search surfaces.
7. 7Validate against a baseline before trusting a result — compare the analytic outcome to a known reference rate (Tesla measured one autopilot fatality per 130 million miles against one per 94 million miles of non-automated driving) so the pattern is judged in context, not in isolation.
8. 8Close the loop to extraction: confirm the output is actionable information a decision-maker can use, since big data is only useful if you can extract useful information from it.

Search a term like 'border collies' on Google and click the top result. Behind that single click, roughly sixty-seven third-party site connections are generated, and logs are created recording the query term itself, the IP address of the device, the time the query was submitted, how long you stayed on each site, and the order in which you visited them. Each of these fields is structured metadata attached to otherwise raw browsing activity, which turns a diffuse interaction into a tabulated record amenable to pattern search. Analytic algorithms then mine these logs across billions of daily searches (over 3.5 billion per day on Google as of 2012) to identify which sites are relevant and which interests cluster together, all without directly identifying the individual. The discovered patterns are shared between commercial enterprises to track interests. The step that adds value is not the collection but the selection: deciding which of the surfaced patterns is important enough to act on.

• If data is unstructured (photos, video, free text) and you have no tagging scheme, then add metadata tags or hashtags first — otherwise existing analytic techniques cannot be applied.
• If response time is critical and data streams continuously (autonomous vehicles, GPS, real-time sensors), then use streaming analytics; if the data can be stored and analysed later, then use batch cluster processing.
• If the dataset is structured, fast, and stable in schema, then keep it in a relational database with SQL; if volume growth is degrading query performance, then move to distributed storage across clusters.
• If a discovered pattern cannot be tied to useful, actionable information, then discard it — volume of patterns is not the goal, importance is.
• If you report an analytic result, then benchmark it against a known baseline rate before drawing conclusions.

• Applying traditional structured-data statistical methods directly to unstructured sources — the tell is that ~80 per cent of your candidate data is text, photos, and images that the tools silently ignore or fail to load.
• Forcing real-time streaming data through traditional statistical and data-processing pipelines — the tell is that response time lags behind the source (a car generating 30 Tb/day cannot wait for batch processing), so decisions arrive too late to act on.
• Treating semi-structured data as fully unstructured and discarding the metadata — the tell is you throw away email headers or hashtags that already carry the structure needed for grouping and search.
• Scaling a relational database beyond its limits instead of distributing — the tell is queries slowing steadily as volume rises, the classic scalability problem.
• Mistaking pattern quantity for insight and reporting every correlation found — the tell is no criterion for which patterns are important, so decision-makers drown in noise.

• ▸Data type: mark structured / semi-structured / unstructured and note the share that is unstructured
• ▸Structuring step: list metadata tags or hashtags you will add to make data searchable
• ▸Response-time need: state whether real-time (streaming analytics) or batch is required
• ▸Storage/compute plan: relational database + SQL, or distributed clusters — justify by data volume
• ▸Technique applied: name the pattern-search algorithm and the source fields it reads
• ▸Importance filter: state the rule for deciding which discovered patterns matter
• ▸Baseline for validation: record the known reference rate you will compare results against
• ▸Extracted information: state the specific actionable output a decision-maker will receive

1. 1Classify your raw input first: structured (fits databases/spreadsheets), semi-structured (email with metadata heading plus unstructured body), or fully unstructured (text, photos, images). Roughly 80% of the world's data falls in the last bucket and resists traditional structured analysis.
2. 2Add structure to unstructured data before extraction by attaching descriptive metadata tags — word tags on images, hashtags on social posts — so the item becomes identifiable and searchable.
3. 3Match the tool to the data type: unstructured data cannot go into traditional databases or spreadsheets, so route it to purpose-built tools; route latency-sensitive high-volume streams (like the 30 Tb/day per autonomous car) to streaming analytics that bypasses traditional statistical processing.
4. 4Capture the incidental data trail, not just the primary result. When a query runs, log the query term, IP address, submission time, dwell time per site, and visit order — value lives in these logs, not only the answer returned.
5. 5Gather from a defined population and locate observations spatially or temporally so patterns become visible — e.g. tie each case to a location and plot it.
6. 6Test whether the extracted pattern supports or refutes a specific hypothesis, then act on it: the extraction is only complete when it informs a decision (shut the pump, apply the brakes).
7. 7Confirm processing capacity matches volume before committing — Hollerith's tabulator cut census processing from eight years to one; without a capacity match, extraction never finishes in a useful window.

John Snow faced a cholera outbreak in London with no structured dataset — only scattered cases and a prevailing belief the disease was airborne. He gathered data directly from local inhabitants and established that affected people all used the same public water pump. He then imposed spatial structure by producing a map showing illness occurred in clusters around the Broad Street pump. That map turned raw case reports into a pattern that supported his hypothesis of contaminated water over airborne spread. He acted on the extracted information by persuading the parish authorities to shut the pump — accomplished by removing the pump handle. The extraction was useful precisely because it moved from raw observation to a spatial pattern to a decision.

• If data arrives as email or social posts, then treat it as semi-structured — extract the metadata/hashtags for structure and process the free text separately.
• If the volume and speed exceed what traditional statistical methods can process (near-instant demands, tens of Tb daily), then use streaming analytics rather than batch processing.
• If an image or document has no searchable handle, then attach metadata tags before attempting extraction.
• If a pattern emerges (a cluster, a shared source), then test it against a stated hypothesis before acting, not after.
• If processing time exceeds the window in which the answer is still useful (census taking eight years for a ten-year cycle), then change the processing technology, not the analysis question.

• Trying to store or analyse unstructured data in traditional databases/spreadsheets — the tell is tools that simply reject or mangle text, photos, and images that make up ~80% of data.
• Keeping only the primary output and discarding the incidental trail — the tell is you have the search results but not the logs (query term, IP, dwell time, visit order) where the behavioural value sits.
• Seeing a pattern and treating it as conclusion rather than hypothesis support — Snow used the cluster map to lend support, not to declare proof; skipping the test invites wrong action.
• Applying batch statistical methods to streaming, latency-critical data — the tell is data piling up unprocessed while decisions (like braking) need it almost instantly.
• Underestimating processing capacity against generation rate — the tell is a backlog that never clears, as with hand-tallied census data that grew beyond what could realistically be recorded.

• ▸Data source: name where it originates (e.g. search logs, telescope, sensors)
• ▸Structure class: mark structured / semi-structured / unstructured
• ▸Structuring step: what tags or metadata you add to make it searchable
• ▸Incidental trail: list the by-product data captured (timestamps, IPs, dwell, order)
• ▸Tool/route: which storage and analysis method fits this type (traditional vs streaming)
• ▸Volume/velocity check: daily size and whether processing beats the useful-time window
• ▸Pattern extracted: the cluster, correlation, or trend you found
• ▸Hypothesis tested: the specific claim this pattern supports or refutes
• ▸Action taken: the decision or intervention the information informs