R for Data Science

1. 1Open with a question, not a plot. Write the question in one line (e.g., 'What type of variation occurs within my variables?'). Per Cox, an approximate answer to the right, vague question beats an exact answer to the wrong one.
2. 2Pick the view that fits the variable type. One variable: histogram (continuous) or bar chart (categorical). Two variables: freqpoly or boxplot for categorical vs continuous. Make the plot fast and imperfect, since speed matters more than polish at this stage.
3. 3Read the plot for leads and write them down: clusters, unusual values, missing values, apparent covariation. For clusters, ask how observations within a cluster are similar, how separate clusters differ, how to describe them, and why the clusters might be misleading.
4. 4When rare values hide behind tall bins, zoom in. In the diamonds `y` histogram, `coord_cartesian(ylim = c(0, 50))` exposes three unusual values (0, ~30, ~60) that the full-scale plot hid. Use `coord_cartesian()` to zoom; `xlim()`/`ylim()` discard data outside the limits.
5. 5Transform to sharpen the comparison. Build a new variable (e.g., `cancelled = is.na(dep_time)`, `sched_dep_time` rescaled to hours) with `mutate()`, then replot. If group sizes differ so much that shapes are unreadable, swap count for density on the y-axis.
6. 6Refine or replace the question from what you saw. Each plot should produce either a sharper question or a new one. Then repeat from step 2. Cleaning is part of this loop: ask whether the data meets your expectations.
7. 7Use models as another way to look, not only for confirmation. Fit and overlay candidate models on the data (e.g., the 10 best grid models over `sim1`) to see what pattern they capture, then refine.
8. 8Triage as you go. Keep investigating every idea early on, then narrow to the few productive areas worth writing up. Keep the code in scripts so each pass can be rerun.

Start with the question: how are the values of `y` in `diamonds` distributed? You draw `geom_histogram(mapping = aes(x = y), binwidth = 0.5)`. The only evidence of a problem is unusually wide y-axis limits, because the common bins are so tall that the rare bins are invisible. You refine the view by adding `coord_cartesian(ylim = c(0, 50))`. Now three unusual values show up: 0, ~30, and ~60. That is a lead, and it prompts a new question: are these data-entry errors or something real? Elsewhere, you ask whether cancelled flights depart at different times. You build `cancelled = is.na(dep_time)` and rescale `sched_dep_time`, then draw a `geom_freqpoly` coloured by `cancelled`. The plot is weak because non-cancelled flights vastly outnumber cancelled ones, so you note the problem and move to a density-based comparison. Each pass turns one plot into a better question.

• If a plot shows tall common bins and you suspect rare values, then zoom the y-axis with `coord_cartesian()` rather than filtering the data.
• If group counts differ greatly and you need to compare shapes, then display density instead of count.
• If you see clusters, then ask how they are similar within, different between, how to describe them, and why they might be misleading, before treating them as real.
• If a lead looks promising after several passes, then narrow to it and prepare to communicate it; if it stays flat, then drop it as a dead end and move on.
• If you want to claim confirmation, then remember each observation can be used only once for that. Once you look more than once, you are exploring, so label the result a hypothesis.

• Polishing the first plot instead of iterating. The tell: you have one beautiful chart and no second question.
• Missing outliers because the full-scale histogram hides them. The tell: the y-axis range looks oddly wide but you never zoom in.
• Comparing groups by raw count when their sizes differ. The tell: the smaller group looks flat, like the non-cancelled vs cancelled freqpoly.
• Treating exploratory findings as confirmed results. The tell: you reused the same observations to find and to 'prove' a pattern.
• Leaving the loop's work only in the console or workspace. The tell: you cannot re-create a promising lead from a script after restarting.

• ▸Pass number: count each cycle so you can see how many leads you generated.
• ▸Question: state the question in one line, vague is fine.
• ▸Variables and type: list the variables involved and whether each is categorical or continuous.
• ▸View or transformation used: name the plot, new variable, or model you tried.
• ▸What I saw: record clusters, unusual values, missing values, or covariation.
• ▸Possible explanations: list data errors, real effects, or misleading artifacts.
• ▸Refined or new question: write the question that the next pass will ask.
• ▸Status: mark each lead as follow up, dead end, or write up.
• ▸Script saved: confirm the code is in a script and reruns cleanly.

1. 1Write each analysis as one document that holds prose, code, and results together. The source describes R Markdown as a unified authoring framework that combines your code, its results, and your prose commentary. If a number appears in your write-up, it should come from a chunk in that document, not from a pasted value.
2. 2Pick the mode by audience. Use notebook mode for analyst-to-analyst communication and report mode for analyst-to-decision-maker communication. The same document can serve both purposes through R Markdown formats, so write once and change the output format.
3. 3Load packages at the top of the script so a reader sees at once what the analysis needs. Check that each package is current; for tidyverse packages, run tidyverse_update(). A bug you hit may already be fixed.
4. 4Make the data travel with the code. For a small example, use an inline CSV: read_csv("a,b,c\n1,2,3\n4,5,6"). For a real object, run dput(mtcars), copy the output, and in the script type mtcars <- and paste. Cut the data to the smallest subset that still shows the problem or result.
5. 5Record reasoning next to the code. Use comments and prose to state why you took an approach, since code communicates meaning to humans as well as to the computer. Write for future-you, who is a collaborator on every project.
6. 6Rewrite while the ideas are fresh. The first version of code is rarely clear, and rewriting is the key to clarity. Ask whether it is obvious what you did, but do not rewrite every function. Balance what you need now against time saved later.
7. 7Strip everything unrelated to the point. Use spaces and concise, informative variable names. The shorter the code, the easier it is to understand and to fix.
8. 8Verify reproducibility by starting a fresh R session and running or pasting in the whole script. Only a clean run counts as proof.
9. 9Keep a running record of successes and failures in an analysis notebook so that you can learn from them later.

You get stuck on a problem in a dataset and want help. Following the source's reprex procedure, you open a new script and put library(tidyverse) at the top, then run tidyverse_update() to confirm your packages are current. You run dput(mtcars), copy the output, and in the script type mtcars <- followed by the paste. You then look for the smallest subset of rows and columns that still reproduces the problem, and delete every line unrelated to it. You add a comment at the line where the problem occurs and check that spacing and variable names are tidy. Last, you start a fresh R session and paste in the whole script. If it fails or behaves differently, the example was not reproducible and you fix it. The source adds that in the course of making a reprex you often figure out the problem yourself.

• If the reader is another analyst, use notebook mode and keep the code and reasoning visible. If the reader is a decision-maker, use report mode, from the same document.
• If you want help from others, build a minimal reproducible example with packages, data, and code, and keep only the smallest data subset that shows the problem.
• If you cannot share the whole dataset, use dput() to turn a subset into R code, or use an inline CSV for a tiny case.
• If code is clear now but you may forget why, rewrite and annotate it while the ideas are fresh. If it is a one-off function, do not over-polish it; balance present need against long-run savings.
• If you have not tested the document in a fresh R session, treat it as not yet reproducible.

• Packages are loaded midway or are implicit. The tell: the script errors for a collaborator on a missing function, or you cannot tell what the example needs.
• Data is described but not included. The tell: others reply asking for the data, or the script only runs on your machine.
• Sharing the full dataset or a long script with unrelated code. The tell: readers cannot find where the problem lies. The source says shorter code is easier to understand and fix.
• Code with no recorded reasoning. The tell: months later you, as future-you, cannot say why you tackled the analysis the way you did.
• Skipping the fresh-session check. The tell: the script works in your session because of leftover objects, but fails when pasted into a clean one.

• ▸Audience and mode: Name the reader and choose notebook (analyst) or report (decision-maker).
• ▸Packages: List the packages loaded at the top of the script and note that you ran tidyverse_update().
• ▸Data source: State how the data is included (read_csv path, inline CSV, or dput() pasted after name <-).
• ▸Smallest subset: Describe how you cut the data to the minimum that still shows the result or problem.
• ▸Reasoning notes: Write one or two sentences on why you chose this approach, to be placed as comments or prose.
• ▸Cleanup: Confirm that unrelated code is removed, names are concise and informative, and spacing is consistent.
• ▸Problem marker: Point to the comment that shows where the issue or key result sits.
• ▸Fresh-session test: Record the date you ran the full script in a new R session and whether it passed.
• ▸Success/failure log entry: Note what worked and what did not, for your analysis notebook.

1. 1Get the data into a usable form first: import it into a data frame, then tidy it so each column is a variable and each row is an observation. Without this, you spend effort fighting the data instead of questioning it.
2. 2Transform to the question: narrow to observations of interest (one city, the last year), create new variables from existing ones (velocity from speed and time), and compute summaries (counts, means). Keep the counts next to every mean.
3. 3Visualize before you model. A good plot shows you things you did not expect, raises new questions, or hints that you are asking the wrong question or need different data. Write down each surprise as a candidate hypothesis.
4. 4Map variables to aesthetics deliberately (x, y, color, shape, alpha) and check what the plot dropped. ggplot2 uses only six shapes at a time, so extra groups go unplotted by default. Count the groups in the plot against the groups in the data.
5. 5Look for the pattern, then ask what produces it. In the batting data, two patterns appear: variation in the aggregate shrinks as data points increase, and skill (ba) correlates positively with opportunities (ab) because teams pick their best players to play. Name the mechanism behind each.
6. 6Challenge your ranking and summaries. Sorting on desc(ba) puts players with 1 at-bat and a 1.000 average on top; the best averages are luck, not skill. Filter (for example ab > 100) or otherwise account for sample size before you read anything into the ordering.
7. 7Once a question is precise, use a model to answer it, knowing every model makes assumptions and cannot question its own. Then return to visualization to see what remains after the strong pattern is removed. Iterate between the two many times.
8. 8Keep every step in a script, not in your workspace. Restart and rerun (Cmd/Ctrl-Shift-F10, then Cmd/Ctrl-Shift-S) so the insight can be re-created from code. Then communicate the result, because understanding that stays with you alone is incomplete.

The source's baseball case. Using the Lahman package, you group Batting by playerID and compute ba = sum(H)/sum(AB) and ab = sum(AB). You then plot ab (x) against ba (y) for players with ab > 100, adding geom_point() and geom_smooth(). Two patterns appear. Scatter in ba is wide at low ab and narrows as ab grows, which is the effect of having more data points. There is also a positive trend between ab and ba, explained by teams controlling who plays and picking their best players. The naive check, arrange(desc(ba)), returns players such as abramge01 with ba = 1 and ab = 1, and the source calls these players clearly lucky, not skilled. The insight is not 'these players are the best'. It is that small samples produce extreme averages, and that opportunity is itself tied to skill. Ranking only becomes meaningful once you account for ab.

• If a ranking is led by extreme values, then check the sample size behind each row before interpreting; filter or adjust for count.
• If a plot surprises you or looks wrong, then treat it as a signal that your question or your data may be wrong before you tune the plot.
• If your question is precise enough to state mathematically, then use a model; if you are still exploring, then plot, because a model cannot surprise you.
• If you reuse the same observations to test a hypothesis, then label the work exploratory; confirmation requires using an observation once and a preregistered plan.
• If a group is missing from a plot, then check the aesthetic's limits (six shapes) before concluding the group is absent from the data.

• Reading luck as skill: a top-of-list result sits on one or two observations. The tell is a perfect or extreme value paired with tiny n.
• Silent data loss in a plot: an aesthetic such as shape drops groups beyond six. The tell is a category you know exists (SUVs) missing from the figure.
• Treating exploration as confirmation: you test a hypothesis on the same data that suggested it. The tell is a clean 'finding' with no pre-written analysis plan and repeated looks at the same observations.
• Trusting the model to find surprises: you fit and accept the output, but a model inherits your assumptions and cannot question them. The tell is that nothing in the output ever contradicts your expectations.
• Insight trapped in the workspace: the result exists only in session objects, not in code. The tell is that you cannot rerun the script from a fresh restart and reproduce it.

• ▸Question: the precise question the data is meant to answer, in one sentence.
• ▸Data form check: confirm each column is a variable and each row an observation; note any fixes.
• ▸Summary and n: the statistic you computed and the number of observations behind each value.
• ▸Pattern observed: the strongest pattern seen in the plot, in plain words.
• ▸Mechanism: what process could produce this pattern (for example, teams pick better players to play more).
• ▸Noise/luck check: how you ruled out small-sample extremes (filter threshold, count column).
• ▸Dropped or hidden data: groups or rows missing from the plot or summary, and why.
• ▸Residual signal: what remains after removing the strong pattern, and the next question it raises.
• ▸Status and reproducibility: exploratory or confirmatory, and the script that regenerates the result from a fresh restart.

1. 1Scope the work against the 80-20 rule before you start. Treat import, tidy, transform, visualize, model, and communicate, with programming running through all of them, as the 80% toolkit. Write down which part of the problem needs something outside it (the remaining 20%) and look up that resource separately, so a niche need does not derail the core workflow.
2. 2Create an RStudio project for each analysis (File → New Project) and store it in a sensible subdirectory. Keep input data, R scripts, results, and figures together. Confirm with getwd() that the project's home directory is the working directory.
3. 3Use relative paths with forward slashes only. Never call setwd(), and never use absolute paths such as /users/hadley or C:. Backslashes need doubling in R, and no one else shares your directory layout, so absolute paths break sharing and rerunning.
4. 4Keep your work in a script saved in the editor, not typed into the console. Build a reset loop: Cmd/Ctrl-Shift-F10 restarts RStudio, then Cmd/Ctrl-Shift-S reruns the whole script. Run it often so hidden state cannot mask errors.
5. 5Tidy before you transform or plot. Check that each column is one variable and each row is one observation. The up-front work means you spend less time munging between representations and more time on the analytic question.
6. 6Write transformations as pipe chains (group_by %>% summarize %>% filter), read as 'then'. This removes the need to name throwaway intermediate data frames. For complex logical steps, store interim values in new variables and check each step.
7. 7Iterate fast on big inputs. When reading a very large file, set n_max to 10,000 or 100,000 while you fix parsing problems. If parsing fails badly, fall back to read_lines() or read_file() and parse the strings yourself.
8. 8Do not use CSV as a cache for interim results. write_csv() loses type information (a date column comes back as character), so you must re-create the column specification on every load. Use a format that preserves types, or restate the specification when you reload.

Take the flights-by-destination analysis. The naive version names every intermediate data frame: one for the grouping, one for the summary, one for the filter. Naming them slows you down even though you never reuse them. Rewrite it as one chain: delays <- flights %>% group_by(dest) %>% summarize(count = n(), dist = mean(distance, na.rm = TRUE), delay = mean(arr_delay, na.rm = TRUE)) %>% filter(count > 20, dest != "HNL"). It reads as group, then summarize, then filter. The filter drops noisy destinations with few flights and Honolulu, which is almost twice as far as the next closest airport. The na.rm = TRUE arguments matter too. Without them, grouping by year, month, and day returns NA for every mean of dep_delay, which is rework you avoid by handling missing values at the summarize step. Put the script in an RStudio project, save plots with relative paths such as ggsave("diamonds.pdf"), and restart and rerun with Cmd/Ctrl-Shift-F10 then Cmd/Ctrl-Shift-S. The whole analysis then regenerates from scratch and works on any collaborator's machine.

• If a path starts with a drive letter, a leading slash, or setwd(), then replace it with a relative path inside an RStudio project.
• If you are about to name a data frame you will never reuse, then pipe the steps into one chain instead.
• If a summary returns NA, then add na.rm = TRUE (or deal with the missing values) before you suspect the data is broken.
• If a file is large or parsing is failing, then read only the first 10,000 to 100,000 rows with n_max until the problems are fixed, and use read_lines() or read_file() for exotic formats.
• If you need to save interim results and column types matter, then do not rely on CSV, because types are lost on reload.

• Working in whatever directory R defaults to, or calling setwd() with a personal path. The tell: the script errors on a collaborator's machine, or you cannot find the output files.
• Running code only in the console, or only partial chunks of a script. The tell: the script works in your live session but fails after a restart.
• Fighting messy data shape inside every function instead of tidying once. The tell: repeated reshaping code between each step and little time left for the question itself.
• Reloading CSV interim files and assuming types survive. The tell: a date column silently arrives as character and the column specification changes between runs.
• Treating the toolkit as covering everything. The tell: you spend hours forcing an unusual problem through the core tools instead of recognizing it as part of the remaining 20% and finding the right resource.

• ▸Project name and folder: the RStudio project name and its location in a sensible subdirectory.
• ▸Working directory check: paste the getwd() output and confirm it matches the project home.
• ▸Path audit: list any absolute paths or setwd() calls found and the relative replacement for each.
• ▸Tidy check: state what the variables and observations are, and whether each column is one variable and each row one observation.
• ▸Pipe chain: write the main transformation as group, summarize, filter steps with no throwaway names.
• ▸Missing-value handling: list each summary that needs na.rm and what is done about NA values.
• ▸Large-file iteration: set the n_max value used while debugging and note where it is removed.
• ▸Interim storage: record which format holds interim results and how column types are preserved.
• ▸The 20%: list the parts of the problem the core toolkit does not cover and the resource you will use for each.

1. 1Classify the mess before choosing a tool. The passages name four kinds: unstructured or semi-structured text (strings, handled with regexps), hierarchical or tree-like structure (lists), non-tidy tables (variables spread across columns, rows or cells), and sheer size (the flights join shows '3.368e+05 more rows'). Write down which kind you have, because each one points to a different part of the toolkit.
2. 2Test whether the data is tidy using the three rules: each value in its own cell, each variable in its own column, each observation in its own row. The source says 'all of the data that you've seen has been tidy. In real life, most data isn't tidy.' If all three rules hold, tidy practices pay off at full strength. If any fail, reshape first.
3. 3For text, build the regexp incrementally and check it with str_view(). Begin with an exact match ('an'), then move to '.', and escape literals. Remember the two escape layers: the regexp \. is written as the string "\\.", and a literal backslash needs four backslashes ("\\\\"). Use writeLines() to see the actual regexp the engine receives.
4. 4For nested data, inspect the shape with str() before you manipulate anything. It 'focuses on the structure, not the contents', so it shows you the depth and the mixed types (chr, int, num, logi) of a list. Use x[[i]] to pull out a single element and x[i] to keep the list wrapper. The source says this distinction matters most for lists.
5. 5When lists must sit alongside tabular data, put them in list-columns of a tibble. Make parallel arguments a data frame with tribble() so 'each column has a name and is the same length as all the other columns', then iterate with pmap(). When the function itself varies, store the function name and its params in the same tribble and call invoke_map().
6. 6Before unnesting, check the lengths. unnest(y, z) fails with 'All nested columns must have the same number of elements.' Use lengths() to confirm that the list-columns match row by row. Unnest columns that agree together, and unnest the rest separately.
7. 7Handle missing values explicitly, because messy data is full of NA. NA is contagious: NA == NA returns NA. Use is.na() to test for it. filter() drops both FALSE and NA, so write filter(df, is.na(x) | x > 1) when you want to keep the missing rows. Check this before you trust any row count.
8. 8Adjust for size and distribution rather than trusting a default view. Boxplots look roughly the same whatever the number of observations, so use varwidth = TRUE or cut_number() to keep bins comparable. Use a scatterplot to expose two-dimensional outliers that one-dimensional views hide. Convert model objects to tidy frames with broom::tidy(), glance() and augment() so that the output stays rectangular.

Take the params example from the list chapter. You want three random draws, each with a different mean, sd and n. Passing the arguments as loose lists is fragile because the order matters and the lengths can silently differ. Instead you build params <- tribble(~mean, ~sd, ~n, 5, 1, 1, 10, 5, 3, -3, 10, 5) and call params %>% pmap(rnorm). The data frame guarantees named columns of equal length, and the result is a list of three vectors of lengths 1, 3 and 5 (for example 4.68 for the first). You then raise the mess one more level: the function itself varies, so you add a tribble with columns f and params holding "runif" with list(min = -1, max = 1), "rnorm" with list(sd = 5), and "rpois" with list(lambda = 10). mutate(sim = invoke_map(f, params, n = 10)) stores the results in a list-column. The structure is now irregular (numeric and integer vectors, different arguments per row), yet it stays in one rectangular tibble. If you tried to unnest columns whose row lengths differ, as in df2 with y of length 1 and 2 and z of length 2 and 1, you would hit the 'same number of elements' error. That error is the signal that the data has outgrown a simple rectangle.

• If the data passes the three tidy rules (value per cell, variable per column, observation per row), then apply the integrated toolkit directly: the gain is highest here.
• If the data is text with an irregular pattern, then reach for stringr and regexps, building the pattern in str_view() one step at a time.
• If the data is hierarchical, then keep it in a list or list-column and iterate with map-family functions. Move it to a flat tibble only once the row lengths match.
• If you want to keep NA rows through a filter, then name them explicitly with is.na(); never rely on a comparison with NA.
• If a multipart expression in filter() gets complicated, then turn it into explicit variables, or simplify it with De Morgan's law or %in%, so you can check your work.

• Counting backslashes wrong in a regexp: the pattern matches nothing or the wrong things. The tell is that str_view() highlights nothing for a literal '.' or '\'. Fix it by printing the pattern with writeLines().
• Assuming NA behaves like a value: filter(df, x > 1) silently drops the NA row, and NA == NA returns NA. The tell is that your row count is lower than expected. Test with is.na() instead.
• Forcing nested data into a rectangle: unnest() on list-columns of unequal length throws 'All nested columns must have the same number of elements'. The tell is the error itself. Check lengths() first.
• Trusting a default plot on messy or uneven data: boxplots look the same regardless of how many points sit in each bin, and 1D views hide unusual combinations of x and y. The tell is a tidy-looking plot that conceals very different group sizes or outliers. Use varwidth, cut_number() or a scatterplot.
• Reading a mixed-type list as if it were an atomic vector: using [ where you needed [[ returns a list rather than the element. The tell is a list wrapper appearing around a value you expected to be a plain element.

• ▸Dataset and size: name the data and note its row and column counts, and whether size alone will slow you down.
• ▸Tidy test: record pass or fail for each of the three rules (value per cell, variable per column, observation per row).
• ▸Mess type: mark text, nested/hierarchical, non-tidy table, missing values, or large volume (more than one is allowed).
• ▸Missing-value plan: state how NA will be detected and whether filters must keep or drop it.
• ▸Structure check: paste the str() output for any list or list-column, and note its depth and mixed types.
• ▸Reshape or nest decision: say whether to reshape to rectangular form or keep it nested, and why.
• ▸Length check: for any unnest, record the lengths() result per row and whether the columns match.
• ▸Tool chosen: name the stringr, purrr, tidyr, broom or dplyr function you will use first.
• ▸Verification: note how you will confirm the result, for example str_view(), a row count, or a scatterplot.

1. 1Place tidying right after import and before transformation. The passages model the workflow as import, tidy, transform, then visualize and model. Do not start summarizing or plotting on data whose shape you have not checked.
2. 2Name the dataset's semantics first, in words, before touching the structure. List the variables (what is measured or described) and state what one observation is (for example, one diamond). The storage must match this list, not the layout of the source file.
3. 3Test the shape with three checks. Each column must be one variable. Each row must be one observation. Each cell must hold one value. Any column whose name or contents mix two variables, or any variable spread across several columns, fails the check.
4. 4Import with explicit types. Supply col_types, building up from the printout readr gives you (for example col_double() for x, col_date() for y). Each column then holds one consistently typed variable, and the import script is reproducible.
5. 5Make parsing failures loud. Add stop_for_problems() so the script stops on any parsing problem instead of silently producing malformed columns.
6. 6Reshape with code, never by hand. If you find an error in a data file, do not modify it directly. Write code that corrects the value and comment why. If you must record a small snippet of data, lay it out clearly with tibble::tribble().
7. 7Confirm that the tidy shape pays off. Check that nrow() equals the number of observations you expect, and that downstream functions (dplyr, ggplot2, modeling) work without special-case reshaping. Tidy data lets you spend effort on questions about the data, not on fighting its form.
8. 8Re-run the whole notebook from a clean state before you stop, so the tidying step is part of a reproducible record.

The passages' challenge.csv import shows the principle at the column level. Read with default guesses, the tail of the file shows y as <chr>, with values like 2019-11-21 and 2018-03-29 stored as text. The data holds one variable, a date, but the storage says character, so the structure does not match the semantics. The fix is to state the semantics in code: read_csv(readr_example("challenge.csv"), col_types = cols(x = col_double(), y = col_date())). The tail now prints x as <dbl> and y as <date>. The same data is now stored as two typed variables, one per column, with one observation per row. The text says to always supply col_types because, if you rely on guesses and the data changes, readr keeps reading it in anyway. Adding stop_for_problems() makes the script stop on any parsing problem. The row-equals-observation rule then pays off downstream: nrow(diamonds) returns 53940 because each row is one diamond, and nrow(diamonds) - nrow(smaller) gives 126 diamonds over 2.5 carats with no reshaping. Likewise broom::augment(model, data) returns one row per row of data, adding residuals and influence statistics.

• If a column holds the right values but the wrong type (dates as <chr>), then declare the type with col_types rather than converting later.
• If you are tempted to fix a bad value in the source file, then leave the file untouched and write a documented correction in code.
• If you need to enter a small amount of data inside R, then lay it out with tibble::tribble() so the rows and columns are readable. Otherwise do data entry outside R.
• If a function needs unusual reshaping to run on your data, then treat that as a sign the data is not tidy and fix the structure first.
• If the import must be reproducible and the file may change, then add stop_for_problems() so changes fail loudly instead of passing silently.

• Trusting the default column guesses. The tell is a column that prints as <chr> when it should be <date> or <dbl>, and a script that keeps running after the data changes.
• Editing the raw data file by hand. The tell is that nobody can explain why a value differs from the source, and the fix cannot be re-applied to a new version of the file.
• Starting transformation or plotting before checking the shape. The tell is that you are writing special-case code for each function instead of answering questions about the data.
• Defining the structure by how the file looks rather than what the data means. The tell is that you cannot state in a sentence what one row represents.
• Treating tidying as a one-off cleanup that is not recorded. The tell is that the notebook will not knit from a clean session, or the fix lives only in your memory.

• ▸Dataset name and source: where the data comes from (file, database, web API).
• ▸One row means: finish the sentence 'One row is one ___'.
• ▸Variables list: every variable, one per intended column.
• ▸Column-to-variable match: for each column, name the one variable it holds, or flag any column that mixes or splits variables.
• ▸Cell check: does every cell hold exactly one value? List the exceptions.
• ▸col_types declared: the type given to each column (for example col_double(), col_date()).
• ▸Parsing safeguard: is stop_for_problems() in the script? yes/no.
• ▸Corrections made in code: each value fixed, the code location, and the reason.
• ▸Verification: nrow() result versus the expected observation count, and the first downstream function run without special-case reshaping.

1. 1Install the whole set once with install.packages("tidyverse") in the console. Get R from the cloud mirror https://cloud.r-project.org rather than picking a mirror yourself. Keep R and RStudio current: a major R release arrives yearly and forces a reinstall of all packages, so postponing makes the upgrade harder.
2. 2Start each script with library(tidyverse). Read the startup message: it lists the core packages loaded (ggplot2, tibble, tidyr, readr, purrr, dplyr) and the conflicts, such as filter() and lag() masking stats functions. Treat the conflict list as a to-do: when you mean the stats version, call it as stats::filter().
3. 3Follow the pipeline order the packages are built around: import (readr), tidy (tidyr), transform (dplyr), visualize (ggplot2), model, communicate (R Markdown). Programming (purrr and your own functions) cuts across every stage. Pick the tool for the stage you are in instead of reaching for a base-R workaround.
4. 4Make the data tidy before any other step: each column a variable, each row an observation. The shared structure is what lets every tidyverse function accept your data without special reshaping, so you spend effort on questions rather than on forcing data into each function's form.
5. 5Import with readr and always supply col_types, building the specification up from the printout readr gives you (for example cols(x = col_double(), y = col_date())). Add stop_for_problems() if you want the script to halt on parsing problems. Your import then does not depend on default guesses that change when the data does.
6. 6Work in tibbles, the tidyverse's opinionated data frames, rather than the traditional data.frame. Hold the same objects from import through dplyr verbs to ggplot2 so nothing needs conversion between stages.
7. 7When you need to name the package a function comes from, use the pkg::fun() form (dplyr::mutate(), nycflights13::flights). Do this to resolve conflicts and to make a script's dependencies readable.
8. 8Bring in a non-tidyverse package only where the tidyverse stops (the book's rough 80-20 rule says the tools cover about 80% of a project). Example: broom::augment(model, data) returns one row per row of data, plus residuals and influence statistics, and fits the tidy-data pattern, so it plugs into the same workflow.

The readr challenge.csv example shows the toolkit's discipline at the import stage. You read it with read_csv(readr_example("challenge.csv")) and let readr guess. Column y comes back as <chr> because the default guess looks at only the first 1000 rows, and those rows give no evidence it is a date. The fix is to state the types: col_types = cols(x = col_double(), y = col_date()). The tail then shows y as <date>, with values like 2019-11-21. A second route, guess_max = 1001, works because one more row is enough for readr to parse correctly. The authors still recommend always supplying col_types. If you rely on the defaults and the data changes, readr keeps reading it in without complaint. When you want a hard stop, stop_for_problems() throws an error on any parsing problem. For diagnosis you can read everything as character with .default = col_character() and apply type_convert(). The result is a typed tibble that flows straight into dplyr and ggplot2 with no ad hoc cleanup.

• If library(tidyverse) reports a conflict (filter(), lag()) and you intend the base or stats function, then call it as stats::filter(); otherwise assume the dplyr version.
• If a file's columns are parsed wrongly or you need a reproducible script, then write explicit col_types from readr's printout instead of trusting the guess; add stop_for_problems() when failures must halt the run.
• If a function demands awkward reshaping of your data, then stop and tidy it first (one variable per column, one observation per row) rather than adapting the function call.
• If a package you want sits outside the tidyverse, then use it only if it returns or accepts tidy data frames (as broom::augment() does), and namespace it with pkg::fun().
• If you are tempted to patch a gap with a base-R idiom mid-pipeline, then check first whether the tidyverse already has a verb for that stage.

• Relying on read_csv's default type guesses. The tell: a column that should be a date arrives as <chr>, or a script that ran last month silently reads changed data.
• Ignoring the conflicts message at load. The tell: filter() or lag() behaves differently than expected because stats masks or is masked by dplyr, and you never namespaced the call.
• Doing analysis on untidy data and fighting each function. The tell: repeated manual reshaping before every plot or summary, with different layouts for different functions.
• Mixing in ad hoc, philosophically inconsistent tools where tidyverse equivalents exist. The tell: a pipeline that alternates between data.frame and tibble behaviors and needs conversions between steps.
• Putting off R upgrades. The tell: a major version jump forces reinstalling every package at once, and the older environment has drifted from current tidyverse behavior.

• ▸Project/script name: identify the analysis you are auditing.
• ▸Setup line: confirm the script starts with library(tidyverse) and record any conflicts reported (e.g., filter(), lag()).
• ▸Import step: list each readr call and whether col_types is explicitly supplied (yes/no).
• ▸Import safeguard: note whether stop_for_problems() or an equivalent check is used, and why.
• ▸Tidy check: state what each column (variable) and each row (observation) is in your main table.
• ▸Transform step: list the dplyr verbs used and any base-R workarounds that could be replaced.
• ▸Visualization step: name the ggplot2 plots and confirm they take the tibble directly with no conversion.
• ▸Non-tidyverse packages: list each one and how its output reconnects to tidy data (e.g., broom::augment()).
• ▸Namespacing: record where pkg::fun() is used to resolve conflicts or document dependencies.

1. 1Spot the repetition before you fix it: find lines that differ only by a variable name. The passage's trigger is four near-identical rescaling lines for df$a, df$b, df$c, df$d, where the copy-paste slip was 'I forgot to change an a to a b'.
2. 2Count the inputs. Read the repeated expression and list what actually varies. In (df$a - min(df$a, na.rm = TRUE)) / (max(df$a, na.rm = TRUE) - min(df$a, na.rm = TRUE)) there is one input, df$a. TRUE is not an input.
3. 3Rewrite the code with a temporary variable that has a general name: x <- df$a, then use x everywhere. This makes the inputs explicit before any function exists.
4. 4Collapse repeated intermediate calculations into named variables. The range was computed three times, so replace it with rng <- range(x, na.rm = TRUE) and use rng[1] and rng[2]. Re-run it and confirm the output is unchanged (0.289 0.751 0.000 ...).
5. 5Only now wrap the working code: pick a name (rescale01), list the arguments inside function(), and put the code in the { } body. Start from working code, because it is easier than writing a function and then debugging it.
6. 6Test with varied inputs: c(0, 5, 10), c(-10, 0, 10), and c(1, 2, 3, NA, 5). Expect 0.0 0.5 1.0 for the first two and 0.00 0.25 0.50 NA 1.00 for the third. Informal checks like these are what later become formal unit tests.
7. 7Replace each original copy with a call (df$a <- rescale01(df$a), and so on). Then note the remaining repetition across columns, which iteration tools handle in Chapter 17 once you know the data structures from Chapter 16.
8. 8When requirements change, edit the function body once. Adding finite = TRUE to range() fixes the Inf failure everywhere the function is called.

The source starts with a rescaling formula copied for columns a, b, c and d, and one copy has an error: an a was never changed to a b. You count the inputs and find one, df$a, so you set x <- df$a. The formula computes min and max of x three times, so you replace that with rng <- range(x, na.rm = TRUE) and (x - rng[1]) / (rng[2] - rng[1]). The output still matches (0.289 0.751 0.000 ...), so you wrap it as rescale01 <- function(x) { ... }. rescale01(c(0, 5, 10)) returns 0.0 0.5 1.0, and c(1, 2, 3, NA, 5) returns 0.00 0.25 0.50 NA 1.00. Later, x <- c(1:10, Inf) makes rescale01() return NaN in the last position and 0 elsewhere. You change one line to range(x, na.rm = TRUE, finite = TRUE), and the result becomes 0.000 0.111 ... 1.000 Inf. The four calls to rescale01() pick up the fix automatically. The same logic applies in ggplot2: geom_point(aes(x = displ, y = hwy)) + geom_smooth(aes(x = displ, y = hwy)) repeats the mappings, so you move them into ggplot(data = mpg, mapping = aes(x = displ, y = hwy)). Switching hwy to cty then means editing one place.

• If two or more lines differ only by a variable name, extract a function and make that varying part its argument.
• If the same intermediate value is computed more than once inside a block (the range three times), name it once before you write the function.
• If you cannot yet make the code work on a simple input, do not write the function yet. Get it working first, then wrap it.
• If several layers in a ggplot share the same mappings, put them in ggplot() as global mappings. Use local mappings in a geom only for what differs, such as aes(color = class), which extends or overrides the global ones for that layer.
• If the function call itself is still repeated across many columns, treat that as remaining duplication and use iteration (Chapter 17) instead of more copies.
• If a small repetition makes an important operation clearer, tolerate it. The author prefers repeating an object name in assignment over %<>% because assignment should be explicit.

• Partial update after copy-paste: one copy keeps a stale variable name (a instead of b) and gives silently wrong results. The tell is that copies of the same formula differ in a column reference.
• Extracting a function from code you have not yet made work, then debugging inside the function. The tell is that you cannot say what output a simple input should give.
• Mis-identifying inputs, such as making TRUE a parameter or leaving df$a hard-coded inside the body. The tell is that the function only works on one column or has arguments that never change.
• Declaring victory after replacing the formula with a function while the near-identical call lines remain. The tell is df$a, df$b, df$c, df$d lines that still differ only by a name.
• Skipping tests on edge inputs (NA, Inf, negative values). The tell is a function that returns NaN or unexpected values, as rescale01() did with Inf before the finite = TRUE fix.
• Repeating the same mapping in every ggplot layer. The tell is that changing hwy to cty requires edits in several places and you may forget one.

• ▸Repeated snippet: paste the lines that look alike.
• ▸What varies between copies: list the parts that differ (for example the column name).
• ▸Inputs identified: name each input and give it a general temporary name (x).
• ▸Internal repeats to name: list sub-calculations computed more than once (such as range) and the variable that will hold each.
• ▸Proposed function name: choose an evocative name that says what it does (rescale01).
• ▸Simple test input and expected output: give one input you can verify by hand.
• ▸Edge-case tests: list NA, Inf, negative values and the output you expect for each.
• ▸Where it will be called: list the call sites replacing each copy.
• ▸Remaining duplication: note any repeated calls left over and the plan for iterating over them.

1. 1Start with visualization and transformation on data that is already imported and tidied. The book's rationale: ingest and tidying are 80% routine and boring and 20% weird and frustrating, which is a bad place to start learning. Begin where the payoff is clear.
2. 2Run library(tidyverse) and make your first ggplot2 plot early. Visualization is the best entry point because you get elegant, informative plots that help you understand data. Record what you learned from that first plot as your payoff memory.
3. 3Treat the payoff as a stored reason to persist. When you later ingest and tidy your own data, your motivation stays high because you know the pain is worth it. Before starting an ingest task, name the plot or question the cleaned data will feed.
4. 4Expect frustration and label it as normal. R is a stickler for punctuation and one misplaced character makes it complain. The source calls this typical and temporary: it happens to everyone, and the only way past it is to keep trying.
5. 5Keep the loop short: look at the data, generate hypotheses, test them quickly, repeat. Data exploration is meant to produce many promising leads, so each cycle should end with something you can see.
6. 6Defer what you cannot yet use. Modeling waits until Part IV, once you have more wrangling and programming tools. Programming is a long-term investment that does not pay off immediately, so do not expect instant returns from it.
7. 7Do the exercises. The source admits it is tempting to skip them but says there is no better way to learn than practicing on real problems.
8. 8Keep your code organized from the start, using the workflow chapters (2, 4, 6) on writing and organizing R code. That prepares you to stay organized when you tackle real projects, which is where tedium peaks.

The book's own sequence is the example. A learner who starts with data ingest and tidying meets a subject that is 80% routine and boring and 20% weird and frustrating, and is likely to quit. The authors reverse the order. In Chapter 1 you load the tidyverse with one line, library(tidyverse), and build ggplot2 plots from data that is already imported and tidied. You see results immediately and understand the data better. Chapter 5 then adds your curiosity and skepticism to ask and answer interesting questions. When you later hit a one-character punctuation error while importing your own file, you have a concrete memory of what the tidy data will make possible. The book states the effect directly: your motivation will stay high because you know the pain is worth it. The frustration is still there, but it is typical and temporary.

• If you are about to begin with ingest and tidying of a new dataset, then first produce one plot from an already-tidy dataset so the payoff is fresh.
• If R rejects your code over a single character, then treat it as normal, check punctuation, and keep trying rather than abandoning the task.
• If you feel pulled toward modeling before you have wrangling and programming skills, then defer it to after Part IV.
• If you are tempted to skip an exercise, then do it anyway, because practice on real problems is the best way to learn.
• If a task feels tedious and unrewarded, then restate what visualization or question it enables before continuing.

• Starting with ingest and tidying. The tell: you spend your first sessions on routine or weird data problems and feel no sense of progress.
• Reading frustration as a sign you are unsuited. The tell: you want to quit after a punctuation error, though the source says it happens to everyone.
• Expecting immediate payoff from programming study. The tell: you drop programming because it does not pay off right away, though it is a long-term investment.
• Skipping exercises because reading feels like enough. The tell: you recognize the code but cannot write it unaided.
• Jumping to modeling before you have the tools. The tell: you apply models without the visualization, tidy-data, and programming background that makes them understandable.

• ▸Upcoming tedious task: name the ingest or tidying job you are about to do.
• ▸Payoff memory: describe the plot or insight you already got from visualization.
• ▸Target payoff: state the plot or question the cleaned data will feed.
• ▸Expected frustration: note the likely errors, such as punctuation mistakes.
• ▸Normalizing reminder: write 'typical and temporary; keep trying'.
• ▸Next visible output: name the first plot you will make once the data is tidy.
• ▸Exercise to complete: pick one exercise to do rather than skip.
• ▸Deferred item: list what you are postponing, such as modeling, until you have the tools.

1. 1Name the one question before you write code. State it as a sentence about the data (e.g., 'How does average delay relate to distance for each destination?'). Every later line either serves that sentence or goes in the bin.
2. 2Write the transformation as a pipe chain, not as named intermediates. Use `flights %>% group_by(dest) %>% summarize(...) %>% filter(...)` and read it aloud as 'group, then summarize, then filter'. This removes the burden of naming throwaway data frames ('Naming things is hard, so this slows down our analysis') and puts the focus on the transformations rather than on what is being transformed.
3. 3Spot duplication and extract the one thing that varies. When you see `col_mean()`, `col_median()` and `col_sd()` differing only in the summary function, turn that function into an argument (`col_summary(df, fun)`). The same logic applies to `f1`, `f2`, `f3`, which collapse to `f(x, i)`. Less code means fewer bugs and a clearer view of what actually changes.
4. 4Replace boilerplate loops with `map_*()` when the loop body is only bookkeeping. `df %>% map_dbl(mean)` puts attention on the operation (`mean`, `median`, `sd`) instead of on `vector()`, `seq_along()` and index handling. Solve the problem for a single element first, then let purrr generalize it.
5. 5Make missing values an explicit decision at the point of aggregation. If a summary returns `NA` for all 365 days, don't hunt through the code. Aggregation functions return `NA` when any input is `NA`, so add `na.rm = TRUE` deliberately and know why you did.
6. 6Isolate surprises with a zoom, then pluck them out with a filter. Use `coord_cartesian(ylim = c(0, 50))` to zoom into rare bins without throwing data away. Then use `filter(y < 3 | y > 20) %>% arrange(y)` to list the odd rows. Look at the 9 rows rather than guessing at the chart.
7. 7Test the effect of oddities by repeating the analysis with and without them. If the results barely change and you can't explain the values, replace them with missing values and move on. If the results change substantially, find the cause before dropping anything.
8. 8Use the stat/position vocabulary to resolve confusion about plots. When a plot shows something you didn't supply (e.g., `count` on a bar chart's y-axis), check the default `stat` in `?geom_bar` (`stat_count()`). When points hide each other (126 visible of 234 observations), choose `position = "jitter"` instead of guessing.

The source's flights example shows the shift in load. The first version creates `by_dest`, then `delay`, then overwrites `delay` with a filtered copy, so you must track three object names that you don't care about. The rewrite is `delays <- flights %>% group_by(dest) %>% summarize(count = n(), dist = mean(distance, na.rm = TRUE), delay = mean(arr_delay, na.rm = TRUE)) %>% filter(count > 20, dest != "HNL")`. It reads as three imperatives: group by destination, summarize distance, delay and count, then filter out noisy points (count of 20 or fewer) and Honolulu, which is almost twice as far away as the next closest airport. If you drop `na.rm = TRUE` from the `mean()` calls, you get `NA` for every group. That is the standard missing-value rule, not a bug in your logic. With the pipe and the explicit `na.rm`, your attention goes to the substantive finding: delay rises with distance up to about 750 miles and then falls. The question 'maybe longer flights make up delays in the air?' is then the only thing left to think about.

• If you are naming an intermediate data frame you never reuse, then use a pipe chain instead.
• If two or more blocks of code differ in only one value or function, then extract that thing into an argument rather than copy-pasting a third time.
• If a for loop is mostly index and storage bookkeeping, then use `map_*()` and describe only the per-element operation.
• If a summary returns `NA` where you expected numbers, then suspect missing inputs and add `na.rm = TRUE` deliberately before changing anything else.
• If unusual values appear, then run the analysis with and without them. Replace them with `NA` only when results barely change and you can't explain them. Otherwise investigate the cause first.

• Copy-pasting a loop and editing one word. The tell is several near-identical functions (`col_mean`, `col_median`, `col_sd`) where you can't see at a glance what differs.
• Naming every intermediate step (`by_dest`, `delay`, then `delay` again). The tell is that you lose track of which version of the object is current, and you spend time on names instead of the question.
• Treating `NA` output as a coding error and rewriting the pipeline. The tell is a column full of `NA` after `summarize()` with no `na.rm` set.
• Dropping outliers without testing their effect. The tell is that you removed values like 0 mm, 32 mm or 59 mm in `y` without ever comparing results with and without them.
• Being puzzled by plot output you didn't supply, such as a `count` axis or fewer points than observations. The tell is that you can't say which stat or position adjustment produced it; the fix is to check the default `stat` or try `jitter`.

• ▸Question sentence: write the single data question this script answers.
• ▸Intermediate objects: list every named intermediate data frame; mark which ones are never reused and can become pipe steps.
• ▸Pipe chain read-aloud: write the chain as 'X, then Y, then Z' and confirm each step is one imperative.
• ▸Duplicated blocks: list code blocks differing in only one value or function, and name the argument that would replace the difference.
• ▸Loops to replace: list each for loop and note whether the body is bookkeeping (replace with `map_*()`) or real logic.
• ▸Missing-value handling: record each aggregation and whether `na.rm = TRUE` was set on purpose, and why.
• ▸Unusual values: note any zoom or filter used to find odd values, and how many rows turned up.
• ▸With/without comparison: state the result with outliers, the result without, and your decision (keep, replace with NA, or investigate).
• ▸Remaining unknowns: list anything on the plot or in the output you cannot yet explain (stat, position, grouping).