Python for Data Analysis

1. 1Name the data form before you touch it. The source lists four: tabular data where each column may have its own type (string, numeric, date), multidimensional arrays, multiple tables linked by key columns (primary/foreign keys), and evenly or unevenly spaced time series. Write down which form or forms you have, because each one implies different preparation.
2. 2State the end use first. The source groups work into interacting with the outside world, preparation, transformation (e.g., aggregating a large table by group variables), modeling, and presentation. Analysis-ready means the data supports the specific downstream group you picked: an aggregation, a chart, or a model input.
3. 3Load from the outside world. Read from the file format or data store you actually have (comma- or tab-delimited text, relational database). After loading, record each column's type, since columns in tabular data may differ in type.
4. 4Run the preparation tasks the source names, in an order you choose and document: clean, munge, combine, normalize, reshape, slice and dice, transform. Keep the target in view. Combine tables through their key columns, and reshape until each unit you will group or plot by is addressable.
5. 5Put the steps in a .py script and run it with %run in IPython. %run starts from an empty namespace, so the result matches `python script.py`. Variables defined up to the point of any exception remain accessible afterward, so you can inspect where it stopped.
6. 6Use plain %run for reproducibility. Use %run -i only when you deliberately need variables already in the interactive namespace, and record that dependency.
7. 7Watch the cost of your building blocks. When assembling a large collection from chunks, call extend on one list rather than repeatedly adding lists with +, which creates a new list and copies every object each time. For repeated membership checks, use a dict or set (constant-time hash lookup), not a list (linear scan).
8. 8Verify with the end use itself. Perform the aggregation, chart, or summary the data was prepared for and check that it runs and gives plausible output. If it needs another cleanup step, the data was not yet analysis-ready.

The source gives a small, concrete case in the loading-and-combining stage: assembling many chunks into one collection. Given `list_of_lists`, the efficient form is `everything = []` then `for chunk in list_of_lists: everything.extend(chunk)`. The alternative, `everything = everything + chunk`, is slower because each addition builds a new list and copies the objects over. Suppose each chunk is one loaded piece of a larger dataset. With extend you get one combined sequence without repeated copying. The same logic applies to checking values: the source notes `'dwarf' in b_list` scans the list linearly, while dicts and sets check in constant time. Put the assembly in a script and run it with %run so it executes in an empty namespace. The source's own test script shows the pattern: after `%run ipython_script_test.py`, you inspect `c` (7.5) and `result` (1.4666666666666666) directly in the shell. Finish by running your intended aggregation on the result. The source does not specify particular pandas calls, so choose them to fit your data form.

• If the data is several tables joined by key columns, then combine on those keys before you aggregate or plot, because the source treats multi-table data as a distinct form.
• If you are building a large collection from chunks, then use extend on a single list instead of repeated + concatenation.
• If you run many membership checks against a large collection, then store it as a set or dict, because a list does a linear scan.
• If you need a clean, repeatable run, then use %run without -i. If the script must see interactive variables, then use %run -i and document it.
• If the intended end use is a group aggregation, then confirm the grouping columns are present and typed correctly before declaring the data ready.

• Hidden interactive state: the script works only because %run -i or earlier cells defined variables. The tell is that it fails on a fresh `python script.py` run.
• Quadratic-style slowdown when assembling data: chunks are joined with `everything = everything + chunk` in a loop. The tell is that runtime balloons as the chunk count grows.
• Treating a tabular column as uniform when columns may be strings, numerics, or dates. The tell is that aggregation or sorting errors out or gives odd results on a column you assumed was numeric.
• Declaring done after loading and cleaning without testing the end use. The tell is that the first aggregation or visualization reveals reshaping or key-matching problems.
• Assuming Ctrl-C will stop a long load. When code has called into compiled extension modules, it may not stop immediately. The tell is that the process keeps running after KeyboardInterrupt, and you must wait or forcibly terminate it.

• ▸Data form: tabular / multidimensional array / multi-table with keys / time series (even or uneven).
• ▸Source and format: the file format or data store you load from.
• ▸Intended end use: aggregation by group, visualization, or modeling input.
• ▸Column inventory: name and type (string, numeric, date, other) for each column.
• ▸Preparation steps applied: cleaning, combining, normalizing, reshaping, transforming. List each and its order.
• ▸Key columns for combining: the key used to join each pair of tables.
• ▸Script and run mode: filename and whether run with %run or %run -i, with the reason.
• ▸Performance check: chunk assembly uses extend, and large lookups use sets or dicts.
• ▸End-use verification: the aggregation or chart you ran and the result you expected.

1. 1Name the task group before you write code. The source sorts data work into five groups: interacting with the outside world (reading and writing files and data stores), preparation, transformation, modeling and computation, and presentation. Insight extraction lives in the last three, so confirm that preparation is finished first.
2. 2Confirm the data is in a structured form. The source lists tabular data, multidimensional arrays, multiple tables linked by key columns, and evenly or unevenly spaced time series. If your data is none of these, extract features into one. The source's example turns a collection of news articles into a word frequency table, which can then feed sentiment analysis.
3. 3Set up the standard environment and follow the import conventions: import numpy as np, import matplotlib.pyplot as plt, import pandas as pd. Work in an IPython shell or Jupyter notebook so every input and output stays visible and re-runnable.
4. 4Produce summaries through transformation. Apply mathematical and statistical operations to groups of data to derive new datasets, such as aggregating a large table by group variables. Use the derived table, not the raw one, as the object you interpret.
5. 5Convert repeated text labels to categoricals where it fits. The source notes that groupby performs better on categoricals. Use ordered=True, or as_ordered(), when the categories have a real order (foo < bar < baz). Use pandas.qcut or pandas.cut to bin numeric data into a categorical.
6. 6Make the presentation step explicit. Produce an interactive or static graphical visualization, or a textual summary, from the derived table. Keep it tied to the group-level result so a reader can check it against the table.
7. 7Choose the modeling library by question type. scikit-learn covers classification, regression, clustering, dimensionality reduction, model selection, and preprocessing. statsmodels covers classical (primarily frequentist) statistics and econometrics: linear regression, GLMs, ANOVA, and time series models such as AR, ARMA, ARIMA, and VAR. Connect your cleaned data to the chosen tool.
8. 8Make the work reproducible. Keep the notebook or %run script and re-run it end to end after any correction. If a calculation turns out wrong, fix the script and rerun it, using command-history search (Ctrl-P, Ctrl-R) to repeat the same %run quickly.

The source's own illustration of insight extraction is the news-article case. You start with a collection of news articles, which is unstructured text and not directly analyzable. You process it into a word frequency table, which is structured data with one row per article or term and numeric counts. That table is the readiness step: it can now be summarized, grouped, and visualized. Next you use it for sentiment analysis, one of the modeling and computation tasks the source lists. For the modeling step, the passages point to scikit-learn for classification and clustering, or statsmodels for classical regression and time series. The output is a textual summary or a chart of the sentiment results. The sequence is feature extraction, then structured table, then group-level transformation, then model, then presentation. The source does not give numeric results for this example, so judge success by whether each stage produces an object the next stage can accept.

• If the data is not tabular, an array, linked tables, or a time series, then extract features into a structured form (for example a word frequency table) before attempting any summary or model.
• If you aggregate by a repeated label column and need speed, then convert it to a categorical, since groupby performs better on categoricals.
• If categories have a meaningful order, then mark the categorical as ordered (ordered=True or as_ordered()) so order-aware functions can use it.
• If the question is classification, clustering, or dimensionality reduction, then reach for scikit-learn. If it is classical statistical inference, ANOVA, or ARIMA-style time series, then reach for statsmodels.
• If a result looks wrong, then fix the script or cell, rerun the full sequence, and do not patch the output by hand.

• Modeling before preparation is complete. The tell: messy or mixed-type columns, or unstructured text, are passed straight to a model and the output is meaningless or errors out.
• Mismatching the library to the question. The tell: you fit a classical regression with ANOVA-style questions in scikit-learn, or expect machine-learning model selection from statsmodels, when each package has a different focus.
• Forgetting to assign a result to a variable. The tell: you must recompute an expensive step. IPython's stored input and output variables can recover it, but better to assign up front.
• Treating an unordered categorical as ordered. The tell: binned or labeled groups sort alphabetically or by appearance, and the order-based result is wrong. Set ordered explicitly.
• Presenting the raw table instead of a derived summary. The tell: the chart or text shows many unaggregated rows and no group-level conclusion a reader can state in one sentence.

• ▸Question: state in one sentence what you want to learn from the data.
• ▸Data form: tabular, array, linked tables by key, or time series? If none, name the features you will extract.
• ▸Preparation status: list the cleaning, combining, and reshaping steps completed and any still open.
• ▸Grouping variable(s): which columns define the groups you will aggregate by?
• ▸Derived table: describe the aggregated or transformed dataset you will interpret.
• ▸Categorical handling: which columns become categoricals, and are they ordered?
• ▸Presentation form: chart (static or interactive) or textual summary, and what it should show.
• ▸Modeling library: scikit-learn or statsmodels, and the specific model family you will use.
• ▸Reproducibility: name the script or notebook and confirm it reruns end to end.

1. 1Work in IPython or Jupyter, not in a script you run blind. Write a piece, run it, inspect the output, then write the next piece. The book's author describes his own setup as 'IPython plus a text editor', with each piece tested and debugged iteratively and the data manipulations checked visually.
2. 2Start with NumPy before pandas. NumPy supplies the ndarray, fast whole-array operations without loops, and broadcasting. Pandas inherits these array-oriented semantics, so understanding arrays makes you much more effective with pandas.
3. 3Replace loops with array operations. Express the computation as one operation on the whole array, then use axis arguments to control direction. On a 3x3 array, `arr.cumsum(axis=0)` accumulates down the rows and gives `[[0,1,2],[3,5,7],[9,12,15]]`. `arr.cumprod(axis=1)` accumulates across the columns and gives `[[0,0,0],[3,12,60],[6,42,336]]`. Predict the output before you run it.
4. 4Learn the statistical methods in Table 4-5 as one family: sum, mean, std, var, min, max, argmin, argmax, cumsum, cumprod. Note the edge behaviors: a zero-length array has sum 0 and mean NaN, and std/var default to denominator n.
5. 5Use boolean coercion to count and test. True becomes 1 and False becomes 0, so `(arr > 0).sum()` counts positive values. Use `any` to test whether at least one value is True and `all` to test whether every value is. Both also work on non-boolean arrays, where non-zero counts as True.
6. 6Move to pandas Series and DataFrame for tabular data, and let the defaults work for you. Descriptive statistics on pandas objects exclude missing data by default. Chain operations, such as `rolling(...).mean()`, `ewm(span=...)`, `pct_change()` and `.corr(...)`, instead of hand-writing loops.
7. 7Check what each operation did to your data. Confirm that `arr.sort()` is in-place and changes the array itself. Look at the `Out[]` cells and plots, and treat any surprise as a gap in your understanding.
8. 8Read the errors and the documentation. Learn to read a traceback such as the `UnicodeDecodeError` that ends with 'can't decode byte 0xb1 in position 0'. Use conda to update conda-installed packages, because updating them with pip can cause environment problems. Follow the pandas release notes as the library changes.

The book computes a rolling correlation between Apple's returns and the S&P 500. First, `spx_rets = spx_px.pct_change()` and `returns = close_px.pct_change()` turn price series into returns. Then `returns.AAPL.rolling(125, min_periods=100).corr(spx_rets)` gives a six-month rolling correlation for one stock, and `corr.plot()` draws it. The book then shows that the same call on the whole DataFrame, `returns.rolling(125, min_periods=100).corr(spx_rets)`, computes the correlation of the S&P 500 with every stock column at once, with no loop and no hand-built DataFrame. Mastery shows in four moves: knowing `pct_change` returns a Series or DataFrame aligned to the original index, knowing `rolling` takes a window and `min_periods`, knowing a Series passed to `corr` is applied against each DataFrame column, and checking the plot in the notebook right away. If you had to write this as a loop over stocks, that signals a gap in your grasp of the array-oriented semantics.

• If you are about to write a Python for-loop over numbers, then first look for a NumPy or pandas whole-array operation, because avoiding loops is the point of the ndarray.
• If you need to count or test a condition, then build a boolean array and call `sum`, `any` or `all` on it.
• If a result is wrong in a multidimensional array, then check the axis argument first (axis=0 down the rows, axis=1 across the columns).
• If you need to install or upgrade a package in an Anaconda or Miniconda environment, then try conda first and do not update conda-installed packages with pip.
• If you are testing a new data manipulation, then run it in IPython or Jupyter and inspect the output before putting it in a script.

• Writing loops where one vectorized call would do. The tell: element-by-element code over arrays, or a loop that builds a DataFrame column by column when a single `rolling(...).corr(...)` handles all columns.
• Misreading axis direction in cumulative or aggregate methods. The tell: `cumsum(axis=0)` and `cumsum(axis=1)` seem interchangeable to you, or the totals come out transposed from what you expected.
• Forgetting that sort is in-place. The tell: `arr.sort()` returns nothing, and the original ordering is gone when you look at `arr` later.
• Treating text encoding errors as random. The tell: a `UnicodeDecodeError` ('invalid start byte') on a file opened with the wrong encoding, and no idea where to look. Non-ASCII text work calls for learning Python's Unicode functionality.
• Breaking the environment with mixed package managers or an old Python. The tell: an environment that misbehaves after you updated a conda package with pip, or Python 2.7 code in a new project (the book uses Python 3.6).

• ▸Task: one-line description of the data operation you are doing.
• ▸Loop check: Did you write a loop? If yes, name the NumPy or pandas operation that replaces it.
• ▸Axis prediction: State the expected output shape and values of any axis-based method before running it.
• ▸Boolean check: List any counts or tests done with `sum`, `any` or `all` on a boolean array.
• ▸Missing-data behavior: Note how NaN is treated by each statistic you call (excluded by default in pandas).
• ▸In-place vs copy: Mark each operation as in-place or returning a new object, and confirm it.
• ▸Notebook verification: Paste the `Out[]` result or plot you inspected and say what you concluded.
• ▸Environment note: Python version and package manager used (conda or pip), and any upgrade performed.
• ▸Gap log: Any error or surprise, plus the documentation page you consulted.

1. 1Name the loop before you replace it. Write down what the loop does per element: a scalar transform (sqrt, exp), a pairwise comparison between two arrays (maximum), a conditional pick, or a group summary. The passages tie each category to a vectorized tool: ufuncs, boolean indexing, array expressions for conditional logic, and group-wise operations.
2. 2Move the data into an ndarray first. The passages say NumPy arrays are the efficient container for numerical data and that tools like pandas share array-oriented semantics. A Python list of numbers will not give you the speedup.
3. 3Replace scalar-by-scalar functions with ufuncs. Use unary ufuncs like np.sqrt(arr) and np.exp(arr) in place of a loop over arr. Use binary ufuncs like np.maximum(x, y) in place of a loop with if x[i] > y[i]. They are described as fast vectorized wrappers for functions that take scalars and return scalars.
4. 4Turn each if-test into a boolean array. Comparisons are vectorized just like arithmetic, so names == 'Bob' returns one boolean per row. Use != or ~(condition) for the complement.
5. 5Select with the boolean array directly: data[names == 'Bob']. Mix in slices or integers to pick columns in the same expression, as in data[names == 'Bob', 2:] or data[names == 'Bob', 3]. Check that the boolean array has the same length as the axis you are indexing.
6. 6Express if-elif-else branching as array expressions, not loops. The book lists this as a core NumPy use case. Build one boolean mask per branch and apply them to the whole array.
7. 7Use the library's own algorithms for sorting, unique, set operations, descriptive statistics and group-wise work. Do not hand-roll these with dicts and counters. Prefer pandas for tabular data, group aggregation, merging and joining.
8. 8Review the result against a small slice. Run the loop version on a few rows, compare it with the vectorized output, then delete the loop.

The book's example has seven names and a 7x4 array of random data: names = np.array(['Bob','Joe','Will','Bob','Will','Joe','Joe']) and data = np.random.randn(7, 4). The goal is to select every row whose name is 'Bob'. The loop version iterates over the seven positions, tests each name, and appends matching rows to a list. The vectorized version is one expression: names == 'Bob' yields array([True, False, False, True, False, False, False]). Passing that to data[names == 'Bob'] returns rows 0 and 3, [0.0929, 0.2817, 0.769, 1.2464] and [1.669, -0.4386, -0.5397, 0.477]. To keep only the last two columns you write data[names == 'Bob', 2:], which gives [[0.769, 1.2464], [-0.5397, 0.477]]. data[names == 'Bob', 3] gives [1.2464, 0.477]. To select everyone except Bob you write data[~(names == 'Bob')], which returns the other five rows. The test, the row filter and the column slice all happen in a single indexing expression with no Python-level iteration.

• If the loop body applies one scalar function to every element, use a ufunc such as np.sqrt, np.exp or np.add instead of looping.
• If the loop body compares two arrays element by element and keeps one, use a binary ufunc like np.maximum(x, y).
• If the loop contains if/elif/else on array values, build boolean arrays and write an array expression instead.
• If you need the complement of a condition, use != or ~(cond) on the boolean array, not a second loop.
• If the data is tabular or needs grouping, merging or time-series handling, move from raw NumPy to pandas rather than writing loops over rows.

• Looping over a NumPy array element by element, such as for i in range(len(arr)), and calling scalar functions. The tell is that a ufunc already exists for the operation and the code is many lines long.
• A boolean mask whose length differs from the axis it indexes. The tell is an error or wrong rows, because the mask must match the axis length.
• Forgetting parentheses when negating a comparison, writing ~names == 'Bob' instead of ~(names == 'Bob'). The tell is a wrong or errored selection.
• Leaving data in plain Python lists or dicts and calling vectorized-sounding functions on it. The tell is no speedup, because the efficiency comes from ndarray storage.
• Rebuilding sorting, unique, set logic or group-wise aggregation by hand. The tell is dict-and-counter code where NumPy or pandas already provides the algorithm.

• ▸Loop location: file and line of the explicit loop or if-elif-else block.
• ▸Per-element action: scalar transform, pairwise compare, conditional pick, or group summary.
• ▸Input container: list, ndarray, Series or DataFrame; note whether conversion is needed.
• ▸Vectorized tool chosen: ufunc, boolean index, array expression, sort/unique/set op, or pandas group-wise operation.
• ▸Boolean mask expression(s): write each condition and any ~ negation.
• ▸Axis and length check: confirm each mask length matches the axis it indexes.
• ▸Combined indexing: slice or integer selections to add alongside the mask.
• ▸Verification: compare against the loop result on a small slice and record the outcome.
• ▸Loop removed: yes or no, with the remaining reason if no.

1. 1Name the target shape before touching code. The source defines structured data as tabular data with one type per column, multiple tables linked by key columns, multidimensional arrays, or time series. Write down which one you are building and what a single row represents.
2. 2Decide whether the data is already structured. A large share of datasets can be transformed into a structured form. If it is not (the source's example is a collection of news articles), extract features into a structured form, such as a word frequency table.
3. 3Put the result in a DataFrame, the tabular, column-oriented object with both row and column labels. Use a Series for a single labeled one-dimensional column. Label both axes deliberately rather than accepting default labels.
4. 4Set the index to the identifier that the data from your different sources shares. pandas aligns data on labels, automatically or explicitly. This prevents errors from misaligned data and from differently indexed data coming from different sources.
5. 5Check each column's dtype. Each column may hold a different type (string, numeric, date). Care about the general kind: float, int, boolean, string, or Python object. For numeric columns, the NumPy array underneath is the efficient container.
6. 6Resolve encodings at the file boundary. Bytes objects mostly appear when reading files. Decode them to Unicode strings with the correct encoding (UTF-8 is preferred, but latin1 and utf-16 variants occur) before putting text into columns.
7. 7For several related tables, keep each as its own DataFrame with a key column or index. This is the primary/foreign-key pattern a SQL user knows. Combine them afterward instead of cramming everything into one wide table.
8. 8Confirm the final frame supports the downstream tasks the source lists: reshaping, slicing and dicing, aggregation, and subset selection. Then hand it to modeling or statistical tools.

The source's own example of structuring unstructured data is a collection of news articles. As raw text it cannot be analyzed directly. You extract features into a word frequency table: one row per article, one column per word, with counts as values. This is tabular, labeled, and numeric, and it can then feed sentiment analysis. The same logic applies to the source's smaller illustration. A dictionary of seven random numbers keyed 0 to 6 is a loose container. As a Series, the keys become the index labels and the values become a one-dimensional labeled array. The pattern is the same in both cases: pick the row unit, label the axes, and make each column a single type. After that, every downstream tool can operate on the data without guessing what it means.

• If the data is already tabular, comma-delimited, or from a relational database, load it into a DataFrame directly and spend your effort on indexes and dtypes.
• If the data is free-form (text, documents), extract features into a table, such as word counts per document, before attempting any analysis.
• If two sources describe the same entities, give both the same index and let pandas align on labels instead of aligning by position.
• If the data is purely numeric and homogeneous, a NumPy array is the efficient container. Use a DataFrame when you need labels or mixed column types.
• If you read bytes from files and see garbled text, identify the encoding and decode explicitly instead of guessing.

• Combining sources by position instead of by label. The tell is values that look plausible but belong to the wrong entity, which is the misalignment error that labeled axes are meant to prevent.
• Leaving columns as generic Python objects. The tell is a numeric or date column that cannot be aggregated or that behaves like text.
• Mixing text encodings. The tell is characters like 'ñ' appearing as escape sequences or garbage, because the bytes were not decoded with the right encoding.
• Forcing everything into one table when it is really several related tables. The tell is repeated values and unclear row meaning, where a key-linked set of tables would have been cleaner.
• Skipping the row-unit decision. The tell is that groupings and counts come out wrong, because you cannot say what one row represents.

• ▸Source and raw form: where the data comes from and what shape it arrives in (file, database, text, array).
• ▸Structure type: tabular, multi-table with keys, multidimensional array, or time series.
• ▸Row unit: what exactly one row represents.
• ▸Feature extraction needed: if the data is unstructured, which features become columns (for example, word counts).
• ▸Index choice: which label identifies rows and is shared across sources.
• ▸Column dtypes: the kind of each column (float, int, boolean, string, date, object).
• ▸Encoding check: the encoding of any text or bytes input and how it is decoded.
• ▸Key columns: how related tables join, equivalent to primary and foreign keys.
• ▸Downstream use: the analysis, aggregation, or model the table must support.

1. 1Work in an execute-explore loop. Run your analysis in IPython or a Jupyter notebook instead of the edit-compile-run cycle. Most data analysis is exploration, trial and error, and iteration, and this loop shortens each pass.
2. 2Replace Python for loops with whole-array operations. Write my_arr * 2 instead of [x * 2 for x in my_list]. NumPy runs C algorithms on a contiguous memory block with no per-element type checking, and the source reports 10 to 100 times (or more) speedups.
3. 3Measure before you optimize. Use %time on both versions, as the source does: 10 repetitions took 26.5 ms of wall time for the NumPy array and 473 ms for the list comprehension. Rewrite only where the gap is real.
4. 4Use the built-in reductions with the axis argument. arr.mean(axis=1), arr.sum(axis=0), and df.sum(axis='columns') replace hand-written loops. Remember that axis=0 collapses down the rows and axis=1 collapses across the columns.
5. 5Let pandas handle missing data. DataFrame reductions skip NA values by default and return NA only if the entire slice is NA. Set skipna=False only when you want NA to propagate. Do not write NaN-filtering code first.
6. 6Express group logic with groupby. Split by one or more keys, then apply an aggregation or transformation (normalization, rank, regression, subset selection) using any function that accepts a pandas object or NumPy array. This replaces SQL workarounds and manual splitting.
7. 7Use the Categorical type for columns that repeat a small set of distinct values. The source says it gives better performance and memory use in some operations. Check unique and value_counts first to confirm the repetition.
8. 8Hand off to modeling libraries through NumPy arrays. Use DataFrame.values (the point of contact between pandas and other libraries) so cleaning in pandas and fitting in statsmodels or scikit-learn needs no custom conversion code.

The source's own benchmark shows the effect. You create my_arr = np.arange(1000000) and my_list = list(range(1000000)), then multiply each by 2, ten times over. The array version, `my_arr * 2`, reports CPU times of 28 ms total and 26.5 ms wall time. The list version, `[x * 2 for x in my_list]`, reports 472 ms total and 473 ms wall time. That is roughly an 18x gap on one trivial operation. Over a session of repeated runs, the difference sets how many iterations you can try before losing focus. The same logic applies downstream. `arr.mean(axis=1)` returns the five row means of a 5x4 array in one call, with no loop to write or debug. Your time goes to reading results, not to building the machinery that produces them.

• If you are exploring or iterating, work in IPython or Jupyter. If the code is stable and reused, move it into modules, which Python caches after the first import.
• If a task loops over elements of a numeric sequence, convert it to a NumPy array operation. Check with %time that the speedup justifies the change.
• If a summary statistic must tolerate gaps, use the pandas method with its default NA skipping. If a gap should invalidate the result, set skipna=False.
• If a column holds repeated values from a small set, check unique and value_counts, then consider Categorical for speed and memory.
• If you need a custom per-group computation beyond what SQL expresses comfortably, use groupby with any function that accepts a pandas object or NumPy array. If you need to cross into modeling, convert with .values.

• Writing Python for loops over large numeric data when a vectorized operation exists. The tell is wall time in hundreds of milliseconds where an array version takes tens.
• Confusing axis meaning. arr.mean(1) averages across columns, while arr.sum(0) sums down the rows. The tell is a result array of the wrong length, or one whose dimension is not reduced as expected.
• Using NumPy reductions on data with missing values and getting NaN back. pandas reductions already skip NA, so the tell is hand-written NaN-filtering code duplicating built-in behavior.
• Optimizing without measuring. You rewrite code that was never the bottleneck. The tell is the absence of any %time comparison before and after.
• Treating exploration as a build-and-compile cycle, with every tweak a full rerun. The tell is that testing one idea takes minutes of setup instead of one executed cell.

• ▸Task: one line describing the manipulation step (e.g., double every value, summarize per group).
• ▸Current approach: loop, list comprehension, manual filtering, or built-in method.
• ▸Baseline timing: %time wall time of the current approach on realistic data size.
• ▸Vectorized or built-in alternative: the NumPy or pandas operation that replaces it.
• ▸Alternative timing: %time wall time of the replacement on the same data.
• ▸Speedup ratio: baseline divided by alternative; decide whether the change is worth making.
• ▸Missing-data handling: confirm whether NA should be skipped (default) or propagated (skipna=False).
• ▸Axis check: state whether the operation runs over axis 0 or 1 and confirm the output shape.
• ▸Environment: IPython/Jupyter for exploration, or module for stable reusable code.

1. 1Scope the job before touching code. Name the data form you hold: tabular with mixed column types, multidimensional arrays, multiple tables linked by key columns, or time series. The source treats these as the structured data the toolchain is built for, so naming the form tells you which tool family applies and shrinks the ecosystem to a manageable part.
2. 2Budget for preparation. The passages say loading, cleaning, transforming, and rearranging often take 80% or more of an analyst's time. Plan the session around wrangling and treat that share as normal, not as a sign you are slow.
3. 3Load with the matching read_ function and start with a few parameters. Pick from read_csv, read_sql, read_pickle, read_sas, read_stata, read_feather and the rest by file format. Then add options by category: indexing, type inference and conversion, datetime parsing, iterating over chunks, and unclean-data handling (skipped rows, footers, comments, thousands separators).
4. 4When a loader has too many options, don't memorize them. The source calls feeling overwhelmed by read_csv's 50+ parameters normal. Find a similar example in the online pandas documentation and adapt its parameters to your file.
5. 5Inspect immediately after loading. Run .head() and check the dtypes. read_csv infers types, while HDF5, Feather, and msgpack store them. Dates and custom types often need extra effort, so verify them first.
6. 6Prefer built-in pandas operations over hand-written loops. Missing data is excluded from descriptive statistics by default. A Series passed to rolling(...).corr() is computed against every column of a DataFrame. Check for an existing method before writing a loop.
7. 7Wrap repeated steps in a named function once you expect to run them more than once (def with positional and keyword arguments), and chain them with .pipe. Pass `result = df[df.col1 < 0].pipe(group_demean, ['key1','key2'], ['col1'])`. Named steps are easier to read, test, and debug.
8. 8When your need falls outside what the book or pandas covers, go to the documentation and release notes, then to the mailing lists or the pandas GitHub site. Treat these as normal channels for deepening expertise.

You are handed a stock-price table and asked how Apple's returns track the S&P 500. You name the form first: a time series. You compute percent changes with `spx_rets = close_px_all['SPX'].pct_change()` and `returns = close_px.pct_change()`. Then you take the rolling correlation with `returns.AAPL.rolling(125, min_periods=100).corr(spx_rets)` and plot it, which gives the six-month AAPL correlation to the S&P 500 (Figure 11-8). Asked to extend this to many stocks, you skip the loop. You call `returns.rolling(125, min_periods=100).corr(spx_rets)`, which correlates the Series with each DataFrame column and plots in one line (Figure 11-9). Asked for something custom, such as the percentile rank of a 2% return over a one-year window, you pass a function to `rolling(250).apply(score_at_2percent)`. Each request maps to a known pattern, so none of them requires learning something new from scratch.

• If a file will not parse, then look for a close example in the pandas documentation and change the relevant parameter category (indexing, types, dates, unclean data) instead of rewriting the loader.
• If the file is very large, then use the chunked iteration options rather than forcing a single load.
• If you would write the same or very similar code twice, then make it a named function; if a loop only repeats a per-column computation, then check for a built-in that handles all columns.
• If you combine boolean conditions on NumPy arrays, then use & and |, not and/or, and wrap each comparison in parentheses.
• If nested comprehensions go past two or three levels, then rewrite for readability; if a module split creates many small interconnected files, then keep fewer, cohesive, longer files.

• Treating cleaning time as personal inefficiency. The tell: you rush or skip validation because 80% of the time on prep feels like failure, though the source reports it as typical.
• Memorizing option lists instead of learning to look them up. The tell: you stall on read_csv's 50+ parameters and don't search the documentation for a similar example.
• Trusting type inference blindly. The tell: dates stay as strings or columns load with the wrong dtype, and you find out only when a later step breaks.
• Using Python's `and`/`or` on boolean arrays. The tell: an error or wrong mask where `(names == 'Bob') | (names == 'Will')` was needed, which erodes confidence in NumPy.
• Reaching for a modeling library before the data is ready, or expecting one chapter to cover a whole library. The tell: you start on statsmodels or scikit-learn with unresolved missing data, though the source says each library warrants its own book and wrangling comes first.

• ▸Task statement: one sentence describing the question to answer.
• ▸Data form: tabular, array, multi-table with keys, or time series.
• ▸Source format and loader: file type and the read_ function chosen.
• ▸Option categories needed: indexing, types, dates, chunking, unclean data (tick those that apply).
• ▸Documentation example found: link or title of the closest matching example.
• ▸Post-load checks: head(), dtypes, missing-value count, date columns parsed.
• ▸Built-in alternative to a loop: the method you checked before writing one.
• ▸Reusable function candidate: any step you will repeat, with a proposed name.
• ▸Unknowns and escalation: what remains unclear and which resource you will consult (docs, release notes, mailing list).