Methods of Operations Research (Dover Books on Computer Science)

1. 1Name the decision and its deadline before any analysis. Write down who decides (the executive department), what they decide, and the date after which the answer no longer changes the outcome. Morse and Kimball's test: 'crucial decisions cannot wait on the slower study.'
2. 2Run the first quick survey of the field. The passages hold that 'the big improvements often come from the first quick survey of a new field; later detailed study may only gain small additional factors.' Spend your first effort here, not on data collection.
3. 3Rank the operational parameters by sensitivity and drop the weak ones from the first analysis. The passages say 'the less sensitive ones must be eliminated from the first analysis in order that any progress can be made at all.' Keep the few that move the outcome.
4. 4Work from the data you have, even if it is incomplete, and state what it can and cannot support. A preliminary analysis on incomplete data can be worth more than a thorough study on adequate data, because the decision gets made on whichever arrives first.
5. 5Check that the data represent operational conditions. Before you rely on a number, ask whether it would hold in battle or in the field. The rocket-sight example shows one pilot on a land range failing this test, while several squadrons firing at a towed sea target passes it.
6. 6Plan interim output into the study. Build in 'interim instructions or a report of first findings' so the decision-maker can act before the full study finishes. Record and tabulate data during testing so you can see whether it is significant and repeat any unexpected condition at once.
7. 7Delegate extraneous duties so your time goes to the analysis. The passages say to divest the analyst of 'gadgeteering' and similar side duties so he has leisure to make the study 'as thorough ... as possible before the crucial decision must be made.'
8. 8Run the exhaustive study in parallel and use it to back up the preliminary finding. The passages say 'the exhaustive study should also be made, to back up the preliminary work.' Revise the recommendation only if the full study contradicts it.

Early in World War II the British Coastal Command dropped depth charges from aircraft, and squadrons argued over the depth-setting. The setting could not be changed in the plane before the attack run, so one average value had to serve every attack. Some squadrons chose 150 feet, which was absurd: submarines at that depth could not be seen, and visible ones near the surface were only shaken. The setting was cut to 50 feet as a compromise, and the argument ran for a year before a numerical analysis settled it. That analysis asked one question: what was the submarine's state of submergence at the moment of the drop? Operational results showed 40 per cent of attacks hit a surfaced submarine and another 10 per cent a partly visible one, so in half the cases, and those with the best accuracy, 50 feet was too deep. The lesson for urgency is that a year of debate cost operations more than a quick tally of attack conditions would have. The tally needed only a rough count and a sensible rule: set the depth for the attack type with the best chance of success.

• If the decision falls due before the thorough study can finish, then act on the preliminary analysis and say plainly what data it lacks.
• If a parameter is not sensitive in the first pass, then drop it from the first analysis rather than modeling everything.
• If early data come from unrepresentative conditions, such as one pilot on a land range, then widen the sample to varied units, directions and conditions before you trust it, even under time pressure.
• If an unexpected result appears during testing, then repeat that condition at once while the facilities are still available.
• If the exhaustive study is still feasible, then run it after the quick survey as backup, and do not substitute it for the quick survey.

• Waiting for adequate data: the analysis is accurate but arrives after the decision, so the tell is a finished report nobody can use. The depth-setting argument ran a year before the numerical analysis.
• Scorning superficial work: the analyst treats a rough first survey as beneath him. The tell is a long list of study phases with no interim findings.
• Treating a rough pass as final: the preliminary view is never checked against the full study. The tell is that no one ever opens the backup analysis.
• Letting side duties eat the analysis time: the analyst drifts into equipment tinkering or other 'gadgeteering'. The tell is that his calendar is full while the operational question stays unanswered.
• Hasty data that misrepresent the operation: a quick test is run in convenient conditions, like several weeks on a land range. The tell is a result that cannot be tied to how the equipment would be used in the field.

• ▸Decision and decision-maker: state the choice and the executive who must make it.
• ▸Decision deadline: the date after which the analysis can no longer change the outcome.
• ▸Quick-survey question: the one question whose answer would move the decision most.
• ▸Key parameters kept: the few most sensitive parameters used in the first pass.
• ▸Parameters dropped: the less sensitive ones set aside, with the reason.
• ▸Data available now and gaps: what you have, and what is missing or unrepresentative of operational conditions.
• ▸Preliminary finding and recommendation: the answer you can give by the deadline, stated with its limits.
• ▸Interim report date: when first findings go to the decision-maker.
• ▸Backup study plan: the exhaustive study to be run afterward, and what result would reverse the recommendation.

1. 1Plot the measure of effectiveness month by month, as the Bay of Biscay analysts did with the operational sweep rate. The chart is your instrument for seeing the see-saw. A flat or falling line after a period of gains signals an enemy response before anyone confirms it.
2. 2Pick the measure before the data. The passages warn that the choice of measure is often the real problem. Antiaircraft guns on merchant vessels looked useless by planes shot down, but they cut enemy accuracy enough to lessen the chance the ship was sunk. Choose a measure that captures what your measure actually changes in the enemy's behavior or results.
3. 3Log every own-side change (new equipment, new tactic) with its date next to the effectiveness line. Without dated introductions you cannot tell a benefit from a countermeasure effect.
4. 4When the line moves, ask what the enemy changed and how it connects to your last move. In the Bay, night attacks pushed submarines to surface by day, which raised day-plane sightings, and then the Germans fitted radar receivers. Trace the chain from your measure to his reaction.
5. 5Run small, deliberate variations when chance fluctuations do not reveal dependence. Radar countermeasure studies shut sets off for periods and tried intermittent operation. Keep changes as small as will give a positive result until you know the direction of the effect. Senior executives usually must authorize such a change.
6. 6Prepare the next counter before the current one fails. The side that counters quickly, before the opponent perfects new tactics and weapons, has the advantage. Have the next measure designed while the enemy's response is still being fielded, as the S-band fit was introduced once the low sweep rate persisted.
7. 7Estimate the enemy's lag. Count design and manufacturing difficulties on his side (the S-band receivers were not available until fall 1943) and watch his interim moves, such as fewer submarines at sea and orders to stay submerged. These also depress your measure.
8. 8Do not rely on small-variation studies alone. Enemy reaction to a quantitative change is often a qualitative change, and the useful move may be a different combination of actions. Use whole-operation analysis or discussions with operating personnel for that, and treat expert recollection as unreliable evidence.

The passages' own case is the Bay of Biscay, 1942–43. In early 1942 air cover over the Bay increased, and in the spring a few radar squadrons were equipped for night flying with searchlights. The effective search rate rose for all plane types: night attacks caught submarines on the surface, which pushed them to submerge at night and surface by day, so day planes found more of them too. The Germans countered by fitting submarines with receivers that could hear the British L-band radar. The operational sweep rate fell abruptly in late summer 1942, to about one-fifth of its earlier value. When the low value persisted for several months, the British fitted S-band radar that the L-band receivers could not detect. The sweep rate rose from the start of 1943 and by midsummer was back to its level of a year earlier. The Germans' S-band receivers were not ready until fall 1943. In the interim they sent fewer submarines out and had them stay submerged in the Bay, and by the time the receivers arrived the sweep rate had dropped to the previous fall's low. The month-to-month chart let analysts time new measures and assess the danger of each enemy measure.

• If your effectiveness measure drops sharply and stays low for months, then assume an enemy countermeasure and introduce a new measure rather than wait for the line to recover.
• If your measure rises after a new tactic, then expect an enemy response and begin work on the next counter immediately; the gain is temporary.
• If the enemy's counter needs design or manufacturing effort, then expect interim behavioral changes from him (fewer sorties, evasive posture) and do not read the resulting dip as the final countermeasure.
• If a tactic looks ineffective on its obvious metric, then check whether it changes the enemy's accuracy or behavior before cutting it (antiaircraft guns on merchant vessels).
• If small-variation tests show only marginal gains, then consider that the enemy's reaction may be qualitative and search for a different combination of actions, using whole-operation analysis or operator discussion, rather than extrapolating.

• Treating a measure's gain as permanent. Tell: no next-generation counter is in development while the sweep-rate-style metric is at its peak.
• Using a measure that misses the real effect. Tell: a useful tactic is about to be dropped because it fails the headline metric, like the antiaircraft guns that did not shoot planes down.
• Reacting slowly to a persistent decline. Tell: the metric sits at a low value for months with no new measure introduced, and the enemy gains time to perfect his tactics.
• Relying on operators' memory of the exciting events instead of impersonal data. Tell: conclusions rest on a few dozen people's recollections, which are biased toward memorable events and underweight the rest.
• Extrapolating from small variations. Tell: predictions assume linear response, but the enemy responds with a qualitative change in counteraction, so results diverge sharply from forecast.

• ▸Measure of effectiveness: define the quantity tracked and why it reflects the real effect (e.g., sweep rate, exchange rate).
• ▸Period and value: record the value for each month in a running chart.
• ▸Own measure introduced: name the new tactic or equipment and its date.
• ▸Observed effect on the metric: note the direction and size of the change after introduction.
• ▸Suspected enemy response: state the hypothesized countermeasure and the chain linking it to your last move.
• ▸Enemy lag estimate: note what design, production, or fielding difficulties delay his counter, and his interim behavioral changes.
• ▸Evidence quality: mark each input as impersonal data or operator opinion.
• ▸Small-variation test: describe any temporary parameter change, its size, and who authorized it.
• ▸Next counter and trigger: name the planned counter-countermeasure and the metric level or duration that triggers it.

1. 1State the operation and the decision-maker in one line before touching data. The passages stress that the study is made at the request of the authority controlling the operation, because a report handed to any other authority 'could not have been acted upon.' If you cannot name who can try out your prediction, stop.
2. 2Acquire enough technical background to understand the fundamentals, but not so much that familiarity with the operation's details handicaps you. Learn what the operation does and how it behaves (for convoys: ships in port and in convoy). Do not adopt the operators' habitual explanations.
3. 3Scrutinize every assumption, including the measures of value and the estimates of results, as a matter of routine procedure. Treat the executive's stated criteria as hypotheses to test, not as givens. Keep this questioning inside the analysis team so initial doubts do not spread to other parts of the organization.
4. 4Before computing anything, choose the measure of effectiveness. Write down at least two candidate criteria and compute each from the same raw data. Where the choice is genuinely ambiguous (accident reduction, traffic flow, or their ratio), put it to the executive in charge for a decision.
5. 5Test the raw count against a normalized criterion. Ask what the total exposed population is and whether you are looking at an absolute number or a percentage of the whole. A flat raw count can hide a real effect, or fake one.
6. 6Move from correlation to mechanism. When two variables correlate, such as COD refusals and delivery delay, do not stop at recording it. Identify the economic, psychological, physical or other factors that produce it, because only an understood mechanism lets you change the outcome.
7. 7Frame the output as a prediction of future operations under a modified procedure, not a description of the past. Check whether the recommended change needs no new equipment, only a change in how existing equipment is used.
8. 8Hand the result back with the decision explicitly left to the executive. State what is skeptical analysis and what is the arbitrary but necessary judgment that remains the executive's to make.

The passages describe the convoy-size problem. Raw sinking data showed that about the same number of ships were lost per attack from large convoys as from small ones. An analyst who accepted that count 'thoughtlessly' would conclude that larger convoys would not help. The impartial analyst first questions the criterion. The correct measure is the percentage lost of the total number of ships crossing the ocean, because a large convoy carries more ships through the same number of attacks. Under that measure the answer becomes apparent: larger convoys reduce average ship loss. The analyst also needs knowledge of how ships behave in port and in convoy, to confirm that the larger size would not bring concomitant deleterious effects. The study was made for the authority controlling convoys, who could try the prediction. The result was a large effect obtained with no change in equipment, only a change in how it was used. The analyst needed no personal stake and no command responsibility to get there, only impersonal curiosity and a refusal to accept the first criterion.

• If the raw data show no difference between options, then re-express the outcome as a rate or percentage of the total exposed population before concluding that there is no effect.
• If the choice of criterion is ambiguous (e.g., accidents vs. flow vs. their ratio), then do not settle it alone. Lay out the candidates and have the executive decide.
• If you find a correlation, then do not report it as the finding. Identify the mechanism behind it before recommending action.
• If your familiarity with the operation's details starts to make its habits feel natural, then step back to the fundamentals. Too great a familiarity may handicap you.
• If the executive reacts with suspicion to your questions, then explain that questioning measures of value is procedure, not criticism. Keep the doubts inside the analysis and let results demonstrate their worth.

• Accepting the raw count as the criterion. The tell: a conclusion like 'large convoys won't help' drawn from equal losses per attack, with no denominator of total ships crossing.
• Describing the past instead of predicting the future. The tell: the report records or explains what happened but contains no forecast of what a changed procedure would produce, as in the COD correlation that was noted but not understood.
• Skepticism that reads as hostility. The tell: the executive concludes you are criticizing him rather than the assumptions, and opposition builds. The cause is questioning delivered without tact, or doubts allowed to spread beyond the analysis team.
• Taking over the decision. The tell: you push your preferred measure of value on an ambiguous problem instead of letting the executive in charge decide, forgetting that the executive remains the one to make the basic decisions.
• Analyzing for someone who cannot act. The tell: a study addressed to an authority with no control over the operation, so the prediction cannot be tried and proved.

• ▸Operation and decision-maker: name the operation and the authority who controls it and can act on the result.
• ▸Candidate measures of effectiveness: list at least two criteria, including one normalized by the total exposed population.
• ▸Raw data versus normalized result: record what each criterion shows from the same data.
• ▸Assumptions and value estimates examined: list each assumption or estimate in the executive's framing and your test of it.
• ▸Mechanism behind any correlation: state the factors (physical, economic, psychological) that explain it, or mark it 'not yet understood'.
• ▸Prediction under the changed procedure: state the forecast outcome, and whether it needs new equipment or only a change in use.
• ▸Technical background gaps: note any fundamentals of the operation you do not yet understand, and any detail-level familiarity that may bias you.
• ▸Decision left to the executive: record the judgments, such as the choice of criterion, that you hand back.
• ▸Containment note: record who has been told about your questions and how doubts are kept within the analysis team.

1. 1Agree terms of reference first. Before any analysis, settle in conversation with the administrative head what questions the group will study and who decides what. The source calls this one of the most important organizational problems in entering a new field.
2. 2Secure direct, personal contact. Have the group report to the officer or executive who makes the decisions, not to a research department. The executive tells you which questions require decision, and you tell him the results directly. Keep the group as small as possible and contact as personal as possible.
3. 3State the division of labor out loud. Tell the executive you provide a quantitative basis for decision, and tell the scientist the executive still makes the basic decisions. Each party names the other's prerogative and its own.
4. 4Expect the executive's reactions in sequence and prepare for each. First: 'scientists are welcome but there is no important problem for them.' Next: suspicion and impatience when you begin scrutinizing assumptions. Do not read either as hostility.
5. 5Explain why you question assumptions. Use tact to show that challenges to measures of value and estimates of results are part of the method and not a wish to criticize. Say this before you start probing, not after the executive objects.
6. 6Contain the doubt. Keep initial questioning of assumptions inside the working pair and do not let it spread to other parts of the organization.
7. 7Deliver an early, successful solution on a real problem. Trust consolidates once the executive sees that the critical questioning produces results. Choose a first problem where a clear answer is likely.
8. 8Treat presentation as part of the work. A project is not complete until the findings are imparted to the executive in a manner that aids his decision. Present in clear, plain language, and respect secrecy needs without drifting into 'secrecy for secrecy's sake.'

The suicide-plane study shows the pattern. Planes diving on ships could be met by violent ship maneuvers or by a steady course relying on antiaircraft fire. The analysts collected 477 cases of Kamikaze attacks: 172 planes (36%) hit their target, and 27 ships were sunk. Only 365 cases had enough detail to classify, so the analysis used those, comparing hit rates by ship size and by maneuvering or not. The finding was stated as guidance for the decider: battleships, cruisers, and carriers should employ radical maneuvers, because the hit percentage was considerably lower when they maneuvered. The scientist also stated the limit of the evidence: the table says nothing about what sort of maneuvers to use. The scientist offers a quantitative basis and its boundary, and the executive keeps the decision on tactics. Trust grows when the analyst reports what the data do not show.

• If the executive says there is no important problem for the scientists, then do not argue the point abstractly. Ask what decisions he faces and pick one a quantitative answer could help.
• If the executive turns suspicious or impatient when you question assumptions, then explain that the scrutiny is the method and not criticism, and confine the questioning to the working pair.
• If an analysis would require changing live operations (a test change in tactics or parameters), then bring it to the senior executive as his decision, and propose the smallest change that can give a positive result.
• If a scientist begins pushing the decision itself or resents an 'arbitrary' call, then restate that the executive must make the basic decisions, which are in part arbitrary, and that your role is the quantitative basis.
• If a trained analyst moves into an executive seat, then expect him to lose usefulness as an analyst, since deciding conflicts with looking at the operation impersonally. Do not use him as both.

• Reporting through a research department. The tell: the analysts do not know which questions the decider actually faces, and he hears results secondhand.
• Treating skepticism as an attack, or letting it spread. The tell: the executive's early welcome turns into suspicion, and doubts about assumptions circulate across the organization.
• Analyst overreach. The tell: the scientist is impatient at arbitrary decisions and presents conclusions as orders, so the executive stops listening.
• Presentation neglected. The tell: the analysis is correct but the executive cannot use it, and the project is treated as finished although the findings were never imparted in a decision-aiding form.
• Mixing analyst and executive roles in one person, or letting security habits become 'secrecy for secrecy's sake.' The tell: impersonal analysis erodes once decisions are required, or information is withheld from the pair that needs it.

• ▸Executive decision-maker: name the one officer or executive who makes the decisions this group serves.
• ▸Decisions he faces: list the operational decisions the analysis is meant to support.
• ▸Reporting line: confirm direct, personal contact with the executive and not via a research department.
• ▸Executive's prerogative: write what remains his alone to decide.
• ▸Scientist's remit: write the questions the group may analyze and the assumptions it may challenge.
• ▸Stage of executive reaction: no problem for you / suspicion and impatience / results seen. Note the current stage.
• ▸Assumption challenges planned: list each assumption to be questioned and the explanation you will give for why.
• ▸Containment: state who inside the organization hears the questioning and who does not.
• ▸First test problem and presentation: name one early solvable problem and how you will present the result plainly, with its limits.

1. 1Write down the measure of effectiveness before touching the data. State it as a ratio or percentage if the raw count could mislead. In the convoy study the criterion was the percentage of ships lost out of all ships crossing the ocean, not ships lost per attack.
2. 2Test the obvious criterion against the raw data and ask what it would conclude. Raw sinking data showed about the same number of ships lost per attack from large convoys as from small, so a careless reading says large convoys do not help. If a criterion yields a conclusion that feels thoughtless, replace it.
3. 3List two or three competing criteria and say what each would recommend. The passages' traffic-system example offers reduction of accidents, increase of traffic flow, and the ratio of the two. Where the choice is not easy, take it to the executive in charge of the operation and have them help decide.
4. 4Ask what the thing under study actually does, not only what it was meant to do. Antiaircraft guns on merchant vessels were judged by planes shot down. The real effect was lower enemy accuracy, which lessened the chance the vessel was sunk.
5. 5Examine who or what the current effort is aimed at. The merchandising concern sent advertising mainly to satisfied customers and judged it by gratifying 'results'. Ask who is not being reached: customers drifting from satisfied to dissatisfied, and potential customers who never became regular ones.
6. 6Confirm that the person who must act is the person asking, or is in the loop. The convoy study was requested by the authority controlling convoys, who could try out and prove its predictions. The same report handed to any other authority could not have been acted upon.
7. 7Check the reformulated problem with a small trial before standardizing. Predict from past data (large convoys, smaller percentage losses), run trials, and reduce the findings to standard practice only when the results justify the prediction. If the data cannot show the dependence, change one parameter temporarily and as little as possible. The senior executive usually has to approve that change.
8. 8Re-state the problem in one sentence, naming the criterion, and get the executive's agreement before analysis proceeds.

The convoy study is the clean case. The raw sinking data showed that about the same number of ships were lost per attack from large convoys as from small ones. Judged by that count, one might thoughtlessly conclude that large convoys would not help. The analyst changed the criterion to the percentage lost of the total number of ships crossing the ocean. On that measure the solution was apparent: large convoys give smaller percentage losses. Much of the problem was solved the moment the criterion was fixed. The study was made at the request of the authority controlling convoys, who could act on it. Enough data on convoys of different sizes existed to make a preliminary prediction. Trials then checked it, and when the results justified it the finding was reduced to standard practice.

• If the raw counts say an intervention makes no difference, then re-express the outcome as a rate against the total exposed (a percentage of everything at risk) before accepting that conclusion.
• If an asset is being judged by its nominal purpose and failing (guns that do not shoot planes down), then ask what other outcome it changes, such as reduced enemy accuracy and fewer vessels sunk, and re-score it on that.
• If several plausible measures exist and they conflict (accidents, flow, or their ratio), then do not choose silently. Take the choice to the executive in charge of the operation.
• If reported results look gratifying but the effort is aimed at people already satisfied, then shift the question to where customers are lost or never won, and look for different criteria.
• If the requester cannot act on the findings, then redirect the problem to the authority who controls the operation before investing in analysis.

• Adopting the criterion that comes with the data: counting losses per attack because that is how sinkings are recorded. The tell is a conclusion that large convoys do not help, which fits the numbers but ignores total ships crossing.
• Judging a measure by its nominal function. The tell is a proposal to remove antiaircraft guns because they 'did not shoot down enemy planes', while the effect on enemy accuracy goes unmeasured.
• Accepting favorable results from the wrong population. The tell is advertising 'results' that look quite gratifying, yet the mailings go mainly to satisfied customers, and nobody asks why customers turn dissatisfied or why prospects never become regular.
• Choosing the measure of effectiveness without the executive. The tell is a technically correct analysis of a metric the person in charge never agreed to, such as accident reduction when they cared about flow.
• Analyzing for someone who cannot act. The tell is a report delivered to an authority outside the operation, which, as the convoy case notes, would have had no effect because it could not be acted upon.

• ▸Problem as first stated: Write the request or assumption in the requester's own words.
• ▸Who controls the operation: Name the executive or authority who can act on and test the findings.
• ▸Obvious criterion: State the measure the raw data invites and what conclusion it gives.
• ▸Alternative criteria: List at least two other measures, such as a rate or percentage of the total at risk, or a ratio of two outcomes.
• ▸What each criterion recommends: Write the decision each measure would lead to, and note where they conflict.
• ▸Hidden effect or hidden population: Note any outcome the thing changes beyond its nominal purpose, and any group the current effort misses.
• ▸Executive's chosen criterion: Record the measure the executive in charge agreed to, and the date.
• ▸Restated problem: Write one sentence naming the real problem and the agreed criterion.
• ▸Check plan: Describe the prediction, the smallest trial change that would test it, and who approves it.

1. 1State the decision the executive must make and the operation under their control. The passage defines OR as 'a quantitative basis for decisions regarding operations under their control', so a finding that does not attach to a decision is not yet a basis.
2. 2Identify the repeated element in the operation (sorties, firing runs, duels, engagements). Repetition is what makes the operation open to scientific attack, even when the similarity between parts is not obvious. Look for it before choosing a measure.
3. 3Pick a measure of effectiveness and decompose it into its factors. In the VLR bomber case the factors are bomb load carried, accuracy, and operational hours spent on targets versus training. Measure each factor separately so the executive can see which one moved the result.
4. 4Standardize the conditions before comparing runs. In AA gunnery, choose a standard firing rate and ammunition quality, then weight each run by jq (firing-rate ratio j times ammunition-quality ratio q). Record firing rate and quality on every run. If the target size differs from the real enemy craft, record the ratio of effective target areas.
5. 5Analyze each type of run separately (altitude, speed, line of approach) rather than pooling. Express results as a curve, such as a hit expectancy curve, not a single average. Then state which parameters the curve is proportional to. Hit expectancy is proportional to firing rate and to ammunition quality, so the effect of changing guns or rate is easy to read off.
6. 6Name the sensitive parameters explicitly. Where a quantity such as an exchange rate E (Blue losses over Red losses) drives the outcome, show the result as a function of it. Where a parameter is directly proportional to the outcome, say so.
7. 7Compare options on the same scale before recommending. For example, ask whether adding new equipment or further training in the old equipment yields more effectiveness in this case. Do not assume numbers, sorties or gadgets are the answer.
8. 8Present the findings in a form the non-scientist can use: plain graphs, a short statement of conclusions, and an oral briefing of the staff on the major points. Flag the non-quantitative factors (politics, morale, tradition) the executive must add. Do not make the decision yourself.

The source's VLR bomber analysis breaks effectiveness into plotted components: bomb load carried, accuracy, and the share of operational hours spent bombing rather than training. The curve for lost effectiveness due to the extra training time is 'a very small amount' next to the rise in bomb load and accuracy that the training in part produced. Multiplied together, the fourth graph shows a VLR plane was ten times as effective in destroying enemy installations in November as in August. A squadron in November outperformed three groups in August, with essentially the same planes. The executive's basis is therefore not 'send more aircraft' but 'the sensitive parameters are loading and accuracy, and training moves them'. The source concludes that these gains far exceeded what more aircraft, more missions or new gadgets could have delivered. It also recommends analyzing other cases so one can estimate whether new equipment or further training would pay more. The decomposition is what makes the ten-fold figure intelligible and actionable, not just large.

• If a result is a single aggregate, then break it into factors (load, accuracy, hours) and plot each, so the sensitive one is visible.
• If runs differ in firing rate or ammunition quality, then weight each run by jq against a standard before pooling.
• If a parameter is proportional to the outcome (e.g., hit expectancy to firing rate), then state it, so the executive can read off the effect of changing guns or rate directly.
• If the proposal is to add numbers (aircraft, sorties, gadgets), then compare against a quality improvement such as training on the same measure before endorsing it.
• If the remaining factors are politics, morale or tradition, then name them for the executive and stop; the decision belongs to the executive.

• Reporting a total without its components. The tell: the executive cannot say which parameter moved the result or what to change.
• Pooling unlike runs (different altitude, speed, firing rate, ammunition quality) into one average. The tell: results shift when the mix of runs changes.
• Defaulting to quantity as the lever. The tell: the recommendation is more aircraft, sorties or bombs, with no check against training or accuracy effects.
• Treating the quantitative basis as the whole story, or making the decision for the executive. The tell: no non-quantitative factors are flagged, or the report reads as an order.
• Finishing the analysis without imparting it intelligibly. The tell: the staff cannot restate the conclusions, and no briefing or plain graph exists.

• ▸Decision and owner: the specific decision the executive must make and the operation it concerns.
• ▸Measure of effectiveness: the single quantity used to compare options.
• ▸Component factors: the parts the measure decomposes into (e.g., load, accuracy, time on task).
• ▸Standardization: the standard conditions and weights applied (e.g., standard firing rate, ammunition quality, jq weights).
• ▸Sensitive parameters: the parameters the outcome depends on most, with how it scales (proportional, ratio, rate).
• ▸Option comparison: quantity-based option versus quality-based option on the same measure.
• ▸Result in plain form: the curve or graph and a one-sentence conclusion the executive can restate.
• ▸Non-quantitative factors for the executive: politics, morale, tradition or other items to add before deciding.
• ▸Briefing record: who was briefed orally and whether they could restate the conclusions.

1. 1Write the objective of the operation in one sentence before you pick any number. The operation exists to achieve something (safe passage of cargo, damage to the enemy, coverage of an area). The measure of effectiveness must express that objective, not a convenient byproduct.
2. 2List 2-4 candidate measures of effectiveness (MOEs) for the whole operation. The source calls this step devising 'various measures of effectiveness of the operation as a whole'. For a traffic system, the candidates are reduction of accidents, increase of traffic flow, or the ratio of the two.
3. 3Collect data on past operations of the same sort, ideally from a scientifically trained observer placed as close to the operation as is feasible. Check that you have many reports on operations that are roughly similar. Statistical analysis is not fruitful without them.
4. 4Compute each candidate MOE from the same data and compare them. Keep the measure that displays most fully how the outcome depends on the operational parameters, and that captures the properties of the outcome the executive most wants.
5. 5Choose the denominator deliberately. Test whether the raw count or the proportion of the total at risk is the right form. Raw losses per attack and percentage lost of all ships crossing give opposite conclusions.
6. 6Take the shortlisted MOEs to the executive who controls the operation and settle the choice together. The source says the executive 'must help decide' when the choice is not easy.
7. 7Use the chosen MOE to find the sensitive parameters. Correlate variations in the MOE with variations in operational details, then drop the less sensitive parameters from the first analysis.
8. 8Predict, then test. Make a preliminary prediction of the MOE under a changed procedure, run a trial change, and compare. Reduce the finding to standard practice only after the results justify the prediction. Keep observing, because the measure must be re-checked as the enemy or conditions change.

The source's convoy study shows the effect of the criterion on the conclusion. Raw sinking data showed that about the same number of ships were lost per attack from large convoys as from small ones. A thoughtless reading concludes that large convoys do not help. The analyst changes the MOE to the percentage lost of the total number of ships crossing the ocean. Under that measure a large convoy loses a smaller fraction, because the same number of losses is spread over more ships. The prediction of smaller percentage losses was then tried out at the request of the authority controlling convoys, and when the trials justified it, it became standard practice. The data did not change; the measure of effectiveness did. A second case, from section 3.4.4, uses the same logic. Antiaircraft guns on merchant vessels were nearly dropped because they did not shoot down enemy planes. The analyst pointed out that the guns reduced the enemy's accuracy enough to lessen the chance that the ship was sunk. The right MOE was the chance the merchant vessel survived, not planes destroyed.

• If two candidate MOEs give opposite verdicts on the same data, then stop and re-derive the denominator (total at risk versus per event) before you present any conclusion.
• If an asset appears to fail on its own output metric (guns that do not down planes), then ask whether it improves the outcome the operation actually cares about (the ship is not sunk).
• If the MOE choice is not clear-cut (accidents, flow, or their ratio), then put the options to the executive in charge and let them settle the choice. Do not decide it alone.
• If you have few similar operations to study, then do not rely on statistics, because large, unique actions give slow progress. Start with the numerous, simple, roughly similar operations.
• If a prediction about the MOE is confirmed by trial, then reduce it to standard practice. If it is not, then return to the parameter analysis.

• Measuring the effect of the tool, not the objective, for example counting planes shot down when the objective is ships saved. The tell: a valuable asset looks useless on your metric.
• Using raw counts where a proportion is needed. The tell: results look flat across sizes or scales (the same losses per attack in large and small convoys) while the exposure differs.
• Choosing the MOE without the executive. The tell: the report is accurate but nobody can act on it, because it addresses a question the decision-maker is not asking.
• Running naive statistics on observational data with too few similar operations. The tell: confident conclusions from a handful of dissimilar cases, with 'serious errors' on retest.
• Measuring effectiveness once and never re-checking. The tell: the enemy or conditions change (as U-boats changed their tactics after radar) and the MOE keeps reporting the old result.

• ▸Objective (one sentence): state what the operation exists to achieve.
• ▸Candidate MOEs (2-4): list each measure you could use for the whole operation.
• ▸Denominator for each: write whether it counts per event, per unit, or as a percentage of the total at risk.
• ▸Data source and sample: list how many roughly similar past operations you have and who observed them.
• ▸Result under each MOE: record the value from the same data for each candidate.
• ▸Conflicts between MOEs: note where candidates disagree and why.
• ▸Executive's decision: record the measure the decision-maker chose and the reason.
• ▸Sensitive parameters: list the parameters that move the chosen MOE most.
• ▸Prediction and trial: state the predicted MOE change, the trial procedure, and the observed result.

1. 1Fix the measure of effectiveness before you change anything. Choose the one number that carries the outcome you care about (U-boat sinkings per attack, bomb load times accuracy times operating hours, ship-months lost, waiting-line length) and record it for the old method over a comparable period.
2. 2Express the result as a ratio, new over old, not as a difference. Read it on the hemibel scale: about 2, about 3, about 10. A ratio near 1 means you have not moved the operation.
3. 3Check that the change cost little. Record what was added in equipment, money, and development time. The passages' strongest cases (depth-charge fuze setting, convoy size, tub rearrangement) changed use, not hardware. Large ratios from a small input are the signature of a real finding.
4. 4Compute the theoretical prediction before the trial and compare it with the observed ratio. For the tubs, 2 wash plus 2 rinse becoming 3 wash plus 1 rinse predicts a 50 per cent flow increase. Observed: the line fell to practically zero. A gap that large points to a self-aggravating mechanism (queues that grow because they are long), so name that mechanism.
5. 5Price the alternative route to the same ratio. Estimate what the same factor would cost by adding aircraft, sorties, explosive charge, or fire-control improvements. The depth-setting factor of 2 would otherwise have taken years and millions of dollars.
6. 6Translate the ratio into forces or units a decision-maker feels. A VLR plane was ten times as effective in November as in August, so one squadron outperformed three groups of August planes.
7. 7Decide against the hemibel threshold. If alternatives differ by less than about a factor of 2, treat them as equivalent and let nonquantitative factors decide. If they differ by an order of magnitude, the numbers should outweigh those factors.
8. 8Report the ratio with its exclusions stated, for example that morale effects are not in the figure. Then confirm it with a controlled follow-up where you can.

Take the VLR bomber group. The analysis breaks effectiveness into parts: bomb load carried, bombing accuracy, and the fraction of operational hours spent bombing. Training pulled hours away from bombing, but the loss was very small compared to the rise in bomb load and accuracy that training partly produced. Multiplying the components gives the overall measure. Plotted for August and November, it shows a ratio of ten: a VLR plane was ten times as effective in November as in August. You then translate the ratio: one squadron in November beat three groups in August, with essentially the same planes. Then you test the alternative: reaching that result by adding aircraft, flying more missions, or adding gadgets would have fallen far short. The 10x is an order-of-magnitude difference, so no unquantified consideration should outweigh it.

• If the new/old ratio is under about 2, then call the options effectively equivalent and decide on nonquantitative grounds.
• If the ratio is 10 or more, then treat the quantitative result as dominant over nonquantitative considerations and act on it.
• If a theory predicts a modest gain (50 per cent) and the observed gain is far larger, then look for a self-aggravating process such as queue growth, and do not assume an error.
• If the ratio came from a change in use rather than in equipment, then compare its cost against the years and millions of dollars an equivalent hardware gain would need, and favor the use change.
• If a gain is measured only in physical results, then state in the report that morale and similar effects are excluded.

• Reporting a difference or percentage without a baseline ratio. The tell: the reader cannot say whether the change is a 2x or a 10x.
• Treating small differences between disparate tactics as real. The tell: you rank options that differ by under a hemibel as if the quantitative comparison were accurate.
• Chasing the factor through numbers or gadgets: more aircraft, sorties, bombs. The tell: the proposal needs large equipment spend while the ratio from better use or training is never computed.
• Ignoring diminishing returns at high skill. Halving error doubles hits only when the error is much larger than the target width. The tell: a projected big gain from refining an already accurate bombardier.
• Using opinion or memory of vivid events instead of recorded outcomes to claim the improvement. The tell: the evidence is 'so-and-so says it worked' with no before/after data.

• ▸Measure of effectiveness: the single outcome number and its units.
• ▸Baseline value and period: old-method value over a comparable span.
• ▸Post-change value and period: new-method value over a comparable span.
• ▸Ratio (new/old) and hemibel band: compute it, then mark under 2, about 2, about 3, or about 10 or more.
• ▸Theoretical prediction: the ratio your model predicted before the trial.
• ▸Prediction gap and suspected mechanism: observed minus predicted, with the process that explains it.
• ▸Added equipment and cost: what the change required; write 'none' if only use changed.
• ▸Alternative-route estimate: cost and time to get the same ratio with more units or hardware.
• ▸Excluded effects and verification plan: what the number leaves out (e.g., morale), and the controlled test to confirm.

1. 1Name the executive who owns the operations: the commanding general, the vice-president in charge of operations, or the director of the governmental activity. Attach the group to that person. The source states the group "should not report to the research department of a military service or of an industrial organization."
2. 2Set up direct, personal contact in both directions. The group learns from the executive which questions need decisions, and the executive hears study results directly from the group. Schedule this contact; do not route it through an intermediary.
3. 3Keep the group as small as possible and keep all contact as personal as possible. The source ties this to OR being a staff function. Add people only when the work requires them.
4. 4Place the group's people with the operational command, as "scientists in the field," not with the service commands or bureaus. Morse's 1942 Anti-Submarine Warfare Operations Research Group worked under Admiral King, Commander in Chief of the U.S. Fleet.
5. 5Take problems from the executive's operations, not from the equipment-development side. The group consults for the user of equipment, not the builder. Keep the focus on strategy and tactics, not on devising or modifying equipment.
6. 6Use the group's access to operating data and field observation. In the mess-kit example, the worker saw the delay on his first day in a new field command and timed the washing and rinsing steps.
7. 7Let the group judge which long-range research is pertinent. The source says long-range work is needed, but it must be directed at the main goal, and the group is best placed to decide whether a given piece of research bears on the problem at hand.
8. 8Deliver reports with conclusions that form a basis for decision, and usually without recommendations. The executive keeps the decision, and the group keeps its freedom of action.

The source's mess-kit case shows what the attachment buys. An operations research worker, on his first day of assignment to a new field command, noticed soldiers queuing to wash and rinse mess kits at four tubs: two for washing and two for rinsing. Because he was embedded with the operational command, he could watch the operation directly. He observed that washing took on average three times as long as rinsing and asked the right question about relative time at each tub. He proposed three washing tubs and one rinsing tub. The change was made with no added equipment, and on most days no waiting line formed. A group attached to a development laboratory or a research branch would not have been standing at the mess line, and the problem would not have reached it. The same logic holds at fleet level: Morse's anti-submarine group sat under Admiral King, so it had access to the executive's questions and to operational data.

• If the proposed reporting line runs through a research department of a service or an industrial organization, then move it to the executive who makes the decisions, because the group needs direct and personal contact with that officer.
• If the group wants to add headcount or layers of liaison, then hold it small, because staff function and personal contact depend on size.
• If a problem is about equipment design or modification, then leave it to scientists attached to service commands or bureaus; the exception is when the group is the only technical presence with an operational command and the work would otherwise not get done.
• If findings are ready, then write conclusions that serve as a basis for decision and usually omit recommendations, because usurping the executive's duties costs the group freedom of action and costs the executive effectiveness.
• If a long-range research topic comes up, then let the group decide whether it is pertinent to the executive's problem, and drop it if it does not serve the main goal.

• Housing the group in a research or service branch. The tell: the group hears about problems second-hand and works on questions the executive did not ask.
• Contact becomes formal or filtered through intermediaries. The tell: the executive cannot say what the group is working on, and the group cannot say which decisions are pending.
• The group drifts into gadgeteering. The tell: its time goes to "playing" with new equipment instead of the refractory problems of strategy and tactics.
• The group grows large and bureaucratic. The tell: contact is no longer personal, and it no longer functions as a staff.
• The group makes the decision instead of informing it. The tell: reports are full of recommendations, and the executive defers to the group instead of deciding.

• ▸Executive served: name the person who makes the decisions over the operations (commanding general, VP operations, agency director).
• ▸Current reporting line: state who the group reports to today and whether it passes through a research or service branch.
• ▸Direct contact schedule: list how and how often the group meets the executive personally, and in which direction information flows.
• ▸Group size: record the headcount and why each person is needed for a staff function.
• ▸Questions received from executive: list the important decision questions the executive has given the group.
• ▸Field access: note which operations, sites, and data the group can observe first-hand.
• ▸Equipment/gadget work: list any equipment work taken on and whether it is the group's job or a service bureau's.
• ▸Long-range research items: for each item, state whether the group judged it pertinent to the main goal.
• ▸Report format: confirm that the report states conclusions as a basis for decision and leaves the decision to the executive.

1. 1Write down the outcome you care about before you look at any data. State the property of success in plain words (ships arriving, enemy planes failing to sink the vessel, traffic moving safely). The measure must track that property, not whatever is easiest to count.
2. 2Collect data on past operations and correlate variations in results with variations in operational parameters (convoy size, gun fitted or not, road width). This gives the rough dependence of results on those parameters.
3. 3Devise several candidate measures of the operation as a whole. For each, ask two questions: does it display most fully how the outcome depends on the parameters, and does it measure the properties of the outcome that are most desirable?
4. 4Test each candidate against the raw data. Be suspicious of a raw count that looks flat across conditions. Convert it to a rate against the total exposed (percentage lost of all ships crossing, not ships lost per attack) and see whether the dependence appears.
5. 5Ask what the thing being evaluated actually does. Antiaircraft guns on merchant ships did not shoot down planes, but they reduced the planes' accuracy and so the chance the ship was sunk. If the obvious measure shows no value, test for an indirect effect on the real outcome.
6. 6Use the chosen measure to make a preliminary prediction (large convoys give smaller percentage losses). Then check it with trials or new operational data under controlled conditions before turning it into standard practice.
7. 7If different measures lead to different optimum operations, do not pick silently. Set out the alternatives in the report, clearly enough for the executive to decide which criterion governs. Present the quantitative picture and the sensitivities, and leave the decision to the executive.

The convoy study is the passage's own case. Raw sinking data showed that about the same number of ships were lost per attack from large convoys as from small ones, so one might thoughtlessly conclude that large convoys would not help. The analysts instead used the percentage lost of the total number of ships crossing the ocean as the criterion. Under that measure the dependence on convoy size appeared: large convoys gave smaller percentage losses. That prediction was then checked by trials, and when the results justified it, it became standard practice. The authority controlling convoys had asked for the study and could act on it, so the finding had an effect. The passage notes that in many cases much of the problem is solved once the criterion is decided.

• If the raw count looks the same across different operating conditions, then divide by the total exposed or at risk and re-examine before concluding the parameter does not matter.
• If the item under evaluation fails on the obvious metric (guns that do not shoot planes down), then check whether it changes the probability of the real outcome (ship not sunk) before dropping it.
• If two reasonable measures (accident reduction, traffic flow increase, or their ratio) point to different best operations, then lay out each alternative in the report and have the executive choose the criterion.
• If a measure shows only favorable results from a favored activity (advertising sent to satisfied customers), then ask whether it measures the failures that matter (customers turning dissatisfied, prospects who never become regular) and consider a different criterion.
• If a candidate measure does not vary with the operational parameters you can control, then discard it, because it cannot guide decisions on them.

• Using a raw count where a proportion is needed: losses per attack look equal for large and small convoys. The tell is a flat count while the exposed population differs greatly across conditions.
• Judging equipment by its apparent function rather than its effect on the outcome: guns are rated by planes shot down. The tell is a proposal to drop something that nobody has tested against the real goal, ship survival.
• Choosing a measure that confirms what is already happening: advertising results look gratifying because the mailings go to satisfied customers. The tell is a high score from a measure that ignores lost or never-converted customers.
• Settling on a measure without telling the executive that alternatives exist. The tell is a report whose optimum changes if a different criterion is used, and the reader is never shown that.
• Studying for an authority that cannot act on the result. The tell is a good analysis delivered to someone without control over the operation, with no effect.

• ▸Operation and decision-maker: name the operation and the executive who controls it and can act on the result.
• ▸Desired property of success: one sentence on what a good outcome means in operational terms.
• ▸Operational parameters under control: list the variables the executive can change (size, width, equipment fitted, tactics).
• ▸Candidate measures: list 2-4 measures (counts, rates, ratios, percentage of total at risk).
• ▸Denominator or exposure: for each candidate, state what total it is measured against.
• ▸Dependence test: for each candidate, record how it varies with each parameter in past data and whether the pattern is clear.
• ▸Desirability test: record whether the candidate tracks the desired property, including indirect effects.
• ▸Conflicts between measures: note where candidates recommend different optimum operations.
• ▸Executive's choice and check: record the criterion the executive selects and the trial that will test the prediction.

1. 1Name the operation and the decision it informs. Morse and Kimball define OR as a quantitative basis for decisions on operations under an executive's control. Write down who decides, what they control, and what outcome measure (for example, U-boat sinkings or ship loss per convoy) the decision moves.
2. 2Study past operations first to establish the facts. The sequence is: study past operations to determine the facts, elaborate theories that explain them, then use both to predict future operations. Pull the reports before you build any model.
3. 3Check that the data suit statistics. Statistical analysis needs a large number of roughly similar operation reports. Bombing a given target type under similar weather and opposition qualifies. Large, rare, complex actions do not, so expect slow progress there.
4. 4Classify the data as observational or experimental. Uncontrolled operations are observational, and naive statistical techniques on them can produce serious errors. Record what was not held constant before you draw conclusions.
5. 5Report the spread as well as the average. Nearly all operational results involve chance, and it is nearly as important to know how far individual operations differ from the expected average as to know how the average depends on the variables. With meager, fragmentary data, estimate how likely the next operations are to resemble the ones analyzed.
6. 6Fit an analytical model to the facts and test the fit. Use a chi-square comparison of observed counts against expected counts. The trial with the smallest χ² fits best, and degrees of freedom s = ν − 1. P(>χ²) is small when χ² exceeds s and near 1 when χ² is much smaller than s.
7. 7Look for self-aggravating (nonlinear) behavior. Waiting lines and traffic jams get worse as they grow, so a small capacity change can produce a larger effect than a linear estimate predicts. Probability theory handles these cases.
8. 8Where past data cannot answer the question, plan an operational experiment with equipment or tactics (Chapter 7's subject). Then take the result to the administrator who has authority to act, because OR done apart from that person is an empty exercise.

The source's clearest case is the depth-setting of aircraft antisubmarine depth charges (section 3.4.8, cited in section 1.2.2). Operators wanted more U-boat sinkings. The route most people would think of was a bigger explosive charge or better fire control. Instead, a detailed probability study, combined with knowledge of the physical properties of depth charge fuzes, led to a recommendation to change only the depth-setting. Sinkings improved by a factor of 2 with no substantial change in equipment. The passage says an equivalent gain from a larger charge or better fire control would have taken years of development and millions of dollars. A second example comes from convoy size (section 3.2.2). The decision to increase convoy size reduced average ship loss, again with no equipment change. A third comes from a laundry line. Converting two wash tubs plus two rinse tubes to three wash tubs plus one rinse tub was predicted to raise flow by 50 per cent. The waiting line dropped to practically zero, which beat the prediction. This illustrates the self-aggravating property of queues.

• If you have many simple, roughly similar operation reports (bombing a given target type, sales, traffic), then use statistical analysis of them. If the action is large, rare, and multi-factor, then expect slow progress and do not claim precision.
• If the data come from uncontrolled operations, then treat them as observational and avoid naive statistics. If the question cannot be settled that way, design an operational experiment.
• If the system's trouble grows as it grows (queues, congested highways), then model it as nonlinear and self-aggravating. Look for a small structural change, not a proportional capacity increase.
• If you need a result at a given effort, then look first at settings and arrangements (fuze depth, convoy size) before new equipment. These gave large effects with no substantial equipment change.
• If the chi-square P(>χ²) is small (χ² above s), then reject the model as a good fit. If χ² is much smaller than s, then accept it. Among competing trials, prefer the smallest χ².
• If no administrator with authority will act on the result, then do not proceed. The source calls such work an empty exercise.

• Treating observational data as if it came from a controlled experiment. The tell: a conclusion about cause drawn from operations in which conditions were not held constant.
• Reporting only the average. The tell: no statement of how far individual operations vary around it, so the next operation's likely outcome is unknown.
• Applying statistics to too few or too dissimilar reports. The tell: a confident figure drawn from a handful of unlike large-scale actions.
• Assuming linear response in a self-aggravating system. The tell: a prediction (like the 50 per cent flow gain) that real results then exceed or miss by a wide margin.
• Defaulting to more equipment (the 'gadgeteer' solution). The tell: a recommendation needing years and millions when a setting or unit-size change was never examined.

• ▸Decision and decision-maker: state who decides what, and which operation they control.
• ▸Outcome measure: name the quantity that shows success (for example, sinkings or ship loss).
• ▸Past-operation data: list the sources and the number of roughly similar reports.
• ▸Data type: mark observational or experimental and note what was uncontrolled.
• ▸Expected average and spread: give the expected value and the typical deviation of individual operations.
• ▸Model and fit check: name the model, then give observed vs expected counts, χ², s = ν − 1, and P(>χ²).
• ▸Self-aggravating effects: note any nonlinearity (queues, jams) that breaks linear prediction.
• ▸Cheapest lever: list settings or arrangements to change before new equipment.
• ▸Experiment or action plan: state the planned test or decision and the authorized administrator.

1. 1Name the operation and pick the measure of effectiveness before collecting anything. Write the measure down and get the executive to confirm it, because the executive must help decide it when the choice is not obvious. Ask whether the thing you are measuring is the real purpose or a proxy for it. Examples: percentage of ships lost, not ships lost per attack; reduced enemy accuracy, not planes shot down.
2. 2Strip the operation to a few broad 'constants' and estimate each to one significant figure, or within a factor of about 3 (a hemibel). Extra precision at this stage is unessential detail. You are looking for large factors of improvement, not small ones.
3. 3Compute the theoretical optimum for each constant where you can, and compare the actual value with it. If the actual value is within a factor of 3 of the optimum, there is little to gain from this component. If it is far below, look for the single faulty component.
4. 4Name the bottleneck as one of three types: inadequate training of crews, incorrect use of equipment, or inadequate equipment. Rule out the first two before recommending new equipment. Fixing the bottleneck can raise effectiveness by hundreds or thousands of per cent.
5. 5Build the factual base from recorded actions, not recollections. Get detailed action reports and witness the operation yourself. Treat interviews with operating personnel as a source of insight and hypotheses, not as data, because people remember the 'exciting one' and forget the rest.
6. 6Use past operations to find how results depend on operational parameters, including chance fluctuations in the operation. If that is not enough, change a parameter temporarily, such as a tactic, a firing or operating pattern, or a layout. Make the change as small as possible until you know which direction the effect runs. Clear this with the senior executive, because it is a serious decision.
7. 7Design the trial as a measured experiment, not a training exercise. Lay it out so the behavior of the forces can be measured quantitatively. Compare the result with your prediction.
8. 8If the trial confirms the prediction, reduce the finding to standard practice. Deliver it to the executive in a form he can act on, and leave the decision to him. The project is not complete until the executive has received the finding in usable form. In a measure/countermeasure contest, speed matters: the side that adapts its tactics first holds the advantage.

Convoy size is the passage's case of tactics changing outcomes with no new equipment. The raw sinking data showed that about the same number of ships were lost per attack from large convoys as from small ones. A naive reading concludes that large convoys do not help. The analyst changed the criterion to the percentage lost of all ships crossing the ocean, and the answer became clear: large convoys lose a smaller percentage. Enough data on convoys of different sizes already existed to support a preliminary prediction. Trials were then instituted to check it. When the results justified the prediction, the finding was reduced to standard practice. The study worked because it was made at the request of the authority controlling convoys, who could act on it; the same report handed to any other authority would have had no effect. The same lesson appears with antiaircraft guns on merchant vessels. They were nearly dropped because they did not shoot down planes, until it was shown that they decreased the enemy's accuracy enough to lessen the chance the vessel was sunk.

• If an operation performs far below its computed optimum, then look first for a faulty component in crew training or equipment use before proposing new equipment.
• If the data cannot say which parameter drives the result, then change one parameter temporarily, keep the change as small as possible, and get the senior executive's approval.
• If the raw numbers suggest a tactic does nothing, then recheck the measure of effectiveness before dropping the tactic. Per-attack losses and planes shot down both misled in the passage's cases.
• If the evidence is operators' opinions or memories, then use it only to generate hypotheses. Base conclusions on action reports and observed operations.
• If a prediction is confirmed in trial, then write it into standard practice, and re-test when the opponent changes tactics or equipment.

• Wrong measure of effectiveness. The tell is a conclusion like 'large convoys don't help' or 'AA guns are useless' that rests on a per-event count or a proxy outcome, not on the percentage lost or the effect on the enemy.
• Reflexive equipment purchase. The tell is a proposal for new gear when no one has compared actual performance with the theoretical optimum or checked crew training and how the existing equipment is used.
• Reliance on expert opinion or polls. The tell is 'so-and-so can tell you all about it' replacing action reports and witnessed operations, and results that match the memorable incidents.
• Over-precise analysis. The tell is weeks spent refining constants to several digits while the factor-of-3 bottleneck goes unfound.
• Analyst or executive role confusion. The tell is a report that issues the decision itself, or one handed to someone who cannot act on it; either way the finding is never put into practice. A related tell is an exercise that exists as training but never yields measurements.

• ▸Operation: describe it in one line, and name the executive who controls it and can act on findings.
• ▸Measure of effectiveness: state the criterion (for example, percentage lost, not number lost) and record the executive's agreement.
• ▸Constants of the operation: list each to one significant figure or within a factor of 3.
• ▸Theoretical optimum vs actual: for each constant, note the optimum and the actual value, and whether the gap exceeds a factor of 3.
• ▸Suspected bottleneck: choose crew training, equipment use, or equipment inadequacy, and cite the recorded evidence (action reports, observed operation) behind it.
• ▸Opinion-derived hypotheses: list claims from operators and mark each as unverified until checked against recorded actions.
• ▸Parameter to vary and size of change: name the tactic or practice, make the change as small as possible, and record the executive's approval.
• ▸Prediction and trial result: state the predicted effect before the trial, then the measured result under controlled conditions.
• ▸Disposition: record whether the result goes to standard practice, to a further trial, or is rejected, and the date it was passed to the executive.

1. 1Open with the source's gate: statistical analysis is not fruitful unless a large number of reports on roughly similar operations are available. Before choosing any method, count how many comparable operations exist and how many have usable records.
2. 2Classify each operation type by the three properties the source names: numerous, simple, roughly similar. Bombing a target of a given type under similar weather and similar enemy opposition passes. Larger actions of warfare or industry, with few occasions, great complexity, and many independent factors, do not.
3. 3Declare the data observational or experimental. Operations are often uncontrolled in the scientific sense, so their records are observational, not experiments. Write that down, because it sets how careful the analysis must be: naive statistical techniques on observational data can lead to serious errors.
4. 4Audit the recording of each operation. For the AA gunnery case, check that the range of every hit was recorded, that the firing rate and ammunition quality (fraction of shells not duds) were recorded for each run, and that the target type and run type (altitude, speed, line of approach) were noted. If a variable that changes between runs was not recorded, you cannot standardize across runs.
5. 5Standardize before pooling. Pick a standard firing rate and standard ammunition quality. Weight each run by jq, where j is the firing-rate ratio and q is the ammunition-quality ratio, so that it counts as jq standard runs. Analyze different run types separately instead of mixing them.
6. 6Compute the number of effective standard runs, not the raw number of runs. Use it as the denominator: rank the hits by decreasing range, number them serially, and divide each serial number by the total standard runs to get average hits outside each range. Plot against range to get the hit expectancy curve.
7. 7Match the scope to the data. Where occasions are few, start with the numerous, simple sub-aspects that can be studied statistically, and expect slow progress on the larger action. Do not force a statistical answer onto the whole action.
8. 8Secure access to the data before you start. The field man should agree with the commanding officer that, as a scientist, he has complete access to the data pertaining to the assigned problem, and he should reach that agreement at the outset of the assignment.

The source's AA gunnery evaluation shows the standard for usable operational data. A drone approaches the gun position using evasive maneuvers like those a real enemy would use, and firing continues at a known rate for the whole run. The range of each hit is recorded. The drone need not match the enemy craft in size, but the ratio of effective target areas must be known. Different run types (altitude, speed, line of approach) are tried and analyzed separately. Firing rate and ammunition quality vary from run to run, so both are recorded. Against a standard rate and standard quality, a run with firing rate j times standard and ammunition q times standard counts as jq standard runs. All hits are ordered by decreasing range and numbered serially. Each serial number divided by the total standard runs gives the average hits outside that range. Plotted against range, this is the hit expectancy curve for a standard run. Because the curve is proportional to firing rate and to ammunition quality, a change in the number of guns or in the firing rate per gun is easy to read off. Without the recorded rates and quality, the runs could not be pooled.

• If you can count many reports of roughly similar operations (numerous, simple, similar conditions), then use statistical analysis; if the occasions are few and complex, expect slow progress and do not promise statistical conclusions.
• If the operations were not controlled, then treat the data as observational and avoid naive statistical techniques, since they can produce serious errors.
• If a condition varies from run to run (firing rate, ammunition quality, run type), then record it during the operation, normalize with a weight such as jq, or split the analysis by type. Never pool the runs raw.
• If the available target or test object differs from the real one, then proceed only if you know the ratio of effective areas.
• If you cannot obtain complete access to the data pertaining to the problem, then settle access terms with the commanding officer before accepting the problem, because the analysis depends on it.

• Treating large-action data as if it were numerous: a handful of big engagements is analyzed statistically. The tell is a conclusion resting on a few occasions with many independent factors.
• Treating uncontrolled operational records as experiments and running naive statistics. The tell is a confident result that confounds conditions the operations never held constant.
• Pooling runs without recording or adjusting for firing rate, ammunition quality, or run type. The tell is a hit expectancy curve that cannot be reconciled with any single standard run.
• Counting raw runs instead of standard runs. The tell is a denominator that ignores jq weighting, so hits per run is biased up or down.
• Starting work without agreed data access. The tell is analysis that stalls or rests on partial records because the commanding officer controls the data.

• ▸Operation type: name the class of operation under study.
• ▸Number of comparable operations on record: count roughly similar, well-documented cases.
• ▸Simplicity and similarity: note whether the operations are numerous, simple, and done under similar conditions (weather, enemy opposition).
• ▸Controlled or observational: state which, and the statistical cautions that follow.
• ▸Variables recorded per operation: list each (e.g., range of hit, firing rate, ammunition quality, run type).
• ▸Variables varying but unrecorded: list the gaps that block standardization.
• ▸Standard conditions and weights: define the standard and the weight formula (e.g., jq).
• ▸Effective standard-run count: total weighted runs available for analysis.
• ▸Data access agreement: record whether complete access is confirmed with the commanding officer.
• ▸Verdict and scope: state whether to analyze statistically, narrow to a sub-aspect, or defer.