Choose a Missing-Data Strategy: A Sensitivity Workshop

A missing reading is not automatically an observation of zero. This workshop uses six fictional entries to compare exclusion, zero filling and mean filling. The objective is to make assumptions visible, not to select a universally correct technique. You will calculate the consequences of each policy, retain the missing count and explain which conclusions the tiny fixture cannot support.

Documentation boundary

Python’s math reference documents isnan for detecting a floating-point NaN. This fixture also uses None as an explicitly declared missing marker. The original cleaning rule checks both representations and preserves a numeric zero as an observed value.

Declare the missing markers and the question

The entries are ten, None, twenty, zero, NaN and thirty. There are four observed values and two missing entries. No reason for missingness is supplied. First calculate the mean among observed readings. Then explore what the six-entry mean would become under two deliberately stated filling rules. Do not treat the scenario calculations as recovered measurements: the missing values remain unknown in the original data.

Write the policy names beside the outputs before running the code. If a report only says “the average is fifteen”, a reader cannot tell whether it describes four observed values or six values after filling. The numerical equality of two results does not make their evidence or assumptions identical. Your answer should preserve that distinction even when the headline number stays unchanged.

Work the three scenarios

from math import isnan

values = [10, None, 20, 0, float("nan"), 30]
observed = [x for x in values if x is not None and not isnan(x)]
missing_count = len(values) - len(observed)
observed_mean = sum(observed) / len(observed)
zero_filled = [x if x is not None and not isnan(x) else 0 for x in values]
mean_filled = [x if x is not None and not isnan(x) else observed_mean for x in values]
assert observed == [10, 20, 0, 30]
assert missing_count == 2
assert observed_mean == 15
assert sum(zero_filled) / len(zero_filled) == 10
assert sum(mean_filled) / len(mean_filled) == 15
assert min(observed) == 0
print("observed4; missing2; observed mean15; zero-fill mean10")

The observed sum is sixty and the observed count is four, giving fifteen. Zero filling keeps the sum sixty but changes the denominator to six, giving ten. Mean filling adds two supplied fifteens, producing ninety divided by six, again fifteen. The valid zero remains in the observed list throughout. It is not a missing marker under the stated contract.

The filling rules are sensitivity scenarios, not recommendations for every dataset. The fixture does not tell you whether absent readings tend to be unusually high, unusually low or unrelated to their values. You therefore cannot identify the true six-entry mean from the supplied information. A responsible handover records the chosen analytical question and any assumption needed to move beyond the observed readings.

Diagnose a hidden denominator change

A faulty calculation adds the observed values but divides by the full six-entry length while claiming to have excluded missing values. It reports ten without acknowledging that it effectively assigned no contribution to the missing entries. Locate the disagreement between the stated policy and denominator. Repair the description or the calculation; do not leave two incompatible claims in the final report.

After missing markers are correctly identified, a faulty extra step filters the observed values by whether they are truthy. That removes the genuine zero. The already-observed list would then contain ten, twenty and thirty, giving a mean of twenty. This is a different dataset created by an unintended selection rule. Check the retained values, not just whether the code completes.

Choose and defend a limited strategy

For a descriptive report of measured readings, submit the observed mean fifteen, observed count four and missing count two out of six. If asked for a completed six-entry scenario, name the filling rule explicitly and retain the original values separately. Explain that a filled value is an analytical assumption. Do not overwrite the missing markers in the original fixture and later present the supplied values as measured facts.

For further practice, suppose the two unknown readings are both thirty in a clearly labelled hypothetical scenario. The six-entry mean would then be twenty. Compare that result with ten under zero filling and fifteen under mean filling. The range illustrates sensitivity to assumptions; it is not a confidence interval. Finish with the table, retained-value list and a short statement of what additional information about missingness would help choose a method.

Questions for this exercise

Official documentation

• NaN detection

Explore the Courses overview or use Contact for general enquiries. This is an original INTIAU editorial self-study exercise; no qualification is awarded.

View the study diagram