Create a Data Dictionary Before Analysis: Worked Workshop

Before calculating an average, decide what one row represents and what each field means. This exercise uses four fictional packets. A zero mass, an unknown mass and an absent field must not collapse into one category. You will write a small data dictionary, apply it to the records and identify the questions that remain outside the dictionary’s scope.

Documentation boundary

The official JSON Schema object reference distinguishes defining a property from requiring its presence. It also distinguishes a property whose value is null from an absent property. This exercise borrows those distinctions for a human-readable dictionary; its small Python checker is custom code, not a complete JSON Schema validator.

Define the unit of observation

One row describes one fictional packet at one inspection. Packet is a nonempty identifier unique within this fixture. Mass is measured in grams and must be nonnegative when known. None records an unknown measurement; it is allowed but must not be included as a measured zero. State must be either sealed or open. All three fields must be present, even when the allowed mass value is unknown.

The dictionary should also say what it does not establish. It does not prove that an instrument was calibrated, that a packet was inspected only once outside this fixture, or that the fictional measurements represent a larger population. A field definition supplies interpretation and validation rules; it does not provide external evidence about how observations were produced.

Apply the dictionary before summarising

records = [
    {"packet": "P01", "mass_g": 0, "state": "sealed"},
    {"packet": "P02", "mass_g": None, "state": "open"},
    {"packet": "P03", "mass_g": -2, "state": "sealed"},
    {"packet": "P04", "mass_g": 4},
]
def issues(row):
    errors = []
    if set(row) != {"packet", "mass_g", "state"}:
        errors.append("field set")
    if not isinstance(row.get("packet"), str) or not row.get("packet"):
        errors.append("packet")
    mass = row.get("mass_g")
    if mass is not None and (type(mass) not in (int, float) or not (0 <= mass < float("inf"))):
        errors.append("mass")
    if row.get("state") not in {"sealed", "open"}:
        errors.append("state")
    return errors
assert [issues(r) for r in records] == [[], [], ["mass"], ["field set", "state"]]
assert len({r["packet"] for r in records}) == len(records)
print("P01 valid zero; P02 allowed unknown; P03/P04 rejected")

P01 is valid with a measured zero. P02 is structurally valid but has an unknown mass. P03 violates the nonnegative mass rule. P04 lacks the required state field, which also prevents it from meeting the state category rule. The worked answer deliberately reports both relevant issues for P04 rather than inventing a state to make the row pass.

The uniqueness check is dataset-wide, separate from the row-level checks. A row can satisfy every field rule while repeating an identifier already used elsewhere. Keep that distinction in your handover. The custom checker assumes the exercise’s stated Python representations and is not a general-purpose validator for every numeric object, schema vocabulary or interchange format.

Diagnose an ambiguous column

Imagine replacing mass_g with a column named value and leaving the unit unspecified. Another learner could read four as kilograms while you intended grams. No amount of arithmetic would resolve the disagreement. Repair the dictionary first: restore the unit, explain the measurement meaning and state the accepted range. A precise column name helps, but the accompanying definition is still necessary.

A second flawed dictionary says “blank means none” without explaining whether that means not measured, not applicable or a measured absence. Under this workshop’s contract only unknown measurement is allowed. If you want multiple missing reasons, add a separate reason field and define its categories. Do not guess a reason from an empty cell after the data has already been collected.

Produce a usable dictionary handover

Submit the field table, the four records, the issue lists and the dataset-wide uniqueness result. Add two example sentences: P01 contributes a measured zero; P02 contributes to the unknown count. Then state which records are eligible for a measured-mass summary after invalid rows are excluded. The current answer has one eligible measurement, so a sweeping statement about typical packet mass would be unsupported.

For a controlled extension, make P02 a measured mass of six without changing its identifier or state. The dictionary stays unchanged, but the number of eligible measurements rises from one to two. Alternatively add a second P01 row and observe that row validity does not settle identifier uniqueness. These variations show why a dictionary should be written before choosing the calculation you hope the dataset will support.

Questions for this exercise

Official documentation

• Object properties

Explore the Courses overview or use Contact for general enquiries. This is an original INTIAU editorial self-study exercise; no qualification is awarded.

View the study diagram