Parsing Messy, Real-World Data

TL;DR: Real-world data is messy, inconsistent formats, missing fields, bad encodings, malformed and unexpected records, and a parser that assumes clean input breaks immediately in production. Defensive parsing means handling the unhappy path deliberately: validate input, decide per record whether to skip, default, or fail, isolate failures so one bad record does not crash the whole batch, and log what you dropped. Ingesting documents and data for AI systems is half the job, so this is a constant, often-screened skill.

Clean input is a fantasy

Demos run on tidy data; production runs on whatever the source actually produces: CSVs with stray commas and inconsistent quoting, JSON with missing or extra fields, mixed encodings (UTF-8 with a stray Latin-1 byte), dates in five formats, truncated records, and inputs nobody anticipated. A parser written for the happy path fails on the first surprise, often taking the whole job down. The skill is assuming mess and handling it on purpose.

Decide what to do with bad records

The core design decision is, per record, what happens when parsing fails:

• Skip and quarantine: drop the bad record to a dead-letter location and log it, so the batch continues and you can inspect failures. The common default for non-critical data.
• Default / coerce: fill a sensible default or normalize the value when safe (and recorded).
• Fail fast: halt only when a record (or a rate of failures) indicates something truly broken upstream, you do not want to silently process garbage either.

The point that separates a junior answer from a senior one: isolate failures. One malformed record must not crash the parse of the other million. And track drop rates, a spike is a data-quality signal.

Worked example: a parser that survives the batch

This is the shape interviewers want. Per-record try/except, a dead-letter list, a typed coercion, and a failure-rate circuit breaker so a corrupt upstream file fails fast instead of quarantining a million rows silently.

import csv, json, logging
from datetime import datetime

def parse_rows(path, max_drop_rate=0.05):
    good, dead = [], []
    with open(path, encoding="utf-8", errors="replace", newline="") as f:
        rows = list(csv.DictReader(f))
    for i, row in enumerate(rows):
        try:
            amount = float(row["amount"])              # coerce; may raise
            ts = parse_date(row.get("date") or "")      # None-safe; normalize 5 formats
            if amount < 0:
                raise ValueError("negative amount")     # validate, not just parse
            good.append({"id": row["id"], "amount": amount, "ts": ts})
        except (KeyError, TypeError, ValueError) as e:
            dead.append({"line": i, "row": row, "error": str(e)})
    if rows and len(dead) / len(rows) > max_drop_rate:  # circuit breaker
        raise RuntimeError(f"drop rate {len(dead)}/{len(rows)} exceeds {max_drop_rate}")
    logging.warning("quarantined %d of %d rows", len(dead), len(rows))
    return good, dead

def parse_date(s):
    for fmt in ("%Y-%m-%d", "%m/%d/%Y", "%d-%b-%Y"):
        try: return datetime.strptime(s.strip(), fmt)
        except ValueError: continue
    raise ValueError(f"unparseable date: {s!r}")

Concretely: feed it a 10,000-row file where 30 rows have a blank amount and 1 row is truncated. You get 9,969 good records, 31 in dead with line numbers and reasons, and the batch finishes. Now corrupt the encoding so 600 rows fail: the drop rate hits 6%, the breaker trips, and you find out at parse time rather than discovering missing revenue a week later. errors="replace" keeps a single bad byte from killing the read; the per-record try keeps a single bad row from killing the loop.

Validate, normalize, and be explicit

• Validate structure and types as you parse (the data-quality checks at the record level). Parsing succeeds means "I got a float," validation means "the float makes sense."
• Normalize into a consistent internal shape (canonical dates, units, encodings) so downstream code is simple.
• Be explicit about assumptions and edge cases; messy data punishes implicit ones. This connects to streaming/backpressure (parse without loading everything, swap list() for a generator on large files) and to building testable parsers you can throw bad inputs at.

Why interviewers probe this

Data ingestion is half of building AI systems (documents for RAG, records for features), and a brittle parser fails on day one, so this is a frequent coding screen. A strong answer assumes messy input, handles the unhappy path with a deliberate per-record policy (skip/default/fail), isolates failures so one bad record does not crash the batch, validates and normalizes, and logs drops. The tell they listen for: do you wrap the loop body, not the loop, and do you watch the drop rate? That production-realism, not clean-input optimism, is the signal.

Common misconceptions

• "Assume the input is well-formed." It never is; design for malformed, missing, and unexpected records.
• "One bad record can crash the batch." Isolate failures; quarantine bad records and keep going.
• "Always skip bad records." Sometimes default/coerce, sometimes fail fast, choose deliberately, and never silently drop everything.
• "Drop rates do not matter." A spike in dropped records is a data-quality alarm; track it and trip a circuit breaker.

Key takeaways

• Real data is messy; parsers must handle the unhappy path, not just the happy path.
• Decide per record whether to skip-and-quarantine, default, or fail fast, and never silently drop everything.
• Isolate failures (wrap the record body) so one bad record does not crash the batch; validate, normalize, and log drops.
• Track the drop rate and trip a circuit breaker on a spike; ingestion is half of AI engineering.