The ingestion layer is where most data quality problems are born and where they are cheapest to fix. A null that enters undetected at ingest becomes a missing dimension in a downstream model, which becomes an unexplained variance in a board-level KPI, which becomes a retrospective investigation consuming two engineers and a week of calendar time. The economics of catching problems at the source versus discovering them in production are not close.
A production-grade ingestion pipeline is not a scheduled script that moves records from A to B. It is a system with contracts. The contract specifies the schema the source is expected to produce, the freshness window within which data must arrive, the cardinality constraints that should hold on key fields, and the volume envelope that separates a slow day from a broken feed. When any of these contracts are violated, the pipeline fails loudly, routes the raw payload to a quarantine zone, and generates an incident with enough context for the on-call engineer to understand the nature of the failure without reading source code.
Change data capture is the architectural pattern that separates reactive pipelines from responsive ones. Polling a source system every four hours produces data that is four hours stale at worst — acceptable for some analytical use cases, completely inadequate for operational intelligence. CDC subscribes to the transaction log of the source database and propagates row-level changes in near real time, enabling downstream consumers to operate on current state rather than the last snapshot. The implementation complexity is real — log format differences across database engines, handling schema changes in the log stream, managing consumer lag during backfill operations — but the operational advantage justifies it for any source that feeds time-sensitive decisions.
Idempotency is the property that separates pipelines that can be safely re-run from pipelines that require surgical intervention when something goes wrong. An idempotent pipeline can be executed multiple times against the same input and produce the same output. This sounds like a basic requirement. Most pipelines built under time pressure do not have it. When a partial failure at two in the morning requires a manual decision about which records were successfully written and which need to be replayed, the cost of not having built idempotency in from the start becomes very concrete. The correct time to enforce this constraint is in the design phase, before the first line of transformation logic is written.