# Data Pipeline Domain
## Approach
- Stages: extract -> validate -> transform -> load, each idempotent.
- Checkpointing and retry with backoff.
- Schema validation at boundaries; fail fast on drift.
- Stream large inputs; avoid loading everything into memory.
- Observability: counts, durations, errors per stage.
## Common Patterns
- ETL/ELT with staging tables.
- Incremental loads (watermarks).
- Dead-letter queues for bad records.
- Idempotent upserts.
## Testing
- Unit: transforms on sample fixtures.
- Integration: end-to-end on small dataset.
- Property: invariants hold across arbitrary inputs.
## Pitfalls
- Partial failures leaving inconsistent state.
- Schema drift breaking downstream.
- Memory blowup on large inputs.
- Non-idempotent reruns causing duplicates.
- Silent data loss on parse errors.
"