# Sync Engine Architecture
## Overview
The sync engine is a Rust binary that pulls HL7/FHIR payloads from legacy hospital systems, normalizes them, and writes them to Postgres. It is designed to be boring, deterministic, and idempotent.
## Crate Structure
- `src/main.rs`: Entry point. Parses CLI args, initializes the database pool, and starts the batch loop.
- `src/sync/batch.rs`: Core batch processing logic. Reads a batch of pending records from the `sync_queue` table, processes them, and updates their status.
- `src/sync/normalize.rs`: Transforms raw HL7/FHIR JSON into our internal canonical schema.
- `src/db/pool.rs`: Manages the Postgres connection pool. Uses `sqlx` for async queries.
## Data Flow
1. **Ingestion**: A background job polls the legacy system's API or reads from a file drop.
2. **Queueing**: Raw payloads are inserted into the `sync_queue` table with a `status = 'pending'`.
3. **Processing**: The batch loop selects up to `N` pending records. For each record:
- Validate the payload.
- Normalize the data.
- Upsert into the `patient_records` table using a unique constraint on `(source_id, record_hash)`.
4. **Commit**: If all records in the batch succeed, update their status to `'completed'`. If any fail, mark them `'failed'` and log the error.
## Idempotency
- Every record has a deterministic `record_hash` computed from its canonical form.
- The `patient_records` table has a unique index on `(source_id, record_hash)`.
- This ensures that reprocessing the same record multiple times results in no duplicate rows.
## Failure Handling
- Failed records are not retried automatically. They are marked `'failed'` and require manual intervention via the admin console.
- The batch loop halts if more than 5% of records in a batch fail, to prevent cascading errors.
## Testing
- Unit tests for `normalize.rs` cover all HL7/FHIR edge cases.
- Integration tests spin up a local Postgres instance and verify that duplicate records are not inserted.
## Notes for Juniors
- Do not modify the `sync_queue` schema without consulting Priya or Marcus.
- Always run `cargo test --test integration` before merging changes to the sync logic.
- The `record_hash` function is in `src/sync/hash.rs`. Do not change it without understanding the implications for idempotency.