# Sync Engine Onboarding Guide
## 1. Core Philosophy
The sync engine is designed for **zero data loss**. We prioritize reliability over speed. If a record cannot be synced safely, it is held in a dead-letter queue rather than dropped. Never assume a message is processed just because the network call succeeded.
## 2. Architecture Overview
- **Ingestion**: HL7/FHIR payloads arrive via HTTP POST.
- **Validation**: Payloads are validated against FHIR schemas. Invalid payloads are rejected immediately with a 400.
- **Queueing**: Valid payloads are written to a Postgres `pending_messages` table with a unique `message_id`.
- **Processing**: A Rust worker loop polls `pending_messages`, processes them, and updates status.
- **Idempotency**: Every message has a unique `message_id`. If a duplicate arrives, the engine ignores it. This is critical for hospital systems that retry on timeouts.
## 3. Data Flow & Transaction Boundaries
1. **Insert**: A new message is inserted into `pending_messages` in a single transaction.
2. **Claim**: The worker claims a message by updating its status to `processing` and setting a `claimed_at` timestamp. This is done in a separate transaction to prevent double-processing.
3. **Execute**: The sync logic runs. If it fails, the status is set to `failed` and the error is logged.
4. **Complete**: On success, the status is set to `synced` and the record is written to the target system.
## 4. Dead-Letter Queue (DLQ)
If a message fails 3 times, it is moved to `dead_letter_queue`. An alert is triggered. **Do not manually delete DLQ records.** They are the audit trail for any data loss incidents.
## 5. Key Tables
- `pending_messages`: Current queue.
- `synced_records`: Historical log of successfully synced records.
- `dead_letter_queue`: Failed records requiring manual review.
## 6. Common Pitfalls
- **Duplicate IDs**: Always check for existing `message_id` before inserting.
- **Long Transactions**: Keep transactions short. Do not hold locks during network calls to the target hospital system.
- **Silent Failures**: Never swallow errors. Log them and update the message status.
## 7. Testing
- Use `cargo test` for unit tests.
- Integration tests are in `tests/sync_flow.rs`. Run them against a local Postgres instance.
- Always test the duplicate ID scenario. It is the most common bug in hospital integrations.
## 8. Next Steps
-