# Sync Engine Architecture Onboarding Guide
Welcome to the Ferrite Sync Engine. This document is designed to help you understand the core components and data flow of the system without needing to reverse-engineer the codebase.
## High-Level Overview
The sync engine is a Rust-based service that moves HL7/FHIR data between legacy hospital systems. It is designed for reliability and data integrity, prioritizing "boring" robustness over flashy features.
## Core Components
1. **Ingest**: Handles incoming HL7/FHIR messages, validates them, and assigns a unique batch ID.
2. **Transform**: Maps legacy data structures to the internal FHIR model. This stage is where most schema-specific logic lives.
3. **Commit**: Writes the transformed data to the Postgres database. This stage is strictly idempotent to prevent duplicate records.
## Data Flow
1. **Message Arrival**: A message arrives at the Ingest stage.
2. **Validation & Batch Assignment**: The message is validated against the expected schema. If valid, a batch ID is generated.
3. **Transformation**: The message is transformed into the internal FHIR model.
4. **Commit**: The transformed data is written to Postgres. If the write fails, the message is sent to the dead-letter queue.
## Key Concepts
- **Idempotency**: The Commit stage is designed to be idempotent. If a message is processed twice, it will not create duplicate records.
- **Dead-Letter Queue**: If a message fails validation or transformation, it is sent to the dead-letter queue for manual review.
- **Batch Processing**: Messages are processed in batches to improve throughput and reduce database load.
## Running the Local Test Suite
To run the local test suite, use the following command:
```bash
cargo test
## Troubleshooting
- **Batch Failures**: Check the dead-letter queue for failed messages.
- **Schema Mismatches**: Ensure that the incoming HL7/FHIR messages match the expected schema.
- **Database Issues**: Check the Postgres connection and ensure that the database is running.
## Next Steps
- Review the code in `src/sync/` to understand the implementation details.
- Run the local test suite to ensure that your environment is set up correctly.
- Reach out to Priya or Marcus if you have questions about the architecture or implementation.