Document AI at scale for a French energy group
A one-year AI and cloud architecture mission: turning multi-page technical PDFs into structured, auditable data with a serverless AWS pipeline.
Architecture at a glance
End-to-end pipeline
S3 ingestion and Step Functions orchestrate Lambda workers, Textract, seven parallel Pydantic AI agents, field validation and a consolidated report, with DynamoDB and RDS state and Logfire tracing.
The problem
Hazardous-material survey reports arrive as multi-page PDFs with heterogeneous layouts: tables, forms, drawings and long text sections. Reading them manually is slow and interpretation varies from one reader to another.
The goal was not a generic chatbot. It was a reproducible pipeline that extracts the same fields from every report, keeps the source of each value and produces a consolidated final document.
The architecture
The pipeline runs on a serverless AWS architecture. Step Functions orchestrate the stages, Lambda functions process pages, S3 and SQS carry documents and work items, and PostgreSQL and DynamoDB hold state and results.
Amazon Textract handles layout, tables and forms. On top of it, specialised Pydantic AI agents process each page in parallel with one responsibility per agent, and a final step compiles the consolidated report.
Model calls run with sequential fallback so a degraded provider does not stop a batch, and normative retrieval grounds extraction in the relevant standards.
Traceability and quality
Every run is traced with Logfire and OpenTelemetry: which page, which model, which prompt version and which result. Evaluation sets are used to compare architectures and catch regressions before they reach production.
The visible outcome is a system that can be operated and audited, not a demonstration. Monitoring, failure handling and cost measurement are part of the delivery.
What stays private
The client name, document volumes, processing costs and rejection rates are not published here. They belong to the client and are shared only with explicit authorisation.
Public outcomes
- A production pipeline from PDF ingestion to a consolidated report.
- Page-level provenance for every extracted value.
- Seven specialised agents working in parallel per page.
- Sequential model fallback across providers.
- Full run tracing and cost measurement.
Stack
- AWS Step Functions
- AWS Lambda
- Amazon S3
- Amazon SQS
- Amazon Textract
- Amazon RDS PostgreSQL
- Amazon DynamoDB
- Pydantic AI
- Logfire
- OpenTelemetry