The problem
Data operations were mostly manual, with no central orchestration layer. That created several important risks:
- Reliance on people to run business-critical jobs.
- Higher chance of operational failures and environment drift.
- Difficulty scaling data workloads with demand.
- Little standardization around deploy, security, and governance.
- Expensive maintenance and component upgrades.
In short, the goal was to move from a reactive, manual setup to a reliable, traceable platform that could grow.
The solution
I contributed to implementing a data architecture on AWS, fully provisioned with Terraform, running Apache Airflow on managed Kubernetes (Amazon EKS), with a focus on automation, security, and scalability.
Implemented architecture (technical view)
- Infrastructure as code with Terraform, organized in reusable stacks/modules.
- Airflow on Kubernetes (EKS), deployed with Helm, with clear separation between:
- application execution;
- persistent layer (state, database, secrets, artifacts).
- Dynamic cluster scaling with node- and workload-level autoscaling.
- Versioned execution images stored in Amazon ECR.
- Centralized secrets in AWS Secrets Manager, synced to Kubernetes with External Secrets.
- Airflow access and authentication integrated with Amazon Cognito (SSO/OAuth and role mapping).
- Operational persistence for metadata, logs, and run state using managed services (relational DB + object storage).
- Security and governance practices, including permission segregation by service/account and IAM roles for EKS workloads.
Outcomes
The delivery improved technical and operational maturity in concrete ways:
✅ Reliability: fewer manual failures and more predictable runs.
✅ Availability: distributed EKS-based architecture with resilient components.
✅ Scalability: capacity to handle load growth without a full redesign.
✅ Security: proper secret handling, centralized auth, and less credential exposure.
✅ Standardization: reproducible, auditable environments with Terraform.
✅ Operational efficiency: maintenance, upgrades, and evolution follow a structured, versioned process.