Building and operating a web-data platform.
I led development of the data engine and worked across its backend, workers, infrastructure, and deployment workflows.
The client name and quantitative results are not public.
The platform
The platform collected and processed data from multiple web sources. Python services, background workers, databases, and AWS services supported its operation.
My work covered the data engine, production components, and the workflows used to release and manage them.
My responsibilities
I worked across development, production operation, and deployment automation.
Data engine
I led development of the engine that collected and processed web data with Scrapy, Django, and Celery.
Production operation
I owned components across services, workers, queues, databases, proxy infrastructure, and AWS runtime services.
Deployment automation
I rebuilt deployment workflows to automate releases, database migrations, and worker and service management.
System components
The collection engine ran alongside application services, queue-based processing, persistent storage, and AWS infrastructure.
- Web-data engine
- Scrapy · proxy infrastructure
- Application services
- Django
- Workers and queues
- Celery · Redis · SQS
- Persistent data
- PostgreSQL
- Cloud services
- ECS · Lambda · CloudWatch
- Delivery tooling
- GitLab CI/CD · Docker · Terraform
Working on a data platform?
Tell me which sources you need, where the data should arrive, and how the system runs today.