Data engineering / Case study
Scalable Data Fusion Engine
As a backend developer, I built data-fusion logic, FastAPI services, ETL workflows and shared security utilities for a system analysing roadway data.
Harshit Makwana
- My role
- Backend developer — contributed to a wider system
- Timeline
- 2023–2024
- Focus
- Data engineering
01 / Context
Integrating roadway data across processing, access and delivery.
The system needed to bring information from multiple sources into a usable backend for roadway insights. My contribution focused on the integration and service layer: processing data, making it accessible through APIs, supporting search and exports, and implementing shared backend utilities.
02 / Ownership
What I built
- Developed data-fusion logic and ETL workflows, with RESTful FastAPI endpoints for data access and manipulation.
- Managed storage and querying with PostgreSQL and Trino, real-time streams and caching with Redis, and search/indexing with Solr.
- Built a dynamic Python export feature for PDF and Excel reports.
- Developed a shared Python library covering virus scanning, SQL/XSS injection defenses, authentication, authorization, encryption/decryption and compression/decompression middleware.
- Integrated Keycloak for access control and used Docker for application containerization.
- Created a Python SOAP data-exchange framework without external libraries, including asynchronous feed APIs and maintainable handling of multiple SOAP services.
03 / Engineering
Technical approach
Connect ingestion to usable services
I implemented data-fusion logic and ETL processing, then exposed data through FastAPI REST services. PostgreSQL and Trino supported storage and querying, Redis handled real-time streams and caching, and Solr provided search and indexing capabilities.
Build reusable backend capabilities
I implemented a common Python library so security and data-handling functionality could be reused across the backend. My work included access control with Keycloak and middleware for encryption, decryption, compression and decompression, alongside scanning and injection-defense functionality.
Support multiple delivery formats
I added dynamic PDF and Excel exports for stakeholders and built a SOAP framework to support data exchange. I structured the SOAP work around multiple services and asynchronous data feeds to improve maintainability.
04 / System view
Workflow at a glance
Source data
Multiple roadway-information sources
Integration
Fusion logic and ETL processing
Backend capabilities
Querying, caching, search and access control
Delivery
REST / SOAP APIs and PDF / Excel reports
05 / Delivery
Outcomes
My backend contributions connected data processing with API access, search, export and shared security capabilities. This was a contribution to the wider Scalable Data Fusion Engine project, with my ownership concentrated on the backend features described here.
Technologies & methods
- Python
- FastAPI
- PostgreSQL
- Trino
- Redis
- Solr
- Keycloak
- Docker
- REST
- SOAP
