Back to the work
Market data · Data Engineer

Stock Data Platform.

Unified market data warehouse, 10 tickers, 25 years of history, 4 data sources. Kafka streaming, Airflow orchestration, TimescaleDB star schema, Dockerized.

Role
Data Engineer
When
Personal
Stack
Python, Kafka, Airflow, TimescaleDB
Scale
25 yrs history per ticker
View live demo ↗GitHub ↗
Stock Data Platform previewLive dashboard
25 yrshistory per ticker
10major US equities
~18Airflow DAGs
7Docker services

The problem

Build a unified market data warehouse combining real-time streaming with batch historical data from multiple sources.

What it does

  1. Kafka producers poll 4 data sources (yfinance, SEC EDGAR, FRED, live streams) every 15 seconds.
  2. ~18 Airflow DAGs orchestrate batch ETL, backfills, and aggregations across 10 major tickers.
  3. 8-table star schema in TimescaleDB with dimensional modeling for prices, fundamentals, earnings, SEC filings, and macro data.

Impact

  • 25 years of historical depth for 10 major US equities (AAPL, NVDA, MSFT, GOOG, AMZN, META, TSLA, JPM, NFLX, DIS).
  • Fully containerized with 7 Docker services, single command deployment.
  • Interactive Dash dashboards with SMA/EMA indicators and time-window filtering.