trading-daemon/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Trading Data Daemon - A modular Python daemon system for downloading and storing trading data from German stock exchanges (EIX, Lang & Schwarz, Deutsche Börse, Gettex, Stuttgart, Börsenag regional exchanges) into QuestDB.

Language: Python 3.12+ | Database: QuestDB (time-series via Influx Line Protocol)

Commands

# Docker Compose (primary deployment method)
docker-compose up -d                    # Start all 5 services
docker-compose logs -f fetcher          # Watch fetcher logs
docker-compose down                     # Stop all services

# Manual Python execution
python3 daemon.py                       # Run main fetcher
python -m src.analytics.worker          # Run analytics worker
python src/metadata/fetcher.py          # Run metadata fetcher
python dashboard/server.py              # Run dashboard API (port 8000)

# Dependencies
pip install -r requirements.txt

Architecture

Five microservices orchestrated via Docker Compose:

┌─────────────────────────────────────────────────────────────────┐
│                         QuestDB                                  │
│              (9000=HTTP, 8812=PostgreSQL, 9009=ILP)             │
└──────────┬──────────────┬──────────────┬──────────────┬─────────┘
           │              │              │              │
    ┌──────┴──────┐ ┌─────┴─────┐ ┌──────┴──────┐ ┌─────┴─────┐
    │   fetcher   │ │ analytics │ │  metadata   │ │ dashboard │
    │ daemon.py   │ │  worker   │ │  fetcher    │ │  :8080    │
    └─────────────┘ └───────────┘ └─────────────┘ └───────────┘

Fetcher (daemon.py): Main orchestrator. Fetches trades from all exchanges daily at 23:00. Uses streaming for EIX (large files), batch processing for others. Handles deduplication via MD5 hash caching.

Analytics Worker (src/analytics/worker.py): Calculates aggregated tables (analytics_exchange_daily) for time periods: 7, 30, 42, 69, 180, 365 days.

Metadata Fetcher (src/metadata/fetcher.py): Enriches ISINs with company/sector data via OpenFIGI API and yfinance.

Dashboard (dashboard/server.py): FastAPI server serving REST endpoints and static UI from dashboard/public/.

Exchange Extensibility Pattern

All exchanges extend src/exchanges/base.py:BaseExchange. To add a new exchange:

1. Create subclass in src/exchanges/
2. Implement fetch_latest_trades() returning list of Trade objects
3. Implement name property
4. Register in daemon.py in STREAMING_EXCHANGES (large data) or STANDARD_EXCHANGES (batch)

Currently registered:

• Streaming: EIX (European Investor Exchange)
• Standard: LS, Xetra, Frankfurt, Quotrix, Gettex, Stuttgart, Börsenag (DUSA/DUSB/DUSC/DUSD/HAMA/HAMB/HANA/HANB)

Database Schema

• trades: exchange, symbol, isin, price, quantity, timestamp
• analytics_exchange_daily: timestamp, exchange, trade_count, volume
• metadata: isin, name, country, continent, sector

Environment Variables

DB_USER=admin              # QuestDB user (default: admin)
DB_PASSWORD=quest          # QuestDB password (default: quest)
DB_HOST=questdb            # QuestDB hostname (default: questdb for Docker)

Code Conventions

• German-language comments and commit messages
• Memory-efficient patterns: streaming/generators for large files, explicit del after processing
• Rate limiting: User-Agent rotation, 0.3-0.5s delays between requests, retry with backoff for 429s