diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..3a7e3d3 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,84 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +Trading Data Daemon - A modular Python daemon system for downloading and storing trading data from German stock exchanges (EIX, Lang & Schwarz, Deutsche Börse, Gettex, Stuttgart, Börsenag regional exchanges) into QuestDB. + +**Language:** Python 3.12+ | **Database:** QuestDB (time-series via Influx Line Protocol) + +## Commands + +```bash +# Docker Compose (primary deployment method) +docker-compose up -d # Start all 5 services +docker-compose logs -f fetcher # Watch fetcher logs +docker-compose down # Stop all services + +# Manual Python execution +python3 daemon.py # Run main fetcher +python -m src.analytics.worker # Run analytics worker +python src/metadata/fetcher.py # Run metadata fetcher +python dashboard/server.py # Run dashboard API (port 8000) + +# Dependencies +pip install -r requirements.txt +``` + +## Architecture + +Five microservices orchestrated via Docker Compose: + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ QuestDB │ +│ (9000=HTTP, 8812=PostgreSQL, 9009=ILP) │ +└──────────┬──────────────┬──────────────┬──────────────┬─────────┘ + │ │ │ │ + ┌──────┴──────┐ ┌─────┴─────┐ ┌──────┴──────┐ ┌─────┴─────┐ + │ fetcher │ │ analytics │ │ metadata │ │ dashboard │ + │ daemon.py │ │ worker │ │ fetcher │ │ :8080 │ + └─────────────┘ └───────────┘ └─────────────┘ └───────────┘ +``` + +**Fetcher (`daemon.py`):** Main orchestrator. Fetches trades from all exchanges daily at 23:00. Uses streaming for EIX (large files), batch processing for others. Handles deduplication via MD5 hash caching. + +**Analytics Worker (`src/analytics/worker.py`):** Calculates aggregated tables (`analytics_exchange_daily`) for time periods: 7, 30, 42, 69, 180, 365 days. + +**Metadata Fetcher (`src/metadata/fetcher.py`):** Enriches ISINs with company/sector data via OpenFIGI API and yfinance. + +**Dashboard (`dashboard/server.py`):** FastAPI server serving REST endpoints and static UI from `dashboard/public/`. + +## Exchange Extensibility Pattern + +All exchanges extend `src/exchanges/base.py:BaseExchange`. To add a new exchange: + +1. Create subclass in `src/exchanges/` +2. Implement `fetch_latest_trades()` returning list of `Trade` objects +3. Implement `name` property +4. Register in `daemon.py` in `STREAMING_EXCHANGES` (large data) or `STANDARD_EXCHANGES` (batch) + +Currently registered: +- **Streaming:** EIX (European Investor Exchange) +- **Standard:** LS, Xetra, Frankfurt, Quotrix, Gettex, Stuttgart, Börsenag (DUSA/DUSB/DUSC/DUSD/HAMA/HAMB/HANA/HANB) + +## Database Schema + +- `trades`: exchange, symbol, isin, price, quantity, timestamp +- `analytics_exchange_daily`: timestamp, exchange, trade_count, volume +- `metadata`: isin, name, country, continent, sector + +## Environment Variables + +```bash +DB_USER=admin # QuestDB user (default: admin) +DB_PASSWORD=quest # QuestDB password (default: quest) +DB_HOST=questdb # QuestDB hostname (default: questdb for Docker) +``` + +## Code Conventions + +- German-language comments and commit messages +- Memory-efficient patterns: streaming/generators for large files, explicit `del` after processing +- Rate limiting: User-Agent rotation, 0.3-0.5s delays between requests, retry with backoff for 429s