me/dagster
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
184 commits 1 branch 0 tags 4.2 MiB
  • Python 94.4%
  • Nix 3.1%
  • TypeScript 1.4%
  • Jupyter Notebook 1%
  • Shell 0.1%
Find a file
Exact
Christian Lengert 2583b4bf4b refactor no spine
2026-08-11 19:22:54 +02:00
.claude/commands refactor bronze 2026-08-05 17:26:00 +02:00
.opencode/plugins refactor no spine 2026-08-08 16:25:30 +02:00
dagster_deployment refactor no spine 2026-08-11 19:22:54 +02:00
dagster_deployment_tests revert: roll back OSM bronze pipeline to shapely/GeoJSONSeq 2026-08-09 00:07:35 +02:00
docs dagster 2026-08-01 13:47:35 +02:00
plans refactor no spine 2026-08-09 21:51:51 +02:00
queries refactor no spine 2026-08-08 17:44:55 +02:00
scripts refactor no spine 2026-08-08 12:40:57 +02:00
.gitignore cleanup: remove dead cache assets, stale lake dirs, and legacy files 2026-08-08 22:51:31 +02:00
AGENTS.md refactor no spine 2026-08-08 17:08:57 +02:00
CLAUDE.md feature store: spine + flood silver with asset checks 2026-08-01 15:38:37 +02:00
DATASETS.md refactor no spine 2026-08-08 19:13:08 +02:00
flake.lock dagster 2026-07-25 14:59:27 +02:00
flake.nix fix: make osmium available to Dagster services and enable run monitoring 2026-08-08 22:21:40 +02:00
pyproject.toml restructure: flatten repo and group defs 2026-08-08 22:11:53 +02:00
README.md docs: add DATASETS.md, SOURCES.md; clean README; remove example scaffolding 2026-08-01 14:07:13 +02:00
SOURCES.md refactor no spine 2026-08-07 17:28:49 +02:00
uv.lock restructure: flatten repo and group defs 2026-08-08 22:11:53 +02:00

README.md

dagster-etl

Dagster pipeline for ingesting open EU datasets into a local data lake (/mnt/pool/data/lake).

Conventions

Country codes: always ISO 3166-1 alpha-2 (DE, FR, GR). TED notices use alpha-3 — convert at ingest. NUTS codes share the same alpha-2 prefix (LEFT(nuts_code, 2)).

Hive partitioning: use key=value directory names for all partitioned assets (year=2024/, country=DE/). Always pass hive_partitioning=true when reading with DuckDB.

Development

nix develop          # activate shell (sets LD_LIBRARY_PATH, syncs uv venv)
dg dev               # start Dagster UI at http://localhost:3000
dagster-etl-run      # run main_job as a one-shot module

Further reading