OpenELT

Open Extract-Load-Transform — a ZFS-backed, medallion-architecture (bronze/silver/gold) storage foundation for heterogeneous bulk data at homelab-to-enterprise scale.

Built for data you don't want to lose but haven't fully decided how you'll use yet: LLM model weights and quants, tokenizers, training corpora, web archives, scientific datasets, whitepapers, git repos, firmware images, media, configs, and anything else that doesn't fit a single-purpose pipeline. Classifies incoming files into content-domain buckets rather than file-format buckets, tracks per-object metadata for downstream keyword/vector indexing, and links related assets (a video, its audio track, its transcript) across independent ingest times without requiring them to arrive together.

Why

Most self-hosted storage setups either dump everything into one flat directory (unsearchable past a few thousand files) or organize by file extension (which breaks the moment one dataset exists as .csv, .sqlite, and .json copies of the same thing). OpenELT organizes by content domain instead, keeps a write-once cold-storage tier (bronze/) separate from rebuildable derived tiers (silver/, gold/), and treats the classification config as data, not code — so adding a new domain never means touching a script.

Architecture

/data/
├── staging/   # dump zone, unvalidated, no snapshots, compression off
├── bronze/    # write-once, checksummed, organized by domain
├── silver/    # extracted text + metadata JSON sidecars (rebuildable)
└── gold/      # elasticsearch (keyword) + vector-db (semantic), rebuildable

bronze/ is the only tier that must never be lost. silver/ and gold/ are derived and can be wiped/rebuilt at any time from bronze/ plus its metadata sidecars — this is what decouples storage from compute: gold/ can live on faster hardware entirely separate from bronze/'s bulk storage.

Full docs:

• docs/ARCHITECTURE.md — layer design, rebuild/backup strategy, schema versioning
• docs/DOMAIN-GRANULARITY.md — domain-vs-tag rule, worked example
• docs/LINKING.md — cross-time asset linking (link_key/work_id)

Features

• Domain-based classification — driven entirely by config/domains.conf; add a domain by adding a line, no code changes
• Content-addressed storage — files land in bronze/ named by sha256, automatic dedup across sources
• Cross-time linking — a transcript ingested today and a video ingested next month, from the same source, resolve to the same work_id automatically
• Rebuildable index tier — gold/ (elasticsearch + vector DB) is a cache over silver/ metadata, never a source of truth
• Zero third-party Python deps — bin/*.py use stdlib only, runs on any Python 3.10+
• POSIX-first shell — bin/*.sh are pure POSIX sh, no bashisms

Requirements

• OpenZFS — required for bin/zfs-init-domains.sh, Linux only:
  • Alpine: apk add zfs zfs-lts (kernel module + userland; zfs-lts matches Alpine's LTS kernel, pick the module package matching your running kernel)
  • Debian: apt install zfsutils-linux (pulls in zfs-dkms via contrib/non-free-firmware, enable those repo sections first if not already)
• Python 3.10+, stdlib only, no third-party packages, for bin/ingest.py
• Shell: POSIX sh. Native on Alpine (ash, /bin/sh) and Debian (bash invoked in POSIX mode, or dash as /bin/sh)

Getting the repo

Alpine may need git installed first — check before installing:

which git || apk add git

Clone via SSH:

git clone git@github.com:ms4x-dev/OpenELT.git
cd OpenELT

Or

Clone via HTTPS:

git clone https://github.com/ms4x-dev/OpenELT.git
cd OpenELT

Make the scripts executable — before running anything in bin/, regardless of OS:

chmod +x bin/*.sh bin/*.py

Quickstart

Alpine (ash)

Run as root, or prefix each command below with sudo.

# 0. install OpenZFS if not already present
which zfs || apk add zfs zfs-lts
rc-service zfs-load zstart
rc-update add zfs-load boot

# 1. create the ZFS pool (if not already present)
zpool create -o ashift=12 data mirror /dev/disk/by-id/DEVICE1 /dev/disk/by-id/DEVICE2

# 2. provision the bronze/silver/gold dataset tree from config/domains.conf
./bin/zfs-init-domains.sh data

# 3. drop files into /data/staging/, then run the ingest pipeline
python3 bin/ingest.py --data-root /data

Debian (bash)

# 0. install OpenZFS if not already present (contrib/non-free-firmware
#    sections must be enabled in /etc/apt/sources.list first)
which zfs || (sudo apt update && sudo apt install zfsutils-linux)

# 1. create the ZFS pool (if not already present)
sudo zpool create -o ashift=12 data mirror /dev/disk/by-id/DEVICE1 /dev/disk/by-id/DEVICE2

# 2. provision the bronze/silver/gold dataset tree from config/domains.conf
sudo bin/zfs-init-domains.sh data

# 3. drop files into /data/staging/, then run the ingest pipeline
python3 bin/ingest.py --data-root /data

Adding a domain

Append one line to config/domains.conf:

name|recordsize|alias1,alias2,...|parent-or-dash

Re-run bin/zfs-init-domains.sh <pool> — idempotent, only creates what's missing. No code changes required. See docs/DOMAIN-GRANULARITY.md for whether something warrants a new domain or is just a tag.

Repo structure

bin/
├── detect-os.sh          # POSIX sh, sourced by zfs-init-domains.sh
├── zfs-init-domains.sh   # POSIX sh, run as root, Linux only
├── classify.py           # domain classification, imported by ingest.py
├── ingest.py             # main pipeline entrypoint
└── link.py                # cross-time asset linking, imported by ingest.py
config/
└── domains.conf          # single source of truth for domains
schema/
└── metadata.schema.json  # JSON Schema for silver metadata sidecars
docs/
├── ARCHITECTURE.md
├── DOMAIN-GRANULARITY.md
└── LINKING.md
requirements.txt          # empty, stdlib-only by design

Roadmap / stubs to wire in

bin/ingest.py ships with two intentional stubs, left unimplemented so the pipeline runs end-to-end without forcing a dependency choice:

• extract_text() — plug in per-domain extraction (pdfplumber for PDF, BeautifulSoup for HTML, GGUF header parsing for models, etc.)
• embed() — plug in a vector DB client (Qdrant, Weaviate, Milvus, or other) for the gold/vector-db tier

Both default to None/"pending" status until implemented — nothing downstream breaks in the meantime.

Contributing

Issues and PRs welcome. Keep bin/*.sh POSIX-compliant (no bashisms) and bin/*.py stdlib-only unless a stub above is being implemented, in which case add the dependency to requirements.txt with a comment explaining what it's for.

License

Licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). Full text in LICENSE.

In short: you may use, modify, and redistribute this software freely, including for commercial purposes, provided that any modified version you run as a network service is also made available under AGPL-3.0 to users of that service, and any redistribution — modified or not — remains under the same license with source made available.