The open source data lakehouse
with Lineage built in.
VektralForge integrates Apache Airflow, Spark, Delta Lake, Trino and Superset into one deployable stack, with end-to-end data lineage via OpenLineage and Marquez. Run it locally with Docker Compose; deploy it to production on K3s.
Your Data
Multicloud ready
Used for
OpenLineage traces every step, from your data to what your team ships.
Overview
What is VektralForge?
VektralForge is an open source data lakehouse platform. It wires together the tools most data teams already reach for — orchestration, distributed processing, transactional table storage, SQL querying and dashboards — into a stack you can run with one command, instead of assembling and gluing each piece by hand.
Each pipeline is automatically traced: It shows where the data came from, what transformed it, when, and where it ended up. Data lineage is a built-in feature, not an add-on that you configure later.
Run it locally with Docker Compose for development, or deploy it to a production Kubernetes cluster on K3s. The project is licensed under Apache 2.0, accepts contributions under a Developer Certificate of Origin rather than a CLA, and is governed independently of its sponsor.
Architecture
Platform architecture
Spark writes, Trino reads. ACID operations on Delta Lake —
MERGE, UPDATE, DELETE, VACUUM — run through Spark; Trino
gives you interactive SQL over the same tables through one shared Hive Metastore, so a table Spark writes is
queryable from Trino without registering it twice.
Data moves through medallion layers — raw (unmodified
source response), bronze (typed Delta tables), silver (cleaned, deduplicated models)
and gold (aggregates ready for dashboards).
Features
Everything the stack gives you
Not a checklist of integrations — a working stack, with two real pipelines already inside it.
Resource library
A set of community-developed pipelines ready to test the stack and explore its capabilities.
Lineage from source to dashboard
OpenLineage and Marquez capture lineage at every step automatically — no manual instrumentation of your DAGs or Spark jobs required.
ACID lakehouse tables
Delta Lake on MinIO gives you transactional guarantees on object storage — Spark writes, Trino reads, one metastore shared between both.
Local-to-production parity
The same stack that runs on your laptop with Docker Compose is the one designed to run in production on K3s — no separate dev-only architecture.
Secrets stay out of the repo
Nothing ships hardcoded. Local development uses a generated .env;
production secrets management with OpenBao is on the roadmap.
Apache 2.0, DCO, no CLA
Copyright stays distributed across every contributor. Relicensing the project would require unanimous consent — which is what makes the open source promise structural, not just a stated intention.
Sponsors
Those who make this community project possible
Founding Sponsor — Aleph Server
An IT infrastructure consultancy based in Chile that created VektralForge and funds its ongoing development. Sponsorship buys visibility and gratitude — it does not buy a veto, a casting vote, or a reserved seat on the Technical Steering Committee.
Interested in sponsoring VektralForge — get in touch at
Documentation
Learn how to use the stack
Start here, then go as deep as you need.
Easy examples to start
Download the library of ready-to-use examples to get started with
make dev-load-example.
Quick Start
Up and running in under 5 minutes
No complicated setup and nothing to request first — clone the repository or pull the image, run one command, and the full stack is live.
git clone https://github.com/vektralforge/vektralforge.git
cd vektralforge
make init-env # Generate keys and prompt for passwords
make setup # Create the .venv with Python 3.12
make dev-up # Bring the stack upRoadmap
Where the project is going
Dates depend on maintainer availability — the repository issues track what's actually in progress.
Phase 1 — Foundation
Local stack, two pipelines, lineage, CI, governance.
Phase 2 — Production
K3s deployment, managed secrets, Kafka streaming.
Phase 3 — Maturity
Data quality checks, metrics, unified access control.
Phase 4 — Exploration
Natural-language query over the catalog.
Community
Join the community
Contribute
Open an issue before anything beyond a typo or obvious fix. Issues labelled
good first issue are a good place to start; every commit needs a DCO sign-off.
Try it and give us your feedback
Spin up the stack in a few minutes and tell us what worked, what didn't, and what you'd build next — early feedback shapes the roadmap.
Jump to Quick Start →Code of Conduct
VektralForge follows the Contributor Covenant. Participation in every project space is expected to stay respectful and harassment-free.
Read the Code of Conduct →