What a data engineer actually does, on Databricks (and a little Fabric)
Magnus Vinjebo · Backstage
AI data consultancy in Copenhagen, built on Databricks. Three kinds of work:
Lakehouses that ingest, clean and model company data
Models and agents that run on top of that data
Dashboards and answers for the business
ERP, CRM, APIs, files, events
systems of recordCopy data in, reliably and incrementally
Lakeflow ConnectAuto LoaderOpen tables in the lakehouse
Delta LakeClean, join, model: bronze → silver → gold
Lakeflow PipelinesBI, ML features, AI agents
SQL · MLflow · GenieThis strip comes back in the top-right corner on the next slides, so you can see where we are.
Auto Loader: pick up only new files, remember progress
# files land in a Unity Catalog volume (spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .option("cloudFiles.schemaLocation", chk) .load("/Volumes/raw/sap/orders/") .writeStream .option("checkpointLocation", chk) .trigger(availableNow=True) .toTable("main.bronze.sap_orders"))
For SaaS apps and databases, Lakeflow Connect gives managed connectors (e.g. Salesforce, SQL Server, SharePoint) with CDC built in.
SELECT * FROM main.silver.orders VERSION AS OF 12; DESCRIBE HISTORY main.silver.orders;
ML link Log the table version with your model run, and you can reproduce the exact training data months later.
main.bronze.sap_orders → main.silver.orders → main.gold.fct_sales
In a notebook
# orders_cleaning_FINAL_v3.ipynb df = spark.read.csv("orders.csv") df = df.dropna() df = df.filter("amount > 0") df.write.parquet("clean_orders") # how many rows did we drop? why?
Works once. Silent about what it threw away.
In a Lakeflow declarative pipeline
from pyspark import pipelines as dp from pyspark.sql import functions as F @dp.materialized_view(comment="One row per order line") @dp.expect_or_drop("valid_amount", "amount > 0") @dp.expect_or_fail("has_id", "order_id IS NOT NULL") def silver_orders(): return (spark.read.table("bronze.sap_orders") .dropDuplicates(["order_id", "line_no"]) .withColumn("amount", F.col("amount").cast("decimal(18,2)")))
expect keep the row, record the violation
expect_or_drop drop it, and count it
expect_or_fail stop the update
Facts = events you count (sales, clicks). Dimensions = the context you slice by (who, what, when, where).
SCD type 2 (slowly changing dimension): keep every version of a row.
| customer | segment | valid_from | valid_to |
|---|---|---|---|
| C-104 | SMB | 2024-01-01 | 2025-06-30 |
| C-104 | Enterprise | 2025-07-01 | 9999-12-31 |
ML link: point-in-time joins. Predicting 2024 churn with today's segment = leaking the future. Join the version that was valid at the label date.
| One-off notebook | Production pipeline | |
|---|---|---|
| Inputs | Hardcoded paths, "final_v3.csv" | Parameterised: environment, dates, catalog |
| Reruns | Run cells top to bottom and hope | Idempotent: rerun any day, same result |
| Correctness | Eyeball df.head() | Unit tests + data expectations |
| Running | You press Shift+Enter | Scheduled or triggered, with retries |
| Failure | Someone notices weeks later | Alert within minutes, with lineage |
| Change | Edit in place | Git → PR → review → CI → deploy |
Notebooks are still great for exploring. The work is turning the result into something that runs without you.
Cron schedule, file arrival, table update, or continuous
Retries, timeouts, and repair run: rerun only the failed tasks
Run history, durations, alerts to email or Teams/Slack, cost per job
prod.sales_gold.fct_sales
catalog . schema . table / view / volume / model / function
-- access control in plain SQL GRANT SELECT ON SCHEMA prod.sales_gold TO `analysts`; -- tag sensitive columns ALTER TABLE prod.silver.customers ALTER COLUMN email SET TAGS ('pii' = 'email');
"If I change this column, what breaks?" is now a click, down to column level.
Formerly Databricks Asset Bundles ("DABs"). Jobs, pipelines, permissions and clusters as YAML in the repo, deployed with one command.
databricks bundle deploy -t prod
CI (GitHub Actions / Azure DevOps) runs pytest on transformation logic, then deploys. Nobody clicks around in prod.
# databricks.yml bundle: name: sales_platform include: - resources/*.yml # jobs + pipelines targets: dev: mode: development # prefixed, paused default: true prod: mode: production workspace: host: https://adb-prod.azuredatabricks.net
Same tables, same permissions, same lineage, whatever the consumer. An agent is only as good as the gold layer it reads.
SaaS analytics inside the Microsoft / Power BI world. Every workload reads and writes OneLake, one Delta lake per tenant.
Both ways Fabric can mirror a Databricks Unity Catalog, so Power BI reads gold tables without copying them.
| Databricks | Fabric | |
|---|---|---|
| Model | Platform in your cloud (Azure, AWS, GCP) | SaaS in your Microsoft tenant |
| Storage | Delta in your storage + Unity Catalog | Delta in OneLake |
| Billing | Pay per compute used (DBUs) | Fixed capacity (F‑SKUs) |
| Strength | Heavy engineering, ML/AI, scale | Power BI-centric teams, low-code |
gold.forecastWhere the time actually went: agreeing on what "a sale" means (returns? internal orders? which date?) and getting history right. Not the model.
Someone owns it, versions it, documents it and is paged when it breaks.
Idempotent, parameterised, tested, scheduled. Explore in notebooks, ship as pipelines.
Expectations, Unity Catalog, time travel: measured, not assumed.
Point-in-time features, versioned training data, models as job tasks.
Magnus Vinjebo · Backstage