Skip to main content

Module 4: The Data Platform

You don't need to be a data engineer to work at Wasteology, but you will run into this stack — in conversation, in lat.md, or in a repo you're asked to touch. This module is a conceptual orientation: what each piece is for and how they fit together. It is deliberately not a how-to-deploy guide; that lives in each project's own documentation and in specialist tooling.

The three pieces, and what each one is for​

ToolRolePlain-language summary
PrefectOrchestrationSchedules and runs data workflows ("flows"), tracks whether each run succeeded, and retries or alerts on failure
dbtTransformationTurns raw loaded data into modeled, tested tables in the warehouse using SQL — the "T" in ETL/ELT
SlingMovementMoves data between systems (e.g. SQL Server to PostgreSQL, or querying a schema directly) without hand-written extract/load code

Each does one job. Prefect doesn't know SQL modeling; dbt doesn't move data between databases; Sling doesn't schedule or retry anything on its own. The platform composes them rather than building one tool that tries to do everything.

How the pieces run​

Prefect flows here don't run on a server that sits idle waiting for work. Wasteology runs Prefect Cloud with Azure Container Instances (ACI) as the execution backend, using what's called a push-pool model: when a flow needs to run, Prefect Cloud provisions a fresh, ephemeral container directly via the Azure API, the flow runs to completion inside it, and the container goes away. There's no persistent worker process to keep alive or patch. This is why you'll hear people refer to "ACI push-pool ETL" as shorthand for "how our scheduled data jobs actually execute."

dbt, by contrast, runs via the dbt Cloud CLI rather than a self-hosted server — the transform logic (SQL models, tests) lives in a git-tracked project, and dbt Cloud executes it.

Sling connections are defined once, centrally, and then reused by name (rather than each project reinventing its own connection config) — which is part of why you'll see the same connection names referenced across multiple unrelated pipelines.

Wastey says

You'll sometimes see a data view inside the orchestration CLI's dashboard that surfaces dbt freshness, Sling tables and row counts, and database schemas side by side, read-only. That's a monitoring convenience — the actual dbt run, dbt test, and ad hoc sling query work stays a CLI operation, not something you click a button for.

Why this composition, not one tool​

It would be simpler, on paper, to have one tool do orchestration, transformation, and movement. Wasteology doesn't, because each of these tools is genuinely best-in-class at its one job, and composing focused tools means a change to how you move data doesn't require touching how you model it, or vice versa. If you're troubleshooting a data problem, the composition also tells you where to look first: is data not arriving at all (a Sling / source problem), not being scheduled or retried (a Prefect problem), or arriving but wrong/stale after transform (a dbt problem)?

Hands-on lab​

This lab is about recognizing the pieces, not operating them — you likely won't have write access to production data flows yet, and that's fine.

  1. Find a project's lat.md/ that documents its data pipeline (ask your onboarding buddy which project is a good example, or look for one with a Prefect flow).
  2. Skim the section describing its flow(s). Identify: what triggers the flow (a schedule, or something else), what it moves or transforms, and where the output lands.
  3. If the project has a dbt model directory, open one model file and read its SQL — you're not expected to understand every line, just to see that it's SQL, not custom code, doing the transformation.
  4. Write one sentence describing which of Prefect, dbt, or Sling you'd suspect first if you were told "yesterday's numbers on a dashboard look stale" — and why.

Explore the data stack​

Click each piece to see what it's responsible for.

Sling moves, dbt transforms, Prefect orchestrates — data lands in the warehouse for reporting and apps.
Sling moves, dbt transforms, Prefect orchestrates — data lands in the warehouse for reporting and apps.
Wasteology's data stackInteractive

Click any step to see what happens and who's responsible.

▸Yesterday's dashboard numbers look stale — where do you look first?

Knowledge check​

Which tool is responsible for scheduling and monitoring whether a data workflow ran successfully?
What does 'ACI push-pool' describe?
If a dashboard's numbers look correct on the source system but wrong after landing in the warehouse's modeled tables, which layer would you suspect first?

Next up: Module 5 — Cloud and Access.