Building a data platform with dbt, Dagster, and ArgoCD

Starting point

At vilisto we operate more than 100,000 smart thermostats at about 400 customer sites. The operational systems (a sharded backend, a battery database, ML evaluation pipelines) each hold part of the data, and none of them is built for analytics. A question such as "how many devices went offline per firmware version last week" required a one-off script against production APIs.

The data platform extracts from these sources, loads into a PostgreSQL warehouse, transforms the data with dbt and feeds Grafana dashboards. It is deployed with ArgoCD.

Architecture

The platform follows the ELT pattern with a medallion layout (bronze, silver, gold):

Operational systems (sharded backend API, MinIO, battery database)
  -> [Extract + Load] Dagster assets -> PostgreSQL raw schema (bronze)
  -> [Transform] dbt staging views (silver) -> mart tables (gold)
  -> [Visualize] Grafana dashboards

It has five components:

1. Sqitch manages the raw schema. Migrations run in a Kubernetes init container before the main application starts.
2. Dagster orchestrates extraction and loading. Each source is a Dagster asset with its own schedule.
3. dbt turns raw JSONB into typed, indexed tables. Staging models flatten and deduplicate; mart models join and add indices.
4. Helm and ArgoCD handle deployment. Schema migrations, secrets and the Dagster webserver are defined in Helm templates.
5. Grafana dashboards query the mart tables directly.

Extraction from the sharded backend

The backend is split across several shards. The API client authenticates against each shard and yields data per customer. Every extraction asset calls the API, removes sensitive fields and upserts the result into the raw schema.

PII handling uses an allowlist. Only the listed configuration parameters are loaded into the warehouse, and a new field in the API response is not loaded until someone adds it to the list.

For daily aggregates (valve movement, temperature, humidity, battery voltage), the client first requests data per customer and falls back to per-device requests if that fails. For customers with thousands of devices the customer-level request is much faster, but it is not always available.

Dagster jobs and resources

Three scheduled jobs load data at different frequencies:

Each asset records metadata (number of customers, thermostats and errors, execution time) that is shown in the Dagster UI. The daily aggregates job uses Dagster's multiprocess executor to extract customers in parallel.

Development and production run the same asset code with different resources. In development the backend API client and the MinIO client are stubs and the database runs locally. In production the resources are the sharded API client, the production database and MinIO. CI runs the full pipeline with stubbed data against a local TimescaleDB, including the Sqitch migrations and all dbt transformations, and fails if any asset does not materialize.

dbt staging and mart models

The raw schema stores API responses as JSONB. The central staging model turns the raw device snapshots into one row per device and day, with typed columns for firmware version, battery data, valve position, connectivity ratios and configuration parameters. It is materialized incrementally with device and date as the unique key, so a run does not reprocess the full history.

The same model uses window functions (lag over each device's snapshots, ordered by query time) to derive values from consecutive snapshots, for example whether the device restarted (uptime decreased) and how many calibrations happened in between.

Mart models join the staging tables with seed data (firmware rollouts, start dates of data collection) and add indices. The largest mart table has indices on date, customer, device, firmware version and hardware version so that Grafana queries stay fast as it grows.

All mart tables are PostgreSQL unlogged tables. They contain derived data that every dbt run rebuilds, so they do not need WAL durability, and unlogged tables are faster to write.

Deployment with Sqitch and ArgoCD

Schema migrations run in a Kubernetes init container that executes sqitch deploy before the Dagster pod starts. If a migration fails, the pod does not start, ArgoCD marks the application as degraded, and no job runs against a partially migrated schema.

Secrets (database credentials, API passwords, MinIO keys) are stored in Git as SealedSecrets. The Helm chart contains the Deployment, Service, PVC, ConfigMap, SealedSecrets and ServiceAccount.

ArgoCD syncs staging automatically from HEAD. For production, I update the image tag in the production values file, merge to main and approve the sync manually in the ArgoCD UI.

Source freshness checks

dbt source freshness checks run in the pipeline, with a warn and an error threshold per source:

If an extraction job stops delivering data without raising an error, the stale source appears as a freshness warning in Grafana.

Known limitations

The platform started as a proof of concept, and three decisions reflect that.

PostgreSQL works as the warehouse at the current size, but the mart tables are growing. A columnar store such as DuckDB or ClickHouse would fit better in the long term.

The extractors are hand-written for the internal systems. Adding sources such as the CRM, Prometheus metrics or GitLab would be easier with Airbyte than with more hand-written extractors.

The largest mart table is not partitioned. Monthly partitions would simplify backfills and retention policies; at present the indices keep queries fast.

Stack