Evaluating battery prediction algorithms on field data before fleet rollout

Background

At vilisto we predict battery lifetime for 100k+ smart thermostats. The algorithms estimate remaining capacity (mAh) and runtime (days) per device, and building managers use these predictions to plan maintenance.

The algorithms live in an internal Python library, and new versions with changed prediction logic are released regularly. Before a new version runs on the fleet, it has to be shown to predict better than the current one. An A/B test is impractical because a battery takes months to run down.

I built two tools for this: an evaluation pipeline that compares algorithm versions on curated field data, and an operational service that runs the selected version on the whole fleet.

Dataset creation from field data

Lab data from test bench measurements under controlled conditions exists, but field data is less regular. Thermostats sit in buildings with different heating patterns, firmware versions change during deployment, and devices go offline for days.

The evaluation pipeline is a Dagster application with monthly partitioned assets. For each month it:

1. fetches valve movement and sensor logs from the analytics database,
2. pulls capacity test data from the device API,
3. records the version history of each device (hardware and firmware at the start of the month, software changes during the month),
4. stores the result as Parquet in MinIO under a deterministic path per month.

The algorithms behave differently depending on hardware revision and firmware. The version history makes it possible to evaluate each prediction against the firmware behavior that produced its inputs.

Manual review of samples

Some devices do not produce usable evaluation data. A thermostat that was offline for two weeks in the middle of the month is not a fair test for a runtime prediction. The edge cases are too varied to define "usable" in code.

After the datasets are built, a reviewer opens a Streamlit UI that shows per-sample plots (voltage curves, valve movement patterns, connectivity gaps) and marks each sample as accepted or rejected. The decisions are stored as a CSV manifest in MinIO next to the dataset, and the evaluation step runs only on accepted samples. The same manifest always yields the same evaluation, so the pipeline stays reproducible while domain experts apply judgment that is hard to express as rules.

Isolated environments per algorithm version

The evaluation compares two or more library versions on the same dataset. They cannot run in one Python process because they are different versions of the same package and may have incompatible dependencies.

The pipeline creates a temporary virtual environment for each version label:

# For each version label (a release, "latest", or a git ref)
# 1. Create a temporary venv
# 2. pip install the library at that version from the private PyPI index
# 3. Run the evaluation in a subprocess
# 4. Collect the results and remove the venv

Version labels can be release versions, latest, or git refs. Dagster partitions the evaluation by month and by library version, so every combination is tracked.

This setup needs no container builds and no per-version CI pipelines. Each venv exists only for the duration of one evaluation run.

Version comparison in MLflow

Each evaluation logs to MLflow:

• per-sample predictions and ground truth (capacity in mAh, runtime in days)
• error distributions (mean, median, percentiles)
• comparison plots across versions

With these runs, a question such as whether a new version reduced the median runtime error on a given month's dataset can be answered in MLflow without writing a new analysis script.

The Dagster assets record metadata (sample count, error count, execution duration) that is shown in the Dagster UI, so a failed or incomplete evaluation run is visible there.

Fleet-wide rollout

The evaluation pipeline processes curated samples from a few hundred devices. After a version is selected, it has to run on all 100k+ devices.

A separate FastAPI application does this. It starts background worker processes, each in its own venv with the target library version:

• workers process devices in batches of 250
• each batch writes a progress checkpoint to disk (JSON and CSV snapshots)
• results are uploaded to MinIO as Parquet when the run is complete
• an admin UI shows progress: devices processed, error count, elapsed time

A full fleet run takes hours. If a worker crashes partway through, the run can continue from the last checkpoint.

The service has three computation modes:

• per-device Q10/Q90 capacity and runtime estimates
• bucketed distribution snapshots (30-day runtime ranges, 10% capacity ranges) for fleet-level health views
• a configuration mode that resets battery parameters across the fleet through the API

How the two tools connect

field data
  -> curated datasets
  -> version evaluation
  -> MLflow comparison
  -> human decision
  -> fleet rollout of the selected version
  -> new field data

Evaluation datasets and fleet results are both stored in MinIO as versioned Parquet.

What I would change

• Dataset versioning: MinIO paths encode year, month, and version, but there is no formal lineage tracking. If the extraction logic changes, old datasets can no longer be compared with new ones, and nothing flags this. DVC or a metadata table would address it.
• Worker orchestration: the FastAPI subprocess pattern works but is fragile. Dagster could also run the fleet jobs and provide retries and checkpointing.
• Gating: a person reads the MLflow results and decides whether to promote a version. An automated gate, for example "promote if the median error improved by more than 5%", would replace this manual step.

Stack