Experiment Tracking as Evidence: Turning Runs into Records

Why Run Logs Are Not Enough

Most data science teams track their experiments in some fashion. They may save terminal output, write notes in a shared document, or use a lightweight tracking library to log metrics. In research-only contexts, this is often sufficient. The scientist remembers the context, the notebook preserves the narrative, and the results speak for themselves.

In regulated environments, this approach fails. The audience for experiment records is not the scientist who ran them—it is an auditor, a quality assessor, or a regulatory reviewer who arrives months or years after the fact, with no shared context and a specific set of questions. They need to understand not just what happened, but exactly how it happened, with what inputs, under what conditions, and whether the process was controlled.

A run log tells you what the model achieved. A record tells you how it got there, with what data, under whose authority, and whether the process was controlled.

The gap between a run log and a record is the gap between scientific convenience and regulatory defensibility. Closing it requires intentional design of the tracking infrastructure, not just better logging habits.

What Should Be Logged

A structured experiment record must capture five categories of metadata. Each serves a distinct purpose in reconstructing the provenance of a model artefact.

Parameters and Hyperparameters

Every configurable value that governs the training process: learning rate schedules, optimiser settings, batch sizes, architectural parameters, regularisation coefficients, early stopping criteria, and random seeds. These must be captured programmatically—not manually transcribed—and stored in a structured, queryable format such as JSON or a key-value store.

The critical requirement is completeness. Partial parameter capture is worse than no capture at all, because it creates a false sense of reproducibility. If a reviewer can see that the learning rate was 0.001 but cannot determine the weight decay setting, the record is incomplete and the experiment is not reproducible from its metadata alone.

Artefacts

Every output of the training process that has downstream value: serialised model weights, evaluation plots, confusion matrices, feature importance rankings, and calibration curves. Artefacts must be stored immutably and linked to the run that produced them. A model binary without a traceable link to its training run is an orphan artefact—it may function correctly but it cannot be defended.

Code Version

The exact state of the codebase at the time of execution. At minimum, this means a Git commit hash. In practice, it should also include:

• The full dependency manifest (e.g., requirements.txt, conda.lock, or poetry.lock)
• Any uncommitted changes (ideally, the system should refuse to run experiments with uncommitted code)
• The container image digest, if execution occurs within a containerised environment

Data Pointers

References to the exact datasets consumed during training and evaluation. These should be immutable identifiers—content hashes, dataset version numbers, or snapshot identifiers—rather than file paths or URLs that may change over time. The pointer must resolve to the same data indefinitely, or the experiment cannot be reproduced.

For large datasets, storing a content hash (such as SHA-256 over the sorted, serialised data) provides a lightweight but robust way to verify that the data has not changed between the original run and any subsequent reproduction attempt.

Environment Hash

A fingerprint of the execution environment: the container image digest, hardware specification (CPU architecture, GPU model, driver version), operating system version, and any runtime flags that affect numerical behaviour (such as deterministic mode settings in deep learning frameworks). Two runs with identical code, data, and parameters can produce different results on different hardware. The environment is part of the experiment, not external to it.

Model Registries and Controlled Promotion

Experiment tracking captures the development phase. But the transition from "interesting result" to "production-grade artefact" requires a separate governance mechanism: the model registry.

A model registry serves as the controlled boundary between experimentation and deployment. It records:

• Model versions with unique identifiers linked to their originating experiment runs
• Stage transitions—from development to staging to production—with timestamps, approvers, and justifications
• Validation evidence attached to each stage transition: test set performance, bias assessments, robustness checks
• Lineage metadata connecting every registered model back to its training data, code version, and parameter set

The MLflow model registry pattern illustrates this well. Models are registered with a name and version. Each version can be transitioned through stages (None, Staging, Production, Archived), and each transition can require an approval step. The registry does not enforce a specific validation protocol—it provides the infrastructure for organisations to implement their own governance rules.

A model registry is not a file server with metadata. It is a governance boundary. Crossing it should require evidence, not just a file upload.

Retention and Access Control

Compliance frameworks impose specific expectations on how long records must be retained and who may access them. Experiment tracking systems in regulated environments must address both.

Retention Policies

In pharmaceutical contexts, records related to product quality may need to be retained for the commercial life of the product plus a defined period (often one year after expiry of the last batch). For clinical trial data, retention requirements may extend to 15 years or more. Experiment tracking systems must support configurable retention policies that prevent premature deletion while enabling eventual archival.

Access Control

Not all experiment records should be visible to all users. Access control must be granular enough to support role-based visibility—data scientists can view and create runs within their projects, quality assurance can review all runs, and external auditors can access read-only views of specific experiments. The principle of least privilege applies to experiment records just as it does to any other regulated data.

Governance Artefacts: Model Cards and Datasheets

Beyond the raw metadata of experiment tracking, regulated environments benefit from structured governance documents that summarise a model's purpose, limitations, and fitness for use. Two formats have emerged as lightweight but effective:

Model Cards

Originally proposed by Mitchell et al. at Google, model cards provide a standardised summary of a trained model's intended use, performance characteristics, ethical considerations, and limitations. In regulated biotech, model cards serve as concise documentation that can accompany a model through the registry and into deployment review. They answer the questions a reviewer needs answered without requiring deep technical expertise.

A model card for a regulated context should include:

• Model purpose and intended use population
• Training data characteristics and known biases
• Performance metrics across relevant subgroups
• Known limitations and failure modes
• Regulatory classification and applicable standards
• Maintenance and monitoring requirements

Datasheets for Datasets

Proposed by Gebru et al., datasheets document the provenance, composition, collection methodology, and intended uses of a dataset. In regulated environments, datasheets complement experiment tracking by providing context that raw data pointers cannot convey. They answer questions about why the data exists, how it was collected, and what biases it may carry—questions that are increasingly relevant to regulatory submissions involving AI-derived insights.

Together, model cards and datasheets form a lightweight documentation layer that bridges the gap between machine-readable experiment metadata and human-readable compliance documentation. They are not substitutes for formal validation protocols, but they significantly reduce the effort required to prepare for audits and regulatory reviews.

Getting Started

The transition from informal tracking to structured evidence does not require a large upfront investment. Begin by instrumenting your training scripts to capture the five metadata categories described above. Use a tracking framework that stores this information in a structured, queryable backend. Ensure that every run produces an immutable record that can be retrieved by its unique identifier.

Once tracking is in place, introduce the model registry as the governance boundary between experimentation and production. Define the evidence required for each stage transition, and automate as much of the evidence collection as possible. The goal is a system where compliance is built into the workflow, not layered on top of it.

The best experiment tracking systems are invisible to the scientist. They capture everything needed for regulatory defensibility without adding friction to the research process. That is the standard to aim for: maximum evidence, minimum overhead.