Pipelines That Scale: Why Directed-Graph Workflows Win

The Limits of Sequential Scripts

Most machine learning projects begin as a collection of scripts: one for data preprocessing, one for feature engineering, one for training, one for evaluation. The scientist executes them in order, passing outputs from one stage to the next through shared directories or naming conventions. This works for early exploration, but it does not scale.

As projects grow, the problems multiply. Dependencies between stages become implicit. A change to the feature engineering step may invalidate the training step, but nothing enforces that relationship. Re-execution is manual and error-prone. Parallelisation opportunities are invisible. And when an auditor asks "what exact sequence of operations produced this model?"—the answer depends on the scientist's memory and discipline, not on a system of record.

Sequential scripts encode workflow knowledge in the scientist's head. Directed graphs encode it in infrastructure.

The solution is to model ML workflows as what they actually are: directed acyclic graphs (DAGs) of components with explicit data dependencies. This is not a new idea—build systems like Make have used dependency graphs for decades—but its application to ML pipelines has become critical as models move from research into regulated production.

Components, Dependencies, and Data Flow

In a graph-based pipeline, each processing step is an isolated component with defined inputs and outputs. Components do not share state except through their declared interfaces. This isolation provides three immediate benefits:

• Testability — Each component can be validated independently with known inputs and expected outputs
• Reusability — A well-defined component can be shared across pipelines without modification
• Auditability — The data flowing between components is explicit and can be inspected at every boundary

Dependencies between components are declared, not implied. If the training component depends on the output of the feature engineering component, that relationship is part of the pipeline definition, not an assumption in a README. The orchestrator can use these declarations to determine execution order, identify what can run in parallel, and skip components whose inputs have not changed.

A Concrete Example

Consider a typical molecular property prediction pipeline in a pharma research context:

1. Data Ingestion — Fetch molecular structures and assay results from a versioned data store
2. Featurisation — Compute molecular fingerprints and descriptors (depends on step 1)
3. Train/Test Split — Partition data using a scaffold-aware splitting strategy (depends on step 2)
4. Model Training — Train a gradient-boosted model on the training partition (depends on step 3)
5. Evaluation — Compute metrics on the held-out test set (depends on steps 3 and 4)
6. Registration — If metrics exceed threshold, register the model in the model registry (depends on step 5)

As a DAG, this pipeline makes the dependency structure visible. Steps 1 through 3 form a linear chain, but evaluation depends on both the split data and the trained model. The registration step includes a conditional gate. An orchestrator can express all of this declaratively and execute it reproducibly.

Kubernetes-Based Portability and Governance

Workflow engines built on Kubernetes—such as those following the Kubeflow Pipelines model—bring a powerful combination of portability and governance to ML pipeline execution.

Portability

Each pipeline component runs in its own container. This means the execution environment is fully specified: the base image, installed libraries, system dependencies, and runtime configuration are all captured in the container definition. A pipeline that runs on a development cluster will produce identical results on a production cluster, because the environment travels with the code.

This portability is particularly valuable in regulated environments where validation is expensive. Once a pipeline has been validated on one Kubernetes cluster, the same containers can be deployed to another cluster with confidence that the computational behaviour is preserved. The infrastructure becomes interchangeable; the pipeline definition is the constant.

Governance Through Infrastructure

Kubernetes provides native mechanisms for governance that map directly to regulated requirements:

• Namespace isolation ensures that development, staging, and production pipelines operate in separate security contexts
• Role-based access control (RBAC) restricts who can create, modify, or execute pipelines in each namespace
• Resource quotas prevent runaway experiments from consuming shared infrastructure
• Audit logging captures every API call, providing a comprehensive record of who did what, and when

These are not ML-specific features. They are standard Kubernetes capabilities. But when combined with a pipeline orchestrator that runs each component as a Kubernetes job, they provide a governance framework that satisfies many regulatory expectations without custom tooling.

Practical Controls for Regulated Pipelines

Beyond the structural benefits of DAG-based execution, regulated pipelines require specific controls that graph-based orchestrators are well-positioned to provide.

Approval Gates

Certain transitions in a pipeline should not proceed automatically. Moving a model from training to production deployment, for example, may require human review and explicit approval. Graph-based orchestrators can model these as gate nodes—components that pause execution until an authorised user provides approval through a defined interface. The approval, the approver's identity, and the timestamp become part of the pipeline's execution record.

Secrets Handling

Pipelines that access external data sources, APIs, or storage systems require credentials. In a regulated environment, these credentials must never appear in pipeline definitions, logs, or artefact metadata. Kubernetes-native secrets management, combined with pipeline-level secret injection, ensures that credentials are available to the components that need them without being persisted in any record. The pipeline definition references a secret by name; the secret's value is resolved at runtime and never logged.

Reproducibility by Design

A well-constructed DAG pipeline is reproducible by construction. Every component runs in a pinned container image. Every input is a versioned artefact. Every parameter is recorded. Re-executing the pipeline with the same inputs and parameters will produce the same outputs—and the orchestrator can verify this by comparing artefact hashes across runs.

Reproducibility is not a feature you add to a pipeline. It is a property that emerges from building pipelines correctly—with isolated components, versioned inputs, and deterministic execution.

Where Workflow Engines Belong in a Platform Roadmap

In the platform tier architecture, workflow engines sit at Tier 03: AI Workflow Engine. This positioning is deliberate. Workflow orchestration depends on capabilities provided by the lower tiers:

Teams should not attempt to deploy a workflow engine before the foundational tiers are in place. A pipeline orchestrator running on unreliable compute, without experiment tracking, and without a model registry is just an expensive cron job. The value of the orchestrator emerges from its integration with the surrounding platform.

Adoption Sequence

A practical adoption sequence for most regulated biotech teams follows this pattern:

1. Establish reliable, isolated compute environments with GPU access (Tier 01)
2. Implement experiment tracking and versioned artefact storage (Tier 02)
3. Define initial pipelines as code, starting with the most critical or repetitive workflows (Tier 03)
4. Add approval gates, model registry integration, and automated compliance checks (Tier 03 + 04)
5. Layer retention policies, audit exports, and regulatory reporting on top (Tier 04)

Each step builds on the previous one. The pipeline definition at step 3 references the tracking infrastructure from step 2, which runs on the compute platform from step 1. Compliance capabilities at steps 4 and 5 wrap around the existing workflow without restructuring it.

From Scripts to Systems

The transition from sequential scripts to directed-graph pipelines is not primarily a technology change. It is a shift in how teams think about their work. Scripts are personal tools; pipelines are shared infrastructure. Scripts encode individual practice; pipelines encode organisational process. Scripts are run; pipelines are executed, recorded, and audited.

For biotech organisations operating under regulatory oversight, this shift is inevitable. The question is not whether to adopt graph-based pipeline orchestration, but when—and whether to do so proactively, as part of a deliberate platform strategy, or reactively, under pressure from an audit finding. The proactive path is less expensive, less disruptive, and produces better outcomes. The graph is the right abstraction. Build on it.