Guide

Release environments on Kubernetes: provision with Crossplane, test with Argo Workflows

A release environment is a complete copy of the platform for exactly one release candidate: its own namespace, database, topics, test data. Crossplane creates it from a single Kubernetes object; Argo Workflows runs the end-to-end tests on it and cleans up afterwards. This article covers how the two fit together, the decisions that come first and where such environments fail.

It is written for platform and test teams that have a shared staging environment today and wait too long for a free slot for each release.

1. Why a shared staging environment is not enough

On an event-driven platform with many teams, in the client project Kafka with Avro schemas and Spring Boot services, every release meets in the same staging environment. The same thing always happens there: team A is testing, team B rolls out a new version in the meantime, and team A’s failed test can no longer be attributed. Someone resets the database, and the test data of three other teams is gone. In the end, a wiki page records who “has” staging when.

In a project where we were responsible for the release tests of a data platform for passenger information, the starting point was exactly that: a basic GitLab CI pipeline deploying to a shared environment, and tests only one person could reproduce from memory. The way out has two halves: environments created per release candidate, and tests that run without people.

2. What a release environment has to contain

Before the first tool comes a list. Which building blocks does a release candidate need for an end-to-end test to tell you anything?

Building block Per environment or shared Why
Namespace with ResourceQuota and LimitRange per environment Isolation, and a ceiling so that ten parallel environments do not drain the cluster
Database (schema or instance) per environment Tests write; shared data makes results unreproducible
Message topics or a topic prefix of its own per environment A consumer may only see events from its own release candidate
Schema registry shared, with a prefix The schemas are versioned; only the subjects need an environment prefix
Data suppliers (simulators) per environment External data sources are replaced by test suppliers that feed in reproducible events
Ingress hostname per environment So that test runners and people can reach the environment
Secrets per environment, generated No production credentials in test environments

The column “per environment or shared” is the real architecture decision. Every shared building block is a source of errors later; every dedicated one costs time to build. A Kafka cluster per environment rarely pays off; a topic prefix per environment almost always does.

3. Crossplane: an environment is a Kubernetes object

Crossplane extends Kubernetes with resource types of your own. For release environments, you define a CompositeResourceDefinition (XRD), say ReleaseEnvironment, and a Composition that describes which objects such an environment produces: namespace, quota, database, topics, secrets, ingress.

apiVersion: platform.example.org/v1alpha1
kind: ReleaseEnvironment
metadata:
  name: rc-2419
spec:
  release: '2419'
  imageTag: 'sha-8f3a1c2'
  ttl: 6h
  dataset: baseline-small

A team does not need to know more than that. The object is created, Crossplane renders the composition, and a few minutes later rc-2419.release.example.test answers. When the object is deleted, every derived resource disappears with it. That is the decisive difference from a shell script that creates environments: Crossplane knows the desired state and reconciles continuously. A namespace someone deleted by hand comes back; an object nobody needs any more leaves nothing behind.

Three things belong in the composition that teams like to forget:

  1. A ttl field and something that enforces it. Crossplane itself deletes nothing after a time. A small CronJob or an Argo workflow that deletes expired ReleaseEnvironment objects keeps the cluster from being full on Friday evening.
  2. The database as a provider resource, not as a pod. If you put Postgres per environment into the namespace as a Deployment, deleting the namespace also deletes the data, which is intended, but annoying when debugging a failed test. A database operator, or a schema in a shared instance with an environment-specific name, is cleaner; the test run can then save a dump as an artefact before the environment goes.
  3. Status fields the test runner can read. The composition writes the hostname, the database secret name and the topic prefix into status. The workflow reads them from there instead of guessing naming conventions.

4. Argo Workflows: the test as a DAG

Argo Workflows runs steps as containers and knows the dependencies between them. An end-to-end release test is a directed graph:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: release-test-
spec:
  entrypoint: main
  onExit: teardown
  templates:
    - name: main
      dag:
        tasks:
          - name: provision
            template: create-environment
          - name: wait-ready
            template: wait-for-status
            dependencies: [provision]
          - name: seed
            template: load-dataset
            dependencies: [wait-ready]
          - name: feed
            template: start-simulators
            dependencies: [seed]
          - name: e2e
            template: run-testrunner
            dependencies: [feed]
          - name: report
            template: publish-report
            dependencies: [e2e]
    - name: teardown
      # deletes the ReleaseEnvironment object, saves logs and dump first

The onExit handler is the part that reliably clears environments away, even when a step fails or a person aborts the workflow. Without it, half-finished environments pile up, and nobody knows which one is still needed.

The steps themselves are ordinary containers. In our case they were a Node.js test runner that feeds in events and checks the output on the target channels, and Spring Boot and Kotlin services that query states and prepare data as test APIs. Argo does not care which language the tests are in. What matters is that every step receives its inputs as parameters and stores its results as artefacts, so the graph stays repeatable.

What a step should return

  • Exit code 0 or non-zero: Argo decides on success, retry or abort from it.
  • A JUnit or JSON report as an artefact: GitLab or the dashboard displays it.
  • Logs of the environment before the teardown: without them, a red test can no longer be analysed after the clean-up.

5. The trigger: GitLab CI or an event

There are two common ways to start the workflow:

From the pipeline. The GitLab CI stage release-test creates a workflow via kubectl or the Argo CLI and waits for the result. Advantage: the result is in the merge request. Disadvantage: a pipeline job blocks a runner for the entire run. For tests that run for an hour, that is expensive.

Via an event. Argo Events or a small webhook service receives the pipeline event and starts the workflow decoupled. The pipeline ends immediately, and the status comes back as a commit status. That scales better but costs one more building block.

The first way is the one to start with. Once more than a handful of workflows run per day, the second pays off.

6. Test data and data suppliers

The hardest part of a release environment is not the infrastructure but the state it is supposed to start in. Two patterns have proven themselves:

  • A versioned baseline dataset. A dump or a collection of fixtures with its own version, living in the repository together with the application. Every environment starts from exactly this state. If a release changes the schema, the same release changes the baseline dataset.
  • Simulators instead of live connections. On platforms fed by many data suppliers, the real sources cannot be redirected into a test environment. One simulator per supplier feeds in recorded or generated events, in a reproducible order and with controlled timestamps. The test then checks what becomes of these events on the output channels.

Time dependencies remain difficult. An event “departure in 5 minutes” is worthless an hour later. Simulators should therefore set times relative to the start of the environment, not absolutely.

7. Legacy: when the test runner itself is the problem

In many teams, test automation has grown historically: a script a tester started years ago that now has a thousand lines and no tests of its own. Building something like that into an Argo workflow only automates the unreliability.

Our order in such cases:

  1. Leave the runner as it is, and first package it in a container that builds reproducibly.
  2. Pull the configuration out of the code: hostnames, credentials and datasets become parameters.
  3. Only then refactor, module by module, with unit tests for the checking logic.
  4. In parallel, bring in the company-wide DevSecOps requirements: dependency scans, signed images, no secrets in the repository. A test runner is production code and is treated like it.

The temptation to rewrite everything is strong. But it costs months in which nobody tests releases. The step-by-step variant delivers results from week one.

8. Costs and limits

Every environment costs CPU, memory and, if cloud databases are involved, money. Three levers:

  • Small datasets. The test data has to cover the cases, not the production size.
  • A short TTL. Six hours is enough for almost every test run; anyone who wants to debug longer extends it deliberately.
  • A limit on parallel environments. A ResourceQuota with an object count (count/releaseenvironments.platform.example.org) in the namespace where environments are requested, or a semaphore in Argo (synchronization) prevents twenty merge requests from creating twenty environments at once.

Where release environments do not help: with tests that need production load, and with errors that only occur with real data suppliers. For those, a staging environment with real connections still makes sense, but as the last stage before the release, not as a playground for everyone.

9. Checklist

Before the build:

  • List of building blocks per environment, with the decision “dedicated or shared” (section 2).
  • Baseline dataset versioned in the repository.
  • Simulators for every external data source, with relative time.

Crossplane:

  • XRD ReleaseEnvironment with release, imageTag, ttl, dataset.
  • Composition creates namespace, quota, LimitRange, database, topics, secrets, ingress.
  • Status fields for hostname, secret name, topic prefix.
  • TTL enforced by a CronJob or a workflow.

Argo Workflows:

  • DAG with provision → wait-ready → seed → feed → e2e → report.
  • onExit teardown saves logs and dump, then deletes the environment object.
  • Reports as artefacts, clean exit codes.
  • Semaphore for the number of parallel environments.

Test runner:

  • Builds reproducibly as a container.
  • Configuration entirely via parameters.
  • Unit tests of its own for the checking logic.
  • Meets the company’s DevSecOps requirements (scans, signatures, no secrets in the code).

10. Frequently asked questions

Do I need Crossplane, or is Helm enough? Helm renders manifests, but has no reconcile loop and knows no cloud resources outside the cluster. For environments made only of Kubernetes objects, a Helm chart plus a clean-up job is enough. As soon as databases, buckets or DNS records outside the cluster come in, Crossplane is the better-suited tool.

Argo Workflows or Tekton? Both can run DAGs. Argo Workflows has the more mature interface, exit handlers and artefact management; Tekton is a vendor-neutral CD Foundation project and the basis of OpenShift Pipelines. If you already use Argo CD, you will usually take Argo Workflows.

How long does it take to build an environment? That depends on the slowest resource, usually the database. Measure it: a timestamp when the object is created and when it reaches the Ready status. Anything over ten minutes is a hint to provision the database as a schema rather than an instance.

Can developers create environments by hand too? Yes, that is one of the advantages: a kubectl apply with a five-line object is enough. Apply the same TTL and the same parallelism limit as for the pipeline.

Where this leads

We run our own hardware in a data centre in Munich. For us, release environments, test orchestration and the platform underneath are one area of work; how we offer this for platforms on AWS and GCP is described under DevOps & SRE for AWS and GCP.

Example from a project: Release testing a data platform for passenger information.

What to measure in such an environment when response times rise under load: Reducing p95 latency on Kubernetes.

German version: Release-Umgebungen auf Kubernetes: mit Crossplane bereitstellen, mit Argo Workflows testen.

Sources

All read on 20 Oct 2026.

Release tests that run without you?

If your releases currently depend on a shared staging environment, we look at which building blocks would have to exist per environment, and what the first step would be, in an intro call.