Company with its own Ceph cluster

Ceph storage encrypted after the fact: every OSD, zero downtime

Frybad's role
Planning, execution in service, handover (secrets, documentation, runbooks)
Stack
Ceph, OSD encryption at rest

Starting point

The storage cluster had just been migrated to Ceph; the OSDs had been created unencrypted. The “encryption at rest” requirement came afterwards, as it often does: from a customer request or an internal policy. Ceph does not encrypt an OSD in place; each one has to be drained, recreated and refilled. The workloads on top were not allowed to notice.

What we did

  • Planning. Inventory of the cluster, OSD order, free space and redundancy during each step, backfill speed, abort criteria. Output: a runbook with checks before and after every OSD.
  • Execution, one OSD at a time. Each OSD was taken out of the cluster, drained, recreated encrypted and taken back in; only once the cluster was fully healthy again did the next one follow. That kept every file within the agreed redundancy the whole time.
  • No maintenance window. The applications on the cluster kept running throughout. The pace was set by the backfill, not by the encryption itself.
  • Handover. The customer received the key secrets, documentation of the new state and our runbooks for operations and restart.

Why it takes time: encrypting one OSD is a matter of minutes; refilling it with data takes hours, and during the refill the cluster is deliberately throttled so the applications do not suffer. The sum of those waits is the project duration.

What this means for your project

If your Ceph cluster runs unencrypted or a migration onto it is coming up, start with Kubernetes platforms, built and run; Ceph is the storage there.

Result

Example from a client project.

Every OSD of the cluster encrypted, zero downtime for the applications.

Secrets, documentation and runbooks handed over; the customer can run operations and restarts without us.

The cluster is large and consists of several drive types and storage classes. Not included: the data volume.

We have taken the same step on our own platform: Since 2 Oct 2026, the storage and system disks of our virtualisation hosts have been encrypted. What we run there is on Infrastructure: where your data lives.

What we would do differently today

Migrate encrypted, not encrypt afterwards. The OSDs had only just been created. Had they been created encrypted during the Ceph migration, the second fill would not have been needed.

Measure backfill speed first. It sets the project duration. One measurement on a single OSD before planning makes the estimate reliable.

Similar situation?

Tell us how big your cluster is and what runs on it. We will tell you how long encrypting it in service would take in your case.