Field report
Encrypting Ceph OSDs after the fact: how we did it while in service
Since 2 Oct 2026 the storage and system disks of our virtualisation hosts have been encrypted. We rebuilt the Ceph OSDs from 24 to 30 Sep 2026 while the cluster stayed in service: one OSD after another, each recreated under the same ID, encrypted with dm-crypt, and the next one only after its data had fully backfilled.
This article shows why we neither rebuilt from scratch nor drained the OSDs first, which steps each OSD went through, how we removed old plaintext leftovers from the disks, what went wrong along the way, and what the encryption protects and what it does not. It ends with a checklist with commands.
1. Why retrofit instead of rebuilding
Ceph only encrypts an OSD if it is created encrypted. There is no switch for an existing OSD. If you want to encrypt a running cluster, you have three routes: build a new cluster next to it and move over, drain each OSD first and then recreate it, or recreate each OSD in place and let the data backfill from the other copies.
We chose the third route. It needs no extra hardware and moves the data of an OSD only once. The price: while an OSD is being rebuilt, its data exists in only two copies under three-way replication. That window lasts from stopping the OSD until the backfill has finished.
The Ceph documentation says this works: encrypted and unencrypted OSDs can run side by side in the same cluster. A cluster can therefore be half rebuilt for days without bothering Ceph or the applications.
2. What we rebuilt
Every OSD of our Ceph cluster. It holds the virtual machines, the object storage and the database volumes of our platforms, including those of Fortlauf, our own product.
One device class at a time, the most important data last. First came a class of SSDs that holds no production data of the applications. We tried the procedure out on it. Then followed the hard disks with the archive, and last the SSDs with the database volumes.
A reboot of every host after each class. An encrypted OSD has to come back up by itself after a reboot. We did not assume that; we checked it on every host individually after each class (section 4).
Operations carried on. For the reboots we moved the virtual machines to other hosts each time. A machine with passed-through hardware cannot be moved; it was off while its host rebooted.
3. How one OSD was rebuilt
Every command was approved individually, and every risky step was preceded by a check that aborts on a wrong value.
Each OSD went through the same steps:
- Identify the device beyond doubt. Read the OSD ID, host and disk from Ceph itself, not from a list or from memory. Note the serial number.
- Check that the cluster can afford to lose a copy. All placement groups
active+clean,ceph osd ok-to-stopconfirmed, no other OSD down. - Set
noout. This stops Ceph from moving the data of the stopped OSD to other OSDs. - Stop the daemon, then
ceph osd destroy. Destroy keeps the ID, the place in the CRUSH map and the weight. That is why the data later lands in the same place again. - Clean up leftovers. Remove the OSD’s old activation units, otherwise they run into nothing on every reboot (section 5).
- Wipe the disk and remove plaintext leftovers (section 4).
- Recreate it encrypted with
ceph-volume lvm create --dmcrypt --osd-id <ID>. - Seven checks before the OSD counts as done: encrypted according to
ceph-volume, to the block device and tocryptsetup(LUKS2, AES-XTS); OSDupandin; device class unchanged; CRUSH weight as before the rebuild; no data movement in other device classes. - Wait until every placement group is
active+cleanagain. - Check on the application side. Fortlauf checks a sample of the archived documents against their stored checksums every day. That check reported no error during the entire rebuild.
- Unset
nooutand compare the health status with the state before the rebuild.
What we checked beforehand. Before the first OSD we checked the procedure against the Ceph documentation and the Ceph source code. Four points changed the procedure:
ceph osd outwould have doubled the work. Setting an OSD “out” before the rebuild copies its data away once and back again after the rebuild. Withnooutand no draining, the data moves only once.- After
destroya clock is running. Ceph automatically marks a destroyed OSD “out” after ten minutes (the default) and starts redistributing its data. Onlynooutstops that. Withoutnoout, one data movement silently becomes two. - A failure while recreating can remove the OSD entirely. If
ceph-volume lvm createfails halfway, it cleans up withpurge-newand takes the place in the CRUSH map with it; then a full redistribution begins. This is a known, open bug in the Ceph tracker (#50878). It did not occur for us; the runbook said to readceph osd treefirst after any failure and never to retry blindly. --crush-device-classhas no effect when an ID is reused. The device class is kept anyway, becausedestroydoes not touch the CRUSH map. We checked it after every OSD with a string comparison instead of relying on the option.
Also: the “Replacing an OSD” guide in the Ceph documentation names no step that stops the daemon, and no noout. But Ceph refuses destroy and zap on a running OSD. On Proxmox VE we deliberately did not use pveceph osd destroy, because it removes the entry from the CRUSH map.
4. What we measured
| What | Result |
|---|---|
| Period | 24 to 30 Sep 2026, with breaks overnight |
| Data backfill per OSD, SSD | mostly 35 to 60 minutes, at most about 90 minutes |
| Data backfill per OSD, hard disk | mostly 70 to 120 minutes, once a little over three hours |
| Overwriting a hard disk with zeros (where needed, section 5) | just under three hours per hard disk |
| Reboot of a host until all its encrypted OSDs were running again | about 2 to 3 minutes, without intervention |
| Final check on 30 Sep 2026 | every OSD encrypted (consistent across the Ceph configuration, ceph-volume and the block devices), every placement group active+clean, CRUSH weights as before the rebuild |
Plaintext leftovers: the part many guides leave out. ceph-volume lvm zap deletes the signatures and overwrites the first 10 MB with zeros. The rest of the disk keeps the old, unencrypted data until something happens to be written over it. If a cluster is only a small fraction full, the new, encrypted OSD only overwrites a small fraction too. Without a separate step, most of the old data would remain in plaintext on a disk that counts as encrypted.
Depending on the device, we used one of three tools, and measured beforehand which one the device supports:
- SSDs:
blkdiscard. Checked beforehand withlsblk -Dthat discard gets through the controller. This takes seconds. A caveat: a plain discard (TRIM) is not a reliable erase; whether and when the firmware actually clears the blocks is up to the device. Where the device supports them,blkdiscard --secure,nvme format --ses=1ornvme sanitizeon NVMe, and ATA Secure Erase or Sanitize on SATA SSDs are the more reliable choice. - Hard disks with ATA Sanitize:
hdparm --sanitize-crypto-scramble. The disk discards its internal key, and every sector is unreadable afterwards. This also takes seconds. - Hard disks where Sanitize did not get through: fully overwritten with zeros (
blkdiscard -z), just under three hours per disk.
5. What went wrong and what we changed
- Sanitize did not get through everywhere. On some hosts the RAID controller in JBOD mode rejected the sanitize command; on others with the same controller model and the same BIOS version it did not.
hdparmreported an error, and nothing was started. The OSD in question stayeddestroyedundernooutuntil the decision was made: these hard disks get overwritten with zeros. That cost just under three hours per disk instead of seconds. Our lesson: whether a command gets through a controller holds per host, not per model. We check it on every host beforehand, read-only. - A success without a success message. On another host the scramble returned exit code 0, but not the line “Completed Without Error” that other hosts showed. Instead of trusting the status text, we read the content: the area that
zaphad previously written with zeros contained random data afterwards. That proved the scramble. - Old activation units ran into nothing. At the first test reboot we found activation units on two hosts from earlier OSD replacements, pointing at OSDs that no longer existed. They tried for about 150 seconds and then reported success anyway. Ceph-volume ends its retries without an error code, even if activation never succeeded. That is why
systemctl statusafter a reboot is no proof. The proof is thatceph osd treeshows every OSD of the host asup. We removed the old units when we rebuilt the OSD in question. - A check stopped us, rightly. Before one OSD, the entry check reported that not every placement group was
active+clean: an ordinary scrub was running. The check aborted, we waited, and it passed on the second attempt. - A counting command counted the header line. In the runbook, one command counted the header line of its output as a device. The first check therefore showed a difference of one. We cleared it up before going on and corrected the runbook.
- The connection to the hosts dropped several times. The cluster carried on unaffected; only our view of it broke off. We therefore started long operations such as overwriting with zeros in a way that a dropped connection does not end them. After every drop we first read the state (destroyed OSDs, flags, placement groups) before we carried on.
None of these points cost any data. Without an explicit check, three of them would have given false confidence: the missing proof of the scramble, the green status of the activation units, and a counting error that can hide a missing device.
6. What the encryption protects and what it does not
Scope: the encryption protects disks that are switched off or removed. Not included: data on a running system.
- Protected: a disk that goes for repair, is decommissioned or goes missing. Before a disk leaves operations with us, it is additionally wiped and the result is recorded.
- Not protected: anything that is allowed to read on a running system. Anyone with valid credentials for an application or a tenant reads the data unencrypted; to them, disk encryption is invisible. Access rights, tenant isolation and monitoring help against that, not dm-crypt.
- No backup. An encrypted OSD does not replace a backup at another location. How we measure database restores is shown in the database restore drill.
- Keys. Without the keys, the data of an encrypted OSD is lost to you as well. Before the first OSD, settle how the keys are secured and how you get to them in an emergency, and rehearse that. How we solve it, we do not publish.
7. When retrofitting is worth it
- When hardware can leave the building: repair, return at the end of a lease, decommissioning, colocation. This is exactly what disk encryption is built for.
- When a security questionnaire asks about “encryption of data at rest” and you want to back the answer with a date and a scope instead of “planned”.
- Before the cluster fills up. The rebuild moves each OSD’s data once. The more data, the longer the window with two copies instead of three.
- When moving to Proxmox VE and Ceph. If you are moving from VMware to Proxmox VE, it is best to create the OSDs encrypted from the start. Then this whole article does not apply.
- And always afterwards: every new OSD is created encrypted. A single unencrypted OSD, added later, brings plaintext back into a pool that everyone believes is encrypted, and nothing warns you.
8. Checklist for your own rebuild
Before:
- Ceph and the hosts’ operating system up to date; only then touch the first OSD, and do not update in between.
- Settled and rehearsed how the keys are secured and how you get to them in an emergency.
- Disk health data read; conspicuous disks on a watch list.
- Measured per device which tool works against plaintext leftovers (
lsblk -D,hdparm -I), on every host individually. - Starting state saved:
ceph osd df tree, device classes, health messages. - Order fixed: first a class without production data, the most important data last.
- An application-side integrity check you can read between OSDs.
ID=12 # example
ceph osd tree | grep -w "osd.$ID" # note class, weight, up/in
CLASS=$(ceph osd tree -f json | jq -r ".nodes[]|select(.id==$ID)|.device_class")
DEV=$(ceph-volume lvm list "$ID" --format json | jq -r ".[\"$ID\"][0].devices[0]")
lsblk -o NAME,SIZE,MODEL,SERIAL "$DEV" # note the serial number
ceph pg stat # all active+clean?
ceph osd ok-to-stop "osd.$ID"
Per OSD:
nooutset, and checked that it is set.
ceph osd set noout
systemctl stop "ceph-osd@$ID"
ceph osd tree | grep -w "osd.$ID" # must show "down" AND "in"
ceph osd destroy "$ID" --yes-i-really-mean-it # never purge, never pveceph osd destroy
ceph osd tree | grep -w "osd.$ID" # "destroyed", weight unchanged
ceph-volume lvm zap --destroy --osd-id "$ID"
- Old activation units of this OSD removed (
systemctl list-units 'ceph-volume@lvm-<ID>-*' --all). - Immediately before wiping the leftovers, identified the device again: host, serial number, no OSD claims it.
lsblk -o NAME,SIZE,SERIAL "$DEV"
ceph-volume lvm list | grep -c "$DEV" # must be 0
lsblk -D -o NAME,DISC-GRAN,DISC-MAX "$DEV" # DISC-MAX > 0: discard works
blkdiscard -v "$DEV" # SSD; not a reliable erase
# more reliable where supported: blkdiscard --secure, nvme format --ses=1, nvme sanitize
# hard disk with Sanitize:
# hdparm --yes-i-know-what-i-am-doing --sanitize-crypto-scramble "$DEV"
# otherwise, takes hours, start it so a dropped connection cannot end it:
# nohup blkdiscard -z -v "$DEV" > zerofill.log 2>&1 &
- Scramble proven by the content, not by the status text: the area zeroed by
zapmust look random afterwards.
# After a scramble: almost all bytes non-zero. Zeros mean: not wiped.
dd if="$DEV" bs=1M count=1 skip=2 2>/dev/null | tr -d '\0' | wc -c
- Recreated encrypted, same ID. On a failure, read
ceph osd treefirst, never retry blindly.
ceph-volume lvm create --objectstore bluestore --data "$DEV" \
--osd-id "$ID" --dmcrypt --crush-device-class "$CLASS"
ceph-volume lvm list "$ID" | grep encrypted # "encrypted 1"
lsblk -o NAME,TYPE "$DEV" | grep -c crypt # at least 1
[ "$(ceph osd tree -f json | jq -r ".nodes[]|select(.id==$ID)|.device_class")" = "$CLASS" ] \
&& echo "class ok" || echo "class wrong"
ceph osd df tree | grep -w "osd.$ID" # compare the weight with the starting state
- Waited until every placement group is
active+clean; duration noted. The first OSD of each class is your estimate for the rest of the class. - Application-side integrity check read.
nooutunset, health messages compared with the starting state.
After each class:
- Rebooted every host individually; the proof is
ceph osd tree, notsystemctl status.
ceph osd set noout
# Move the guests off the host, then reboot. Afterwards:
ceph osd tree # every OSD of the host up and in?
grep -E 'failed activating|successfully triggered' /var/log/ceph/ceph-volume-systemd.log | tail -20
lsblk -o NAME,TYPE | grep -c crypt # = encrypted OSDs on this host
ceph osd unset noout
- If an encrypted OSD does not come back by itself after a reboot: stop, find the cause, and only then the next class.
After:
- Number of encrypted OSDs equals the number of all OSDs, from at least two sources.
- Every future OSD created encrypted (
ceph-volume lvm create --dmcrypt, on Proxmox VEpveceph osd create … --encrypted 1). - Every disk wiped before it leaves operations, and the result recorded.
Where this fits
We build and run Kubernetes platforms with Ceph as storage, on our own hardware or yours: Kubernetes platforms, built and run. Where the data behind this rebuild lives, and our other dated measurements: Where your data lives.
German version: Ceph-OSDs nachträglich verschlüsseln: wie wir es im laufenden Betrieb gemacht haben.
Sources
Read on 12 Oct 2026.
- Ceph documentation: Adding/Removing OSDs, section “Replacing an OSD”. Sequence
safe-to-destroy,destroy,zap, recreate with--osd-id. - Ceph documentation: ceph-volume, Encryption. Encryption with dm-crypt via
--dmcrypt; encrypted and unencrypted volumes side by side. - Ceph tracker #50878: “ceph-volume can purge OSD being re-created”.
The matching serviceKubernetes platforms, built and run