- ceph-users - lists.ceph.io

Replacing an OSD in Squid
by Patrick Begou 08 Sep '26

08 Sep '26

Hi I'm replacing the HDD of osd.2 in my squid cluster following the provided documentation: https://docs.ceph.com/en/squid/rados/operations/add-or-rm-osds/ The OSD was yet out and the daemon down. The osd.2 is destroyed: cephosddestroy2--yes-i-really-mean-it I've stopped the node and replaced the HDD. Ceph (graphic board) see the HDD as available. But I'm unable to go further with the ceph-volume commands: # ceph-volume lvm zap /dev/sda --> RuntimeError: /dev/mapper/osvg-rootvol not found. # ceph-volume lvm prepare --osd-id 2 --data /dev/sda --> RuntimeError: /dev/mapper/osvg-rootvol not found. Something wrong in my cluster or a step is missing in the documentation ? Thank Patrick

2 10

Trouble With Attaching OSDs From A Previous Cluster To A New One
by Jacek Rużyczka 07 Sep '26

07 Sep '26

Hi folks, I've now been struggling for five weeks with my broken Ceph cluster (4 nodes on Ubuntu 24, containerised with Docker) and can't find a solution, not even with help from Google's AI. Here is what happened: 1. The cluster FS was no longer accessible, and even ceph -s no longer worked (error 110). The MON count had gone down to 2, and an hour later, all 4 OSD daemons were also gone. The corresponding containers had also disappeared (docker ps -a no longer displayed them) together with their logs. 2. A week later, one of the nodes got frozen. Accessing its serial console returned a bunch of low-level errors you can read in this forum thread if you wish: https://community.mixtile.com/t/node-3-unreachable-100-cpu-load/941/6 3. It turned out that some essential files on the nodes (like ceph.conf or keyrings) had simply disappeared. Further research revealed trouble with the internal flash storage (eMMC) of the nodes. So, the data on them was apparently corrupt, what forced me to ressort to "safer" eMMC settings, re-flash them, and re-install Ceph (with the FSID of the old cluster, of course). After that, 4 MONs and 2 MGRs worked, but: 4. Re-integrating the existing OSDs (on the NVMe SSDs, not on the internal eMMCs) did not work: Eugen's script https://heiterbiswolkig.blogs.nde.ag/2026/06/28/cephadm-recover-mon-store-u… ended me up with a "No such file or directory" error. Google's AI made a bunch of suggestions, which did not help either (mostly due to "Permission denied" errors). It also told me to shut down one of the MONs, extract a monmap, and inject it into the new cluster. But the "permission denied" errors persisted for at least two weeks until we could get rid of them. 5. Two OSDs could get revivif <https://dict.leo.org/englisch-deutsch/revivify>ied, but ceph -s marked them as "in", but "down" whatever we did. Extracting the keys from the OSDs and adding them to the config of the "new" cluster didn't help. So I got stuck in a dead end. 6. Now, Google's AI wants me to extract the raw data (which is hopefully still complete) from the drives for whatever purpose and store it on another computer, but I'm afraid that doing this won't help me either. So I need some reliable expert advice in order to bring my cluster back to life. What shall I do here? Are the files in my cluster maybe also corrupted by the abovementioned incident? Thank you. Regards Jacek

1 0

Mount Client/Superblock Sharing Broken Bugfix
by Benjamin Huth 07 Sep '26

07 Sep '26

Hey folks, I opened a bug report on the ceph issue tracker a couple months ago detailing how the mount sharing for the cephfs kernel client was broken and how it was impacting me as a talos user with lots of pods sharing the same, readonly cephfs mounts. The bug report is here: https://tracker.ceph.com/issues/78440 I noticed some work being done in approximately the same area by Xiubo Li from Clyso, and was hoping someone could piggyback my fix into what he's doing: https://github.com/ceph/ceph-client/commit/b9d944f1c8c2044d08609fdc238d27df… My patch looks like this and is compatible with his new SB_ACTIVE flag: fs/ceph/super.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/ceph/super.c b/fs/ceph/super.c index 72935f665f11..13a8f246f1a0 100644 --- a/fs/ceph/super.c +++ b/fs/ceph/super.c @@ -1265,7 +1265,8 @@ static int ceph_compare_super(struct super_block *sb, struct fs_context *fc) doutc(cl, "fsid doesn't match\n"); return 0; } - if (fc->sb_flags != (sb->s_flags & ~SB_BORN)) { + if (fc->sb_flags != (sb->s_flags & + ~(SB_BORN | SB_NODIRATIME | SB_NOATIME | SB_ACTIVE))) { doutc(cl, "flags differ\n"); return 0; } base-commit: dc173b37415e8f738fc4de477490056b479ddc9f I've been running this patch on my talos cluster and it does correctly resolve pods not sharing the same mount and inode cache. Thanks!

1 0

Authentication looks broken
by Sake 06 Sep '26

06 Sep '26

Hi all! We upgraded a staging environment from 19.2.4 to 19.2.6 and rotated all of the keys without issues for the cluster. But for the Cephfs clients we noticed something strange. We created new keys for the Cephfs (kernel) clients so they could switch-over with just a remount. All went fine and thus we removed the old keys. Still everything kept working. Only then we rebooted one of the nodes to apply a new kernel. But it wasn't accessible after a couple of minutes, so we checked the console output and there where authentication errors for mds and mon. With a reset the node got online again, but without it's cephfs mount. When investigating the issue, we forgot to adjust the client name in the fstab. It was using the old name instead of the new name. So it looks like when mounting a cephfs filesystem, the backend isn't checking if the combination of client name and key is correct. Just if either is correct, you can mount the filesystem. Also, why isn't the umount failing after some time when authentication fails and then reboots the machine? Are others hitting this issue also? I'm going to check this on another system later on, but I thought I share this observation. System info Ceph is deployed via Cephadm Clients are running on RHEL 9 Using kernel mounts Installed ceph-common 19.2.6 on clients Kind regards, Sake

1 0

Unable to connect to filesystem Tentacle 20.2.4
by Johan 05 Sep '26

05 Sep '26

I have a cephadm managed cluster, Tentacle 20.2.4 and Debian 12 & 13. On the Debian 12 Bookworm servers I can connect to the cluster. id: <ceph-id> health: HEALTH_OK (muted: AUTH_INSECURE_CLIENT_KEY_TYPE(13d AUTH_INSECURE_KEYS_ALLOWED(13d)) services: mon: 3 daemons, quorum lagenhet4,lagenhet3,lagenhet5 (age 117m) [leader: lagenhet4] mgr: lagenhet3.rbkplm(active, since 2h), standbys: lagenhet4.xontzc mds: 1/1 daemons up, 1 standby osd: 8 osds: 8 up (since 2h), 8 in (since 12d) ... Worth noting is that my osd-disks are encrypted and those keys are not rotated. Keys for the crash services are not rotated either (a number of bootstrap-users are left too, but read in a earlier conversation that those should be safe to delete). One client that I previously have been using to connect to a cephfs have had the key rotated, but I can't connect to the filesystem. If the keyfile is written as before [client.fsuser] key = <key> or [client.fsuser] key = <key> caps mds = "allow rw fsname=filesystem" caps mon = "allow r fsname=filesystem" caps osd = "allow rw tag cephfs data=filesystem" and only replace the key I get the error adding ceph secret key to kernel failed: Unknown error 524 couldn't append secret option: -524 when I try to mount the filesystem. If I only have the key itself in the keyfile I get mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorised To test if I have proper authorisation I created a new user sudo ceph fs authorize newuser client.newuser / rw | sudo tee /etc/ceph/ceph.client.newuser.keyring and I still get the same errors. Looking at the keys in the dashboard it shows the same keys as I'm using. I tested to create a new filesystem (information regarding previous filesystems are removed) # ceph osd pool create dummy_data pool 'dummy_data' created # ceph osd pool create dummy_meta pool 'dummy_meta' created # ceph fs new dummyfs dummy_meta dummy_data Pool 'dummy_data' (id '21') has pg autoscale mode 'on' but is not marked as bulk. Consider setting the flag by running # ceph osd pool set dummy_data bulk true new fs with metadata pool 22 and data pool 21 # ceph fs ls name: dummyfs, metadata pool: dummy_meta, data pools: [dummy_data ] # sudo ceph orch apply mds dummyfs Scheduled mds.dummyfs update... # ceph mds stat <deleted>:1 dummyfs:1 {<deleted>:0=<deleted>.lagenhet4.kvscuw=up:active,dummyfs:0=dummyfs.lagenhet5.jpqcdl=up:active} 2 up:standby # ceph fs authorize dummyfs client.dummyuser / rw | tee /etc/ceph/ceph.client.dummyuser.keyring [client.dummyuser] key = AgBvWpxqEJPpOiAAjr6ziBqhdz23pCcU/2pCeY2ZGHNVKV4pAt13jCeKxr0= caps mds = "allow rw fsname=dummyfs" caps mon = "allow r fsname=dummyfs" caps osd = "allow rw tag cephfs data=dummyfs" # mount -t ceph dummyuser@.dummyfs=/ /datastorage/dummyfs adding ceph secret key to kernel failed: Unknown error 524 couldn't append secret option: -524 I manually change keyring-file to include only the key. # mount -t ceph dummyuser@.dummyfs=/ /datastorage/dummyfs mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized This procedure to create a filesystem has worked before. I also tested this procedure on my main cluster, Tentacle 20.2.1, and there was no problem and didn't have to edit keyring-file. Anyone have any idea what could be wrong? /Johan

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Johan 05 Sep '26

05 Sep '26

Sorry for the delayed answer, I have been setting up a new server to try handle the situation. Im using cephadm. The conainers are running fine as far as I can see. Checking both my two clusters I realise ceph-common was not updated since Reef. It hasn't been any problem until now, easy to forget when you have to manually update the release in apt source-file. I now have a server running Debian 12 Bookworm in the cluster (and updated ceph-common) and I can without problem connect to the cluster. id: <ceph-id> health: HEALTH_OK (muted: AUTH_INSECURE_CLIENT_KEY_TYPE(13d AUTH_INSECURE_KEYS_ALLOWED(13d)) services: mon: 3 daemons, quorum lagenhet4,lagenhet3,lagenhet5 (age 117m) [leader: lagenhet4] mgr: lagenhet3.rbkplm(active, since 2h), standbys: lagenhet4.xontzc mds: 1/1 daemons up, 1 standby osd: 8 osds: 8 up (since 2h), 8 in (since 12d) ... I have observed some other problems that I will address in another mail. /Johan Den 2026-08-26 kl. 08:44, skrev Eugen Block: > Hi, > > according to the "OS Recommendations" [0] Ceph Tentacle is not available > for Debian 13. Umbrella will be built for Trixie. > Are you using cephadm managed clusters or package based? With cephadm > you're not that strictly tied to the installed package versions. > > Regards, > Eugen > > [0] https://docs.ceph.com/en/latest/start/os-recommendations/ > > Zitat von Johan <ceph-users(a)ceph.io>: > >> Ok, >> >> on Debian 12 it was not a problem upgrading to ceph 20.2.4 using apt >> and https://download.ceph.com/debian-tentacle/ >> >> With Debian 13 which I have at my troublesome cluster I get several >> dependency-issues. >> ceph-common is dependent on libicu72 which doesn't have any candidates >> python3-cephfs (and others) is dependent on python3 < 3.12 but >> 3.13.5-1 will be installed >> etc >> >> Is this because I perhaps hadn't upgraded ceph-common and others >> properly before upgrading Debian from 12 to 13? >> >> /Johan >> >> Den 2026-08-24 kl. 11:06, skrev Johan: >>> Im not a professional computer tech but have two ceph clusters >>> (cephadm, Trixie, Debian 12/13). >>> >>> I upgraded the least sensitive one using the dashboard. When I >>> upgrade the admin client I now get >>> auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error >>> setting modifier for [client.admin] type=key val=<key>: Malformed >>> input [buffer:3] >>> auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) >>> Input/ output error >>> >>> I get the same respons when using a ordinary user to connect the >>> filesystem. >>> >>> I found this issue describing the same error >>> https://github.com/rook/rook/issues/18240 >>> >>> As they, in above issue, discussed which version/image of the toolbox >>> was used I checked my version of ceph at the command line. I get >>> ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef >>> (stable) >>> I would guess this is the problem. >>> >>> How do I get to run up-to-date version on the command line? >>> >>> /Johan >>> >>> >>> Den 2026-08-19 kl. 18:09, skrev Patrick Donnelly: >>>> Today, August 19th, 2026, the Ceph project has released Tentacle >>>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>>> components. >>>> >>>> This is the sixth backport release in the Squid series. >>>> >>>> This is the fourth backport release in the Tentacle series. >>>> >>>> Additional details will be posted on the Ceph website/blog: >>>> https://ceph.io/en/news/blog/ >>>> >>>> We strongly recommend that all Ceph operators upgrade to one of these >>>> releases as soon as possible. >>>> >>>> >>>> ## Critical Upgrade Steps >>>> >>>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>>> This is the first time Ceph has introduced a new key type for CephX >>>> credentials. Therefore, a new procedure exists for upgrading and >>>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>>> cluster, the operator should familiarize themselves with the procedure >>>> documented at: >>>> >>>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/ >>>> index.html#upgrading-and-rotating-cephx-keys >>>> >>>> These steps are for package-based deployments of Ceph. In other >>>> sections, the document also includes new information about key >>>> rotation procedures, monitor authentication controls, and emergency >>>> recovery procedures. >>>> >>>> Deployments using cephadm will automate the process except for client >>>> keys. You may notice that Cephadm spends more time than normal on the >>>> upgrade after updating all daemon images. This is due to a new process >>>> rotating the OSD and MDS keys. Please be patient. There are plans to >>>> improve this in the future. >>>> >>>> Deployments using Rook will also automate rotation of some client keys >>>> with some exclusions. Look to the Rook project’s announcement for more >>>> details. >>>> >>>> Client and kernel upgrades are recommended to support aes256k. Note >>>> that upstream Linux kernel client support began in kernel 7.0 and has >>>> been backported to CentOS Stream 9 and 10. Check with your >>>> distribution vendor for backported key support before rotating client >>>> keys used by the kernel. >>>> >>>> As part of upgrading a Ceph cluster, six new health warnings and >>>> errors will be generated. This is normal. As part of the process to >>>> upgrade/rotate entity keys, you will address the warnings and errors >>>> as you go. These are documented beginning here: >>>> >>>> https://docs.ceph.com/en/latest/rados/operations/health-checks/ >>>> index.html#auth-insecure-keys-creatable >>>> >>>> The CephX documentation has been brought up to date and vetted for >>>> accuracy. Those changes are live here: >>>> >>>> https://docs.ceph.com/en/latest/architecture/index.html#high- >>>> availability-authentication >>>> >>>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>>> requests with `host` and `x-amz-` headers not included in the signed >>>> subset. Unfortunately, the REST client used in multisite was >>>> generating such improperly signed requests. >>>> >>>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>>> option to true before you begin to upgrade. After all clusters are >>>> upgraded, set the option to ``false`` again. >>>> >>>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>>> secrets stored in the Monitor config-key store will be forthcoming. In >>>> the meantime, operators of cephadm-managed clusters are strongly >>>> encouraged to rotate the cephadm SSH key, as an established process >>>> exists: >>>> >>>> See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh- >>>> configuration >>>> >>>> Rotating other secrets is advised but the process is not yet >>>> established. Operators should assess their cluster's potential >>>> exposure to determine if immediate Monitor config-key secret rotation >>>> is required for their environment. >>>> >>>> >>>> ## CVE Fixes >>>> >>>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>>> of AES-CBC. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >>>> >>>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>>> in improper verification of a cryptographic signature. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >>>> >>>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>>> subscription handler. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >>>> >>>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>>> cryptographic signatures in RGW's SigV4 verifier. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >>>> >>>> ## Other Media >>>> >>>> A Cephalocon 2025 talk covers some of the details surrounding the new >>>> CephX key upgrade process. >>>> >>>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and- >>>> cephx- keys-patrick-donnelly-ibm-yehuda-sadeh-weinraub-ubiquiti >>>> >>>> ## Getting Ceph >>>> ------------ >>>> * Source at https://github.com/ceph/ceph >>>> * v19.2.6 Tarball at https://download.ceph.com/tarballs/ >>>> ceph-19.2.6.tar.gz >>>> * v20.2.4 Tarball at https://download.ceph.com/tarballs/ >>>> ceph-20.2.4.tar.gz >>>> * Containers at https://quay.io/repository/ceph/ceph >>>> * For packages, see https://docs.ceph.com/en/latest/install/get- >>>> packages/ >>>> * For upgrade with cephadm, see: >>>> https://docs.ceph.com/en/latest/cephadm/upgrade/ >>>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Issue with encrypted OSDs when upgrading from version 19.2.4 to 19.2.6 using cephadm
by Eugen Block 04 Sep '26

04 Sep '26

I did a lot of tests with a virtual test cluster in the past two weeks with encrypted OSDs and I haven't encountered that yet. I would check for ceph-osd and ceph-volume package on the affected hosts. Or for remainders from previous cephadm installations, anything not properly cleaned up maybe? What do the OSDs log when they try to start up? It sounds like some stale systemd units or something, hard to tell. Have you rebooted the hosts in question? On one of my lab clusters I noticed issues with systemd when I re-bootstrap a cluster without cephadm multiple times. I have a bash script that works just fine on a freshly rebooted system, but after wiping everything and starting over, at some point the script doesn't work anymore, especially when it comes to create OSDs. But this might be unrelated, just mentioning it. Zitat von Eugeniusz Pokora <ceph-users(a)ceph.io>: > When upgrading Ceph cluster from version 19.2.4 to 19.2.6 using > cephadm, an issue occurred with encrypted OSDs. > > 2026-09-02 10:53:28,494 7f24e08141c0 DEBUG Determined image: > 'quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089' > 2026-09-02 10:53:28,501 7f24e08141c0 INFO Reconfig daemon osd.36 ... > 2026-09-02 10:53:29,885 7f24e08141c0 INFO Stopping osd.36 to update > osd_key bluestore label > 2026-09-02 10:53:31,670 7f24e08141c0 INFO Opening osd.36 with crypsetup > 2026-09-02 10:53:33,955 7f24e08141c0 INFO Non-zero exit code 5 from > /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host > --entrypoint /tmp/cryptsetup_action.sh --privileged --group-add=disk > --init -e > CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 -e NODE_NAME=k08r06cph03 -v /dev:/dev -v /tmp/cephadm-osd-36-rotate-helper.sh:/tmp/cryptsetup_action.sh > quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 > 2026-09-02 10:53:33,956 7f24e08141c0 INFO /usr/bin/podman: stderr > Cannot use device > /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or > mounted). > 2026-09-02 10:53:33,956 7f24e08141c0 ERROR ERROR: Got error rotating > osd keyring while using cryptsetup tool > Out: > Err:Cannot use device > /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or > mounted). > > The issue did not occur on all hosts in the cluster (fortunately, it > was a test env), but it affected all OSDs on affected host. After > error occurred OSDs remained in a down state and had to be up > manually. > After several attempts, the update was successfully completed, but > we had to rotate the OSD keyrings ourselves. > A few hours latter when monmap was changed and cephadm scheduled a > reconfig this problem happened again, on 2 out of 4 nodes of the > cluster all osds goes down due to cephadm reconfig and this same > error: "Cannot use device... which is in use (already mapped or > mounted)". > > Has anyone encountered similar problems? > > Regards > Eugeniusz P

1 0

Re: Upgrade 19.2.6 to 20.2.4 changes auth_cipher config
by Patrick Donnelly 04 Sep '26

04 Sep '26

On Fri, Sep 4, 2026 at 3:06 AM Eugen Block <ceph-users(a)ceph.io> wrote: > > Oops, I hit send a bit too early. > > I wanted to ask if this is by design or if it's worth a bug report. It looks like a bug. Please make a ticket. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

19.2.7 QE validation status
by Yuri Weinstein 04 Sep '26

04 Sep '26

Details of this release are summarized here: https://tracker.ceph.com/issues/78483#note-7 Release Notes (drafts) - https://github.com/ceph/ceph/pull/71539 https://github.com/ceph/ceph.io/pull/1198 LRC/SRC upgrade - TBD (Patrick,Dan / The Core Team) PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! Seeking approvals/reviews for: smoke: Laura rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson, Eric fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/quincy-x: PASSED upgrade/reef-x: PASSED upgrade/telemetry-upgrade: PASSED upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: FAILED (suspect https://tracker.ceph.com/issues/79948) upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED powercycle: PASSED TIA

1 0

Maintenance Notification: Mailing List Migration 9/4
by David Galloway 04 Sep '26

04 Sep '26

Hello, Just a quick heads up that the mailing lists will be migrating back into the Sepia lab on 9/4 at 10am Eastern time. Aside from no longer needing to pay AWS for the service, additional benefits we’ll see from this migration include: * Upgrade from mailman 3.3.1 to 3.3.10. This version includes a task runner that will automatically clean up stuck list subscriptions. This was the main cause for the Web UI hanging and going down over the past few years. * Postgres DB migration from 9.6 to highly available Crunchy DB based on Postgres 16 * Lab-wide SMTP relay * Modern anti-spam stack (postscreen, rspamd) * In-house DKIM signing (mail-tester.com rates us a 8.9/10 score) * Mailing alias (vs. an entire list) support * Proper reverse DNS (helps our spam score) The migration itself shouldn’t take more than 30 minutes. It’s already been tested and everything that can be staged ahead of time is already ready. -- David Galloway Ceph Engineering Labs – Infrastructure Architect david.galloway(a)ibm.com IBM

1 3