- ceph-users - lists.ceph.io

Re: NVMe ARM cluster: throughput plateaus and initial OSD hotspot
by Maged Mokhtar 25 Feb '26

25 Feb '26

i would try simple things first: 1) disable scrub/deepscrub and re-run the test. if this was the cause, you can re-enable scrub again but lower the scrub load 2) disable pg auto scaling, manually set the pg_num yourself on new pool and re-run test. /maged On 25/02/2026 10:47, Lina SADI via ceph-users wrote: > > Hello Ceph users, > > I am currently running performance investigations on a small ARM-based > Ceph cluster (Raspberry Pi + NVMe), and I would really appreciate > feedback from experienced users or developers. We are trying to > understand a non-trivial behavior that appears systematic rather than > random. > > Below is a technical summary of our setup and methodology. I am > attaching the figures directly so that the detailed interpretation can > be derived visually from the data. > > _CLUSTER ARCHITECTURE > _Hardware: > > * > > 3× Raspberry Pi storage nodes > > * > > Each node equipped with an identical 512 GB NVMe device > > * > > Nodes interconnected through a 1 Gb/s Ethernet switch > > * > > 1 additional Raspberry Pi acting as a client on the same network > > Software topology (Ceph Pacific 16.2.15 – default configuration): > > * > > Node 1 (umbriel): MON + MGR + OSD > > * > > Node 2 (elara): MON + MDS + OSD > > * > > Node 3 (kerberos): MON + OSD > > * > > Client node: continuous camera ingestion pipeline writing frames > into a default CephFS pool . > > _EXPERIMENT OBJECTIVE > _Our goal is to establish a stable performance baseline before > injecting failure scenarios (e.g., verifying whether images are lost > during Ceph faults). However, even with identical workloads on an > empty cluster, the system consistently evolves through several > distinct performance phases. > > _WORKLOAD DESCRIPTION > _The workload consists of a simple 2-hour execution of the camera > program (~42 FPS) with a frame size of 512 KB: > > Collected information: > > * > > Ceph metrics: cluster health, PG state, and OSD performance > (commit/apply latency) > > * > > System metrics on all nodes: CPU, memory, disk and network usage, > as well as temperature and Raspberry Pi throttling flags > > * > > Client metrics: latency and throughput > > Metrics were collected in parallel with offline parsing to minimize > perturbation. > > _OBSERVATIONS (see attached figures) > _Across multiple executions, performance does not drift gradually but > instead evolves through three clear plateaus affecting client latency, > client throughput, OSD commit/apply latency, and network traffic, > while the cluster remains HEALTH_OK. CPU usage stays stable, and > memory shows periodic cache-related fluctuations. *The most notable > behavior appears at the disk level*: one NVMe starts with much higher > utilization (~95%) while the others remain around ~40%, then activity > progressively converges until all OSDs stabilize at similar levels. > DVFS and thermal throttling do not explain the phenomenon, as running > experiments with throttling already active still produces the same > three phases. > > Our main questions are therefore: > > * > > *What internal Ceph mechanism could explain these three > performance phases?* > > * > > *Why does one OSD initially receive more load before balancing > occurs?* > > Any pointers on relevant metrics, configuration aspects, or known > behaviors on small ARM-based clusters would be extremely valuable. > > Thank you very much for your time and insights. > > Screenshot 2026-02-23 164425.png > Screenshot 2026-02-23 164440.pngScreenshot 2026-02-24 112112.png > Screenshot 2026-02-23 164517.png > Screenshot 2026-02-23 164536.png > Screenshot 2026-02-23 164552.png > Screenshot 2026-02-23 164602.png > Screenshot 2026-02-23 164624.png > Screenshot 2026-02-23 164658.png > > > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io

1 0

NVMe ARM cluster: throughput plateaus and initial OSD hotspot
by Lina SADI 25 Feb '26

25 Feb '26

Hello Ceph users, I am currently running performance investigations on a small ARM-based Ceph cluster (Raspberry Pi + NVMe), and I would really appreciate feedback from experienced users or developers. We are trying to understand a non-trivial behavior that appears systematic rather than random. Below is a technical summary of our setup and methodology. I am attaching the figures directly so that the detailed interpretation can be derived visually from the data. *CLUSTER ARCHITECTURE*Hardware: - 3× Raspberry Pi storage nodes - Each node equipped with an identical 512 GB NVMe device - Nodes interconnected through a 1 Gb/s Ethernet switch - 1 additional Raspberry Pi acting as a client on the same network Software topology (Ceph Pacific 16.2.15 – default configuration): - Node 1 (umbriel): MON + MGR + OSD - Node 2 (elara): MON + MDS + OSD - Node 3 (kerberos): MON + OSD - Client node: continuous camera ingestion pipeline writing frames into a default CephFS pool . *EXPERIMENT OBJECTIVE*Our goal is to establish a stable performance baseline before injecting failure scenarios (e.g., verifying whether images are lost during Ceph faults). However, even with identical workloads on an empty cluster, the system consistently evolves through several distinct performance phases. *WORKLOAD DESCRIPTION*The workload consists of a simple 2-hour execution of the camera program (~42 FPS) with a frame size of 512 KB: Collected information: - Ceph metrics: cluster health, PG state, and OSD performance (commit/apply latency) - System metrics on all nodes: CPU, memory, disk and network usage, as well as temperature and Raspberry Pi throttling flags - Client metrics: latency and throughput Metrics were collected in parallel with offline parsing to minimize perturbation. *OBSERVATIONS (see attached figures)*Across multiple executions, performance does not drift gradually but instead evolves through three clear plateaus affecting client latency, client throughput, OSD commit/apply latency, and network traffic, while the cluster remains HEALTH_OK. CPU usage stays stable, and memory shows periodic cache-related fluctuations. *The most notable behavior appears at the disk level*: one NVMe starts with much higher utilization (~95%) while the others remain around ~40%, then activity progressively converges until all OSDs stabilize at similar levels. DVFS and thermal throttling do not explain the phenomenon, as running experiments with throttling already active still produces the same three phases. Our main questions are therefore: - *What internal Ceph mechanism could explain these three performance phases?* - *Why does one OSD initially receive more load before balancing occurs?* Any pointers on relevant metrics, configuration aspects, or known behaviors on small ARM-based clusters would be extremely valuable. Thank you very much for your time and insights. [image: Screenshot 2026-02-23 164425.png] [image: Screenshot 2026-02-23 164440.png][image: Screenshot 2026-02-24 112112.png] [image: Screenshot 2026-02-23 164517.png] [image: Screenshot 2026-02-23 164536.png] [image: Screenshot 2026-02-23 164552.png] [image: Screenshot 2026-02-23 164602.png] [image: Screenshot 2026-02-23 164624.png] [image: Screenshot 2026-02-23 164658.png]

1 0

Broken dashboard after upgrade to 20.2.0
by Nicola Mori 25 Feb '26

25 Feb '26

Dear Ceph users, I just successfully upgraded my cehpadm-managed cluster to 20.2.0, but the status reports the following error: Module 'dashboard' has failed: No module named 'smb' I upgraded using my self-built Ceph image based on Ubuntu 22.04 and making use of the official Ceph .deb packages, and I noticed that the pysmb package was actually not present. So I rebuilt another image in which I install it with sudo pip3 install pysmb (since I found no .deb), and this should have fixed the problem since from a shell in a container I can launch python3 and load the smb module. But still I get the dashboard error, even after restarting the managers, and the dashboard is unreachable. Any hint about how can I fix this? Thank you, Nicola

1 2

metrics duplicate in prometheus after migrate and upgrade to cephadm
by Francesco Usseglio Gaudi 24 Feb '26

24 Feb '26

Hi all, we have a 10 nodes cluster used for S3 with radosgw. Initially the cluster has been installed with ceph-ansible and was at version 18. Some months ago an external consultant converted to cephadm and upgrade to 19.2.3 Almost everything is ok but we have some issues with dashboard and grafana graphs. In Dashboard → Cluster → OSDs → Overall Performance we can't see "OSD Read Latencies" and "OSD Write Latencies" graphs, they have red triangle icons with this error popup: "execution: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{ceph_daemon=\"osd.0\", cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\", instance=\"ceph_cluster\", job=\"ceph\"}, {ceph_daemon=\"osd.0\", cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\", instance=\"node1\", job=\"ceph-exporter\"}];many-to-many matching not allowed: matching labels must be unique on one side" After some my investigations I found out there are some duplicate metrics in prometheus db, one coming from ceph (I think mgr module?) and one from ceph-exporter, for example this metric: ceph_osd_op_r_latency_sum be there with two distinct values with different job labels (values are exactly the same) Every node has a running ceph-exporter container and there are 3 nodes with mgr services. I have tried to modify grafana dashboard queries to use only one of two metrics specifying the job label but they are readonly (I can make a copy and modifying it that works, but i don't know how to use this new grafana dashboard in ceph dashboard). I have read from ceph documentation and I understand ceph-exporter is no longer required but on another ceph cluster recently set up with cephadm version 19.2.3 from the very start, ceph-exporter container are presents on every nodes, so I'm not sure if I should keep ceph-exporter containers running Which is the right way of extract these metrics from ceph? Thank you all Francesco Usseglio

1 0

Re: since squid (19.2.3) orch puts OSD block.db and data on same device
by Adrien Georget 24 Feb '26

24 Feb '26

Hi Boris, The workaround we used to replace disks affected by this bug is to force devices in the osd creation using the orchestrator without yaml specs. VG and LV are first created manually and then we used something like this : |ceph orch daemon add osd <host>:data_devices=/dev/data01,db_devices=/dev/fast01/db01| See the advanced OSD creation : https://docs.ceph.com/en/reef/cephadm/services/osd/#creating-new-osds Adrien Le 24/02/2026 à 14:57, Boris via ceph-users a écrit : > Thanks Marek, but this did not work here, because the SSDs still have other > block.db LVs on it. > > > Am Di., 24. Feb. 2026 um 14:46 Uhr schrieb Marek Szuba via ceph-users < > ceph-users(a)ceph.io>: > >> On 2026-02-24 11:15, Robert Sander via ceph-users wrote: >> >>> The orchestrator in Ceph 19 and 20 has a bug with hybrid OSDs: >>> >>> https://tracker.ceph.com/issues/72696 >>> >>> There is a patch available but it has not been merged yet. >> Meanwhile, a workaround that has worked for us is: >> * tell the orchestrator to delete unwanted non-hybrid OSDs _without >> zapping the drives_; >> * manually wipe both underlying devices, with as little time in >> between the two as possible; >> * explicitly tell the orchestrator to refresh its device list, so that >> it sees both available drives at the same time. >> >> -- >> MS >> _______________________________________________ >> ceph-users mailing list --ceph-users(a)ceph.io >> To unsubscribe send an email toceph-users-leave(a)ceph.io >> >

2 1

Save the date(s) - 2026 Ceph Meetup in Berlin, Germany and Frankfurt/Main, Germany - interested people are welcome !
by Matthias Muench 24 Feb '26

24 Feb '26

Hi people in Germany, we run two meetup groups in Berlin and Frankfurt with meetings nearly quarterly. For each location, we provide also a quick introduction to Ceph before the main event. Currently, we have a new set of appointments. You can join us by RSVP to the events - check out the meetup groups: - Ceph Berlin meetup: https://www.meetup.com/ceph-berlin/ - Ceph Frankfurt meetup: https://www.meetup.com/ceph-frankfurt/ What are the next dates ? ( March set, the other meetups are in planning for now - but the dates are likely those) *Ceph Berlin: * - 02.03.2026 - 02.06.2026 - 14.09.2026 - 30.11.2026 * Ceph Frankfurt: * - 12.03.2026 - 24.06.2026 - 09.09.2026 - 01.12.2026 Any suggestions ? Meet us in person or on the meetup group pages - you can open a discussion and see what people think about your idea. Happy to see you soon, -matt -- —————————————————— Matthias Muench Principal Specialist Solution Architect, OpenShift EMEA Storage Specialist matthias.muench(a)redhat.com Phone: +49-160-92654111 Red Hat GmbH Technopark II Werner-von-Siemens-Ring 12 85630 Grasbrunn Germany _______________________________________________________________________ Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross

1 0

Re: since squid (19.2.3) orch puts OSD block.db and data on same device
by Boris 24 Feb '26

24 Feb '26

Thanks Marek, but this did not work here, because the SSDs still have other block.db LVs on it. Am Di., 24. Feb. 2026 um 14:46 Uhr schrieb Marek Szuba via ceph-users < ceph-users(a)ceph.io>: > On 2026-02-24 11:15, Robert Sander via ceph-users wrote: > > > The orchestrator in Ceph 19 and 20 has a bug with hybrid OSDs: > > > > https://tracker.ceph.com/issues/72696 > > > > There is a patch available but it has not been merged yet. > > Meanwhile, a workaround that has worked for us is: > * tell the orchestrator to delete unwanted non-hybrid OSDs _without > zapping the drives_; > * manually wipe both underlying devices, with as little time in > between the two as possible; > * explicitly tell the orchestrator to refresh its device list, so that > it sees both available drives at the same time. > > -- > MS > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: since squid (19.2.3) orch puts OSD block.db and data on same device
by Marek Szuba 24 Feb '26

24 Feb '26

On 2026-02-24 11:15, Robert Sander via ceph-users wrote: > The orchestrator in Ceph 19 and 20 has a bug with hybrid OSDs: > > https://tracker.ceph.com/issues/72696 > > There is a patch available but it has not been merged yet. Meanwhile, a workaround that has worked for us is: * tell the orchestrator to delete unwanted non-hybrid OSDs _without zapping the drives_; * manually wipe both underlying devices, with as little time in between the two as possible; * explicitly tell the orchestrator to refresh its device list, so that it sees both available drives at the same time. -- MS

1 0

Re: since squid (19.2.3) orch puts OSD block.db and data on same device
by Boris 24 Feb '26

24 Feb '26

Ah nice. Thanks Robert! Am Di., 24. Feb. 2026 um 12:16 Uhr schrieb Robert Sander via ceph-users < ceph-users(a)ceph.io>: > Am 24.02.26 um 12:04 PM schrieb Boris via ceph-users: > > Hi, > > we've upgraded from reef to squid lately and are encountering the problem > > that the orchestrator now puts the block.db together with the data on the > > same HDD. > > > > We've set the orch config to use non rotational devices for the block.db > > and the rotational devices for the data. We use one SSD/NVMe for multiple > > HDDs. > > The orchestrator in Ceph 19 and 20 has a bug with hybrid OSDs: > > https://tracker.ceph.com/issues/72696 > > There is a patch available but it has not been merged yet. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

network ring recommendations wanted
by Kai Zimmer 24 Feb '26

24 Feb '26

Dear listers, for cluster network i have currently 3x Mikrotik switches CRS518-16XS-2XQ (connected via 100 GBit) and for the "public" connection to clients via 2x10 GBit i have 5x Mikrotik switches CRS326-24S+2Q+RM (connected via 40 GBit) available. Looking at the hardware to me it seems there is only oen option to build network rings. MLAG seems to be supported only for up to two devices (not enough for me), RSTP seems unsupported. I read about people using MSTP (which is supported) but i'm unsure if it is suitable for ceph needs. I'm not talking about a high performance setup here, rather than file server and virtualization needs. Any thoughts/comments? Best regards, Kai

1 0