How is your pg distribution on your osd devices? Do you have enough assigned pgs? Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- On 2023. Jan 27., at 23:30, Victor Rodriguez &lt;vrodriguez(a)soltecsis.com&gt; wrote: Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ Ah yes, checked that too. Monitors and OSD's report with ceph config show-with-defaults that bluefs_buffered_io is set to true as default setting (it isn't overriden somewere). On 1/27/23 17:15, Wesley Dillingham wrote: I hit this issue once on a nautilus cluster and changed the OSD parameter bluefs_buffered_io = true (was set at false). I believe the default of this parameter was switched from false to true in release 14.2.20, however, perhaps you could still check what your osds are configured with in regard to this config item. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Fri, Jan 27, 2023 at 8:52 AM Victor Rodriguez &lt;vrodriguez(a)soltecsis.com&gt; wrote: Hello, Asking for help with an issue. Maybe someone has a clue about what's going on. Using ceph 15.2.17 on Proxmox 7.3. A big VM had a snapshot and I removed it. A bit later, nearly half of the PGs of the pool entered snaptrim and snaptrim_wait state, as expected. The problem is that such operations ran extremely slow and client I/O was nearly nothing, so all VMs in the cluster got stuck as they could not I/O to the storage. Taking and removing big snapshots is a normal operation that we do often and this is the first time I see this issue in any of my clusters. Disks are all Samsung PM1733 and network is 25G. It gives us plenty of performance for the use case and never had an issue with the hardware. Both disk I/O and network I/O was very low. Still, client I/O seemed to get queued forever. Disabling snaptrim (ceph osd set nosnaptrim) stops any active snaptrim operation and client I/O resumes back to normal. Enabling snaptrim again makes client I/O to almost halt again. I've been playing with some settings: ceph tell 'osd.*' injectargs '--osd-max-trimming-pgs 1' ceph tell 'osd.*' injectargs '--osd-snap-trim-sleep 30' ceph tell 'osd.*' injectargs '--osd-snap-trim-sleep-ssd 30' ceph tell 'osd.*' injectargs '--osd-pg-max-concurrent-snap-trims 1' None really seemed to help. Also tried restarting OSD services. This cluster was upgraded from 14.2.x to 15.2.17 a couple of months. Is there any setting that must be changed which may cause this problem? I have scheduled a maintenance window, what should I look for to diagnose this problem? Any help is very appreciated. Thanks in advance. Victor _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io -- _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

After some investigation this is what I'm seeing: - OSD processes get stuck at least at 100% CPU if I ceph osd unset nosnaptrim. They keep at 100% CPU even if I ceph osd set nosnaptrim. They stayed like that for at least 26 hours. Some quick benchmarks don't show a reduction of the performance of the cluster. - Restarting a OSD lowers it's CPU usage to typical levels, as expected, but it also usually sets some other OSD in a different host to typical levels. - All OSDs in this cluster take quite a bit to start: between 35 to 70 seconds depending on the OSD. Clearly much longer than any other OSD in any of my clusters. - I believe that the size of the rocksdb database is dumped in the OSD log when an automatic compact operation is triggered. The "sum" sizes of these OSD range between 2.5 and 5.1 GB. Thats way bigger that those in any other cluster I have. - ceph daemon osd.* calc_objectstore_db_histogram is giving values for num_pgmeta_omap (I don't know what it is) way bigger than those on any other of my clusters for some OSD. Also, values are not similar among the OSD which hold the same PGs. osd.0:    "num_pgmeta_omap": 17526766, osd.1:    "num_pgmeta_omap": 2653379, osd.2:    "num_pgmeta_omap": 12358703, osd.3:    "num_pgmeta_omap": 6404975, osd.6:    "num_pgmeta_omap": 19845318, osd.7:    "num_pgmeta_omap": 6043083, osd.12:   "num_pgmeta_omap": 18666776, osd.13:    "num_pgmeta_omap": 615846, osd.14:    "num_pgmeta_omap": 13190188, - Compacting the OSD barely reduces rocksdb size and does not reduce num_pgmeta_omap at all. - This is the only cluster I have were there are some RBD images that I mount directly from some clients, that is, they are not disks for QEMU/Proxmox VMs. Maybe I have something misconfigured related to this?  This cluster is at least two and half years old an never had this issue with snaptrims. Thanks in advance! On 1/27/23 17:29, Victor Rodriguez wrote: --

On 1/29/23 00:50, Matt Vandermeulen wrote: That's an interesting approach! Thanks! On preliminary tests seems that just running snaptrim on a single PG of a single OSD still makes the cluster barely usable. I have to increase osd_snap_trim_sleep_ssd to ~1 so the cluster remains usable by getting a third of its performance. After a while, a few PG got trimmed and feels like some of them are harder to trim than others, as some need a higher osd_snap_trim_sleep_ssd value to let the cluster perform. I don't know how long this is going to take... Maybe recreating the OSD's and dealing with the rebalance is a better option? There's something ugly going on here... I would really like to put my finger on it.

Looks like this is going to take a few days. I hope to manage the available performance for VMs with osd_snap_trim_sleep_ssd. I'm wondering if after that long snaptrim process you went through, was your cluster was stable again and snapshots/snaptrims did work properly? On 1/29/23 16:01, Matt Vandermeulen wrote: --

I'm reading that thread on spinics. Thanks for pointing that out. The cluster was upgraded from Nautilus to Octopus first week of December. No previous version has been used in this cluster. I just don't remember if we used snapshots after the upgrade. bluestore_fsck_quick_fix_on_mount is at default of "true". Did not notice anything unusual during or after the upgrade. Is there any way to check if the OMAP conversion is done for an OSD? Maybe it tries to do the conversion every time I restart an OSD and fails? (given they take nearly a minute to start). I still have 13 PGs pending snaptrims and client I/O is severely affected even while doing on OSD at a time and osd_snap_trim_sleep_ssd=5 :\ On 1/30/23 09:23, Frank Schilder wrote: --

Hi, Josh already suggested, but I will one more time. We had similar behaviour upgrading from Nautilus to Pacific. In our case compacting the OSDs did the trick. For us there was no performance impact running the compaction (ceph osd daemon osd.0 compact) although we run them in batches and not all at once on all OSDs just in case. Also, no need to restart OSDs for this operation. https://www.spinics.net/lists/ceph-users/msg74774.html Regards, Ana On 30-01-2023 10:23, Victor Rodriguez wrote: > I'm reading that thread on spinics. Thanks for pointing that out. > > The cluster was upgraded from Nautilus to Octopus first week of > December. No previous version has been used in this cluster. I just > don't remember if we used snapshots after the upgrade. > bluestore_fsck_quick_fix_on_mount is at default of "true". Did not > notice anything unusual during or after the upgrade. > > Is there any way to check if the OMAP conversion is done for an OSD? > Maybe it tries to do the conversion every time I restart an OSD and > fails? (given they take nearly a minute to start). I still have 13 PGs > pending snaptrims and client I/O is severely affected even while doing > on OSD at a time and osd_snap_trim_sleep_ssd=5 :\ > > > On 1/30/23 09:23, Frank Schilder wrote: >> Hi Victor, >> >> out of curiosity, did you upgrade the cluster recently to octopus? We >> and others observed this behaviour when following one out of two >> routes to upgrade OSDs. There was a thread "Octopus OSDs extremely >> slow during upgrade from mimic", which seems to have been lost with >> the recent mail list outage. If it is relevant, I could copy pieces I >> have into here. >> >> Best regards, >> ================= >> Frank Schilder >> AIT Risø Campus >> Bygning 109, rum S14 >> >> ________________________________________ >> From: Victor Rodriguez &lt;vrodriguez(a)soltecsis.com&gt; >> Sent: 29 January 2023 22:40:46 >> To: ceph-users(a)ceph.io >> Subject: [ceph-users] Re: Very slow snaptrim operations blocking >> client I/O >> >> Looks like this is going to take a few days. I hope to manage the >> available performance for VMs with osd_snap_trim_sleep_ssd. >> >> I'm wondering if after that long snaptrim process you went through, was >> your cluster was stable again and snapshots/snaptrims did work properly? >> >> >> On 1/29/23 16:01, Matt Vandermeulen wrote: >>> I should have explicitly stated that during the recovery, it was still >>> quite bumpy for customers.  Some snaptrims were very quick, some took >>> what felt like a really long time.  This was however a cluster with a >>> very large number of volumes and a long, long history of snapshots. >>> I'm not sure what the difference will be from our case versus a single >>> large volume with a big snapshot. >>> >>> >>> >>> On 2023-01-28 20:45, Victor Rodriguez wrote: >>>> On 1/29/23 00:50, Matt Vandermeulen wrote: >>>>> I've observed a similar horror when upgrading a cluster from >>>>> Luminous to Nautilus, which had the same effect of an overwhelming >>>>> amount of snaptrim making the cluster unusable. >>>>> >>>>> In our case, we held its hand by setting all OSDs to have zero max >>>>> trimming PGs, unsetting nosnaptrim, and then slowly enabling >>>>> snaptrim a few OSDs at a time.  It was painful to babysit but it >>>>> allowed the cluster to catch up without falling over. >>>> >>>> That's an interesting approach! Thanks! >>>> >>>> On preliminary tests seems that just running snaptrim on a single PG >>>> of a single OSD still makes the cluster barely usable. I have to >>>> increase osd_snap_trim_sleep_ssd to ~1 so the cluster remains usable >>>> by getting a third of its performance. After a while, a few PG got >>>> trimmed and feels like some of them are harder to trim than others, >>>> as some need a higher osd_snap_trim_sleep_ssd value to let the >>>> cluster perform. >>>> >>>> I don't know how long this is going to take... Maybe recreating the >>>> OSD's and dealing with the rebalance is a better option? >>>> >>>> There's something ugly going on here... I would really like to put my >>>> finger on it. >>>> >>>> >>>>> On 2023-01-28 19:43, Victor Rodriguez wrote: >>>>>> After some investigation this is what I'm seeing: >>>>>> >>>>>> - OSD processes get stuck at least at 100% CPU if I ceph osd unset >>>>>> nosnaptrim. They keep at 100% CPU even if I ceph osd set >>>>>> nosnaptrim. They stayed like that for at least 26 hours. Some quick >>>>>> benchmarks don't show a reduction of the performance of the cluster. >>>>>> >>>>>> - Restarting a OSD lowers it's CPU usage to typical levels, as >>>>>> expected, but it also usually sets some other OSD in a different >>>>>> host to typical levels. >>>>>> >>>>>> - All OSDs in this cluster take quite a bit to start: between 35 to >>>>>> 70 seconds depending on the OSD. Clearly much longer than any other >>>>>> OSD in any of my clusters. >>>>>> >>>>>> - I believe that the size of the rocksdb database is dumped in the >>>>>> OSD log when an automatic compact operation is triggered. The "sum" >>>>>> sizes of these OSD range between 2.5 and 5.1 GB. Thats way bigger >>>>>> that those in any other cluster I have. >>>>>> >>>>>> - ceph daemon osd.* calc_objectstore_db_histogram is giving values >>>>>> for num_pgmeta_omap (I don't know what it is) way bigger than those >>>>>> on any other of my clusters for some OSD. Also, values are not >>>>>> similar among the OSD which hold the same PGs. >>>>>> >>>>>> osd.0:    "num_pgmeta_omap": 17526766, >>>>>> osd.1:    "num_pgmeta_omap": 2653379, >>>>>> osd.2:    "num_pgmeta_omap": 12358703, >>>>>> osd.3:    "num_pgmeta_omap": 6404975, >>>>>> osd.6:    "num_pgmeta_omap": 19845318, >>>>>> osd.7:    "num_pgmeta_omap": 6043083, >>>>>> osd.12:   "num_pgmeta_omap": 18666776, >>>>>> osd.13:    "num_pgmeta_omap": 615846, >>>>>> osd.14:    "num_pgmeta_omap": 13190188, >>>>>> >>>>>> - Compacting the OSD barely reduces rocksdb size and does not >>>>>> reduce num_pgmeta_omap at all. >>>>>> >>>>>> - This is the only cluster I have were there are some RBD images >>>>>> that I mount directly from some clients, that is, they are not >>>>>> disks for QEMU/Proxmox VMs. Maybe I have something misconfigured >>>>>> related to this?  This cluster is at least two and half years old >>>>>> an never had this issue with snaptrims. >>>>>> >>>>>> Thanks in advance! >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> -- >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

An update on this for the record: To fully solve this I've had to destroy each OSD and create then again, one by one. I could have done it one host at a time but I've preferred to be on the safest side just in case something else went wrong. The values for num_pgmeta_omap (which I don't know what it is, yet) on the new OSD were similar to other clusters (I've seen from 300000 to 700000 aprox), so I believe the characteristics of the data in the cluster does not determine how big or small num_pgmeta_omap should be. One thing I've noticed is that /bad/ or /damanged/ OSDs (i.e. those showing high CPU usage and poor performance doing the trim operation) took much more time to calculate their histogram, even if their num_pgmeta_opmap was low: (/bad //OSD)/: # time ceph daemon osd.1 calc_objectstore_db_histogram | grep "num_pgmeta_omap"     "num_pgmeta_omap": 673208, real    1m14,549s user    0m0,075s sys    0m0,025s (/good new OSD/): #  time ceph daemon osd.1 calc_objectstore_db_histogram | grep "num_pgmeta_omap"     "num_pgmeta_omap": 434298, real    0m18,022s user    0m0,078s sys    0m0,023s Maybe is worth checking that histogram from time to time as a way to measure the OSD "health"? Again, thanks everyone. On 1/30/23 18:18, Victor Rodriguez wrote: --