- ceph-users - lists.ceph.io

Cephfs mds node already exists crashes mds
by Tarrago, Eli (RIS-BCT) 20 Aug '24

20 Aug '24

Good Morning Ceph Users, I’m currently engaged in troubleshooting an issue and I wanted to post here to get some feedback. If there is no response or feedback that this looks like a bug, then I’ll write up a bug report. Cluster: Reef 18.2.4 Ubuntu 20.04 ceph -s cluster: id: 93e49b2e-2b56-4faa-xxxx-88ceb615daac health: HEALTH_WARN 1 filesystem is degraded 49 large omap objects 1 MDSs report oversized cache 1 MDSs report slow requests 152 pgs not deep-scrubbed in time 14 daemons have recently crashed services: mon: 3 daemons, quorum mon01,mon03,mon02 (age 2w) mgr: mgr01(active, since 2w), standbys: mgr02, mgr03 mds: 16/16 daemons up, 1 standby, 1 hot standby osd: 170 osds: 167 up (since 3d), 165 in (since 3d) rgw: 3 daemons active (3 hosts, 1 zones) The error message received when mds crashes are: Aug 18 09:14:32 node02 ceph-mds[4098736]: log_channel(cluster) log [ERR] : loaded dup inode 0x100009e4175 [2,head] v12 at /volume/db2/RDP/sql02/innodb/20240817.103010/idm_dw/tmp/text-base, but inode 0x100009e417> Aug 18 09:14:32 node02 ceph-mds[4098736]: log_channel(cluster) log [ERR] : loaded dup inode 0x100009e4176 [2,head] v10 /volume/db2/RDP/sql02/innodb/20240817.103010/idm_dw/tmp/props, but inode 0x100009e4176.he> Current status of cephfs for this volume ceph fs status cephfs - 93 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 clientreplay mds0206 126M 126M 123k 73 1 active mds0102 Reqs: 0 /s 10 13 11 0 2 active mds0204 Reqs: 0 /s 12.9k 12.9k 63.8k 58 3 active mds0203 Reqs: 0 /s 12.9k 12.9k 63.9k 56 4 active mds0207 Reqs: 0 /s 12.7k 12.7k 63.8k 7 5 active mds0201 Reqs: 0 /s 12.8k 12.8k 63.8k 3 6 active mds0106 Reqs: 0 /s 12.8k 12.8k 63.8k 9 7 active mds0105 Reqs: 0 /s 12.8k 12.8k 63.8k 4 8 active mds0205 Reqs: 0 /s 12.4k 12.4k 61.7k 1 9 active mds0208 Reqs: 0 /s 12.5k 12.5k 61.7k 0 10 active mds0107 Reqs: 0 /s 12.5k 12.5k 61.7k 1 11 active mds0104 Reqs: 0 /s 12.4k 12.5k 61.7k 0 12 active mds0101 Reqs: 0 /s 12.5k 12.5k 61.7k 0 13 active mds0103 Reqs: 0 /s 12.5k 12.5k 61.7k 0 14 active mds0108 Reqs: 0 /s 12.4k 12.4k 61.7k 0 POOL TYPE USED AVAIL cephfs_metadata metadata 232G 325T cephfs_data data 439T 325T This error cause mds0206, rank0 to restart. This happened 14 times in the last 48 hours. ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

1 0

CephFS troubleshooting
by Eugenio Tampieri 20 Aug '24

20 Aug '24

Hello, I'm writing to troubleshoot an otherwise functional Ceph quincy cluster that has issues with cephfs. I cannot mount it with ceph-fuse (it gets stuck), and if I mount it with NFS I can list the directories but I cannot read or write anything. Here's the output of ceph -s cluster: id: 3b92e270-1dd6-11ee-a738-000c2937f0ec health: HEALTH_WARN mon ceph-storage-a is low on available space 1 daemons have recently crashed too many PGs per OSD (328 > max 250) services: mon: 5 daemons, quorum ceph-mon-a,ceph-storage-a,ceph-mon-b,ceph-storage-c,ceph-storage-d (age 105m) mgr: ceph-storage-a.ioenwq(active, since 106m), standbys: ceph-mon-a.tiosea mds: 1/1 daemons up, 2 standby osd: 4 osds: 4 up (since 104m), 4 in (since 24h) rbd-mirror: 2 daemons active (2 hosts) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 13 pools, 481 pgs objects: 231.83k objects, 648 GiB usage: 1.3 TiB used, 1.8 TiB / 3.1 TiB avail pgs: 481 active+clean io: client: 1.5 KiB/s rd, 8.6 KiB/s wr, 1 op/s rd, 0 op/s wr Best regards, Eugenio Tampieri

2 1

The snaptrim queue of PGs has not decreased for several days.
by Giovanna Ratini 20 Aug '24

20 Aug '24

Hello all, We use Ceph (v18.2.2) and Rook (1.14.3) as the CSI for a Kubernetes environment. Last week, we had a problem with the MDS falling behind on trimming every 4-5 days (GitHub issue link). We resolved the issue using the steps outlined in the GitHub issue. We have 3 hosts (I know, I need to increase this as soon as possible, and I will!) and 6 OSDs. After running the commands: ceph config set mds mds_dir_max_commit_size 80, ceph fs fail <fs_name>, and ceph fs set <fs_name> joinable true, After that, the snaptrim queue for our PGs has stopped decreasing. All PGs of our CephFS are in either active+clean+snaptrim_wait or active+clean+snaptrim states. For example, the PG 3.12 is in the active+clean+snaptrim state, and its snap_trimq_len was 4077 yesterday but has increased to 4538 today. I increased the osd_snap_trim_priority to 10 (ceph config set osd osd_snap_trim_priority 10), but it didn't help. Only the PGs of our CephFS have this problem. Do you have any ideas on how we can resolve this issue? Thanks in advance, Giovanna p.s. I'm not a ceph expert :-). Faulkener asked me for more information, so here it is: MDS Memory: 11GB mds_cache_memory_limit: 11,811,160,064 bytes root@kube-master02:~# ceph fs snap-schedule status / { "fs": "rook-cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "3h", "retention": {"h": 24, "w": 4}, "start": "2024-05-05T00:00:00", "created": "2024-05-05T17:28:18", "first": "2024-05-05T18:00:00", "last": "2024-08-15T18:00:00", "last_pruned": "2024-08-15T18:00:00", "created_count": 817, "pruned_count": 817, "active": true } I do not understand if the snapshots in the PGs are correlated with the snapshots on CephFS. Until we encountered the issue with the "MDS falling behind on trimming every 4-5 days," we didn't have any problems with snapshots. Could someone explain me this or send me to the documentation? Thank you

3 15

Bug with Cephadm module osd service preventing orchestrator start
by Benjamin Huth 20 Aug '24

20 Aug '24

Hey there, so I went to upgrade my ceph from 18.2.2 to 18.2.4 and have encountered a problem with my managers. After they had been upgraded, my ceph orch module broke because the cephadm module would not load. This obviously halted the update because you can't really update without the orchestrator. Here are the logs related to why the cephadm module fails to start: https://pastebin.com/SzHbEDVA and the relevent part here: "backtrace": [ " File \\"/usr/share/ceph/mgr/cephadm/module.py\\", line 591, in __init__\\n self.to_remove_osds.load_from_store()", " File \\"/usr/share/ceph/mgr/cephadm/services/osd.py\\", line 918, in load_from_store\\n osd_obj = OSD.from_json(osd, rm_util=self.rm_util)", " File \\"/usr/share/ceph/mgr/cephadm/services/osd.py\\", line 783, in from_json\\n return cls(**inp)", "TypeError: __init__() got an unexpected keyword argument 'original_weight'" ] Unfortunately, I am at a loss to what passes this the original weight argument. I have attempted to migrate back to 18.2.2 and successfully redeployed a manager of that version, but it also has the same issue with the cephadm module. I believe this may be because I recently started several OSD drains, then canceled them, causing this to manifest once the managers restarted. I went through a good bit of the source and found the module at fault: https://github.com/ceph/ceph/blob/e0dd396793b679922e487332a2a4bc48e024a42f/… as well as the PR that caused the issue: https://github.com/ceph/ceph/commit/ba7fac074fb5ad072fcad10862f75c0a26a7591d I unfortunately am not familiar enough with the ceph source to find the ceph-config values I need to delete or smart enough to fix this myself. Any help would be super appreciated. Thanks!

2 7

Prometheus and "404" error on console
by Tim Holloway 19 Aug '24

19 Aug '24

Although I'm seeing this in Pacific, it appears to be a perennial issue with no well-documented solution. The dashboard home screen is flooded with popups saying "404 - Not Found Could not reach Prometheus's API on http://ceph1234.mydomain.com:9095/api/v1 " If I was a slack-jawed PHB casually wandering into the operations center and saw that, I'd probably doubt that Ceph was a good product decision. The "404" indicates that the Prometheus server is running and accepting requests, but all the "404" says is that whatever requests it's receiving are meaningless to it. Presumably either some handler needs to be jacked into Prometheus or the sender needs to be notified to desist. Unfortunately, "404" doesn't provide any clues. Ideally, ceph should log something more specific by default, but failing that, if anyone knows how to shut it up (cleanly), I'd appreciate knowing! Tim

2 2

CLT meeting notes August 19th 2024
by Adam King 19 Aug '24

19 Aug '24

- [travisn] Arm64 OSDs crashing on v18.2.4, need a fix in v18.2.5 - https://tracker.ceph.com/issues/67213 - tcmalloc issue, solved by rebuilding the gperftools package - Travis to reach out to Rongqi Sun about the issue - moving away from tcmalloc would probably cause performance regressions and is therefore undesirable - [zdover] - ceph-users list moderation minute - the whole Python mess seems to be fixed. Release branches on docs.ceph.com are now updating. - [venky] - CVE testing process/documentation - https://docs.ceph.com/en/reef/dev/release-process/?highlight=ceph%20release… - does not include the testing procedure - currently, even when pointed towards branch on private repo, tries to pull packages from ceph-ci - Venky and Zac Dover to coordinate on creating the docs - should this documentation be publicly available? - only once CI/build issues that require the builds being in public places are fixed - just in a google doc for now - [Eric I] - How do people feel that Read the Docs is super-imposing ads over our docs? So far I've only seen one for www.ethicalads.io . - comes with Read the Docs, not enough of an issue to justify moving away from it - [DanV] - Cephalocon CFP is closed : 175 submissions! (for around 50 sessions) - Program Committee reviewing until the end of August. - On-site Dev Summit being planned for Dec 3, will go live here: https://indico.cern.ch/e/ceph-developer-summit - 3-4 rooms at CERN IT for dev whiteboarding. power users session in the afternoon. - If you still want to submit a CFP, talk to Dan ASAP - still looking for additional sponsors - 19.1.1 - almost ready, need ceph-volume approval and Gibba upgrade - 17.2.8 - mark PRs if you need them in the release - Yuri to start testing soon - core team to potentially do scrub of backport PRs

1 0

cephadm module fails to load with "got an unexpected keyword argument"
by Alex Sanderson 19 Aug '24

19 Aug '24

Hi everyone, I recently upgraded from Quincy to Reef v18.2.4 and my dashboard and mgr systems have been broken since. Since the upgrade I was slowly removing and zapping osd's that still had the 64k "bluestore_bdev_block_size" and decided to have a look at the dashboard problem. I restarted the mgrs one at a time and they showed in status that they working but actually the cephadm module was failing. The systems were all upgraded to 18 via orch from 17.2.7 and are running the official docker images. This is the error message: debug 2024-08-13T10:08:11.736+0000 7fd30dbe0640 -1 mgr load Failed to construct class in 'cephadm' debug 2024-08-13T10:08:11.736+0000 7fd30dbe0640 -1 mgr load Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 619, in __init__ self.to_remove_osds.load_from_store() File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 924, in load_from_store osd_obj = OSD.from_json(osd, rm_util=self.rm_util) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 789, in from_json return cls(**inp) TypeError: __init__() got an unexpected keyword argument 'original_weight' debug 2024-08-13T10:08:11.736+0000 7fd30dbe0640 -1 mgr operator() Failed to run module in active mode ('cephadm') The config-key responsible was mgr/cephadm/osd_remove_queue This is what it looked like before. After removing the original_weight field and setting the variable again, the cephadm module loads and orch works. It seems like a bug. [{"osd_id": 89, "started": true, "draining": true, "stopped": false, "replace": false, "force": true, "zap": true, "hostname": "goanna", "original_weight": 0.930999755859375, "drain_started_at": "2024-08-12T13:21:04.458019Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2024-08-12T13:20:40.021185Z"}, {"osd_id": 37, "started": true, "draining": true, "stopped": false, "replace": false, "force": true, "zap": true, "hostname": "gsceph1osd05", "original_weight": 4, "drain_started_at": "2024-08-10T06:30:37.569931Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2024-08-10T06:30:19.729143Z"}, {"osd_id": 47, "started": true, "draining": true, "stopped": false, "replace": false, "force": true, "zap": true, "hostname": "gsceph1osd07", "original_weight": 4, "drain_started_at": "2024-08-10T09:54:49.132830Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2024-08-10T09:54:34.367655Z"}] I thought I should put this out there in case anyone else was having a weird issue with a keyword argument problem. It did not fix the problem with the dashboard, still working on that. Alex

2 1

memory leak in mds?
by Dario Graña 19 Aug '24

19 Aug '24

Hi all, We’re experiencing an issue with CephFS. I think we are facing this issue <https://tracker.ceph.com/issues/64977>. The main symptom is that the MDS starts using a lot of memory within a few minutes and finally it gets killed by OS (Out Of Memory). Sometimes it happens once a week and sometimes 2 times a day. We are running ceph quincy 17.2.7 on both the cluster and clients. I have read through some emails on the mailing list about it, but I didn't find a workaround. Does anyone have any suggestions? Thanks in advance. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

3 4

Re: ceph device ls missing disks
by Alfredo Rezinovsky 18 Aug '24

18 Aug '24

No, I don´t monitor drivers in non-CEPH nodes. All my non-CEPH nodes are disposables. Only CEPH has data I can't lose El jue, 15 ago 2024 a las 20:49, Anthony D'Atri (<anthony.datri(a)gmail.com>) escribió: > Do you monitor OS drives on non-Ceph nodes? > > > > > On Aug 15, 2024, at 8:17 AM, Alfredo Rezinovsky <alfrenovsky(a)gmail.com> > wrote: > > > > With: "ceph device ls" I can see which physical disks are used by which > > daemons. > > > > OSD data and wal disks are ok. > > > > MON appears using operating system disks, or the disks where > /var/lib/ceph > > is mounted, also OK. > > > > The problem is that non monitor nodes operating system disks are missing > > when they are really being used by all the daemons in the node. > > > > The problem with this is that there are no monitoring for those disks > when > > they are essential and failure points for all the daemons running in > those > > nodes. > > > > I think the OS devices should appear used by all the daemons having > > essential information in those disks. > > > > -- > > Alfrenovsky > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Alfrenovsky

1 0

Identify laggy PGs
by Boris 18 Aug '24

18 Aug '24

Hi, currently we encouter laggy PGs and I would like to find out what is causing it. I suspect it might be one or more failing OSDs. We had flapping OSDs and I synced one out, which helped with the flapping, but it doesn't help with the laggy ones. Any tooling to identify or count PG performance and map that to OSDs? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

5 10