- ceph-users - lists.ceph.io

Re: MDS crash
by Alexey GERASIMOV 27 Apr '24

27 Apr '24

I don't know why, but I miss my topic when I reply to it. Moderators, please delete unnecessary topics and move my answer to the correct topic.

1 0

Re: MDS crash
by Alexey GERASIMOV 27 Apr '24

27 Apr '24

Colleagues, thank you for the advice to check the operability of MGRs. In fact, it is strange also: we checked our nodes for the network issues (ip connectivity, sockets, ACL, DNS) and find nothing wrong - but suddenly just the restart of all MGRs solved the problem with stale PGs and with ceph commands hang! So, we are at the start point again - ceph is working except MDS daemons crash. But now we see some additional errors in MDS logs when try to start the daemon: dir 0x1000dd10fa0 object missing on disk; some files may be lost (/volumes/csi/csi-vol-2eb40f89-f2e1-11ee-b657-3aa98da4c4a6/1080803d-1277-4ad8-ae80-a004bd3a5699/gallery/pc-12083932925583528732) dir 0x1000dd10f9d object missing on disk; some files may be lost (/volumes/csi/csi-vol-2eb40f89-f2e1-11ee-b657-3aa98da4c4a6/1080803d-1277-4ad8-ae80-a004bd3a5699/cadserver-filevault/project-files/661fb14d341d3746ea5c2a8f I promiced to create the bug, so will do it later a bit. But should I try to do something more from my side also? What I did exactly last time: cephfs-journal-tool journal reset cephfs-table-tool all reset session cephfs-data-scan scan_extents cephfs-data-scan scan_inodes cephfs-data-scan scan_links cephfs-data-scan cleanup And one more question: is it possible to access to cephfs content directly, without MDS?

1 0

rbd-mirror get status updates quicker
by Stefan Kooman 27 Apr '24

27 Apr '24

Hi, We're testing with rbd-mirror (mode snapshot) and try to get status updates about snapshots as fast a possible. We want to use rbd-mirror as a migration tool between two clusters and keep downtime during migration as short as possible. Therefore we have tuned the following parameters and set them to 1 second (default 30 seconds): rbd_mirror_pool_replayers_refresh_interval rbd_mirror_image_state_check_interval rbd_mirror_sync_point_update_age However, on the destination cluster, the "last_update:" field is only updated every 30 seconds. Is this tunable? Goal is to determine when the last snapshot that is made on the source has made it to the target and a demote (source) and promote (target) can be initiated. Gr. Stefan

2 1

Re: Add node-exporter using ceph orch
by Robert Sander 26 Apr '24

26 Apr '24

On 4/26/24 15:47, Vahideh Alinouri wrote: > The result of this command shows one of the servers in the cluster, > but I have node-exporter daemons on all servers. The default service specification looks like this: service_type: node-exporter service_name: node-exporter placement: host_pattern: '*' If you apply this YAML code the orchestrator should deploy one node-exporter daemon to each host of the cluster. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Add node-exporter using ceph orch
by Vahideh Alinouri 26 Apr '24

26 Apr '24

Hi guys, I have tried to add node-exporter to the new host in ceph cluster by the command mentioned in the document. ceph orch apply node-exporter hostname I think there is a functionality issue because cephadm log print node-exporter was applied successfully, but it didn't work! I tried the below command and it worked! ceph orch daemon add node-exporter hostname Which way is the correct way?

2 1

Setup Ceph over RDMA
by Vahideh Alinouri 26 Apr '24

26 Apr '24

Hi guys, I need setup Ceph over RDMA, but I faced many issues! The info regarding my cluster: Ceph version is Reef Network cards are Broadcom RDMA. RDMA connection between OSD nodes are OK. I just found ms_type = async+rdma config in document and apply it using ceph config set global ms_type async+rdma After this action the cluster crashes. I tried to cluster back, and I did: Put ms_type async+posix in ceph.conf Restart all MON services The cluster is back, but I don't have any active mgr. All OSDs are down too. Is there any order to do for setting up Ceph over RDMA? Thanks

1 1

MDS crash
by alexey.gerasimov＠opencascade.com 26 Apr '24

26 Apr '24

Dear colleagues, hope that anybody can help us. The initial point: Ceph cluster v15.2 (installed and controlled by the Proxmox) with 3 nodes based on physical servers rented from a cloud provider. CephFS is installed also. Yesterday we discovered that some of the applications stopped working. During the investigation we recognized that we have the problem with Ceph, more precisely with СephFS - MDS daemons suddenly crashed. We tried to restart them and found that they crashed again immediately after the start. The crash information: 2024-04-17T17:47:42.841+0000 7f959ced9700 1 mds.0.29134 recovery_done -- successful recovery! 2024-04-17T17:47:42.853+0000 7f959ced9700 1 mds.0.29134 active_start 2024-04-17T17:47:42.881+0000 7f959ced9700 1 mds.0.29134 cluster recovered. 2024-04-17T17:47:43.825+0000 7f959aed5700 -1 ./src/mds/OpenFileTable.cc: In function 'void OpenFileTable::commit(MDSContext*, uint64_t, int)' thread 7f959aed5700 time 2024-04-17T17:47:43.831243+0000 ./src/mds/OpenFileTable.cc: 549: FAILED ceph_assert(count > 0) Next hours we read the tons of articles, studied the documentation, and checked the common state of Ceph cluster by the various diagnostic commands – but didn’t find anything wrong. At evening we decided to upgrade it up to v16, and finally to v17.2.7. Unfortunately, it didn’t solve the problem, MDS continue to crash with the same error. The only difference that we found is “1 MDSs report damaged metadata” in the output of ceph -s – see it below. I supposed that it may be the well-known bug, but couldn’t find the same one on https://tracker.ceph.com - there are several bugs associated with file OpenFileTable.cc but not related to ceph_assert(count > 0) We tried to check the source code of OpenFileTable.cc also, here is a fragment of it, in function OpenFileTable::_journal_finish int omap_idx = anchor.omap_idx; unsigned& count = omap_num_items.at(omap_idx); ceph_assert(count > 0); So, we guess that the object map is empty for some object in Ceph, and it is unexpected behavior. But again, we found nothing wrong in our cluster… Next, we started with https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ article – tried to reset the journal (despite that it was Ok all the time) and wipe the sessions using cephfs-table-tool all reset session command. No result… Now I decided to continue following this article and run cephfs-data-scan scan_extents command, it is working just now. But I have a doubt that it will solve the issue because of no problem with our objects in Ceph. Is it the new bug? or something else? Any idea is welcome! The important outputs: ----- ceph -s cluster: id: 4cd1c477-c8d0-4855-a1f1-cb71d89427ed health: HEALTH_ERR 1 MDSs report damaged metadata insufficient standby MDS daemons available 83 daemons have recently crashed 3 mgr modules have recently crashed services: mon: 3 daemons, quorum asrv-dev-stor-2,asrv-dev-stor-3,asrv-dev-stor-1 (age 22h) mgr: asrv-dev-stor-2(active, since 22h), standbys: asrv-dev-stor-1 mds: 1/1 daemons up osd: 18 osds: 18 up (since 22h), 18 in (since 29h) data: volumes: 1/1 healthy pools: 5 pools, 289 pgs objects: 29.72M objects, 5.6 TiB usage: 21 TiB used, 47 TiB / 68 TiB avail pgs: 287 active+clean 2 active+clean+scrubbing+deep io: client: 2.5 KiB/s rd, 172 KiB/s wr, 261 op/s rd, 195 op/s wr -----ceph fs dump e29480 enable_multiple, ever_enabled_multiple: 0,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 1 Filesystem 'cephfs' (1) fs_name cephfs epoch 29480 flags 12 joinable allow_snaps allow_multimds_snaps created 2022-11-25T15:56:08.507407+0000 modified 2024-04-18T16:52:29.970504+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 14728 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=156636152} failed damaged stopped data_pools [5] metadata_pool 6 inline_data disabled balancer standby_count_wanted 1 [mds.asrv-dev-stor-1{0:156636152} state up:active seq 6 laggy since 2024-04-18T16:52:29.970479+0000 addr [v2:172.22.2.91:6800/2487054023,v1:172.22.2.91:6801/2487054023] compat {c=[1],r=[1],i=[7ff]}] -----cephfs-journal-tool --rank=cephfs:0 journal inspect Overall journal integrity: OK -----ceph pg dump summary version 41137 stamp 2024-04-18T21:17:59.133536+0000 last_osdmap_epoch 0 last_pg_scan 0 PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG sum 29717605 0 0 0 0 6112544251872 13374192956 28493480 1806575 1806575 OSD_STAT USED AVAIL USED_RAW TOTAL sum 21 TiB 47 TiB 21 TiB 68 TiB -----ceph pg dump pools POOLID OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG 8 31771 0 0 0 0 131337887503 2482 140 401246 401246 7 839707 0 0 0 0 3519034650971 736 61 399328 399328 6 1319576 0 0 0 0 421044421 13374189738 28493279 206749 206749 5 27526539 0 0 0 0 2461702171417 0 0 792165 792165 2 12 0 0 0 0 48497560 0 0 6991 6991

5 7

Cephadm stacktrace on copying ceph.conf
by Jesper Agerbo Krogh [JSKR] 25 Apr '24

25 Apr '24

Hi. We're currently getting these errors - and I seem to be missing a clear overview over the cause and how to debug. 3/26/24 9:38:09 PM[ERR]executing _write_files((['dkcphhpcadmin01', 'dkcphhpcmgt028', 'dkcphhpcmgt029', 'dkcphhpcmgt031', 'dkcphhpcosd033', 'dkcphhpcosd034', 'dkcphhpcosd035', 'dkcphhpcosd036', 'dkcphhpcosd037', 'dkcphhpcosd038', 'dkcphhpcosd039', 'dkcphhpcosd040', 'dkcphhpcosd041', 'dkcphhpcosd042', 'dkcphhpcosd043', 'dkcphhpcosd044'],)) failed. Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 240, in _write_remote_file await asyncssh.scp(f.name, (conn, tmp_path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc asyncssh.sftp.SFTPFailure: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 79, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1088, in _write_files self._write_client_files(client_files, host) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1107, in _write_client_files self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 261, in write_remote_file host, path, content, mode, uid, gid, addr)) File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async return self.event_loop.get_result(coro) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result return asyncio.run_coroutine_threadsafe(coro, self._loop).result() File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/ssh.py", line 249, in _write_remote_file raise OrchestratorError(msg) orchestrator._interface.OrchestratorError: Unable to write dkcphhpcmgt028:/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied 3/26/24 9:38:09 PM[ERR]Unable to write dkcphhpcmgt028:/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 240, in _write_remote_file await asyncssh.scp(f.name, (conn, tmp_path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc asyncssh.sftp.SFTPFailure: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied 3/26/24 9:38:09 PM[INF]Updating dkcphhpcmgt028:/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf It seem to be related to the permissions that the manager writes the files with and the process copying them around. $ sudo ceph -v [sudo] password for adminjskr: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Best regards, Jesper Agerbo Krogh Director Digitalization Digitalization Topsoe A/S Haldor Topsøes Allé 1 2800 Kgs. Lyngby Denmark Phone (direct): 27773240     Read more attopsoe.com Topsoe A/S and/or its affiliates. This e-mail message (including attachments, if any) is confidential and may be privileged. It is intended only for the addressee. Any unauthorised distribution or disclosure is prohibited. Disclosure to anyone other than the intended recipient does not constitute waiver of privilege. If you have received this email in error, please notify the sender by email and delete it and any attachments from your computer system and records.

2 1

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

Slow/blocked reads and writes
by Fábio Sato 24 Apr '24

24 Apr '24

Hello all, I am trying to troubleshoot a ceph cluster version 18.2.2 having users reporting slow and blocked reads and writes. When running "ceph status" I am seeing many warnings about its health state: cluster: id: cc881230-e0dd-11ee-aa9e-37c4e4e5e14b health: HEALTH_WARN 6 clients failing to respond to capability release 2 clients failing to advance oldest client/flush tid 1 MDSs report slow requests 1 MDSs behind on trimming Too many repaired reads on 11 OSDs Degraded data redundancy: 2 pgs degraded 105 pgs not deep-scrubbed in time 109 pgs not scrubbed in time 1 mgr modules have recently crashed 12 slow ops, oldest one blocked for 97678 sec, daemons [osd.11,osd.12,osd.15,osd.16,osd.19,osd.20,osd.28,osd.3,osd.32,osd.34]... have slow ops. services: mon: 3 daemons, quorum file03-xx,file04-xx,file05-xx (age 17h) mgr: file03-xx.xxxxxx(active, since 2w), standbys: file04-xx.xxxxxx mds: 1/1 daemons up, 1 standby osd: 44 osds: 44 up (since 17h), 44 in (since 39h); 492 remapped pgs data: volumes: 1/1 healthy pools: 3 pools, 2065 pgs objects: 66.44M objects, 140 TiB usage: 281 TiB used, 304 TiB / 586 TiB avail pgs: 16511162/134215883 objects misplaced (12.302%) 1508 active+clean 487 active+remapped+backfill_wait 53 active+clean+scrubbing+deep 8 active+clean+scrubbing 5 active+remapped+backfilling 2 active+recovering+degraded+repair 2 active+recovering+repair io: recovery: 47 MiB/s, 37 objects/s When checking the output of `ceph -w` I am flooded with crc error messages like the examples below: 2024-04-24T19:15:40.430334+0000 osd.32 [ERR] 3.566 full-object read crc 0xa5da7fa != expected 0xffffffff on 3:66a8d8f5:::10001c72400.00000007:head 2024-04-24T19:15:40.430507+0000 osd.39 [ERR] 3.270 full-object read crc 0xa1bc3a1e != expected 0xffffffff on 3:0e44aa2f:::1000265a625.00000003:head 2024-04-24T19:15:40.494249+0000 osd.28 [ERR] 3.469 full-object read crc 0x8e757c06 != expected 0xffffffff on 3:962852f4:::10001c72c2d.00000001:head 2024-04-24T19:15:40.529771+0000 osd.32 [ERR] 3.566 full-object read crc 0xa5da7fa != expected 0xffffffff on 3:66a8d8f5:::10001c72400.00000007:head 2024-04-24T19:15:40.582128+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.583350+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.662945+0000 osd.28 [ERR] 3.469 full-object read crc 0x8e757c06 != expected 0xffffffff on 3:962852f4:::10001c72c2d.00000001:head 2024-04-24T19:15:40.698197+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.699389+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.769191+0000 osd.28 [ERR] 3.469 full-object read crc 0x8e757c06 != expected 0xffffffff on 3:962852f4:::10001c72c2d.00000001:head 2024-04-24T19:15:40.834344+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.835513+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head I suspect this is the main issue affecting the cluster health state and performance so I am trying to address this first. The "expected 0xffffffff" crc seems like a bug to me and I found an open ticket (https://tracker.ceph.com/issues/53240) with similar error messages but I am not sure this is related to my case. Could someone point me to the steps to solve these errors? Cheers, -- Fabio

1 0