- ceph-users - lists.ceph.io

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

cephfs read hang after cluster stuck, but need attach the process to continue
by zxcs 13 Dec '23

13 Dec '23

Hi, experts, we are using cephfs with 16.2.* with multi active mds, and recently we see a strange thing, we have some c++ code about read file from cephfs. the client code just call very base read(), and when the cluster hit mds has slow request, and later the cluster back to normal. the read hang. then we need to gdb attach -p `pid of c++ process`, just attach, do nothing, the code then continue running. Our question is why this happens? and config could tune or we need change out c++ read code? Thanks xz

1 0

Cephfs too many repaired copies on osds
by zxcs 13 Dec '23

13 Dec '23

Hi, Experts, we are using cephfs with 16.2.* with multi active mds, and recently we see an osd report “full object read crc *** != expected oxffffffff on ****:head” “missing primary copy of ***: will try to read copies on **” from `ceph -s`, could see OSD_TOO_MANY_REPAIRS: Too many repaired reads on ** OSDs. we don’t know how to fix this , could you are please help shed some light here? Thanks a ton! Thanks xz

2 2

mds.0.journaler.pq(ro) _finish_read got error -2
by Eugen Block 12 Dec '23

12 Dec '23

Hi, I'm trying to help someone with a broken CephFS. We managed to recover basic ceph functionality but the CephFS is still inaccessible (currently read-only). We went through the disaster recovery steps but to no avail. Here's a snippet from the startup logs: ---snip--- mds.0.41 Booting: 2: waiting for purge queue recovered mds.0.journaler.pq(ro) _finish_probe_end write_pos = 14797504512 (header had 14789452521). recovered. mds.0.purge_queue operator(): open complete mds.0.purge_queue operator(): recovering write_pos monclient: get_auth_request con 0x55c280bc5c00 auth_method 0 monclient: get_auth_request con 0x55c280ee0c00 auth_method 0 mds.0.journaler.pq(ro) _finish_read got error -2 mds.0.purge_queue _recover: Error -2 recovering write_pos mds.0.purge_queue _go_readonly: going readonly because internal IO failed: No such file or directory mds.0.journaler.pq(ro) set_readonly mds.0.41 unhandled write error (2) No such file or directory, force readonly... mds.0.cache force file system read-only force file system read-only ---snip--- I've added the dev mailing list, maybe someone can give some advice how to continue from here (we could try to recover with an empty metadata pool). Or is this FS lost? Thanks! Eugen

3 4

Announcing go-ceph v0.25.0
by Anoop C S 12 Dec '23

12 Dec '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.25.0 More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S

1 0

MDS recovery with existing pools
by Eugen Block 12 Dec '23

12 Dec '23

Hi, following up on the previous thread (After hardware failure tried to recover ceph and followed instructions for recovery using OSDS), we were able to get ceph back into a healthy state (including the unfound object). Now the CephFS needs to be recovered and I'm having trouble to fully understand the docs [1] which the next steps would be. We ran the following which according to [1] sets the state to existing but failed: ceph fs new <fs_name> <metadata_pool> <data_pool> --force --recover But how to continue from here? Should we expect an active MDS at this point or not? Because the "ceph fs status" output still shows rank 0 as failed. We then tried: ceph fs set <fs_name> joinable true But apparently it was already joinable, nothing changed. Before doing anything (destructive) from the advanced options [2] I wanted to ask the community, how to get on from here. I pasted the mds logs at the bottom, I'm not really sure if the current state is expected or not. Apparently, the journal recovers but the purge_queue does not: mds.0.41 Booting: 2: waiting for purge queue recovered mds.0.journaler.pq(ro) _finish_probe_end write_pos = 14797504512 (header had 14789452521). recovered. mds.0.purge_queue operator(): open complete mds.0.purge_queue operator(): recovering write_pos monclient: get_auth_request con 0x55c280bc5c00 auth_method 0 monclient: get_auth_request con 0x55c280ee0c00 auth_method 0 mds.0.journaler.pq(ro) _finish_read got error -2 mds.0.purge_queue _recover: Error -2 recovering write_pos mds.0.purge_queue _go_readonly: going readonly because internal IO failed: No such file or directory mds.0.journaler.pq(ro) set_readonly mds.0.41 unhandled write error (2) No such file or directory, force readonly... mds.0.cache force file system read-only force file system read-only Is this expected because the "--recover" flag prevents an active MDS or not? Before running "ceph mds rmfailed ..." and/or "ceph fs reset <file system name>" with the --yes-i-really-mean-it flag I'd like to ask for your input. In which case should we run those commands? The docs are not really clear to me. Any input is highly appreciated! Thanks! Eugen [1] https://docs.ceph.com/en/latest/cephfs/recover-fs-after-mon-store-loss/ [2] https://docs.ceph.com/en/latest/cephfs/administration/#advanced-cephfs-admi… ---snip--- Dec 07 15:35:48 node02 bash[692598]: debug -90> 2023-12-07T13:35:47.730+0000 7f4cd855f700 1 mds.storage.node02.hemalk Updating MDS map to version 41 from mon.0 Dec 07 15:35:48 node02 bash[692598]: debug -89> 2023-12-07T13:35:47.730+0000 7f4cd855f700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Dec 07 15:35:48 node02 bash[692598]: debug -88> 2023-12-07T13:35:47.730+0000 7f4cd855f700 5 asok(0x55c27fe86000) register_command objecter_requests hook 0x55c27fe16310 Dec 07 15:35:48 node02 bash[692598]: debug -87> 2023-12-07T13:35:47.730+0000 7f4cd855f700 10 monclient: _renew_subs Dec 07 15:35:48 node02 bash[692598]: debug -86> 2023-12-07T13:35:47.730+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -85> 2023-12-07T13:35:47.730+0000 7f4cd855f700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) Dec 07 15:35:48 node02 bash[692598]: debug -84> 2023-12-07T13:35:47.730+0000 7f4cd855f700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Dec 07 15:35:48 node02 bash[692598]: debug -83> 2023-12-07T13:35:47.730+0000 7f4cd855f700 4 mds.0.0 apply_blocklist: killed 0, blocklisted sessions (0 blocklist entries, 0) Dec 07 15:35:48 node02 bash[692598]: debug -82> 2023-12-07T13:35:47.730+0000 7f4cd855f700 1 mds.0.41 handle_mds_map i am now mds.0.41 Dec 07 15:35:48 node02 bash[692598]: debug -81> 2023-12-07T13:35:47.734+0000 7f4cd855f700 1 mds.0.41 handle_mds_map state change up:standby --> up:replay Dec 07 15:35:48 node02 bash[692598]: debug -80> 2023-12-07T13:35:47.734+0000 7f4cd855f700 5 mds.beacon.storage.node02.hemalk set_want_state: up:standby -> up:replay Dec 07 15:35:48 node02 bash[692598]: debug -79> 2023-12-07T13:35:47.734+0000 7f4cd855f700 1 mds.0.41 replay_start Dec 07 15:35:48 node02 bash[692598]: debug -78> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: opening inotable Dec 07 15:35:48 node02 bash[692598]: debug -77> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -76> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: opening sessionmap Dec 07 15:35:48 node02 bash[692598]: debug -75> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -74> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: opening mds log Dec 07 15:35:48 node02 bash[692598]: debug -73> 2023-12-07T13:35:47.734+0000 7f4cd855f700 5 mds.0.log open discovering log bounds Dec 07 15:35:48 node02 bash[692598]: debug -72> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: opening purge queue (async) Dec 07 15:35:48 node02 bash[692598]: debug -71> 2023-12-07T13:35:47.734+0000 7f4cd855f700 4 mds.0.purge_queue open: opening Dec 07 15:35:48 node02 bash[692598]: debug -70> 2023-12-07T13:35:47.734+0000 7f4cd855f700 1 mds.0.journaler.pq(ro) recover start Dec 07 15:35:48 node02 bash[692598]: debug -69> 2023-12-07T13:35:47.734+0000 7f4cd855f700 1 mds.0.journaler.pq(ro) read_head Dec 07 15:35:48 node02 bash[692598]: debug -68> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -67> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: loading open file table (async) Dec 07 15:35:48 node02 bash[692598]: debug -66> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -65> 2023-12-07T13:35:47.734+0000 7f4cd855f700 2 mds.0.41 Booting: 0: opening snap table Dec 07 15:35:48 node02 bash[692598]: debug -64> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -63> 2023-12-07T13:35:47.734+0000 7f4cd1d52700 4 mds.0.journalpointer Reading journal pointer '400.00000000' Dec 07 15:35:48 node02 bash[692598]: debug -62> 2023-12-07T13:35:47.734+0000 7f4cd1d52700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -61> 2023-12-07T13:35:47.734+0000 7f4cd4557700 2 mds.0.cache Memory usage: total 316452, rss 43088, heap 198940, baseline 198940, 0 / 0 inodes have caps, 0 caps, 0 caps per inode Dec 07 15:35:48 node02 bash[692598]: debug -60> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _renew_subs Dec 07 15:35:48 node02 bash[692598]: debug -59> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: _send_mon_message to mon.node02 at v2:10.40.99.12:3300/0 Dec 07 15:35:48 node02 bash[692598]: debug -58> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 1 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -57> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 2 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -56> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 3 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -55> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 4 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -54> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 5 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -53> 2023-12-07T13:35:47.734+0000 7f4cd855f700 10 monclient: handle_get_version_reply finishing 6 version 10835 Dec 07 15:35:48 node02 bash[692598]: debug -52> 2023-12-07T13:35:47.734+0000 7f4cdb565700 10 monclient: get_auth_request con 0x55c280bc5800 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -51> 2023-12-07T13:35:47.734+0000 7f4cdbd66700 10 monclient: get_auth_request con 0x55c280dc6800 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -50> 2023-12-07T13:35:47.734+0000 7f4cdad64700 10 monclient: get_auth_request con 0x55c280dc7800 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -49> 2023-12-07T13:35:47.734+0000 7f4cd3555700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 14789115904, expire 14789452521, write 14789452521, stream_format 1). probing for end of log (from 14789452521)... Dec 07 15:35:48 node02 bash[692598]: debug -48> 2023-12-07T13:35:47.734+0000 7f4cd3555700 1 mds.0.journaler.pq(ro) probing for end of the log Dec 07 15:35:48 node02 bash[692598]: debug -47> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 1 mds.0.journaler.mdlog(ro) recover start Dec 07 15:35:48 node02 bash[692598]: debug -46> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 1 mds.0.journaler.mdlog(ro) read_head Dec 07 15:35:48 node02 bash[692598]: debug -45> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 4 mds.0.log Waiting for journal 0x200 to recover... Dec 07 15:35:48 node02 bash[692598]: debug -44> 2023-12-07T13:35:47.738+0000 7f4cdbd66700 10 monclient: get_auth_request con 0x55c280dc7c00 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -43> 2023-12-07T13:35:47.738+0000 7f4cd2553700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 1416940748800, expire 1416947000701, write 1417125359769, stream_format 1). probing for end of log (from 1417125359769)... Dec 07 15:35:48 node02 bash[692598]: debug -42> 2023-12-07T13:35:47.738+0000 7f4cd2553700 1 mds.0.journaler.mdlog(ro) probing for end of the log Dec 07 15:35:48 node02 bash[692598]: debug -41> 2023-12-07T13:35:47.738+0000 7f4cdb565700 10 monclient: get_auth_request con 0x55c280e2fc00 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -40> 2023-12-07T13:35:47.738+0000 7f4cdad64700 10 monclient: get_auth_request con 0x55c280ee0400 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -39> 2023-12-07T13:35:47.738+0000 7f4cd2553700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1417129492480 (header had 1417125359769). recovered. Dec 07 15:35:48 node02 bash[692598]: debug -38> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 4 mds.0.log Journal 0x200 recovered. Dec 07 15:35:48 node02 bash[692598]: debug -37> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 4 mds.0.log Recovered journal 0x200 in format 1 Dec 07 15:35:48 node02 bash[692598]: debug -36> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 2 mds.0.41 Booting: 1: loading/discovering base inodes Dec 07 15:35:48 node02 bash[692598]: debug -35> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 0 mds.0.cache creating system inode with ino:0x100 Dec 07 15:35:48 node02 bash[692598]: debug -34> 2023-12-07T13:35:47.738+0000 7f4cd1d52700 0 mds.0.cache creating system inode with ino:0x1 Dec 07 15:35:48 node02 bash[692598]: debug -33> 2023-12-07T13:35:47.742+0000 7f4cdbd66700 10 monclient: get_auth_request con 0x55c280dc7400 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -32> 2023-12-07T13:35:47.742+0000 7f4cd2553700 2 mds.0.41 Booting: 2: replaying mds log Dec 07 15:35:48 node02 bash[692598]: debug -31> 2023-12-07T13:35:47.742+0000 7f4cd2553700 2 mds.0.41 Booting: 2: waiting for purge queue recovered Dec 07 15:35:48 node02 bash[692598]: debug -30> 2023-12-07T13:35:47.742+0000 7f4cd3555700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 14797504512 (header had 14789452521). recovered. Dec 07 15:35:48 node02 bash[692598]: debug -29> 2023-12-07T13:35:47.742+0000 7f4cd3555700 4 mds.0.purge_queue operator(): open complete Dec 07 15:35:48 node02 bash[692598]: debug -28> 2023-12-07T13:35:47.742+0000 7f4cd3555700 4 mds.0.purge_queue operator(): recovering write_pos Dec 07 15:35:48 node02 bash[692598]: debug -27> 2023-12-07T13:35:47.742+0000 7f4cdb565700 10 monclient: get_auth_request con 0x55c280bc5c00 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -26> 2023-12-07T13:35:47.742+0000 7f4cdad64700 10 monclient: get_auth_request con 0x55c280ee0c00 auth_method 0 Dec 07 15:35:48 node02 bash[692598]: debug -25> 2023-12-07T13:35:47.746+0000 7f4cd3555700 0 mds.0.journaler.pq(ro) _finish_read got error -2 Dec 07 15:35:48 node02 bash[692598]: debug -24> 2023-12-07T13:35:47.746+0000 7f4cd3555700 -1 mds.0.purge_queue _recover: Error -2 recovering write_pos Dec 07 15:35:48 node02 bash[692598]: debug -23> 2023-12-07T13:35:47.746+0000 7f4cd3555700 1 mds.0.purge_queue _go_readonly: going readonly because internal IO failed: No such file or directory Dec 07 15:35:48 node02 bash[692598]: debug -22> 2023-12-07T13:35:47.746+0000 7f4cd3555700 1 mds.0.journaler.pq(ro) set_readonly Dec 07 15:35:48 node02 bash[692598]: debug -21> 2023-12-07T13:35:47.746+0000 7f4cd3555700 -1 mds.0.41 unhandled write error (2) No such file or directory, force readonly... Dec 07 15:35:48 node02 bash[692598]: debug -20> 2023-12-07T13:35:47.746+0000 7f4cd3555700 1 mds.0.cache force file system read-only Dec 07 15:35:48 node02 bash[692598]: debug -19> 2023-12-07T13:35:47.746+0000 7f4cd3555700 0 log_channel(cluster) log [WRN] : force file system read-only ---snip---

2 5

Deleting files from lost+found in 18.2.0
by Thomas Widhalm 11 Dec '23

11 Dec '23

Hi, I saw in the changelogs that it should now finally be possible to delete files from lost+found with 18.x . I upgraded recently but still I can't delete or move files from there. I tried to change permissions but still every time I try it, I get "read only filesystem" but only when acting on files in lost+found. I searched the documentation but I can't find anything related. Is there a special trick or a flag I have to set? Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

1 0

Ceph 17.2.7 to 18.2.0 issues
by pclark6063＠outlook.com 11 Dec '23

11 Dec '23

Hi All, Recently I upgraded my cluster from Quincy to Reef. Everything appeared to go smoothly and without any issues arising. I was forced to poweroff the cluster, performing the ususal procedures beforehand and everything appears to have come back fine. Every service reports green across the board except.... If i try to copy any files from a cephfs mountpoint whether kernel or fuse the actual copy will hang. ls/stat etc all work which indicates metadata appears fine but copying always hangs. I can copy objects direct using the rados toolset which indicates the underlying data exists. The system itself reports no errors and thinks its healthy. The entire cluster and cephfs clients are all Rocky9. Any advice would be much appreciatd. I'd find this easier to deal with if the cluster actually gave me an error....

3 4

reef 18.2.1 QE Validation status
by Yuri Weinstein 11 Dec '23

11 Dec '23

Details of this release are summarized here: https://tracker.ceph.com/issues/63443#note-1 Seeking approvals/reviews for: smoke - Laura, Radek, Prashant, Venky (POOL_APP_NOT_ENABLE failures) rados - Neha, Radek, Travis, Ernesto, Adam King rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade/quincy-x (reef) - Laura PTL powercycle - Brad perf-basic - Laura, Prashant (POOL_APP_NOT_ENABLE failures) Please reply to this email with approval and/or trackers of known issues/PRs to address them. TIA YuriW

12 30

Osd full
by Mohamed LAMDAOUAR 11 Dec '23

11 Dec '23

Hello the team, We initially had a cluster of 3 machines with 4 osd on each machine, we added 4 machines in the cluster (each machine with 4 osd) We launched the balancing but it never finished, still in progress. But the big issue: we have an osd full and all the pools on this osd are read only. *ceph osd df *: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 20 hdd 9.09569 1.00000 9.1 TiB 580 GiB 576 GiB 1.2 GiB 3.1 GiB 8.5 TiB 6.23 0.32 169 up 21 hdd 9.09569 1.00000 9.1 TiB 1.5 TiB 1.5 TiB 252 MiB 7.7 GiB 7.6 TiB 16.08 0.82 247 up 22 hdd 9.09569 1.00000 9.1 TiB 671 GiB 667 GiB 204 MiB 4.1 GiB 8.4 TiB 7.21 0.37 136 up 23 hdd 9.09569 1.00000 9.1 TiB 665 GiB 660 GiB 270 MiB 4.5 GiB 8.4 TiB 7.14 0.37 124 up 0 hdd 9.09569 1.00000 9.1 TiB 1.2 TiB 1.2 TiB 87 MiB 6.0 GiB 7.9 TiB 13.30 0.68 230 up 1 hdd 9.09569 1.00000 9.1 TiB 1.3 TiB 1.3 TiB 347 MiB 6.6 GiB 7.8 TiB 14.01 0.72 153 up 2 hdd 9.09569 0.65009 9.1 TiB 1.8 TiB 1.8 TiB 443 MiB 7.3 GiB 7.3 TiB 20.00 1.03 147 up 3 hdd 9.09569 1.00000 9.1 TiB 617 GiB 611 GiB 220 MiB 5.8 GiB 8.5 TiB 6.62 0.34 101 up 4 hdd 9.09569 0.80005 9.1 TiB 2.0 TiB 2.0 TiB 293 MiB 8.2 GiB 7.1 TiB 22.12 1.13 137 up 5 hdd 9.09569 1.00000 9.1 TiB 857 GiB 852 GiB 157 MiB 4.9 GiB 8.3 TiB 9.20 0.47 155 up 6 hdd 9.09569 1.00000 9.1 TiB 580 GiB 575 GiB 678 MiB 4.4 GiB 8.5 TiB 6.23 0.32 114 up 7 hdd 9.09569 0.50000 9.1 TiB 7.7 TiB 7.7 TiB 103 MiB 16 GiB 1.4 TiB 85.03 4.36 201 up 24 hdd 9.09569 1.00000 9.1 TiB 1.2 TiB 1.2 TiB 133 MiB 6.2 GiB 7.9 TiB 13.11 0.67 225 up 25 hdd 9.09569 0.34999 9.1 TiB 8.3 TiB 8.2 TiB 101 MiB 17 GiB 860 GiB 90.77 4.66 159 up 26 hdd 9.09569 1.00000 9.1 TiB 665 GiB 661 GiB 292 MiB 3.8 GiB 8.4 TiB 7.14 0.37 107 up 27 hdd 9.09569 1.00000 9.1 TiB 427 GiB 423 GiB 241 MiB 3.4 GiB 8.7 TiB 4.58 0.24 103 up 8 hdd 9.09569 1.00000 9.1 TiB 845 GiB 839 GiB 831 MiB 5.9 GiB 8.3 TiB 9.07 0.47 163 up 9 hdd 9.09569 1.00000 9.1 TiB 727 GiB 722 GiB 162 MiB 4.8 GiB 8.4 TiB 7.80 0.40 169 up 10 hdd 9.09569 0.80005 9.1 TiB 1.9 TiB 1.9 TiB 742 MiB 7.5 GiB 7.2 TiB 21.01 1.08 136 up 11 hdd 9.09569 1.00000 9.1 TiB 733 GiB 727 GiB 498 MiB 5.2 GiB 8.4 TiB 7.87 0.40 163 up 12 hdd 9.09569 1.00000 9.1 TiB 892 GiB 886 GiB 318 MiB 5.6 GiB 8.2 TiB 9.58 0.49 254 up 13 hdd 9.09569 1.00000 9.1 TiB 759 GiB 755 GiB 37 MiB 4.0 GiB 8.4 TiB 8.15 0.42 134 up 14 hdd 9.09569 0.85004 9.1 TiB 2.3 TiB 2.3 TiB 245 MiB 7.7 GiB 6.8 TiB 24.96 1.28 142 up 15 hdd 9.09569 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 435 MiB 16 GiB 1.8 TiB 80.17 4.11 213 up 16 hdd 9.09569 1.00000 9.1 TiB 784 GiB 781 GiB 104 MiB 3.6 GiB 8.3 TiB 8.42 0.43 247 up 17 hdd 9.09569 1.00000 9.1 TiB 861 GiB 856 GiB 269 MiB 5.1 GiB 8.3 TiB 9.25 0.47 102 up 18 hdd 9.09569 1.00000 9.1 TiB 1.9 TiB 1.9 TiB 962 MiB 8.2 GiB 7.2 TiB 21.15 1.09 283 up 19 hdd 9.09569 1.00000 9.1 TiB 893 GiB 888 GiB 291 MiB 4.6 GiB 8.2 TiB 9.59 0.49 148 up TOTAL 255 TiB 50 TiB 49 TiB 9.7 GiB 187 GiB 205 TiB 19.49 MIN/MAX VAR: 0.24/4.66 STDDEV: 19.63 *ceph health detail |grep -i wrn* [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set [WRN] OSD_NEARFULL: 2 nearfull osd(s) [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 16 pgs backfill_toofull [WRN] PG_NOT_DEEP_SCRUBBED: 1360 pgs not deep-scrubbed in time [WRN] PG_NOT_SCRUBBED: 53 pgs not scrubbed in time [WRN] POOL_NEARFULL: 36 pool(s) nearfull Thanks the team ;)

2 1