- ceph-users - lists.ceph.io

physical vs osd performance
by Curt 10 Jan '24

10 Jan '24

Hello all, Looking at grafana reports, can anyone point me to documentation that outlines physical vs osd? https://docs.ceph.com/en/latest/monitoring/ gives some basic info, but I'm trying to get a better understanding. For instance, physical latency is 20ms and osd is 200ms, these are just made up for this example, why the huge difference? Same thing if for bytes or iops, just using latency as an example. Thanks, Curt

1 0

Re: Stuck in upgrade process to reef
by Igor Fedotov 10 Jan '24

10 Jan '24

Hi Jan, indeed this looks like some memory allocation problem - may be OSD's RAM usage threshold reached or something? Curious if you have any custom OSD settings or may be any memory caps for Ceph containers? Could you please set debug_bluestore to 5/20 and debug_prioritycache to 10 and try to start OSD once again. Please monitor process RAM usage along the process and share the resulting log. Thanks, Igor On 10/01/2024 11:20, Jan Marek wrote: > Hi Igor, > > I've tried to repair osd.1 with command: > > ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.1 --command repair > > and then start osd.1 ceph-osd podman service. > > It semms, that there is problem with memory allocation, see > attached log... > > Sincerely > Jan > > Dne Út, led 09, 2024 at 02:23:32 CET napsal(a) Igor Fedotov: >> Hi Marek, >> >> I haven't looked through those upgrade logs yet but here are some comments >> regarding last OSD startup attempt. >> >> First of answering your question >> >>> _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) >>> Is it a mandatory part of fsck? >> This is caused by previous non-graceful OSD process shutdown. BlueStore is unable to find up-to-date allocation map and recovers it from RocksDB. And since fsck is a read-only procedure the recovered allocmap is not saved - hence all the following BlueStore startups (within fsck or OSD init) cause another rebuild attempt. To avoid that you might want to run repair instead of fsck - this will persist up-to-date allocation map and avoid its rebuilding on the next startup. This will work till the next non-graceful shutdown only - hence unsuccessful OSD attempt might break the allocmap state again. >> >> Secondly - looking at OSD startup log one can see that actual OSD log ends with that allocmap recovery as well: >> >>> 2024-01-09T11:25:30.718449+01:00 osd1 ceph-osd[1734062]: bluestore(/var/lib/ceph/osd/ceph-1) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... >> Subsequent log line indicating OSD daemon termination is from systemd: >>> 2024-01-09T11:25:33.516258+01:00 osd1 systemd[1]: Stopping ceph-2c565e24-7850-47dc-a751-a6357cbbaf2a(a)osd.1.service - Ceph osd.1 for 2c565e24-7850-47dc-a751-a6357cbbaf2a... >> And honestly these lines provide almost no clue why termination happened. No obvious OSD failures or something are shown. Perhaps containerized environment hides the details e.g. by cutting off OSD log's tail. >> So you might want to proceed the investigation by running repair prior to starting the OSD as per above. This will result in no alloc map recovery and hopefully workaround the problem during startup - if the issue is caused by allocmap recovery. >> Additionally you might want to increase debug_bluestore log level for osd.1 before starting it up to get more insight on what's happening. >> >> Alternatively you might want to play with OSD log target settings to write OSD.1 log to some file rather than using system wide logging infra - hopefully this will be more helpful. >> >> Thanks, >> Igor >> >> On 09/01/2024 13:31, Jan Marek wrote: >>> Hi Igor, >>> >>> I've sent you logs via filesender.cesnet.cz, if someone would >>> be interested, they are here: >>> >>> https://filesender.cesnet.cz/?s=download&token=047b1ec4-4df0-4e8a-90fc-3170… >>> >>> Some points: >>> >>> 1) I've found, that on the osd1 server was bad time (3 minutes in >>> future). I've corrected that. Yes, I know, that it's bad, but we >>> moved servers to any other net segment, where they have no access >>> to the timeservers in Internet, then I must reconfigure it to use >>> our own NTP servers. >>> >>> 2) I've tried to start osd.1 service by this sequence: >>> >>> a) >>> >>> ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.1 --command fsck >>> >>> (without setting log properly :-( ) >>> >>> b) >>> >>> export CEPH_ARGS="--log-file osd.1.log --debug-bluestore 5/20" >>> ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.1 --command fsck >>> >>> - here I have one question: Why is it in this log stil this line: >>> >>> _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) >>> >>> Is it a mandatory part of fsck? >>> >>> Log is attached. >>> >>> c) >>> >>> systemctl start ceph-2c565e24-7850-47dc-a751-a6357cbbaf2a(a)osd.1.service >>> >>> still crashing, gzip-ed log attached too. >>> >>> Many thanks for exploring problem. >>> >>> Sincerely >>> Jan Marek >>> >>> Dne Po, led 08, 2024 at 12:00:05 CET napsal(a) Igor Fedotov: >>>> Hi Jan, >>>> >>>> indeed fsck logs for the OSDs other than osd.0 look good so it would be >>>> interesting to see OSD startup logs for them. Preferably to have that for >>>> multiple (e.g. 3-4) OSDs to get the pattern. >>>> >>>> Original upgrade log(s) would be nice to see as well. >>>> >>>> You might want to use Google Drive or any other publicly available file >>>> sharing site for that. >>>> >>>> >>>> Thanks, >>>> >>>> Igor >>>> >>>> On 05/01/2024 10:25, Jan Marek wrote: >>>>> Hi Igor, >>>>> >>>>> I've tried to start only osd.1, which seems to be fsck'd OK, but >>>>> it crashed :-( >>>>> >>>>> I search logs and I've found, that I have logs from 22.12.2023, >>>>> when I've did a upgrade (I have set logging to journald). >>>>> >>>>> Would you be interested in those logs? This file have 30MB in >>>>> bzip2 format, how I can share it with you? >>>>> >>>>> It contains crash log from start osd.1 too, but I can cut out >>>>> from it and send it to list... >>>>> >>>>> Sincerely >>>>> Jan Marek >>>>> >>>>> Dne Čt, led 04, 2024 at 02:43:48 CET napsal(a) Jan Marek: >>>>>> Hi Igor, >>>>>> >>>>>> I've ran this oneliner: >>>>>> >>>>>> for i in {0..12}; do export CEPH_ARGS="--log-file osd."${i}".log --debug-bluestore 5/20" ; ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.${i} --command fsck ; done; >>>>>> >>>>>> On osd.0 it crashed very quickly, on osd.1 it is still working. >>>>>> >>>>>> I've send those logs in one e-mail. >>>>>> >>>>>> But! >>>>>> >>>>>> I've tried to list disk devices in monitor view, and I've got >>>>>> very interesting screenshot - some part I've emphasized by red >>>>>> rectangulars. >>>>>> >>>>>> I've got a json from syslog, which was as a part cephadm call, >>>>>> where it seems to be correct (for my eyes). >>>>>> >>>>>> Can be this coincidence for this problem? >>>>>> >>>>>> Sincerely >>>>>> Jan Marek >>>>>> >>>>>> Dne Čt, led 04, 2024 at 12:32:47 CET napsal(a) Igor Fedotov: >>>>>>> Hi Jan, >>>>>>> >>>>>>> may I see the fsck logs from all the failing OSDs to see the pattern. IIUC >>>>>>> the full node is suffering from the issue, right? >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Igor >>>>>>> >>>>>>> On 1/2/2024 10:53 AM, Jan Marek wrote: >>>>>>>> Hello once again, >>>>>>>> >>>>>>>> I've tried this: >>>>>>>> >>>>>>>> export CEPH_ARGS="--log-file /tmp/osd.0.log --debug-bluestore 5/20" >>>>>>>> ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0 --command fsck >>>>>>>> >>>>>>>> And I've sending /tmp/osd.0.log file attached. >>>>>>>> >>>>>>>> Sincerely >>>>>>>> Jan Marek >>>>>>>> >>>>>>>> Dne Ne, pro 31, 2023 at 12:38:13 CET napsal(a) Igor Fedotov: >>>>>>>>> Hi Jan, >>>>>>>>> >>>>>>>>> this doesn't look like RocksDB corruption but rather like some BlueStore >>>>>>>>> metadata inconsistency. Also assertion backtrace in the new log looks >>>>>>>>> completely different from the original one. So in an attempt to find any >>>>>>>>> systematic pattern I'd suggest to run fsck with verbose logging for every >>>>>>>>> failing OSD. Relevant command line: >>>>>>>>> >>>>>>>>> CEPH_ARGS="--log-file osd.N.log --debug-bluestore 5/20" >>>>>>>>> bin/ceph-bluestore-tool --path <path-to-osd> --command fsck >>>>>>>>> >>>>>>>>> Unlikely this will fix anything it's rather a way to collect logs to get >>>>>>>>> better insight. >>>>>>>>> >>>>>>>>> >>>>>>>>> Additionally you might want to run similar fsck for a couple of healthy OSDs >>>>>>>>> - curious if it succeeds as I have a feeling that the problem with crashing >>>>>>>>> OSDs had been hidden before the upgrade and revealed rather than caused by >>>>>>>>> it. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Igor >>>>>>>>> >>>>>>>>> On 12/29/2023 3:28 PM, Jan Marek wrote: >>>>>>>>>> Hello Igor, >>>>>>>>>> >>>>>>>>>> I'm attaching a part of syslog creating while starting OSD.0. >>>>>>>>>> >>>>>>>>>> Many thanks for help. >>>>>>>>>> >>>>>>>>>> Sincerely >>>>>>>>>> Jan Marek >>>>>>>>>> >>>>>>>>>> Dne St, pro 27, 2023 at 04:42:56 CET napsal(a) Igor Fedotov: >>>>>>>>>>> Hi Jan, >>>>>>>>>>> >>>>>>>>>>> IIUC the attached log is for ceph-kvstore-tool, right? >>>>>>>>>>> >>>>>>>>>>> Can you please share full OSD startup log as well? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Igor >>>>>>>>>>> >>>>>>>>>>> On 12/27/2023 4:30 PM, Jan Marek wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> I've problem: my ceph cluster (3x mon nodes, 6x osd nodes, every >>>>>>>>>>>> osd node have 12 rotational disk and one NVMe device for >>>>>>>>>>>> bluestore DB). CEPH is installed by ceph orchestrator and have >>>>>>>>>>>> bluefs storage on osd. >>>>>>>>>>>> >>>>>>>>>>>> I've started process upgrade from version 17.2.6 to 18.2.1 by >>>>>>>>>>>> invocating: >>>>>>>>>>>> >>>>>>>>>>>> ceph orch upgrade start --ceph-version 18.2.1 >>>>>>>>>>>> >>>>>>>>>>>> After upgrade of mon and mgr processes orchestrator tried to >>>>>>>>>>>> upgrade the first OSD node, but they are falling down. >>>>>>>>>>>> >>>>>>>>>>>> I've stop the process of upgrade, but I have 1 osd node >>>>>>>>>>>> completely down. >>>>>>>>>>>> >>>>>>>>>>>> After upgrade I've got some error messages and I've found >>>>>>>>>>>> /var/lib/ceph/crashxxxx directories, I attach to this message >>>>>>>>>>>> files, which I've found here. >>>>>>>>>>>> >>>>>>>>>>>> Please, can you advice, what now I can do? It seems, that rocksdb >>>>>>>>>>>> is even non-compatible or corrupted :-( >>>>>>>>>>>> >>>>>>>>>>>> Thanks in advance. >>>>>>>>>>>> >>>>>>>>>>>> Sincerely >>>>>>>>>>>> Jan Marek >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>>> -- >>>>>>>>>>> Igor Fedotov >>>>>>>>>>> Ceph Lead Developer >>>>>>>>>>> >>>>>>>>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>>>>>>>> >>>>>>>>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>>>>>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>>>>>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>>>>>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Igor Fedotov >>>>>>>>> Ceph Lead Developer >>>>>>>>> >>>>>>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>>>>>> >>>>>>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>>>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>>>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>>>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>> -- >>>>>>> Igor Fedotov >>>>>>> Ceph Lead Developer >>>>>>> >>>>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>>>> >>>>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>> -- >>>>>> Ing. Jan Marek >>>>>> University of South Bohemia >>>>>> Academic Computer Centre >>>>>> Phone: +420389032080 >>>>>> http://www.gnu.org/philosophy/no-word-attachments.cs.html >>>>> >>>>> >>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

mds crashes after up:replay state
by Lars Köppel 09 Jan '24

09 Jan '24

Hello everyone, we are running a small cluster with 3 nodes and 25 osds per node. And Ceph version 17.2.6. Recently the active mds crashed and since then the new starting mds has always been in the up:replay state. In the output of the command 'ceph tell mds.cephfs:0 status' you can see that the journal is completely read in. As soon as it's finished, the mds crashes and the next one starts reading the journal. At the moment I have the journal inspection running ('cephfs-journal-tool --rank=cephfs:0 journal inspect'). Does anyone have any further suggestions on how I can get the cluster running again as quickly as possible? Best regards Lars [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel(a)ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai

4 7

How to configure something like osd_deep_scrub_min_interval?
by Frank Schilder 09 Jan '24

09 Jan '24

Hi folks, I am fighting a bit with odd deep-scrub behavior on HDDs and discovered a likely cause of why the distribution of last_deep_scrub_stamps is so weird. I wrote a small script to extract a histogram of scrubs by "days not scrubbed" (more precisely, intervals not scrubbed; see code) to find out how (deep-) scrub times are distributed. Output below. What I expected is along the lines that HDD-OSDs try to scrub every 1-3 days, while they try to deep-scrub every 7-14 days. In other words, OSDs that have been deep-scrubbed within the last 7 days would *never* be in scrubbing+deep state. However, what I see is completely different. There seems to be no distinction between scrub- and deep-scrub start times. This is really unexpected as nobody would try to deep-scrub HDDs every day. Weekly to bi-weekly is normal, specifically for large drives. Is there a way to configure something like osd_deep_scrub_min_interval (no, I don't want to run cron jobs for scrubbing yet)? In the output below, I would like to be able to configure a minimum period of 1-2 weeks before the next deep-scrub happens. How can I do that? The observed behavior is very unusual for RAID systems (if its not a bug in the report script). With this behavior its not surprising that people complain about "not deep-scrubbed in time" messages and too high deep-scrub IO load when such a large percentage of OSDs is needlessly deep-scrubbed after 1-6 days again already. Sample output: # scrub-report dumped pgs Scrub report: 4121 PGs not scrubbed since 1 intervals (6h) 3831 PGs not scrubbed since 2 intervals (6h) 4012 PGs not scrubbed since 3 intervals (6h) 3986 PGs not scrubbed since 4 intervals (6h) 2998 PGs not scrubbed since 5 intervals (6h) 1488 PGs not scrubbed since 6 intervals (6h) 909 PGs not scrubbed since 7 intervals (6h) 771 PGs not scrubbed since 8 intervals (6h) 582 PGs not scrubbed since 9 intervals (6h) 2 scrubbing 431 PGs not scrubbed since 10 intervals (6h) 333 PGs not scrubbed since 11 intervals (6h) 1 scrubbing 265 PGs not scrubbed since 12 intervals (6h) 195 PGs not scrubbed since 13 intervals (6h) 116 PGs not scrubbed since 14 intervals (6h) 78 PGs not scrubbed since 15 intervals (6h) 1 scrubbing 72 PGs not scrubbed since 16 intervals (6h) 37 PGs not scrubbed since 17 intervals (6h) 5 PGs not scrubbed since 18 intervals (6h) 14.237* 19.5cd* 19.12cc* 19.1233* 14.40e* 33 PGs not scrubbed since 20 intervals (6h) 23 PGs not scrubbed since 21 intervals (6h) 16 PGs not scrubbed since 22 intervals (6h) 12 PGs not scrubbed since 23 intervals (6h) 8 PGs not scrubbed since 24 intervals (6h) 2 PGs not scrubbed since 25 intervals (6h) 19.eef* 19.bb3* 4 PGs not scrubbed since 26 intervals (6h) 19.b4c* 19.10b8* 19.f13* 14.1ed* 5 PGs not scrubbed since 27 intervals (6h) 19.43f* 19.231* 19.1dbe* 19.1788* 19.16c0* 6 PGs not scrubbed since 28 intervals (6h) 2 PGs not scrubbed since 30 intervals (6h) 19.10f6* 14.9d* 3 PGs not scrubbed since 31 intervals (6h) 19.1322* 19.1318* 8.a* 1 PGs not scrubbed since 32 intervals (6h) 19.133f* 1 PGs not scrubbed since 33 intervals (6h) 19.1103* 3 PGs not scrubbed since 36 intervals (6h) 19.19cc* 19.12f4* 19.248* 1 PGs not scrubbed since 39 intervals (6h) 19.1984* 1 PGs not scrubbed since 41 intervals (6h) 14.449* 1 PGs not scrubbed since 44 intervals (6h) 19.179f* Deep-scrub report: 3723 PGs not deep-scrubbed since 1 intervals (24h) 4621 PGs not deep-scrubbed since 2 intervals (24h) 8 scrubbing+deep 3588 PGs not deep-scrubbed since 3 intervals (24h) 8 scrubbing+deep 2929 PGs not deep-scrubbed since 4 intervals (24h) 3 scrubbing+deep 1705 PGs not deep-scrubbed since 5 intervals (24h) 4 scrubbing+deep 1904 PGs not deep-scrubbed since 6 intervals (24h) 5 scrubbing+deep 1540 PGs not deep-scrubbed since 7 intervals (24h) 7 scrubbing+deep 1304 PGs not deep-scrubbed since 8 intervals (24h) 7 scrubbing+deep 923 PGs not deep-scrubbed since 9 intervals (24h) 5 scrubbing+deep 557 PGs not deep-scrubbed since 10 intervals (24h) 7 scrubbing+deep 501 PGs not deep-scrubbed since 11 intervals (24h) 2 scrubbing+deep 363 PGs not deep-scrubbed since 12 intervals (24h) 2 scrubbing+deep 377 PGs not deep-scrubbed since 13 intervals (24h) 1 scrubbing+deep 383 PGs not deep-scrubbed since 14 intervals (24h) 2 scrubbing+deep 252 PGs not deep-scrubbed since 15 intervals (24h) 2 scrubbing+deep 116 PGs not deep-scrubbed since 16 intervals (24h) 5 scrubbing+deep 47 PGs not deep-scrubbed since 17 intervals (24h) 2 scrubbing+deep 10 PGs not deep-scrubbed since 18 intervals (24h) 2 PGs not deep-scrubbed since 19 intervals (24h) 19.1c6c* 19.a01* 1 PGs not deep-scrubbed since 20 intervals (24h) 14.1ed* 2 PGs not deep-scrubbed since 21 intervals (24h) 19.1322* 19.10f6* 1 PGs not deep-scrubbed since 23 intervals (24h) 19.19cc* 1 PGs not deep-scrubbed since 24 intervals (24h) 19.179f* PGs marked with a * are on busy OSDs and not eligible for scrubbing. The script (pasted here because attaching doesn't work): # cat bin/scrub-report #!/bin/bash # Compute last scrub interval count. Scrub interval 6h, deep-scrub interval 24h. # Print how many PGs have not been (deep-)scrubbed since #intervals. ceph -f json pg dump pgs 2>&1 > /root/.cache/ceph/pgs_dump.json echo "" T0="$(date +%s)" scrub_info="$(jq --arg T0 "$T0" -rc '.pg_stats[] | [ .pgid, (.last_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*6)|ceil), (.last_deep_scrub_stamp[:19]+"Z" | (($T0|tonumber) - fromdateiso8601)/(60*60*24)|ceil), .state, (.acting | join(" ")) ] | @tsv ' /root/.cache/ceph/pgs_dump.json)" # less <<<"$scrub_info" # 1 2 3 4 5..NF # pg_id scrub-ints deep-scrub-ints status acting[] awk <<<"$scrub_info" '{ for(i=5; i<=NF; ++i) pg_osds[$1]=pg_osds[$1] " " $i if($4 == "active+clean") { si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 } else if($4 ~ /scrubbing\+deep/) { deep_scrubbing[$3]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else if($4 ~ /scrubbing/) { scrubbing[$2]++ for(i=5; i<=NF; ++i) osd[$i]="busy" } else { unclean[$2]++ unclean_d[$3]++ si_mx=si_mx<$2 ? $2 : si_mx dsi_mx=dsi_mx<$3 ? $3 : dsi_mx pg_sn[$2]++ pg_sn_ids[$2]=pg_sn_ids[$2] " " $1 pg_dsn[$3]++ pg_dsn_ids[$3]=pg_dsn_ids[$3] " " $1 for(i=5; i<=NF; ++i) osd[$i]="busy" } } END { print "Scrub report:" for(si=1; si<=si_mx; ++si) { if(pg_sn[si]==0 && scrubbing[si]==0 && unclean[si]==0) continue; printf("%7d PGs not scrubbed since %2d intervals (6h)", pg_sn[si], si) if(scrubbing[si]) printf(" %d scrubbing", scrubbing[si]) if(unclean[si]) printf(" %d unclean", unclean[si]) if(pg_sn[si]<=5) { split(pg_sn_ids[si], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "Deep-scrub report:" for(dsi=1; dsi<=dsi_mx; ++dsi) { if(pg_dsn[dsi]==0 && deep_scrubbing[dsi]==0 && unclean_d[dsi]==0) continue; printf("%7d PGs not deep-scrubbed since %2d intervals (24h)", pg_dsn[dsi], dsi) if(deep_scrubbing[dsi]) printf(" %d scrubbing+deep", deep_scrubbing[dsi]) if(unclean_d[dsi]) printf(" %d unclean", unclean_d[dsi]) if(pg_dsn[dsi]<=5) { split(pg_dsn_ids[dsi], pgs) osds_busy=0 for(pg in pgs) { split(pg_osds[pgs[pg]], osds) for(o in osds) if(osd[osds[o]]=="busy") osds_busy=1 if(osds_busy) printf(" %s*", pgs[pg]) if(!osds_busy) printf(" %s", pgs[pg]) } } printf("\n") } print "" print "PGs marked with a * are on busy OSDs and not eligible for scrubbing." } ' Don't forget the last "'" when copy-pasting. Thanks for any pointers. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 6

Re: Stuck in upgrade process to reef
by Igor Fedotov 09 Jan '24

09 Jan '24

Hi Marek, I haven't looked through those upgrade logs yet but here are some comments regarding last OSD startup attempt. First of answering your question >_init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) >Is it a mandatory part of fsck? This is caused by previous non-graceful OSD process shutdown. BlueStore is unable to find up-to-date allocation map and recovers it from RocksDB. And since fsck is a read-only procedure the recovered allocmap is not saved - hence all the following BlueStore startups (within fsck or OSD init) cause another rebuild attempt. To avoid that you might want to run repair instead of fsck - this will persist up-to-date allocation map and avoid its rebuilding on the next startup. This will work till the next non-graceful shutdown only - hence unsuccessful OSD attempt might break the allocmap state again. Secondly - looking at OSD startup log one can see that actual OSD log ends with that allocmap recovery as well: > 2024-01-09T11:25:30.718449+01:00 osd1 ceph-osd[1734062]: bluestore(/var/lib/ceph/osd/ceph-1) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... Subsequent log line indicating OSD daemon termination is from systemd: > 2024-01-09T11:25:33.516258+01:00 osd1 systemd[1]: Stopping ceph-2c565e24-7850-47dc-a751-a6357cbbaf2a(a)osd.1.service - Ceph osd.1 for 2c565e24-7850-47dc-a751-a6357cbbaf2a... And honestly these lines provide almost no clue why termination happened. No obvious OSD failures or something are shown. Perhaps containerized environment hides the details e.g. by cutting off OSD log's tail. So you might want to proceed the investigation by running repair prior to starting the OSD as per above. This will result in no alloc map recovery and hopefully workaround the problem during startup - if the issue is caused by allocmap recovery. Additionally you might want to increase debug_bluestore log level for osd.1 before starting it up to get more insight on what's happening. Alternatively you might want to play with OSD log target settings to write OSD.1 log to some file rather than using system wide logging infra - hopefully this will be more helpful. Thanks, Igor On 09/01/2024 13:31, Jan Marek wrote: > Hi Igor, > > I've sent you logs via filesender.cesnet.cz, if someone would > be interested, they are here: > > https://filesender.cesnet.cz/?s=download&token=047b1ec4-4df0-4e8a-90fc-3170… > > Some points: > > 1) I've found, that on the osd1 server was bad time (3 minutes in > future). I've corrected that. Yes, I know, that it's bad, but we > moved servers to any other net segment, where they have no access > to the timeservers in Internet, then I must reconfigure it to use > our own NTP servers. > > 2) I've tried to start osd.1 service by this sequence: > > a) > > ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.1 --command fsck > > (without setting log properly :-( ) > > b) > > export CEPH_ARGS="--log-file osd.1.log --debug-bluestore 5/20" > ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.1 --command fsck > > - here I have one question: Why is it in this log stil this line: > > _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) > > Is it a mandatory part of fsck? > > Log is attached. > > c) > > systemctl start ceph-2c565e24-7850-47dc-a751-a6357cbbaf2a(a)osd.1.service > > still crashing, gzip-ed log attached too. > > Many thanks for exploring problem. > > Sincerely > Jan Marek > > Dne Po, led 08, 2024 at 12:00:05 CET napsal(a) Igor Fedotov: >> Hi Jan, >> >> indeed fsck logs for the OSDs other than osd.0 look good so it would be >> interesting to see OSD startup logs for them. Preferably to have that for >> multiple (e.g. 3-4) OSDs to get the pattern. >> >> Original upgrade log(s) would be nice to see as well. >> >> You might want to use Google Drive or any other publicly available file >> sharing site for that. >> >> >> Thanks, >> >> Igor >> >> On 05/01/2024 10:25, Jan Marek wrote: >>> Hi Igor, >>> >>> I've tried to start only osd.1, which seems to be fsck'd OK, but >>> it crashed :-( >>> >>> I search logs and I've found, that I have logs from 22.12.2023, >>> when I've did a upgrade (I have set logging to journald). >>> >>> Would you be interested in those logs? This file have 30MB in >>> bzip2 format, how I can share it with you? >>> >>> It contains crash log from start osd.1 too, but I can cut out >>> from it and send it to list... >>> >>> Sincerely >>> Jan Marek >>> >>> Dne Čt, led 04, 2024 at 02:43:48 CET napsal(a) Jan Marek: >>>> Hi Igor, >>>> >>>> I've ran this oneliner: >>>> >>>> for i in {0..12}; do export CEPH_ARGS="--log-file osd."${i}".log --debug-bluestore 5/20" ; ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.${i} --command fsck ; done; >>>> >>>> On osd.0 it crashed very quickly, on osd.1 it is still working. >>>> >>>> I've send those logs in one e-mail. >>>> >>>> But! >>>> >>>> I've tried to list disk devices in monitor view, and I've got >>>> very interesting screenshot - some part I've emphasized by red >>>> rectangulars. >>>> >>>> I've got a json from syslog, which was as a part cephadm call, >>>> where it seems to be correct (for my eyes). >>>> >>>> Can be this coincidence for this problem? >>>> >>>> Sincerely >>>> Jan Marek >>>> >>>> Dne Čt, led 04, 2024 at 12:32:47 CET napsal(a) Igor Fedotov: >>>>> Hi Jan, >>>>> >>>>> may I see the fsck logs from all the failing OSDs to see the pattern. IIUC >>>>> the full node is suffering from the issue, right? >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Igor >>>>> >>>>> On 1/2/2024 10:53 AM, Jan Marek wrote: >>>>>> Hello once again, >>>>>> >>>>>> I've tried this: >>>>>> >>>>>> export CEPH_ARGS="--log-file /tmp/osd.0.log --debug-bluestore 5/20" >>>>>> ceph-bluestore-tool --path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0 --command fsck >>>>>> >>>>>> And I've sending /tmp/osd.0.log file attached. >>>>>> >>>>>> Sincerely >>>>>> Jan Marek >>>>>> >>>>>> Dne Ne, pro 31, 2023 at 12:38:13 CET napsal(a) Igor Fedotov: >>>>>>> Hi Jan, >>>>>>> >>>>>>> this doesn't look like RocksDB corruption but rather like some BlueStore >>>>>>> metadata inconsistency. Also assertion backtrace in the new log looks >>>>>>> completely different from the original one. So in an attempt to find any >>>>>>> systematic pattern I'd suggest to run fsck with verbose logging for every >>>>>>> failing OSD. Relevant command line: >>>>>>> >>>>>>> CEPH_ARGS="--log-file osd.N.log --debug-bluestore 5/20" >>>>>>> bin/ceph-bluestore-tool --path <path-to-osd> --command fsck >>>>>>> >>>>>>> Unlikely this will fix anything it's rather a way to collect logs to get >>>>>>> better insight. >>>>>>> >>>>>>> >>>>>>> Additionally you might want to run similar fsck for a couple of healthy OSDs >>>>>>> - curious if it succeeds as I have a feeling that the problem with crashing >>>>>>> OSDs had been hidden before the upgrade and revealed rather than caused by >>>>>>> it. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Igor >>>>>>> >>>>>>> On 12/29/2023 3:28 PM, Jan Marek wrote: >>>>>>>> Hello Igor, >>>>>>>> >>>>>>>> I'm attaching a part of syslog creating while starting OSD.0. >>>>>>>> >>>>>>>> Many thanks for help. >>>>>>>> >>>>>>>> Sincerely >>>>>>>> Jan Marek >>>>>>>> >>>>>>>> Dne St, pro 27, 2023 at 04:42:56 CET napsal(a) Igor Fedotov: >>>>>>>>> Hi Jan, >>>>>>>>> >>>>>>>>> IIUC the attached log is for ceph-kvstore-tool, right? >>>>>>>>> >>>>>>>>> Can you please share full OSD startup log as well? >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Igor >>>>>>>>> >>>>>>>>> On 12/27/2023 4:30 PM, Jan Marek wrote: >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I've problem: my ceph cluster (3x mon nodes, 6x osd nodes, every >>>>>>>>>> osd node have 12 rotational disk and one NVMe device for >>>>>>>>>> bluestore DB). CEPH is installed by ceph orchestrator and have >>>>>>>>>> bluefs storage on osd. >>>>>>>>>> >>>>>>>>>> I've started process upgrade from version 17.2.6 to 18.2.1 by >>>>>>>>>> invocating: >>>>>>>>>> >>>>>>>>>> ceph orch upgrade start --ceph-version 18.2.1 >>>>>>>>>> >>>>>>>>>> After upgrade of mon and mgr processes orchestrator tried to >>>>>>>>>> upgrade the first OSD node, but they are falling down. >>>>>>>>>> >>>>>>>>>> I've stop the process of upgrade, but I have 1 osd node >>>>>>>>>> completely down. >>>>>>>>>> >>>>>>>>>> After upgrade I've got some error messages and I've found >>>>>>>>>> /var/lib/ceph/crashxxxx directories, I attach to this message >>>>>>>>>> files, which I've found here. >>>>>>>>>> >>>>>>>>>> Please, can you advice, what now I can do? It seems, that rocksdb >>>>>>>>>> is even non-compatible or corrupted :-( >>>>>>>>>> >>>>>>>>>> Thanks in advance. >>>>>>>>>> >>>>>>>>>> Sincerely >>>>>>>>>> Jan Marek >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>> -- >>>>>>>>> Igor Fedotov >>>>>>>>> Ceph Lead Developer >>>>>>>>> >>>>>>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>>>>>> >>>>>>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>>>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>>>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>>>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>>>>>> >>>>>>> -- >>>>>>> Igor Fedotov >>>>>>> Ceph Lead Developer >>>>>>> >>>>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>>>> >>>>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> -- >>>>> Igor Fedotov >>>>> Ceph Lead Developer >>>>> >>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>> >>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> -- >>>> Ing. Jan Marek >>>> University of South Bohemia >>>> Academic Computer Centre >>>> Phone: +420389032080 >>>> http://www.gnu.org/philosophy/no-word-attachments.cs.html >>> >>> >>> >>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

REST API Endpoint Failure - Request For Where To Look To Resolve
by duluxoz 09 Jan '24

09 Jan '24

Hi All, A little help please. TL/DR: Please help with error message: ~~~ REST API failure, code : 500 Unable to access the configuration object Unable to contact the local API endpoint (https://localhost:5000/api) ~~~ The Issue ------------ I've been through the documentation and can't find what I'm looking for - possibly because I'm not really sure what it is I *am* looking for, so if someone can point me in the right direction I would really appreciate it. I get the above error message when I run the `gwcli` command from inside a cephadm shell. What I'm trying to do is set up a set of iSCSI Gateways in our Ceph-Reef 18.2.1 Cluster (yes, I know its being depreciated as of Nov 22 - or whatever). We recently migrated 7 upgraded from a manual install of Quincy to a CephAdm install of Reef - everything went AOK *except* for the iSCSI Gateways. So we tore them down and then rebuilt them as per the latest documentation. So now we've got 3 gateways as per the Service page of the Dashboard and I'm trying to create the targets. I tried via the Dashboard but had errors, so instead I went in to do it via gwcli and hit the above error (which I now bevel to be the cause of the GUI creation I encountered. I have absolutely no experience with podman or containers in general, and can't work out how to fix the issue. So I'm requesting some help - not to solve the problem for me, but to point me in the right direction to solve it myself. :-) So, anyone? Cheers Dulux-Oz

2 9

Re: rbd persistent cache configuration
by Peter 08 Jan '24

08 Jan '24

rbd --version ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)

2 1

Radosgw not syncing files/folders with slashes in object name
by Matt Dunavant 08 Jan '24

08 Jan '24

Hi all, I need some help troubleshooting a strange issue where my relatively newly setup 2 ceph clusters (17.2.6) are setup for replication and I can get files with just regular names (example.txt for example) to sync but anything with a slash or a folder type (folder1/folder2/example.txt for example) won't sync over. Not sure exactly why this would be the case as I'm pretty sure slashes are allowed in object names, https://docs.ceph.com/en/latest/radosgw/layout/. Any ideas or something obvious I'm missing? Sync status looks normal and I have tested this with a variety of new and old buckets and the behavior always stays the same, nothing with a slash syncs but everything without does. Thanks in advance, -Matt Dunavant

1 0

Network Flapping Causing Slow Ops and Freezing VMs
by mahnoosh shahidi 08 Jan '24

08 Jan '24

Hi all, I hope this message finds you well. We recently encountered an issue on one of our OSD servers, leading to network flapping and subsequently causing significant performance degradation across our entire cluster. Although the OSDs were correctly marked as down in the monitor, slow ops persisted until we resolved the network issue. This incident resulted in a major disruption, especially affecting VMs with mapped RBD images, leading to their freezing. In light of this, I have two key questions for the community: 1. Why did slow ops persist even after marking the affected server as down in the monitor? 2.Are there any recommended configurations for OSD suicide or OSD down reports that could help us better handle similar network-related issues in the future? Best Regards, Mahnoosh

3 5

cephadm bootstrap on 3 network clusters
by Luis Domingues 08 Jan '24

08 Jan '24

Hi, I am bootstrapping a ceph cluster using cephadm, and our cluster uses 3 networks. We have - 1 network as public network (10.X.X.0/24) (pub) - 1 network as cluster network (10.X.Y.0/24) (cluster) - 1 network for management (172.Z.Z.0/24) (mgmt) The nodes are reachable using SSH only on mgmt network. However, they are reachable for our services using pub network. I want my MONs to be bind to this pub network. But when I bootstrap my cluster, I set my MON IP and CLUSTER NETWORK, and then the bootstrap process tries to add my bootstrap node using the MON IP. And then fails because it cannot reach the node. If I apply proper spec after it works fine, but the bootstrap process did not finish properly. Is there an option to tell cephadm to not use MON IP but another one for accessing the node during the bootstrap? If I tell it --skip-prepare-host, it tries to connect to it anyway, and then fails. Thanks, Luis Domingues Proton AG

4 6