- ceph-users - lists.ceph.io

recommendation for barebones server with 8-12 direct attach NVMe?
by Drew Weaver 17 Jan '24

17 Jan '24

Hello, So we were going to replace a Ceph cluster with some hardware we had laying around using SATA HBAs but I was told that the only right way to build Ceph in 2023 is with direct attach NVMe. Does anyone have any recommendation for a 1U barebones server (we just drop in ram disks and cpus) with 8-10 2.5" NVMe bays that are direct attached to the motherboard without a bridge or HBA for Ceph specifically? Thanks, -Drew

7 19

Re: Stuck in upgrade process to reef
by Igor Fedotov 17 Jan '24

17 Jan '24

Hi Jan, I've just fired an upstream ticket for your case, see https://tracker.ceph.com/issues/64053 for more details. You might want to tune (or preferably just remove) your custom bluestore_cache_.*_ratio settings to fix the issue. This is reproducible and fixable in my lab this way. Hope this helps. Thanks, Igor On 15/01/2024 12:54, Jan Marek wrote: > Hi Igor, > > I've tried to start ceph-sod daemon as you advice me and I'm > sending log osd.1.start.log > > About memory: According to 'top' podman ceph daemon don't reach > 2% of whole server memory (64GB)... > > I have switch on autotune of memory... > > My ceph config dump - see attached dump.txt > > Sincerely > Jan Marek > > Dne Čt, led 11, 2024 at 04:02:02 CET napsal(a) Igor Fedotov: >> Hi Jan, >> >> unfortunately this wasn't very helpful. Moreover the log looks a bit messy - >> looks like a mixture of outputs from multiple running instances or >> something. I'm not an expert in using containerized setups though. >> >> Could you please simplify things by running ceph-osd process manually like >> you did for ceph-objectstore-tool. And enforce log output to a file. Command >> line should look somewhat the following: >> >> ceph-osd -i 0 --log-to-file --log-file <some-file> --debug-bluestore 5/20 >> --debug-prioritycache 10 >> >> Please don't forget to run repair prior to that. >> >> >> Also you haven't answered my questions about custom [memory] settings and >> RAM usage during OSD startup. It would be nice to hear some feedback. >> >> >> Thanks, >> >> Igor >> >> On 11/01/2024 16:47, Jan Marek wrote: >>> Hi Igor, >>> >>> I've tried to start osd.1 with debug_prioritycache and >>> debug_bluestore 5/20, see attached file... >>> >>> Sincerely >>> Jan >>> >>> Dne St, led 10, 2024 at 01:03:07 CET napsal(a) Igor Fedotov: >>>> Hi Jan, >>>> >>>> indeed this looks like some memory allocation problem - may be OSD's RAM >>>> usage threshold reached or something? >>>> >>>> Curious if you have any custom OSD settings or may be any memory caps for >>>> Ceph containers? >>>> >>>> Could you please set debug_bluestore to 5/20 and debug_prioritycache to 10 >>>> and try to start OSD once again. Please monitor process RAM usage along the >>>> process and share the resulting log. >>>> >>>> >>>> Thanks, >>>> >>>> Igor >>>> >>>> On 10/01/2024 11:20, Jan Marek wrote: >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Upgrading nautilus / centos7 to octopus / ubuntu 20.04. - Suggestions and hints?
by Götz Reinicke 17 Jan '24

17 Jan '24

Hi, As I’v read and thought a lot about the migration as this is a bigger project, I was wondering if anyone has done that already and might share some notes or playbooks, because in all readings there where some parts missing or miss understandable to me. I do have some different approaches in mind, so may be you have some suggestions or hints. a) upgrade nautilus on centos 7 with the few missing features like dashboard and prometheus. After that migrate one node after an other to ubuntu 20.04 with octopus and than upgrade ceph to the recent stable version. b) migrate one node after an other to ubuntu 18.04 with nautilus and then upgrade to octupus and after that to ubuntu 20.04. or c) upgrade one node after an other to ubuntu 20.04 with octopus and join it to the cluster until all nodes are upgraded. For test I tried c) with a mon node, but adding that to the cluster fails with some failed state, still probing for the other mons. (I dont have the right log at hand right now.) So my questions are: a) What would be the best (most stable) migration path and b) is it in general possible to add a new octopus mon (not upgraded one) to a nautilus cluster, where the other mons are still on nautilus? I hope my thoughts and questions are understandable :) Thanks for any hint and suggestion. Best . Götz

7 11

ceph pg mark_unfound_lost delete results in confused ceph
by Oliver Dzombic 16 Jan '24

16 Jan '24

Hi, after osd.15 died in the wrong moment there is: #ceph health detail [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg stale pg 10.17 is stuck stale for 3d, current state stale+active+undersized+degraded, last acting [15] [WRN] PG_DEGRADED: Degraded data redundancy: 172/57063399 objects degraded (0.000%), 1 pg degraded, 1 pg undersized pg 10.17 is stuck undersized for 3d, current state stale+active+undersized+degraded, last acting [15] which will never resolv as there is no osd.15 anymore. So a ceph pg 10.17 mark_unfound_lost delete was executed. ceph seems to be a bit confused about pg 10.17 now: While this worked before, its not working anymore # ceph pg 10.17 query Error ENOENT: i don't have pgid 10.17 And while this was pointing to 15 the map changed now to 5 and 6 ( which is correct ): # ceph pg map 10.17 osdmap e14425 pg 10.17 (10.17) -> up [5,6] acting [5,6] According to ceph health, ceph assumes that osd.15 is still somehow in charge. The pg map seems to think that 10.17 is on osd.5 and osd.6 But pg 10.17 seems not to be really existing, as a query will fail. Any idea whats going wrong and howto fix this? Thank you! -- Mit freundlichen Gruessen / Best regards Oliver Dzombic Layer7 Networks mailto:info@layer7.net Anschrift: Layer7 Networks GmbH Zum Sonnenberg 1-3 63571 Gelnhausen HRB 96293 beim Amtsgericht Hanau Geschäftsführung: Oliver Dzombic UST ID: DE259845632

1 1

[quincy 17.2.7] ceph orchestrator not doing anything
by Boris 16 Jan '24

16 Jan '24

Happy new year everybody. I just found out that the orchestrator in one of our clusters is not doing anything. What I tried until now: - disabling / enabling cephadm (no impact) - restarting hosts (no impact) - starting upgrade to same version (no impact) - starting downgrade (no impact) - forcefully removing hosts and adding them again (now I have no daemons anymore) - applying new configurations (no impact) The orchestrator just does nothing. Cluster itself is fine. I also checked the SSH connecability from all hosts to all hosts ( https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors) The logs always show a message like "took the task" but then nothing happens. Cheers Boris

2 2

Email duplicates.
by Roman Pashin 16 Jan '24

16 Jan '24

1 0

[v18.2.1] problem with wrong osd device symlinks after upgrade to 18.2.1
by Reto Gysi 16 Jan '24

16 Jan '24

Hi ceph community I noticed the following problem after upgrading my ceph instance on Debian 12.4 from 17.2.7 to 18.2.1: I had placed bluestore block.db for hdd osd's on raid1/mirrored logical volumes on 2 nvme devices, so that if a single block.db nvme device fails, that not all hdd osds fail. That worked fine under 17.2.7 and had no problems during host/osd restarts. During the upgrade to 18.2.1 the osd's wouldn't with the block.db on mirrored lv wouldn't start anymore because the block.db symlink was updated to pointing to the wrong device mapper device, and the osd startup failed with error message that block.db device is busy. OSD1: 2024-01-05T19:56:43.592+0000 7fdde9f43640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _minimal_open_bluefs add block device(/var/lib/ceph/osd/ceph-1/block.db) returned: (16) Device or resource busy 2024-01-05T19:56:43.592+0000 7fdde9f43640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _open_db failed to prepare db environment: 2024-01-05T19:56:43.592+0000 7fdde9f43640 1 bdev(0x55a2d5014000 /var/lib/ceph/osd/ceph-1/block) close 2024-01-05T19:56:43.892+0000 7fdde9f43640 -1 osd.1 0 OSD:init: unable to mount object store the symlink was updated to point to lrwxrwxrwx 1 ceph ceph 111 Jan 5 20:57 block -> /dev/mapper/ceph--dec5bd7c--d84f--40d9--ba14--6bd8aadf2957-osd--block--cdd02721--6876--4db8--bdb2--12ac6c70127c lrwxrwxrwx 1 ceph ceph 48 Jan 5 20:57 block.db -> /dev/mapper/optane-ceph--db--osd1_rimage_1_iorig the correct symlink would have been: lrwxrwxrwx 1 ceph ceph 111 Jan 5 20:57 block -> /dev/mapper/ceph--dec5bd7c--d84f--40d9--ba14--6bd8aadf2957-osd--block--cdd02721--6876--4db8--bdb2--12ac6c70127c lrwxrwxrwx 1 ceph ceph 48 Jan 5 20:57 block.db -> /dev/mapper/optane-ceph--db--osd1 To continue with the upgrade I converted one by one all the block.db lvm logical volumes back to linear volumes, and fixed the symlinks manually. converting the lv's back to linear was necessary, because even when I fixed the symlink manually, after a osd restart the symlink would be created wrong again if the block.db would point to a raid1 lv. Here's any example how the symlink looked before an osd was touched by the 18.2.1 upgrade: OSD2: lrwxrwxrwx 1 ceph ceph 93 Jan 4 03:38 block -> /dev/ceph-17a894d6-3a64-4e5e-9fa0-8dd3b5f4bf33/osd-block-3cd7a5af-9002-47a7-b4c2-540381d53be7 lrwxrwxrwx 1 ceph ceph 24 Jan 4 03:38 block.db -> /dev/optane/ceph-db-osd2 Here's what the output of lvs -a -o +devices looked like for OSD1 block.db device when it was an raid1 lv: LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices ceph-db-osd1 optane rwi-a-r--- 44.00g 100.00 ceph-db-osd1_rimage_0(0),ceph-db-osd1_rimage_1(0) [ceph-db-osd1_rimage_0] optane gwi-aor--- 44.00g [ceph-db-osd1_rimage_0_iorig] 100.00 ceph-db-osd1_rimage_0_iorig(0) [ceph-db-osd1_rimage_0_imeta] optane ewi-ao---- 428.00m /dev/sdg(55482) [ceph-db-osd1_rimage_0_imeta] optane ewi-ao---- 428.00m /dev/sdg(84566) [ceph-db-osd1_rimage_0_iorig] optane -wi-ao---- 44.00g /dev/sdg(9216) [ceph-db-osd1_rimage_0_iorig] optane -wi-ao---- 44.00g /dev/sdg(82518) [ceph-db-osd1_rimage_1] optane gwi-aor--- 44.00g [ceph-db-osd1_rimage_1_iorig] 100.00 ceph-db-osd1_rimage_1_iorig(0) [ceph-db-osd1_rimage_1_imeta] optane ewi-ao---- 428.00m /dev/sdj(55392) [ceph-db-osd1_rimage_1_imeta] optane ewi-ao---- 428.00m /dev/sdj(75457) [ceph-db-osd1_rimage_1_iorig] optane -wi-ao---- 44.00g /dev/sdj(9218) [ceph-db-osd1_rimage_1_iorig] optane -wi-ao---- 44.00g /dev/sdj(73409) [ceph-db-osd1_rmeta_0] optane ewi-aor--- 4.00m /dev/sdg(55388) [ceph-db-osd1_rmeta_1] optane ewi-aor--- 4.00m /dev/sdj(9217) It would be good if the symlinks were recreated pointing to the correct device even when they point to a raid1 lv. Not sure if this problem has been reported yet. Cheers Reto

2 4

Recomand number of k and m erasure code
by Phong Tran Thanh 16 Jan '24

16 Jan '24

Hi ceph user! I need to determine which erasure code values (k and m) to choose for a Ceph cluster with 10 nodes. I am using the reef version with rbd. Furthermore, when using a larger k, for example, ec6+2 and ec4+2, which erasure coding performance is better, and what are the criteria for choosing the appropriate erasure coding? Please help me Email: tranphong079(a)gmail.com Skype: tranphong079

3 4

RGW - user created bucket with name of already created bucket
by Ondřej Kukla 15 Jan '24

15 Jan '24

Hello, I would like to share a quite worrying experience I’ve just found on one of my production clusters. User successfully created a bucket with name of a bucket that already exists! He is not bucket owner - the original user is, but he is able to see it when he does ListBuckets over s3 api. (Both accounts are able to do it now - only the original owner is able to interact with it) This bucket is also counted to the new users usage stats. Has anyone noticed this before? This cluster is running on Quincy - 17.2.6. Is there a way to detach the bucket from the new owner so he doesn’t have a bucket that doesn’t belong to him? Regards, Ondrej

2 3

3 DC with 4+5 EC not quite working
by Torkil Svensgaard 14 Jan '24

14 Jan '24

We are looking to create a 3 datacenter 4+5 erasure coded pool but can't quite get it to work. Ceph version 17.2.7. These are the hosts (there will eventually be 6 hdd hosts in each datacenter): -33 886.00842 datacenter 714 -7 209.93135 host ceph-hdd1 -69 69.86389 host ceph-flash1 -6 188.09579 host ceph-hdd2 -3 233.57649 host ceph-hdd3 -12 184.54091 host ceph-hdd4 -34 824.47168 datacenter DCN -73 69.86389 host ceph-flash2 -2 201.78067 host ceph-hdd5 -81 288.26501 host ceph-hdd6 -31 264.56207 host ceph-hdd7 -36 1284.48621 datacenter TBA -77 69.86389 host ceph-flash3 -21 190.83224 host ceph-hdd8 -29 199.08838 host ceph-hdd9 -11 193.85382 host ceph-hdd10 -9 237.28154 host ceph-hdd11 -26 187.19536 host ceph-hdd12 -4 206.37102 host ceph-hdd13 We did this: ceph osd erasure-code-profile set DRCMR_k4m5_datacenter_hdd plugin=jerasure k=4 m=5 technique=reed_sol_van crush-root=default crush-failure-domain=datacenter crush-device-class=hdd ceph osd pool create cephfs.hdd.data erasure DRCMR_k4m5_datacenter_hdd ceph osd pool set cephfs.hdd.data allow_ec_overwrites true ceph osd pool set cephfs.hdd.data pg_autoscale_mode warn Didn't quite work: " [WARN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg incomplete pg 33.0 is creating+incomplete, acting [104,219,NONE,NONE,NONE,41,NONE,NONE,NONE] (reducing pool cephfs.hdd.data min_size from 5 may help; search ceph.com/docs for 'incomplete') " I then manually changed the crush rule from this: " rule cephfs.hdd.data { id 7 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type datacenter step emit } " To this: " rule cephfs.hdd.data { id 7 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type datacenter step chooseleaf indep 3 type host step emit } " Based on some testing and dialogue I had with Red Hat support last year when we were on RHCS, and it seemed to work. Then: ceph fs add_data_pool cephfs cephfs.hdd.data ceph fs subvolumegroup create hdd --pool_layout cephfs.hdd.data I started copying data to the subvolume and increased pg_num a couple of times: ceph osd pool set cephfs.hdd.data pg_num 256 ceph osd pool set cephfs.hdd.data pg_num 2048 But at some point it failed to activate new PGs eventually leading to this: " [WARN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.cephfs.ceph-flash1.agdajf(mds.0): 64 slow metadata IOs are blocked > 30 secs, oldest blocked for 25455 secs [WARN] MDS_TRIM: 1 MDSs behind on trimming mds.cephfs.ceph-flash1.agdajf(mds.0): Behind on trimming (997/128) max_segments: 128, num_segments: 997 [WARN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive pg 33.6f6 is stuck inactive for 8h, current state activating+remapped, last acting [50,79,116,299,98,219,164,124,421] pg 33.6fa is stuck inactive for 11h, current state activating+undersized+degraded+remapped, last acting [17,408,NONE,196,223,290,73,39,11] pg 33.705 is stuck inactive for 11h, current state activating+undersized+degraded+remapped, last acting [33,273,71,NONE,411,96,28,7,161] pg 33.721 is stuck inactive for 7h, current state activating+remapped, last acting [283,150,209,423,103,325,118,142,87] pg 33.726 is stuck inactive for 11h, current state activating+undersized+degraded+remapped, last acting [234,NONE,416,121,54,141,277,265,19] [WARN] PG_DEGRADED: Degraded data redundancy: 1818/1282640036 objects degraded (0.000%), 3 pgs degraded, 3 pgs undersized pg 33.6fa is stuck undersized for 7h, current state activating+undersized+degraded+remapped, last acting [17,408,NONE,196,223,290,73,39,11] pg 33.705 is stuck undersized for 7h, current state activating+undersized+degraded+remapped, last acting [33,273,71,NONE,411,96,28,7,161] pg 33.726 is stuck undersized for 7h, current state activating+undersized+degraded+remapped, last acting [234,NONE,416,121,54,141,277,265,19] [WARN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups Pool cephfs.hdd.data has 1024 placement groups, should have 2048 [WARN] SLOW_OPS: 3925 slow ops, oldest one blocked for 26012 sec, daemons [osd.17,osd.234,osd.283,osd.33,osd.50] have slow ops. " We had thought it would only affect that particular data pool but eventually everything ground to a halt, also RBD on unrelated replicated pools. Looking at it now I guess that was because the 5 OSDs were blocked for everything and not just the PGs for that data pool? We tried restarting the 5 blocked OSDs to no avail and eventually resorted to deleting the cephfs.hdd.data data pool to restore service. Any suggestions as to what we did wrong? Something to do with min_size? The crush rule? Thanks. Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

4 5