March 2024 - ceph-users - lists.ceph.io

Robust cephfs design/best practice
by Szabo, Istvan (Agoda) 15 Mar '24

15 Mar '24

Hi, I'd like to add cephfs to our production objectstore/block storage cluster so I'd like to collect hands on experiences like, good to know/be careful/avoid etc ... other than ceph documentation. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 2

RGW - tracking new bucket creation and bucket usage
by Ondřej Kukla 15 Mar '24

15 Mar '24

Hello I’m looking for suggestions how to track bucket creation over s3 api and bucket usage (num of objects and size) of all buckets in time. In our RGW setup, we have a custom client panel, where like 85% percent of buckets are created which is easy for us to then track the newly created buckets and assigned them to our clients, however we also enable them to create the buckets using the s3 api and there the tracking bucket to client is harder. We also want to periodically track the bucket usage stats (num of objects and size) of all buckets in time. Now we are using the GetBucketInfo from the AdminOPS api - https://docs.ceph.com/en/quincy/radosgw/adminops/#id44 with the stats=true option GET /admin/bucket?stats=1 which returns all buckets with the number of objects and size we then parse. We also use it for the tracking of newly created buckets as it gives you all of the buckets with their owners at one place, but we are not using the metadata in that case. This however in our environment, where we have close to ~400 buckets isn’t scalable and the calls can take from 10 to 400 to even 6000s to complete. So I’m looking for suggestions on how to get this data in time better/smarter. I know there is a bunch of prometheus exporters like https://github.com/pcuzner/rgw-exporter but they are usually just using the same AdminOPS api so that doesn’t seem like a better solution. I’m also working on sending all of our RGW ops logs to elastic which would give me the ability to easily query the CreateBucket calls, but that only solves the bucket creation and not the usage stats. How would you handle this? Do you have any suggestions? Thanks a lot. Ondrej

3 2

[REEF][cephadm] new cluster all pg unknown
by wodel youchi 15 Mar '24

15 Mar '24

Hi, I am creating a new ceph cluster using REEF. This is my host_specs file [root@controllera config]# cat hosts-specs2.yml service_type: host hostname: computehci01 addr: 20.1.0.2 location: chassis: chassis1 --- service_type: host hostname: computehci02 addr: 20.1.0.3 location: chassis: chassis1 --- service_type: host hostname: computehci03 addr: 20.1.0.4 location: chassis: chassis1 --- service_type: host hostname: computehci04 addr: 20.1.0.5 location: chassis: chassis2 --- service_type: host hostname: computehci05 addr: 20.1.0.6 location: chassis: chassis2 --- service_type: host hostname: computehci06 addr: 20.1.0.7 location: chassis: chassis2 --- service_type: host hostname: computehci07 addr: 20.1.0.8 location: chassis: chassis3 --- service_type: host hostname: computehci08 addr: 20.1.0.9 location: chassis: chassis3 --- service_type: host hostname: computehci09 addr: 20.1.0.10 location: chassis: chassis3 --- service_type: host hostname: computehci10 addr: 20.1.0.11 location: chassis: chassis3 --- service_type: host hostname: computehci11 addr: 20.1.0.12 location: chassis: chassis4 --- service_type: host hostname: computehci12 addr: 20.1.0.13 location: chassis: chassis4 --- service_type: host hostname: computehci13 addr: 20.1.0.14 location: chassis: chassis4 --- service_type: host hostname: computehci14 addr: 20.1.0.15 location: chassis: chassis4 --- service_type: host hostname: computehci15 addr: 20.1.0.16 location: chassis: chassis5 --- service_type: host hostname: computehci16 addr: 20.1.0.17 location: chassis: chassis5 --- service_type: host hostname: computehci17 addr: 20.1.0.18 location: chassis: chassis5 --- service_type: host hostname: computehci18 addr: 20.1.0.19 location: chassis: chassis5 --- service_type: host hostname: computehci19 addr: 20.1.0.20 location: chassis: chassis6 --- service_type: host hostname: computehci20 addr: 20.1.0.21 location: chassis: chassis6 --- service_type: host hostname: computehci21 addr: 20.1.0.22 location: chassis: chassis6 --- service_type: host hostname: computehci22 addr: 20.1.0.24 location: chassis: chassis7 --- service_type: host hostname: computehci23 addr: 20.1.0.25 location: chassis: chassis7 --- service_type: host hostname: computehci24 addr: 20.1.0.26 location: chassis: chassis7 --- service_type: host hostname: computehci25 addr: 20.1.0.28 location: chassis: chassis8 --- service_type: host hostname: computehci26 addr: 20.1.0.29 location: chassis: chassis8 --- service_type: host hostname: computehci27 addr: 20.1.0.30 location: chassis: chassis8 --- service_type: host hostname: controllera addr: 20.1.0.23 --- service_type: host hostname: controllerb addr: 20.1.0.27 --- service_type: host hostname: controllerc addr: 20.1.0.31 --- service_type: mon placement: hosts: - controllera - controllerb - controllerc --- service_type: mgr placement: hosts: - controllera - controllerb - controllerc --- service_type: osd service_id: default_drive_group placement: hosts: - computehci01 - computehci02 - computehci03 - computehci04 - computehci05 - computehci06 - computehci07 - computehci08 - computehci09 - computehci10 - computehci11 - computehci12 - computehci13 - computehci14 - computehci15 - computehci16 - computehci17 - computehci18 - computehci19 - computehci20 - computehci21 - computehci22 - computehci23 - computehci24 - computehci25 - computehci26 - computehci27 spec: data_devices: rotational: 0 All osds were added but, pg still unknown state I've created a pool, but it didn't change anything. [root@controllerb ~]# ceph -s cluster: id: be250ade-e1f2-11ee-a6ff-3cecef2872f0 health: HEALTH_WARN Reduced data availability: 1 pg inactive services: mon: 3 daemons, quorum controllera,controllerc,controllerb (age 3h) mgr: controllerc.jevbkl(active, since 21s), standbys: controllera.zwlolp, controllerb.vqkdga osd: 108 osds: 108 up (since 2m), 108 in (since 24m) data: pools: 2 pools, 33 pgs objects: 0 objects, 0 B usage: 5.1 GiB used, 330 TiB / 330 TiB avail * pgs: 100.000% pgs unknown 33 unknown* Did I miss something? Regards.

2 6

PSA: CephFS/MDS config defer_client_eviction_on_laggy_osds
by Venky Shankar 15 Mar '24

15 Mar '24

If you are using CephFS on Pacific v16.2.14(+), the MDS config `defer_client_eviction_on_laggy_osds' is enabled by default. This config is used to not evict cephfs clients if OSDs are laggy[1]. However, this can result in a single client holding up the MDS in servicing other clients. To avoid this, please disable the config by executing ceph config set mds defer_client_eviction_on_laggy_osds false This issue doesn't affect any Quincy or Reef releases. Furthermore, the config is being disabled by default[2] till the fix is being worked on. [1]: https://tracker.ceph.com/issues/58023 [2]: https://tracker.ceph.com/issues/64685 -- Cheers, Venky

1 0

Ceph Dashboard Clear Cache
by ashar.khan＠rapidcompute.com 15 Mar '24

15 Mar '24

Hello Ceph members, How do I clear the Ceph dashboard cache? Kindly guide me on how to do this. Thanks

1 0

Re: ceph metrics units
by Konstantin Shalygin 14 Mar '24

14 Mar '24

Hi, > On 14 Mar 2024, at 19:29, Denis Polom <denispolom(a)gmail.com> wrote: > > so metric itself is miliseconds and after division on _count it's in seconds? > > This is two metrics for long running averages [1], the query that produces "seconds" unit looks like this (irate(ceph_osd_op_r_latency_sum[1m]) / irate(ceph_osd_op_r_latency_count[1m])) * on (instance, ceph_daemon) group_left (device) ceph_disk_occupation * on (instance, ceph_daemon) group_left (device_class, hostname) ceph_osd_metadata k [1] https://docs.ceph.com/en/latest/mgr/prometheus/?highlight=%3Cname%3E_sum#st…

1 0

ceph metrics units
by Denis Polom 14 Mar '24

14 Mar '24

Hi guys, do you know if there is some table of Ceph metrics and units that should be used for them? I currently struggling with ceph_osd_op_r_latency_sum ceph_osd_op_w_latency_sum if they are in ms or seconds? Any idea please? Thx!

2 1

Fwd: Ceph fs snapshot problem
by Marcus 14 Mar '24

14 Mar '24

Hi all, I have just setup a small ceph cluster with ceph fs. The setup is reef 18.2.1 on Debian bookworm. The system is up and running the way it should, though I have a problem with ceph fs snapshots. When I read the doc I should be able to make a snapshot in any directory in the filesystem. I can do a snapshot in the root of the filesystem but if I try somewhere else I get: Operation not permitted This is the same if I do it with mkdir or with ceph fs subvolume snapshot create ... I have created an auth client with rws: [client.snap-mount] key = **** caps mds = "allow rws fsname=gds-common" caps mon = "allow r fsname=gds-common" caps osd = "allow rw tag cephfs data=gds-common" Where the filsystem is called gds-common, saved in a file on the client: /etc/ceph/ceph.client.snap-mount.keyring I mount ceph fs with: mount -t ceph :/ -o name=snap-mount /mnt If I create a snapshot in root, it works fine, as in: mkdir /mnt/.snap/mysnap I also notice that in every subdir there is a "snapshot dir" as well with the name _mysnap_1, as in: /mnt/dir/.snap/_mysnap_1 My guess that is is a part of the snapshot system, this "snapshot" dissapear when the snapshot is removed with: rmdir /mnt/.snap/mysnap If I try to make a snapshot in another directory this does not work: mkdir /mnt/dir/.snap/othersnap Get the error: cannot create directory ‘/mnt/dir/.snap/othersnap’: Operation not permitted It is the same thing on the commandline, root works: ceph fs subvolume snapshot create gds-common / fromcmd But not in a subdir: ceph fs subvolume snapshot create gds-common /dir dirsnap Error EINVAL: invalid value specified for ceph.dir.subvolume I also notice that when you use any command of type: ceph fs subvolume snapshot ... You get a new directory (volumes) in the root: /mnt/volumes/_legacy/6666cd76f96956469e7be39d750cc7d9.meta I do not know if I am missing something, some lacking of config or so. Thanks for your help!! Best regards Marcus

1 0

18.2.2 dashboard really messed up.
by Harry G Coin 14 Mar '24

14 Mar '24

Looking at ceph -s, all is well. Looking at the dashboard, 85% of my capacity is 'warned', and 95% is 'in danger'. There is no hint given as to the nature of the danger or reason for the warning. Though apparently with merely 5% of my ceph world 'normal', the cluster reports 'ok'. Which, you know, seems contradictory. I've used just under 40% of capacity. Further down the dashboard, all the subsections of 'Cluster Utilization' are '1' and '0.5' with nothing whatever in the graphics area. Previous versions of ceph presented a normal dashboard. It's just a little half rack, 5 hosts, a few physical drives each, been running ceph for a couple years now. Orchestrator is cephadm. It's just about as 'plain vanilla' at it gets. I've had to mute one alert, because cephadm refresh aborts when it finds drives on any host that have nothing to do with ceph that don't have a blkid_ip 'TYPE' key. Seems unrelated to a totally messed up dashboard. (The tracker for that is here: https://tracker.ceph.com/issues/63502 ). Any idea what the steps are to get useful stuff back on the dashboard? Any idea where I can learn what my 85% danger and 95% warning is 'about'? (You'd think 'danger' (The volcano is blowing up now!) would be worse than 'warning' (the volcano might blow up soon) , so how can warning+danger > 100%, or if not additive how can warning < danger?) Here's a bit of detail: root@noc1:~# ceph -s cluster: id: 4067126d-01cb-40af-824a-881c130140f8 health: HEALTH_OK (muted: CEPHADM_REFRESH_FAILED) services: mon: 5 daemons, quorum noc4,noc2,noc1,noc3,sysmon1 (age 70m) mgr: noc2.yhyuxd(active, since 82m), standbys: noc4.tvhgac, noc3.sybsfb, noc1.jtteqg mds: 1/1 daemons up, 3 standby osd: 27 osds: 27 up (since 20m), 27 in (since 2d) data: volumes: 1/1 healthy pools: 16 pools, 1809 pgs objects: 12.29M objects, 17 TiB usage: 44 TiB used, 67 TiB / 111 TiB avail pgs: 1793 active+clean 9 active+clean+scrubbing 7 active+clean+scrubbing+deep io: client: 5.6 MiB/s rd, 273 KiB/s wr, 41 op/s rd, 58 op/s wr

2 3

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

bluestore_min_alloc_size and bluefs_shared_alloc_size
by Joel Davidow 13 Mar '24

13 Mar '24

Summary ---------- The relationship of the values configured for bluestore_min_alloc_size and bluefs_shared_alloc_size are reported to impact space amplification, partial overwrites in erasure coded pools, and storage capacity as an osd becomes more fragmented and/or more full. Previous discussions including this topic ---------------------------------------- comment #7 in bug 63618 in Dec 2023 - https://tracker.ceph.com/issues/63618#note-7 pad writeup related to bug 62282 likely from late 2023 - https://pad.ceph.com/p/RCA_62282 email sent 13 Sept 2023 in mail list discussion of cannot create new osd - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/5M4QAXJDCN… comment #9 in bug 58530 likely from early 2023 - https://tracker.ceph.com/issues/58530#note-9 email sent 30 Sept 2021 in mail list discussion of flapping osds - https://www.mail-archive.com/ceph-users@ceph.io/msg13072.html email sent 25 Feb 2020 in mail list discussion of changing allocation size - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/B3DGKH6THFG… Current situation ----------------- We have three Ceph clusters that were originally built via cephadm on octopus and later upgraded to pacific. All osds are HDD (will be moving to wal+db on SSD) and were resharded after the upgrade to enable rocksdb sharding. The value for bluefs_shared_alloc_size has remained unchanged at 65535. The value for bluestore_min_alloc_size_hdd was 65535 in octopus but is reported as 4096 by ceph daemon osd.<id> config show in pacific. However, the osd label after upgrading to pacific retains the value of 65535 for bfm_bytes_per_block. BitmapFreelistManager.h in Ceph source code (src/os/bluestore/BitmapFreelistManager.h) indicates that bytes_per_block is bdev_block_size. This indicates that the physical layout of the osd has not changed from 65535 despite the return of the ceph dameon command reporting it as 4096. This interpretation is supported by the Minimum Allocation Size part of the Bluestore configuration reference for quincy (https://docs.ceph.com/en/quincy/rados/configuration/bluestore-config-ref/#m…) Questions ---------- What are the pros and cons of the following three cases with two variations per case - when using co-located wal+db on HDD and when using separate wal+db on SSD: 1) bluefs_shared_alloc_size, bluestore_min_alloc_size, and bfm_bytes_per_block all equal 2) bluefs_shared_alloc_size greater than but a multiple of bluestore_min_alloc_size with bfm_bytes_per_block equal to bluestore_min_alloc_size 3) bluefs_shared_alloc_size greater than but a multiple of bluestore_min_alloc_size with bfm_bytes_per_block equal to bluefs_shared_alloc_size

4 6

ceph osd different size to create a cluster for Openstack : asking for advice
by wodel youchi 13 Mar '24

13 Mar '24

Hi, I need some guidance from you folks... I am going to deploy a ceph cluster in HCI mode for an openstack platform. My hardware will be : - 03 control nodes : - 27 osd nodes : each node has 03x3.8To nvme + 01x1.9To nvme disks (those disks will all be used as OSDs) In my Openstack I will be creating all sorts of pools : RBD, Cephfs and RGW. I am planning to create two crush rules using the disk size as a parameter. Then divide my pools between the two rules. - RBD to use the 3.8To disks since I need more space here. - Cephfs and RGW to use 1.9To disks. Is this a good configuration? Regards

2 1

ceph osd crush reweight rounding issue
by Stefan Kooman 13 Mar '24

13 Mar '24

Hi, After some tests with OSD crush weight I wanted to restore the original weight of the OSD. But that proves to be difficult. See the following example: Situation before OSD crush reweight has taken place ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -11 0.39197 datacenter dc1 -5 0.39197 host node1 2 hdd 0.09798 osd.2 up 1.00000 1.00000 3 hdd 0.09798 osd.3 up 1.00000 1.00000 6 ssd 0.09798 osd.6 up 1.00000 1.00000 7 ssd 0.09798 osd.7 up 1.00000 1.00000 -10 0.39197 datacenter dc2 -3 0.39197 host node2 0 hdd 0.09798 osd.0 up 1.00000 1.00000 1 hdd 0.09798 osd.1 up 1.00000 1.00000 4 ssd 0.09798 osd.4 up 1.00000 1.00000 5 ssd 0.09798 osd.5 up 1.00000 1.00000 -1 0 root default ceph osd crush reweight osd.5 0.1 ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -11 0.39197 datacenter dc1 -5 0.39197 host node1 2 hdd 0.09798 osd.2 up 1.00000 1.00000 3 hdd 0.09798 osd.3 up 1.00000 1.00000 6 ssd 0.09798 osd.6 up 1.00000 1.00000 7 ssd 0.09798 osd.7 up 1.00000 1.00000 -10 0.39392 datacenter dc2 -3 0.39392 host node2 0 hdd 0.09798 osd.0 up 1.00000 1.00000 1 hdd 0.09798 osd.1 up 1.00000 1.00000 4 ssd 0.09798 osd.4 up 1.00000 1.00000 5 ssd 0.09999 osd.5 up 1.00000 1.00000 -1 0 root default ceph osd crush reweight osd.5 0.09798 reweighted item id 5 name 'osd.5' to 0.09798 in crush map ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -11 0.39197 datacenter dc1 -5 0.39197 host node1 2 hdd 0.09798 osd.2 up 1.00000 1.00000 3 hdd 0.09798 osd.3 up 1.00000 1.00000 6 ssd 0.09798 osd.6 up 1.00000 1.00000 7 ssd 0.09798 osd.7 up 1.00000 1.00000 -10 0.39191 datacenter dc2 -3 0.39191 host dc2 0 hdd 0.09798 osd.0 up 1.00000 1.00000 1 hdd 0.09798 osd.1 up 1.00000 1.00000 4 ssd 0.09798 osd.4 up 1.00000 1.00000 5 ssd 0.09798 osd.5 up 1.00000 1.00000 -1 0 root default The difference is (0.39197 - 0.09798) = .00006. If we divide this by four OSDs per host bucket it is .000015. So each OSD could be reweighted to 0.097995 to give the host / data center buckets equal weights. This indeed works, after reweighting all OSDs with 0.097995 -10 0.39197 datacenter dc2 -3 0.39197 host node2 0 hdd 0.09799 osd.0 up 1.00000 1.00000 1 hdd 0.09799 osd.1 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -1 0 root default Note however that the OSD weights now differ from (identical) OSDs in the other datacenter. Why is this such a big deal? Well, Ceph "stretch mode" requires each bucket to be of equal weight. And IMHO it should be easily possible to set the weight Ceph CRUSH internally uses to do its calculations. Anyone has more info on how to do this? Gr. Stefan P.s. Sure, I could reweight all OSDs to a new weight, but that's more of a workaround than a proper way of fixing it.

1 0

Hanging request in S3
by Christian Kugler 12 Mar '24

12 Mar '24

Hi, I am having some trouble with some S3 requests and I am at a loss. After upgrading to reef a couple of weeks ago some requests get stuck and never return. The two Ceph clusters are set up to sync the S3 realm bidirectionally. The bucket has 479 shards (dynamic resharding) at the moment. Putting an object (/etc/services) into the bucket via s3cmd works, and deleting it works as well. So I know it is not just the entire bucket that is somehow faulty. When I try to delete a specific prefix it the request for listing all objects never comes back. In the example below I only included the request in question which I aborted with ^C. $ s3cmd rm -r s3://sql20/pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/ -d [...snip...] DEBUG: Canonical Request: GET /sql20/ prefix=pgbackrest%2Fbackup%2Fadrpb%2F20240130-200410F%2Fpg_data%2Fbase%2F16560%2F host:[...snip...] x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20240306T183435Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20240306T183435Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=VL0FRB7CYGMHBGCD419M/20240306/[...snip...]/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=45b133675535ab611bbf2b9a7a6e40f9f510c0774bf155091dc9a05b76856cb7', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(sql20): [...snip...] DEBUG: ConnMan.get(): re-using connection: [...snip...]#1 DEBUG: format_uri(): /sql20/?prefix=pgbackrest%2Fbackup%2Fadrpb%2F20240130-200410F%2Fpg_data%2Fbase%2F16560%2F DEBUG: Sending request method_string='GET', uri='/sql20/?prefix=pgbackrest%2Fbackup%2Fadrpb%2F20240130-200410F%2Fpg_data%2Fbase%2F16560%2F', headers={'x-amz-date': '20240306T183435Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=VL0FRB7CYGMHBGCD419M/20240306/[...snip...]/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=45b133675535ab611bbf2b9a7a6e40f9f510c0774bf155091dc9a05b76856cb7', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) ^CDEBUG: Response: {} See ya! The request did not show up normally in the logs so I set debug_rgw=20 and debug_ms=20 via ceph config set. I tried to isolate the request and looked for its request id: 13321243250692796422 The following is a grep for the request id: Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket verifying op params Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket pre-executing Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket check rate limiting Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket executing Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket list_objects_ordered: starting attempt 1 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket cls_bucket_list_ordered: request from each of 479 shard(s) for 8 entries to get 1001 total entries Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101438318.gz from shard 437 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101438318.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: skipping pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101438318.gz[] Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457659_fsm.gz from shard 202 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457659_fsm.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: skipping pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457659_fsm.gz[] Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457662_fsm.gz from shard 420 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457662_fsm.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket manifest: total_size = 59 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: setting s->obj_tag to 107ace7a-a829-4d1c-9cb8-9db30644b786.395658.17352021385636272460 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: got pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457662_fsm.gz Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457721.gz from shard 444 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457721.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: skipping pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457721.gz[] Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458889.gz from shard 116 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458889.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket manifest: total_size = 1040 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: setting s->obj_tag to 107ace7a-a829-4d1c-9cb8-9db30644b786.395658.2135820855302089522 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: got pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458889.gz Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458903.gz from shard 452 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458903.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: skipping pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458903.gz[] Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458955.gz from shard 55 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458955.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cls_bucket_list_ordered: skipping pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458955.gz[] Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101600036.gz from shard 455 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101600036.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket manifest: total_size = 4403 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket get_obj_state: setting s->obj_tag to 107ace7a-a829-4d1c-9cb8-9db30644b786.395658.13169705319307666922 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cls_bucket_list_ordered: got pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101600036.gz Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101600044.gz from shard 407 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101600044.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket manifest: total_size = 6264856 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket get_obj_state: setting s->obj_tag to 107ace7a-a829-4d1c-9cb8-9db30644b786.395658.12884446303569321109 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket bucket index object: rechenzentrum.rgw.buckets.index:.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3.1.34 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cache get: name=rechenzentrum.rgw.log++bucket.sync-source-hints.sql20 : hit (negative entry) Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cache get: name=rechenzentrum.rgw.log++bucket.sync-target-hints.sql20 : hit (negative entry) Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket reflect(): flow manager (bucket=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3): adding source pipe: {s={b=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3,z=3caabb9a-4e3b-4b8a-8222-34c33dd63210,az=0},d={b=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3,z=107ace7a-a829-4d1c-9cb8-9db30644b786,az=0}} Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket reflect(): flow manager (bucket=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3): adding dest pipe: {s={b=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3,z=107ace7a-a829-4d1c-9cb8-9db30644b786,az=0},d={b=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3,z=3caabb9a-4e3b-4b8a-8222-34c33dd63210,az=0}} Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=3caabb9a-4e3b-4b8a-8222-34c33dd63210,az=0},d={b=*,z=107ace7a-a829-4d1c-9cb8-9db30644b786,az=0}} Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=107ace7a-a829-4d1c-9cb8-9db30644b786,az=0},d={b=*,z=3caabb9a-4e3b-4b8a-8222-34c33dd63210,az=0}} Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket chain_cache_entry: cache_locator= Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket chain_cache_entry: couldn't find cache locator Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket couldn't put bucket_sync_policy cache entry, might have raced with data changes Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket RGWDataChangesLog::add_entry() bucket.name=sql20 shard_id=34 now=2024-03-06T18:36:17.978389+0000 cur_expiration=1970-01-01T00:00:00.000000+0000 I don't see any clear error but somehow the last view lines are odd to me: - When before it said: flow manager (bucket=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3) it has no more bucket: flow manager (bucket=) - no cache locator found. No idea if this is okay or not - The cur_expiration a few lines later is set to unix time 0 (1970-01-01T00:00:00.000000+0000) - I did this multiple times and it seems to always be shard 34 that has the issue Did someone see something like this before? Any ideas how to remedy the situation or at least where to or what to look for? Best, Christian

2 2

Ceph-Cluster integration with Ovirt-Cluster
by ankit＠eurus.net 12 Mar '24

12 Mar '24

Hi Guys, I have a running ovirt-4.3 cluster with 1 manager and 4 hypervisors nodes and for storage using traditional SAN storage which is connect using iscsi. where i can create VM's and assign storge from SAN. This is running fine since a decade but now i want to move from traditional SAN storage to ceph storage cluster due to slow speed and scalability issues of storage. But i am a very newbie in ceph but able to install ceph-cluster reef_v18 in my lab environment with 5 VM's (2 Mon's and Manager and 3 OSD's) on Debian 12. And i have installed ovirt 4.5 with 1 hypervisor on VM's on centos stream 9 and want to integrate it with ceph-cluster so that i can use ceph-cluster storage like i am using SAN with my ovirt and I have tried to do some google and read ceph documentation but did not find anything regarding this. So how can i do it? If you need any other information, please let me know! Many Thanks! PJ111288

1 0

Elasticsearch sync module | Ceph Issue
by Lokendra Rathour 12 Mar '24

12 Mar '24

Hi Team, We are working on Elasticsearch sync module integration with ceph. Ceph version: 18.2.5 (reef) Elasticsearch: 8.2.1 Problem statement: The Syncing between the zones are not happenning: Links followed to perform the integration: https://ceph.io/en/news/blog/2017/new-luminous-rgw-metadata-search/ https://www.suse.com/c/rgw-elasticsearch/ Current status of radosgw-admin sync status: realm 1fc3fe5f-2bb8-4134-8200-4b26479f7709 (gold) zonegroup 59d8be05-0a31-4610-8f7f-d80a6d221f2e (zone-zg) zone ce1e53a5-5f30-42c5-bcca-a69b60411e1a (zone) current time 2024-03-11T02:31:17Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) data sync source: e012b80d-cb04-4ab2-b78f-13b1dea0156d (zone-es) not syncing from zone Configuration of ceph.conf: [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = d5b53cd2-bef4-41a3-8667-135a3e02fc25 mon host = [v2:[fd00:fd00:fd00:9900::34]:3300,v1:[fd00:fd00:fd00:9900::34]:6789],[v2:[fd00:fd00:fd00:9900::35]:3300,v1:[fd00:fd00:fd00:9900::35]:6789],[v2:[fd00:fd00:fd00:9900::36]:3300,v1:[fd00:fd00:fd00:9900::36]:6789] public network = fd00:fd00:fd00:9900::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 31380104806 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[fd00:fd00:fd00:9900::34]:8080 rgw thread pool size = 512 [client.rgw.zone] rgw_frontends="beast port=8000" rgw_zone=zone host = storagenode1 keyring = /etc/ceph/ceph.client.zone.keyring log file = /var/log/radosgw/rgw.zone.radosgw.log [client.rgw.zone-es] rgw_frontends="beast port=8002" rgw_zone=zone-es host = storagenode1 keyring = /etc/ceph/ceph.client.zone-es.keyring log file = /var/log/radosgw/rgw.zone-es.radosgw.log Could anyone please help. -- ~ Lokendra skype: lokendrarathour

1 0

General best practice for stripe unit and count if I want to change object size
by Nathan Morrison 11 Mar '24

11 Mar '24

This was asked in reddit and was requested to post here: So in RBD, say I want to make an image that's got an object size of 1M instead of the default 4M (if it will be a VM say, and likely not have too many big files in it, just OS files mostly). I also know I don't wanna go too crazy and make 4k objects or the cluster will blow up with number of objects. What's a good rule of thumb on what to set the stripe unit and count in relation to object size? Also how can I see the stripe unit and count for an image, it seems "rbd info <img-name>" doesn't show it, only object size (or order). Would this be sensible (assuming old img is default 4M obj size) or really stupid and why? rbd cp --object-size 1M --stripe-unit 1M --stripe-count 1 pool/old-img pool/new-img I'm trying to get my head around striping and I tried using this page (https://docs.ceph.com/en/latest/dev/file-striping/) to understand what's going on low level, but it's just leaving me more confused the more I read it.

2 1

Telemetry endpoint down?
by Konstantin Shalygin 11 Mar '24

11 Mar '24

Hi, seems telemetry endpoint is down for a some days? We have connection errors from multiple places 1:ERROR Mar 10 00:46:10.653 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 2:ERROR Mar 10 01:48:20.061 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 3:ERROR Mar 10 02:50:29.473 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 4:ERROR Mar 10 03:52:38.877 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 5:ERROR Mar 10 04:54:48.285 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 6:ERROR Mar 10 05:56:57.693 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 7:ERROR Mar 10 06:59:07.105 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 8:ERROR Mar 10 08:01:16.509 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 9:ERROR Mar 10 09:03:25.917 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 <http://telemetry.ceph.com:443/> Thanks, k

2 2

AMPQS support in Nautilus
by Manuel Negron 11 Mar '24

11 Mar '24

Hello, ive been trying to setup bucket notifications using amqs (according to the documentation is possible) however when checking the logs the push fails saying "invalid schema". any ideas wha the issue could be ? kind regards

2 1

v18.2.2 Reef (hot-fix) released
by Yuri Weinstein 11 Mar '24

11 Mar '24

We're happy to announce the 2nd hotfix release in the Reef series. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2024/v18-2-2-reef-released/ Notable Changes --------------- * mgr/Prometheus: refine the orchestrator availability check to prevent against crashes in the prometheus module during startup. Introduce additional checks to handle daemon_ids generated within the Rook environment, thus preventing potential issues during RGW metrics metadata generation. Related tracker: https://tracker.ceph.com/issues/64721 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.2.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2

1 0

MANY_OBJECT_PER_PG on 1 pool which is cephfs_metadata
by Edouard FAZENDA 11 Mar '24

11 Mar '24

Hello Ceph community, I have since this morning a warning about MANY_OBJECT_PER_PG on 1 pool which is cephfs_metadata # ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average [WRN] MANY_OBJECTS_PER_PG: 1 pools have many more objects per pg than average pool cephfs_metadata objects per pg (154151) is more than 10.0215 times cluster average (15382) I have the autoscaling on on all the pool : # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE device_health_metrics 9523k 3.0 26827G 0.0000 1.0 1 on cephfs_data 5389G 2.0 26827G 0.4018 1.0 512 on cephfs_metadata 19365M 2.0 26827G 0.0014 4.0 16 on .rgw.root 1323 3.0 26827G 0.0000 1.0 32 on default.rgw.log 23552 3.0 26827G 0.0000 1.0 32 on default.rgw.control 0 3.0 26827G 0.0000 1.0 32 on default.rgw.meta 11911 3.0 26827G 0.0000 4.0 8 on default.rgw.buckets.index 0 3.0 26827G 0.0000 4.0 8 on default.rgw.buckets.data 497.0G 3.0 26827G 0.0556 1.0 32 on kubernetes 177.2G 2.0 26827G 0.0132 1.0 32 on default.rgw.buckets.non-ec 432 3.0 26827G 0.0000 1.0 32 on Actually the pg_num is 16 for the cephfs_metdata pool , but it does not define NEW_PG_NUM Here the replicated size of all my pool # ceph osd dump | grep 'replicated size' pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 189372 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth pool 10 'cephfs_data' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 189346 lfor 0/0/183690 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs pool 11 'cephfs_metadata' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 187861 lfor 0/187861/187859 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 18 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 5265 flags hashpspool stripe_width 0 application rgw pool 19 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 5267 flags hashpspool stripe_width 0 application rgw pool 20 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 5269 flags hashpspool stripe_width 0 application rgw pool 21 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 5398 lfor 0/5398/5396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 22 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 7491 lfor 0/7491/7489 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 23 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 7500 flags hashpspool stripe_width 0 application rgw pool 24 'kubernetes' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 189363 lfor 0/0/7560 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 25 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 23983 flags hashpspool stripe_width 0 application rgw Why the autoscaler is not acting to increase the pg_num of the pool in warning ? As the pgcalc is not available on ceph now , do you thing it is a good idea to increase manually the pg_num of the cephfs_metadata , but which value should I set ? I have 18 OSD Thanks for the help Best Regards, Edouard FAZENDA Technical Support Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 <https://www.csti.ch/> www.csti.ch

3 2

Dashboard building issue "RuntimeError: memory access out of bounds"?
by 张东川 11 Mar '24

11 Mar '24

Hi there, I was building ceph with the tag "v19.0.0" on Milkv Pioneer board (RISCV arch, OS is fedora-riscv 6.1.55) I ran "./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_SPDK=ON" then just went to the "build" folder and ran "ninja" command. But it failed with the following dashboard error: [1/2] cd /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend && . /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/node-env/bin/activate && npm config set cache /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/node-env/.npm --userconfig /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/node-env/.npmrc && deactivate [2/2] dashboard frontend is being created FAILED: src/pybind/mgr/dashboard/frontend/dist /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/dist cd /mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend && . /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/node-env/bin/activate && DASHBOARD_FRONTEND_LANGS="" npm run build:localize -- --output-path /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/dist --configuration=production --progress=false && deactivate > ceph-dashboard(a)0.0.0 build:localize > node cd --env --pre && ng build --localize --output-path /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/dist --configuration=production --progress=false [cd.js] './angular.backup.json' already exists, restoring it into './angular.json'} [cd.js] Preparing build of EN. [cd.js] 'src/environments/environment.tpl.ts' was copied to 'src/environments/environment.prod.ts' [cd.js] 'src/environments/environment.tpl.ts' was copied to 'src/environments/environment.ts' [cd.js] Writing to ./angular.json [cd.js] Placeholders were replace in 'src/environments/environment.prod.ts' [cd.js] Placeholders were replace in 'src/environments/environment.ts'     TypeScript compiler options "target" and "useDefineForClassFields" are set to "ES2022" and "false" respectively by the Angular CLI. To control ECMA version and features use the Browerslist configuration. For more information, see https://angular.io/guide/build#configuring-browser-compatibility     NOTE: You can set the "target" to "ES2022" in the project's tsconfig to remove this warning. wasm://wasm/7f04b08a:1 RuntimeError: memory access out of bounds     at wasm://wasm/7f04b08a:wasm-function[2]:0x322     at WasmHash.digest (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/webpack/lib/util/hash/wasm-hash.js:138:11)     at BatchedHash.digest (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/webpack/lib/util/hash/BatchedHash.js:64:20)     at /mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/webpack/lib/DefinePlugin.js:595:38     at _next42 (eval at create (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/tapable/lib/HookCodeFactory.js:19:10), <anonymous>:36:1)     at _next20 (eval at create (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/tapable/lib/HookCodeFactory.js:19:10), <anonymous>:97:1)     at Hook.eval [as call] (eval at create (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/tapable/lib/HookCodeFactory.js:19:10), <anonymous>:141:1)     at Hook.CALL_DELEGATE [as _call] (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/tapable/lib/Hook.js:14:14)     at Compiler.newCompilation (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/webpack/lib/Compiler.js:1122:26)     at /mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/webpack/lib/Compiler.js:1166:29     at eval (eval at create (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:13:1) Node.js v18.17.0 ninja: build stopped: subcommand failed. Has anyone met this same issue before? How to fix it??? Please let me know if further info is needed.Thanks a lot. (I will try to build on the latest mainline code tonight to see if the same issue still exists) Best Regards, Dongchuan

1 0

Adding new OSD's - slow_ops and other issues.
by jskr＠topsoe.com 11 Mar '24

11 Mar '24

Hi. We have a cluster working very nicely since it was put up more than a year ago. Now we needed to add more NVMe drives to expand. After setting all the "no" flags.. we added them using $ ceph orch osd add .... The twist is that we have managed to get the default weights set to 1 for all disks not 7.68 (as the default for the ceph orch command. Thus we did a subsequent reweight to change weight -- and then removed the "no" flags. As a consequence we had a bunch of OSD's delivering slow_ops and -- after manually restarting osd's to get rid of them - the system returned to normal. ... second try... Same drill - but somehow the ceph orch command failed to bring the new OSD online before we ran the reweight command ... and it works flawlessly ... third try ... Same drill - but now ceph orch brought the new OSD into the system - and we saw excactly the same problem again. Being a bit wiser - we forcefully restarted the new OSD.. and everything whet back into normal mode again. Thus it seems like the "reweight" command on online OSD's have a bad effect on our setup - causing major service disruption. 1) Is it possible to "bulk" change default weights on all OSD's without a huge data movement going on? 2) or Is it possible to instrurct "ceph orch osd add" to set default weight before it putting the new OSD into the system? I would not expect above to be expected behaviour - if someone has ideas about what goes on more than above please share? Setup: # ceph version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 43 7.68 TB NVMe's over 12 OSD hosts - all connected using 2x 100GbitE Thanks Jesper

1 0

Connect to Ceph Cluster from other network
by Ramin Najjarbashi 10 Mar '24

10 Mar '24

Hello Ceph community, I am currently facing an issue connecting to my Ceph cluster from an external system and would appreciate your guidance. Here are some details about my setup: - Ceph Public Network: 172.0.0.1/24 - Ceph Private Network: 172.10.0.1/24 - Monitor (MON) Nodes: - mon1: 172.0.3.1 - mon2: 172.0.3.2 - mon3: 172.0.3.3 My network node has an internet IP (1.1.1.1), and I'm using HAProxy with the following configuration: ```bash backend admin_mons_v2 mode tcp server mon1 172.0.3.1:3300 check server mon2 172.0.3.2:3300 check server mon3 172.0.3.3:3300 check backend admin_mons_v1 mode tcp server mon1 172.0.3.1:6789 check server mon2 172.0.3.2:6789 check server mon3 172.0.3.3:6789 check frontend mon_v1 bind *:6789 mode tcp use_backend admin_mons_v1 frontend mon_v2 bind *:3300 mode tcp use_backend admin_mons_v2 ``` I've created a new authentication for my local system (IP: 2.2.2.2) using the following command: ```bash ceph auth get-or-create client.gateway mon 'allow * network 0.0.0.0/0' ``` And I've created a configuration file on my local system as follows: ```ini [global] fsid = MY_CLUSTER_FSID mon host = [v1:1.1.1.1:3300,v2:1.1.1.1:6789] ``` However, when attempting to connect to the Ceph cluster using the `ceph` or `radosgw-admin` commands, I encounter a connection timeout issue: ```bash 2024-03-10T20:00:10.551+0000 ffff7a214d38 0 monclient(hunting): authenticate timed out after 300 ``` Interestingly, I can connect to ports 3300 and 6789 using `telnet` from my local system: ```bash telnet 1.1.1.1 3300 Trying 1.1.1.1... Connected to 1.1.1.1. Escape character is '^]'. ceph v2 telnet 1.1.1.1 6789 Trying 1.1.1.1... Connected to 1.1.1.1. Escape character is '^]'. ceph v027?? ``` I would appreciate any insights or guidance on how to successfully connect to my Ceph cluster from my local system. Thank you for your time and assistance! Best regards,

1 0

PG damaged "failed_repair"
by Romain Lebbadi-Breteau 10 Mar '24

10 Mar '24

Hi, We're a student club from Montréal where we host an Openstack cloud with a Ceph backend for storage of virtual machines and volumes using rbd. Two weeks ago we received an email from our ceph cluster saying that some pages were damaged. We ran "sudo ceph pg repair <pg-id>" but then there was an I/O error on the disk during the recovery ("An unrecoverable disk media error occurred on Disk 4 in Backplane 1 of Integrated RAID Controller 1." and "Bad block medium error is detected at block 0x1377e2ad on Virtual Disk 3 on Integrated RAID Controller 1." messages on iDRAC). After that, the PG we tried to repair was in the state "active+recovery_unfound+degraded". After a week, we ran the command "sudo ceph pg 2.1b mark_unfound_lost revert" to try to recover the damaged PG. We tried to boot the virtual machine that had crashed because of this incident, but the volume seemed to have been completely erased, the "mount" command said there was no filesystem on it, so we recreated the VM from a backup. A few days later, the same PG was once again damaged, and since we knew the physical disk on the OSD hosting one part of the PG had problems, we tried to "out" the OSD from the cluster. That resulted in the two other OSDs hosting copies of the problematic PG to go down, which caused timeouts on our virtual machines, so we put the OSD back in. We then tried to repair the PG again, but that failed and the PG is now "active+clean+inconsistent+failed_repair", and whenever it goes down, two other OSDs from two other hosts go down too after a few minutes, so it's impossible to replace the disk right now, even if we have new ones available. We have backups for most of our services, but it would be very disrupting to delete the whole cluster, and we don't know that to do with the broken PG and the OSD that can't be shut down. Any help would be really appreciated, we're not experts with Ceph and Openstack, and it's likely we handled things wrong at some point, but we really want to go back to a healthy Ceph. Here are some information about our cluster : romain:step@alpha-cen ~ $ sudo ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 1 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 2.1b is active+clean+inconsistent+failed_repair, acting [3,11,0] romain:step@alpha-cen ~ $ sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 70.94226 root default -7 20.00792 host alpha-cen 3 hdd 1.81879 osd.3 up 1.00000 1.00000 6 hdd 1.81879 osd.6 up 1.00000 1.00000 12 hdd 1.81879 osd.12 up 1.00000 1.00000 13 hdd 1.81879 osd.13 up 1.00000 1.00000 15 hdd 1.81879 osd.15 up 1.00000 1.00000 16 hdd 9.09520 osd.16 up 1.00000 1.00000 17 hdd 1.81879 osd.17 up 1.00000 1.00000 -5 23.64874 host beta-cen 1 hdd 5.45749 osd.1 up 1.00000 1.00000 4 hdd 5.45749 osd.4 up 1.00000 1.00000 8 hdd 5.45749 osd.8 up 1.00000 1.00000 11 hdd 5.45749 osd.11 up 1.00000 1.00000 14 hdd 1.81879 osd.14 up 1.00000 1.00000 -3 27.28560 host gamma-cen 0 hdd 9.09520 osd.0 up 1.00000 1.00000 5 hdd 9.09520 osd.5 up 1.00000 1.00000 9 hdd 9.09520 osd.9 up 1.00000 1.00000 romain:step@alpha-cen ~ $ sudo rados list-inconsistent-obj 2.1b {"epoch":9787,"inconsistents":[]} romain:step@alpha-cen ~ $ sudo ceph pg 2.1b query https://pastebin.com/gsKCPCjr Best regards, Romain Lebbadi-Breteau

3 6

PGs increasing number
by Michel Niyoyita 09 Mar '24

09 Mar '24

Hello team, I have increased my volumes pool which was 128 PGs to 256 PGs , the activity started yesterday 5PM , It started when it was 5.733% of misplaced object , after 4 to 5 hours it reaches to 5.022 % after that it come back to the initial percentage 5.733% , kindly help to solve the issue. I am using Ceph Pacific deployed using ceph-ansible with ubuntu OS. the cluster is in production with openstack hypervisor. Best regards Michel

3 4

Which RHEL/Fusion/CentOS/Rocky Package Contains cephfs-shell?
by duluxoz 09 Mar '24

09 Mar '24

Hi All, The subject pretty much says it all: I need to use cephfs-shell and its not installed on my Ceph Node, and I can't seem to locate which package contains it - help please. :-) Cheers Dulux-Oz

2 3

CephFS On Windows 10
by duluxoz 09 Mar '24

09 Mar '24

Hi All, I'm looking for some pointers/help as to why I can't get my Win10 PC to connect to our Ceph Cluster's CephFS Service. Details are as follows: Ceph Cluster: - IP Addresses: 192.168.1.10, 192.168.1.11, 192.168.1.12 - Each node above is a monitor & an MDS - Firewall Ports: open (ie 3300, etc) - CephFS System Name: my_cephfs - Log files: nothing jumps out at me Windows PC: - Keyring file created and findable: ceph.client.me.keyring - dokany installed - ceph-for-windows installed - Can ping all three ceph nodes - Connection command: ceph-dokan -l v -o -id me --debug --client_fs my_cephfs -c C:\ProgramData\Ceph\ceph.conf Ceph.conf contents: ~~~ [global] mon_host = 192.168.1.10, 192.168.1.11, 192.168.1.12 log to stderr = true log to syslog = true run dir = C:/ProgramData/ceph crash dir = C:/logs/ceph debug client = 2 [client] keyring = C:/ProgramData/ceph/ceph.client.me.keyring log file = C:/logs/ceph/$name.$pid.log admin socket = C:/ProgramData/ceph/$name.$pid.asok ~~~ Windows Logfile contents (ieC:/logs/ceph/client.me.NNNN.log): ~~~ 2024-02-28T18:26:45.201+1100 1 0 monclient(hunting): authenticate timed out after 300 2024-02-28T18:31:45.203+1100 1 0 monclient(hunting): authenticate timed out after 300 2024-02-28T18:36:45.205+1100 1 0 monclient(hunting): authenticate timed out after 300 ~~~ Additional info from Windows CLI: ~~~ failed to fetch mon config (--no-mon-config to skip) ~~~ So I've gone through the doco and done some Google-foo and I can't work out *why* I'm getting a failure; why I'm getting the authentication failure. I know it'll be something simple, something staring me in the face, but I'm at the point where I can't see the forest for the trees - please help. Any help greatly appreciated Thanks in advance Cheers Dulux-Oz

3 4

reef 18.2.2 (hot-fix) QE validation status
by Yuri Weinstein 08 Mar '24

08 Mar '24

Details of this release are summarized here: https://tracker.ceph.com/issues/64721#note-1 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke - in progress rados - Radek, Laura? quincy-x - in progress Also need approval from Travis, Redouane for Prometheus fix testing.

7 10

Ceph-storage slack access
by Matthew Vernon 08 Mar '24

08 Mar '24

Hi, How does one get an invite to the ceph-storage slack, please? Thanks, Matthew

5 10

Re: pgcalc tool removed (or moved?) from ceph.com ?
by John Zachary Dover 08 Mar '24

08 Mar '24

PGcalc has been restored to docs.ceph.com in this PR: https://github.com/ceph/ceph/pull/56045 PGcalc can be accessed here: https://docs.ceph.com/en/reef/rados/operations/pgcalc/ Zac Dover Upstream Docs Ceph Foundation On Thu, Aug 26, 2021 at 9:56 AM Mike Perez <miperez(a)redhat.com> wrote: > Hi all, > > I attempted to migrate this to the new website, but it's going to > require some CSS work. I'm not the best with CSS, and the old website > has a very large CSS file to dissect from. I'll see what I can come up > with tomorrow. > > https://github.com/ceph/ceph.io/issues/265 > > On Thu, Jul 8, 2021 at 12:33 AM Christian Rohmann > <christian.rohmann(a)inovex.de> wrote: > > > > Hey Dominik, > > > > On 05/07/2021 09:55, Dominik Csapak wrote: > > > Hi, > > > > > > just wanted to ask if it is intentional that > > > > > > http://ceph.com/pgcalc/ > > > > > > results in a 404 error? > > > > > > is there any alternative url? > > > it is still linked from the offical docs. > > > > It's available at https://ceph.com/pgcalc/ just now (with cert not > > matching), but there apparently are people working on migrating the > > whole website .... > > > > * ceph.com redirects to https://old.ceph.com/ with matching Let's > > Encrypt certificate > > * but https://ceph.com/pgcalc/ is not rewritten to the old.ceph.com > > domain and thus the certificate error because the cert is only valid vor > > old.ceph.com > > > > > > Regards > > > > Christian > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > -- > Mike Perez > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Journal size recommendations
by Christian Rohmann 08 Mar '24

08 Mar '24

On 01.03.22 19:57, Eugen Block wrote: > can you be more specific what exactly you are looking for? Are you > talking about the rocksDB size? And what is the unit for 5012? It’s > really not clear to me what you’re asking. And since the > recommendations vary between different use cases you might want to > share more details about your use case. FWIW, I suppose OP was asking about this setting: https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval… And reading https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#journal… states "This section applies only to the older Filestore OSD back end. Since Luminous BlueStore has been default and preferred." It's totally obsolete with bluestore. Regards Christian

1 0

rgw dynamic bucket sharding will hang io
by nuabo tan 08 Mar '24

08 Mar '24

When reshard occurs, io will be blocked, why has this serious problem not been solved?

2 2

Upgarde from 16.2.1 to 16.2.2 pacific stuck
by Edouard FAZENDA 08 Mar '24

08 Mar '24

Dear Ceph Community, I am in the process of upgrading ceph pacific 16.2.1 to 16.2.2 , I have followed the documentation : https://docs.ceph.com/en/pacific/cephadm/upgrade/ My cluster is in Healthy state , but the upgrade is not going forward , as on the cephadm logs I have the following : # Ceph -W cephadm 2024-03-06T08:39:11.653447+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Need to upgrade myself (mgr.rke-sh1-1.qskoyj) 2024-03-06T08:39:12.281386+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Updating mgr.rke-sh1-2.lxmguj 2024-03-06T08:39:12.286096+0000 mgr.rke-sh1-1.qskoyj [INF] Deploying daemon mgr.rke-sh1-2.lxmguj on rke-sh1-2 2024-03-06T08:39:19.347877+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered out host rke-sh1-1: could not verify host allowed virtual ips 2024-03-06T08:39:19.347989+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered out host rke-sh1-3: could not verify host allowed virtual ips 2024-03-06T08:39:19.366355+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Need to upgrade myself (mgr.rke-sh1-1.qskoyj) 2024-03-06T08:39:19.965822+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Updating mgr.rke-sh1-2.lxmguj 2024-03-06T08:39:19.969089+0000 mgr.rke-sh1-1.qskoyj [INF] Deploying daemon mgr.rke-sh1-2.lxmguj on rke-sh1-2 2024-03-06T08:39:26.961455+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered out host rke-sh1-1: could not verify host allowed virtual ips 2024-03-06T08:39:26.961502+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered out host rke-sh1-3: could not verify host allowed virtual ips 2024-03-06T08:39:26.973897+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Need to upgrade myself (mgr.rke-sh1-1.qskoyj) 2024-03-06T08:39:27.623773+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Updating mgr.rke-sh1-2.lxmguj 2024-03-06T08:39:27.628115+0000 mgr.rke-sh1-1.qskoyj [INF] Deploying daemon mgr.rke-sh1-2.lxmguj on rke-sh1-2 My public_network is set : root@rke-sh1-1:~# ceph config dump | grep public_network mon advanced public_network 10.10.71.0/24 * Do you have an idea why I have the following error : Filtered out host: could not verify host allowed virtual ips Current state of the upgrade : # ceph orch upgrade status { "target_image": "docker.io/ceph/ceph@sha256:8cdd8c7dfc7be5865255f0d59c048a1fb8d1335f6923996e 2c2d9439499b5cf2", "in_progress": true, "services_complete": [], "progress": "0/35 ceph daemons upgraded", "message": "Currently upgrading mgr daemons" } progress: Upgrade to 16.2.2 (24m) [............................] Thanks for the help. Best Regards, Edouard FAZENDA Technical Support Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 <https://www.csti.ch/> www.csti.ch

3 7

All MGR loop crash
by David C. 08 Mar '24

08 Mar '24

Hello everybody, I'm encountering strange behavior on an infrastructure (it's pre-production but it's very ugly). After a "drain" on monitor (and a manager). MGRs all crash on startup: Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 standby mgrmap(e 1310) v1 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map received map epoch 1310 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map active in map: 1 active is 99148504 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map Activating! Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr handle_mgr_map I am now activating Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 mgrmap(e 1310) v1 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Got map version 1310 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc handle_mgr_map Active mgr is now Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgrc reconnect No active mgr available yet Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr init waiting for OSDMap... Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _renew_subs Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _send_mon_message to mon.idf-pprod-osd3 at v2:X.X.X.X:3300/0 <http://10.191.10.3:3300/0> Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: monclient: _reopen_session rank -1 Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: *** Caught signal (Aborted) ** in thread 7f9a07a27640 thread_name:mgr-fin ceph version 17.2.6-170.el9cp (59bbeb8815ec3aeb3c8bba1e1866f8f6729eb840) quincy (stable) 1: /lib64/libc.so.6(+0x54db0) [0x7f9a2364ddb0] 2: /lib64/libc.so.6(+0xa154c) [0x7f9a2369a54c] 3: raise() 4: abort() 5: /usr/lib64/ceph/libceph-common.so.2(+0x1c1fa8) [0x7f9a23ce2fa8] 6: /usr/lib64/ceph/libceph-common.so.2(+0x444425) [0x7f9a23f65425] 7: /usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0] 8: /usr/lib64/ceph/libceph-common.so.2(+0x4442e0) [0x7f9a23f652e0] 9: (MonClient::_add_conns()+0x242) [0x7f9a23f5fa42] 10: (MonClient::_reopen_session(int)+0x428) [0x7f9a23f60518] 11: (Mgr::init()+0x384) [0x5604667a6434] 12: /usr/bin/ceph-mgr(+0x1af271) [0x5604667ae271] 13: /usr/bin/ceph-mgr(+0x11364d) [0x56046671264d] 14: (Finisher::finisher_thread_entry()+0x175) [0x7f9a23d10645] 15: /lib64/libc.so.6(+0x9f802) [0x7f9a23698802] 16: /lib64/libc.so.6(+0x3f450) [0x7f9a23638450] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. I have the impression that the MGRs are ejected by the monitors, however after debugging monitor, I don't see anything abnormal on the monitor side (if I haven't missed something). All we can see is that we get an exception on the "_add_conn" method ( https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc #L775) Version : 17.2.6-170.el9cp (RHCS6)

3 6

Announcing Ceph Day NYC 2024 - April 26th!
by Dan van der Ster 08 Mar '24

08 Mar '24

Hi everyone, Ceph Days are coming to New York City again this year, co-hosted by Bloomberg Engineering and Clyso! We're planning a full day of Ceph content, well timed to learn about the latest and greatest Squid release. https://ceph.io/en/community/events/2024/ceph-days-nyc/ We're opening the CFP for presenters today -- it will close on March 26th so please get your proposals in quickly! Registration is also open now and space is limited, so book now to reserve your seat! Looking forward to seeing you there! -- dan

2 1

Remove cluster_network without routing
by Torkil Svensgaard 07 Mar '24

07 Mar '24

Hi Cephadm Reef 18.2.0. We would like to remove our cluster_network without stopping the cluster and without having to route between the networks. global advanced cluster_network 192.168.100.0/24 * global advanced public_network 172.21.12.0/22 * The documentation[1] states: " You may specifically assign static IP addresses or override cluster_network settings using the cluster_addr setting for specific OSD daemons. " So for one OSD at a time I could set cluster_addr to override the cluster_network IP and use the public_network IP instead? As the containers are using host networking they have access to both IPs and will just layer 2 the traffic, avoiding routing? When all OSDs are running with a public_network IP set via cluster_addr we can just delete the cluster_network setting and then remove all the ceph_addr settings, as with no cluster_network setting the public_network setting will be used? We tried with one OSD and it seems to work. Anyone see a problem with this approach? Thanks Torkil [1] https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#id3 -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 2

Disable signature url in ceph rgw
by marc＠singer.services 07 Mar '24

07 Mar '24

Hi Ceph users We are using Ceph Pacific (16) in this specific deployment. In our use case we do not want our users to be able to generate signature v4 URLs because they bypass the policies that we set on buckets (e.g IP restrictions). Currently we have a sidecar reverse proxy running that filters requests with signature URL specific request parameters. This is obviously not very efficient and we are looking to replace this somehow in the future. 1. Is there an option in RGW to disable this signed URLs (e.g returning status 403)? 2. If not is this planned or would it make sense to add it as a configuration option? 3. Or is the behaviour of not respecting bucket policies in RGW with signature v4 URLs a bug and they should be actually applied? Thanks you for your help and let me know if you have any questions Marc Singer

4 4

Minimum amount of nodes needed for stretch mode?
by Stefan Kooman 07 Mar '24

07 Mar '24

Hi, TL;DR Failure domain considered is data center. Cluster in stretch mode [1]. - What is the minimum amount of monitor nodes (apart from tie breaker) needed per failure domain? - What is the minimum amount of storage nodes needed per failure domain? - Are device classes supported with stretch mode? - is min_size = 1 in "degraded stretch mode" a hard coded requirement or can this be changed to it at leave min_size = 2 (yes, I'm aware that no other OSD may go down in the surviving data center or PGs will become unavailable). I've converted a (test) 3 node replicated cluster (2 storage nodes, 1 node with monitor only, min_size=2, size=4) setup to a "stretch mode" setup [1]. That works as expected. CRUSH rule (adjusted to work with 1 host and 2 OSDs per device class per data center only) rule stretch_rule { id 5 type replicated step take dc1 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit step take dc2 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit } The documentation seems to suggest that 2 storage nodes and 2 monitor nodes are needed at a minimum. Is that correct? I wonder why? For a minimal (as possible) cluster I don't see the need for one additional monitor per datacenter. Does the tiebreaker monitor function as a normal monitor (apart from it not allowed to become leader)? When stretch rules with device classes are used things don't work as expected anymore. Example crush rule: rule stretch_rule_ssd { id 4 type replicated step take dc1 class ssd step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit step take dc2 class ssd step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit } A similar crush rule for hdd exists. When I change the crush_rule for one of the pools to use stretch_rule_ssd the PGs on OSDs with device class ssd become inactive as soon as one of the data centers goes offline (and "degraded stretched mode" has been activated, and only 1 bucket, data center, is needed for peering). I don't understand why. Another issue with this is that as soon as the datacenter is online again, the recovery will never finish by itself and a "ceph osd force_healthy_stretch_mode --yes-i-really-mean-it" is needed to get HEALTH_OK Can anyone explain to me why this is? Gr. Stefan [1]: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/

2 2

ceph-volume fails when adding spearate DATA and DATA.DB volumes
by service.plant＠ya.ru 07 Mar '24

07 Mar '24

Hi all! I;ve faced an issue I couldnt even google. Trying to create OSD with two separate LVM for data.db and data, gives me intresting error ``` root@ceph-uvm2:/# ceph-volume lvm prepare --bluestore --data ceph-block-0/block-0 --block.db ceph-db-0/db-0 --> Incompatible flags were found, some values may get ignored --> Cannot use None (None) with --bluestore (bluestore) --> Incompatible flags were found, some values may get ignored --> Cannot use --bluestore (bluestore) with --block.db (bluestore) Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e3b7b9e5-6399-4e61-8634-41a991bb1948 stderr: 2024-03-04T11:45:52.263+0000 7f71c3a89700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-03-04T11:45:52.263+0000 7f71c3a89700 -1 AuthRegistry(0x7f71bc064978) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2024-03-04T11:45:52.263+0000 7f71c3a89700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-03-04T11:45:52.263+0000 7f71c3a89700 -1 AuthRegistry(0x7f71bc067fd0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2024-03-04T11:45:52.267+0000 7f71c3a89700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-03-04T11:45:52.267+0000 7f71c3a89700 -1 AuthRegistry(0x7f71c3a87ea0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2024-03-04T11:45:52.267+0000 7f71c1024700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2024-03-04T11:45:52.267+0000 7f71c2026700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2024-03-04T11:45:52.267+0000 7f71c1825700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2024-03-04T11:45:52.267+0000 7f71c3a89700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication stderr: [errno 13] RADOS permission denied (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id ``` And here is output which shows my created volumes ``` root@ceph-uvm2:/# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert block-0 ceph-block-0 -wi-a----- 16.37t db-0 ceph-db-0 -wi-a----- 326.00g ``` CEPH was rolled out by using cephadm and all the commands I do under cephadm shell I have no idea what is the reason of that error and that is why I am here. Any help is appreciated. Thanks in advance.

2 2

erasure-code-lrc Questions regarding repair
by Ansgar Jazdzewski 07 Mar '24

07 Mar '24

hi folks, I currently test erasure-code-lrc (1) in a multi-room multi-rack setup. The idea is to be able to repair a disk-failures within the rack itself to lower bandwidth-usage ```bash ceph osd erasure-code-profile set lrc_hdd \ plugin=lrc \ crush-root=default \ crush-locality=rack \ crush-failure-domain=host \ crush-device-class=hdd \ mapping=__DDDDD__DDDDD__DDDDD__DDDDD \ layers=' [ [ "_cDDDDD_cDDDDD_cDDDDD_cDDDDD", "" ], [ "cDDDDDD_____________________", "" ], [ "_______cDDDDDD______________", "" ], [ "______________cDDDDDD_______", "" ], [ "_____________________cDDDDDD", "" ], ]' \ crush-steps='[ [ "choose", "room", 4 ], [ "choose", "rack", 1 ], [ "chooseleaf", "host", 7 ], ]' ``` The roule picks 4 out of 5 rooms and keeps the PG in one rack like expected! However it looks like the PG will not move to another Room if the PG is undersized or the entire Room or Rack is down! Questions: * do I miss something to allow LRC (PG's) to move across Racks/Rooms for repair? * Is it even possible to build such a 'Multi-stage' grushmap? Thanks for your help, Ansgar 1) https://docs.ceph.com/en/quincy/rados/operations/erasure-code-jerasure/

2 2

Running dedicated RGWs for async tasks
by Marc Singer 07 Mar '24

07 Mar '24

Hello Ceph Users Since we are running a big S3 cluster we would like to externalize the RGW daemons that do async tasks, like: * Garbage collection * Lifecycle policies * Calculating and updating quotas Would this be possible to do in the configuration? Which config values would I need to set on the exposed RGWs and which on the Async Task RGWs? Thanks for your input. Marc

2 1

Unable to map RBDs after running pg-upmap-primary on the pool
by Torkil Svensgaard 07 Mar '24

07 Mar '24

Hi I tried to do offline read optimization[1] this morning but I am now unable to map the RBDs in the pool. I did this prior to running the pg-upmap-primary commands suggested by the optimizer, as suggested by the latest documentation[2]: " ceph osd set-require-min-compat-client reef " The command didn't complain and the documentation stated it should failed if any pre-reef clients were connected so I thought all was well and rad the pg_upmap_primary commands. Can I simply remove the applied pg_upmap_primary settings somehow to my RBDs back while investigating? Mvh. Torkil [1] https://docs.ceph.com/en/reef/rados/operations/read-balancer/ [2] https://docs.ceph.com/en/latest/rados/operations/read-balancer/ -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

1 1

Ceph is constantly scrubbing 1/4 of all PGs and still have pigs not scrubbed in time
by thymus_03fumbler＠icloud.com 07 Mar '24

07 Mar '24

I recently switched from 16.2.x to 18.2.x and migrated to cephadm, since the switch the cluster is constantly scrubbing, 24/7 up to 50 PGs simultaneously and up to 20 deep scrubs simultaneously in a cluster that has only 12 (in use) OSDs. Furthermore it still manages to regularly have a warning with ‘pgs not scrubbed in time’ I have tried various settings, like osd_deep_scrub_interval, osd_max_scrubs, mds_max_scrub_ops_in_progress etc. All those get ignored. Please advice. Here is an output of ceos config dump: WHO MASK LEVEL OPTION VALUE RO global advanced auth_client_required cephx * global advanced auth_cluster_required cephx * global advanced auth_service_required cephx * global advanced auth_supported cephx * global basic container_image quay.io/ceph/ceph@sha256:aca35483144ab3548a7f670db9b79772e6fc51167246421c66c0bd56a6585468 * global basic device_failure_prediction_mode local global advanced mon_allow_pool_delete true global advanced mon_data_avail_warn 20 global advanced mon_max_pg_per_osd 400 global advanced osd_max_pg_per_osd_hard_ratio 10.000000 global advanced osd_pool_default_pg_autoscale_mode on mon advanced auth_allow_insecure_global_id_reclaim false mon advanced mon_crush_min_required_version firefly * mon advanced mon_warn_on_pool_no_redundancy false mon advanced public_network 10.79.0.0/16 * mgr advanced mgr/balancer/active true mgr advanced mgr/balancer/mode upmap mgr advanced mgr/cephadm/manage_etc_ceph_ceph_conf_hosts label:admin * mgr advanced mgr/cephadm/migration_current 6 * mgr advanced mgr/dashboard/GRAFANA_API_PASSWORD admin * mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false * mgr advanced mgr/dashboard/GRAFANA_API_URL https://10.79.79.12:3000 * mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://10.79.79.12:9095 * mgr advanced mgr/devicehealth/enable_monitoring true mgr advanced mgr/orchestrator/orchestrator cephadm osd advanced osd_map_cache_size 250 osd advanced osd_map_share_max_epochs 50 osd advanced osd_mclock_profile high_client_ops osd advanced osd_pg_epoch_persisted_max_stale 50 osd.0 basic osd_mclock_max_capacity_iops_hdd 380.869888 osd.1 basic osd_mclock_max_capacity_iops_hdd 441.000000 osd.10 basic osd_mclock_max_capacity_iops_ssd 13677.906485 osd.11 basic osd_mclock_max_capacity_iops_hdd 274.411212 osd.13 basic osd_mclock_max_capacity_iops_hdd 198.492501 osd.2 basic osd_mclock_max_capacity_iops_hdd 251.592009 osd.3 basic osd_mclock_max_capacity_iops_hdd 208.197434 osd.4 basic osd_mclock_max_capacity_iops_hdd 196.544082 osd.5 basic osd_mclock_max_capacity_iops_ssd 12739.225456 osd.6 basic osd_mclock_max_capacity_iops_hdd 211.288660 osd.7 basic osd_mclock_max_capacity_iops_hdd 210.543236 osd.8 basic osd_mclock_max_capacity_iops_hdd 242.241594 osd.9 basic osd_mclock_max_capacity_iops_hdd 559.933780 mds.plexfs basic mds_join_fs plexfs Here is a ceph -s output services: mon: 3 daemons, quorum lxt-prod-ceph-util02,lxt-prod-ceph-util01,lxt-prod-ceph-util03 (age 3w) mgr: lxt-prod-ceph-util02.iyrhxj(active, since 3w), standbys: lxt-prod-ceph-util03.wvstpe mds: 1/1 daemons up osd: 14 osds: 14 up (since 4w), 14 in (since 4w) data: volumes: 1/1 healthy pools: 4 pools, 193 pgs objects: 14.48M objects, 52 TiB usage: 71 TiB used, 39 TiB / 110 TiB avail pgs: 131 active+clean 47 active+clean+scrubbing 15 active+clean+scrubbing+deep

3 2

Ceph Cluster Config File Locations?
by duluxoz 07 Mar '24

07 Mar '24

Hi All, I don't know how its happened (bad backup/restore, bad config file somewhere, I don't know) but my (DEV) Ceph Cluster is in a very bad state, and I'm looking for pointers/help in getting it back running (unfortunate, a complete rebuild/restore is *not* an option). This is on Ceph Reef (on Rocky 9) which was converted to CephAdm from a manual install a few weeks ago (which worked). Five days ago everything when "t!ts-up" (an Ozzie technical ICT term meaning nothing works :-) ) So, my (first?) issue is that I can't get any Managers to come up clean. Each one tries to connect on an ip subnet which doesn't exist any longer and hasn't for a couple of years. The second issue is that (possible because of the first) every `ceph orch` command just hangs. Cephadm commands work fine. I've checked, checked, and checked again that the individual config files all point towards the correct ip subnet for the monitors, and I cannot find any trace of the old subnet's ip address in any config file (that I can find). For the record I am *not* a "podman guy" so there may be something there that's causing my issue(s?) but I don't know where to even begin to look. Any/all logs simply start that the Manager(s) try to come up, can't find an address in the "old" subnet, and so fail - nothing else helpful (at least to me). I've even pulled a copy of the monmap and its showing the correct ip subnet addresses for the monitors. The firewalls are all set correctly and an audit2allow shows nothing is out of place, as does disabling SELinux (ie no change). A `ceph -s` shows I've got no active managers (and that a monitor is down - that's my third issue), plus a whole bunch of osds and pgs aren't happy either. I have, though, got a monitor quorum. So, what should I be looking at / where should I be looking? Any help is greatly *greatly* appreciated. Cheers Dulux-Oz

3 3

ceph Quincy to Reef non cephadm upgrade
by sarda.ravi＠gmail.com 07 Mar '24

07 Mar '24

I want to perform non cephadm upgrade from Quincy to Reef. Reason for not using cephadm is do not want to go for ceph in containers. My test deployment is as given below. Total cluster hosts : 5 ceph-mon hosts: 3 ceph-mgr hosts: 3 (ceph-mgr active on one node, and other ceph-mgr each on ceph-mon host) ceph-mds : 1 ceph-osd : 5 (one ceph-osd on each of the host in the cluster.) While I try to follow the steps - https://docs.ceph.com/en/latest/releases/reef/#upgrading-non-cephadm-cluste… - on the step - Upgrade monitors by installing the new packages and restarting the monitor daemons. when I try to upgrade only ceph-mon using "apt upgrade ceph-mon" command it upgrades all packages including ceph-mgr, ceph-mds, ceph-osd etc. as ceph-mon package has dependency on these packages. My question is - does this mean I need to upgrade all ceph packages (ceph, ceph-common) and restart only monitor daemon first? Or there is any way I can upgrade only ceph-mon pacakge first, then ceph-mgr, ceph-osd and so on?

2 1

回复：Re: How to build ceph without QAT?
by 张东川 07 Mar '24

07 Mar '24

Hi Hualong and llya, Thanks for your help. More info: I am trying to buid ceph on Milkv Pioneer board (RISCV arch, OS is fedora-riscv 6.1.55). The ceph code being used was downloaded from github last week (master branch) Currently I am working on environment cleanup (I suspect my work environment is not clean). I will try to switch to V19.0.0 and rebuild. Will let you know if further help needed. Thanks. Best Regards, Dongchuan Ilya Dryomov<idryomov(a)gmail.xn--com> -8s7w 2024年3月6日周三 21:13 写道： On Wed, Mar 6, 2024 at 7:41 AM Feng, Hualong <hualong.feng(a)intel.com> wrote: > > Hi Dongchuan > > Could I know which version or which commit that you are building and your environment: system, CPU, kernel? > > ./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo   this command should be OK without QAT. Hi Hualong, I don't think this is true.  In main, both WITH_QATLIB and WITH_QATZIP default to ON unless the system is aarch64.  IIRC I needed to append -D WITH_QATLIB=OFF -D WITH_QATZIP=OFF to build without QAT. Thanks,                 Ilya > > Thanks > -Hualong > > > -----Original Message----- > > From: 张东川 <zhangdongchuan(a)metastonecorp.com> > > Sent: Wednesday, March 6, 2024 9:51 AM > > To: ceph-users <ceph-users(a)ceph.io> > > Subject: [ceph-users] How to build ceph without QAT? > > > > Hi guys, > > > > > > I tried both following commands. > > Neither of them worked. > > > > > > "./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_QAT=OFF > > -DWITH_QATDRV=OFF -DWITH_QATZIP=OFF" > > "ARGS="-DWITH_QAT=OFF -DWITH_QATDRV=OFF - > > DWITH_QATZIP=OFF" ./do_cmake.sh - > > DCMAKE_BUILD_TYPE=RelWithDebInfo" > > > > > > I still see errors like: > > make[1]: *** [Makefile:4762: > > quickassist/lookaside/access_layer/src/sample_code/performance/framew > > ork/linux/user_space/cpa_sample_code-cpa_sample_code_utils.o] Error 1 > > > > > > > > > > So what's the proper way to configure build commands? > > Thanks a lot. > > > > > > Best Regards, > > Dongchuan > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email > > to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

change ip node and public_network in cluster
by farhad kh 06 Mar '24

06 Mar '24

I have implemented a ceph cluster with cephadm which has three monitors and three OSDs each node have one interface 192.168.0.0/24 network. I want to change the address of the machines to the range 10.4.4.0/24. Is there a solution for this change without data loss and failure? i change the pubic_network in mon and change the ip node but its not worked . how can i sovle this problem? ``` ceph orch host ls HOST ADDR LABELS STATUS ceph-01 192.168.0.130 _admin,rgw ceph-02 192.168.0.131 _admin,rgw ceph-03 192.168.0.132 _admin,rgw 3 hosts in cluster ```` [root@ceph-01 ~]# ceph config get mon public_network 192.168.0.0/24 ```` [root@ceph-01 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 112s ago 9M count:1 ceph-exporter 3/3 114s ago 8M * crash 3/3 114s ago 9M * grafana ?:3000 1/1 112s ago 8M count:1 mgr 2/2 113s ago 9M count:2 mon 3/3 114s ago 8M count:3 node-exporter ?:9100 3/3 114s ago 9M * osd.dashboard-admin-1685787597651 6 114s ago 8M * prometheus ?:9095 1/1 112s ago 3M count:1 ````

3 3

Ceph Leadership Team Meeting Minutes - March 6, 2024
by Ernesto Puerta 06 Mar '24

06 Mar '24

Hi Cephers, These are the topics covered in today's meeting: - *Releases* - *Hot fixes Releases* - *18.2.2* - https://github.com/ceph/ceph/pull/55491 - reef: mgr/prometheus: fix orch check to prevent Prometheus crash - https://github.com/ceph/ceph/pull/55709 - reef: debian/*.postinst: add adduser as a dependency and specify --home when adduser - https://github.com/ceph/ceph/pull/55712 - reef: src/osd/OSDMap.cc: Fix encoder to produce same bytestream - [Laura] When/who can upgrade the LRC for this release? (Dan will after we do some last checks today) - Gibba has been upgraded with no apparent issues - *17.2.8* (on hold based on whether the osdmap fix requirement) - *No longer needed* - osdmap fix (not needed: https://github.com/ceph/ceph/blob/quincy/src/osd/OSDMap.cc#L3087-L3089) - Rook requests to include this c-v fix that blocks OSDs in some scenarios (not worthy of its own hotfix, just please include if we do the hotfix) - https://github.com/ceph/ceph/pull/54522 - quincy: ceph-volume: fix a regression in raw list - As crc fix is not needed, the Rook request can be included in a regular quincy release - *Regular releases* - *18.2.3* - exporter fixes for rook and debian-derived users make this more urgent than quincy - mgr usage of pyO3/cryptography an issue for debian - and possibly centos9 (https://tracker.ceph.com/issues/64213#note-2) - see notes from 02/07 - Any updates on potentially dropping modules or another fix? Adam? - Squid *19.1.0* - CephFS waiting for 2 feature PR - RGW PRs - NVMe? To be confirmed with Aviv - squid blockers: - build centos 9 containers: https://github.com/ceph/ceph-container/pull/2183 - ceph-object-corpus: https://github.com/ceph/ceph-object-corpus/pull/17 (testing in https://github.com/ceph/ceph/pull/54735) - Milestone for squid blockers (use to tag blockers for the first 19.1.0 RC): https://github.com/ceph/ceph/milestone/21 - Squid RCs and community testing - https://pad.ceph.com/p/squid_scale_testing - Target date March ~20 - *17.2.9* - need jammy builds for quincy before a squid release. maybe we can just build them for the 17.2.7 release? (do you mean the 17.2.8 release?) - *Meeting time - *change days to Monday or Thursday? (added by Josh - who has a conflict on Wednesdays now) - Thursday has several conflicting community meetings - Any objections to Monday at the same time? - Note the change to US daylight savings next week - Let's do a poll (Doodle) - *debian-reef_OLD email thread "[ceph-users] debian-reef_OLD?"* - Fixed by Yuri - *CDM APAC tonight*: https://tracker.ceph.com/projects/ceph/wiki/CDM_06-MAR-2024 - *Sepia Lab*: - PSA: https://github.com/ceph/ceph/pull/55820 merged (squid crontab additions and overhaul to nightlies) - New grafana widget for smithi node utilization: - https://grafana-route-grafana.apps.os.sepia.ceph.com/d/teuthology/teutholog… - (Basically: unlocked machine * hours / total machine * hours ) - [Zac] *ceph-exporter release notes question from Jan Horacek* (from the upstream community) - Route to Juanmi Olmo - [Zac] - *Eugen Block's question about removing sensitive information from ceph-users mailing list* - No easy way to request/remove sensitive information. - [Zac] - *Anthony D'Atri submits Index HQ in Toronto as a possible venue for Cephalocon 2024* - Venue already booked (Patrick) - [Zac] - *CQ issue 4 -- submit your requests before 25 Mar 2024* -- zac.dover(a)proton.me Kind Regards, Ernesto Puerta

1 0

Ceph reef mon is not starting after host reboot
by ankit＠eurus.net 06 Mar '24

06 Mar '24

Hi guys, i am very newbie to ceph-cluster but after multiple attempts, i was able to install ceph-reef cluster on debian-12 by cephadm tool on test environment with 2 mons and 3 OSD's om VM's. All was seeming good and i was exploring more about it so i rebooted cluster and found that now i am not able to access ceph dashboard and i have try to check this root@ceph-mon-01:/# ceph orch ls 2024-03-01T08:53:05.051+0000 7ff7602b8700 0 monclient(hunting): authenticate timed out after 300 [errno 110] RADOS timed out (error connecting to the cluster) i have not configured RADOS. And i have no clue about it. Any help would be very appreciated? the same issue.

2 1

Ceph storage project for virtualization
by egoitz＠ramattack.net 06 Mar '24

06 Mar '24

Hi! I have been reading some ebooks of Ceph and some doc and learning about it. The goal of all it, is the fact of creating a rock solid storage por virtual machines. After all the learning I have not been able to answer by myself to this question so I was wondering if perhaps you could clarify my doubt. Let's imagine three datacenters, each one with for instance, 4 virtualization hosts. As I was planning to build a solution for diferent hypervisors I have been thinking in the following env. - I planed to have my Ceph storage (with different pools inside) with OSDs in three different datacenters (as failure point). - Each datacenter's hosts, will be accessing to a NFS redundant service in the own datacenter. - Each NFS redundant service of each datacenter will be composed by two NFS gateways accessing to the OSDs of the placement group located in the own datacenter. I planned achieving this with OSD weights and getting with that the fact that the crush algorithm to build the map so that each datacenter accesses end up having as master, the OSD of the own datacenter in the placement group. Obviously, slave OSD replicas will exist in the other three datacenters or even I don't discard the fact of using erasure coding in some manner. - The NFS gateways could be a NFS redundant gateway service from Ceph (I have seen now they have developed something for this purpose https://docs.ceph.com/en/quincy/mgr/nfs/) or perhaps two different Debian machines, accessing to Ceph with rados and sharing to the hypervisors that information over NFS. In case of Debian machines I have heard good results using pacemaker/corosync for providing HA to that NFS (between 0,5 and 3 seconds for fail over and service up again). What do you think about this plan?. Do you see it feasible?. We will work too with KVM and there we could access to Ceph directly but I would needed to provide too storage por Xen and Vmware. Thank you so much in advance, Cheers!

2 4

Re: How to build ceph without QAT?
by Feng, Hualong 06 Mar '24

06 Mar '24

Hi Dongchuan Could I know which version or which commit that you are building and your environment: system, CPU, kernel? ./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo this command should be OK without QAT. Thanks -Hualong > -----Original Message----- > From: 张东川 <zhangdongchuan(a)metastonecorp.com> > Sent: Wednesday, March 6, 2024 9:51 AM > To: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] How to build ceph without QAT? > > Hi guys, > > > I tried both following commands. > Neither of them worked. > > > "./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_QAT=OFF > -DWITH_QATDRV=OFF -DWITH_QATZIP=OFF" > "ARGS="-DWITH_QAT=OFF -DWITH_QATDRV=OFF - > DWITH_QATZIP=OFF" ./do_cmake.sh - > DCMAKE_BUILD_TYPE=RelWithDebInfo" > > > I still see errors like: > make[1]: *** [Makefile:4762: > quickassist/lookaside/access_layer/src/sample_code/performance/framew > ork/linux/user_space/cpa_sample_code-cpa_sample_code_utils.o] Error 1 > > > > > So what's the proper way to configure build commands? > Thanks a lot. > > > Best Regards, > Dongchuan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email > to ceph-users-leave(a)ceph.io

2 1

Monitoring Ceph Bucket and overall ceph cluster remaining space
by Michael Worsham 06 Mar '24

06 Mar '24

Is there an easy way to poll the ceph cluster buckets in a way to see how much space is remaining? And is it possible to see how much ceph cluster space is remaining overall? I am trying to extract the data from our Ceph cluster and put it into a format that our SolarWinds can understand in whole number integers, so we can monitor bucket allocated space and overall cluster space in the cluster as a whole. Via Canonical support, the said I can do something like "sudo ceph df -f json-pretty" to pull the information, but what is it I need to look at from the output (see below) to display over to SolarWinds? { "stats": { "total_bytes": 960027263238144, "total_avail_bytes": 403965214187520, "total_used_bytes": 556062049050624, "total_used_raw_bytes": 556062049050624, "total_used_raw_ratio": 0.57921481132507324, "num_osds": 48, "num_per_pool_osds": 48, "num_per_pool_omap_osds": 48 }, "stats_by_class": { "ssd": { "total_bytes": 960027263238144, "total_avail_bytes": 403965214187520, "total_used_bytes": 556062049050624, "total_used_raw_bytes": 556062049050624, "total_used_raw_ratio": 0.57921481132507324 } }, And a couple of data pools... { "name": "default.rgw.jv-va-pool.data", "id": 65, "stats": { "stored": 4343441915904, "objects": 17466616, "kb_used": 12774490932, "bytes_used": 13081078714368, "percent_used": 0.053900588303804398, "max_avail": 76535973281792 } }, { "name": "default.rgw.jv-va-pool.index", "id": 66, "stats": { "stored": 42533675008, "objects": 401, "kb_used": 124610380, "bytes_used": 127601028363, "percent_used": 0.00055542576592415571, "max_avail": 76535973281792 } }, This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

2 4

InvalidAccessKeyId
by ashar.khan＠rapidcompute.com 06 Mar '24

06 Mar '24

Dear Team, I am facing one issue, which could be a possible bug, but not able to find any solution. We are unable to create a bucket on ceph from ceph-dashboard. Bucket is getting created with a fresh/different name but once I am trying with a deleted bucket name then it is not getting created. I have tested on (Octopus and Pacific version). Steps to reproduce. a. Create user RGW user from ceph Dashboard. (user1) b. Create a bucket with the bucket owner as user1. (bucket1) c. Delete the bucket d Delete user. e. Create a user again with the same name (user1) f. Create a new bucket with the bucket owner as user1. (Bucket2) I get below error message: RGW REST API failed request with status code 403 (b'{"Code":"InvalidAccessKeyId","RequestId":"tx00000457ff5169168a9e3-00648afcd0' b'-fdd1c2-dev","HostId":"fdd1c2-dev-india"}')

1 0

How to build ceph without QAT?
by 张东川 06 Mar '24

06 Mar '24

Hi guys, I tried both following commands. Neither of them worked. "./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_QAT=OFF -DWITH_QATDRV=OFF -DWITH_QATZIP=OFF" "ARGS="-DWITH_QAT=OFF -DWITH_QATDRV=OFF -DWITH_QATZIP=OFF" ./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo" I still see errors like: make[1]: *** [Makefile:4762: quickassist/lookaside/access_layer/src/sample_code/performance/framework/linux/user_space/cpa_sample_code-cpa_sample_code_utils.o] Error 1 So what's the proper way to configure build commands? Thanks a lot. Best Regards, Dongchuan

1 0

debian-reef_OLD?
by Daniel Brown 05 Mar '24

05 Mar '24

I likely missed an announcement, and if so, please forgive me. I’m seeing some failure for when running apt on a cluster of ubuntu machines — looks like a directory has changed on https://download.ceph.com/ Was: debian-reef/ Now appears to be: debian-reef_OLD/ Was reef pulled?

4 4

Number of pgs
by Nikolaos Dandoulakis 05 Mar '24

05 Mar '24

Hi all, Pretty sure not the first time you see a thread like this. Our cluster consists of 12 nodes/153 OSDs/1.2 PiB used, 708 TiB /1.9 PiB avail The data pool is 2048 pgs big exactly the same number as when the cluster started. We have no issues with the cluster, everything runs as expected and very efficiently. We support about 1000 clients. The question is should we increase the number of pgs? If you think so, what is the sensible number to go to? 4096? More? I will eagerly await for your response. Best, Nick P.S. Yes, autoscaler is off :) The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th' ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

3 5

Number of pgs
by ndandoul＠ed.ac.uk 05 Mar '24

05 Mar '24

Hi all, Pretty sure not the very first time you see a thread like this. Our cluster consists of 12 nodes/153 OSDs/1.2 PiB used, 708 TiB /1.9 PiB avail The data pool is 2048 pgs big exactly the number when the cluster started. We have no issues with the cluster, everything runs as expected and very efficiently. We support about 1000 clients. The question is should we increase the number of pgs? If you think so, what is the sensible number to go to? 4096? More? I will eagerly await for your response. P.S. Yes, autoscaler is off :)

1 0

Upgraded 16.2.14 to 16.2.15
by Zakhar Kirpichenko 05 Mar '24

05 Mar '24

Hi, I have upgraded my test and production cephadm-managed clusters from 16.2.14 to 16.2.15. The upgrade was smooth and completed without issues. There were a few things which I noticed after each upgrade: 1. RocksDB options, which I provided to each mon via their configuration files, got overwritten during mon redeployment and I had to re-add mon_rocksdb_options back. 2. Monitor debug_rocksdb option got silently reset back to the default 4/5, I had to set it back to 1/5. 3. For roughly 2 hours after the upgrade, despite the clusters being healthy and operating normally, all monitors would run manual compactions very often and write to disks at very high rates. For example, production monitors had their rocksdb:low0 thread write to store.db: monitors without RocksDB compression: ~8 GB/5 min, or ~96 GB/hour; monitors with RocksDB compression: ~1.5 GB/5 min, or ~18 GB/hour. After roughly 2 hours with no changes to the cluster the write rates dropped to ~0.4-0.6 GB/5 min and ~120 MB/5 min respectively. The reason for frequent manual compactions and high write rates wasn't immediately apparent. 4. Crash deployment broke ownership of /var/lib/ceph/FSID/crash /var/lib/ceph/FSID/crash/posted, despite I already fixed it manually after the upgrade to 16.2.14 which had broken it as well. 5. Mgr RAM usage appears to be increasing at a slower rate than it did with 16.2.14, although it's too early to tell whether the issue with mgrs randomly consuming all RAM and getting OOM-killed has been fixed - with 16.2.14 this would normally take several days. Overall, things look good. Thanks to the Ceph team for this release! Zakhar

4 9

PGs with status active+clean+laggy
by mori.ricardo＠gmail.com 05 Mar '24

05 Mar '24

Dear community, I have a ceph quincy cluster with 5 nodes currently. But only 3 with SSDs and others with nvme. On separate pools. I have had many alerts from PGs with active-clean-laggy status. This has caused problems with slow writing. I wanted to know how to troubleshoot properly. I checked several things related to the network, I have 10 GB cards on all nodes and everything seems to be correct. Many thanks

1 0

Uninstall ceph rgw
by Albert Shih 05 Mar '24

05 Mar '24

Hi everyone, I'm currently trying to understand how to deploy rgw, so I test few things but now I'm not sure what's are installed what not. First I try to install according to https://docs.ceph.com/en/quincy/cephadm/services/rgw/ then I see in that page they are https://docs.ceph.com/en/quincy/mgr/rgw/#mgr-rgw-module so now I got some rgw daemon running. But I like to clean up and «erase» everything about rgw ? not only to try to understand but also because I think I mixted up between realm and zonegroup... Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mar. 05 mars 2024 11:01:30 CET

2 2

PGs with status active+clean+laggy
by ricardomori＠soujmv.com 05 Mar '24

05 Mar '24

Dear community, I have a ceph quincy cluster with 5 nodes currently. But only 3 with SSDs. I have had many alerts from PGs with active-clean-laggy status. This has caused problems with slow writing. I wanted to know how to troubleshoot properly. I checked several things related to the network, I have 10 GB cards on all nodes and everything seems to be correct. Many thanks

3 2

[RGW] Restrict a subuser to access only one specific bucket
by Huy Nguyen 05 Mar '24

05 Mar '24

Hi community, I have a user that owns some buckets. I want to create a subuser that has permission to access only one bucket. What can I do to archive this? Thanks

2 1

Slow RGW multisite sync due to "304 Not Modified" responses on primary zone
by Alam Mohammad 05 Mar '24

05 Mar '24

Hi, We have 2 clusters (v18.2.1) primarily used for RGW which has over 2+ billion RGW objects. They are also in multisite configuration totaling to 2 zones and we've got around 2 Gbps of bandwidth dedicated (P2P) for the multisite traffic. We see that using "radosgw-admin sync status" on the zone 2, all the 128 shards are recovering and unfortunately there is very less data transfer from primary zone ie., the link utilization is barely 100 Mbps / 2 Gbps. Our objects are quite small as well like avg. of 1 MB in size. On further inspection, we noticed the rgw access the logs at primary site are mostly yielding "304 Not Modified" for RGWs at site-2. Is this expected? Here are some of the logs (information is redacted) root@host-04:~# tail -f /var/log/haproxy-msync.log Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:33730 [12/Feb/2024:05:06:51.047] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/1/0/0 0/0 "GET /bucket1/object1.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:59730 [12/Feb/2024:05:06:51.048] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/3/1/0 0/0 "GET /bucket1/object91.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" We also took a look at our grafana instance and out of 1000 requests / second, 200 are "200 OK" and 800 are "304 Not Modified". Sync threads are run on only 2 rgw daemons per zone and are behind a Load Balancer. "# radosgw-admin sync error list" also contains around 20 errors which are mostly automatically recoverable. As we understand, does it mean that RGW multisite sync logs in the log pool are yet to be generated or some sort? Please provide us some insights and let us know how to resolve this. Thanks, Saif

2 2

OSDs not balanced
by Ml Ml 04 Mar '24

04 Mar '24

Hello, i wonder why my autobalancer is not working here: root@ceph01:~# ceph -s cluster: id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df health: HEALTH_ERR 1 backfillfull osd(s) 1 full osd(s) 1 nearfull osd(s) 4 pool(s) full => osd.17 was too full (92% or something like that) root@ceph01:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE ... %USE ... PGS TYPE NAME -25 209.50084 - 213 TiB ... 69.56 ... - datacenter xxx-dc-root -19 84.59369 - 86 TiB ... 56.97 ... - rack RZ1.Reihe4.R10 -3 35.49313 - 37 TiB ... 57.88 ... - host ceph02 2 hdd 1.70000 1.00000 1.7 TiB ... 58.77 ... 44 osd.2 3 hdd 1.00000 1.00000 2.7 TiB ... 22.14 ... 25 osd.3 7 hdd 2.50000 1.00000 2.7 TiB ... 58.84 ... 70 osd.7 9 hdd 9.50000 1.00000 9.5 TiB ... 63.07 ... 268 osd.9 13 hdd 2.67029 1.00000 2.7 TiB ... 53.59 ... 65 osd.13 16 hdd 2.89999 1.00000 2.7 TiB ... 59.35 ... 71 osd.16 19 hdd 1.70000 1.00000 1.7 TiB ... 48.98 ... 37 osd.19 23 hdd 2.38419 1.00000 2.4 TiB ... 59.33 ... 64 osd.23 24 hdd 1.39999 1.00000 1.7 TiB ... 51.23 ... 39 osd.24 28 hdd 3.63869 1.00000 3.6 TiB ... 64.17 ... 104 osd.28 31 hdd 2.70000 1.00000 2.7 TiB ... 64.73 ... 76 osd.31 32 hdd 3.39999 1.00000 3.3 TiB ... 67.28 ... 101 osd.32 -9 22.88817 - 23 TiB ... 56.96 ... - host ceph06 35 hdd 7.15259 1.00000 7.2 TiB ... 55.71 ... 182 osd.35 36 hdd 5.24519 1.00000 5.2 TiB ... 53.75 ... 128 osd.36 45 hdd 5.24519 1.00000 5.2 TiB ... 60.91 ... 144 osd.45 48 hdd 5.24519 1.00000 5.2 TiB ... 57.94 ... 139 osd.48 -17 26.21239 - 26 TiB ... 55.67 ... - host ceph08 37 hdd 6.67569 1.00000 6.7 TiB ... 58.17 ... 174 osd.37 40 hdd 9.53670 1.00000 9.5 TiB ... 58.54 ... 250 osd.40 46 hdd 5.00000 1.00000 5.0 TiB ... 52.39 ... 116 osd.46 47 hdd 5.00000 1.00000 5.0 TiB ... 50.05 ... 112 osd.47 -20 59.11053 - 60 TiB ... 82.47 ... - rack RZ1.Reihe4.R9 -4 23.09996 - 24 TiB ... 79.92 ... - host ceph03 5 hdd 1.70000 0.75006 1.7 TiB ... 87.24 ... 66 osd.5 6 hdd 1.70000 0.44998 1.7 TiB ... 47.30 ... 36 osd.6 10 hdd 2.70000 0.85004 2.7 TiB ... 83.23 ... 100 osd.10 15 hdd 2.70000 0.75006 2.7 TiB ... 74.26 ... 88 osd.15 17 hdd 0.50000 0.85004 1.6 TiB ... 91.44 ... 67 osd.17 20 hdd 2.00000 0.85004 1.7 TiB ... 88.41 ... 68 osd.20 21 hdd 2.79999 0.75006 2.7 TiB ... 77.25 ... 91 osd.21 25 hdd 1.70000 0.90002 1.7 TiB ... 78.31 ... 60 osd.25 26 hdd 2.70000 1.00000 2.7 TiB ... 82.75 ... 99 osd.26 27 hdd 2.70000 0.90002 2.7 TiB ... 84.26 ... 101 osd.27 63 hdd 1.89999 0.90002 1.7 TiB ... 84.15 ... 65 osd.63 -13 36.01057 - 36 TiB ... 84.12 ... - host ceph05 11 hdd 7.15259 0.90002 7.2 TiB ... 85.45 ... 273 osd.11 39 hdd 7.20000 0.85004 7.2 TiB ... 80.90 ... 257 osd.39 41 hdd 7.20000 0.75006 7.2 TiB ... 74.95 ... 239 osd.41 42 hdd 9.00000 1.00000 9.5 TiB ... 92.00 ... 392 osd.42 43 hdd 5.45799 1.00000 5.5 TiB ... 84.84 ... 207 osd.43 -21 65.79662 - 66 TiB ... 74.29 ... - rack RZ3.Reihe3.R10 -2 28.49664 - 29 TiB ... 74.79 ... - host ceph01 0 hdd 2.70000 1.00000 2.7 TiB ... 73.82 ... 88 osd.0 1 hdd 3.63869 1.00000 3.6 TiB ... 73.47 ... 121 osd.1 4 hdd 2.70000 1.00000 2.7 TiB ... 74.63 ... 89 osd.4 8 hdd 2.70000 1.00000 2.7 TiB ... 77.10 ... 92 osd.8 12 hdd 2.70000 1.00000 2.7 TiB ... 78.76 ... 94 osd.12 14 hdd 5.45799 1.00000 5.5 TiB ... 78.86 ... 193 osd.14 18 hdd 1.89999 1.00000 2.7 TiB ... 63.79 ... 76 osd.18 22 hdd 1.70000 1.00000 1.7 TiB ... 74.85 ... 57 osd.22 30 hdd 1.70000 1.00000 1.7 TiB ... 76.34 ... 59 osd.30 64 hdd 3.29999 1.00000 3.3 TiB ... 73.48 ... 110 osd.64 -11 12.39999 - 12 TiB ... 73.40 ... - host ceph04 34 hdd 5.20000 1.00000 5.2 TiB ... 72.81 ... 171 osd.34 44 hdd 7.20000 1.00000 7.2 TiB ... 73.83 ... 236 osd.44 -15 24.89998 - 25 TiB ... 74.15 ... - host ceph07 66 hdd 7.20000 1.00000 7.2 TiB ... 74.07 ... 236 osd.66 67 hdd 7.20000 1.00000 7.2 TiB ... 73.74 ... 236 osd.67 68 hdd 3.29999 1.00000 3.3 TiB ... 72.99 ... 110 osd.68 69 hdd 7.20000 1.00000 7.2 TiB ... 75.18 ... 241 osd.69 -1 0 - 0 B ... 0 ... - root default TOTAL 213 TiB ... 69.56 root@ceph01:~# ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.256761", "last_optimize_started": "Mon Mar 4 10:25:10 2024", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } Where as reweight-by-utilization would change quite a bit!: root@ceph01:~# ceph osd test-reweight-by-utilization 110 .5 10 moved 260 / 6627 (3.92334%) avg 127.442 stddev 81.345 -> 78.169 (expected baseline 11.18) min osd.17 with 24 -> 16 pgs (0.188321 -> 0.125547 * mean) max osd.42 with 401 -> 320 pgs (3.14652 -> 2.51094 * mean) oload 110 max_change 0.5 max_change_osds 10 average_utilization 0.6956 overload_utilization 0.7652 osd.42 weight 1.0000 -> 0.7561 osd.6 weight 0.4500 -> 0.6616 osd.17 weight 0.8500 -> 0.6466 osd.20 weight 0.8500 -> 0.6688 osd.5 weight 0.7501 -> 0.6004 osd.11 weight 0.9000 -> 0.7326 osd.43 weight 1.0000 -> 0.8199 osd.27 weight 0.9000 -> 0.7430 osd.63 weight 0.9000 -> 0.7440 osd.10 weight 0.8500 -> 0.7104 no change root@ceph01:~# ceph versions { "mon": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 2 }, "osd": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 52 }, "mds": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4 }, "overall": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 61 } } Cheers, Michael

5 4

v16.2.15 Pacific released
by Yuri Weinstein 04 Mar '24

04 Mar '24

We're happy to announce the 15th, and expected to be the last, backport release in the Pacific series. https://ceph.io/en/news/blog/2024/v16-2-15-pacific-released/ Notable Changes --------------- * `ceph config dump --format <json|xml>` output will display the localized option names instead of their normalized version. For example, "mgr/prometheus/x/server_port" will be displayed instead of "mgr/prometheus/server_port". This matches the output of the non pretty-print formatted version of the command. * CephFS: MDS evicts clients who are not advancing their request tids, which causes a large buildup of session metadata, resulting in the MDS going read-only due to the RADOS operation exceeding the size threshold. The `mds_session_metadata_threshold` config controls the maximum size that an (encoded) session metadata can grow. * RADOS: The `get_pool_is_selfmanaged_snaps_mode` C++ API has been deprecated due to its susceptibility to false negative results. Its safer replacement is `pool_is_in_selfmanaged_snaps_mode`. * RBD: When diffing against the beginning of time (`fromsnapname == NULL`) in fast-diff mode (`whole_object == true` with `fast-diff` image feature enabled and valid), diff-iterate is now guaranteed to execute locally if exclusive lock is available. This brings a dramatic performance improvement for QEMU live disk synchronization and backup use cases. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.15.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 618f440892089921c3e944a991122ddc44e60516

2 1

[Quincy] cannot configure dashboard to listen on all ports
by wodel youchi 04 Mar '24

04 Mar '24

Hi, ceph dashboard fails to listen on all IPs. log_channel(cluster) log [ERR] : Unhandled exception from module 'dashboard' while running on mgr.controllera: OSError("No socket could be created -- (('0.0.0.0', 8443): [Errno -2] Name or service not known) -- (('::', 8443, 0, 0): ceph version 17.2.7 quincy (stable) Regards.

1 0

ceph-crash NOT reporting crashes due to wrong permissions on /var/lib/ceph/crash/posted (Debian / Ubuntu packages)
by Christian Rohmann 04 Mar '24

04 Mar '24

Hey ceph-users, I just noticed issues with ceph-crash using the Debian /Ubuntu packages (package: ceph-base): While the /var/lib/ceph/crash/posted folder is created by the package install, it's not properly chowned to ceph:ceph by the postinst script. This might also affect RPM based installs somehow, but I did not look into that. I opened a bug report with all the details and two ideas to fix this: https://tracker.ceph.com/issues/64548 The wrong ownership causes ceph-crash to NOT work at all. I myself missed quite a few crash reports. All of them were just sitting around on the machines, but were reported right after I did chown ceph:ceph /var/lib/ceph/crash/posted systemctl restart ceph-crash.service You might want to check if you might be affected as well. Failing to post crashes to the local cluster results in them not being reported back via telemetry. Regards Christian

3 3

Ceph orch doesn't execute commands and doesn't report correct status of daemons
by wodel youchi 03 Mar '24

03 Mar '24

Hi, I have finished the conversion from ceph-ansible to cephadm yesterday. Everything seemed to be working until this morning, I wanted to redeploy rgw service to specify the network to be used. So I deleted the rgw services with ceph orch rm, then I prepared a yml file with the new conf. I applied the file and the new rgw service was started but it was launched with an external image, so I wanted to redeploy using my local image so I did a redeploy ... and then nothing happened, I get the rescheduled message but nothing happened, then I restarted one of the controllers, the orchestrator doesn't seem to be aware that some service have restarted??? PS : I don't fully master the cephadm command line and use. Regards.

2 3

[Quincy] NFS ingress mode haproxy-protocol not recognized
by wodel youchi 03 Mar '24

03 Mar '24

Hi; I tried to create an NFS cluster using this command : [root@controllera ceph]# ceph nfs cluster create mynfs "3 controllera controllerb controllerc" --ingress --virtual_ip 20.1.0.201 --ingress-mode haproxy-protocol Invalid command: haproxy-protocol not in default|keepalive-only And I got this error : Invalid command haproxy-protocol I am using Quincy : ceph version 17.2.7 (...) quincy (stable) Is it not supported yet? Regards.

2 1

Question about erasure coding on cephfs
by Erich Weiler 03 Mar '24

03 Mar '24

Hi Y'all, We have a new ceph cluster online that looks like this: md-01 : monitor, manager, mds md-02 : monitor, manager, mds md-03 : monitor, manager store-01 : twenty 30TB NVMe OSDs store-02 : twenty 30TB NVMe OSDs The cephfs storage is using erasure coding at 4:2. The crush domain is set to "osd". (I know that's not optimal but let me get to that in a minute) We have a current regular single NFS server (nfs-01) with the same storage as the OSD servers above (twenty 30TB NVME disks). We want to wipe the NFS server and integrate it into the above ceph cluster as "store-03". When we do that, we would then have three OSD servers. We would then switch the crush domain to "host". My question is this: Given that we have 4:2 erasure coding, would the data rebalance evenly across the three OSD servers after we add store-03 such that if a single OSD server went down, the other two would be enough to keep the system online? Like, with 4:2 erasure coding, would 2 shards go on store-01, then 2 shards on store-02, and then 2 shards on store-03? Is that how I understand it? Thanks for any insight! -erich

3 2

Re: has anyone enabled bdev_enable_discard?
by jsterr＠deckplosion.de 02 Mar '24

02 Mar '24

Is there any update on this? Did someone test the option and has performance values before and after? Is there any good documentation regarding this option?

7 7

Re: Scrub stuck and 'pg has invalid (post-split) stat'
by Eugen Block 01 Mar '24

01 Mar '24

Please don't drop the list from your response. The first question coming to mind is, why do you have a cache-tier if all your pools are on nvme decices anyway? I don't see any benefit here. Did you try the suggested workaround and disable the cache-tier? Zitat von Cedric <yipikai7(a)gmail.com>: > Thanks Eugen, see attached infos. > > Some more details: > > - commands that actually hangs: ceph balancer status ; rbd -p vms ls ; > rados -p vms_cache cache-flush-evict-all > - all scrub running on vms_caches pgs are stall / start in a loop > without actually doing anything > - all io are 0 both from ceph status or iostat on nodes > > On Tue, Feb 20, 2024 at 10:00 AM Eugen Block <eblock(a)nde.ag> wrote: >> >> Hi, >> >> some more details would be helpful, for example what's the pool size >> of the cache pool? Did you issue a PG split before or during the >> upgrade? This thread [1] deals with the same problem, the described >> workaround was to set hit_set_count to 0 and disable the cache layer >> until that is resolved. Afterwards you could enable the cache layer >> again. But keep in mind that the code for cache tier is entirely >> removed in Reef (IIRC). >> >> Regards, >> Eugen >> >> [1] >> https://ceph-users.ceph.narkive.com/zChyOq5D/ceph-strange-issue-after-addin… >> >> Zitat von Cedric <yipikai7(a)gmail.com>: >> >> > Hello, >> > >> > Following an upgrade from Nautilus (14.2.22) to Pacific (16.2.13), we >> > encounter an issue with a cache pool becoming completely stuck, >> > relevant messages below: >> > >> > pg xx.x has invalid (post-split) stats; must scrub before tier agent >> > can activate >> > >> > In OSD logs, scrubs are starting in a loop without succeeding for all >> > pg of this pool. >> > >> > What we already tried without luck so far: >> > >> > - shutdown / restart OSD >> > - rebalance pg between OSD >> > - raise the memory on OSD >> > - repeer PG >> > >> > Any idea what is causing this? any help will be greatly appreciated >> > >> > Thanks >> > >> > Cédric >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 5

What's up with 16.2.15?
by Zakhar Kirpichenko 01 Mar '24

01 Mar '24

Hi, We randomly got several Pacific package updates to 16.2.15 available for Ubuntu 20.04. As far as I can see, 16.2.15 hasn't been released and there's been no release announcement. The updates seem to be no longer available. What's going on with 16.2.15? /Z

2 1

Renaming an OSD node
by Deep Dish 01 Mar '24

01 Mar '24

Hello. We have a requirement to change the hostname on some of our OSD nodes. All of our nodes are Ubuntu 22.04 based and have been deployed using 17.2.7 Orchestrator. 1. Is there a procedure to rename the existing node, without rebuilding and have it detected by Ceph Orchestrator? If not, 2. To minimize impact on cluster (rebuilding OSDs / balancing, etc) Is it possible to REINTRODUCE existing OSDs into the cluster a the newly rebuilt node? Is there a ceph orch process to scan local node OSDs, detect and create OSD daemons? Thank you.

2 1

Slow RGW multisite sync due to "304 Not Modified" responses on primary zone
by praveenkumargpk17＠gmail.com 01 Mar '24

01 Mar '24

Hi, We have 2 clusters (v18.2.1) primarily used for RGW which has over 2+ billion RGW objects. They are also in multisite configuration totaling to 2 zones and we've got around 2 Gbps of bandwidth dedicated (P2P) for the multisite traffic. We see that using "radosgw-admin sync status" on the zone 2, all the 128 shards are recovering and unfortunately there is very less data transfer from primary zone ie., the link utilization is barely 100 Mbps / 2 Gbps. Our objects are quite small as well like avg. of 1 MB in size. On further inspection, we noticed the rgw access the logs at primary site are mostly yielding "304 Not Modified" for RGWs at site-2. Is this expected? Here are some of the logs (information is redacted) root@host-04:~# tail -f /var/log/haproxy-msync.log Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:33730 [12/Feb/2024:05:06:51.047] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/1/0/0 0/0 "GET /bucket1/object1.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:59730 [12/Feb/2024:05:06:51.048] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/3/1/0 0/0 "GET /bucket1/object91.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" We also took a look at our Grafana instance and out of 1000 requests / second, 200 requests are getting "200 OK" and 800 requests are getting "304 Not Modified". Sync threads are run on only 2 rgw daemons per zone and are behind a Load Balancer. "# radosgw-admin sync error list" also contains around 20 errors which are mostly automatically recoverable. As we understand, does it mean that RGW multisite sync logs in the log pool are yet to be generated or some sort? Please provide us some insights and let us know how to resolve this. Thanks , Praveen

1 0