- ceph-users - lists.ceph.io

ceph metrics units
by Denis Polom 14 Mar '24

14 Mar '24

Hi guys, do you know if there is some table of Ceph metrics and units that should be used for them? I currently struggling with ceph_osd_op_r_latency_sum ceph_osd_op_w_latency_sum if they are in ms or seconds? Any idea please? Thx!

2 1

Fwd: Ceph fs snapshot problem
by Marcus 14 Mar '24

14 Mar '24

Hi all, I have just setup a small ceph cluster with ceph fs. The setup is reef 18.2.1 on Debian bookworm. The system is up and running the way it should, though I have a problem with ceph fs snapshots. When I read the doc I should be able to make a snapshot in any directory in the filesystem. I can do a snapshot in the root of the filesystem but if I try somewhere else I get: Operation not permitted This is the same if I do it with mkdir or with ceph fs subvolume snapshot create ... I have created an auth client with rws: [client.snap-mount] key = **** caps mds = "allow rws fsname=gds-common" caps mon = "allow r fsname=gds-common" caps osd = "allow rw tag cephfs data=gds-common" Where the filsystem is called gds-common, saved in a file on the client: /etc/ceph/ceph.client.snap-mount.keyring I mount ceph fs with: mount -t ceph :/ -o name=snap-mount /mnt If I create a snapshot in root, it works fine, as in: mkdir /mnt/.snap/mysnap I also notice that in every subdir there is a "snapshot dir" as well with the name _mysnap_1, as in: /mnt/dir/.snap/_mysnap_1 My guess that is is a part of the snapshot system, this "snapshot" dissapear when the snapshot is removed with: rmdir /mnt/.snap/mysnap If I try to make a snapshot in another directory this does not work: mkdir /mnt/dir/.snap/othersnap Get the error: cannot create directory ‘/mnt/dir/.snap/othersnap’: Operation not permitted It is the same thing on the commandline, root works: ceph fs subvolume snapshot create gds-common / fromcmd But not in a subdir: ceph fs subvolume snapshot create gds-common /dir dirsnap Error EINVAL: invalid value specified for ceph.dir.subvolume I also notice that when you use any command of type: ceph fs subvolume snapshot ... You get a new directory (volumes) in the root: /mnt/volumes/_legacy/6666cd76f96956469e7be39d750cc7d9.meta I do not know if I am missing something, some lacking of config or so. Thanks for your help!! Best regards Marcus

1 0

18.2.2 dashboard really messed up.
by Harry G Coin 14 Mar '24

14 Mar '24

Looking at ceph -s, all is well. Looking at the dashboard, 85% of my capacity is 'warned', and 95% is 'in danger'. There is no hint given as to the nature of the danger or reason for the warning. Though apparently with merely 5% of my ceph world 'normal', the cluster reports 'ok'. Which, you know, seems contradictory. I've used just under 40% of capacity. Further down the dashboard, all the subsections of 'Cluster Utilization' are '1' and '0.5' with nothing whatever in the graphics area. Previous versions of ceph presented a normal dashboard. It's just a little half rack, 5 hosts, a few physical drives each, been running ceph for a couple years now. Orchestrator is cephadm. It's just about as 'plain vanilla' at it gets. I've had to mute one alert, because cephadm refresh aborts when it finds drives on any host that have nothing to do with ceph that don't have a blkid_ip 'TYPE' key. Seems unrelated to a totally messed up dashboard. (The tracker for that is here: https://tracker.ceph.com/issues/63502 ). Any idea what the steps are to get useful stuff back on the dashboard? Any idea where I can learn what my 85% danger and 95% warning is 'about'? (You'd think 'danger' (The volcano is blowing up now!) would be worse than 'warning' (the volcano might blow up soon) , so how can warning+danger > 100%, or if not additive how can warning < danger?) Here's a bit of detail: root@noc1:~# ceph -s cluster: id: 4067126d-01cb-40af-824a-881c130140f8 health: HEALTH_OK (muted: CEPHADM_REFRESH_FAILED) services: mon: 5 daemons, quorum noc4,noc2,noc1,noc3,sysmon1 (age 70m) mgr: noc2.yhyuxd(active, since 82m), standbys: noc4.tvhgac, noc3.sybsfb, noc1.jtteqg mds: 1/1 daemons up, 3 standby osd: 27 osds: 27 up (since 20m), 27 in (since 2d) data: volumes: 1/1 healthy pools: 16 pools, 1809 pgs objects: 12.29M objects, 17 TiB usage: 44 TiB used, 67 TiB / 111 TiB avail pgs: 1793 active+clean 9 active+clean+scrubbing 7 active+clean+scrubbing+deep io: client: 5.6 MiB/s rd, 273 KiB/s wr, 41 op/s rd, 58 op/s wr

2 3

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

bluestore_min_alloc_size and bluefs_shared_alloc_size
by Joel Davidow 13 Mar '24

13 Mar '24

Summary ---------- The relationship of the values configured for bluestore_min_alloc_size and bluefs_shared_alloc_size are reported to impact space amplification, partial overwrites in erasure coded pools, and storage capacity as an osd becomes more fragmented and/or more full. Previous discussions including this topic ---------------------------------------- comment #7 in bug 63618 in Dec 2023 - https://tracker.ceph.com/issues/63618#note-7 pad writeup related to bug 62282 likely from late 2023 - https://pad.ceph.com/p/RCA_62282 email sent 13 Sept 2023 in mail list discussion of cannot create new osd - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/5M4QAXJDCN… comment #9 in bug 58530 likely from early 2023 - https://tracker.ceph.com/issues/58530#note-9 email sent 30 Sept 2021 in mail list discussion of flapping osds - https://www.mail-archive.com/ceph-users@ceph.io/msg13072.html email sent 25 Feb 2020 in mail list discussion of changing allocation size - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/B3DGKH6THFG… Current situation ----------------- We have three Ceph clusters that were originally built via cephadm on octopus and later upgraded to pacific. All osds are HDD (will be moving to wal+db on SSD) and were resharded after the upgrade to enable rocksdb sharding. The value for bluefs_shared_alloc_size has remained unchanged at 65535. The value for bluestore_min_alloc_size_hdd was 65535 in octopus but is reported as 4096 by ceph daemon osd.<id> config show in pacific. However, the osd label after upgrading to pacific retains the value of 65535 for bfm_bytes_per_block. BitmapFreelistManager.h in Ceph source code (src/os/bluestore/BitmapFreelistManager.h) indicates that bytes_per_block is bdev_block_size. This indicates that the physical layout of the osd has not changed from 65535 despite the return of the ceph dameon command reporting it as 4096. This interpretation is supported by the Minimum Allocation Size part of the Bluestore configuration reference for quincy (https://docs.ceph.com/en/quincy/rados/configuration/bluestore-config-ref/#m…) Questions ---------- What are the pros and cons of the following three cases with two variations per case - when using co-located wal+db on HDD and when using separate wal+db on SSD: 1) bluefs_shared_alloc_size, bluestore_min_alloc_size, and bfm_bytes_per_block all equal 2) bluefs_shared_alloc_size greater than but a multiple of bluestore_min_alloc_size with bfm_bytes_per_block equal to bluestore_min_alloc_size 3) bluefs_shared_alloc_size greater than but a multiple of bluestore_min_alloc_size with bfm_bytes_per_block equal to bluefs_shared_alloc_size

4 6

ceph osd different size to create a cluster for Openstack : asking for advice
by wodel youchi 13 Mar '24

13 Mar '24

Hi, I need some guidance from you folks... I am going to deploy a ceph cluster in HCI mode for an openstack platform. My hardware will be : - 03 control nodes : - 27 osd nodes : each node has 03x3.8To nvme + 01x1.9To nvme disks (those disks will all be used as OSDs) In my Openstack I will be creating all sorts of pools : RBD, Cephfs and RGW. I am planning to create two crush rules using the disk size as a parameter. Then divide my pools between the two rules. - RBD to use the 3.8To disks since I need more space here. - Cephfs and RGW to use 1.9To disks. Is this a good configuration? Regards

2 1

ceph osd crush reweight rounding issue
by Stefan Kooman 13 Mar '24

13 Mar '24

Hi, After some tests with OSD crush weight I wanted to restore the original weight of the OSD. But that proves to be difficult. See the following example: Situation before OSD crush reweight has taken place ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -11 0.39197 datacenter dc1 -5 0.39197 host node1 2 hdd 0.09798 osd.2 up 1.00000 1.00000 3 hdd 0.09798 osd.3 up 1.00000 1.00000 6 ssd 0.09798 osd.6 up 1.00000 1.00000 7 ssd 0.09798 osd.7 up 1.00000 1.00000 -10 0.39197 datacenter dc2 -3 0.39197 host node2 0 hdd 0.09798 osd.0 up 1.00000 1.00000 1 hdd 0.09798 osd.1 up 1.00000 1.00000 4 ssd 0.09798 osd.4 up 1.00000 1.00000 5 ssd 0.09798 osd.5 up 1.00000 1.00000 -1 0 root default ceph osd crush reweight osd.5 0.1 ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -11 0.39197 datacenter dc1 -5 0.39197 host node1 2 hdd 0.09798 osd.2 up 1.00000 1.00000 3 hdd 0.09798 osd.3 up 1.00000 1.00000 6 ssd 0.09798 osd.6 up 1.00000 1.00000 7 ssd 0.09798 osd.7 up 1.00000 1.00000 -10 0.39392 datacenter dc2 -3 0.39392 host node2 0 hdd 0.09798 osd.0 up 1.00000 1.00000 1 hdd 0.09798 osd.1 up 1.00000 1.00000 4 ssd 0.09798 osd.4 up 1.00000 1.00000 5 ssd 0.09999 osd.5 up 1.00000 1.00000 -1 0 root default ceph osd crush reweight osd.5 0.09798 reweighted item id 5 name 'osd.5' to 0.09798 in crush map ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -11 0.39197 datacenter dc1 -5 0.39197 host node1 2 hdd 0.09798 osd.2 up 1.00000 1.00000 3 hdd 0.09798 osd.3 up 1.00000 1.00000 6 ssd 0.09798 osd.6 up 1.00000 1.00000 7 ssd 0.09798 osd.7 up 1.00000 1.00000 -10 0.39191 datacenter dc2 -3 0.39191 host dc2 0 hdd 0.09798 osd.0 up 1.00000 1.00000 1 hdd 0.09798 osd.1 up 1.00000 1.00000 4 ssd 0.09798 osd.4 up 1.00000 1.00000 5 ssd 0.09798 osd.5 up 1.00000 1.00000 -1 0 root default The difference is (0.39197 - 0.09798) = .00006. If we divide this by four OSDs per host bucket it is .000015. So each OSD could be reweighted to 0.097995 to give the host / data center buckets equal weights. This indeed works, after reweighting all OSDs with 0.097995 -10 0.39197 datacenter dc2 -3 0.39197 host node2 0 hdd 0.09799 osd.0 up 1.00000 1.00000 1 hdd 0.09799 osd.1 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -1 0 root default Note however that the OSD weights now differ from (identical) OSDs in the other datacenter. Why is this such a big deal? Well, Ceph "stretch mode" requires each bucket to be of equal weight. And IMHO it should be easily possible to set the weight Ceph CRUSH internally uses to do its calculations. Anyone has more info on how to do this? Gr. Stefan P.s. Sure, I could reweight all OSDs to a new weight, but that's more of a workaround than a proper way of fixing it.

1 0

Hanging request in S3
by Christian Kugler 12 Mar '24

12 Mar '24

Hi, I am having some trouble with some S3 requests and I am at a loss. After upgrading to reef a couple of weeks ago some requests get stuck and never return. The two Ceph clusters are set up to sync the S3 realm bidirectionally. The bucket has 479 shards (dynamic resharding) at the moment. Putting an object (/etc/services) into the bucket via s3cmd works, and deleting it works as well. So I know it is not just the entire bucket that is somehow faulty. When I try to delete a specific prefix it the request for listing all objects never comes back. In the example below I only included the request in question which I aborted with ^C. $ s3cmd rm -r s3://sql20/pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/ -d [...snip...] DEBUG: Canonical Request: GET /sql20/ prefix=pgbackrest%2Fbackup%2Fadrpb%2F20240130-200410F%2Fpg_data%2Fbase%2F16560%2F host:[...snip...] x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20240306T183435Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20240306T183435Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=VL0FRB7CYGMHBGCD419M/20240306/[...snip...]/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=45b133675535ab611bbf2b9a7a6e40f9f510c0774bf155091dc9a05b76856cb7', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(sql20): [...snip...] DEBUG: ConnMan.get(): re-using connection: [...snip...]#1 DEBUG: format_uri(): /sql20/?prefix=pgbackrest%2Fbackup%2Fadrpb%2F20240130-200410F%2Fpg_data%2Fbase%2F16560%2F DEBUG: Sending request method_string='GET', uri='/sql20/?prefix=pgbackrest%2Fbackup%2Fadrpb%2F20240130-200410F%2Fpg_data%2Fbase%2F16560%2F', headers={'x-amz-date': '20240306T183435Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=VL0FRB7CYGMHBGCD419M/20240306/[...snip...]/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=45b133675535ab611bbf2b9a7a6e40f9f510c0774bf155091dc9a05b76856cb7', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) ^CDEBUG: Response: {} See ya! The request did not show up normally in the logs so I set debug_rgw=20 and debug_ms=20 via ceph config set. I tried to isolate the request and looked for its request id: 13321243250692796422 The following is a grep for the request id: Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket verifying op params Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket pre-executing Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket check rate limiting Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket executing Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket list_objects_ordered: starting attempt 1 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.000000000s s3:list_bucket cls_bucket_list_ordered: request from each of 479 shard(s) for 8 entries to get 1001 total entries Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101438318.gz from shard 437 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101438318.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: skipping pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101438318.gz[] Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457659_fsm.gz from shard 202 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457659_fsm.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: skipping pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457659_fsm.gz[] Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457662_fsm.gz from shard 420 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457662_fsm.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket manifest: total_size = 59 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: setting s->obj_tag to 107ace7a-a829-4d1c-9cb8-9db30644b786.395658.17352021385636272460 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: got pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457662_fsm.gz Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457721.gz from shard 444 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457721.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: skipping pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101457721.gz[] Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458889.gz from shard 116 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458889.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket manifest: total_size = 1040 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: setting s->obj_tag to 107ace7a-a829-4d1c-9cb8-9db30644b786.395658.2135820855302089522 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: got pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458889.gz Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458903.gz from shard 452 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458903.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: skipping pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458903.gz[] Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458955.gz from shard 55 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.332010120s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458955.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cls_bucket_list_ordered: skipping pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101458955.gz[] Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101600036.gz from shard 455 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101600036.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket manifest: total_size = 4403 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket get_obj_state: setting s->obj_tag to 107ace7a-a829-4d1c-9cb8-9db30644b786.395658.13169705319307666922 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cls_bucket_list_ordered: got pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101600036.gz Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cls_bucket_list_ordered: currently processing pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101600044.gz from shard 407 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket get_obj_state: rctx=0x7f74bdc6f860 obj=sql20:pgbackrest/backup/adrpb/20240130-200410F/pg_data/base/16560/101600044.gz state=0x55d4237419e8 s->prefetch_data=0 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket manifest: total_size = 6264856 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket get_obj_state: setting s->obj_tag to 107ace7a-a829-4d1c-9cb8-9db30644b786.395658.12884446303569321109 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket bucket index object: rechenzentrum.rgw.buckets.index:.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3.1.34 Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cache get: name=rechenzentrum.rgw.log++bucket.sync-source-hints.sql20 : hit (negative entry) Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket cache get: name=rechenzentrum.rgw.log++bucket.sync-target-hints.sql20 : hit (negative entry) Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket reflect(): flow manager (bucket=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3): adding source pipe: {s={b=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3,z=3caabb9a-4e3b-4b8a-8222-34c33dd63210,az=0},d={b=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3,z=107ace7a-a829-4d1c-9cb8-9db30644b786,az=0}} Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket reflect(): flow manager (bucket=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3): adding dest pipe: {s={b=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3,z=107ace7a-a829-4d1c-9cb8-9db30644b786,az=0},d={b=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3,z=3caabb9a-4e3b-4b8a-8222-34c33dd63210,az=0}} Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=3caabb9a-4e3b-4b8a-8222-34c33dd63210,az=0},d={b=*,z=107ace7a-a829-4d1c-9cb8-9db30644b786,az=0}} Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=107ace7a-a829-4d1c-9cb8-9db30644b786,az=0},d={b=*,z=3caabb9a-4e3b-4b8a-8222-34c33dd63210,az=0}} Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket chain_cache_entry: cache_locator= Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket chain_cache_entry: couldn't find cache locator Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket couldn't put bucket_sync_policy cache entry, might have raced with data changes Mär 06 19:36:17 radosgw[8318]: req 13321243250692796422 0.336010247s s3:list_bucket RGWDataChangesLog::add_entry() bucket.name=sql20 shard_id=34 now=2024-03-06T18:36:17.978389+0000 cur_expiration=1970-01-01T00:00:00.000000+0000 I don't see any clear error but somehow the last view lines are odd to me: - When before it said: flow manager (bucket=sql20:3caabb9a-4e3b-4b8a-8222-34c33dd63210.10724501.3) it has no more bucket: flow manager (bucket=) - no cache locator found. No idea if this is okay or not - The cur_expiration a few lines later is set to unix time 0 (1970-01-01T00:00:00.000000+0000) - I did this multiple times and it seems to always be shard 34 that has the issue Did someone see something like this before? Any ideas how to remedy the situation or at least where to or what to look for? Best, Christian

2 2

Ceph-Cluster integration with Ovirt-Cluster
by ankit＠eurus.net 12 Mar '24

12 Mar '24

Hi Guys, I have a running ovirt-4.3 cluster with 1 manager and 4 hypervisors nodes and for storage using traditional SAN storage which is connect using iscsi. where i can create VM's and assign storge from SAN. This is running fine since a decade but now i want to move from traditional SAN storage to ceph storage cluster due to slow speed and scalability issues of storage. But i am a very newbie in ceph but able to install ceph-cluster reef_v18 in my lab environment with 5 VM's (2 Mon's and Manager and 3 OSD's) on Debian 12. And i have installed ovirt 4.5 with 1 hypervisor on VM's on centos stream 9 and want to integrate it with ceph-cluster so that i can use ceph-cluster storage like i am using SAN with my ovirt and I have tried to do some google and read ceph documentation but did not find anything regarding this. So how can i do it? If you need any other information, please let me know! Many Thanks! PJ111288

1 0

Elasticsearch sync module | Ceph Issue
by Lokendra Rathour 12 Mar '24

12 Mar '24

Hi Team, We are working on Elasticsearch sync module integration with ceph. Ceph version: 18.2.5 (reef) Elasticsearch: 8.2.1 Problem statement: The Syncing between the zones are not happenning: Links followed to perform the integration: https://ceph.io/en/news/blog/2017/new-luminous-rgw-metadata-search/ https://www.suse.com/c/rgw-elasticsearch/ Current status of radosgw-admin sync status: realm 1fc3fe5f-2bb8-4134-8200-4b26479f7709 (gold) zonegroup 59d8be05-0a31-4610-8f7f-d80a6d221f2e (zone-zg) zone ce1e53a5-5f30-42c5-bcca-a69b60411e1a (zone) current time 2024-03-11T02:31:17Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) data sync source: e012b80d-cb04-4ab2-b78f-13b1dea0156d (zone-es) not syncing from zone Configuration of ceph.conf: [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = d5b53cd2-bef4-41a3-8667-135a3e02fc25 mon host = [v2:[fd00:fd00:fd00:9900::34]:3300,v1:[fd00:fd00:fd00:9900::34]:6789],[v2:[fd00:fd00:fd00:9900::35]:3300,v1:[fd00:fd00:fd00:9900::35]:6789],[v2:[fd00:fd00:fd00:9900::36]:3300,v1:[fd00:fd00:fd00:9900::36]:6789] public network = fd00:fd00:fd00:9900::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 31380104806 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[fd00:fd00:fd00:9900::34]:8080 rgw thread pool size = 512 [client.rgw.zone] rgw_frontends="beast port=8000" rgw_zone=zone host = storagenode1 keyring = /etc/ceph/ceph.client.zone.keyring log file = /var/log/radosgw/rgw.zone.radosgw.log [client.rgw.zone-es] rgw_frontends="beast port=8002" rgw_zone=zone-es host = storagenode1 keyring = /etc/ceph/ceph.client.zone-es.keyring log file = /var/log/radosgw/rgw.zone-es.radosgw.log Could anyone please help. -- ~ Lokendra skype: lokendrarathour

1 0