- ceph-users - lists.ceph.io

General best practice for stripe unit and count if I want to change object size
by Nathan Morrison 11 Mar '24

11 Mar '24

This was asked in reddit and was requested to post here: So in RBD, say I want to make an image that's got an object size of 1M instead of the default 4M (if it will be a VM say, and likely not have too many big files in it, just OS files mostly). I also know I don't wanna go too crazy and make 4k objects or the cluster will blow up with number of objects. What's a good rule of thumb on what to set the stripe unit and count in relation to object size? Also how can I see the stripe unit and count for an image, it seems "rbd info <img-name>" doesn't show it, only object size (or order). Would this be sensible (assuming old img is default 4M obj size) or really stupid and why? rbd cp --object-size 1M --stripe-unit 1M --stripe-count 1 pool/old-img pool/new-img I'm trying to get my head around striping and I tried using this page (https://docs.ceph.com/en/latest/dev/file-striping/) to understand what's going on low level, but it's just leaving me more confused the more I read it.

2 1

Telemetry endpoint down?
by Konstantin Shalygin 11 Mar '24

11 Mar '24

Hi, seems telemetry endpoint is down for a some days? We have connection errors from multiple places 1:ERROR Mar 10 00:46:10.653 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 2:ERROR Mar 10 01:48:20.061 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 3:ERROR Mar 10 02:50:29.473 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 4:ERROR Mar 10 03:52:38.877 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 5:ERROR Mar 10 04:54:48.285 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 6:ERROR Mar 10 05:56:57.693 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 7:ERROR Mar 10 06:59:07.105 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 8:ERROR Mar 10 08:01:16.509 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 9:ERROR Mar 10 09:03:25.917 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 <http://telemetry.ceph.com:443/> Thanks, k

2 2

AMPQS support in Nautilus
by Manuel Negron 11 Mar '24

11 Mar '24

Hello, ive been trying to setup bucket notifications using amqs (according to the documentation is possible) however when checking the logs the push fails saying "invalid schema". any ideas wha the issue could be ? kind regards

2 1

v18.2.2 Reef (hot-fix) released
by Yuri Weinstein 11 Mar '24

11 Mar '24

We're happy to announce the 2nd hotfix release in the Reef series. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2024/v18-2-2-reef-released/ Notable Changes --------------- * mgr/Prometheus: refine the orchestrator availability check to prevent against crashes in the prometheus module during startup. Introduce additional checks to handle daemon_ids generated within the Rook environment, thus preventing potential issues during RGW metrics metadata generation. Related tracker: https://tracker.ceph.com/issues/64721 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.2.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2

1 0

MANY_OBJECT_PER_PG on 1 pool which is cephfs_metadata
by Edouard FAZENDA 11 Mar '24

11 Mar '24

Hello Ceph community, I have since this morning a warning about MANY_OBJECT_PER_PG on 1 pool which is cephfs_metadata # ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average [WRN] MANY_OBJECTS_PER_PG: 1 pools have many more objects per pg than average pool cephfs_metadata objects per pg (154151) is more than 10.0215 times cluster average (15382) I have the autoscaling on on all the pool : # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE device_health_metrics 9523k 3.0 26827G 0.0000 1.0 1 on cephfs_data 5389G 2.0 26827G 0.4018 1.0 512 on cephfs_metadata 19365M 2.0 26827G 0.0014 4.0 16 on .rgw.root 1323 3.0 26827G 0.0000 1.0 32 on default.rgw.log 23552 3.0 26827G 0.0000 1.0 32 on default.rgw.control 0 3.0 26827G 0.0000 1.0 32 on default.rgw.meta 11911 3.0 26827G 0.0000 4.0 8 on default.rgw.buckets.index 0 3.0 26827G 0.0000 4.0 8 on default.rgw.buckets.data 497.0G 3.0 26827G 0.0556 1.0 32 on kubernetes 177.2G 2.0 26827G 0.0132 1.0 32 on default.rgw.buckets.non-ec 432 3.0 26827G 0.0000 1.0 32 on Actually the pg_num is 16 for the cephfs_metdata pool , but it does not define NEW_PG_NUM Here the replicated size of all my pool # ceph osd dump | grep 'replicated size' pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 189372 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth pool 10 'cephfs_data' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 189346 lfor 0/0/183690 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs pool 11 'cephfs_metadata' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 187861 lfor 0/187861/187859 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 18 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 5265 flags hashpspool stripe_width 0 application rgw pool 19 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 5267 flags hashpspool stripe_width 0 application rgw pool 20 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 5269 flags hashpspool stripe_width 0 application rgw pool 21 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 5398 lfor 0/5398/5396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 22 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 7491 lfor 0/7491/7489 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 23 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 7500 flags hashpspool stripe_width 0 application rgw pool 24 'kubernetes' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 189363 lfor 0/0/7560 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 25 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 23983 flags hashpspool stripe_width 0 application rgw Why the autoscaler is not acting to increase the pg_num of the pool in warning ? As the pgcalc is not available on ceph now , do you thing it is a good idea to increase manually the pg_num of the cephfs_metadata , but which value should I set ? I have 18 OSD Thanks for the help Best Regards, Edouard FAZENDA Technical Support Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 <https://www.csti.ch/> www.csti.ch

3 2

Dashboard building issue "RuntimeError: memory access out of bounds"?
by 张东川 11 Mar '24

11 Mar '24

Hi there, I was building ceph with the tag "v19.0.0" on Milkv Pioneer board (RISCV arch, OS is fedora-riscv 6.1.55) I ran "./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_SPDK=ON" then just went to the "build" folder and ran "ninja" command. But it failed with the following dashboard error: [1/2] cd /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend && . /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/node-env/bin/activate && npm config set cache /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/node-env/.npm --userconfig /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/node-env/.npmrc && deactivate [2/2] dashboard frontend is being created FAILED: src/pybind/mgr/dashboard/frontend/dist /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/dist cd /mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend && . /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/node-env/bin/activate && DASHBOARD_FRONTEND_LANGS="" npm run build:localize -- --output-path /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/dist --configuration=production --progress=false && deactivate > ceph-dashboard(a)0.0.0 build:localize > node cd --env --pre && ng build --localize --output-path /mnt/ssd/projects/test/ceph/build/src/pybind/mgr/dashboard/frontend/dist --configuration=production --progress=false [cd.js] './angular.backup.json' already exists, restoring it into './angular.json'} [cd.js] Preparing build of EN. [cd.js] 'src/environments/environment.tpl.ts' was copied to 'src/environments/environment.prod.ts' [cd.js] 'src/environments/environment.tpl.ts' was copied to 'src/environments/environment.ts' [cd.js] Writing to ./angular.json [cd.js] Placeholders were replace in 'src/environments/environment.prod.ts' [cd.js] Placeholders were replace in 'src/environments/environment.ts'     TypeScript compiler options "target" and "useDefineForClassFields" are set to "ES2022" and "false" respectively by the Angular CLI. To control ECMA version and features use the Browerslist configuration. For more information, see https://angular.io/guide/build#configuring-browser-compatibility     NOTE: You can set the "target" to "ES2022" in the project's tsconfig to remove this warning. wasm://wasm/7f04b08a:1 RuntimeError: memory access out of bounds     at wasm://wasm/7f04b08a:wasm-function[2]:0x322     at WasmHash.digest (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/webpack/lib/util/hash/wasm-hash.js:138:11)     at BatchedHash.digest (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/webpack/lib/util/hash/BatchedHash.js:64:20)     at /mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/webpack/lib/DefinePlugin.js:595:38     at _next42 (eval at create (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/tapable/lib/HookCodeFactory.js:19:10), <anonymous>:36:1)     at _next20 (eval at create (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/tapable/lib/HookCodeFactory.js:19:10), <anonymous>:97:1)     at Hook.eval [as call] (eval at create (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/tapable/lib/HookCodeFactory.js:19:10), <anonymous>:141:1)     at Hook.CALL_DELEGATE [as _call] (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/tapable/lib/Hook.js:14:14)     at Compiler.newCompilation (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/webpack/lib/Compiler.js:1122:26)     at /mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/webpack/lib/Compiler.js:1166:29     at eval (eval at create (/mnt/ssd/projects/test/ceph/src/pybind/mgr/dashboard/frontend/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:13:1) Node.js v18.17.0 ninja: build stopped: subcommand failed. Has anyone met this same issue before? How to fix it??? Please let me know if further info is needed.Thanks a lot. (I will try to build on the latest mainline code tonight to see if the same issue still exists) Best Regards, Dongchuan

1 0

Adding new OSD's - slow_ops and other issues.
by jskr＠topsoe.com 11 Mar '24

11 Mar '24

Hi. We have a cluster working very nicely since it was put up more than a year ago. Now we needed to add more NVMe drives to expand. After setting all the "no" flags.. we added them using $ ceph orch osd add .... The twist is that we have managed to get the default weights set to 1 for all disks not 7.68 (as the default for the ceph orch command. Thus we did a subsequent reweight to change weight -- and then removed the "no" flags. As a consequence we had a bunch of OSD's delivering slow_ops and -- after manually restarting osd's to get rid of them - the system returned to normal. ... second try... Same drill - but somehow the ceph orch command failed to bring the new OSD online before we ran the reweight command ... and it works flawlessly ... third try ... Same drill - but now ceph orch brought the new OSD into the system - and we saw excactly the same problem again. Being a bit wiser - we forcefully restarted the new OSD.. and everything whet back into normal mode again. Thus it seems like the "reweight" command on online OSD's have a bad effect on our setup - causing major service disruption. 1) Is it possible to "bulk" change default weights on all OSD's without a huge data movement going on? 2) or Is it possible to instrurct "ceph orch osd add" to set default weight before it putting the new OSD into the system? I would not expect above to be expected behaviour - if someone has ideas about what goes on more than above please share? Setup: # ceph version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 43 7.68 TB NVMe's over 12 OSD hosts - all connected using 2x 100GbitE Thanks Jesper

1 0

Connect to Ceph Cluster from other network
by Ramin Najjarbashi 10 Mar '24

10 Mar '24

Hello Ceph community, I am currently facing an issue connecting to my Ceph cluster from an external system and would appreciate your guidance. Here are some details about my setup: - Ceph Public Network: 172.0.0.1/24 - Ceph Private Network: 172.10.0.1/24 - Monitor (MON) Nodes: - mon1: 172.0.3.1 - mon2: 172.0.3.2 - mon3: 172.0.3.3 My network node has an internet IP (1.1.1.1), and I'm using HAProxy with the following configuration: ```bash backend admin_mons_v2 mode tcp server mon1 172.0.3.1:3300 check server mon2 172.0.3.2:3300 check server mon3 172.0.3.3:3300 check backend admin_mons_v1 mode tcp server mon1 172.0.3.1:6789 check server mon2 172.0.3.2:6789 check server mon3 172.0.3.3:6789 check frontend mon_v1 bind *:6789 mode tcp use_backend admin_mons_v1 frontend mon_v2 bind *:3300 mode tcp use_backend admin_mons_v2 ``` I've created a new authentication for my local system (IP: 2.2.2.2) using the following command: ```bash ceph auth get-or-create client.gateway mon 'allow * network 0.0.0.0/0' ``` And I've created a configuration file on my local system as follows: ```ini [global] fsid = MY_CLUSTER_FSID mon host = [v1:1.1.1.1:3300,v2:1.1.1.1:6789] ``` However, when attempting to connect to the Ceph cluster using the `ceph` or `radosgw-admin` commands, I encounter a connection timeout issue: ```bash 2024-03-10T20:00:10.551+0000 ffff7a214d38 0 monclient(hunting): authenticate timed out after 300 ``` Interestingly, I can connect to ports 3300 and 6789 using `telnet` from my local system: ```bash telnet 1.1.1.1 3300 Trying 1.1.1.1... Connected to 1.1.1.1. Escape character is '^]'. ceph v2 telnet 1.1.1.1 6789 Trying 1.1.1.1... Connected to 1.1.1.1. Escape character is '^]'. ceph v027?? ``` I would appreciate any insights or guidance on how to successfully connect to my Ceph cluster from my local system. Thank you for your time and assistance! Best regards,

1 0

PG damaged "failed_repair"
by Romain Lebbadi-Breteau 10 Mar '24

10 Mar '24

Hi, We're a student club from Montréal where we host an Openstack cloud with a Ceph backend for storage of virtual machines and volumes using rbd. Two weeks ago we received an email from our ceph cluster saying that some pages were damaged. We ran "sudo ceph pg repair <pg-id>" but then there was an I/O error on the disk during the recovery ("An unrecoverable disk media error occurred on Disk 4 in Backplane 1 of Integrated RAID Controller 1." and "Bad block medium error is detected at block 0x1377e2ad on Virtual Disk 3 on Integrated RAID Controller 1." messages on iDRAC). After that, the PG we tried to repair was in the state "active+recovery_unfound+degraded". After a week, we ran the command "sudo ceph pg 2.1b mark_unfound_lost revert" to try to recover the damaged PG. We tried to boot the virtual machine that had crashed because of this incident, but the volume seemed to have been completely erased, the "mount" command said there was no filesystem on it, so we recreated the VM from a backup. A few days later, the same PG was once again damaged, and since we knew the physical disk on the OSD hosting one part of the PG had problems, we tried to "out" the OSD from the cluster. That resulted in the two other OSDs hosting copies of the problematic PG to go down, which caused timeouts on our virtual machines, so we put the OSD back in. We then tried to repair the PG again, but that failed and the PG is now "active+clean+inconsistent+failed_repair", and whenever it goes down, two other OSDs from two other hosts go down too after a few minutes, so it's impossible to replace the disk right now, even if we have new ones available. We have backups for most of our services, but it would be very disrupting to delete the whole cluster, and we don't know that to do with the broken PG and the OSD that can't be shut down. Any help would be really appreciated, we're not experts with Ceph and Openstack, and it's likely we handled things wrong at some point, but we really want to go back to a healthy Ceph. Here are some information about our cluster : romain:step@alpha-cen ~ $ sudo ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 1 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 2.1b is active+clean+inconsistent+failed_repair, acting [3,11,0] romain:step@alpha-cen ~ $ sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 70.94226 root default -7 20.00792 host alpha-cen 3 hdd 1.81879 osd.3 up 1.00000 1.00000 6 hdd 1.81879 osd.6 up 1.00000 1.00000 12 hdd 1.81879 osd.12 up 1.00000 1.00000 13 hdd 1.81879 osd.13 up 1.00000 1.00000 15 hdd 1.81879 osd.15 up 1.00000 1.00000 16 hdd 9.09520 osd.16 up 1.00000 1.00000 17 hdd 1.81879 osd.17 up 1.00000 1.00000 -5 23.64874 host beta-cen 1 hdd 5.45749 osd.1 up 1.00000 1.00000 4 hdd 5.45749 osd.4 up 1.00000 1.00000 8 hdd 5.45749 osd.8 up 1.00000 1.00000 11 hdd 5.45749 osd.11 up 1.00000 1.00000 14 hdd 1.81879 osd.14 up 1.00000 1.00000 -3 27.28560 host gamma-cen 0 hdd 9.09520 osd.0 up 1.00000 1.00000 5 hdd 9.09520 osd.5 up 1.00000 1.00000 9 hdd 9.09520 osd.9 up 1.00000 1.00000 romain:step@alpha-cen ~ $ sudo rados list-inconsistent-obj 2.1b {"epoch":9787,"inconsistents":[]} romain:step@alpha-cen ~ $ sudo ceph pg 2.1b query https://pastebin.com/gsKCPCjr Best regards, Romain Lebbadi-Breteau

3 6

PGs increasing number
by Michel Niyoyita 09 Mar '24

09 Mar '24

Hello team, I have increased my volumes pool which was 128 PGs to 256 PGs , the activity started yesterday 5PM , It started when it was 5.733% of misplaced object , after 4 to 5 hours it reaches to 5.022 % after that it come back to the initial percentage 5.733% , kindly help to solve the issue. I am using Ceph Pacific deployed using ceph-ansible with ubuntu OS. the cluster is in production with openstack hypervisor. Best regards Michel

3 4