- ceph-users - lists.ceph.io

Stuck OSD down/out + workaround
by Mazzystr 10 Jun '24

10 Jun '24

Hi ceph users, I've seen this happen a couple times and been meaning to ask the group about it. Sometimes I get a failed block device and I have to replace it. My normal process is - * stop the osd process * remove the osd from crush map * rm -rf /var/lib/ceph/osd/<cluster>-<osd>/* * run mkfs * start osd process * add osd to crush map Sometimes the osd will get "stuck" and it doesn't want to flip to up/in. I have to actually rm the osd.id from ceph itself then recreate the id then do all the above. Does anyone know why this could be? I'm on reef 18.2 but I've seen this a lot over the years. Thanks, /Chris

1 0

Question regarding bluestore labels
by Bailey Allison 10 Jun '24

10 Jun '24

I have a question regarding bluestore labels, specifically for a block.db partition. To make a long story short, we are currently in a position where checking the label of a block.db partition and it appears corrupted. I have seen another thread on here suggesting to copy the label from a working OSD to the non working OSD, then re-adding the correct value to the labels with ceph-bluestore-tool. Where this was mentioned this was with an OSD in mind, would the same logic apply if we were working with a db device instead? This is assuming the only issue with the db is the corrupted label, and there is no other issues. Regards, Bailey

2 2

tuning for backup target cluster
by Lukasz Borek 10 Jun '24

10 Jun '24

Hi Everyone, I'm putting together a HDD cluster with an ECC pool dedicated to the backup environment. Traffic via s3. Version 18.2, 7 OSD nodes, 12 * 12TB HDD + 1NVME each, 4+2 ECC pool. Wondering if there is some general guidance for startup setup/tuning in regards to s3 object size. Files are read from fast storage (SSD/NVME) and written to s3. Files sizes are 10MB-1TB, so it's not standard s3. traffic. Backup for big files took hours to complete. My first shot would be to increase default bluestore_min_alloc_size_hdd, to reduce the number of stored objects, but I'm not sure if it's a good direccion? Any other parameters worth checking to support such a traffic pattern? Thanks! -- Łukasz

4 13

Are we logging IRC channels?
by Alvaro Soto 08 Jun '24

08 Jun '24

Hi Community!!! Are we logging IRC channels? I ask this because a lot of people only use Slack, and the Slack we use doesn't have a subscription, so messages are lost after 90 days (I believe) I believe it's important to keep track of the technical knowledge we see each day over IRC+Slack Cheers! -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

5 9

Excessively Chatty Daemons RHCS v5
by Joshua Arulsamy 07 Jun '24

07 Jun '24

Hi, I recently upgraded my RHCS cluster from v4 to v5 and moved to containerized daemons (podman) along the way. I noticed that there are a huge number of logs going to journald on each of my hosts. I am unsure why there are so many. I tried changing the logging level at runtime with commands like these (from the ceph docs): ceph tell osd.\* config set debug_osd 0/5 I tried adjusting several different subsystems (also with 0/0) but I noticed that logs seem to come at the same rate/content. I'm not sure what to try next? Is there a way to trace where logs are coming from? Some of the sample log entries are events like this on the OSD nodes: Jun 04 10:34:02 pf-osd1 ceph-osd-0[182875]: 2024-06-04T10:34:02.470-0600 7fc049c03700 -1 osd.0 pg_epoch: 703151 pg[35.39s0( v 703141'789389 (701266'780746,703141'789389] local-lis/les=702935/702936 n=48162 ec=63726/27988 lis/c=702935/702935 les/c/f=702936/702936/0 sis=702935) [0,194,132,3,177,159,83,18,149,14,145]p0(0) r=0 lpr=702935 crt=703141'789389 lcod 703141'789388 mlcod 703141'789388 active+clean planned DEEP_SCRUB_ON_ERROR] scrubber <NotActive/>: handle_scrub_reserve_grant: received unsolicited reservation grant from osd 177(4) (0x55fdea6c4000) These are very verbose messages and occur roughly every 0.5 second per daemon. On a cluster with 200 daemons this is getting unmanageable and is flooding my syslog servers. Any advice on how to tame all the logs would be greatly appreciated! Best, Josh Joshua Arulsamy HPC Systems Architect Advanced Research Computing Center University of Wyoming jarulsam(a)uwyo.edu

2 1

[Ceph orch] Could not start rgw service with IPv6 network
by Huy Nguyen 06 Jun '24

06 Jun '24

Hi, In traditional way, I can make the radosgw listen on IPv6 like this: # ceph.conf [...] rgw_frontends = beast port=8080 endpoint=[2405:f980:0:2:f816:3eff:fed9:8d16] In Cephadm, It seems specifying IPv6 network does not work (IPv4 does), and the rgw services do not become active. # rgw.yaml service_type: rgw service_id: ceph-lab-1 placement: label: rgw count_per_host: 2 networks: - 2405:f980:0000:0002::/64 #- 10.3.52.0/22 spec: rgw_frontend_port: 8080 # ceph orch apply -i rgw.yaml # ceph orch ls [...] rgw.ceph-lab-1 ?:8080 0/2 - 38m count-per-host:2;label:rgw Does any one have any idea? Where is the log location? Thanks

2 2

Rebalance OSDs after adding disks?
by tpDev Tester 06 Jun '24

06 Jun '24

Hi, I have been curiuos about ceph for long time and now I started to experiment to find out, how it works. The idea I like most is, that ceph can provide growing storage without the need to move from storage x to storage y on consumer side. I started with a 3 node cluster where each node got one OSD (2x 500GB, 1x 1TB). I created the pool and the filesystem, mounted the filesystem and filled it with data. All three OSDs filled up evenly and when the pool reached 70%, I added two 1TB OSDs (each on the node with 500GB). Now I expected some activity to rebalance the fill level of the OSDs, but nothing special happened. The new OSDs got some data, but the 500GB OSDs ran into 95%, the pool reached 100%, everything got stuck with 2 OSDs filled up just 20% and left 80% free space? I searched the documentation how to initiate the rebalance/redistribution by hand but was unable to find anything. Can someone please point me to the docs how I can expand the capacity of the pool without such problems. Thanks in advance Thomas

2 2

CORS Problems
by mailing-lists 05 Jun '24

05 Jun '24

Dear Cephers, I am facing a Problem. I have updated our ceph cluster form 17.2.3 to 17.2.7 last week and i've just gotten complains about a website that is not able to use s3 via CORS anymore. (GET works, PUT does not). I am using cephadm and i have deployed 3 rgws + 2 ingress services. The bucket shows: aws s3api get-bucket-cors { "CORSRules": [ { "AllowedHeaders": [ "*" ], "AllowedMethods": [ "GET", "PUT" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [ "ETag", "Accept-Ranges", "Content-Encoding", "Content-Range" ] } ] } Which is supposed to be a working config (at least this was working until the update). Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https:/EXAMPLE.com[.....] (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 403. I can not 100% say that the update is the culprit, but I haven't touched anything else and before the update it was working. Did I miss an important change? Best inDane

2 3

Error EINVAL: check-host failed - Failed to add host
by isnraju26＠yahoo.com 05 Jun '24

05 Jun '24

Hello, Please help: CEPH cluster using docker. using below command for the bootstrap with provided key and pub cephadm -v bootstrap --mon-ip <ip> --allow-overwrite --ssh-private-key id_ed25519 --ssh-public-key id_ed25519.pub able to ssh directly with id_ed25519 key. RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=<hostname> -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp1sw6a5s0:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpethxlwxr:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add <hostname> <ip>: Error EINVAL: check-host failed: Unable to write <host>:/var/lib/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed also validated below and it was successful: > ceph cephadm get-ssh-config > ssh_config > ceph config-key get mgr/cephadm/ssh_identity_key > key > ssh -F ssh_config -i key root@<hostname> --- root@<hostname>:~/.ssh# ceph orch host add <hostname> <ip> Error EINVAL: check-host failed: Unable to write <hostname>:/var/lib/ceph/fe8ecd30-1da2-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed root@<hostname>:~/.ssh# Thanks, Surya

2 5

CephFS metadata pool size
by Lars Köppel 05 Jun '24

05 Jun '24

Hello everyone, we have a cluster with 72 HDDs with 16TB and 3 SSDs with 4TB each in 3 Nodes. The 3 SSDs are used to store the metadata for the CephFS filesystem. After the update to 18.2.2 the size of the metadata pool went from around 2 TiB to over 3.5 TiB filling up the OSDs. After a few days they started to come back down but they did it not at the same rate. So one was stuck at 90% while the other ones came down to 80% - still way too high. After a few more days the highest one hit the full ratio limit and stopped all writes to the filesystem. I zapped the drive and let it rebuild at the moment. From my estimations, it will end up at ~ 1.9 TiB (~50%) where it should be. The other 2 OSDs are still filling up slowly and will hit full ratio soon. It looks like there is a leak of data that is not properly cleaned up by the MDS or OSD. But it will only rebuild the data that are actually in use. Can someone help me to cleanup the other 2 OSDs without rebuilding them because this takes a week for each of them. Any help is welcome. Best Regards Lars Here is the current output of status: cluster: id: xxxx health: HEALTH_WARN 1 MDSs report oversized cache 1 clients failing to respond to cache pressure noscrub,nodeep-scrub flag(s) set 1 nearfull osd(s) Degraded data redundancy: 440823880/3637190859 objects degraded (12.120%), 121 pgs degraded, 121 pgs undersized 3 pool(s) nearfull 1 pools have too many placement groups services: mon: 3 daemons, quorum storage02,storage01,storage03 (age 2w) mgr: storage02.wcuuzg(active, since 2w), standbys: storage03.kevzol, storage01.kwcjsc mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 75 osds: 75 up (since 2d), 75 in (since 2d); 121 remapped pgs flags noscrub,nodeep-scrub data: volumes: 1/1 healthy pools: 4 pools, 2209 pgs objects: 1.21G objects, 285 TiB usage: 866 TiB used, 323 TiB / 1.2 PiB avail pgs: 440823880/3637190859 objects degraded (12.120%) 2088 active+clean 101 active+undersized+degraded+remapped+backfill_wait 20 active+undersized+degraded+remapped+backfilling io: client: 113 MiB/s rd, 58 MiB/s wr, 640 op/s rd, 1.11k op/s wr recovery: 0 B/s, 2.22k keys/s, 1.22k objects/s Output of ceph fs status: cephfs - 27 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.storage01.ieoyov Reqs: 441 /s 4885k 3576k 2247k 399k 0-s standby-replay cephfs.storage02.uonttf Evts: 419 /s 14.2M 7672k 5298k 0 POOL TYPE USED AVAIL metadata metadata 2460G 389G data data 854T 70.6T ssd-data data 28.5G 454G STANDBY MDS cephfs.storage03.pdgzsg MDS version: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel(a)ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai

1 0