February 2024 - ceph-users

Ceph Storage || Deploy/Install/Bootstrap a Ceph Cluster || Cephadm Orchestrator CLI method
by ankit＠eurus.net 09 Feb '24

09 Feb '24

Hi Guys, I am newbie and trying to install Ceph Storage cluster and following this https://docs.ceph.com/en/latest/cephadm/install/#cephadm-deploying-new-clus… ============================================================= OS - Ubuntu 22.04.3 LTS (Jammy Jellyfish) 4 node Cluster - mon1,mgr1,2 OSD nodes mon1 node can ssh all nodes via root to sudo ceph-user and ceph-user to ceph-user on other nodes basic requirements are done like podman, python3, systemd,ntp, lvm. =================================================================== cephadm bootstrap --mon-ip 192.168.2.125 - after running this i am getting following error. ceph-user@mon1:~$ sudo cephadm bootstrap --mon-ip 192.168.2.125 Creating directory /etc/ceph for ceph.conf Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chrony.service is enabled and running Repeating the final host check... podman (/usr/bin/podman) version 3.4.4 is present systemctl is present lvcreate is present Unit chrony.service is enabled and running Host looks OK Cluster fsid: 90813682-c656-11ee-9ca3-0800274ff361 Verifying IP 192.168.2.125 port 3300 ... Verifying IP 192.168.2.125 port 6789 ... Mon IP `192.168.2.125` is in CIDR network `192.168.2.0/24` Mon IP `192.168.2.125` is in CIDR network `192.168.2.0/24` Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Pulling container image quay.io/ceph/ceph:v17... Ceph version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting mon public_network to 192.168.2.0/24 Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 9283 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr not available, waiting (3/15)... mgr not available, waiting (4/15)... mgr not available, waiting (5/15)... mgr not available, waiting (6/15)... mgr not available, waiting (7/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Generating ssh key... Wrote public SSH key to /etc/ceph/ceph.pub Adding key to root@localhost authorized_keys... Adding host mon1... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Deploying ceph-exporter service with default placement... Non-zero exit code 22 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=mon1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/90813682-c656-11ee-9ca3-0800274ff361:/var/log/ceph:z -v /tmp/ceph-tmpnjonhex7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3gil6lbb:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch apply ceph-exporter /usr/bin/ceph: stderr Error EINVAL: Usage: /usr/bin/ceph: stderr ceph orch apply -i <yaml spec> [--dry-run] /usr/bin/ceph: stderr ceph orch apply <service_type> [--placement=<placement_string>] [--unmanaged] /usr/bin/ceph: stderr Traceback (most recent call last): File "/usr/sbin/cephadm", line 9653, in <module> main() File "/usr/sbin/cephadm", line 9641, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2205, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 5774, in command_bootstrap prepare_ssh(ctx, cli, wait_for_mgr_restart) File "/usr/sbin/cephadm", line 5275, in prepare_ssh cli(['orch', 'apply', t]) File "/usr/sbin/cephadm", line 5708, in cli return CephContainer( File "/usr/sbin/cephadm", line 4144, in run out, _, _ = call_throws(self.ctx, self.run_cmd(), File "/usr/sbin/cephadm", line 1853, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=mon1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/90813682-c656-11ee-9ca3-0800274ff361:/var/log/ceph:z -v /tmp/ceph-tmpnjonhex7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3gil6lbb:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch apply ceph-exporter What i am doing wrong or missing? Please help. Many Thanks AS

4 5

PSA: Long Standing Debian/Ubuntu build performance issue (fixed, backports in progress)
by Mark Nelson 09 Feb '24

09 Feb '24

Hi Folks, Recently we discovered a flaw in how the upstream Ubuntu and Debian builds of Ceph compile RocksDB. It causes a variety of performance issues including slower than expected write performance, 3X longer compaction times, and significantly higher than expected CPU utilization when RocksDB is heavily utilized. The issue has now been fixed in main. Igor Fedotov, however, observed during the performance meeting today that there were no backports for the fix in place. He also rightly pointed out that it would be helpful to make an announcement about the issue given the severity for the affected users. I wanted to give a bit more background and make sure people are aware and understand what's going on. 1) Who's affected? Anyone running an upstream Ubuntu/Debian build of Ceph from the last several years. External builds from Canonical and Gentoo suffered from this issue as well, but were fixed independently. 2) How can you check? There's no easy way to tell at the moment. We are investigating if running "strings" on the OSD executable may provide a clue. For now, assume that if you are using our Debian/Ubuntu builds in a non-container configuration you are affected. Proxmox for instance was affected prior to adopting the fix. 3) Are Cephadm deployments affected? Not as far as we know. Ceph container builds are compiled slightly differently from stand-alone Debian builds. They do not appear to suffer from the bug. 4) What versions of Ceph will get the fix? Casey Bodley kindly offered to backport the fix to both Reef and Quincy. He also verified that the fix builds properly with Pacific. We now have 3 separate backport PRs for the releases here: https://github.com/ceph/ceph/pull/55500 https://github.com/ceph/ceph/pull/55501 https://github.com/ceph/ceph/pull/55502 Please feel free to reply if you have any questions! Thanks, Mark -- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson(a)clyso.com We are hiring: https://www.clyso.com/jobs/

4 4

How to solve data fixity
by Michal Strnad 09 Feb '24

09 Feb '24

Hi all! In the context of a repository-type project, we need to address a situation where we cannot use periodic checks in Ceph (scrubbing) due to the project's nature. Instead, we need the ability to write a checksum into the metadata of the uploaded file via API. In this context, we are not concerned about individual file parts, but rather the file as a whole. Users will calculate the checksum and write it. Based on this hash, we should be able to trigger a check of the given files. We are aware that tools like s3cmd can write MD5 hashes to file metadata, but is there a more general approach? Does anyone have experience with this, or can you suggest a tool that can accomplish this? Thx Michal

4 5

RGW Index pool(separated SSD) tuning factor
by Jaemin Joo 09 Feb '24

09 Feb '24

Hi everyone, I confirmed that write performance has increased too much even if I apply just SSD for the index pool of rgw. I know that ~200 Bytes per object in the index pool is created. When I checked the index pool size, it's around 300 bytes ~ 400 bytes calculated. like me, If it uses index pool applying separated SSD devices, I guess there is the tuning factor like block size to reduce index pool size. Is there the tuning factor or recommendation about separated SSD for the index pool?

1 0

Does it impact write performance when SSD applies into block.wal (not block.db)
by Jaemin Joo 09 Feb '24

09 Feb '24

Hi everyone, I saw the bluestore can separate block.db, block.wal. In my case, I'd like to apply hybrid device which uses SSD, HDD to improve the small data write performance. but I don't have enough SSD to cover block.db and block.wal. so I think it can impact performance even though SSD applies into just block.wal. I just know that block.wal depends on rocksdb cache size as parameters. SSD might not need too much. 1. When I use SSD just into block.wal, Does it impact the write performance of the small data? 2. Should I make lv of ssd per osd for block.wal? 3. How much SSD do I need for block.wal relative to HDD(if I have 100TB)?

3 2

Adding a new monitor fails
by Tim Holloway 08 Feb '24

08 Feb '24

I just jacked in a completely new, clean server and I've been trying to get a Ceph (Pacific) monitor running on it. The "ceph orch daemon add" appears to install all/most of what's necessary, but when the monitor starts, it shuts down immediately, and in the manner of Ceph containers immediately erases itself and the container log, so it's not possible to see what its problem is. I looked at manual installation, but the docs appear to be oriented towards old-style non-container implementation and don't account for the newer /var/lib/ceph/*fsid*/ approach. Any tips? Last few lines in the system journal are like this: Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.938+0000 7f26810ae700 4 rocksdb: (Original Log Time 2024/02/06-16:09:58.938432) [compaction/compaction_job.cc:760] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 2] max score 0.00, MB/sec: 351.7 rd, 351.7 wr, level 6, files in(4, 0) out(2) MB in(92.8, 0.0) out(92.8), read-write-amplify(2.0) write- amplify(1.0) OK, records in: 2858, records dropped: 0 output_compression: NoCompression Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.938+0000 7f26810ae700 4 rocksdb: (Original Log Time 2024/02/06-16:09:58.938452) EVENT_LOG_v1 {"time_micros": 1707235798938446, "job": 6, "event": "compaction_finished", "compaction_time_micros": 276718, "compaction_time_cpu_micros": 73663, "output_level": 6, "num_output_files": 2, "total_output_size": 97309398, "num_input_records": 2858, "num_output_records": 2858, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 2]} Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.940+0000 7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798941291, "job": 6, "event": "table_file_deletion", "file_number": 14} Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.943+0000 7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798943980, "job": 6, "event": "table_file_deletion", "file_number": 12} Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.946+0000 7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798946734, "job": 6, "event": "table_file_deletion", "file_number": 10} Feb 06 11:09:58 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:58.946+0000 7f26810ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707235798946789, "job": 6, "event": "table_file_deletion", "file_number": 4} Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+0000 7f26818af700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+0000 7f26818af700 -1 mon.dell02@-1(synchronizing) e161 *** Got Signal Terminated *** Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.450+0000 7f26818af700 1 mon.dell02@-1(synchronizing) e161 shutdown Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.452+0000 7f2691a95880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work Feb 06 11:09:59 dell02.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-dell02[1357545]: debug 2024-02-06T16:09:59.452+0000 7f2691a95880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete Feb 06 11:09:59 dell02.mousetech.com bash[1357898]: ceph-278fcd86-0861- 11ee-a7df-9c5c8e86cf8f-mon-dell02

2 6

Performance issues with writing files to Ceph via S3 API
by Renann Prado 08 Feb '24

08 Feb '24

Hello, I have an issue at my company where we have an underperforming Ceph instance. The issue that we have is that sometimes writing files to Ceph via S3 API (our only option) takes up to 40s, which is too long for us. We are a bit limited on what we can do to investigate why it's performing so badly, because we have a service provider in between, so getting to the bottom of this really is not that easy. That being said, the way we use the S3 APi (again, Ceph under the hood) is by writing all files (multiple millions) to the root, so we don't use *no* folder-like structure e.g. we write */<uuid>* instead of */this/that/<uuid>* . The question is: Does anybody know whether Ceph has performance gains when you create a folder structure vs when you don't? Looking at Ceph's documentation I could not find such information. Best regards, *Renann Prado*

2 3

ceph error connecting to the cluster
by arimbidhea3＠gmail.com 08 Feb '24

08 Feb '24

hello, i was tried to create osd but when i run ceph command the output is like this : root@pod-deyyaa-ceph1:~# sudo ceph -s 2024-02-02T16:01:23.627+0700 7fc762f37640 0 monclient(hunting): authenticate timed out after 300 [errno 110] RADOS timed out (error connecting to the cluster) can anyone give me a hint or help me to solve this

2 1

ceph orch upgrade to 18.2.1 seems stuck on MDS?
by Nigel Williams 08 Feb '24

08 Feb '24

kicked off ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.1 and just MDS left to do but upgrade has been sitting for hours on this root@rdx-00:~# ceph orch upgrade status { "target_image": " quay.io/ceph/ceph@sha256:a4e86c750cc11a8c93453ef5682acfa543e3ca08410efefa30f520b54f41831f ", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "osd", "mgr", "mon", "crash" ], "progress": "1267/1306 daemons upgraded", "message": "Currently upgrading osd daemons", "is_paused": false } root@rdx-00:~# message seems erroneous as all the OSDs seem to be done.

1 1

Snapshot automation/scheduling for rbd?
by Jeremy Hansen 08 Feb '24

08 Feb '24

Can rbd image snapshotting be scheduled like CephFS snapshots? Maybe I missed it in the documentation but it looked like scheduling snapshots wasn’t a feature for block images. I’m still running Pacific. We’re trying to devise a sufficient backup plan for Cloudstack and other things residing in Ceph. Thanks. -jeremy

3 6

Help: Balancing Ceph OSDs with different capacity
by Jasper Tan 08 Feb '24

08 Feb '24

Hi I have recently onboarded new OSDs into my Ceph Cluster. Previously, I had 44 OSDs of 1.7TiB each and was using it for about a year. About 1 year ago, we onboarded an additional 20 OSDs of 14TiB each. However I observed that many of the data were still being written onto the original 1.7TiB OSDs instead of the 14TiB ones. Overtime, this caused a bottleneck as the 1.7TiB OSDs reached nearfull capacity. I have tried to perform a reweight (both crush reweight and reweight) to reduce the number of PGs on each 1.7TiB. This worked temporarily but resulted in many objects being misplaced and PGs being in a Warning state. Subsequently I have also tried using crush-compat balancer mode instead of upmap but did not see significant improvement. The latest changes I made was to change backfill-threshold to 0.85, hoping that PGs will no longer be assigned to OSDs that are >85% utilization. However, this did not change the situation much as I see many OSDs above >85% utilization today. Attached is a report from ceph report command. For now I have OUT-ed two of my OSDs which have reached 95% capacity. I would greatly appreciate it if someone can provide advice on this matter. Thanks Jasper Tan -- -- -- *The contents of this e-mail message and any attachments are confidential and are intended solely for addressee. The information may also be legally privileged. This transmission is sent in trust, for the sole purpose of delivery to the intended recipient. If you have received this transmission in error, any use, reproduction or dissemination of this transmission is strictly prohibited. If you are not the intended recipient, please immediately NOTIFY the sender by reply e-mail or phone and DELETE this message and its attachments, if any.*

3 3

Direct ceph mount on desktops
by Tim Holloway 07 Feb '24

07 Feb '24

Back when I was battline Octopus, I had problems getting ganesha's NFS to work reliably. I resolved this by doing a direct (ceph) mount on my desktop machine instead of an NFS mount. I've since been plagued by ceph "laggy OSD" complaints that appear to be due to a non-responsive client and I'm suspecting that the client in question is the desktop machine when it's suspended while the ceph mount is in effect. So the question is: Should ceph native mounts be used on general client machines which may hibernate or otherwise go offline?

2 3

Ceph as rootfs?
by Jeremy Hansen 06 Feb '24

06 Feb '24

Is it possible to use Ceph as a root filesystem for a pxe booted host? Thanks

5 4

CompleteMultipartUpload takes a long time to finish
by Ondřej Kukla 06 Feb '24

06 Feb '24

Hello, For some time now I’m struggling with the time it takes to CompleteMultipartUpload on one of my rgw clusters. I have a customer with ~8M objects in one bucket uploading quite a large files. From 100GB to like 800GB. I’ve noticed when they are uploading ~200GB files that the requests started timeouting on a LB we have infront of the rgw. When I’ve started going through the logs I’ve noticed that the CompleteMultipartUpload request took like 700s to finish. Which seemed ok-ish, but the number seem quite large. However, when they started uploading 750GB files the time to complete the multipart upload ended around 2500s -> more than 40minutes which seems like a way to much. Do you have a similar experience? Is there anything we can do to improve this? How much time does the CompleteMultipartUpload takes on your clusters? The cluster is running on version 17.2.6. Regards, Ondrej

2 2

Re: CompleteMultipartUpload takes a long time to finish
by Ondřej Kukla 05 Feb '24

05 Feb '24

Hello Anthony, The replicated index pool has about 20TiB of free space and we are using Intel P5510 NVMe Enterprise SSDs so I guess the HW shouldn’t be the issue. Yes, I’m able to change the timeout on our LB, but I’m not sure if I want to set it to 40minutes+… Ondrej > On 5. 2. 2024, at 20:09, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > Do you have sufficient capacity in the non-ec pool? Is it on fast media? > > You should be able to increase the timeout on your LB. > >> On Feb 5, 2024, at 13:51, Ondřej Kukla <ondrej(a)kuuk.la> wrote: >> >> Hello, >> >> For some time now I’m struggling with the time it takes to CompleteMultipartUpload on one of my rgw clusters. >> >> I have a customer with ~8M objects in one bucket uploading quite a large files. From 100GB to like 800GB. >> >> >> I’ve noticed when they are uploading ~200GB files that the requests started timeouting on a LB we have infront of the rgw. >> >> When I’ve started going through the logs I’ve noticed that the CompleteMultipartUpload request took like 700s to finish. Which seemed ok-ish, but the number seem quite large. >> >> However, when they started uploading 750GB files the time to complete the multipart upload ended around 2500s -> more than 40minutes which seems like a way to much. >> >> >> Do you have a similar experience? Is there anything we can do to improve this? How much time does the CompleteMultipartUpload takes on your clusters? >> >> The cluster is running on version 17.2.6. >> >> Regards, >> >> Ondrej >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: RBD Image Returning 'Unknown Filesystem LVM2_member' On Mount - Help Please
by duluxoz 05 Feb '24

05 Feb '24

~~~ Hello, I think that /dev/rbd* devices are flitered "out" or not filter "in" by the fiter option in the devices section of /etc/lvm/lvm.conf. So pvscan (pvs, vgs and lvs) don't look at your device. ~~~ Hi Gilles, So the lvm filter from the lvm.conf file is set to the default of `filter = [ "a|.*|" ]`, so that's accept every block device, so no luck there :-( ~~~ For Ceph based LVM volumes, you would do this to import: Map every one of the RBDs to the host Include this in /etc/lvm/lvm.conf: types = [ "rbd", 1024 ] pvscan vgscan pvs vgs If you see the VG: vgimportclone -n <make a name for VG> /dev/rbd0 /dev/rbd1 ... --import Now you should be able to vgchange -a y <your VG> and see the LVs ~~~ Hi Alex, Did the above as you suggested - the rbd devices (3 of them, none of which were originally part of an lvm on the ceph servers - at least, not set up manually by me) still do not show up using pvscan, etc. So I still can't mount any of them (not without re-creating a fs, anyway, and thus losing the data I'm trying to read/import) - they all return the same error message (see original post). Anyone got any other ideas? <hopeful tone in voice> :-) Cheers Dulux-Oz

4 4

Throughput metrics missing iwhen updating Ceph Quincy to Reef
by Jose Vicente 05 Feb '24

05 Feb '24

4 8

How can I clone data from a faulty bluestore disk?
by Carl J Taylor 05 Feb '24

05 Feb '24

Hi, I have a small cluster with some faulty disks within it and I want to clone the data from the faulty disks onto new ones. The cluster is currently down and I am unable to do things like ceph-bluestore-fsck but ceph-bluestore-tool bluefs-export does appear to be working. Any help would be appreciated Many thanks Carl

5 5

RBD mirroring to an EC pool
by Jan Kasprzak 05 Feb '24

05 Feb '24

Hello, Ceph users, I would like to use my secondary Ceph cluster for backing up RBD OpenNebula volumes from my primary cluster using mirroring in image+snapshot mode. Because it is for backups only, not a cold-standby, I would like to use erasure coding on the secondary side to save a disk space. Is it supported at all? I tried to create a pool: secondary# ceph osd pool create one-mirror erasure k6m2 secondary# ceph osd pool set one-mirror allow_ec_overwrites true set pool 13 allow_ec_overwrites to true secondary# rbd mirror pool enable --site-name secondary one-mirror image 2024-02-02T11:00:34.123+0100 7f95070ad5c0 -1 librbd::api::Mirror: mode_set: failed to allocate mirroring uuid: (95) Operation not supported When I created a replicated pool instead, this step worked: secondary# ceph osd pool create one-mirror-repl replicated secondary# rbd mirror pool enable --site-name secondary one-mirror-repl image secondary# So, is RBD mirroring supported with erasure-coded pools at all? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

Problem starting radosgw-admin and rados hangs when .rgw.root is incomplete
by Carl J Taylor 05 Feb '24

05 Feb '24

Hi, Can anyone shed light on this please? I have had our cluster crashed and now managed to get everything back up and running, osds have nearly rebalanced but I am seeing issues with rgw. 2024-02-05T01:29:56.272+0000 7f7237e75f40 20 rados->read ofs=0 len=0 2024-02-05T01:29:56.276+0000 7f7237e75f40 20 rados_obj.operate() r=-2 bl.length=0 2024-02-05T01:29:56.276+0000 7f7237e75f40 20 realm 2024-02-05T01:29:56.276+0000 7f7237e75f40 20 rados->read ofs=0 len=0 2024-02-05T01:29:56.276+0000 7f7237e75f40 20 rados_obj.operate() r=-2 bl.length=0 2024-02-05T01:29:56.276+0000 7f7237e75f40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-02-05T01:29:56.276+0000 7f7237e75f40 20 rados->read ofs=0 len=0 2024-02-05T01:29:56.276+0000 7f7237e75f40 20 rados_obj.operate() r=-2 bl.length=0 2024-02-05T01:29:56.276+0000 7f7237e75f40 20 rados->read ofs=0 len=0 2024-02-05T01:29:56.276+0000 7f7237e75f40 20 rados_obj.operate() r=0 bl.length=17 2024-02-05T01:29:56.276+0000 7f7237e75f40 20 rados->read ofs=0 len=0 The .rgw.root and .rgw.index are both marked incomplete and one pg for each was restored from a bad disk. Both pools are now showing a status of peering_blocked_by_history_les_bound. I do have some other pgs with important data that can be recovered from the disks but it is not essential that is done straight away. I need to get RGW running so I can delete old data and free up some space to allow backfiling to complete. Version is 18.2.1 running under cephadm data: pools: 19 pools, 801 pgs objects: 9.23M objects, 4.7 TiB usage: 9.8 TiB used, 2.7 TiB / 12 TiB avail pgs: 2.122% pgs not active 435947/18456424 objects degraded (2.362%) 559225/18456424 objects misplaced (3.030%) 758 active+clean 17 incomplete 12 active+undersized+degraded+remapped+backfill_toofull 12 active+remapped+backfill_toofull 2 active+clean+scrubbing+deep If anyone can suggest a known way of recovering from this your advice would be appreciated. Kind regards Carl.

1 0

RADOSGW Multi-Site Sync Metrics
by Rhys Powell 04 Feb '24

04 Feb '24

Hi All, I am in the process of implementing multi-site RGW instance and have successfully set up a POC and confirmed the functionality. I am working on metrics and alerting for this service, and I am not seeing metrics available for the output shown by radosgw-admin sync status --rgw-realm=<<realm-name>> Sample output: [@cepha-cn02 ~]# radosgw-admin sync status --rgw-realm=<<realm-name>> realm a207b396-8d1b-408b-851e-10ad545861b7 (realm-name) zonegroup 77e8924b-05e3-4d86-b887-aedd7fe5306c (zonegroup-name) zone a26c27b2-d6ac-4eab-a4ce-1036ce2d37dc (zone-name) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 8c7d69db-85ae-45f4-b4ec-f712fad4af07 (zone-name) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source I'd like to measure, track, and alert on shard status during sync operations. Is there a way to expose these metrics? I'm struggling to find guidance or details. Thanks in advance Rhys Rhys Powell (He/Him) KORE<https://www.korewireless.com/> | Senior Systems Engineer (m) rpowell(a)korewireless.com<mailto:rpowell@korewireless.com> LinkedIn<https://www.linkedin.com/company/kore-wireless/> | Twitter<https://twitter.com/KORE_Wireless>| Instagram<https://www.instagram.com/kore_wireless/> Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.

2 1

RBD Image Returning 'Unknown Filesystem LVM2_member' On Mount - Help Please
by duluxoz 04 Feb '24

04 Feb '24

Hi All, All of this is using the latest version of RL and Ceph Reef I've got an existing RBD Image (with data on it - not "critical" as I've got a back up, but its rather large so I was hoping to avoid the restore scenario). The RBD Image used to be server out via an (Ceph) iSCSI Gateway, but we are now looking to use plain old kernal module. The RBD Image has been RBD Mapped to the client's /dev/rbd0 location. So now I'm trying a straight `mount /dev/rbd0 /mount/old_image/` as a test What I'm getting back is `mount: /mount/old_image/: unknown filesystem type 'LVM2_member'.` All my Google Foo is telling me that to solve this issue I need to reformat the image with a new file system - which would mean "losing" the data. So my question is: How can I get to this data using rbd kernal modules (the iSCSI Gateway is no longer available, so not an option), or am I stuck with the restore option? Or is there something I'm missing (which would not surprise me in the least)? :-) Thanks in advance (as always, you guys and gals are really, really helpful) Cheers Dulux-Oz

5 7

OSD read latency grows over time
by Roman Pashin 02 Feb '24

02 Feb '24

Hello Ceph users, we see strange issue on last recent Ceph installation v17.6.2. We store data on HDD pool, index pool is on SSD. Each OSD store its wal on NVME partition. Benchmarks didn't expose any issues with cluster, but since we placed production load on it we see constantly growing OSD latency time (osd_read_latency) on SSD disks (where Index pool located). Latency is constantly growing day-by-day, but disks are not utilized even for 50%. Interesting, that when we move Index pool from SSD to NVME disks (disk space allows it for now) - osd latency drops to zero and start increasing from the ground. Also we noticed, that any change of pg_num for index pool (from 256 to 128 for instance) also drops latency to zero. And it starts its growth again (https://postimg.cc/5YHk9bby) From client perspective it looks like one operation takes longer and longer each other day and operation time drops each time when we do some change on index pool. I've enabled debug_optracker 10/0 and it shows, that OSD spend most time in `queued_for_pg` state, but physical disk utilization is about 10-20%. Also per logs I see, that longest operation is Listbucket, but it is strange, that with less than 100'000 items in bucket list even with 'max_keys=1' takes 3-40 seconds. If it matters client is Apache Flink doing checkpoints via S3 protocol. Here is an example of operation with debug_optracking logs: 2023-12-29T16:24:28.873353+0300, event: throttled, op: osd_op(client.1227774 .0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:28.861549+0300, event: header_read, op: osd_op(client. 1227774.0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:28.873358+0300, event: all_read, op: osd_op(client.1227774 .0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:28.873359+0300, event: dispatched, op: osd_op(client. 1227774.0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:28.873389+0300, event: queued_for_pg, op: osd_op(client. 1227774.0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.077528+0300, event: reached_pg, op: osd_op(client. 1227774.0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.077561+0300, event: started, op: osd_op(client.1227774 .0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27. 9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.077714+0300, event: waiting for subops from 59,494, op: osd_op(client.1227774.0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27.9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.146157+0300, event: sub_op_commit_rec, op: osd_op(client.1227774.0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27.9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.146166+0300, event: op_commit, op: osd_op(client.1227774 .0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27. 9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.146191+0300, event: sub_op_commit_rec, op: osd_op(client.1227774.0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27.9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.146204+0300, event: commit_sent, op: osd_op(client. 1227774.0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27.9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.146216+0300, event: done, op: osd_op(client.1227774.0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27.9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) Does anybody faced with the same issue? I'd be very grateful for any idea because at this point I've got stuck with what to tune and where to look at. Cluster setup - replication 3x, 15 servers in 3 datacenters with datacenter as failure domain. 7x HDD (data), 2x SSD (index), 1x NVME (wal + OS). ceph config - https://pastebin.com/pCqxXhT3 OSD read latency graph - https://postimg.cc/5YHk9bby -- Thank you, Roman

9 17

Re: OSD read latency grows over time
by Cory Snyder 02 Feb '24

02 Feb '24

1024 PGs on NVMe. From: Anthony D'Atri <anthony.datri(a)gmail.com> Sent: Friday, February 2, 2024 2:37 PM To: Cory Snyder <csnyder(a)1111systems.com> Subject: Re: [ceph-users] OSD read latency grows over time Thanks. What type of media are your index OSDs? How many PGs? > On Feb 2, 2024, at 2: 32 PM, Cory Snyder <csnyder@ 1111systems. com> wrote: > > Yes, we changed osd_memory_target to 10 GB on just our index OSDs. These OSDs have over ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. Report Suspicious ZjQcmQRYFpfptBannerEnd Thanks. What type of media are your index OSDs? How many PGs? > On Feb 2, 2024, at 2:32 PM, Cory Snyder <csnyder(a)1111systems.com> wrote: > > Yes, we changed osd_memory_target to 10 GB on just our index OSDs. These OSDs have over 300 GB of lz4 compressed bucket index omap data. Here is a graph showing the latencies before/after that single change: > > https://urldefense.com/v3/__https://pasteboard.co/IMCUWa1t3Uau.png__;!!J0dt… > > Cory Snyder > > > From: Anthony D'Atri <anthony.datri(a)gmail.com> > Sent: Friday, February 2, 2024 2:15 PM > To: Cory Snyder <csnyder(a)1111systems.com> > Cc: ceph-users <ceph-users(a)ceph.io> > Subject: Re: [ceph-users] OSD read latency grows over time > > You adjusted osd_memory_target? Higher than the default 4GB? Another thing that we've found is that rocksdb can become quite slow if it doesn't have enough memory for internal caches. As our cluster usage has grown, we've needed to increase > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > Report Suspicious > > ZjQcmQRYFpfptBannerEnd > You adjusted osd_memory_target? Higher than the default 4GB? > > > > Another thing that we've found is that rocksdb can become quite slow if it doesn't have enough memory for internal caches. As our cluster usage has grown, we've needed to increase OSD memory in accordance with bucket index pool usage. One one cluster, we found that increasing OSD memory improved rocksdb latencies by over 10x. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Unable to mount ceph
by Albert Shih 02 Feb '24

02 Feb '24

Hi, A little basic question. I created a volume with ceph fs volume then a subvolume called «erasure» I can see that with root@cthulhu1:/etc/ceph# ceph fs subvolume info cephfs erasure { "atime": "2024-02-02 11:02:07", "bytes_pcent": "undefined", "bytes_quota": "infinite", "bytes_used": 0, "created_at": "2024-02-02 11:02:07", "ctime": "2024-02-02 14:12:30", "data_pool": "data_ec", "features": [ "snapshot-clone", "snapshot-autoprotect", "snapshot-retention" ], "gid": 0, "mode": 16877, "mon_addrs": [ "145.238.187.184:6789", "145.238.187.185:6789", "145.238.187.186:6789", "145.238.187.188:6789", "145.238.187.187:6789" ], "mtime": "2024-02-02 14:12:30", "path": "/volumes/_nogroup/erasure/998e3bdf-f92b-4508-99ed-69f03a7303e9", "pool_namespace": "", "state": "complete", "type": "subvolume", "uid": 0 } From the mon server I was able to mount the «partition» with mount -t ceph admin@fXXXXXXX-c0f2-11ee-9307-f7e3b9f03075.cephfs=/volumes/_nogroup/erasure/998e3bdf-f92b-4508-99ed-69f03a7303e9 /mnt but on my test client I'm unable to mount root@ceph-vo-m:/etc/ceph# mount -t ceph vo@fxxxxxxx-c0f2-11ee-9307-f7e3b9f03075.cephfs=/volumes/_nogroup/erasure/998e3bdf-f92b-4508-99ed-69f03a7303e9/ /vo --verbose parsing options: rw source mount path was not specified unable to parse mount source: -22 root@ceph-vo-m:/etc/ceph# So I copy the /etc/ceph/ceph.conf on my client Put the /etc/ceph/ceph.client.vo.keyring on my client No firewall between the client/cluster. Weird part is when I run a tcpdump on my client I didn't see any tcp activity. Anyway to debug this pb ? Thanks Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 02 févr. 2024 16:21:01 CET

1 1

XFS on top of RBD, overhead
by Ruben Vestergaard 02 Feb '24

02 Feb '24

Hi group, Today I conducted a small experiment to test an assumption of mine, namely that Ceph incurs a substantial network overhead when doing many small files. One RBD was created, and on top of that an XFS containing 1.6 M files, each with size 10 kiB: # rbd info libvirt/bobtest rbd image 'bobtest': size 20 GiB in 5120 objects order 22 (4 MiB objects) [...] # df -h /space Filesystem Size Used Avail Use% Mounted on /dev/rbd0 20G 20G 181M 100% /space # ls -lh /space |head total 19G -rw-r--r--. 1 root root 10K Feb 2 14:13 xaa -rw-r--r--. 1 root root 10K Feb 2 14:13 xab -rw-r--r--. 1 root root 10K Feb 2 14:13 xac -rw-r--r--. 1 root root 10K Feb 2 14:13 xad -rw-r--r--. 1 root root 10K Feb 2 14:13 xae -rw-r--r--. 1 root root 10K Feb 2 14:13 xaf -rw-r--r--. 1 root root 10K Feb 2 14:13 xag -rw-r--r--. 1 root root 10K Feb 2 14:13 xah -rw-r--r--. 1 root root 10K Feb 2 14:13 xai # ls /space |wc -l 1638400 All files contain pseudorandom (i.e. incompressible) junk. My assumption was, that as the backend RBD block size is 4 MiB, it would be necessary for the client machine to download at least that 4 MiB worth of data on any given request, even if the file in the XFS is only 10 kB. I.e. I cat(1) a small file, the RBD client grabs the relevant 4 MiB block from Ceph, from this the small amount of requested data is extracted and presented to userspace. That's not what I see, however. My testing procedure is as follows: I have a list of all the files on the RBD, order randomized, stored in root's home folder -- this to make sure that I can pick file names at random by going through the list from the top, and not causing network traffic by listing files directly in the target FS. I then reboot the node to ensure that all caches are empty and start an iftop(1) to monitor network usage. Mapping the RBD and mounting the XFS results in 5.29 MB worth of data read from the network. Reading one file at random from the XFS results in approx. 200 kB of network read. Reading 100 files at random results in approx. 3.83 MB of network read. Reading 1000 files at random results in approx. 36.2 MB of network read. Bottom line is that reading any 10 kiB of actual data results in approximately 37 kiB data being transferred over the network. Overhead, sure, but nowhere near what I expected, which was 4 MiB per block of data "hit" in the backend. Is the RBD client performing partial object reads? Is that even a thing? Cheers, Ruben Vestergaard

3 3

PG upmap corner cases that silently fail
by Andras Pataki 02 Feb '24

02 Feb '24

Hi cephers, I've been looking into better balancing our clusters with upmaps lately, and ran into upmap cases that behave in a less than ideal way. If there is any cycle in the upmaps like ceph osd pg-upmap-items <pgid> a b b a or ceph osd pg-upmap-items <pgid> a b b c c a the upmap validation passes, the upmap gets added to the osdmap, but then gets silently ignored. Obviously this is for EC pools - irrelevant for replicated pools where the order of OSDs is not significant. The relevant code OSDMap::_apply_upmap even has a comment about this: if (q != pg_upmap_items.end()) { // NOTE: this approach does not allow a bidirectional swap, // e.g., [[1,2],[2,1]] applied to [0,1,2] -> [0,2,1]. for (auto& r : q->second) { // make sure the replacement value doesn't already appear ... I'm trying to understand the reasons for this limitation: is it the case that this is just a matter of convenience of coding (OSDMap::_apply_upmap could do this correctly with a bit more careful approach), or is there some inherent limitation somewhere else that prevents these cases from working? I did notice that just updating crush weights (without using upmaps) produces similar changes to the UP set (swaps OSDs in EC pools sometimes), so the OSDs seem to be perfectly capable of doing backfills for osdmap changes that shuffle the order of OSDs in the UP set. Some insight/history here would be appreciated. Either way, the behavior of validation passing on an upmap and then the upmap getting silently ignored is not ideal. I do realize that all clients would have to agree on this code, since clients independently execute it to find the OSDs to access (so rolling out a change to this is challenging). Andras

1 0

Understanding subvolumes
by Matthew Melendy 02 Feb '24

02 Feb '24

In our department we're getting starting with Ceph 'reef', using Ceph FUSE client for our Ubuntu workstations. So far so good, except I can't quite figure out one aspect of subvolumes. When I do the commands: [root@ceph1 ~]# ceph fs subvolumegroup create cephfs csvg [root@ceph1 ~]# ceph fs subvolume create cephfs staff csvg --size 2000000000000 I get these results: - A subvolume group csvg is created on volume cephfs - A subvolume called staff is created in csvg subvolume (like /volumes/csvg/staff ) however there is no size limit set at this folder in the Ceph dashboard view - A folder with an random UUID name is created inside the subvolume staff ( like /volumes/csvg/staff/6a1b3de5-f6ab-4878-aea3-3c3c6f96ffcf ); this folder does have a size set on it of 2TB My questions are: - what's the purpose of this UUID, and is it a requirement? - which directory should be mounted for my clients, staff/ or staff/{UUID}, for the size limit to take effect? - is there any way to hide or disable this UUID for client mounts? (eg in /etc/fstab) [root@ceph1 ~]# ceph fs subvolume getpath cephfs staff csvg /volumes/csvg/staff/6a1b3de5-f6ab-4878-aea3-3c3c6f96ffcf [root@ceph1 ~]# ceph fs subvolume ls cephfs csvg [ { "name": "staff" } ] -- Sincerely, Matthew Melendy IT Services Specialist CS System Services Group FEC 3550, University of New Mexico

4 3

Ceph Dashboard failed to execute login
by Michel Niyoyita 02 Feb '24

02 Feb '24

Hello team, I failed to login to my ceph dashboard which is running pacific as version and deployed using ceph-ansible . I have set admin password using the following command : "ceph dashboard ac-user-set-password admin -i ceph-dash-pass" where ceph-dash-pass possesses the real password. I am getting the following output : "{"username": "admin", "password": "$2b$12$Ge/2cpg0ZGjRPnBC2YREP.E5oVyNvV4SC9HU4PMsWWMBtC9UvL7mG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1706866328, "enabled": false, "pwdExpirationDate": null, "pwdUpdateRequired": false}" Once I login to the dashboard , still i get the same error message. I am guessing it is because the above "enabled" field is set to false . Ho w to set that field to true ? or if there is other alternative to set it you can advise. Thank you

2 2

Cannot recreate monitor in upgrade from pacific to quincy (leveldb -> rocksdb)
by Mark Schouten 02 Feb '24

02 Feb '24

Hi, During an upgrade from pacific to quincy, we needed to recreate the mons because the mons were pretty old and still using leveldb. So step one was to destroy one of the mons. After that we recreated the monitor, and although it starts, it remains in state ‘probing’, as you can see below. No matter what I tried, it won’t come up. I’ve seen quite some messages that the MTU might be an issue, but that seems to be ok: root@proxmox03:/var/log/ceph# fping -b 1472 10.10.10.{1..3} -M 10.10.10.1 is alive 10.10.10.2 is alive 10.10.10.3 is alive Does anyone have an idea how to fix this? I’ve tried destroying and recreating the mon a few times now. Could it be that the leveldb mons only support mon.$id notation for the monitors? root@proxmox03:/var/log/ceph# ceph daemon mon.proxmox03 mon_status { "name": “proxmox03”, "rank": 2, "state": “probing”, "election_epoch": 0, "quorum": [], "features": { "required_con": “2449958197560098820”, "required_mon": [ “kraken”, “luminous”, “mimic”, "osdmap-prune”, “nautilus”, “octopus”, “pacific”, "elector-pinging” ], "quorum_con": “0”, "quorum_mon": [] }, "outside_quorum": [ “proxmox03” ], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 0, "fsid": "39b1e85c-7b47-4262-9f0a-47ae91042bac”, "modified": "2024-01-23T21:02:12.631320Z”, "created": "2017-03-15T14:54:55.743017Z”, "min_mon_release": 16, "min_mon_release_name": “pacific”, "election_strategy": 1, "disallowed_leaders: ": “”, "stretch_mode": false, "tiebreaker_mon": “”, "removed_ranks: ": “2”, "features": { "persistent": [ “kraken”, “luminous”, “mimic”, "osdmap-prune”, “nautilus”, “octopus”, “pacific”, "elector-pinging” ], "optional": [] }, "mons": [ { "rank": 0, "name": “0”, "public_addrs": { "addrvec": [ { "type": “v2”, "addr": "10.10.10.1:3300”, "nonce": 0 }, { "type": “v1”, "addr": "10.10.10.1:6789”, "nonce": 0 } ] }, "addr": "10.10.10.1:6789/0”, "public_addr": "10.10.10.1:6789/0”, "priority": 0, "weight": 0, "crush_location": “{}” }, { "rank": 1, "name": “1”, "public_addrs": { "addrvec": [ { "type": “v2”, "addr": "10.10.10.2:3300”, "nonce": 0 }, { "type": “v1”, "addr": "10.10.10.2:6789”, "nonce": 0 } ] }, "addr": "10.10.10.2:6789/0”, "public_addr": "10.10.10.2:6789/0”, "priority": 0, "weight": 0, "crush_location": “{}” }, { "rank": 2, "name": “proxmox03”, "public_addrs": { "addrvec": [ { "type": “v2”, "addr": "10.10.10.3:3300”, "nonce": 0 }, { "type": “v1”, "addr": "10.10.10.3:6789”, "nonce": 0 } ] }, "addr": "10.10.10.3:6789/0”, "public_addr": "10.10.10.3:6789/0”, "priority": 0, "weight": 0, "crush_location": “{}” } ] }, "feature_map": { "mon": [ { "features": “0x3f01cfbdfffdffff”, "release": “luminous”, "num": 1 } ] }, "stretch_mode": false } — Mark Schouten CTO, Tuxis B.V. +31 318 200208 / mark(a)tuxis.nl

2 6

6 pgs not deep-scrubbed in time
by Michel Niyoyita 01 Feb '24

01 Feb '24

Hello team, I have a cluster in production composed by 3 osds servers with 20 disks each deployed using ceph-ansibleand ubuntu OS , and the version is pacific . These days is in WARN state caused by pgs which are not deep-scrubbed in time . I tried to deep-scrubbed some pg manually but seems that the cluster can be slow, would like your assistance in order that my cluster can be in HEALTH_OK state as before without any interuption of service . The cluster is used as openstack backend storage. Best Regards Michel ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 6 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones) data: pools: 10 pools, 385 pgs objects: 5.97M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep io: client: 59 MiB/s rd, 860 MiB/s wr, 155 op/s rd, 665 op/s wr root@ceph-osd3:~# ceph health detail HEALTH_WARN 6 pgs not deep-scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 6 pgs not deep-scrubbed in time pg 6.78 not deep-scrubbed since 2024-01-11T16:07:54.875746+0200 pg 6.60 not deep-scrubbed since 2024-01-13T19:44:26.922000+0200 pg 6.5c not deep-scrubbed since 2024-01-13T09:07:24.780936+0200 pg 4.12 not deep-scrubbed since 2024-01-13T09:09:22.176240+0200 pg 10.d not deep-scrubbed since 2024-01-12T08:04:02.078062+0200 pg 5.f not deep-scrubbed since 2024-01-12T06:06:00.970665+0200

6 27

Re: Performance improvement suggestion
by quaglio＠bol.com.br 01 Feb '24

01 Feb '24

1 1

Syslog server log naming
by Torkil Svensgaard 01 Feb '24

01 Feb '24

We have ceph (currently 18.2.0) log to an rsyslog server with the following file name format: template (name="DynaFile" type="string" string="/tank/syslog/%fromhost-ip%/%hostname%/%programname%.log") Around May 25th this year something changed so instead of getting the usual program log names we are also getting a lot of diffent logs with weird names. Here's an ls excerpt: " ... -rw-------. 1 root root 517K Feb 1 05:42 hardcore_raman.log -rw-------. 1 root root 198K Feb 1 05:42 sleepy_moser.log -rw-------. 1 root root 203K Feb 1 05:42 friendly_gagarin.log -rw-------. 1 root root 1.1K Feb 1 05:42 goofy_hypatia.log -rw-------. 1 root root 164K Feb 1 05:42 kind_chebyshev.log -rw-------. 1 root root 11K Feb 1 05:42 magical_archimedes.log -rw-------. 1 root root 373K Feb 1 05:42 busy_bardeen.log -rw-------. 1 root root 178K Feb 1 05:42 trusting_euler.log -rw-------. 1 root root 526K Feb 1 05:42 inspiring_golick.log -rw-------. 1 root root 369K Feb 1 06:12 condescending_ganguly.log -rw-------. 1 root root 191K Feb 1 06:12 mystifying_torvalds.log -rw-------. 1 root root 475K Feb 1 06:12 charming_nash.log -rw-------. 1 root root 168K Feb 1 06:12 zealous_sinoussi.log -rw-------. 1 root root 325K Feb 1 06:12 amazing_booth.log -rw-------. 1 root root 516K Feb 1 06:12 great_ardinghelli.log -rw-------. 1 root root 313K Feb 1 06:12 magical_bell.log -rw-------. 1 root root 22K Feb 1 06:12 nifty_swartz.log -rw-------. 1 root root 426 Feb 1 06:12 upbeat_beaver.log -rw-------. 1 root root 166K Feb 1 06:13 funny_lederberg.log -rw-------. 1 root root 164K Feb 1 06:13 frosty_murdock.log -rw-------. 1 root root 374K Feb 1 06:13 elastic_banach.log -rw-------. 1 root root 308K Feb 1 06:13 inspiring_cohen.log -rw-------. 1 root root 176K Feb 1 06:13 angry_wu.log -rw-------. 1 root root 662 Feb 1 06:42 admiring_kalam.log -rw-------. 1 root root 3.1K Feb 1 06:43 thirsty_colden.log -rw-------. 1 root root 4.5M Feb 1 07:01 run-parts.log -rw-------. 1 root root 16M Feb 1 07:01 CROND.log -rw-------. 1 root root 109M Feb 1 07:06 python3.log -rw-------. 1 root root 3.4M Feb 1 07:29 systemd-journald.log -rw-------. 1 root root 596M Feb 1 07:34 sudo.log -rw-------. 1 root root 549 Feb 1 07:44 interesting_rosalind.log -rw-------. 1 root root 342K Feb 1 07:45 beautiful_hamilton.log -rw-------. 1 root root 348K Feb 1 07:45 cool_ride.log -rw-------. 1 root root 15G Feb 1 07:45 conmon.log -rw-------. 1 root root 395K Feb 1 07:45 compassionate_satoshi.log -rw-------. 1 root root 11K Feb 1 07:45 hardcore_noether.log -rw-------. 1 root root 223K Feb 1 07:45 wizardly_johnson.log -rw-------. 1 root root 270M Feb 1 07:49 sshd.log -rw-------. 1 root root 111M Feb 1 07:49 systemd-logind.log -rw-------. 1 root root 1.6G Feb 1 07:50 systemd.log -rw-------. 1 root root 119M Feb 1 07:54 rsyslogd.log -rw-------. 1 root root 94G Feb 1 07:55 ceph-8ee2d228-ed21-4580-8bbf-064.log -rw-------. 1 root root 1.1G Feb 1 07:56 podman.log -rw-------. 1 root root 1.8G Feb 1 07:58 ceph-mgr.log -rw-------. 1 root root 213G Feb 1 07:58 ceph-osd.log -rw-------. 1 root root 48G Feb 1 07:58 ceph-mon.log " Those are container names or something like that? The file content seems to be assorted bits from the ceph disk tool: " # cat goofy_hypatia.log Jun 7 04:12:43 dopey goofy_hypatia[3224681]: 167 167 Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> passed data devices: 22 physical, 0 LVM Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> relative data size: 1.0 Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> All data devices are unavailable Jun 24 09:00:08 dopey goofy_hypatia[2319188]: [] Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> passed data devices: 22 physical, 0 LVM Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> relative data size: 1.0 Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> All data devices are unavailable 2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> passed data devices: 2 physical, 0 LVM 2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> relative data size: 1.0 2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> All data devices are unavailable 2023-12-26T09:38:59+01:00 dopey goofy_hypatia[893177]: 167 167 2023-12-26T10:41:14+01:00 dopey goofy_hypatia[1057467]: 167 167 2024-01-05T21:00:37+01:00 dopey goofy_hypatia[3395314]: {} 2024-02-01T05:42:17+01:00 dopey goofy_hypatia[845150]: 167 167 " Anyone else had this issue? Suggestions on how to get a real program name instead? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 3

how can install latest dev release?
by garcetto 01 Feb '24

01 Feb '24

good morning, how can i install latest dev release using cephadm? thank you.

3 4

Merging two ceph clusters
by Nico Schottelius 01 Feb '24

01 Feb '24

Good morning, in the spirit of the previous thread, I am wondering if anyone ever succeeded in merging two separate ceph clusters into one? Background from my side: we are running multiple ceph clusters in k8s/rook, but we still have some Nautilus/Devuan based clusters that are hard-to-impossible to upgrade. One of the ideas was to shift over the existing workload of the older clusters into the newer clusters. In the best case the whole process would be possible while keeping everything running in the new and old cluster. One clear challenge is that the two clusters obviously have a different FSID. The pool (names) in the two clusters differ, but the pool IDs are overlapping. If anyone has ever done this or has ideas on how to accomplish this, I would be very interested in hearing your opinion. Greetings from the Swiss mountains, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Changing A Ceph Cluster's Front- And/Or Back-End Networks IP Address(es)
by duluxoz 01 Feb '24

01 Feb '24

Hi All, Quick Q: How easy/hard is it to change the IP networks of: 1) A Ceph Cluster's "Front-End" Network? 2) A Ceph Cluster's "Back-End" Network? Is it a "simply" matter of: a) Placing the Nodes in maintenance mode b) Changing a config file (I assume it's /etc/ceph/ceph.conf) on each Node c) Rebooting the Nodes d) Taking each Node out of Maintenance Mode Thanks in advance Cheers Dulux-Oz

2 1