- ceph-users - lists.ceph.io

orch adoption and disk encryption without cephx?
by Boris 18 Aug '24

18 Aug '24

Hi, I have some legacy clusters that I can not move to cephx due to customer workload. (We have a plan to move everything iterative to a new cluster, but that might still take a lot time). I would like to adopt the orchestrator and use the ceph disk encryption feature. Is this possible without using cephx? It is only rbd workload. I ask, because I remember that RGW actually required cephx after some update, because there might be certificates in the mon db. And it took me some time to find the problem and the solution back then. I would like to not encounter this with rbd workload :) Cheers Boris

1 0

Accidentally created systemd units for OSDs
by Dan O'Brien 17 Aug '24

17 Aug '24

I was [poorly] following the instructions for migrating the wal/db to an SSD https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-fo… and I didn't add the '--no-systemd' when I did 'ceph-volume lvm activate' command (3 f***ing times). The result is that I've "twinned" 3 of my OSDs: There's a container version managed by cephadm and there's an instantiated systemd unit that runs directly. Surprisingly, this has not done a lot of damage, but it does result in the dashboard reporting 3 failed cephadm daemons when the "native" OSDs start before the containerized ones. I've disabled the systemd units for ceph-osd(a)9.service, ceph-osd(a)11.service and ceph-osd(a)25.service, but I'd like to remove them completely. I will eventually badger The Google into giving me an answer, but could someone tell me what I need to do? The semester starts soon and I don't really have the bandwidth for this right now. Thanks in advance. I will forever be in your debt. (Seriously, I'm ready to give you a kidney, if you need it.)

4 8

Ceph Logging Configuration and "Large omap objects found"
by Janek Bevendorff 16 Aug '24

16 Aug '24

Hi, I have a bunch of long-standing struggles with the way Ceph handles logging and I cannot figured out how to solve them. These issues are basically the following: - The log config options are utterly confusing and very badly documented - Mon file logs are spammed with DBG-level cluster logs, no matter what I configure - syslog/journald gets only part of the messages, but file logs must be turned off due to the spam above - "audit" channel logs cannot be configured at all From this, my following needs and questions arise, perhaps you can help: - I want to receive cluster and audit logs on the MONs with level "warn" or at most "info" - I want everything to go to journald - Where to cluster logs go if "clog_to_monitors" is off? - What's the difference between the "mon_cluster_log_*" and "clog_*" settings? - What the hell does "mon_cluster_log_to_syslog_facility" do and what does "audit=local0" mean or do? A very annoying symptom of the wonky logging config is that I cannot debug the infamous "LARGE_OMAP_OBJECTS" warning. It says "Search the cluster log for 'Large omap object found' for more details.", but I cannot do that, because without enabling the file-logging flood gates, I never receive the required cluster log info at the monitors and there seems to be no other way to debug this than to grep the cluster log (why??). My current log config is the following: global advanced clog_to_monitors true global basic err_to_syslog true global basic log_to_file false global basic log_to_stderr false global basic log_to_syslog true mon advanced mon_cluster_log_file /dev/null mon advanced mon_cluster_log_to_file false mon advanced mon_cluster_log_to_stderr false mon advanced mon_cluster_log_to_syslog true mon advanced mon_cluster_log_to_syslog_facility daemon mon advanced mon_cluster_log_to_syslog_level warn Any help solving this conundrum is much appreciated. Thanks! Janek

2 10

Last Call for Cephalocon T-Shirt Contest
by Noah Lehman 16 Aug '24

16 Aug '24

Hi Ceph community, The deadline for the Cephalocon t-shirt design contest is tomorrow, Friday August 16th. You don’t need to be an artist or a graphics designer, all we ask is that it be original art! If you'd like to submit a proposal for the official t-shirt of Cephalocon 2024, you can do so here <https://ceph.io/en/news/blog/2024/cephalocon24-tshirt-contest/>. Best, Noah

1 0

ceph device ls missing disks
by Alfredo Rezinovsky 15 Aug '24

15 Aug '24

With: "ceph device ls" I can see which physical disks are used by which daemons. OSD data and wal disks are ok. MON appears using operating system disks, or the disks where /var/lib/ceph is mounted, also OK. The problem is that non monitor nodes operating system disks are missing when they are really being used by all the daemons in the node. The problem with this is that there are no monitoring for those disks when they are essential and failure points for all the daemons running in those nodes. I think the OS devices should appear used by all the daemons having essential information in those disks. -- Alfrenovsky

1 0

Cephadm Upgrade Issue
by Alex Hussein-Kershaw (HE/HIM) 15 Aug '24

15 Aug '24

Hi Folks, I'm prototyping the upgrade process for our Ceph Clusters. I've adopted the Cluster following the docs, that works nicely 🙂 I then load my docker image into a locally running container registry, as I'm in a disconnected environment. I have a test Cluster with 3 VMs and no data, adopted at Octopus and upgrading to Pacific. I'm running a MON, MGR, MDS and OSD on each VM. I then attempt to upgrade: ceph orch upgrade start --image localhost:5000/ceph/pacific:v16.2.15 Lots of logs below, but the summary appears to be that we initially fail to upgrade the managers and get into a bad state. It looks like there is some confusion in manager naming, and we end up with two managers on each machine instead of one. Eventually Ceph reports a health warning: $ ceph -s cluster: id: e773d9c2-6d8d-4413-8e8f-e38f248f5959 health: HEALTH_ERR 1 failed cephadm daemon(s) Module 'cephadm' has failed: 'cephadm' That does seem to eventually clean its self up and the upgrade appears to have completed ("ceph versions" shows everything on Pacific), but it feels a bit bumpy. Hoping someone has some guidance here. The containers on one host during upgrade are shown below. Notice I somehow have two managers, where the names are a single character different (a "-" replaced with a "."): $ docker ps | grep mgr 2143b6f0e0e6 localhost:5000/ceph/pacific:v16.2.15 "/usr/bin/ceph-mgr -…" About a minute ago Up About a minute ceph-e773d9c2-6d8d-4413-8e8f-e38f248f5959-mgr.raynor-sc-2 59c8cfddac64 ceph-daemon:v5.0.12-stable-5.0-octopus-centos-8 "/usr/bin/ceph-mgr -…" 14 minutes ago Up 14 minutes ceph-e773d9c2-6d8d-4413-8e8f-e38f248f5959-mgr-raynor-sc-2 In the output of "ceph -w" I see this sort of stuff: 2024-08-14T13:45:13.003405+0000 mon.raynor-sc-1 [INF] Manager daemon raynor-sc-3 is now available 2024-08-14T13:45:23.179699+0000 mon.raynor-sc-1 [ERR] Health check failed: Module 'cephadm' has failed: 'cephadm' (MGR_MODULE_ERROR) 2024-08-14T13:45:22.372376+0000 mgr.raynor-sc-3 [ERR] Unhandled exception from module 'cephadm' while running on mgr.raynor-sc-3: 'cephadm' 2024-08-14T13:45:24.761961+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:24.766395+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 2024-08-14T13:45:31.800989+0000 mon.raynor-sc-1 [INF] Manager daemon raynor-sc-3 is now available 2024-08-14T13:45:32.874227+0000 mon.raynor-sc-1 [INF] Health check cleared: MGR_MODULE_ERROR (was: Module 'cephadm' has failed: 'cephadm') 2024-08-14T13:45:32.874269+0000 mon.raynor-sc-1 [INF] Cluster is now healthy 2024-08-14T13:45:33.664602+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:33.671809+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 2024-08-14T13:45:34.050292+0000 mon.raynor-sc-1 [INF] Manager daemon raynor-sc-3 is now available 2024-08-14T13:45:38.260385+0000 mon.raynor-sc-1 [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-08-14T13:45:43.462665+0000 mgr.raynor-sc-3 [ERR] Unhandled exception from module 'cephadm' while running on mgr.raynor-sc-3: 'cephadm' 2024-08-14T13:45:44.770711+0000 mon.raynor-sc-1 [ERR] Health check failed: Module 'cephadm' has failed: 'cephadm' (MGR_MODULE_ERROR) 2024-08-14T13:45:45.668379+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:45.673206+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 2024-08-14T13:45:45.673316+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:45.689515+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:45.694315+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 2024-08-14T13:45:47.671192+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:47.674805+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 2024-08-14T13:45:47.675037+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:47.697264+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:47.700886+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 And in the output of "ceph -W cephadm" I see: 2024-08-14T13:40:32.214742+0000 mgr.raynor-sc-1 [INF] Upgrade: First pull of localhost:5000/ceph/pacific:v16.2.15 2024-08-14T13:40:34.108767+0000 mgr.raynor-sc-1 [INF] Upgrade: Target is localhost:5000/ceph/pacific:v16.2.15 with id 3c4eff6082ae7530e7eda038765ce400beb1bc1b8df67dffb45910eb45b06b2c 2024-08-14T13:40:34.112388+0000 mgr.raynor-sc-1 [INF] Upgrade: Checking mgr daemons... 2024-08-14T13:40:34.112722+0000 mgr.raynor-sc-1 [INF] Upgrade: Need to upgrade myself (mgr.raynor-sc-1) 2024-08-14T13:40:35.456432+0000 mgr.raynor-sc-1 [INF] It is presumed safe to stop ['mgr.raynor-sc-2'] 2024-08-14T13:40:35.456620+0000 mgr.raynor-sc-1 [INF] Upgrade: It is presumed safe to stop ['mgr.raynor-sc-2'] 2024-08-14T13:40:35.456771+0000 mgr.raynor-sc-1 [INF] Upgrade: Redeploying mgr.raynor-sc-2 2024-08-14T13:40:35.481790+0000 mgr.raynor-sc-1 [INF] Deploying daemon mgr.raynor-sc-2 on raynor-sc-2 2024-08-14T13:42:37.608895+0000 mgr.raynor-sc-1 [INF] refreshing raynor-sc-3 facts 2024-08-14T13:42:39.744098+0000 mgr.raynor-sc-1 [INF] refreshing raynor-sc-1 facts 2024-08-14T13:42:40.081740+0000 mgr.raynor-sc-1 [INF] refreshing raynor-sc-2 facts 2024-08-14T13:42:40.937375+0000 mgr.raynor-sc-1 [INF] Applying drive group all-available-devices on host raynor-sc-1... 2024-08-14T13:42:40.937732+0000 mgr.raynor-sc-1 [INF] Applying drive group all-available-devices on host raynor-sc-2... 2024-08-14T13:42:40.938079+0000 mgr.raynor-sc-1 [INF] Applying drive group all-available-devices on host raynor-sc-3... 2024-08-14T13:42:46.226231+0000 mgr.raynor-sc-1 [INF] Upgrade: Target is localhost:5000/ceph/pacific:v16.2.15 with id 3c4eff6082ae7530e7eda038765ce400beb1bc1b8df67dffb45910eb45b06b2c 2024-08-14T13:42:46.229604+0000 mgr.raynor-sc-1 [INF] Upgrade: Checking mgr daemons... 2024-08-14T13:42:46.229727+0000 mgr.raynor-sc-1 [INF] Upgrade: Need to upgrade myself (mgr.raynor-sc-1) 2024-08-14T13:42:47.696110+0000 mgr.raynor-sc-1 [INF] It is presumed safe to stop ['mgr.raynor-sc-3'] My take on this is that we first start to attempt upgrade of the mgr on raynor-sc-2, and don't seem to detect it's not quite worked. Interestingly it seems that there is a two minute gap between the deploying line and the following line, I wonder if something is failing to come up and we're proceeding after a timer expires? Any pointers are much appreciated. Many thanks, Alex

3 5

Bug with Cephadm module osd service preventing orchestrator start
by benjaminmhuth＠gmail.com 14 Aug '24

14 Aug '24

Hey there, so I went to upgrade my ceph from 18.2.2 to 18.2.4 and have encountered a problem with my managers. After they had been upgraded, my ceph orch module broke because the cephadm module would not load. This obviously halted the update because you can't really update without the orchestrator. Here are the logs related to why the cephadm module fails to start: https://pastebin.com/SzHbEDVA and the relevent part here: "backtrace": [ " File \"/usr/share/ceph/mgr/cephadm/module.py\", line 591, in __init__\n self.to_remove_osds.load_from_store()", " File \"/usr/share/ceph/mgr/cephadm/services/osd.py\", line 918, in load_from_store\n osd_obj = OSD.from_json(osd, rm_util=self.rm_util)", " File \"/usr/share/ceph/mgr/cephadm/services/osd.py\", line 783, in from_json\n return cls(**inp)", "TypeError: __init__() got an unexpected keyword argument 'original_weight'" ] Unfortunately, I am at a loss to what passes this the original weight argument. I have attempted to migrate back to 18.2.2 and successfully redeployed a manager of that version, but it also has the same issue with the cephadm module. I believe this may be because I recently started several OSD drains, then canceled them, causing this to manifest once the managers restarted. I've attempted to dig into this myself a bit and found the associated function in the ceph repo: https://github.com/ceph/ceph/blob/e0dd396793b679922e487332a2a4bc48e024a42f/… My issue may also have to do with this PR: https://github.com/ceph/ceph/commit/ba7fac074fb5ad072fcad10862f75c0a26a7591d I unfortunately do not have enough familiarity with the source to see where these values are being set, or to know what keys to remove from the ceph config. Hopefully someone who knows about this will see this and take a look for me! Thanks!

1 0

The snaptrim queue of PGs has not decreased for several days.
by Giovanna Ratini 14 Aug '24

14 Aug '24

Hello all, We use Ceph (v18.2.2) and Rook (1.14.3) as the CSI for a Kubernetes environment. Last week, we had a problem with the MDS falling behind on trimming every 4-5 days (GitHub issue link <https://github.com/rook/rook/issues/14220>). We resolved the issue using the steps outlined in the GitHub issue. We have 3 hosts (I know, I need to increase this as soon as possible, and I will!) and 6 OSDs. After running the commands: |ceph config set mds mds_dir_max_commit_size 80|, | | |ceph fs fail <fs_name>|, and | | |ceph fs set <fs_name> joinable true|, After that, the snaptrim queue for our PGs has stopped decreasing. All PGs of our CephFS are in either |active+clean+snaptrim_wait| or |active+clean+snaptrim| states. For example, the PG |3.12| is in the |active+clean+snaptrim| state, and its |snap_trimq_len| was 4077 yesterday but has increased to 4538 today. I increased the |osd_snap_trim_priority| to 10 (|ceph config set osd osd_snap_trim_priority 10|), but it didn't help. Only the PGs of our CephFS have this problem. Do you have any ideas on how we can resolve this issue? Thanks in advance, Giovanna

1 0

rbd du USED greater than PROVISIONED
by Murilo Morais 14 Aug '24

14 Aug '24

Good morning everyone! I am confused about the listing of the total amount used on a volume. It says that more than the amount provisioned is being used. The image contains a snapshot. Below is the output of the "rbd du" command: user@abc2:~# rbd info osr1_volume_ssd/volume-6e5f90ac-78e9-465e-8705-a4d476ebc019 rbd image 'volume-6e5f90ac-78e9-465e-8705-a4d476ebc019': size 1000 GiB in 256000 objects order 22 (4 MiB objects) snapshot_count: 1 id: 3ec2fef4cd0ad8 block_name_prefix: rbd_data.3ec2fef4cd0ad8 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Fri Jan 5 22:48:45 2024 access_timestamp: Wed Aug 14 10:46:06 2024 modify_timestamp: Wed Aug 14 10:47:02 2024 user@abc2:~# rbd du osr1_volume_ssd/volume-6e5f90ac-78e9-465e-8705-a4d476ebc019 NAME PROVISIONED USED volume-6e5f90ac-78e9-465e-8705-a4d476ebc019@snapshot-23af41b0-ca7b-45df-991f-ffbe95512dbe 1000 GiB 710 GiB volume-6e5f90ac-78e9-465e-8705-a4d476ebc019 1000 GiB 404 GiB <TOTAL> 1000 GiB 1.1 TiB In this case, is that really the case or is it some kind of BUG? For the record, we are still on version 17.2.3.

2 2

Cephadm and the "--data-dir" Argument
by Alex Hussein-Kershaw (HE/HIM) 14 Aug '24

14 Aug '24

Hi Folks, I'm trying to use the --data-dir argument of cephadm when bootstrapping a Storage Cluster. It looks like exactly what I need, where my use case is that I want to data files onto a persistent disk, such that I can below away my VMs while retaining the files. Everything looks good and the bootstrap command completes. For reference I am running this command: "sudo cephadm --image "ceph/squid:v19.1.0" --docker --data-dir /cephconfig/var/lib/ceph bootstrap --mon-ip 10.235.22.23 --ssh-user qs-admin --ssh-private-key /home/qs-admin/.ssh/id_rsa --ssh-public-key /home/qs-admin/.ssh/id_rsa.pub --output-dir /cephconfig/etc/ceph --skip-dashboard --skip-monitoring-stack --skip-pull --config my.conf" However, when I then try to continue with the deployment of my Storage Cluster, I find that I can't authenticate with the monitors. I run the suggested command to drop into a cephadm shell which then can't speak to the Storage Cluster. For example: $ ceph -s 2024-08-12T10:47:07.862+0000 7f998e59c640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] [errno 13] RADOS permission denied (error connecting to the cluster) In the MON logs at the same time I can see: "cephx server client.admin: unexpected key: req.key=2c62e1471f111d12 expected_key=d18ce06d18f116b4" In the systemd unit files created I see: ... ExecStart=/bin/bash /var/lib/ceph/64415fba-58b0-11ef-9d27-005056014e4f/%i/unit.run ExecStop=-/bin/bash -c 'bash /var/lib/ceph/64415fba-58b0-11ef-9d27-005056014e4f/%i/unit.stop' ExecStopPost=-/bin/bash /var/lib/ceph/64415fba-58b0-11ef-9d27-005056014e4f/%i/unit.poststop ... Which does not contain my data directory. Looking at the source template it appears that it should: ceph/src/cephadm/cephadmlib/templates/ceph.service.j2 at 616fbc1b181ce15e49281553b35ca215d2aa1053 · ceph/ceph (github.com)<https://github.com/ceph/ceph/blob/616fbc1b181ce15e49281553b35ca215d2aa1053/…> Manually modifying the unit file, reloading systemd and restarting the mon makes the authentication issue go away, although cephadm seems to be periodically rewriting my file and undoing the changes. Is there a templating bug in here? I note that there are no other variables being templated from the ctx in this jinja2 template so it seems likely it is broken. Many thanks, Alex

2 4