- ceph-users - lists.ceph.io

Remove cluster_network without routing
by Torkil Svensgaard 07 Mar '24

07 Mar '24

Hi Cephadm Reef 18.2.0. We would like to remove our cluster_network without stopping the cluster and without having to route between the networks. global advanced cluster_network 192.168.100.0/24 * global advanced public_network 172.21.12.0/22 * The documentation[1] states: " You may specifically assign static IP addresses or override cluster_network settings using the cluster_addr setting for specific OSD daemons. " So for one OSD at a time I could set cluster_addr to override the cluster_network IP and use the public_network IP instead? As the containers are using host networking they have access to both IPs and will just layer 2 the traffic, avoiding routing? When all OSDs are running with a public_network IP set via cluster_addr we can just delete the cluster_network setting and then remove all the ceph_addr settings, as with no cluster_network setting the public_network setting will be used? We tried with one OSD and it seems to work. Anyone see a problem with this approach? Thanks Torkil [1] https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#id3 -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 2

Disable signature url in ceph rgw
by marc＠singer.services 07 Mar '24

07 Mar '24

Hi Ceph users We are using Ceph Pacific (16) in this specific deployment. In our use case we do not want our users to be able to generate signature v4 URLs because they bypass the policies that we set on buckets (e.g IP restrictions). Currently we have a sidecar reverse proxy running that filters requests with signature URL specific request parameters. This is obviously not very efficient and we are looking to replace this somehow in the future. 1. Is there an option in RGW to disable this signed URLs (e.g returning status 403)? 2. If not is this planned or would it make sense to add it as a configuration option? 3. Or is the behaviour of not respecting bucket policies in RGW with signature v4 URLs a bug and they should be actually applied? Thanks you for your help and let me know if you have any questions Marc Singer

4 4

Minimum amount of nodes needed for stretch mode?
by Stefan Kooman 07 Mar '24

07 Mar '24

Hi, TL;DR Failure domain considered is data center. Cluster in stretch mode [1]. - What is the minimum amount of monitor nodes (apart from tie breaker) needed per failure domain? - What is the minimum amount of storage nodes needed per failure domain? - Are device classes supported with stretch mode? - is min_size = 1 in "degraded stretch mode" a hard coded requirement or can this be changed to it at leave min_size = 2 (yes, I'm aware that no other OSD may go down in the surviving data center or PGs will become unavailable). I've converted a (test) 3 node replicated cluster (2 storage nodes, 1 node with monitor only, min_size=2, size=4) setup to a "stretch mode" setup [1]. That works as expected. CRUSH rule (adjusted to work with 1 host and 2 OSDs per device class per data center only) rule stretch_rule { id 5 type replicated step take dc1 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit step take dc2 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit } The documentation seems to suggest that 2 storage nodes and 2 monitor nodes are needed at a minimum. Is that correct? I wonder why? For a minimal (as possible) cluster I don't see the need for one additional monitor per datacenter. Does the tiebreaker monitor function as a normal monitor (apart from it not allowed to become leader)? When stretch rules with device classes are used things don't work as expected anymore. Example crush rule: rule stretch_rule_ssd { id 4 type replicated step take dc1 class ssd step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit step take dc2 class ssd step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit } A similar crush rule for hdd exists. When I change the crush_rule for one of the pools to use stretch_rule_ssd the PGs on OSDs with device class ssd become inactive as soon as one of the data centers goes offline (and "degraded stretched mode" has been activated, and only 1 bucket, data center, is needed for peering). I don't understand why. Another issue with this is that as soon as the datacenter is online again, the recovery will never finish by itself and a "ceph osd force_healthy_stretch_mode --yes-i-really-mean-it" is needed to get HEALTH_OK Can anyone explain to me why this is? Gr. Stefan [1]: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/

2 2

ceph-volume fails when adding spearate DATA and DATA.DB volumes
by service.plant＠ya.ru 07 Mar '24

07 Mar '24

Hi all! I;ve faced an issue I couldnt even google. Trying to create OSD with two separate LVM for data.db and data, gives me intresting error ``` root@ceph-uvm2:/# ceph-volume lvm prepare --bluestore --data ceph-block-0/block-0 --block.db ceph-db-0/db-0 --> Incompatible flags were found, some values may get ignored --> Cannot use None (None) with --bluestore (bluestore) --> Incompatible flags were found, some values may get ignored --> Cannot use --bluestore (bluestore) with --block.db (bluestore) Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e3b7b9e5-6399-4e61-8634-41a991bb1948 stderr: 2024-03-04T11:45:52.263+0000 7f71c3a89700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-03-04T11:45:52.263+0000 7f71c3a89700 -1 AuthRegistry(0x7f71bc064978) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2024-03-04T11:45:52.263+0000 7f71c3a89700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-03-04T11:45:52.263+0000 7f71c3a89700 -1 AuthRegistry(0x7f71bc067fd0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2024-03-04T11:45:52.267+0000 7f71c3a89700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2024-03-04T11:45:52.267+0000 7f71c3a89700 -1 AuthRegistry(0x7f71c3a87ea0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2024-03-04T11:45:52.267+0000 7f71c1024700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2024-03-04T11:45:52.267+0000 7f71c2026700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2024-03-04T11:45:52.267+0000 7f71c1825700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2024-03-04T11:45:52.267+0000 7f71c3a89700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication stderr: [errno 13] RADOS permission denied (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id ``` And here is output which shows my created volumes ``` root@ceph-uvm2:/# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert block-0 ceph-block-0 -wi-a----- 16.37t db-0 ceph-db-0 -wi-a----- 326.00g ``` CEPH was rolled out by using cephadm and all the commands I do under cephadm shell I have no idea what is the reason of that error and that is why I am here. Any help is appreciated. Thanks in advance.

2 2

erasure-code-lrc Questions regarding repair
by Ansgar Jazdzewski 07 Mar '24

07 Mar '24

hi folks, I currently test erasure-code-lrc (1) in a multi-room multi-rack setup. The idea is to be able to repair a disk-failures within the rack itself to lower bandwidth-usage ```bash ceph osd erasure-code-profile set lrc_hdd \ plugin=lrc \ crush-root=default \ crush-locality=rack \ crush-failure-domain=host \ crush-device-class=hdd \ mapping=__DDDDD__DDDDD__DDDDD__DDDDD \ layers=' [ [ "_cDDDDD_cDDDDD_cDDDDD_cDDDDD", "" ], [ "cDDDDDD_____________________", "" ], [ "_______cDDDDDD______________", "" ], [ "______________cDDDDDD_______", "" ], [ "_____________________cDDDDDD", "" ], ]' \ crush-steps='[ [ "choose", "room", 4 ], [ "choose", "rack", 1 ], [ "chooseleaf", "host", 7 ], ]' ``` The roule picks 4 out of 5 rooms and keeps the PG in one rack like expected! However it looks like the PG will not move to another Room if the PG is undersized or the entire Room or Rack is down! Questions: * do I miss something to allow LRC (PG's) to move across Racks/Rooms for repair? * Is it even possible to build such a 'Multi-stage' grushmap? Thanks for your help, Ansgar 1) https://docs.ceph.com/en/quincy/rados/operations/erasure-code-jerasure/

2 2

Running dedicated RGWs for async tasks
by Marc Singer 07 Mar '24

07 Mar '24

Hello Ceph Users Since we are running a big S3 cluster we would like to externalize the RGW daemons that do async tasks, like: * Garbage collection * Lifecycle policies * Calculating and updating quotas Would this be possible to do in the configuration? Which config values would I need to set on the exposed RGWs and which on the Async Task RGWs? Thanks for your input. Marc

2 1

Unable to map RBDs after running pg-upmap-primary on the pool
by Torkil Svensgaard 07 Mar '24

07 Mar '24

Hi I tried to do offline read optimization[1] this morning but I am now unable to map the RBDs in the pool. I did this prior to running the pg-upmap-primary commands suggested by the optimizer, as suggested by the latest documentation[2]: " ceph osd set-require-min-compat-client reef " The command didn't complain and the documentation stated it should failed if any pre-reef clients were connected so I thought all was well and rad the pg_upmap_primary commands. Can I simply remove the applied pg_upmap_primary settings somehow to my RBDs back while investigating? Mvh. Torkil [1] https://docs.ceph.com/en/reef/rados/operations/read-balancer/ [2] https://docs.ceph.com/en/latest/rados/operations/read-balancer/ -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

1 1

Ceph is constantly scrubbing 1/4 of all PGs and still have pigs not scrubbed in time
by thymus_03fumbler＠icloud.com 07 Mar '24

07 Mar '24

I recently switched from 16.2.x to 18.2.x and migrated to cephadm, since the switch the cluster is constantly scrubbing, 24/7 up to 50 PGs simultaneously and up to 20 deep scrubs simultaneously in a cluster that has only 12 (in use) OSDs. Furthermore it still manages to regularly have a warning with ‘pgs not scrubbed in time’ I have tried various settings, like osd_deep_scrub_interval, osd_max_scrubs, mds_max_scrub_ops_in_progress etc. All those get ignored. Please advice. Here is an output of ceos config dump: WHO MASK LEVEL OPTION VALUE RO global advanced auth_client_required cephx * global advanced auth_cluster_required cephx * global advanced auth_service_required cephx * global advanced auth_supported cephx * global basic container_image quay.io/ceph/ceph@sha256:aca35483144ab3548a7f670db9b79772e6fc51167246421c66c0bd56a6585468 * global basic device_failure_prediction_mode local global advanced mon_allow_pool_delete true global advanced mon_data_avail_warn 20 global advanced mon_max_pg_per_osd 400 global advanced osd_max_pg_per_osd_hard_ratio 10.000000 global advanced osd_pool_default_pg_autoscale_mode on mon advanced auth_allow_insecure_global_id_reclaim false mon advanced mon_crush_min_required_version firefly * mon advanced mon_warn_on_pool_no_redundancy false mon advanced public_network 10.79.0.0/16 * mgr advanced mgr/balancer/active true mgr advanced mgr/balancer/mode upmap mgr advanced mgr/cephadm/manage_etc_ceph_ceph_conf_hosts label:admin * mgr advanced mgr/cephadm/migration_current 6 * mgr advanced mgr/dashboard/GRAFANA_API_PASSWORD admin * mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false * mgr advanced mgr/dashboard/GRAFANA_API_URL https://10.79.79.12:3000 * mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://10.79.79.12:9095 * mgr advanced mgr/devicehealth/enable_monitoring true mgr advanced mgr/orchestrator/orchestrator cephadm osd advanced osd_map_cache_size 250 osd advanced osd_map_share_max_epochs 50 osd advanced osd_mclock_profile high_client_ops osd advanced osd_pg_epoch_persisted_max_stale 50 osd.0 basic osd_mclock_max_capacity_iops_hdd 380.869888 osd.1 basic osd_mclock_max_capacity_iops_hdd 441.000000 osd.10 basic osd_mclock_max_capacity_iops_ssd 13677.906485 osd.11 basic osd_mclock_max_capacity_iops_hdd 274.411212 osd.13 basic osd_mclock_max_capacity_iops_hdd 198.492501 osd.2 basic osd_mclock_max_capacity_iops_hdd 251.592009 osd.3 basic osd_mclock_max_capacity_iops_hdd 208.197434 osd.4 basic osd_mclock_max_capacity_iops_hdd 196.544082 osd.5 basic osd_mclock_max_capacity_iops_ssd 12739.225456 osd.6 basic osd_mclock_max_capacity_iops_hdd 211.288660 osd.7 basic osd_mclock_max_capacity_iops_hdd 210.543236 osd.8 basic osd_mclock_max_capacity_iops_hdd 242.241594 osd.9 basic osd_mclock_max_capacity_iops_hdd 559.933780 mds.plexfs basic mds_join_fs plexfs Here is a ceph -s output services: mon: 3 daemons, quorum lxt-prod-ceph-util02,lxt-prod-ceph-util01,lxt-prod-ceph-util03 (age 3w) mgr: lxt-prod-ceph-util02.iyrhxj(active, since 3w), standbys: lxt-prod-ceph-util03.wvstpe mds: 1/1 daemons up osd: 14 osds: 14 up (since 4w), 14 in (since 4w) data: volumes: 1/1 healthy pools: 4 pools, 193 pgs objects: 14.48M objects, 52 TiB usage: 71 TiB used, 39 TiB / 110 TiB avail pgs: 131 active+clean 47 active+clean+scrubbing 15 active+clean+scrubbing+deep

3 2

Ceph Cluster Config File Locations?
by duluxoz 07 Mar '24

07 Mar '24

Hi All, I don't know how its happened (bad backup/restore, bad config file somewhere, I don't know) but my (DEV) Ceph Cluster is in a very bad state, and I'm looking for pointers/help in getting it back running (unfortunate, a complete rebuild/restore is *not* an option). This is on Ceph Reef (on Rocky 9) which was converted to CephAdm from a manual install a few weeks ago (which worked). Five days ago everything when "t!ts-up" (an Ozzie technical ICT term meaning nothing works :-) ) So, my (first?) issue is that I can't get any Managers to come up clean. Each one tries to connect on an ip subnet which doesn't exist any longer and hasn't for a couple of years. The second issue is that (possible because of the first) every `ceph orch` command just hangs. Cephadm commands work fine. I've checked, checked, and checked again that the individual config files all point towards the correct ip subnet for the monitors, and I cannot find any trace of the old subnet's ip address in any config file (that I can find). For the record I am *not* a "podman guy" so there may be something there that's causing my issue(s?) but I don't know where to even begin to look. Any/all logs simply start that the Manager(s) try to come up, can't find an address in the "old" subnet, and so fail - nothing else helpful (at least to me). I've even pulled a copy of the monmap and its showing the correct ip subnet addresses for the monitors. The firewalls are all set correctly and an audit2allow shows nothing is out of place, as does disabling SELinux (ie no change). A `ceph -s` shows I've got no active managers (and that a monitor is down - that's my third issue), plus a whole bunch of osds and pgs aren't happy either. I have, though, got a monitor quorum. So, what should I be looking at / where should I be looking? Any help is greatly *greatly* appreciated. Cheers Dulux-Oz

3 3

ceph Quincy to Reef non cephadm upgrade
by sarda.ravi＠gmail.com 07 Mar '24

07 Mar '24

I want to perform non cephadm upgrade from Quincy to Reef. Reason for not using cephadm is do not want to go for ceph in containers. My test deployment is as given below. Total cluster hosts : 5 ceph-mon hosts: 3 ceph-mgr hosts: 3 (ceph-mgr active on one node, and other ceph-mgr each on ceph-mon host) ceph-mds : 1 ceph-osd : 5 (one ceph-osd on each of the host in the cluster.) While I try to follow the steps - https://docs.ceph.com/en/latest/releases/reef/#upgrading-non-cephadm-cluste… - on the step - Upgrade monitors by installing the new packages and restarting the monitor daemons. when I try to upgrade only ceph-mon using "apt upgrade ceph-mon" command it upgrades all packages including ceph-mgr, ceph-mds, ceph-osd etc. as ceph-mon package has dependency on these packages. My question is - does this mean I need to upgrade all ceph packages (ceph, ceph-common) and restart only monitor daemon first? Or there is any way I can upgrade only ceph-mon pacakge first, then ceph-mgr, ceph-osd and so on?

2 1