- ceph-users - lists.ceph.io

no mds services
by Ex Calibur 15 Sep '24

15 Sep '24

Hello, I'm following this guide to upgrade our cephs: https://ainoniwa.net/pelican/2021-08-11a.html (Proxmox VE 6.4 Ceph upgrade Nautilus to Octopus) It's a requirement to upgrade our ProxMox environnement. Now I've reached the point at that guide where i have to "Upgrade all CephFS MDS daemons" But before I started this piece, I checked the status. root@pmnode1:~# ceph status cluster: id: xxxxxxxxxxxxxxx health: HEALTH_ERR noout flag(s) set 1 scrub errors Possible data damage: 1 pg inconsistent 2 pools have too many placement groups services: mon: 3 daemons, quorum pmnode1,pmnode2,pmnode3 (age 19h) mgr: pmnode2(active, since 19h), standbys: pmnode1 osd: 15 osds: 12 up (since 12h), 12 in (since 19h) flags noout data: pools: 3 pools, 513 pgs objects: 398.46k objects, 1.5 TiB usage: 4.5 TiB used, 83 TiB / 87 TiB avail pgs: 512 active+clean 1 active+clean+inconsistent root@pmnode1:~# ceph mds metadata [] as you can see there is no mds service running. What can be wrong and how to solve this? Thank you in advance.

3 2

Numa pinning best practices
by Szabo, Istvan (Agoda) 15 Sep '24

15 Sep '24

Hi, Haven't really found a proper descripton in case of 2 socket how to pin osds to numa node, only this: https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment… Tuning for All Flash Deployments - Ceph - Ceph <https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment…> Redmine tracker.ceph.com Is there anybody have some good how to on this topic? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 5

[RGW][cephadm] How to configure RGW as code and independantely of daemon names ?
by Gilles Mocellin 14 Sep '24

14 Sep '24

Hello Cephers, In our journey to migrate to cephadm, I face a problem with RGW: We use Ceph with OpenStack, so we have to configure keystone authentification, with many rgw_keystone_* variables in /etc/ceph/ceph.conf. It's easy with ceph-ansible, and the section in ceph.conf has always the same name, constructed from the inventory. With cephadm, daemon names for RGW contain random characters, and change at every redeploy. As we dont't have ceph.conf anymore, the configuration should be in ceph config. But how can we have it set once, and preserved between name changes ? I wish I could set thoses config vars in the RGW spec file, having a good "as code" approach, but no, extra params are only for the radosgw binary. It would also be fine if we could have defined ceph confg keys with wildcards, but no. How do you handle rados GW config, with cephadm and ceph config ? -- Gilles

5 6

Re: ceph-mgr perf throttle-msgr - what is caused fails?
by Konstantin Shalygin 14 Sep '24

14 Sep '24

As I said before, currently Prometheus module performance degradation is only one _visible_ issues. I named things like is as indicator (of feature problem's) k Sent from my iPhone > On 12 Sep 2024, at 23:18, Eugen Block <eblock(a)nde.ag> wrote: > > But did you notice any actual issues or did you just see that value being that high without any connection to an incident?

2 2

bluefs _allocate unable to allocate on bdev 2
by Szabo, Istvan (Agoda) 13 Sep '24

13 Sep '24

Hi, Since yesterday on ceph octopus it started to crash multiple osds in the cluster and I can see this error in most of the logs: 2024-09-12T06:13:35.805+0700 7f98b8b27700 1 bluefs _allocate failed to allocate 0xf0732 on bdev 1, free 0x40000; fallback to bdev 2 2024-09-12T06:13:35.805+0700 7f98b8b27700 1 bluefs _allocate unable to allocate 0xf0732 on bdev 2, free 0xffffffffffffffff; fallback to slow device expander The osds are 57% full so I think space issue shouldn't happen. I'm using ssds so I don't have separated wal/rocksdb. Running some compaction now but I don't think it will happen on a long run. What could be this issue and how to fix? Ty ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 7

squid 19.2.0 QE validation status
by Yuri Weinstein 12 Sep '24

12 Sep '24

Details of this release are summarized here: https://tracker.ceph.com/issues/67779#note-1 Release Notes - TBD Gibba upgrade -TBD LRC upgrade - TBD It was decided and agreed upon that there would be limited testing for this release, given it is based on 19.1.1 rather than a full rebase. Seeking approvals/reviews for: (some reruns are still in progress) rgw - Eric, Adam E fs - Venky orch - Adam King quincy-x - Laura, Neha reef-x - Laura, Neha Pls let me know if any tests were missed from this list.

7 18

lifecycle policy on non-replicated buckets
by Christopher Durham 12 Sep '24

12 Sep '24

Hi, I have a multisite system with two sites on 18.2.2, on Rocky 8. I have set up a sync policy to allow replication between sites. I have also createda policy for a given bucket that prevents replication on that given bucket. This allworks just fine, and objects I create in that bucket on side A do not get replicatedto side B, and objects I create in that bucket on side B do not get replicated to side A, butreplication for all other buckets works fine. This is great. But I still want to have a lifecycle policy that deletes objects after say, 4 days. If I create and install the json for this policy, via s3api put-bucket-lifecycle-configuration then only on the master side, side A, do objects get deleted after 4 days. Any objects on side B never get deleted. What am I doing wrong? Thanks -Chris

2 4

ceph-mgr perf throttle-msgr - what is caused fails?
by Konstantin Shalygin 12 Sep '24

12 Sep '24

Hi, seems something in mgr is throttle due val > max. I'm right? root@mon1# ceph daemon /var/run/ceph/ceph-mgr.mon1.asok perf dump | jq '."throttle-msgr_dispatch_throttler-mgr-0x55930f4aed20"' { "val": 104856554, "max": 104857600, "get_started": 0, "get": 9700833, "get_sum": 654452218418, "get_or_fail_fail": 1323887918, "get_or_fail_success": 9700833, "take": 0, "take_sum": 0, "put": 9698716, "put_sum": 654347361864, "wait": { "avgcount": 0, "sum": 0, "avgtime": 0 } } The question is - how-to determine what exactly? Another fail_fail in perf counters is zero. mgr is not in container, and have resources to work Thanks, k

2 3

Successfully using dm-cache
by Michael Lipp 12 Sep '24

12 Sep '24

Just in case anybody is interested: Using dm-cache works and boosts performance -- at least for my use case. The "challenge" was to get 100 (identical) Linux-VMs started on a three node hyperconverged cluster. The hardware is nothing special, each node has a Supermicro server board with a single CPU with 24 cores and 4 x 4 TB hard disks. And there's that extra 1 TB NVMe... I know that the general recommendation is to use the NVMe for WAL and metadata, but this didn't seem appropriate for my use case and I'm still not quite sure about failure scenarios with this configuration. So instead I made each drive a logical volume (managed by an OSD) and added 85 GiB NVMe to each LV as read-only cache. Each VM uses as system disk an RBD based on a snapshot from the master image. The idea was that with this configuration, all VMs should share most (actually almost all) of the data on their system disk and this data should be available from the cache. Well, it works. When booting the 100 VMs, almost all read operations are satisfied from the cache. So I get close to NVMe speed but have payed for conventional hard drives only (well, SSDs aren't that much more expensive nowadays, but the hardware is 4 years old). So, nothing sophisticated, but as I couldn't find anything about this kind of setup, it might be of interest nevertheless. - Michael

13 27

RGW sync gets stuck every day
by Olaf Seibert 12 Sep '24

12 Sep '24

Hi all, we have some Ceph clusters with RGW replication between them. It seems that in the last month at least, it gets stuck at around the same time ~every day. Not 100% the same time, and also not 100% of the days, but in the more recent days seem to happen more, and for longer. With "stuck" I mean that the "oldest incremental change not applied" is getting 5 or more minutes old, and not changing. In the past this seemed to resolve itself in a short time, but recently it didn't. It remained stuck at the same place for several hours. Also, on several different occasions I noticed that the shard number in question was the same. We are using Ceph 18.2.2, image id 719d4c40e096. The output on one end looks like this (I redacted out some of the data because I don't know how much of the naming would be sensitive information): root@zone2:/# radosgw-admin sync status --rgw-realm backup realm xxxxxxxx-xxxx-xxxx-xxxx-8ddf4576ebab (backup) zonegroup xxxxxxxx-xxxx-xxxx-xxxx-58af9051e063 (backup) zone xxxxxxxx-xxxx-xxxx-xxxx-e1223ae425a4 (zone2-backup) current time 2024-08-04T10:22:00Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) data sync source: xxxxxxxx-xxxx-xxxx-xxxx-e8db1c51b705 (zone1-backup) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [30,90,95] oldest incremental change not applied: 2024-08-04T10:05:54.015403+0000 [30] while on the other side it looks ok (not more than half a minute behind): root@zone1:/# radosgw-admin sync status --rgw-realm backup realm xxxxxxxx-xxxx-xxxx-xxxx-8ddf4576ebab (backup) zonegroup xxxxxxxx-xxxx-xxxx-xxxx-58af9051e063 (backup) zone xxxxxxxx-xxxx-xxxx-xxxx-e8db1c51b705 (zone1-backup) current time 2024-08-04T10:23:05Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: xxxxxxxx-xxxx-xxxx-xxxx-e1223ae425a4 (zone2-backup) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 4 shards behind shards: [89,92,95,98] oldest incremental change not applied: 2024-08-04T10:22:53.175975+0000 [95] With some experimenting, we found that redeploying the RGWs on this side resolves the situation: "ceph orch redeploy rgw.zone1-backup". The shards go into "Recovering" state and after a short time it is "caught up with source" as well. Redeploying stuff seems like a much too big hammer to get things going again. Surely there must be something more reasonable? Also, any ideas about how we can find out what is causing this? It may be that some customer has some job running every 24 hours, but that shouldn't cause the replication to get stuck. Thanks in advance, -- Olaf Seibert Site Reliability Engineer SysEleven GmbH Boxhagener Straße 80 10245 Berlin T +49 30 233 2012 0 F +49 30 616 7555 0 https://www.syseleven.de https://www.linkedin.com/company/syseleven-gmbh/ Current system status always at: https://www.syseleven-status.net/ Company headquarters: Berlin Registered court: AG Berlin Charlottenburg, HRB 108571 Berlin Managing directors: Andreas Hermann, Jens Ihlenfeld, Norbert Müller, Jens Plogsties

3 4