- ceph-users - lists.ceph.io

Ceph rgw ldap user acl and quotas
by Ignazio Cassano 19 Apr '23

19 Apr '23

Hello Everyone, We are going to install ceph object storage with ldap autenticazione. We would like to know if acl and quotas on objects and buckets work fine with ldap users. Thanks Ignazio

1 0

Unprivileged Ceph containers
by Stephen Smith6 19 Apr '23

19 Apr '23

Hey folks – just thought I’d ask here – today cephadm creates unit files for Ceph containers with --privileged in the “podman run” call. I saw https://github.com/ceph/ceph-container/blob/main/src/daemon/README.md but I’m not sure how this relates to Ceph when deployed with cephadm? Thanks, Eric

1 0

upgrading from el7 / nautilus
by Marc 19 Apr '23

19 Apr '23

Sorry for addressing this again. But I think there are quite a few still with Nautilus, that are planning such upgrade. Nautilus is currently available for el7, el8 Octopus is currently available for el7, el8 Pacific is currently available for el8, el9 Quincy is currently available for el8, el9 el7 eol 2024 I would like to end up at el9 with a the Pacific release that does not have these performance issues[1] However with the currently offered ceph releases I am more or less forced to an upgrade path of: upgrade hosts from el7 -> el8, upgrade from nautilus -> pacific, upgrade el8 -> el9 - disadvantage upgrading hosts with some co-located services to el8 and then again to el9 - disadvantage having to wait with el9 upgrade until pacific performance issue is fixed. the alternative I have been looking at is building Nautilus for el9 (centos9 stream) which seems to go ok (site-test-packages is not building though). This would result in a more efficient upgrade of: upgrade hosts from el7 -> el9, upgrade from nautilus -> pacific - advantage is only 1 os upgrade. - advantage is I can stay on Nautilus until the pacifc performance issue is fixed. Up until now I can't remember having to much issues with building things from source. However with this storage I would like to have more reassurance. 1. can anyone share experiences with building for different os? 2. what is with all these build warnings, is this common? 3. is it possible that @ceph they can build these rpms and have this go through their default (automated) testing/deploy procedures? I assume with such nice testing facilities this should not take that much time, not? 4. Can anyone recommend some testing procedures? I have a 3 node test cluster. I was thinking of upgrading one host to el9 and run these el9 Nautilus rpms on it, see how it holds. ps. I am not really interested yet in adding complexity with podman. pps. Would be nice if someone from ceph could advice on this. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg19012.html

2 1

HBA or RAID-0 + BBU
by Murilo Morais 19 Apr '23

19 Apr '23

Good evening everyone! Guys, about the P420 RAID controller, I have a question about the operation mode: What would be better: HBA or RAID-0 with BBU (active write cache)? Thanks in advance!

5 4

pacific el7 rpms
by Marc 19 Apr '23

19 Apr '23

Is there ever going to be rpms in https://download.ceph.com/rpm-pacific/el7/

2 2

Consequence of maintaining hundreds of clones of a single RBD image snapshot
by Eyal Barlev 19 Apr '23

19 Apr '23

Hello, My use-case involves creating hundreds of clones (~1,000) of a single RBD image snapshot. I assume watchers exist for each clone, due to the copy-on-write nature of clones. Should I expect a penalty for maintaining such a large number of clones: cpu, memory, performance? If such penalty does exist, we might opt to flatten some of the clones. Is consistency guaranteed during the flattening process? In other words, can I write to a clone while it is being flattened? Perspectivus

3 2

metadata sync
by Christopher Durham 17 Apr '23

17 Apr '23

Hi, After doing a 'radosgw-admin metadata sync init' on a secondary site in a 2-cluster multisite configuration, (See my previous post on doing a quincy upgade with sts roles manually synced between primary and secondary), and letting it sync all metadata from a master zone, I get the following as a large omap object warning on the secondary site: (from ceph health detail): "1 large objects found in pool <zone>.rgw.log" And in ceph.log: "Large omap object found. Object: 87.adcf1188:::meta.full-sync.index.0:head PG 87.abcdfdff (87.0) Key count: 1234567 Size (bytes) 0" I know about large omap objects, and because this is in the log pool, I should be able to delete it with the rados utility. (And do a deep scrub on the pg. Am I correct? Looks like old bucket metadata is still there even though the buckets have been deleted. -Chris

1 0

MGR Memory Leak in Restful
by Chris Palmer 17 Apr '23

17 Apr '23

We've hit a memory leak in the Manager Restful interface, in versions 17.2.5 & 17.2.6. On our main production cluster the active MGR grew to about 60G until the oom_reaper killed it, causing a successful failover and restart of the failed one. We can then see that the problem is recurring, actually on all 3 of our clusters. We've traced this to when we enabled full Ceph monitoring by Zabbix last week. The leak is about 20GB per day, and seems to be proportional to the number of PGs. For some time we just had the default settings, and no memory leak, but had not got around to finding why many of the Zabbix items were showing as Access Denied. We traced this to the MGR's MON CAPS which were "mon 'profile mgr'". The MON logs showed recurring: log_channel(audit) log [DBG] : from='mgr.284576436 192.168.xxx.xxx:0/2356365' entity='mgr.host1' cmd=[{"format": "json", "prefix": "pg dump"}]: access denied Changing the MGR CAPS to "mon 'allow *'" and restarting the MGR immediately allowed that to work, and all the follow-on REST calls worked. log_channel(audit) log [DBG] : from='mgr.283590200 192.168.xxx.xxx:0/1779' entity='mgr.host1' cmd=[{"format": "json", "prefix": "pg dump"}]: dispatch However it has also caused the memory leak to start. We've reverted the CAPS and are back to how we were. Two questions: 1) No matter what the REST consumer is doing, the MGR should not accumulate memory, especially as we can see that the REST TCP connections have wrapped up. Is there anything more we can do to diagnose this? 2) Setting "allow *" worked, but is there are better setting just to allow the "pg dump" call (in addition to profile mgr)? Thanks, Chris

1 0

unable to deploy ceph -- failed to read label for XXX No such file or directory
by Radoslav Bodó 17 Apr '23

17 Apr '23

hello, during basic experimentation I'm running into wierd situaltion when adding osd to test cluster. The test cluster is created as 3x XEN DomU Debian Bookworm (test1-3), 4x CPU, 8GB RAM, xvda root, xvbd swap, 4x xvdj,k,l,m 20GB (LVM volumes in Dom0, propagated via xen phy device) and cleaned with `wipefs -a` ``` apt-get install cephadm ceph-common cephadm bootstrap --mon-ip 10.0.0.101 ceph orch host add test2 ceph orch host add test3 ``` when adding OSDs the first host gets created OSDs as expected, but during creating OSDs on second host the output gets wierd, even when adding each device separately the output shows that `ceph orch` tries to create multiple osds at once ``` root@test1:~# for xxx in j k l m; do ceph orch daemon add osd test2:/dev/xvd$xxx; done Created osd(s) 0,1,2,3 on host 'test2' Created osd(s) 0,1 on host 'test2' Created osd(s) 2,3 on host 'test2' Created osd(s) 1 on host 'test2' ``` the syslog on test2 node shows an errors ``` 2023-04-16T20:57:02.528456+00:00 test2 bash[10426]: cephadm 2023-04-16T20:57:01.389951+0000 mgr.test1.ucudzp (mgr.14206) 1691 : cephadm [INF] Found duplicate OSDs: osd.0 in status running on test1, osd.0 in status error on test2 2023-04-16T20:57:02.528748+00:00 test2 bash[10426]: cephadm 2023-04-16T20:57:01.391346+0000 mgr.test1.ucudzp (mgr.14206) 1692 : cephadm [INF] Removing daemon osd.0 from test2 -- ports [] 2023-04-16T20:57:02.528943+00:00 test2 bash[10426]: cluster 2023-04-16T20:57:02.350564+0000 mon.test1 (mon.0) 743 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-04-16T20:57:17.972962+00:00 test2 bash[20098]: stderr: failed to read label for /dev/ceph-48f3646c-7070-4a37-b9a4-ed0a4a983965/osd-block-11a0dc2b-f8e1-4694-813f-2309ab6a5c1d: (2) No such file or directory 2023-04-16T20:57:17.973064+00:00 test2 bash[20098]: stderr: 2023-04-16T20:57:17.962+0000 7fad2451c540 -1 bluestore(/dev/ceph-48f3646c-7070-4a37-b9a4-ed0a4a983965/osd-block-11a0dc2b-f8e1-4694-813f-2309ab6a5c1d) _read_bdev_label failed to open /dev/ceph-48f3646c-7070-4a37-b9a4-ed0a4a983965/osd-block-11a0dc2b-f8e1-4694-813f-2309ab6a5c1d: (2) No such file or directory 2023-04-16T20:57:17.973181+00:00 test2 bash[20098]: --> Failed to activate via lvm: command returned non-zero exit status: 1 2023-04-16T20:57:17.973278+00:00 test2 bash[20098]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-04-16T20:57:17.973368+00:00 test2 bash[20098]: --> Failed to activate any OSD(s) ``` the ceph and cephadm binaries are installed from debian bookworm ``` ii ceph-common 16.2.11+ds-2 amd64 common utilities to mount and interact with a ceph storage cluster ii cephadm 16.2.11+ds-2 amd64 utility to bootstrap ceph daemons with systemd and containers ``` management session script can be found at https://pastebin.com/raw/FiX7DMHS none of the googled symptoms helped me to understand why is this situation happening nor how to troubleshoot or debug the issues. I'd understand that the nodes are very log on RAM to get this experiment running, but the behavior does not really look like OOM issue. any idea would be appreciated thanks bodik

1 1

RADOSGW zone data-pool migration.
by Gaël THEROND 17 Apr '23

17 Apr '23

Hi everyone, quick question regarding radosgw zone data-pool. I’m currently planning to migrate an old data-pool that was created with inappropriate failure-domain to a newly created pool with appropriate failure-domain. If I’m doing something like: radosgw-admin zone modify —rgw-zone default —data-pool <new_pool> Will data from the old pool be migrated to the new one or do I need to do something else to migrate those data out of the old pool? I’ve read a lot of mail archive with peoples willing to do that but I can’t get a clear answer from those archives. I’m running on nautilus release of it ever help. Thanks a lot!

1 0