- ceph-users - lists.ceph.io

The pg_num from 1024 reduce to 32 spend much time, is there way to shorten the time？
by Louis Koo 08 Jun '23

08 Jun '23

ceph version is 16.2.13; The pg_num is 1024, and the target_pg_num is 32; there is no any data in the pool of ".rgw.buckets.index", but it spend much time to reduce the pg num.

5 12

rbd ls failed with operation not permitted
by zyz 08 Jun '23

08 Jun '23

When set the user's auth and then ls namespace, it is ok. But when I set the user's auth with namespace, ls namespace returns with error, but why?

2 1

[RGW] what is log_meta and log_data config in a multisite config?
by Gilles Mocellin 08 Jun '23

08 Jun '23

Hi Cephers, In a multisite config, with one zonegroup and 2 zones, when I look at `radiosgw-admin zonegroup get`, I see by defaut these two parameters : "log_meta": "false", "log_data": "true", Where can I find documentation on these, I can't find. I set log_meta to true, because, why not ? Is it a bad thing ?

2 2

Question about xattr and subvolumes
by Dario Graña 08 Jun '23

08 Jun '23

Hi, I'm installing a new instance (my first) of Ceph. Our cluster runs AlmaLinux9 + Quincy. Now I'm dealing with CephFS and quotas. I read documentation about setting up quotas with virtual attributes (xattr) and creating volumes and subvolumes with a prefixed size. I cannot distinguish which is the best option for us. Currently we create a directory with a project name and some subdirectories inside. I would like to understand the difference between both options. Thanks in advance. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

keep rbd command history ever executed
by huxiaoyu＠horebdata.cn 08 Jun '23

08 Jun '23

Dear Ceph folks, In a Ceph cluster there could be multiple points (e.g. librbd clients) being able to execute rbd commands. My question is that , is there a methold to reliably record or keep a full rbd command history that ever being executed? This would be helpful for auditors as well as for system operators. any ideas? Samuel huxiaoyu(a)horebdata.cn

1 0

RadosGW S3 API Multi-Tenancy
by Brad House 08 Jun '23

08 Jun '23

I'm in the process of exploring if it is worthwhile to add RadosGW to our existing ceph cluster. We've had a few internal requests for exposing the S3 API for some of our business units, right now we just use the ceph cluster for VM disk image storage via RBD. Everything looks pretty straight forward until we hit multitenancy. The page on multi-tenancy doesn't dive into permission delegation: https://docs.ceph.com/en/quincy/radosgw/multitenancy/ The end goal I want is to be able to create a single user per tenant (Business Unit) which will act as their 'administrator', where they can then do basically whatever they want under their tenant sandbox (though I don't think we need more advanced cases like creations of roles or policies, just create/delete their own users, buckets, objects). I was hopeful this would just work, and I asked on the ceph IRC channel on OFTC and was told once I grant a user caps="users=*", they would then be allowed to create users *outside* of their own tenant using the Rados Admin API and that I should explore IAM roles. I think it would make sense to add a feature, such as a flag that can be set on a user, to ensure they stay in their "sandbox". I'd assume this is probably a common use-case. Anyhow, if its possible to do today using iam roles/policies, then great, unfortunately this is my first time looking at this stuff and there are some things not immediately obvious. I saw this online about AWS itself and creating a permissions boundary, but that's for allowing creation of roles within a boundary: https://www.qloudx.com/delegate-aws-iam-user-and-role-creation-without-givi… I'm not sure what "Action" is associated with the Rados Admin API create user for applying a boundary that the user can only create users with the same tenant name. https://docs.ceph.com/en/quincy/radosgw/adminops/#create-user Any guidance on this would be extremely helpful. Thanks! -Brad

1 1

16.2.13: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
by Zakhar Kirpichenko 08 Jun '23

08 Jun '23

Hi, I'm having an issue with crash daemons on Pacific 16.2.13 hosts. ceph-crash throws the following error on all hosts: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ceph-crash runs in docker, the container has the directory mounted: -v /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z The mount works correctly: 18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls -al crash/posted/ total 8 drwx------ 2 nobody nogroup 4096 May 6 2021 . drwx------ 3 nobody nogroup 4096 May 6 2021 .. 18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# touch crash/posted/a 18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# docker exec -it c0cd2b8022d8 bash [root@ceph02 /]# ls -al /var/lib/ceph/crash/posted/ total 8 drwx------ 2 nobody nobody 4096 Jun 1 18:26 . drwx------ 3 nobody nobody 4096 May 6 2021 .. -rw-r--r-- 1 root root 0 Jun 1 18:26 a I.e. the directory actually exists and is correctly mounted in the crash container, yet ceph-crash says it doesn't exist. How can I convince it that the directory is there? Best regards, Zakhar

3 5

Bucket resharding in multisite without data replication
by Danny Webb 08 Jun '23

08 Jun '23

Hi Ceph users, We have 3 clusters running Pacific 16.2.9 all setup in a multisite configuration with no data replication (we wanted to use per bucket policies but never got them working to our satisfaction). All of the resharding documentation I've found regarding multisite is centred around multisite with data replication and having to reshard from the primary region and re-replicating data. But our data is spread amongst the regions and may not be in the primary region. Testing has shown that resharding from the primary region in the case of a bucket with data only in a remote region results in the remote bucket losing it's ability to list contents (seemingly breaking the index in the remote region). Is there a way (besides waiting for reef and dynamic bucket resharding for multisite) to reshard buckets in this setup? Cheers, Danny Danny Webb Principal OpenStack Engineer Danny.Webb(a)thehutgroup.com [THG Ingenuity Logo]<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en>

1 0

How to secure erasing a rbd image without encryption?
by huxiaoyu＠horebdata.cn 08 Jun '23

08 Jun '23

Dear ceph folks, I bumped into an very interesting challenge, how to secure erase a rbd image data without any encryption? The motivation is to ensure that there is no information leak on OSDs after deleting a user specified rbd image, without the extra burden of using rbd encryption. any ideas, suggestions are highly appreciated, Samuel huxiaoyu(a)horebdata.cn

4 3

Issues in installing old dumpling version to add a new monitor
by Cloud List 08 Jun '23

08 Jun '23

Hi, I have a very old Ceph cluster running the old dumpling version 0.67.1. One of the three monitors suffered a hardware failure and I am setting up a new server to replace the third monitor running Ubuntu 22.04 LTS (all the other monitors are using the old Ubuntu 12.04 LTS). I used ceph-deploy to deploy the cluster initially, and I can't use it since it's a very old version of ceph-deploy -- having issues with apt-key being deprecated and since ceph-deploy is no longer maintained, I can't upgrade it. And even if I can, I am not too sure if it works since the dumpling version is no longer in Ceph's official repository. So I tried to install it manually by cloning it from git: git clone -b dumpling https://github.com/ceph/ceph.git But when I tried to run "git submodule update --init" or "./autogen.sh" as per the README file, I am encountering this error: ==== root@ceph-mon-04:~/ceph-dumpling/ceph# git submodule update --init Submodule 'ceph-object-corpus' (git://ceph.com/git/ceph-object-corpus.git) registered for path 'ceph-object-corpus' Submodule 'src/libs3' (git://github.com/ceph/libs3.git) registered for path 'src/libs3' Cloning into '/root/ceph-dumpling/ceph/ceph-object-corpus'... fatal: repository 'https://ceph.com/git/ceph-object-corpus.git/' not found fatal: clone of 'git://ceph.com/git/ceph-object-corpus.git' into submodule path '/root/ceph-dumpling/ceph/ceph-object-corpus' failed Failed to clone 'ceph-object-corpus'. Retry scheduled Cloning into '/root/ceph-dumpling/ceph/src/libs3'... Cloning into '/root/ceph-dumpling/ceph/ceph-object-corpus'... fatal: repository 'https://ceph.com/git/ceph-object-corpus.git/' not found fatal: clone of 'git://ceph.com/git/ceph-object-corpus.git' into submodule path '/root/ceph-dumpling/ceph/ceph-object-corpus' failed Failed to clone 'ceph-object-corpus' a second time, aborting root@ceph-mon-04:~/ceph-dumpling/ceph# git submodule update --init --recursive Cloning into '/root/ceph-dumpling/ceph/ceph-object-corpus'... fatal: repository 'https://ceph.com/git/ceph-object-corpus.git/' not found fatal: clone of 'git://ceph.com/git/ceph-object-corpus.git' into submodule path '/root/ceph-dumpling/ceph/ceph-object-corpus' failed Failed to clone 'ceph-object-corpus'. Retry scheduled Cloning into '/root/ceph-dumpling/ceph/ceph-object-corpus'... fatal: repository 'https://ceph.com/git/ceph-object-corpus.git/' not found fatal: clone of 'git://ceph.com/git/ceph-object-corpus.git' into submodule path '/root/ceph-dumpling/ceph/ceph-object-corpus' failed Failed to clone 'ceph-object-corpus' a second time, aborting root@ceph-mon-04:~/ceph-dumpling/ceph# ==== It seems that the repositories required for the submodules are no longer there. Anyone can advise me on the correct direction on how can I install the dumpling version of Ceph for me to add a new monitor? At the moment only 2 monitors out of 3 are up and I am worried that the cluster will be down if I lose another monitor. $ ceph status cluster 1660b11f-1074-4f5d-aa7c-64b479397a2f health HEALTH_WARN 1 mons down, quorum 0,1 ceph-mon-01,ceph-mon-02 What approach I should take: - Continue trying the manual installation/compiling route? - Continue trying the ceph-deploy route (by fixing the apt-key deprecation issue)? - Try to install the same old OS (Ubuntu 12.04 LTS) on the new server (not too sure if I still have the ISO) and see if it works? - Try to upgrade the current cluster and then add the monitor later after upgrade? (is it risky to upgrade with HEALTH_WARN status)? Any advice is greatly appreciated. Best regards, -ip-

3 2