- ceph-users - lists.ceph.io

user and bucket not sync ( permission denied )
by Guillaume Morin 07 Mar '23

07 Mar '23

Hello, i have an issue about my multisite configuration. pacific 16.2.9 My problem: i have a permission denied on the the master zone when i use the command below. $ radosgw-admin sync status realm 8df19226-a200-48fa-bd43-1491d32c636c (myrealm) zonegroup 29592d75-224d-49b6-bc36-2703efa4f67f (myzonegroup) zone 6cce41f3-a54b-47c2-981f-3b56ca0a4489 (myzone) metadata sync no sync (zone is master) 2023-03-07T22:31:16.466+0100 7f96a3e7a840 0 ERROR: failed to fetch datalog info data sync source: f2b20676-2672-4a92-a7ee-f3eb2efb12c6 (mysecondaryzone) failed to retrieve sync info: (13) Permission denied because on secondary zone (read only) , i see a 403 error about the permission denied from the master node 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 ====== starting new request req=0x7f1fd418c620 ===== 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 req 2604939314198041770 0.000000000s op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 ====== req done req=0x7f1fd418c620 op status=0 http_status=403 latency=0.000000000s ====== 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 beast: 0x7f1fd418c620: 10......... - - [07/Mar/2023:00:00:53.309 +0100] "POST /admin/realm/period?period=395f9f13-d941-4ccf-a0cf-6c5d6d6579c2&epoch=76&rgwx-zonegroup=29592d75-224d-49b6-bc36-2703efa4f67f HTTP/1.1" 403 194 - - - latency=0.000000000s 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 ====== starting new request req=0x7f1fd4411620 ===== 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 req 7374970752399537975 0.000000000s op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 ====== req done req=0x7f1fd4411620 op status=0 http_status=403 latency=0.000000000s ====== 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 beast: 0x7f1fd4411620: 10......... - - [07/Mar/2023:00:00:53.441 +0100] "POST /admin/log?type=data&notify&source-zone=6cce41f3-a54b-47c2-981f-3b56ca0a4489&rgwx-zonegroup=29592d75-224d-49b6-bc36-2703efa4f67f HTTP/1.1" 403 194 - - - latency=0.000000000s No issue when i use the command to check sync on secondary zone I don't understand because on secondary zone, pull realm and period with a user with flag system and admin works, the sync works for objects but not for users and buckets. When i list user and bucket on secondary zone, there are nothing but i have my objects on pool bucket.data !! i think the 403 was due because my user with flag system doesn't exist on secondary zone but i don't understand why user and bucket are not syncronized ??!! Access key and secret key are set on master zone and secondary zone, endpoint also I have an other cluster with a similary configuration and i don't have any issue Can someone help me ? Sorry for my english Regards Guillaume

1 1

Issue upgrading 17.2.0 to 17.2.5
by aellahib＠gmail.com 07 Mar '23

07 Mar '23

I initially ran the upgrade fine but it failed @ around 40/100 on an osd, so after waiting for along time i thought I'd try restarting it and then restarting the upgrade. I am stuck with the below debug error, I have tested docker pull from other servers and they dont fail for the ceph images but on ceph it does. If i even try to redeploy or add or remove mon damons for example it comes up with the same error related to the images. The error that ceph is giving me is: 2023-03-02T07:22:45.063976-0700 mgr.mgr-node.idvkbw [DBG] _run_cephadm : args = [] 2023-03-02T07:22:45.070342-0700 mgr.mgr-node.idvkbw [DBG] args: --image stop --no-container-init pull 2023-03-02T07:22:45.081086-0700 mgr.mgr-node.idvkbw [DBG] Running command: which python3 2023-03-02T07:22:45.180052-0700 mgr.mgr-node.idvkbw [DBG] Running command: /usr/bin/python3 /var/lib/ceph/5058e342-dac7-11ec-ada3-01065e90228d/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e --image stop --no-container-init pull 2023-03-02T07:22:46.500561-0700 mgr.mgr-node.idvkbw [DBG] code: 1 2023-03-02T07:22:46.500787-0700 mgr.mgr-node.idvkbw [DBG] err: Pulling container image stop... Non-zero exit code 1 from /usr/bin/docker pull stop /usr/bin/docker: stdout Using default tag: latest /usr/bin/docker: stderr Error response from daemon: pull access denied for stop, repository does not exist or may require 'docker login': denied: requested access to the resource is denied ERROR: Failed command: /usr/bin/docker pull stop

4 8

Creating a role for quota management
by anantha.adiga＠intel.com 07 Mar '23

07 Mar '23

Hello, Can you provide details on how to create a role for allowing a set of users to set quota on CephFS pools ?

3 2

Re: Very slow backfilling
by Joffrey 07 Mar '23

07 Mar '23

My Ceph Version is 17.2.5 and all configuration about osd_scrub* are defaults. I tried some updates on osd-max-backfills but no change. I have many HDD with NVME for db and all are connected in a 25G network. Yes, it's the same PG since 4 days. I got a failure on a HDD and get many days of recovery+backfilling last 2 weeks. Perhaps the 'not in time' warning is related to this. 'Jof Le jeu. 2 mars 2023 à 14:25, Anthony D'Atri <aad(a)dreamsnake.net> a écrit : > Run `ceph health detail`. > > Is it the same PG backfilling for a long time, or a different one over > time? > > That it’s remapped makes me think that what you’re seeing is the balancer > doing its job. > > As far as the scrubbing, do you limit the times when scrubbing can happen? > Are these HDDs? EC? > > > On Mar 2, 2023, at 07:20, Joffrey <joff.au(a)gmail.com> wrote: > > > > Hi, > > > > I have many 'not {deep-}scrubbed in time' and a1 PG remapped+backfilling > > and I don't understand why this backfilling is taking so long. > > > > root@hbgt-ceph1-mon3:/# ceph -s > > cluster: > > id: c300532c-51fa-11ec-9a41-0050569c3b55 > > health: HEALTH_WARN > > 15 pgs not deep-scrubbed in time > > 13 pgs not scrubbed in time > > > > services: > > mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3 > > (age 36h) > > mgr: hbgt-ceph1-mon2.nteihj(active, since 2d), standbys: > > hbgt-ceph1-mon1.thrnnu, hbgt-ceph1-mon3.gmfzqm > > osd: 60 osds: 60 up (since 13h), 60 in (since 13h); 1 remapped pgs > > rgw: 3 daemons active (3 hosts, 2 zones) > > > > data: > > pools: 13 pools, 289 pgs > > objects: 67.74M objects, 127 TiB > > usage: 272 TiB used, 769 TiB / 1.0 PiB avail > > pgs: 288 active+clean > > 1 active+remapped+backfilling > > > > io: > > client: 3.3 KiB/s rd, 1.5 MiB/s wr, 3 op/s rd, 8 op/s wr > > recovery: 790 KiB/s, 0 objects/s > > > > > > What can I do to understand this slow recovery (is it the backfill > action ?) > > > > Thanks you > > > > 'Jof > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

3 8

Theory about min_size and its implications
by stefan.pinter＠bearingpoint.com 06 Mar '23

06 Mar '23

Hi! it is unclear for us what min_size means besides what it does. i hope someone can clear this up :) scenario: size is 3 and min_size is 2 2 rooms with 100 OSDs each and this crush rule "op": "take", "item": -10, "item_name": "default" "op": "choose_firstn", "num": 2, "type": "room" "op": "chooseleaf_firstn", "num": 2, "type": "host" "op": "emit" so if one room goes down/offline, around 50% of the PGs would be left with only 1 replica making them read-only. if we'd set min_size to 1 and one room goes down, user wouldn't still be able to access all PGs - but what is the problem with only one active PG? someone pointed out "split brain" but I am unsure about this. i think what happens in the worst case is this: only 1 PG is available, client writes changes to this PG, the disk of this 1 PG dies as well - so i guess we'd need to restore the data from the 2 offline PGs in the room that is down and we would have lots of trouble with restoring and also with data inconsistency, right? thank you!

5 8

upgrade problem from 1.6 to 1.7 related with osd
by Oğuz Yarımtepe 06 Mar '23

06 Mar '23

Hi, I have a cluster with rook operator running the ceph version 1.6 and upgraded first rook operator and then the ceph cluster definition. Everything was fine, every component except from osds are upgraded. Below is the reason of OSD not being upgraded: not updating OSD 1 on node "some-node-name". node no longer exists in the storage spec. if the user wishes to remove OSDs from the node, they must do so manually. Rook will not remove OSDs from nodes that are removed from the storage spec in order to prevent accidental data loss Any idea or anyone had seen it before? Regards. -- Oğuz Yarımtepe http://about.me/oguzy

1 0

mds readonly, mds all down
by kreept.sama＠gmail.com 06 Mar '23

06 Mar '23

Hello. We trying to resolve some issue with ceph. Our openshift cluster is blocked and we tried do almost all. Actual state is: MDS_ALL_DOWN: 1 filesystem is offline MDS_DAMAGE: 1 mds daemon damaged FS_DEGRADED: 1 filesystem is degraded MON_DISK_LOW: mon be is low on available space RECENT_CRASH: 1 daemons have recently crashed We try to perform cephfs-journal-tool --rank=gml-okd-cephfs:all event recover_dentries summary cephfs-journal-tool --rank=gml-okd-cephfs:all journal reset cephfs-table-tool gml-okd-cephfs:all reset session ceph mds repaired 0 ceph config rm mds mds_verify_scatter ceph config rm mds mds_debug_scatterstat ceph tell gml-okd-cephfs scrub start / recursive repair force After these commands, mds rises but an error appears: MDS_READ_ONLY: 1 MDSs are read only We also tried to create new fs with new metadata pool, delete and recreate old fs with same name with old\new metadatapool. We got rid of the errors, but the Openshift cluster did not want to work with the old persistence volumes. The pods wrote an error that they could not find it, while it was present and moreover, this volume was associated with pvc. Now we have rolled back the cluster and are trying to remove the mds error. Any ideas what to try? Thanks

2 2

orchestrator issues on ceph 16.2.9
by Adrian Nicolae 06 Mar '23

06 Mar '23

Hi, I have some orchestrator issues on our cluster running 16.2.9 with rgw only services. We first noticed these issues a few weeks ago when adding new hosts to the cluster - the orch was not detecting the new drives to build the osd containers for them. Debugging the mgr logs, I noticed that the mgr was crashing due to the dashboard module. I disabled the dashboard module and the new drives were detected and added to the cluster. Now we have other similar issues : we have a failed drive. The failure was detected , the osd was marked as down and the rebalancing is finished. I want to remove the failed osd from the cluster but it looks like the orch is not working : - I launched the osd removal with 'ceph orch osd rm 92 --force' where 92 is the osd id in question - I checked the progress but nothing happens even after a few days : ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT *92 node10 started 0 False True False* - the osd process is stopped on that host and from the orch side I can see this : ceph orch ps --daemon_type osd --daemon_id 92 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID osd.92 node10 error 11h ago 4w - 4096M <unknown> <unknown> - I have the same long refresh interval on other osds as well. I know it should be 10 minutes or so : NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.93 node09 running (4w) 11h ago 4w 5573M 4096M 16.2.9 3520ead5eb19 d2f658e9e37b osd: *116 osds: 115 up* (since 11d), 115 in (since 11d) 90 hdd 16.37109 1.00000 16 TiB 4.0 TiB 4.0 TiB 0 B 17 GiB 12 TiB 24.66 0.97 146 up * 92 hdd 0 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down* 94 hdd 16.37109 1.00000 16 TiB 4.0 TiB 4.0 TiB 0 B 17 GiB 12 TiB 24.66 0.97 146 up - I activated debug 20 on the mgr but I can't see any errors or other clues regarding the osd removal. I also switched to the standby manager with 'ceph mgr fail'. The mgr switch works but still nothing happens - It's not only the osd removal thing. I also tried to deploy new rgw services by applying rgw labels on 2 new hosts , we have specs for building rgw containers when detecting the label. Again, nothing happens. I'm planning to upgrade to 16.2.11 to see if this solves the issues but I'm not very confident, I didn't see anything regarding this in the changelogs. Is there anything else I can try to debug this issue ? Thanks.

2 1

Creating a role for allowing users to set quota on CpehFS pools
by ananda a 06 Mar '23

06 Mar '23

Hello, How do I create a role to allow users to set quota on specific Ceph FS pools. ? Thank you

1 0

deep scrub and long backfilling
by xadhoom76＠gmail.com 05 Mar '23

05 Mar '23

ceph 16.2.11, is safe to enable scrub and deep scrub during backfilling ? I have log recovery-backfilling due to a new crushmap , backfilling is going slow and deep scrub interval as expired so I have many pgs not deep-scrubbed in time. Best regards Alessandro

4 3