March 2023 - ceph-users - lists.ceph.io

Difficulty with rbd-mirror on different networks.

by Murilo Morais

Good evening everyone. I'm having trouble with rbd-mirror. In test environment I have the following scenario: DC1: public_network: 172.20.0.0/24, 192.168.0.0/24 --mon-ip 172.20.0.1 ip: 192.168.0.1 DC2: public_network: 172.21.0.0/24, 192.168.0.0/24 --mon-ip 172.21.0.1 ip 192.168.0.2 If I add the peer as being from the 192.168.0.0/24 network, the mirror does not work, both machines see each other, if I add a machine within the same block as the other (the first public_network network of each), that is, DC1 with 172.21.0.2/24 and DC2 with 172.20.0.2/24, the mirror works. Why can't I make the mirror work on the 192.168.0.0/24 network? Thank you very much in advance.

1 year, 1 month

1
0
0 0

Dashboard for Object Servers using wrong hostname

by Wyll Ingersoll

I have an ochestrated (cephadm) ceph cluster (16.2.11) with 2 radosgw services on 2 separate hosts without HA (i.e. no ingress/haproxy in front). Both of the rgw servers use SSL and have a properly signed certificate. We can access them with standard S3 tools like s3cmd, cyberduck, etc. The problem seems to be that the the Ceph mgr dashboard fails to access the RGW API because it uses the shortname "gw01" instead of the FQDN "gw01.domain.com" when forming the S3 signature which makes the S3 signature check fail and we get the following error: Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","RequestId":"tx00000521ceca28974e94b-006408e' b'f93-454bbb4e-default","HostId":"454bbb4e-default-default"}') It seems that the ceph mgr (which we have restarted several times) uses just the short hostname from the inventory and I don't see how to tell it to use the FQDN. Neither is it possible to configure the RGW to listen on an alternate non-SSL port on the cluster private network since the service spec for RGW only allows to set the rgw_frontend_port and rgw_frontend_type, but not the full frontend spec (which would allow for multiple listeners). When we did have HA (haproxy) ingress configured, we ran into issues with the user clients getting lots of 503 errors due to some interaction between the RGW and the haproxy so we gave up on that config and now talk directly to the RGW over SSL which is working well. Any suggestions? thanks, Wyllys Ingersoll

1 year, 1 month

1
0
0 0

Problem with cephadm and deploying 4 ODSs on nvme Storage

by claas.goltz＠contact-software.com

Hi Community, currently i’m installing an nvme only storage cluster with cephadm from scratch (v.17.2.5). Everything works fine. Each of my nodes (6) has 3 enterprise nvme’s with 7TB capacity. At the beginning I only installed one OSD per nvme, now I want to use four instead of one but I’m struggling with that. Frist of all I set the following option in my cluster: ceph orch apply osd --all-available-devices --unmanaged=true As I understand this option should prevent cephadm to automatically fetch new, available disks and deploying OSD daemons. But that seems not to work. If I delete and purge my OSD and zapping the disk with ceph orch device zap ceph-nvme01 /dev/nvme2n1 –force the disk came available for the cluster and seconds later it deploys the same OSD ID than it has before. I checked that the old OSD was completely removed and the docker container was not started. My next try was to set: ceph orch host label add ceph-nvme01 _no_schedule purge the OSD zapping the disk and following: ceph orch daemon add osd ceph-nvme01:/dev/nvme2n1,osds_per_device=4 removing the _no_schedule flag and again: the old OSD was recreated and not 4. So where is my mistake? Thank you!

1 year, 1 month

3
3
0 0

Upgrade problem from 1.6 to 1.7

by Oğuz Yarımtepe

Hi, I have a cluster with rook operator running the ceph version 1.6 and upgraded first rook operator and then the ceph cluster definition. Everything was fine, every component except from osds are upgraded. Below is the reason of OSD not being upgraded: not updating OSD 1 on node "some-node-name". node no longer exists in the storage spec. if the user wishes to remove OSDs from the node, they must do so manually. Rook will not remove OSDs from nodes that are removed from the storage spec in order to prevent accidental data loss Any idea or anyone had seen it before? Regards.

1 year, 1 month

2
1
0 0

s3 lock api get-object-retention

by garcetto

good morning, i am using ver 17 but cannot have s3 lock get-object-retention working: $ aws --profile=user01 --endpoint-url=http://x.x.x.x:80 s3api list-object-versions --bucket demo-compliance { "ETag": "\"b8d6acf7d330d241a8ef851694365b94\"", "Size": 32, "StorageClass": "STANDARD", "Key": "testme.txt", "VersionId": "M632.ygI0AaMm8PXYXHrPNl3Y2aLOU6", "IsLatest": true, "LastModified": "2023-03-08T10:58:00.257Z", "Owner": { "DisplayName": "user01", "ID": "user01" } $ aws --profile=user01 --endpoint-url=http://x.x.x.x:80 s3api get-object-retention --bucket demo-compliance --key "testme.txt" An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectRetention operation: Unknown thank you.

1 year, 1 month

1
0
0 0

user and bucket not sync ( permission denied )

by Guillaume Morin

Hello, i have an issue about my multisite configuration. pacific 16.2.9 My problem: i have a permission denied on the the master zone when i use the command below. $ radosgw-admin sync status realm 8df19226-a200-48fa-bd43-1491d32c636c (myrealm) zonegroup 29592d75-224d-49b6-bc36-2703efa4f67f (myzonegroup) zone 6cce41f3-a54b-47c2-981f-3b56ca0a4489 (myzone) metadata sync no sync (zone is master) 2023-03-07T22:31:16.466+0100 7f96a3e7a840 0 ERROR: failed to fetch datalog info data sync source: f2b20676-2672-4a92-a7ee-f3eb2efb12c6 (mysecondaryzone) failed to retrieve sync info: (13) Permission denied because on secondary zone (read only) , i see a 403 error about the permission denied from the master node 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 ====== starting new request req=0x7f1fd418c620 ===== 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 req 2604939314198041770 0.000000000s op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 ====== req done req=0x7f1fd418c620 op status=0 http_status=403 latency=0.000000000s ====== 2023-03-07T00:00:53.309+0100 7f1ec8f21700 1 beast: 0x7f1fd418c620: 10......... - - [07/Mar/2023:00:00:53.309 +0100] "POST /admin/realm/period?period=395f9f13-d941-4ccf-a0cf-6c5d6d6579c2&epoch=76&rgwx-zonegroup=29592d75-224d-49b6-bc36-2703efa4f67f HTTP/1.1" 403 194 - - - latency=0.000000000s 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 ====== starting new request req=0x7f1fd4411620 ===== 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 req 7374970752399537975 0.000000000s op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 ====== req done req=0x7f1fd4411620 op status=0 http_status=403 latency=0.000000000s ====== 2023-03-07T00:00:53.441+0100 7f1e7e68c700 1 beast: 0x7f1fd4411620: 10......... - - [07/Mar/2023:00:00:53.441 +0100] "POST /admin/log?type=data&notify&source-zone=6cce41f3-a54b-47c2-981f-3b56ca0a4489&rgwx-zonegroup=29592d75-224d-49b6-bc36-2703efa4f67f HTTP/1.1" 403 194 - - - latency=0.000000000s No issue when i use the command to check sync on secondary zone I don't understand because on secondary zone, pull realm and period with a user with flag system and admin works, the sync works for objects but not for users and buckets. When i list user and bucket on secondary zone, there are nothing but i have my objects on pool bucket.data !! i think the 403 was due because my user with flag system doesn't exist on secondary zone but i don't understand why user and bucket are not syncronized ??!! Access key and secret key are set on master zone and secondary zone, endpoint also I have an other cluster with a similary configuration and i don't have any issue Can someone help me ? Sorry for my english Regards Guillaume

1 year, 1 month

1
1
0 0

Issue upgrading 17.2.0 to 17.2.5

by aellahib＠gmail.com

I initially ran the upgrade fine but it failed @ around 40/100 on an osd, so after waiting for along time i thought I'd try restarting it and then restarting the upgrade. I am stuck with the below debug error, I have tested docker pull from other servers and they dont fail for the ceph images but on ceph it does. If i even try to redeploy or add or remove mon damons for example it comes up with the same error related to the images. The error that ceph is giving me is: 2023-03-02T07:22:45.063976-0700 mgr.mgr-node.idvkbw [DBG] _run_cephadm : args = [] 2023-03-02T07:22:45.070342-0700 mgr.mgr-node.idvkbw [DBG] args: --image stop --no-container-init pull 2023-03-02T07:22:45.081086-0700 mgr.mgr-node.idvkbw [DBG] Running command: which python3 2023-03-02T07:22:45.180052-0700 mgr.mgr-node.idvkbw [DBG] Running command: /usr/bin/python3 /var/lib/ceph/5058e342-dac7-11ec-ada3-01065e90228d/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e --image stop --no-container-init pull 2023-03-02T07:22:46.500561-0700 mgr.mgr-node.idvkbw [DBG] code: 1 2023-03-02T07:22:46.500787-0700 mgr.mgr-node.idvkbw [DBG] err: Pulling container image stop... Non-zero exit code 1 from /usr/bin/docker pull stop /usr/bin/docker: stdout Using default tag: latest /usr/bin/docker: stderr Error response from daemon: pull access denied for stop, repository does not exist or may require 'docker login': denied: requested access to the resource is denied ERROR: Failed command: /usr/bin/docker pull stop

1 year, 1 month

4
8
0 0

Creating a role for quota management

by anantha.adiga＠intel.com

Hello, Can you provide details on how to create a role for allowing a set of users to set quota on CephFS pools ?

1 year, 1 month

3
2
0 0

Re: Very slow backfilling

by Joffrey

My Ceph Version is 17.2.5 and all configuration about osd_scrub* are defaults. I tried some updates on osd-max-backfills but no change. I have many HDD with NVME for db and all are connected in a 25G network. Yes, it's the same PG since 4 days. I got a failure on a HDD and get many days of recovery+backfilling last 2 weeks. Perhaps the 'not in time' warning is related to this. 'Jof Le jeu. 2 mars 2023 à 14:25, Anthony D'Atri <aad(a)dreamsnake.net> a écrit : > Run `ceph health detail`. > > Is it the same PG backfilling for a long time, or a different one over > time? > > That it’s remapped makes me think that what you’re seeing is the balancer > doing its job. > > As far as the scrubbing, do you limit the times when scrubbing can happen? > Are these HDDs? EC? > > > On Mar 2, 2023, at 07:20, Joffrey <joff.au(a)gmail.com> wrote: > > > > Hi, > > > > I have many 'not {deep-}scrubbed in time' and a1 PG remapped+backfilling > > and I don't understand why this backfilling is taking so long. > > > > root@hbgt-ceph1-mon3:/# ceph -s > > cluster: > > id: c300532c-51fa-11ec-9a41-0050569c3b55 > > health: HEALTH_WARN > > 15 pgs not deep-scrubbed in time > > 13 pgs not scrubbed in time > > > > services: > > mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3 > > (age 36h) > > mgr: hbgt-ceph1-mon2.nteihj(active, since 2d), standbys: > > hbgt-ceph1-mon1.thrnnu, hbgt-ceph1-mon3.gmfzqm > > osd: 60 osds: 60 up (since 13h), 60 in (since 13h); 1 remapped pgs > > rgw: 3 daemons active (3 hosts, 2 zones) > > > > data: > > pools: 13 pools, 289 pgs > > objects: 67.74M objects, 127 TiB > > usage: 272 TiB used, 769 TiB / 1.0 PiB avail > > pgs: 288 active+clean > > 1 active+remapped+backfilling > > > > io: > > client: 3.3 KiB/s rd, 1.5 MiB/s wr, 3 op/s rd, 8 op/s wr > > recovery: 790 KiB/s, 0 objects/s > > > > > > What can I do to understand this slow recovery (is it the backfill > action ?) > > > > Thanks you > > > > 'Jof > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 year, 1 month

3
8
0 0

Theory about min_size and its implications

by stefan.pinter＠bearingpoint.com

Hi! it is unclear for us what min_size means besides what it does. i hope someone can clear this up :) scenario: size is 3 and min_size is 2 2 rooms with 100 OSDs each and this crush rule "op": "take", "item": -10, "item_name": "default" "op": "choose_firstn", "num": 2, "type": "room" "op": "chooseleaf_firstn", "num": 2, "type": "host" "op": "emit" so if one room goes down/offline, around 50% of the PGs would be left with only 1 replica making them read-only. if we'd set min_size to 1 and one room goes down, user wouldn't still be able to access all PGs - but what is the problem with only one active PG? someone pointed out "split brain" but I am unsure about this. i think what happens in the worst case is this: only 1 PG is available, client writes changes to this PG, the disk of this 1 PG dies as well - so i guess we'd need to restore the data from the 2 offline PGs in the room that is down and we would have lots of trouble with restoring and also with data inconsistency, right? thank you!

1 year, 1 month

5
8
0 0

2024

2023

2022

2021

2020

2019

ceph-users March 2023