- ceph-users - lists.ceph.io

Question about RGW S3 Select
by Dave S 05 Oct '23

05 Oct '23

Hi Everyone, I've been trying to get S3 Select working on our system and whenever I send a query I get the following in the Payload (Result 200 from RGW): # aws --endpoint-url http://cephtest1 s3api select-object-content --bucket test1 --expression-type SQL --input-serialization '{"CSV": {"Fieldter": "\"" , "RecordDelimiter" : "\n" , "QuoteEscapeCharacter" : "\\" , "FileHeaderInfo": "USE" }, "CompressionType": "NONE"}' --output-serialization '{"CSV": {"FieldDelimiter": ":", "RecordDelimiter":"\t", "QuoteFields": "ALWAYS"}}' --key sample_data.csv --expression 'SELECT * from s3object' /dev/stderr <Payload> <Records> <Payload> failure -->SELECT * from s3object<--- </Payload></Records></Payload> I also get the same behaviour when accessing via boto3/python. The same command/code works when accessing other S3 services. Am I missing some config or something? The test1/sample_data.csv file is there and the account is able to get the sample_data.csv data. I've tried uploading a version with unix endings and Dos endings for the data file. This is the data file from the AWS example. We're running Pacific and the RGWs are deployed as podman containers on Rocky 8.8: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Thanks, -Dave

2 1

Calling all Ceph users and developers! Submit a topic for the next User + Dev Meeting!
by Laura Flores 04 Oct '23

04 Oct '23

Hi Ceph users and developers, We are gearing up for the next User + Developer Monthly Meeting, happening October 19th at 10am EST. If you are interested in being a guest speaker, you are invited to submit a focus topic to this Google form: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… Examples of what we're looking for in a focus topic include: - Feature requests / RFEs - User feedback - Knowledge sharing (upgrades, workloads, etc.) - Ideas for longterm improvement (user-facing) - Share the use case of your cluster Any Ceph user or developer is eligible to submit! Reach out to me with any questions. - Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Issue with radosgw-admin reshard when bucket belongs to user with tenant on ceph quincy (17.2.6)
by christoph.weber+cephmailinglist＠xpecto.com 04 Oct '23

04 Oct '23

Hi everybody, I tried to reshard a bucket belonging to the tenant "test-tenant", but got an "No such file or directory" error. $ radosgw-admin reshard add --bucket test-tenant/test-bucket --num-shards 40 $ radosgw-admin reshard process 2023-10-04T12:12:52.470+0200 7f654237afc0 0 process_single_logshard: Error during resharding bucket test-tenant/test-bucket:(2) No such file or directory $ radosgw-admin reshard list [ { "time": "2023-10-04T10:12:46.528741Z", "tenant": "", "bucket_name": "test-tenant/test-bucket", "bucket_id": "43e570fd-7573-403f-ab86-a75e12e60146.24142.3", "new_instance_id": "", "old_num_shards": 23, "tentative_new_num_shards": 40 } ] (I also tried --bucket test-bucket) I then tried to remove the incomplete reshard-job $ radosgw-admin reshard cancel --bucket test-tenant/test-bucket $ radosgw-admin reshard list [] $ radosgw-admin reshard process 2023-10-04T12:33:04.251+0200 7fb728e16fc0 0 INFO: RGWReshardLock::lock found lock on reshard.0000000000 to be held by another RGW process; skipping for now With $ radosgw-admin lc reshard fix --bucket test-tenant/test-bucket and restarting the rgw containers, I could get rid of the reshard.0000000000 lock. Unfortunately updating to the latest 17.2 version did not help. $ ceph orch upgrade start --ceph-version 17.2.6 $ ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) But after transferring the bucket to a different RGW user without tenant, resharding worked fine: $ reshard add --bucket test-bucket-without-tenant-user --num-shards 40 $ radosgw-admin reshard process 2023-10-04T12:44:43.599+0200 7f75c908ffc0 1 execute INFO: reshard of bucket "test-bucket-without-tenant-user" from "test-bucket-without-tenant-user:43e570fd-7573-403f-ab86-a75e12e60146.74748.1" to "test-bucket-without-tenant-user:43e570fd-7573-403f-ab86-a75e12e60146.74819.1" completed successfully This solved my problem, but I wanted to report it, as it may affect other installations too. Is this problem known? I could not find information for this.

1 0

RGW multisite - requesting help for fixing error_code: 125
by Jayanth Reddy 04 Oct '23

04 Oct '23

Hello Users, We're running 2 Ceph clusters with v17.2.6 and noticing the error message in # radosgw-admin sync error list *"message": "failed to sync bucket instance: (125) Operation canceled"* We've the output as below, [ { "shard_id": 0, "entries": [ { "id": "1_1690711173.869335_133603.1", "section": "data", "name": "b1:d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1:2897", "timestamp": "2023-07-30T09:59:33.869335Z", "info": { "source_zone": "d09d3d16-8601-448b-bf3d-609b8a29647d", "error_code": 125, "message": "failed to sync bucket instance: (125) Operation canceled" } }, { "id": "1_1690711175.505687_133683.1", "section": "data", "name": "b1:d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1:1719", "timestamp": "2023-07-30T09:59:35.505687Z", "info": { "source_zone": "d09d3d16-8601-448b-bf3d-609b8a29647d", "error_code": 125, "message": "failed to sync bucket instance: (125) Operation canceled" } }, and with around 26236 errors # radosgw-admin sync error list | grep -i "(125) Operation canceled" | wc -l 26236 I'm trying to fix these by rewriting objects but I'm having trouble finding the exact object names and the procedure. Any help is really appreciated! Thanks, Jayanth

2 1

ceph luminous client connect to ceph reef always permission denied
by Pureewat Kaewpoi 04 Oct '23

04 Oct '23

Hi All ! We have a new installed cluster with ceph reef. but our old client still using ceph luminous. The problem is when using any command to ceph cluster It will hang and no any output. This is a output from command ceph osd pool ls --debug-ms 1 2023-10-02 23:35:22.727089 7fc93807c700 1 Processor -- start 2023-10-02 23:35:22.729256 7fc93807c700 1 -- - start start 2023-10-02 23:35:22.729790 7fc93807c700 1 -- - --> MON-1:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc930174cb0 con 0 2023-10-02 23:35:22.730724 7fc935e72700 1 -- CLIENT:0/187462963 learned_addr learned my addr CLIENT:0/187462963 2023-10-02 23:35:22.732091 7fc927fff700 1 -- CLIENT:0/187462963 <== mon.0 MON-1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2762451217 0 0) 0x7fc920002310 con 0x7fc93017d0f0 2023-10-02 23:35:22.732228 7fc927fff700 1 -- CLIENT:0/187462963 --> MON-1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x7fc914000fc0 con 0 2023-10-02 23:35:22.733237 7fc927fff700 1 -- CLIENT:0/187462963 <== mon.0 MON-1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3693167043 0 0) 0x7fc920002830 con 0x7fc93017d0f0 2023-10-02 23:35:22.733428 7fc927fff700 1 -- CLIENT:0/187462963 --> MON-1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc914002e10 con 0 2023-10-02 23:35:22.733451 7fc927fff700 1 -- CLIENT:0/187462963 <== mon.0 MON-1:6789/0 3 ==== mon_map magic: 0 v1 ==== 532+0+0 (3038142027 0 0) 0x7fc920000e50 con 0x7fc93017d0f0 2023-10-02 23:35:22.734365 7fc927fff700 1 -- CLIENT:0/187462963 <== mon.0 MON-1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (3147563293 0 0) 0x7fc920001640 con 0x7fc93017d0f0 2023-10-02 23:35:22.734597 7fc927fff700 1 -- CLIENT:0/187462963 --> MON-1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- 0x7fc9301755e0 con 0 2023-10-02 23:35:22.734678 7fc93807c700 1 -- CLIENT:0/187462963 --> MON-1:6789/0 -- mon_subscribe({mgrmap=0+}) v2 -- 0x7fc930180750 con 0 2023-10-02 23:35:22.734805 7fc93807c700 1 -- CLIENT:0/187462963 --> MON-1:6789/0 -- mon_subscribe({osdmap=0}) v2 -- 0x7fc930180f00 con 0 2023-10-02 23:35:22.734891 7fc935e72700 1 -- CLIENT:0/187462963 >> MON-1:6789/0 conn(0x7fc93017d0f0 :-1 s=STATE_OPEN pgs=754 cs=1 l=1).read_bulk peer close file descriptor 13 2023-10-02 23:35:22.734917 7fc935e72700 1 -- CLIENT:0/187462963 >> MON-1:6789/0 conn(0x7fc93017d0f0 :-1 s=STATE_OPEN pgs=754 cs=1 l=1).read_until read failed 2023-10-02 23:35:22.734922 7fc935e72700 1 -- CLIENT:0/187462963 >> MON-1:6789/0 conn(0x7fc93017d0f0 :-1 s=STATE_OPEN pgs=754 cs=1 l=1).process read tag failed 2023-10-02 23:35:22.734926 7fc935e72700 1 -- CLIENT:0/187462963 >> MON-1:6789/0 conn(0x7fc93017d0f0 :-1 s=STATE_OPEN pgs=754 cs=1 l=1).fault on lossy channel, failing 2023-10-02 23:35:22.734966 7fc927fff700 1 -- CLIENT:0/187462963 >> MON-1:6789/0 conn(0x7fc93017d0f0 :-1 s=STATE_CLOSED pgs=754 cs=1 l=1).mark_down 2023-10-02 23:35:22.735062 7fc927fff700 1 -- CLIENT:0/187462963 --> MON-2:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fc914005580 con 0 2023-10-02 23:35:22.735077 7fc927fff700 1 -- CLIENT:0/187462963 --> MON-3:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fc914005910 con 0 2023-10-02 23:35:22.737246 7fc927fff700 1 -- CLIENT:0/187462963 <== mon.2 MON-3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2138308960 0 0) 0x7fc920002fd0 con 0x7fc91400b0c0 2023-10-02 23:35:22.737443 7fc927fff700 1 -- CLIENT:0/187462963 --> MON-3:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x7fc914014f10 con 0 2023-10-02 23:35:22.737765 7fc927fff700 1 -- CLIENT:0/187462963 <== mon.1 MON-2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3855879565 0 0) 0x7fc928002390 con 0x7fc91400f730 2023-10-02 23:35:22.737799 7fc927fff700 1 -- CLIENT:0/187462963 --> MON-2:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x7fc914015850 con 0 2023-10-02 23:35:22.737966 7fc927fff700 1 -- CLIENT:0/187462963 <== mon.2 MON-3:6789/0 2 ==== auth_reply(proto 2 -13 (13) Permission denied) v1 ==== 24+0+0 (2583972696 0 0) 0x7fc920003240 con 0x7fc91400b0c0 2023-10-02 23:35:22.737981 7fc927fff700 1 -- CLIENT:0/187462963 >> MON-3:6789/0 conn(0x7fc91400b0c0 :-1 s=STATE_OPEN pgs=464 cs=1 l=1).mark_down 2023-10-02 23:35:22.738096 7fc927fff700 1 -- CLIENT:0/187462963 <== mon.1 MON-2:6789/0 2 ==== auth_reply(proto 2 -13 (13) Permission denied) v1 ==== 24+0+0 (2583972696 0 0) 0x7fc928002650 con 0x7fc91400f730 2023-10-02 23:35:22.738110 7fc927fff700 1 -- CLIENT:0/187462963 >> MON-2:6789/0 conn(0x7fc91400f730 :-1 s=STATE_OPEN pgs=344 cs=1 l=1).mark_down By the way I have using same keyring with ceph nautilus client it work well without any problem. What should I do next ? Where to debug or where to fix this issue. Thanks

2 1

Balancer blocked as autoscaler not acting on scaling change
by bc10＠sanger.ac.uk 04 Oct '23

04 Oct '23

Hi Folks, We are currently running with one nearfull OSD and 15 nearfull pools. The most full OSD is about 86% full but the average is 58% full. However, the balancer is skipping a pool on which the autoscaler is trying to complete a pg_num reduction from 131,072 to 32,768 (default.rgw.buckets.data pool). However, the autoscaler has been working on this for the last 20 days, it works through a list of objects that are misplaced but when it gets close to the end, more objects get added to the list. This morning I observed the list get down to c. 7,000 objects misplaced with 2 PGs active+remapped+backfilling, one PG completed the backfilling then the list shot up to c. 70,000 objects misplaced with 3 PGs active+remapped+backfilling. Has anyone come across this behaviour before? If so, what was your remediation? Thanks in advance for sharing. Bruno Cluster details: 3,068 OSDs when all running, c. 60 per storage node OS: Ubuntu 20.04 Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive Use case: S3 storage and OpenStack backend, all pools three-way replicated

5 4

Re: VM hangs when overwriting a file on erasure coded RBD
by peter.linder＠fiberdirekt.se 04 Oct '23

04 Oct '23

Yes, this is all set up. It was working fine until after the problem with the osd host that lost the cluster/sync network occured. There are a few other VMs that keep running along fine without this issue. I've restarted the problematic VM without success (that is, creating a file works, but overwriting it still hangs right away). fsck runs fine so reading the whole image works. I'm kind of stumped as to what can cause this. Because of the lengthy recovery, and then pg autoscaler currenty doing things there are currently lots of PGs that haven't been scrubbed, but I doubt that is an issue here. Den 2023-09-29 kl. 18:52, skrev Anthony D'Atri: > EC for RBD wasn't possible until Luminous IIRC, so I had to ask. You have a replicated metadata pool defined? Does proxmox know that this is an EC pool? When connecting it needs to know both the metata and data pools. > >> On Sep 29, 2023, at 12:49, peter.linder(a)fiberdirekt.se wrote: >> >> (sorry for duplicate emails) >> >> This turns out to be a good question actually. >> >> The cluster is running Quincy, 17.2.6. >> >> The compute node that is running the VM is proxmox, version 7.4-3. Supposedly this is fairly new, but the version of librbd1 claims to be 14.2.21 when I check with "apt list". We are not using proxmox's own ceph cluster release. However we haven't had any issues with this setup before, but we haven't been using neither erasure coded pools nor had the node-half-dead problem for such a long time. >> >> The VM is configured using proxmox which is not libvirt but similar, and krbd is not enabled. I don't know for sure if proxmox has its own librbd linked in qemu/kvm. >> >> "ceph features" looks like this: >> >> { >> "mon": [ >> { >> "features": "0x3f01cfbf7ffdffff", >> "release": "luminous", >> "num": 5 >> } >> ], >> "osd": [ >> { >> "features": "0x3f01cfbf7ffdffff", >> "release": "luminous", >> "num": 24 >> } >> ], >> "client": [ >> { >> "features": "0x3f01cfb87fecffff", >> "release": "luminous", >> "num": 4 >> }, >> { >> "features": "0x3f01cfbf7ffdffff", >> "release": "luminous", >> "num": 12 >> } >> ], >> "mgr": [ >> { >> "features": "0x3f01cfbf7ffdffff", >> "release": "luminous", >> "num": 2 >> } >> ] >> } >> >> Regards, >> >> Peter >> >> >> Den 2023-09-29 kl. 17:55, skrev Anthony D'Atri: >>> Which Ceph releases are installed on the VM and the back end? Is the VM using librbd through libvirt, or krbd? >>> >>>> On Sep 29, 2023, at 09:09, Peter Linder <peter.linder(a)fiberdirekt.se> wrote: >>>> >>>> Dear all, >>>> >>>> I have a problem that after an OSD host lost connection to the sync/cluster rear network for many hours (the public network was online), a test VM using RBD cant overwrite its files. I can create a new file inside it just fine, but not overwrite it, the process just hangs. >>>> >>>> The VM's disk is on an erasure coded data pool and a replicated pool in front of it. EC overwrites is on for the pool. >>>> >>>> The custer consists of 5 hosts and 4 OSDs on each, and separate hosts for compute. There is a public and separate cluster network, separated. In this case, the AOC cable to the cluster network went link down on a host and it had to be replaced and the host was rebooted. Recovery took about a week to complete. The host was half-down for about 12 hours like this. >>>> >>>> I have some other VMs as well with images in the same pool (totally 4), and they seem to work fine, it is just this one that cant overwrite. >>>> >>>> I'm thinking there is somehow something wrong with just this image? >>>> >>>> Regards, >>>> >>>> Peter >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Slow recovery and inaccurate recovery figures since Quincy upgrade
by Iain Stott 04 Oct '23

04 Oct '23

Hi, We run 3 production clusters in a multi-site setup. They were deployed with Ceph-Ansible but recently switched to cephadm while on the Pacific release. Shortly after migrating to cephadm they were upgraded to Quincy. Since moving to Quincy, the recovery on one of the replica sites has tanked quite severely. We use 8Tb SAS HDDs for OSDs with WAL and DB on nVME drives. Before the upgrade it would take 1-2 days to resilver a OSD, but recently we replaced a drive and it took 6 days to resilver. Nothing else has changed with the configuration of the cluster, and the other 2 are performing recovery fine as far as we can see. Does anyone have any ideas of what could be the issue here or anywhere we can check what is going on?? Also while recovering, we've noticed inaccurate recovery information in ceph -s. We've seen the recovery section of ceph -s reporting its running at less than 10 keys/s, but looking at the Degraded data redundancy we see this drop multiple 100s of keys per second. Does anyone have any advice they can offer on this too please?? Cheers Iain [ceph: root@gb4-li-cephgw-001 /]# ceph -s; sleep 3; ceph -s cluster: id: 6dabcf41-90d7-4e90-b259-1cc0bf298052 health: HEALTH_WARN noout,norebalance flag(s) set Degraded data redundancy: 59610/371260626 objects degraded (0.016%), 1 pg degraded, 1 pg undersized services: mon: 3 daemons, quorum gb4-li-cephgw-001,gb4-li-cephgw-002,gb4-li-cephgw-003 (age 2h) mgr: gb4-li-cephgw-003(active, since 2h), standbys: gb4-li-cephgw-002.iqmxgu, gb4-li-cephgw-001 osd: 72 osds: 72 up (since 39m), 72 in (since 3d); 1 remapped pgs flags noout,norebalance rgw: 3 daemons active (3 hosts, 1 zones) data: pools: 11 pools, 1457 pgs objects: 63.76M objects, 173 TiB usage: 251 TiB used, 275 TiB / 526 TiB avail pgs: 59610/371260626 objects degraded (0.016%) 1452 active+clean 4 active+clean+scrubbing+deep 1 active+undersized+degraded+remapped+backfilling io: client: 249 MiB/s rd, 611 KiB/s wr, 355 op/s rd, 402 op/s wr recovery: 13 KiB/s, 3 keys/s, 2 objects/s progress: Global Recovery Event (39m) [===========================.] (remaining: 1s) cluster: id: 6dabcf41-90d7-4e90-b259-1cc0bf298052 health: HEALTH_WARN noout,norebalance flag(s) set Degraded data redundancy: 59116/371260644 objects degraded (0.016%), 1 pg degraded, 1 pg undersized services: mon: 3 daemons, quorum gb4-li-cephgw-001,gb4-li-cephgw-002,gb4-li-cephgw-003 (age 2h) mgr: gb4-li-cephgw-003(active, since 2h), standbys: gb4-li-cephgw-002.iqmxgu, gb4-li-cephgw-001 osd: 72 osds: 72 up (since 39m), 72 in (since 3d); 1 remapped pgs flags noout,norebalance rgw: 3 daemons active (3 hosts, 1 zones) data: pools: 11 pools, 1457 pgs objects: 63.76M objects, 173 TiB usage: 251 TiB used, 275 TiB / 526 TiB avail pgs: 59116/371260644 objects degraded (0.016%) 1452 active+clean 4 active+clean+scrubbing+deep 1 active+undersized+degraded+remapped+backfilling io: client: 258 MiB/s rd, 595 KiB/s wr, 346 op/s rd, 387 op/s wr recovery: 15 KiB/s, 2 keys/s, 2 objects/s progress: Global Recovery Event (39m) [===========================.] (remaining: 1s) [ceph: root@gb4-li-cephgw-001 /]# ceph -s; sleep 3; ceph -s cluster: id: 6dabcf41-90d7-4e90-b259-1cc0bf298052 health: HEALTH_WARN noout,norebalance flag(s) set Degraded data redundancy: 58503/371260638 objects degraded (0.016%), 1 pg degraded, 1 pg undersized services: mon: 3 daemons, quorum gb4-li-cephgw-001,gb4-li-cephgw-002,gb4-li-cephgw-003 (age 2h) mgr: gb4-li-cephgw-003(active, since 2h), standbys: gb4-li-cephgw-002.iqmxgu, gb4-li-cephgw-001 osd: 72 osds: 72 up (since 39m), 72 in (since 3d); 1 remapped pgs flags noout,norebalance rgw: 3 daemons active (3 hosts, 1 zones) data: pools: 11 pools, 1457 pgs objects: 63.76M objects, 173 TiB usage: 251 TiB used, 275 TiB / 526 TiB avail pgs: 58503/371260638 objects degraded (0.016%) 1452 active+clean 4 active+clean+scrubbing+deep 1 active+undersized+degraded+remapped+backfilling io: client: 245 MiB/s rd, 278 KiB/s wr, 247 op/s rd, 183 op/s wr recovery: 16 KiB/s, 2 keys/s, 2 objects/s progress: Global Recovery Event (39m) [===========================.] (remaining: 1s) cluster: id: 6dabcf41-90d7-4e90-b259-1cc0bf298052 health: HEALTH_WARN noout,norebalance flag(s) set Degraded data redundancy: 58157/371260644 objects degraded (0.016%), 1 pg degraded, 1 pg undersized services: mon: 3 daemons, quorum gb4-li-cephgw-001,gb4-li-cephgw-002,gb4-li-cephgw-003 (age 2h) mgr: gb4-li-cephgw-003(active, since 2h), standbys: gb4-li-cephgw-002.iqmxgu, gb4-li-cephgw-001 osd: 72 osds: 72 up (since 39m), 72 in (since 3d); 1 remapped pgs flags noout,norebalance rgw: 3 daemons active (3 hosts, 1 zones) data: pools: 11 pools, 1457 pgs objects: 63.76M objects, 173 TiB usage: 251 TiB used, 275 TiB / 526 TiB avail pgs: 58157/371260644 objects degraded (0.016%) 1452 active+clean 4 active+clean+scrubbing+deep 1 active+undersized+degraded+remapped+backfilling io: client: 243 MiB/s rd, 285 KiB/s wr, 252 op/s rd, 197 op/s wr recovery: 13 KiB/s, 0 keys/s, 1 objects/s progress: Global Recovery Event (39m) [===========================.] (remaining: 1s) [ceph: root@gb4-li-cephgw-001 /]# Iain Stott OpenStack Engineer Iain.Stott(a)thehutgroup.com [THG Ingenuity Logo]<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en>

3 4

Re: set proxy for ceph installation
by Eugen Block 04 Oct '23

04 Oct '23

Did you apply the changes to the containers.conf file on all hosts? The MGR daemon is issuing the cephadm commands on the remote hosts, so it would need that as well. That setup works for me quite well for years now. What distro is your host running on? We mostly use openSUSE or SLES, but I also had that successfully running on Ubuntu VMs the same way. Zitat von Majid Varzideh <m.varzideh(a)gmail.com>: > Thanks for your responses. > locally it is ok and i can get images without any issue. Also I set proxy > parameters in containers.conf but no result. > it seems that env proxy parameters are not known to user issuing get > commands, > > > On Wed, Sep 27, 2023 at 11:00 AM Eugen Block <eblock(a)nde.ag> wrote: > >> You'll need a containers.conf file: >> >> # cat /etc/containers/containers.conf >> [engine] >> env = ["http_proxy=<IP>:<PORT>", "https_proxy=<IP>:<PORT>", >> "no_proxy=localhost"] >> >> Restarting the container should apply the change. Make sure you also >> have the correct no_proxy settings, for example so the ceph servers >> don't communicate via proxy with each other. >> >> Zitat von Majid Varzideh <m.varzideh(a)gmail.com>: >> >> > hi friends >> > i have deployed my first node in cluster. we dont have direct internet on >> > my server so i have to set proxy for that.i set it /etc/environment >> > /etc/profile but i get bellow error >> > 2023-09-26 17:09:38,254 7f04058b4b80 DEBUG >> > >> -------------------------------------------------------------------------------- >> > cephadm ['--image', 'quay.io/ceph/ceph:v17', 'pull'] >> > 2023-09-26 17:09:38,302 7f04058b4b80 INFO Pulling container image >> > quay.io/ceph/ceph:v17... >> > 2023-09-26 17:09:42,083 7f04058b4b80 INFO Non-zero exit code 125 from >> > /usr/bin/podman pull quay.io/ceph/ceph:v17 >> > 2023-09-26 17:09:42,083 7f04058b4b80 INFO /usr/bin/podman: stderr Trying >> to >> > pull quay.io/ceph/ceph:v17... >> > 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr >> > time="2023-09-26T17:09:38+03:30" level=warning msg="Failed, retrying in >> 1s >> > ... (1/3). Error: initializing source docker://quay.io/ceph/ceph:v17: >> > pinging container registry quay.io: Get \"https://quay.io/v2/\": dial >> tcp >> > 34.228.154.221:443: connect: connection refused" >> > 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr >> > time="2023-09-26T17:09:39+03:30" level=warning msg="Failed, retrying in >> 1s >> > ... (2/3). Error: initializing source docker://quay.io/ceph/ceph:v17: >> > pinging container registry quay.io: Get \"https://quay.io/v2/\": dial >> tcp >> > 3.220.246.53:443: connect: connection refused" >> > 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr >> > time="2023-09-26T17:09:40+03:30" level=warning msg="Failed, retrying in >> 1s >> > ... (3/3). Error: initializing source docker://quay.io/ceph/ceph:v17: >> > pinging container registry quay.io: Get \"https://quay.io/v2/\": dial >> tcp >> > 18.213.60.205:443: connect: connection refused" >> > 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr Error: >> > initializing source docker://quay.io/ceph/ceph:v17: pinging container >> > registry quay.io: Get "https://quay.io/v2/": dial tcp 34.231.182.47:443: >> > connect: connection refused >> > 2023-09-26 17:09:42,084 7f04058b4b80 ERROR ERROR: Failed command: >> > /usr/bin/podman pull quay.io/ceph/ceph:v17 >> > >> > >> > would you please help. >> > thanks, >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 0

cephfs health warn
by Ben 04 Oct '23

04 Oct '23

Hi, see below for details of warnings. the cluster is running 17.2.5. the warnings have been around for a while. one concern of mine is num_segments growing over time. clients with warn of MDS_CLIENT_OLDEST_TID increase from 18 to 25 as well. The nodes are with kernel 4.19.0-91.82.42.uelc20.x86_64. It looks like bugs with client library. And rebooting nodes with problem will fix it for short period of time? Any suggestions from community for fixing? Thanks, Ben [root@8cd2c0657c77 /]# ceph health detail HEALTH_WARN 6 hosts fail cephadm check; 2 clients failing to respond to capability release; 25 clients failing to advance oldest client/flush tid; 3 MDSs report slow requests; 3 MDSs behind on trimming [WRN] CEPHADM_HOST_CHECK_FAILED: 6 hosts fail cephadm check host host15w (192.168.31.33) failed check: Unable to reach remote host host15w. Process exited with non-zero exit status 1 host host20w (192.168.31.38) failed check: Unable to reach remote host host20w. Process exited with non-zero exit status 1 host host19w (192.168.31.37) failed check: Unable to reach remote host host19w. Process exited with non-zero exit status 1 host host17w (192.168.31.35) failed check: Unable to reach remote host host17w. Process exited with non-zero exit status 1 host host18w (192.168.31.36) failed check: Unable to reach remote host host18w. Process exited with non-zero exit status 1 host host16w (192.168.31.34) failed check: Unable to reach remote host host16w. Process exited with non-zero exit status 1 [WRN] MDS_CLIENT_LATE_RELEASE: 2 clients failing to respond to capability release mds.code-store.host18w.fdsqff(mds.1): Client k8s-node36 failing to respond to capability release client_id: 460983 mds.code-store.host16w.vucirx(mds.3): Client failing to respond to capability release client_id: 460983 [WRN] MDS_CLIENT_OLDEST_TID: 25 clients failing to advance oldest client/flush tid mds.code-store.host18w.fdsqff(mds.1): Client k8s-node36 failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host18w.fdsqff(mds.1): Client failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host18w.fdsqff(mds.1): Client k8s-node32 failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host15w.reolpx(mds.5): Client k8s-node34 failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host15w.reolpx(mds.5): Client k8s-node32 failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host15w.reolpx(mds.5): Client failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host18w.rtyvdy(mds.7): Client k8s-node34 failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host18w.rtyvdy(mds.7): Client failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host18w.rtyvdy(mds.7): Client k8s-node36 failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host17w.kcdopb(mds.2): Client failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host17w.kcdopb(mds.2): Client failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host17w.kcdopb(mds.2): Client k8s-node34 failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host17w.kcdopb(mds.2): Client k8s-node24 failing to advance its oldest client/flush tid. client_id: 12072730 mds.code-store.host20w.bfoftp(mds.4): Client k8s-node32 failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host20w.bfoftp(mds.4): Client k8s-node36 failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host19w.ywrmiz(mds.6): Client k8s-node24 failing to advance its oldest client/flush tid. client_id: 12072730 mds.code-store.host19w.ywrmiz(mds.6): Client k8s-node34 failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host19w.ywrmiz(mds.6): Client failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host19w.ywrmiz(mds.6): Client failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host16w.vucirx(mds.3): Client failing to advance its oldest client/flush tid. client_id: 460983 mds.code-store.host16w.vucirx(mds.3): Client failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host16w.vucirx(mds.3): Client failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host17w.pdziet(mds.0): Client k8s-node32 failing to advance its oldest client/flush tid. client_id: 239797 mds.code-store.host17w.pdziet(mds.0): Client k8s-node34 failing to advance its oldest client/flush tid. client_id: 460226 mds.code-store.host17w.pdziet(mds.0): Client k8s-node36 failing to advance its oldest client/flush tid. client_id: 460983 [WRN] MDS_SLOW_REQUEST: 3 MDSs report slow requests mds.code-store.host15w.reolpx(mds.5): 4 slow requests are blocked > 5 secs mds.code-store.host20w.bfoftp(mds.4): 6 slow requests are blocked > 5 secs mds.code-store.host16w.vucirx(mds.3): 97 slow requests are blocked > 5 secs [WRN] MDS_TRIM: 3 MDSs behind on trimming mds.code-store.host15w.reolpx(mds.5): Behind on trimming (25831/128) max_segments: 128, num_segments: 25831 mds.code-store.host20w.bfoftp(mds.4): Behind on trimming (27605/128) max_segments: 128, num_segments: 27605 mds.code-store.host16w.vucirx(mds.3): Behind on trimming (28676/128) max_segments: 128, num_segments: 28676

2 8