- ceph-users - lists.ceph.io

User + Dev Meetup Tomorrow!
by Laura Flores 24 May '24

24 May '24

Hi all, The User + Dev Meetup will be held tomorrow at 10:00 AM EDT. We will be discussing the results of the latest survey, and users who attend will have the opportunity to provide additional feedback in real time. See you there! Laura Flores Meeting Details: https://www.meetup.com/ceph-user-group/events/300883526/ -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

3 4

quincy rgw with ceph orch and two realms only get answers from first realm
by Boris 24 May '24

24 May '24

Hi, we are currently in the process of adopting the main s3 cluster to orchestrator. We have two realms (one for us and one for the customer). The old config worked fine and depending on the port I requested, I got different x-amz-request-id header back: x-amz-request-id: tx00000307170ac0d734ab4-0066508120-94aa0f66-eu-central-1 x-amz-request-id: tx00000a9d8fc1821bbe258-00665081d1-949b2447-eu-customer-1 I then deployed the orchestrator config to other systems and checked, but now I get always an answer from the eu-central-1 zone and never from the customer zone. The test is very simple and works fine for the old rgw config, but fails for the new orchestrator config: curl [IPv6]:7481/somebucket -v All rgw instances are running on ceph version 17.2.7. Configs attached below. The old ceph.conf looked like this: [client.eu-central-1-s3db1-old] rgw_frontends = beast endpoint=[::]:7480 rgw_region = eu rgw_zone = eu-central-1 rgw_dns_name = example.com rgw_dns_s3website_name = eu-central-1.example.com [client.eu-customer-1-s3db1] rgw_frontends = beast endpoint=[::]:7481 rgw_region = eu-customer rgw_zone = eu-customer-1 rgw_dns_name = s3.customer.domain rgw_dns_s3website_name = eu-central-1.s3.customer.domain And this is the new service.yaml service_type: rgw service_id: ab12 placement: label: rgw config: debug_rgw: 0 rgw_thread_pool_size: 2048 rgw_dns_name: example.com rgw_dns_s3website_name: eu-central-1.example.com rgw_enable_gc_threads: false spec: rgw_frontend_port: 7480 rgw_frontend_type: beast rgw_realm: company rgw_zone: eu-central-1 rgw_zonegroup: eu --- service_type: rgw service_id: customer placement: label: rgw config: debug_rgw: 0 rgw_dns_name: s3.customer.domain rgw_dns_s3website_name: eu-central-1.s3.customer.domain rgw_enable_gc_threads: false spec: rgw_frontend_port: 7481 rgw_frontend_type: beast rgw_realm: customer rgw_zone: eu-customer-1 rgw_zonegroup: eu-customer -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 1

cephadm bootstraps cluster with bad CRUSH map(?)
by Matthew Vernon 24 May '24

24 May '24

Hi, I'm probably Doing It Wrong here, but. My hosts are in racks, and I wanted ceph to use that information from the get-go, so I tried to achieve this during bootstrap. This has left me with a single sad pg: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive pg 1.0 is stuck inactive for 33m, current state unknown, last acting [] ceph osd tree shows that CRUSH picked up my racks OK, eg. -3 45.11993 rack B4 -2 45.11993 host moss-be1001 1 hdd 3.75999 osd.1 up 1.00000 1.00000 But root seems empty: -1 0 root default and if I decompile the crush map, indeed: # buckets root default { id -1 # do not change unnecessarily id -14 class hdd # do not change unnecessarily # weight 0.00000 alg straw2 hash 0 # rjenkins1 } which does indeed look empty, whereas I have rack entries that contain the relevant hosts. And the replication rule: rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type rack step emit } I passed this config to bootstrap with --config: [global] osd_crush_chooseleaf_type = 3 and an initial spec file with host entries like this: service_type: host hostname: moss-be1001 addr: 10.64.16.40 location: rack: B4 labels: - _admin - NVMe Once the cluster was up I used an osd spec file that looked like: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe" I could presumably fix this up by editing the crushmap (to put the racks into the default bucket), but what did I do wrong? Was this not a reasonable thing to want to do with cephadm? I'm running ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) Thanks, Matthew

3 9

Help with deep scrub warnings
by Nicola Mori 24 May '24

24 May '24

Dear Ceph users, in order to reduce the deep scrub load on my cluster I set the deep scrub interval to 2 weeks, and tuned other parameters as follows: # ceph config get osd osd_deep_scrub_interval 1209600.000000 # ceph config get osd osd_scrub_sleep 0.100000 # ceph config get osd osd_scrub_load_threshold 0.300000 # ceph config get osd osd_deep_scrub_randomize_ratio 0.100000 # ceph config get osd osd_scrub_min_interval 259200.000000 # ceph config get osd osd_scrub_max_interval 1209600.000000 In my admittedly poor knowledge of Ceph's deep scrub procedures, these settings should spread the deep scrub operations in two weeks instead of the default one week, lowering the scrub frequency and the related load. But I'm currently getting warnings like: [WRN] PG_NOT_DEEP_SCRUBBED: 56 pgs not deep-scrubbed in time pg 3.1e1 not deep-scrubbed since 2024-02-22T00:22:55.296213+0000 pg 3.1d9 not deep-scrubbed since 2024-02-20T03:41:25.461002+0000 pg 3.1d5 not deep-scrubbed since 2024-02-20T09:52:57.334058+0000 pg 3.1cb not deep-scrubbed since 2024-02-20T03:30:40.510979+0000 . . . I don't understand the first one, since the deep scrub interval should be two weeks so I don''t expect warnings for PGs which have been deep-scrubbed less than 14 days ago (at the moment I'm writing it's Tue Mar 5 07:39:07 UTC 2024). Moreover, I don't understand why the deep scrub for so many PGs is lagging behind. Is there something wrong in my settings? Thanks in advance for any help, Nicola

4 4

Traefik front end with RGW
by Reid Guyett 23 May '24

23 May '24

Hello, We are considering moving from Nginx to Traefik as the frontend for our RGW services. Prior to putting into production I ran it through s3-tests and noticed that all of the tests involving metadata (x-amz-meta-*) are failing because they are expected to be lowercase (test_s3.py::test_object_set_get_metadata_none_to_good - KeyError: 'meta1'). Traefik / Go converts these to have the first letter uppercase (X-Amz-Meta-Key). Poking around I found https://github.com/traefik/traefik/issues/466 and it points out RFC 2616 which says these should be case insensitive. AWS S3 says "User-defined metadata is a set of key-value pairs. Amazon S3 stores user-defined metadata keys in lowercase." over at https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html. Is anybody using Traefik for their frontend to RGW? Do you mitigate this if so and how? Thanks for any input. Reid

1 0

Best practice regarding rgw scaling
by Szabo, Istvan (Agoda) 23 May '24

23 May '24

Hi, Wonder what is the best practice to scale RGW, increase the thread numbers or spin up more gateways? * Let's say I have 21000 connections on my haproxy * I have 3 physical gateway servers so let's say each of them need to server 7000 connections This means with 512 thread pool size each of them needs 13 gateway altogether 39 in the cluster. or 3 gateway and each 8192 rgw thread? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 2

Status of 18.2.3
by Sake Ceph 23 May '24

23 May '24

I was wondering what happened to the release of 18.2.3? Validation started on April 13th and as far as I know there have been a couple of builds and some extra bug fixes. Is there a way to follow a release or what is holding it back? Normally I wouldn't ask about a release and just wait, but I really need some fixes of this release. Kind regards, Sake

2 2

rgw can't find zone
by stephan.budach＠jvm.com 23 May '24

23 May '24

I am running two Ceph clusters on 18.2.2. On both clusters, I initially configured rgw using the following config from a yaml file: service_type: rgw service_id: s3.jvm.gh79 placement: label: rgw count_per_host: 1 spec: rgw_realm: jvm rgw_zone: gh79 ssl: true rgw_frontend_port: 9443 rgw_frontend_type: beast rgw_frontend_ssl_certificate: | Note, that I didn't configure anything else regarding realms, zonegroups or zones. I then modified the tags on 3 of my cephnodes to include rgw and the services got installed, configured and started. I am pretty sure, that I did this after I updated Ceph to 18.2.2. A couple of days ago, one of my cephnodes, hosting a rgw rebootet and the rgw service didn't come up again. When I looked at the logs I saw that the rgw service had an error when looking up the zone - which seems non-existent at this point: Mai 23 12:43:00 bceph06 radosgw[3608419]: deferred set uid:gid to 167:167 (ceph:ceph) Mai 23 12:43:00 bceph06 radosgw[3608419]: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable), process radosgw, pid 2 Mai 23 12:43:00 bceph06 radosgw[3608419]: framework: beast Mai 23 12:43:00 bceph06 radosgw[3608419]: framework conf key: ssl_port, val: 9443 Mai 23 12:43:00 bceph06 radosgw[3608419]: framework conf key: ssl_certificate, val: config://rgw/cert/rgw.s3.jvm.gh79 Mai 23 12:43:00 bceph06 radosgw[3608419]: init_numa not setting numa affinity Mai 23 12:43:01 bceph06 radosgw[3608419]: rgw main: ERROR: could not find zone (gh79) Mai 23 12:43:01 bceph06 radosgw[3608419]: rgw main: ERROR: failed to start notify service ((2) No such file or directory Mai 23 12:43:01 bceph06 radosgw[3608419]: rgw main: ERROR: failed to init services (ret=(2) No such file or directory) Mai 23 12:43:01 bceph06 ceph-5ccaf98c-ec2a-11ee-8293-bc2411f9733d-rgw-s3-jvm-gh79-bceph06-rwkczj[3608415]: 2024-05-23T10:43:01.014+0000 7fe0b87b3a80 -1 Couldn't init storage provider (RADOS) Mai 23 12:43:01 bceph06 radosgw[3608419]: Couldn't init storage provider (RADOS) The rados zone configs look like this at the moment: root@cephnode01:/# radosgw-admin zone list { "default_info": "e67d13fc-8343-4c88-9b1f-a8286f46f44a", "zones": [ "default" ] } So, why did it work in the first place and what would be the easiest way of resolving this?

1 0

Re: Pacific 16.2.15 and ceph-volume no-longer creating LVM on block.db partition
by Bruno Canning 23 May '24

23 May '24

I should add, we are not using cephadm. From: Bruno Canning Sent: 23 May 2024 11:36 To: ceph-users(a)ceph.io Subject: Pacific 16.2.15 and ceph-volume no-longer creating LVM on block.db partition Hi Folks, After recently upgrading from 16.2.13 to 16.2.15, when I run: ceph-volume lvm create --data /dev/sda --block.db /dev/nvme0n1p1 to create a new OSD after replacement of a failed disk, ceph-volume no-longer creates a volume group/logical volume on the block.db partition*. This is the old behaviour we saw whilst running Octopus but we have not seen this since our upgrade to Pacific until now. Any ideas why this is happening? Nothing is jumping out of the release notes, but they do mention a lot of work was done on the LVM code in ceph-volume since 16.2.13. Is this now a configuration setting somewhere? Background: Our storage nodes are 60 HDDs per chassis with two 2TB NVMe drives, each containing 30 partitions used as block.db storage, one partition and one HDD per OSD. We are running Ubuntu 20.04 and Ceph Pacific 16.2.15, just upgraded from 16.2.13. We are using the official Ubuntu packages. *so /var/lib/ceph/osd/ceph-1090/block.db points to /dev/nvme0n1p1 and not /dev/ceph-11992480-4ea6-4c1a-80b0-e025a66539b2/osd-db-52655aa3-9102-4d92-8a63-f2c93eed984f which would be on /dev/nvme0n1p1 ---------------------------------------------------------------------- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.

1 0

Pacific 16.2.15 and ceph-volume no-longer creating LVM on block.db partition
by Bruno Canning 23 May '24

23 May '24

Hi Folks, After recently upgrading from 16.2.13 to 16.2.15, when I run: ceph-volume lvm create --data /dev/sda --block.db /dev/nvme0n1p1 to create a new OSD after replacement of a failed disk, ceph-volume no-longer creates a volume group/logical volume on the block.db partition*. This is the old behaviour we saw whilst running Octopus but we have not seen this since our upgrade to Pacific until now. Any ideas why this is happening? Nothing is jumping out of the release notes, but they do mention a lot of work was done on the LVM code in ceph-volume since 16.2.13. Is this now a configuration setting somewhere? Background: Our storage nodes are 60 HDDs per chassis with two 2TB NVMe drives, each containing 30 partitions used as block.db storage, one partition and one HDD per OSD. We are running Ubuntu 20.04 and Ceph Pacific 16.2.15, just upgraded from 16.2.13. We are using the official Ubuntu packages. *so /var/lib/ceph/osd/ceph-1090/block.db points to /dev/nvme0n1p1 and not /dev/ceph-11992480-4ea6-4c1a-80b0-e025a66539b2/osd-db-52655aa3-9102-4d92-8a63-f2c93eed984f which would be on /dev/nvme0n1p1 ---------------------------------------------------------------------- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.

1 0