- ceph-users - lists.ceph.io

Num values for 3 DC 4+2 crush rule
by Torkil Svensgaard 16 Mar '24

16 Mar '24

I was just looking at our crush rules as we need to change them from failure domain host to failure domain datacenter. The replicated ones seem trivial but what about this one for EC 4+2? rule rbd_ec_data { id 0 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } We already have this crush rule for EC 4+5: " rule cephfs.hdd.data { id 7 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type datacenter step chooseleaf indep 3 type host step emit } " I don't understand the "num" argument for the choose step. The documentation[1] says: " If {num} == 0, choose pool-num-replicas buckets (as many buckets as are available). If pool-num-replicas > {num} > 0, choose that many buckets. If {num} < 0, choose pool-num-replicas - {num} buckets. " Num is 0 but it's not replicated so how does this translate to picking 3 of 3 datacenters? I am thinking we should just change 3 to 2 for the chooseleaf line for the 4+2 rule since for 4+5 each DC needs 3 shards and for 4+2 each DC needs 2 shards. Comments? Mvh. Torkil [1] https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/ -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

2 1

[Urgent] Ceph system Down, Ceph FS volume in recovering
by nguyenvandiep＠baoviet.com.vn 16 Mar '24

16 Mar '24

We have 6 node ( 3 OSD-node and 3 service node), t2/3 OSD nodes was powered off and we got big problem pls check ceph-s result below now we cannot start mds service, ( we tried to start but it stopped after 2 minute) Now my application cannot access to NFS exported Folder What should we do [root@cephgw01 /]# ceph -s cluster: id: 258af72a-cff3-11eb-a261-d4f5ef25154c health: HEALTH_WARN 3 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available 1 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 21 pgs backfill_toofull 15 pool(s) nearfull 11 daemons have recently crashed services: mon: 6 daemons, quorum cephgw03,cephosd01,cephgw01,cephosd03,cephgw02,cephosd02 (age 30h) mgr: cephgw01.vwoffq(active, since 17h), standbys: cephgw02.nauphz, cephgw03.aipvii mds: 1/1 daemons up osd: 29 osds: 29 up (since 40h), 29 in (since 29h); 402 remapped pgs rgw: 2 daemons active (2 hosts, 1 zones) tcmu-runner: 18 daemons active (2 hosts) data: volumes: 0/1 healthy, 1 recovering pools: 15 pools, 1457 pgs objects: 36.87M objects, 25 TiB usage: 75 TiB used, 41 TiB / 116 TiB avail pgs: 17759672/110607480 objects misplaced (16.056%) 1055 active+clean 363 active+remapped+backfill_wait 18 active+remapped+backfilling 14 active+remapped+backfill_toofull 7 active+remapped+backfill_wait+backfill_toofull io: client: 2.0 MiB/s rd, 395 KiB/s wr, 73 op/s rd, 19 op/s wr recovery: 32 MiB/s, 45 objects/s

6 49

Cephfs error state with one bad file
by Sake Ceph 15 Mar '24

15 Mar '24

Hi again, hopefully for the last time with problems. We had a MDS crash earlier with the MDS staying in failed state and used a command to reset the filesystem (this was wrong, I know now, thanks Patrick Donnelly for pointing this out). I did a full scrub on the filesystem and two files were damaged. One of those got repaired, but the following file keeps giving errors and can't be removed. What can I do now? Below some information. # ceph tell mds.atlassian-prod:0 damage ls [ { "damage_type": "backtrace", "id": 2244444901, "ino": 1099534008829, "path": "/app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01" } ] Trying to repair the error (online research shows this should work for a backtrace damage type) ---------- # ceph tell mds.atlassian-prod:0 scrub start /app1/shared/data/repositories/11271 recursive,repair,force { "return_code": 0, "scrub_tag": "d10ead42-5280-4224-971e-4f3022e79278", "mode": "asynchronous" } Cluster logs after this ---------- 1/2/24 9:37:05 AM [INF] scrub summary: idle 1/2/24 9:37:02 AM [INF] scrub summary: idle+waiting paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub summary: active paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub summary: idle+waiting paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub queued for path: /app1/shared/data/repositories/11271 But the error doesn't disappear and still can't remove the file. On the client trying to remove the file (we got a backup) ---------- $ rm -f /mnt/shared_disk-app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01 rm: cannot remove '/mnt/shared_disk-app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01': Input/output error Best regards, Sake

2 1

MDS subtree pinning
by Sake Ceph 15 Mar '24

15 Mar '24

Hi! As I'm reading through the documentation about subtree pinning, I was wondering if the following is possible. We've got the following directory structure. / /app1 /app2 /app3 /app4 Can I pin /app1 to MDS rank 0 and 1, the directory /app2 to rank 2 and finally /app3 and /app4 to rank 3? I would like to load balance the subfolders of /app1 to 2 (or 3) MDS servers. Best regards, Sake

2 2

Robust cephfs design/best practice
by Szabo, Istvan (Agoda) 15 Mar '24

15 Mar '24

Hi, I'd like to add cephfs to our production objectstore/block storage cluster so I'd like to collect hands on experiences like, good to know/be careful/avoid etc ... other than ceph documentation. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 2

RGW - tracking new bucket creation and bucket usage
by Ondřej Kukla 15 Mar '24

15 Mar '24

Hello I’m looking for suggestions how to track bucket creation over s3 api and bucket usage (num of objects and size) of all buckets in time. In our RGW setup, we have a custom client panel, where like 85% percent of buckets are created which is easy for us to then track the newly created buckets and assigned them to our clients, however we also enable them to create the buckets using the s3 api and there the tracking bucket to client is harder. We also want to periodically track the bucket usage stats (num of objects and size) of all buckets in time. Now we are using the GetBucketInfo from the AdminOPS api - https://docs.ceph.com/en/quincy/radosgw/adminops/#id44 with the stats=true option GET /admin/bucket?stats=1 which returns all buckets with the number of objects and size we then parse. We also use it for the tracking of newly created buckets as it gives you all of the buckets with their owners at one place, but we are not using the metadata in that case. This however in our environment, where we have close to ~400 buckets isn’t scalable and the calls can take from 10 to 400 to even 6000s to complete. So I’m looking for suggestions on how to get this data in time better/smarter. I know there is a bunch of prometheus exporters like https://github.com/pcuzner/rgw-exporter but they are usually just using the same AdminOPS api so that doesn’t seem like a better solution. I’m also working on sending all of our RGW ops logs to elastic which would give me the ability to easily query the CreateBucket calls, but that only solves the bucket creation and not the usage stats. How would you handle this? Do you have any suggestions? Thanks a lot. Ondrej

3 2

[REEF][cephadm] new cluster all pg unknown
by wodel youchi 15 Mar '24

15 Mar '24

Hi, I am creating a new ceph cluster using REEF. This is my host_specs file [root@controllera config]# cat hosts-specs2.yml service_type: host hostname: computehci01 addr: 20.1.0.2 location: chassis: chassis1 --- service_type: host hostname: computehci02 addr: 20.1.0.3 location: chassis: chassis1 --- service_type: host hostname: computehci03 addr: 20.1.0.4 location: chassis: chassis1 --- service_type: host hostname: computehci04 addr: 20.1.0.5 location: chassis: chassis2 --- service_type: host hostname: computehci05 addr: 20.1.0.6 location: chassis: chassis2 --- service_type: host hostname: computehci06 addr: 20.1.0.7 location: chassis: chassis2 --- service_type: host hostname: computehci07 addr: 20.1.0.8 location: chassis: chassis3 --- service_type: host hostname: computehci08 addr: 20.1.0.9 location: chassis: chassis3 --- service_type: host hostname: computehci09 addr: 20.1.0.10 location: chassis: chassis3 --- service_type: host hostname: computehci10 addr: 20.1.0.11 location: chassis: chassis3 --- service_type: host hostname: computehci11 addr: 20.1.0.12 location: chassis: chassis4 --- service_type: host hostname: computehci12 addr: 20.1.0.13 location: chassis: chassis4 --- service_type: host hostname: computehci13 addr: 20.1.0.14 location: chassis: chassis4 --- service_type: host hostname: computehci14 addr: 20.1.0.15 location: chassis: chassis4 --- service_type: host hostname: computehci15 addr: 20.1.0.16 location: chassis: chassis5 --- service_type: host hostname: computehci16 addr: 20.1.0.17 location: chassis: chassis5 --- service_type: host hostname: computehci17 addr: 20.1.0.18 location: chassis: chassis5 --- service_type: host hostname: computehci18 addr: 20.1.0.19 location: chassis: chassis5 --- service_type: host hostname: computehci19 addr: 20.1.0.20 location: chassis: chassis6 --- service_type: host hostname: computehci20 addr: 20.1.0.21 location: chassis: chassis6 --- service_type: host hostname: computehci21 addr: 20.1.0.22 location: chassis: chassis6 --- service_type: host hostname: computehci22 addr: 20.1.0.24 location: chassis: chassis7 --- service_type: host hostname: computehci23 addr: 20.1.0.25 location: chassis: chassis7 --- service_type: host hostname: computehci24 addr: 20.1.0.26 location: chassis: chassis7 --- service_type: host hostname: computehci25 addr: 20.1.0.28 location: chassis: chassis8 --- service_type: host hostname: computehci26 addr: 20.1.0.29 location: chassis: chassis8 --- service_type: host hostname: computehci27 addr: 20.1.0.30 location: chassis: chassis8 --- service_type: host hostname: controllera addr: 20.1.0.23 --- service_type: host hostname: controllerb addr: 20.1.0.27 --- service_type: host hostname: controllerc addr: 20.1.0.31 --- service_type: mon placement: hosts: - controllera - controllerb - controllerc --- service_type: mgr placement: hosts: - controllera - controllerb - controllerc --- service_type: osd service_id: default_drive_group placement: hosts: - computehci01 - computehci02 - computehci03 - computehci04 - computehci05 - computehci06 - computehci07 - computehci08 - computehci09 - computehci10 - computehci11 - computehci12 - computehci13 - computehci14 - computehci15 - computehci16 - computehci17 - computehci18 - computehci19 - computehci20 - computehci21 - computehci22 - computehci23 - computehci24 - computehci25 - computehci26 - computehci27 spec: data_devices: rotational: 0 All osds were added but, pg still unknown state I've created a pool, but it didn't change anything. [root@controllerb ~]# ceph -s cluster: id: be250ade-e1f2-11ee-a6ff-3cecef2872f0 health: HEALTH_WARN Reduced data availability: 1 pg inactive services: mon: 3 daemons, quorum controllera,controllerc,controllerb (age 3h) mgr: controllerc.jevbkl(active, since 21s), standbys: controllera.zwlolp, controllerb.vqkdga osd: 108 osds: 108 up (since 2m), 108 in (since 24m) data: pools: 2 pools, 33 pgs objects: 0 objects, 0 B usage: 5.1 GiB used, 330 TiB / 330 TiB avail * pgs: 100.000% pgs unknown 33 unknown* Did I miss something? Regards.

2 6

PSA: CephFS/MDS config defer_client_eviction_on_laggy_osds
by Venky Shankar 15 Mar '24

15 Mar '24

If you are using CephFS on Pacific v16.2.14(+), the MDS config `defer_client_eviction_on_laggy_osds' is enabled by default. This config is used to not evict cephfs clients if OSDs are laggy[1]. However, this can result in a single client holding up the MDS in servicing other clients. To avoid this, please disable the config by executing ceph config set mds defer_client_eviction_on_laggy_osds false This issue doesn't affect any Quincy or Reef releases. Furthermore, the config is being disabled by default[2] till the fix is being worked on. [1]: https://tracker.ceph.com/issues/58023 [2]: https://tracker.ceph.com/issues/64685 -- Cheers, Venky

1 0

Ceph Dashboard Clear Cache
by ashar.khan＠rapidcompute.com 15 Mar '24

15 Mar '24

Hello Ceph members, How do I clear the Ceph dashboard cache? Kindly guide me on how to do this. Thanks

1 0

Re: ceph metrics units
by Konstantin Shalygin 14 Mar '24

14 Mar '24

Hi, > On 14 Mar 2024, at 19:29, Denis Polom <denispolom(a)gmail.com> wrote: > > so metric itself is miliseconds and after division on _count it's in seconds? > > This is two metrics for long running averages [1], the query that produces "seconds" unit looks like this (irate(ceph_osd_op_r_latency_sum[1m]) / irate(ceph_osd_op_r_latency_count[1m])) * on (instance, ceph_daemon) group_left (device) ceph_disk_occupation * on (instance, ceph_daemon) group_left (device_class, hostname) ceph_osd_metadata k [1] https://docs.ceph.com/en/latest/mgr/prometheus/?highlight=%3Cname%3E_sum#st…

1 0