- ceph-users - lists.ceph.io

OSD CPU and write latency increase after upgrade from 15.2.16 to 17.2.6
by Tony Yao 11 Dec '23

11 Dec '23

Hi， Recently, I upgraded Ceph from 15.2.16 to 17.2.6, but I found that OSD CPU usage increased from 30% to 90% or more, and OSD subop_w_latency increased from 600us to 5ms. This is incredible. My hardware environment: 12 nodes x 12 NVMe (Intel P4510 4T) I tried to set the OSD configuration to the default value and saw no improvement either. Have you ever encountered this problem？ What might have gone wrong? Best regards, Tony Yao

1 1

ceph fs (meta) data inconsistent
by Frank Schilder 11 Dec '23

11 Dec '23

Dear fellow cephers, today we observed a somewhat worrisome inconsistency on our ceph fs. A file created on one host showed up as 0 length on all other hosts: [user1@host1 h2lib]$ ls -lh total 37M -rw-rw---- 1 user1 user1 12K Nov 1 11:59 dll_wrapper.py [user2@host2 h2lib]# ls -l total 34 -rw-rw----. 1 user1 user1 0 Nov 1 11:59 dll_wrapper.py [user1@host1 h2lib]$ cp dll_wrapper.py dll_wrapper.py.test [user1@host1 h2lib]$ ls -l total 37199 -rw-rw---- 1 user1 user1 11641 Nov 1 11:59 dll_wrapper.py -rw-rw---- 1 user1 user1 11641 Nov 1 13:10 dll_wrapper.py.test [user2@host2 h2lib]# ls -l total 45 -rw-rw----. 1 user1 user1 0 Nov 1 11:59 dll_wrapper.py -rw-rw----. 1 user1 user1 11641 Nov 1 13:10 dll_wrapper.py.test Executing a sync on all these hosts did not help. However, deleting the problematic file and replacing it with a copy seemed to work around the issue. We saw this with ceph kclients of different versions, it seems to be on the MDS side. How can this happen and how dangerous is it? ceph fs status (showing ceph version): # ceph fs status con-fs2 - 1662 clients ======= RANK STATE MDS ACTIVITY DNS INOS 0 active ceph-15 Reqs: 14 /s 2307k 2278k 1 active ceph-11 Reqs: 159 /s 4208k 4203k 2 active ceph-17 Reqs: 3 /s 4533k 4501k 3 active ceph-24 Reqs: 3 /s 4593k 4300k 4 active ceph-14 Reqs: 1 /s 4228k 4226k 5 active ceph-13 Reqs: 5 /s 1994k 1782k 6 active ceph-16 Reqs: 8 /s 5022k 4841k 7 active ceph-23 Reqs: 9 /s 4140k 4116k POOL TYPE USED AVAIL con-fs2-meta1 metadata 2177G 7085G con-fs2-meta2 data 0 7085G con-fs2-data data 1242T 4233T con-fs2-data-ec-ssd data 706G 22.1T con-fs2-data2 data 3409T 3848T STANDBY MDS ceph-10 ceph-08 ceph-09 ceph-12 MDS version: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) There is no health issue: # ceph status cluster: id: abc health: HEALTH_WARN 3 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 9w) mgr: ceph-25(active, since 7w), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1279 up (since 2d), 1279 in (since 5d) task status: data: pools: 14 pools, 25065 pgs objects: 2.20G objects, 3.9 PiB usage: 4.9 PiB used, 8.2 PiB / 13 PiB avail pgs: 25039 active+clean 26 active+clean+scrubbing+deep io: client: 799 MiB/s rd, 55 MiB/s wr, 3.12k op/s rd, 1.82k op/s wr The inconsistency seems undiagnosed, I couldn't find anything interesting in the cluster log. What should I look for and where? I moved the folder to another location for diagnosis. Unfortunately, I don't have 2 clients any more showing different numbers, I see a 0 length now everywhere for the moved folder. I'm pretty sure though that the file still is non-zero length. Thanks for any pointers. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 18

How to replace a disk with minimal impact on performance
by Michal Strnad 08 Dec '23

08 Dec '23

Hi guys! Based on our observation of the impact of the balancer on the performance of the entire cluster, we have drawn conclusions that we would like to discuss with you. - A newly created pool should be balanced before being handed over to the user. This, I believe, is quite evident. - When replacing a disk, it is advisable to exchange it directly for a new one. As soon as the OSD replacement occurs, the balancer should be invoked to realign any improperly placed PGs during the disk outage and disk recovery. Perhaps an even better method is to pause recovery and backfilling before removing the disk, remove the disk itself, promptly add a new one, and then resume recovery and backfilling. It's essential to per form all of this as quickly as possible (using a script). Ad. We are using a community balancer developed by Jonas Jelton because the built-in one does not meet our requirements. What are your thoughts on this? Michal

2 1

nfs export over RGW issue in Pacific
by Adiga, Anantha 07 Dec '23

07 Dec '23

Hi, oot@a001s016:~# cephadm version Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) root@a001s016:~# root@a001s016:~# cephadm shell Inferring fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 Inferring config /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s016/config Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 root@a001s016:~# ceph version ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) --------------------- But, Cephadm does not show "nfs export create rgw...." nfs export create cephfs <fsname> <clusterid> <binding> [--readonly] [<path>] nfs export rm <clusterid> <binding> nfs export delete <clusterid> <binding> nfs export ls <clusterid> [--detailed] nfs export get <clusterid> <binding> nfs export update --------------------- However, Ceph Dashboard allows to create the export see below: Access Type RW Cluster nfs-1 Daemons nfs-1.0.0.zp3110b001a0101.uckows, nfs-1.1.0.zp3110b001a0102.hhpebb, nfs-1.2.0.zp3110b001a0103.bbkpcb, nfs-1.3.0.zp3110b001a0104.zujkso NFS Protocol NFSv4 Object Gateway User admin Path buc-cluster-inventory Pseudo /rgwnfs_cluster_inventory Squash no_root_squash Storage Backend Object Gateway Transport TCP --------------------- While nfs export is created, pseudo "/rgwnfs_cluster_inventory", cephadm is not listing it # ceph nfs export ls nfs-1 [ "/cluster_inventory", "/oob_crashdump" ] # Anantha

2 2

Difficulty adding / using a non-default RGW placement target & storage class
by Anthony D'Atri 07 Dec '23

07 Dec '23

I'm having difficulty adding and using a non-default placement target & storage class and would appreciate insights. Am I going about this incorrectly? Rook does not yet have the ability to do this, so I'm adding it by hand. Following instructions on the net I added a second bucket pool, placement target, and storageclass, and created a user defaulting to the new pg/sc, but I get an error when trying to create a bucket: [rook@rook-ceph-tools-5ff8d58445-gkl5w .aws]$ s5cmd --endpoint-url http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc mb s3://foofoobars ERROR "mb s3://foofoobars": InvalidLocationConstraint: status code: 400, request id: tx0000057b71002881d48ca-0065495d54-1abe555-ceph-objectstore, host id: I found an article suggesting that the placement target and/or storage class should have the api_name prepended, so I tried setting either or both to "ceph-objectstore:HDD-EC" / "ceph-objectstore:GLACIER" with no success. I suspect that I'm missing something subtle -- or that Rook has provisioned these bits in an atypical fashion. Log entry: /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.543+0000 7f6573a9f700 1 ====== starting new request req=0x7f64818ba730 ===== /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.546+0000 7f6570a99700 0 req 6320538205097380042 0.003000009s s3:create_bucket could not find user default placement id HDD-EC/GLACIER within zonegroup /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.546+0000 7f6570a99700 1 ====== req done req=0x7f64818ba730 op status=-2208 http_status=400 latency=0.003000009s ====== /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log:2023-11-06T21:40:36.546+0000 7f6570a99700 1 beast: 0x7f64818ba730: 10.233.90.156 - aad [06/Nov/2023:21:40:36.543 +0000] "PUT /foofoobars HTTP/1.1" 400 266 - "aws-sdk-go/1.40.25 (go1.18.3; linux; amd64)" - latency=0.003000009s [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ ceph -v ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Here's the second buckets pool, constrained to HDDs. AFAICT it can share the index and data_extra_pool created for the default / STANDARD pt/sc by Rook. I initially omitted ec_overwrites but enabled it after creation. pool 19 'ceph-objectstore.rgw.buckets.data' erasure profile ceph-objectstore.rgw.buckets.data_ecprofile size 6 min_size 5 crush_rule 10 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 165350 lfor 0/156300/165341 flags hashpspool,ec_overwrites stripe_width 16384 application rook-ceph-rgw pool 21 'ceph-objectstore.rgw.buckets.data.hdd' erasure profile ceph-objectstore.rgw.buckets.data_ecprofile_hdd size 6 min_size 5 crush_rule 11 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 167193 lfor 0/0/164453 flags hashpspool,ec_overwrites stripe_width 16384 application rook-ceph-rgw [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin zonegroup get { "id": "d994155c-2a9c-4e37-ae30-64fd2934ff99", "name": "ceph-objectstore", "api_name": "ceph-objectstore", "is_master": "true", "endpoints": [ "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "72035401-a6d9-426b-8c89-9a17e268825f", "zones": [ { "id": "72035401-a6d9-426b-8c89-9a17e268825f", "name": "ceph-objectstore", "endpoints": [ "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "HDD-EC", "tags": [], "storage_classes": [ "GLACIER" ] }, { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "51fb8875-31ac-40ef-ab21-0ffd4e229f15", "sync_policy": { "groups": [] } } [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin zone get { "id": "72035401-a6d9-426b-8c89-9a17e268825f", "name": "ceph-objectstore", "domain_root": "ceph-objectstore.rgw.meta:root", "control_pool": "ceph-objectstore.rgw.control", "gc_pool": "ceph-objectstore.rgw.log:gc", "lc_pool": "ceph-objectstore.rgw.log:lc", "log_pool": "ceph-objectstore.rgw.log", "intent_log_pool": "ceph-objectstore.rgw.log:intent", "usage_log_pool": "ceph-objectstore.rgw.log:usage", "roles_pool": "ceph-objectstore.rgw.meta:roles", "reshard_pool": "ceph-objectstore.rgw.log:reshard", "user_keys_pool": "ceph-objectstore.rgw.meta:users.keys", "user_email_pool": "ceph-objectstore.rgw.meta:users.email", "user_swift_pool": "ceph-objectstore.rgw.meta:users.swift", "user_uid_pool": "ceph-objectstore.rgw.meta:users.uid", "otp_pool": "ceph-objectstore.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "HDD-EC", "val": { "index_pool": "ceph-objectstore.rgw.buckets.index", "storage_classes": { "GLACIER": { "data_pool": "ceph-objectstore.rgw.buckets.data.hdd" }, "STANDARD": {} # <------------- seems like this shouldn't be here? }, "data_extra_pool": "ceph-objectstore.rgw.buckets.non-ec", "index_type": 0 } }, { "key": "default-placement", "val": { "index_pool": "ceph-objectstore.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "ceph-objectstore.rgw.buckets.data" } }, "data_extra_pool": "ceph-objectstore.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "", "notif_pool": "ceph-objectstore.rgw.log:notif" } [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin user info --uid=aad { "user_id": "aad", "display_name": "Anthony", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "aad", "access_key": "xxxx", "secret_key": "yyyyyy" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "HDD-EC", "default_storage_class": "GLACIER", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] }

2 1

Moving from ceph-ansible to cephadm and upgrading from pacific to octopus
by wodel youchi 07 Dec '23

07 Dec '23

Hi, I have an Openstack platform deployed with Yoga and ceph-ansible pacific on Rocky 8. Now I need to do an upgrade to Openstack zed with octopus on Rocky 9. This is the path of the upgrade I have traced - upgrade my nodes to Rocky 9 keeping Openstack yoga with ceph-ansible pacific. - convert ceph pacific from ceph-ansible to cephadm. - stop Openstack platform yoga - upgrade ceph pacific to octopus - upgrade Openstack yoga to zed. Any thoughts or guide lines to keep in mind and follow regarding ceph convertion and upgrade. Ps : on my ceph I have rbd, rgw and cephfs pools. Regards.

1 0

About delete old bucket lifecycle
by VÔ VI 07 Dec '23

07 Dec '23

Hi community, I am have multiple bucket was delete but lifecycle of bucket still exist, how i can delete it with radosgw-admin, because user can't access to bucket for delete lifecycle. User for this bucket does not exist. root@ceph:~# radosgw-admin lc list [ { "bucket": ":r30203:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.8", "started": "Wed, 06 Dec 2023 10:43:55 GMT", "status": "COMPLETE" }, { "bucket": ":r30304:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.13", "started": "Wed, 06 Dec 2023 10:43:54 GMT", "status": "COMPLETE" }, { "bucket": ":ec3204cam04:f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.1", "started": "Wed, 06 Dec 2023 10:44:30 GMT", "status": "COMPLETE" }, { "bucket": ":r30105:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.5", "started": "Wed, 06 Dec 2023 10:44:40 GMT", "status": "COMPLETE" }, { "bucket": ":r30303:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.14", "started": "Wed, 06 Dec 2023 10:44:40 GMT", "status": "COMPLETE" }, { "bucket": ":ec3201cam02:f3fec4b6-a248-4f3f-be75-b8055e61233a.56439.2", "started": "Wed, 06 Dec 2023 10:43:56 GMT", "status": "COMPLETE" }, Thanks to the community.

2 2

Assistance Needed with Ceph Cluster Slow Ops Issue
by Peter 06 Dec '23

06 Dec '23

Dear all, I am reaching out regarding an issue with our Ceph cluster that has been recurring every six hours. Upon investigating the problem using the "ceph daemon dump_historic_slow_ops" command, I observed that the issue appears to be related to slow operations, specifically getting stuck at "Waiting for RW Locks." The wait times often range from one to two seconds. Our cluster use SAS SSD disks from Samsung for the storage pool in question. While these disks are of high quality and should provide sufficient speed, the problem persists. The slow ops occurrence is consistent every six hours. I would greatly appreciate any insights or suggestions you may have to address and resolve this issue. If there are specific optimizations or configurations that could improve the situation, please advise. below are some output: root@lasas003:~# ceph -v ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable) "events": [ { "event": "initiated", "time": "2023-12-06T08:34:18.501644-0800", "duration": 0 }, { "event": "throttled", "time": "2023-12-06T08:34:18.501644-0800", "duration": 3.067e-06 }, { "event": "header_read", "time": "2023-12-06T08:34:18.501647-0800", "duration": 3.5429999999999998e-06 }, { "event": "all_read", "time": "2023-12-06T08:34:18.501650-0800", "duration": 9.3399999999999997e-07 }, { "event": "dispatched", "time": "2023-12-06T08:34:18.501651-0800", "duration": 3.2830000000000002e-06 }, { "event": "queued_for_pg", "time": "2023-12-06T08:34:18.501654-0800", "duration": 1.3819939990000001 }, { "event": "reached_pg", "time": "2023-12-06T08:34:19.883648-0800", "duration": 5.7980000000000002e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:19.883654-0800", "duration": 4.2484711649999998 }, { "event": "reached_pg", "time": "2023-12-06T08:34:24.132125-0800", "duration": 1.0667e-05 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:24.132136-0800", "duration": 2.1593527840000002 }, { "event": "reached_pg", "time": "2023-12-06T08:34:26.291489-0800", "duration": 3.292e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:26.291492-0800", "duration": 0.43918164700000001 }, { "event": "reached_pg", "time": "2023-12-06T08:34:26.730674-0800", "duration": 5.1529999999999996e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:26.730679-0800", "duration": 1.0531516869999999 }, { "event": "reached_pg", "time": "2023-12-06T08:34:27.783831-0800", "duration": 5.1329999999999998e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:27.783836-0800", "duration": 1.232525088 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.016361-0800", "duration": 3.844e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.016365-0800", "duration": 0.0051385700000000003 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.021503-0800", "duration": 4.7600000000000002e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.021508-0800", "duration": 0.0092808779999999994 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.030789-0800", "duration": 4.0690000000000003e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.030793-0800", "duration": 0.55757725499999999 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.588370-0800", "duration": 5.5060000000000003e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.588376-0800", "duration": 0.0064168929999999999 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.594793-0800", "duration": 7.0690000000000004e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.594800-0800", "duration": 0.0026404089999999998 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.597440-0800", "duration": 3.3440000000000001e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.597444-0800", "duration": 0.0051126670000000004 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.602556-0800", "duration": 5.0200000000000002e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.602561-0800", "duration": 0.0040569960000000002 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.606618-0800", "duration": 5.0989999999999998e-06 }, { "event": "waiting for rw locks", "time": "2023-12-06T08:34:29.606623-0800", "duration": 0.0068874100000000001 }, { "event": "reached_pg", "time": "2023-12-06T08:34:29.613511-0800", "duration": 1.4636e-05 }, { "event": "started", "time": "2023-12-06T08:34:29.613525-0800", "duration": 0.00028943699999999997 }, { "event": "done", "time": "2023-12-06T08:34:29.613815-0800", "duration": 11.112171102 } Thank you in advance for your assistance.

2 3

EC Profiles & DR
by duluxoz 06 Dec '23

06 Dec '23

Hi All, Looking for some help/explanation around erasure code pools, etc. I set up a 3-node Ceph (Quincy) cluster with each box holding 7 OSDs (HDDs) and each box running Monitor, Manager, and iSCSI Gateway. For the record the cluster runs beautifully, without resource issues, etc. I created an Erasure Code Profile, etc: ~~~ ceph osd erasure-code-profile set my_ec_profile plugin=jerasure k=4 m=2 crush-failure-domain=osd ceph osd crush rule create-erasure my_ec_rule my_ec_profile ceph osd crush rule create-replicated my_replicated_rule default host ~~~ My Crush Map is: ~~~ # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 15 osd.15 class hdd device 16 osd.16 class hdd device 17 osd.17 class hdd device 18 osd.18 class hdd device 19 osd.19 class hdd device 20 osd.20 class hdd # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root # buckets host ceph_1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.0 weight 5.34769 item osd.1 weight 5.45799 item osd.2 weight 5.45799 item osd.3 weight 5.45799 item osd.4 weight 5.45799 item osd.5 weight 5.45799 item osd.6 weight 5.45799 } host ceph_2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.7 weight 5.34769 item osd.8 weight 5.45799 item osd.9 weight 5.45799 item osd.10 weight 5.45799 item osd.11 weight 5.45799 item osd.12 weight 5.45799 item osd.13 weight 5.45799 } host ceph_3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 38.09564 alg straw2 hash 0 # rjenkins1 item osd.14 weight 5.34769 item osd.15 weight 5.45799 item osd.16 weight 5.45799 item osd.17 weight 5.45799 item osd.18 weight 5.45799 item osd.19 weight 5.45799 item osd.20 weight 5.45799 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 114.28693 alg straw2 hash 0 # rjenkins1 item ceph_1 weight 38.09564 item ceph_2 weight 38.09564 item ceph_3 weight 38.09564 } # rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_replicated_rule { id 1 type replicated step take default step chooseleaf firstn 0 type host step emit } rule my_ec_rule { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step choose indep 3 type host step chooseleaf indep 2 type osd step emit } # end crush map ~~~ Finally I create a pool: ~~~ ceph osd pool create my_pool 32 32 erasure my_ec_profile my_ec_rule ceph osd pool application enable my_meta_pool rbd rbd pool init my_meta_pool rbd pool init my_pool rbd create --size 16T my_pool/my_disk_1 --data-pool my_pool --image-feature journaling ~~~ So all this is to have some VMs (oVirt VMs, for the record) with automatic fall-over in the case of a Ceph Node loss - ie I was trying to "replicate" a 3-Disk RAID 5 array across the Ceph Nodes, so that I could loose a Node and still have a working set of VMs. However, I took one of the Ceph Nodes down (gracefully) for some maintenance the other day and I lost *all* the VMs (ie oVirt complained that there was no active pool). As soon as I brought the down node back up everything was good again. So my question is: What did I do wrong with my config? Sound I, for example, change the EC Profile to `k=2, m=1`, but how is that practically different from `k=4, m=2` - yes, the later spreads the pool over more disks, but it should still only put 2 disks on each node, shouldn't it? Thanks in advance Cheers Dulux-Oz

11 18

CLT Meeting minutes 2023-12-06
by Laura Flores 06 Dec '23

06 Dec '23

Closing the loop (blocked waiting for Neha's input): how are we using Gibba on a day-to-day basis? Is it only used for checking reef point releases? - To be discussed again next week, as Neha had a conflict [Nizam] http://old.ceph.com/pgcalc is not working anymore, is there any replacement for pgcalc page in the new ceph site? - Last attempt at a fix https://github.com/ceph/ceph.io/issues/265 - Nizam will take a look; the PR needs some CSS work - We may not even need this link due to the autoscaler, etc. - Perhaps add some kind of "banner" to the pgcalc page to make it clear that users should look to the autoscaler Update on 18.2.1 bluestore issue? - Fix has been raised; also not as serious as initially suspected - Fix is in testing -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

2 1