- ceph-users - lists.ceph.io

Health warning - POOL_TARGET_SIZE_BYTES_OVERCOMMITED
by Sake Paulusma 13 Feb '23

13 Feb '23

The RATIO for cephfs.application-acc.data shouldn't be over 1.0, I believe this triggered the error. All weekend I was thinking about this issue, but couldn't find an option to correct this. But minutes after posting I found a blog about the autoscaler (https://ceph.io/en/news/blog/2022/autoscaler_tuning) and it speaks about the option to set the Rate. Shouldn't this option be set to 2 when using a stretched cluster and not 4?

1 0

Limited set of permissions for an RGW user (S3)
by Gauvain Pocentek 13 Feb '23

13 Feb '23

Hi list, A little bit of background: we provide S3 buckets using RGW (running quincy), but users are not allowed to manage their buckets, just read and write objects in them. Buckets are created by an admin user, and read/write permissions are given to end users using S3 bucket policies. We set the users quota to 0 for everything to forbid them to create buckets. This is not really scalable and a bit annoying for the users. So we are trying to find a solution to allow users to create their own buckets but with a limited set of APIs available (no policy change for example). The ceph doc says that policies cannot be applied on users, groups or roles yet. Is there any other way to achieve this? Any feedback will be appreciated. Thanks! Gauvain

1 0

Re%3A%20%5Bceph-users%5D%20Re%3A%20Exit%20yolo%20mode%20by%20increasing%20size/min_size%20does%20not%20%28really%29%20work
by Stefan Pinter 13 Feb '23

13 Feb '23

Hi! Thank you 😊 Your message was very helpful! The main reason why “ceph df“ went to “100% USAGE” was because of the crush rule said this: "min_size": 2 "max_size": 2 And the new “size” was 3, so the rule did not want to work with the pool. After creating a new rule and setting the pools to this new rule, it seems to work (in our test cluster) rule newrule { ruleset 4 type replicated min_size 3 max_size 3 step take default step choose firstn 2 type room step chooseleaf firstn 2 type host step emit } Thanks again! Further testing now 😊 ________________________________ BearingPoint GmbH Sitz: Wien Firmenbuchgericht: Handelsgericht Wien Firmenbuchnummer: FN 175524z The information in this email is confidential and may be legally privileged. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system.

1 0

recovery for node disaster
by farhad kh 13 Feb '23

13 Feb '23

I have a cluster of three nodes, with three replicas per pool on cluster nodes --------- HOST ADDR LABELS STATUS apcepfpspsp0101 192.168.114.157 _admin mon apcepfpspsp0103 192.168.114.158 mon _admin apcepfpspsp0105 192.168.114.159 mon _admin 3 hosts in cluster --------- # ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] --------- epoch 1033 fsid 9c35e594-2392-11ed-809a-005056ae050c created 2022-08-24T09:53:36.481866+0000 modified 2023-02-12T18:57:34.447536+0000 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 51 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client luminous require_osd_release quincy stretch_mode_enabled false pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr pool 2 'k8s-rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 541 lfor 0/0/44 flags hashpspool,selfmanaged_snaps max_bytes 75161927680 stripe_width 0 application rbd pool 3 'k8s-cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 543 lfor 0/0/57 flags hashpspool max_bytes 5368709120 stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 4 'k8s-cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 542 lfor 0/0/57 flags hashpspool max_bytes 32212254720 stripe_width 0 application cephfs ----------- Is it possible to recover data when two nodes with all physical disks are lost for any reason? What is the maximum number of fault tolerance for the cluster? For this purpose, consider the default settings . what changes should I make to increase fault tolerance?

2 1

Subject: OSDs added, remapped pgs and objects misplaced cycling up and down
by Chris Dunlop 13 Feb '23

13 Feb '23

Hi, ceph-16.2.9 I've added some new osds - some added to existing hosts and some on newly-commissioned hosts. The new osds were added to the data side of an existing EC 8.3 pool. I've been waiting for the system to finish remapping / backfilling for some time. Originally the number of remapped pgs and objects misplaced was steadily decreasing. However in the last few days both remapped pgs and objects misplaced have increased several times. Both then steadily decrease before jumping up again. See below for some details on this. I'm guessing these are associated stats, i.e. the number of objects misplaced reflects the objects in the remapped pgs so it's not surprising they're moving in concert. I thought at first that this cycling might be due to the balancer (which was enabled), but the cycling is continuing after the balancer was disabled, e.g.: $ ts; ceph balancer status 2023-02-09 12:39:14 { "active": true, "last_optimize_duration": "0:00:00.000313", "last_optimize_started": "Thu Feb 9 01:38:55 2023", "mode": "upmap", "optimize_result": "Too many objects (0.078537 > 0.050000) are misplaced; try again later", "plans": [] } $ ts; ceph balancer off 2023-02-09 13:03:27 ceph balancer off 4 days later, the balancer is still inactive, and "last_optimize_started" indicates it hasn't run since before it was turned off: $ ts; ceph balancer status 2023-02-13 10:18:18 { "active": false, "last_optimize_duration": "0:00:00.000420", "last_optimize_started": "Thu Feb 9 02:02:56 2023", "mode": "upmap", "optimize_result": "Too many objects (0.078101 > 0.050000) are misplaced; try again later", "plans": [] } Since the balancer was turned off the cycling continues: "misplaced" gets down to 5% and then rapidly increases to 9%, in conjunction with "starting backfill" messages, before starting to slowly decrease once again. See below for a log extract. Is this "sawtooth" pattern of remapped pgs and misplaced objects a normal consequence of adding OSDs? Will it eventually settle done itself or do I need to "do something"? Is there some way of telling how much work remains until it all settles down? Cheers, Chris ------------------------------------------------------------------------------ # # General health # scrubs disabled until added OSDs complete backfilling etc. # $ ceph -s cluster: id: c6618970-0ce0-4cb2-bc9a-dd5f29b62e24 health: HEALTH_WARN noout,noscrub,nodeep-scrub flag(s) set 5945 pgs not deep-scrubbed in time 5945 pgs not scrubbed in time 5 slow ops, oldest one blocked for 464 sec, mon.k2 has slow ops services: mon: 5 daemons, quorum k2,b2,b4,k1,b5 (age 5d) mgr: b4(active, since 3M), standbys: b5, b2 osd: 82 osds: 82 up (since 2w), 82 in (since 2w); 12 remapped pgs flags noout,noscrub,nodeep-scrub data: pools: 16 pools, 5945 pgs objects: 70.35M objects, 88 TiB usage: 186 TiB used, 242 TiB / 428 TiB avail pgs: 50050192/700705500 objects misplaced (7.143%) 5933 active+clean 10 active+remapped+backfill_wait 2 active+remapped+backfilling io: client: 2.2 MiB/s rd, 1.8 MiB/s wr, 395 op/s rd, 122 op/s wr recovery: 24 MiB/s, 24 objects/s # # remapped pgs steadily decreasing then started periodically increasing # (extract from periodic "ceph -s") # 2023-02-06 00:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 26 remapped pgs 2023-02-06 04:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 25 remapped pgs 2023-02-06 06:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 24 remapped pgs 2023-02-06 09:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 23 remapped pgs 2023-02-06 12:30:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 22 remapped pgs 2023-02-06 14:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 21 remapped pgs 2023-02-06 14:30:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 20 remapped pgs 2023-02-06 20:30:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 19 remapped pgs 2023-02-07 00:00:00 osd: 82 osds: 82 up (since 8d), 82 in (since 7d); 18 remapped pgs 2023-02-07 01:00:00 osd: 82 osds: 82 up (since 8d), 82 in (since 7d); 17 remapped pgs 2023-02-07 11:00:00 osd: 82 osds: 82 up (since 8d), 82 in (since 8d); 15 remapped pgs 2023-02-07 16:30:00 osd: 82 osds: 82 up (since 8d), 82 in (since 8d); 13 remapped pgs 2023-02-07 22:30:00 osd: 82 osds: 82 up (since 9d), 82 in (since 8d); 11 remapped pgs 2023-02-08 15:30:00 osd: 82 osds: 82 up (since 9d), 82 in (since 9d); 9 remapped pgs 2023-02-08 21:30:00 osd: 82 osds: 82 up (since 10d), 82 in (since 9d); 15 remapped pgs <<< increase 2023-02-09 07:30:00 osd: 82 osds: 82 up (since 10d), 82 in (since 9d); 13 remapped pgs 2023-02-09 13:03:27 ceph balancer off 2023-02-09 22:30:00 osd: 82 osds: 82 up (since 11d), 82 in (since 10d); 11 remapped pgs 2023-02-10 14:00:00 osd: 82 osds: 82 up (since 11d), 82 in (since 11d); 9 remapped pgs 2023-02-10 19:00:00 osd: 82 osds: 82 up (since 11d), 82 in (since 11d); 15 remapped pgs <<< increase 2023-02-11 04:30:00 osd: 82 osds: 82 up (since 12d), 82 in (since 12d); 13 remapped pgs 2023-02-11 19:00:00 osd: 82 osds: 82 up (since 12d), 82 in (since 12d); 11 remapped pgs 2023-02-12 12:00:00 osd: 82 osds: 82 up (since 13d), 82 in (since 13d); 9 remapped pgs 2023-02-12 18:00:00 osd: 82 osds: 82 up (since 13d), 82 in (since 13d); 15 remapped pgs <<< increase 2023-02-12 20:30:00 osd: 82 osds: 82 up (since 13d), 82 in (since 13d); 14 remapped pgs 2023-02-13 04:00:00 osd: 82 osds: 82 up (since 2w), 82 in (since 13d); 12 remapped pgs # # objects misplaced increases when the remapped pgs increases # 2023-02-06 00:00:00 pgs: 62898818/685798373 objects misplaced (9.172%) 2023-02-06 04:00:00 pgs: 61995946/686058837 objects misplaced (9.037%) 2023-02-06 06:00:00 pgs: 61090940/686122303 objects misplaced (8.904%) 2023-02-06 09:00:00 pgs: 59285766/686306445 objects misplaced (8.638%) 2023-02-06 12:30:00 pgs: 57626520/686387322 objects misplaced (8.396%) 2023-02-06 14:00:00 pgs: 56813753/686388870 objects misplaced (8.277%) 2023-02-06 14:30:00 pgs: 56578343/686389185 objects misplaced (8.243%) 2023-02-06 20:30:00 pgs: 54255700/686969448 objects misplaced (7.898%) 2023-02-07 00:00:00 pgs: 53089526/687266865 objects misplaced (7.725%) 2023-02-07 01:00:00 pgs: 52810518/687391707 objects misplaced (7.683%) 2023-02-07 11:00:00 pgs: 50169973/688319189 objects misplaced (7.289%) 2023-02-07 16:30:00 pgs: 49312642/689410730 objects misplaced (7.153%) 2023-02-07 22:30:00 pgs: 48244798/689771838 objects misplaced (6.994%) 2023-02-08 15:30:00 pgs: 38058106/691166947 objects misplaced (5.506%) 2023-02-08 21:30:00 pgs: 62851681/691618387 objects misplaced (9.088%) <<< increase 2023-02-09 07:30:00 pgs: 57693559/693214621 objects misplaced (8.323%) 2023-02-09 13:03:27 ceph balancer off 2023-02-09 22:30:00 pgs: 48248474/694618590 objects misplaced (6.946%) 2023-02-10 14:00:00 pgs: 37872501/696136659 objects misplaced (5.440%) 2023-02-10 19:00:00 pgs: 64062339/696645992 objects misplaced (9.196%) <<< increase 2023-02-11 04:30:00 pgs: 52183052/699438360 objects misplaced (7.461%) 2023-02-11 19:00:00 pgs: 49167191/699631520 objects misplaced (7.028%) 2023-02-12 12:00:00 pgs: 39485735/700248360 objects misplaced (5.639%) 2023-02-12 18:00:00 pgs: 63540276/700256808 objects misplaced (9.074%) <<< increase 2023-02-12 20:30:00 pgs: 61766426/700291865 objects misplaced (8.820%) 2023-02-13 04:00:00 pgs: 57622457/700365805 objects misplaced (8.227%) --- annotated log extract --- ## ## "misplaced" decreasing... ## Feb 12 17:50:20 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:18.782191+0000 mgr.b4 (mgr.17297882) 4129637 : cluster [DBG] pgmap v4094135: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 178 op/s; 35014018/700256808 objects misplaced (5.000%); 34 MiB/s, 34 objects/s recovering Feb 12 17:50:20 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:18.782191+0000 mgr.b4 (mgr.17297882) 4129637 : cluster [DBG] pgmap v4094135: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 178 op/s; 35014018/700256808 objects misplaced (5.000%); 34 MiB/s, 34 objects/s recovering Feb 12 17:50:27 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:26.833542+0000 mgr.b4 (mgr.17297882) 4129641 : cluster [DBG] pgmap v4094141: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 2.7 MiB/s rd, 4.9 MiB/s wr, 226 op/s; 35012325/700256808 objects misplaced (5.000%); 37 MiB/s, 36 objects/s recovering Feb 12 17:50:30 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:28.848020+0000 mgr.b4 (mgr.17297882) 4129642 : cluster [DBG] pgmap v4094143: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 436 op/s; 35011830/700256808 objects misplaced (5.000%); 36 MiB/s, 35 objects/s recovering ## ## shortly after, "misplaced" has increased, followed by the "starting backfill" lines ## I'm not sure if the line ordering here is a quirk of journalctl, maybe the "starting backfill" came first which would make some sense ## Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:30.860404+0000 mgr.b4 (mgr.17297882) 4129643 : cluster [DBG] pgmap v4094145: 5945 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5934 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 419 KiB/s rd, 1.2 MiB/s wr, 388 op/s; 44579289/700256808 objects misplaced (6.366%); 31 MiB/s, 30 objects/s recovering Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.031822+0000 osd.64 (osd.64) 362 : cluster [DBG] 33.2cs0 starting backfill to osd.25(4) from (0'0,0'0] MAX to 639282'24881196 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.031951+0000 osd.64 (osd.64) 363 : cluster [DBG] 33.6cs0 starting backfill to osd.25(4) from (0'0,0'0] MAX to 639282'24593871 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.044285+0000 osd.64 (osd.64) 364 : cluster [DBG] 33.6cs0 starting backfill to osd.51(5) from (0'0,0'0] MAX to 639282'24593871 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.044655+0000 osd.64 (osd.64) 365 : cluster [DBG] 33.2cs0 starting backfill to osd.51(5) from (0'0,0'0] MAX to 639282'24881196 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.057888+0000 osd.64 (osd.64) 366 : cluster [DBG] 33.6cs0 starting backfill to osd.63(7) from (0'0,0'0] MAX to 639282'24593871 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.058039+0000 osd.64 (osd.64) 367 : cluster [DBG] 33.2cs0 starting backfill to osd.63(7) from (0'0,0'0] MAX to 639282'24881196 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.077517+0000 osd.64 (osd.64) 368 : cluster [DBG] 33.6cs0 starting backfill to osd.68(2) from (0'0,0'0] MAX to 639282'24593871 ## (more "starting backfill" lines elided here) ## ## "misplaced" continues to increase rapidly to 9%... ## Feb 12 17:50:34 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:32.871242+0000 mgr.b4 (mgr.17297882) 4129644 : cluster [DBG] pgmap v4094148: 5945 pgs: 2 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5932 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 424 KiB/s rd, 1.2 MiB/s wr, 440 op/s; 54133989/700256808 objects misplaced (7.731%); 31 MiB/s, 31 objects/s recovering Feb 12 17:50:36 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:34.886503+0000 mgr.b4 (mgr.17297882) 4129645 : cluster [DBG] pgmap v4094149: 5945 pgs: 2 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5932 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 442 KiB/s rd, 1.1 MiB/s wr, 483 op/s; 54133670/700256808 objects misplaced (7.731%); 41 MiB/s, 41 objects/s recovering Feb 12 17:50:37 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:36.901531+0000 mgr.b4 (mgr.17297882) 4129646 : cluster [DBG] pgmap v4094150: 5945 pgs: 4 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 281 KiB/s rd, 902 KiB/s wr, 324 op/s; 63666090/700256808 objects misplaced (9.092%); 17 MiB/s, 17 objects/s recovering ## ## then starts to fall again... ## Feb 12 17:50:42 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:40.930869+0000 mgr.b4 (mgr.17297882) 4129648 : cluster [DBG] pgmap v4094152: 5945 pgs: 4 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 407 KiB/s rd, 1.4 MiB/s wr, 479 op/s; 63665652/700256808 objects misplaced (9.092%); 28 MiB/s, 28 objects/s recovering Feb 12 17:50:43 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:42.948421+0000 mgr.b4 (mgr.17297882) 4129649 : cluster [DBG] pgmap v4094153: 5945 pgs: 3 active+remapped+backfilling, 12 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 455 KiB/s rd, 1.5 MiB/s wr, 542 op/s; 63665652/700256808 objects misplaced (9.092%); 27 MiB/s, 26 objects/s recovering Feb 12 17:50:46 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:44.961474+0000 mgr.b4 (mgr.17297882) 4129650 : cluster [DBG] pgmap v4094154: 5945 pgs: 3 active+remapped+backfilling, 12 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 436 KiB/s rd, 1.4 MiB/s wr, 518 op/s; 63664579/700256808 objects misplaced (9.092%); 37 MiB/s, 37 objects/s recovering Feb 12 17:50:50 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:48.991308+0000 mgr.b4 (mgr.17297882) 4129652 : cluster [DBG] pgmap v4094156: 5945 pgs: 3 active+remapped+backfilling, 12 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 513 KiB/s rd, 1.6 MiB/s wr, 534 op/s; 63664502/700256808 objects misplaced (9.092%); 33 MiB/s, 33 objects/s recovering ------------------------------------------------------------------------------

2 2

Quincy: Stuck on image permissions
by hicks＠cgi.cz 12 Feb '23

12 Feb '23

Hello guys, could someone help me with this? We've been long-time CEPH users... runing several Mimic + Pacific CEPH clusters. Dozens of disk per cluster, typically. BUT... now I have this brand new Quincy cluster and I'm not able to give CLIENT (Quincy on Rocky 8) rw access to ONE IMAGE on Quincy cluster (cephadm / Rocky 9). I'm using something what worked for us for ages: rbd auth ls: client.xxx key: ... caps: [mon] profile rbd caps: [osd] allow rwx pool prod object_prefix rbd_data.600d1c6723ae; allow rwx pool prod object_prefix rbd_header.600d1c6723ae; allow rx pool prod object_prefix rbd_id.xxx-data rbd info: rbd image 'xxx-data': size 2 TiB in 524288 objects order 22 (4 MiB objects) snapshot_count: 2 id: 600d1c6723ae block_name_prefix: rbd_data.600d1c6723ae format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: rados ls: rbd_data.600d1c6723ae.000000000003958d rbd_header.600d1c6723ae rbd_id.xxx-data BUT... it DOES NOT WORK. When I try it to map on client it says: 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::GetMetadataRequest: 0x7f3a1c001f40 handle_metadata_list: failed to retrieve image metadata: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::RefreshRequest: failed to retrieve pool metadata: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::OpenRequest: failed to refresh image: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::ImageState: 0x555eff78cfc0 failed to open image: (1) Operation not permitted rbd: error opening image xxx-data: (1) Operation not permitted The mapping and access DOES work when I put "osd allow *" into ceph auth. What is the recommended syntax for Quincy? btw: this use case should be mentioned in the manual I think... Thanks!

2 1

Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1
by duluxoz 11 Feb '23

11 Feb '23

Hi All, Sorry if this was mentioned previously (I obviously missed it if it was) but can we upgrade a Ceph Quincy Host/Cluster from Rocky Linux (RHEL) v8.6/8.7 to v9.1 (yet), and if so, what is / where can I find the procedure to do this - ie is there anything "special" that needs to be done because of Ceph, or can we just do a "simple" v8.x +> v9.1 upgrade? Thanks in advance Cheers Dulux-Oz

3 6

Re: Exit yolo mode by increasing size/min_size does not (really) work
by stefan 11 Feb '23

11 Feb '23

ah, thank you so much Eugen! this makes sense! i will report what we will have changed and if it worked or not :) i wish you a nice weekend!

1 0

Exit yolo mode by increasing size/min_size does not (really) work
by Stefan Pinter 10 Feb '23

10 Feb '23

Hi! 😊 It would be very kind of you to help us with that! We have pools in our ceph cluster that are set to replicated size 2 min_size 1. Obviously we want to go to size 3 / min_size 2 but we experience problems with that. USED goes to 100% instantly and MAX AVAIL goes to 0. Write operations seemed to stop. POOLS: NAME ID USED %USED MAX AVAIL OBJECTS Pool1 24 35791G 35.04 66339G 8927762 Pool2 25 11610G 14.89 66339G 3004740 Pool3 26 17557G 100.00 0 2666972 Before the change it was like this: NAME ID USED %USED MAX AVAIL OBJECTS Pool1 24 35791G 35.04 66339G 8927762 Pool2 25 11610G 14.89 66339G 3004740 Pool3 26 17558G 20.93 66339G 2667013 This was quite surprising to us as we’d expect USED to go to something like 30%. Going back to 2/1 also gave us back the 20.93% usage instantly. What’s the matter here? Thank you and best regards Stefan ________________________________ BearingPoint GmbH Sitz: Wien Firmenbuchgericht: Handelsgericht Wien Firmenbuchnummer: FN 175524z The information in this email is confidential and may be legally privileged. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system.

3 5

issue in connecting Openstack(Kolla-ansible) manila with external ceph (cephadm)
by Haitham Abdulaziz 10 Feb '23

10 Feb '23

i deployed kolla-ansible & cephadm on virtual machines (kvm) . My ceph cluster is on 3 vms with 12 vCPU each and 24gb of ram i used cephadm to deploy ceph ceph -s : -------------------------- cluster: id: a0e5ad36-a54c-11ed-9aea-5254008c2a3e health: HEALTH_OK services: mon: 3 daemons, quorum ceph0,ceph1,ceph2 (age 6h) mgr: ceph0.dzutak(active, since 24h), standbys: ceph1.aizuyc mds: 3/3 daemons up, 6 standby osd: 9 osds: 9 up (since 24h), 9 in (since 24h) data: volumes: 3/3 healthy pools: 9 pools, 257 pgs objects: 70 objects, 7.3 KiB usage: 76 MiB used, 780 GiB / 780 GiB avail pgs: 257 active+clean -------------------------- my openstack deployment is AIO on a single node , now i wanna link them together so i started with manila & native cephfs thinking its the easist following this doc : https://docs.openstack.org/manila/latest/admin/cephfs_driver.html#authorizi… i created the user -------------------------- client.manila key: AQC7ot9jfiDsIxAA57fb7S6bVMnr5IadsnukHQ== caps: [mgr] allow rw caps: [mon] allow r caps: [osd] allow rw pool=ganesha_rados_store and created a file system called manila -------------------------- my ceph.conf -------------------------- [global] fsid = a0e5ad36-a54c-11ed-9aea-5254008c2a3e mon_host = [v2:192.168.122.25:3300/0,v1:192.168.122.25:6789/0] [v2:192.168.122.115:3300/0,v1:192.168.122.115:6789/0] [v2:192.168.122.14:3300/0,v1:192.168.122.14:6789/0] -------------------------- i moved the files as to the openstack node and trying to connect them together but it didnt go will , Viewing the logs shows -------------------------- <AIO@cephfsnative1: manila.exception.ShareBackendException: json_command failed - prefix=fs volume ls, argdict={'format': 'json'} - exception message: Bad target type 'mon-mgr'. --------------------------where should i start to fix this issue ?

2 1