February 2023 - ceph-users

by farhad kh

I have a cluster of three nodes, with three replicas per pool on cluster nodes --------- HOST ADDR LABELS STATUS apcepfpspsp0101 192.168.114.157 _admin mon apcepfpspsp0103 192.168.114.158 mon _admin apcepfpspsp0105 192.168.114.159 mon _admin 3 hosts in cluster --------- # ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] --------- epoch 1033 fsid 9c35e594-2392-11ed-809a-005056ae050c created 2022-08-24T09:53:36.481866+0000 modified 2023-02-12T18:57:34.447536+0000 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 51 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client luminous require_osd_release quincy stretch_mode_enabled false pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr pool 2 'k8s-rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 541 lfor 0/0/44 flags hashpspool,selfmanaged_snaps max_bytes 75161927680 stripe_width 0 application rbd pool 3 'k8s-cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 543 lfor 0/0/57 flags hashpspool max_bytes 5368709120 stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 4 'k8s-cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 542 lfor 0/0/57 flags hashpspool max_bytes 32212254720 stripe_width 0 application cephfs ----------- Is it possible to recover data when two nodes with all physical disks are lost for any reason? What is the maximum number of fault tolerance for the cluster? For this purpose, consider the default settings . what changes should I make to increase fault tolerance?

1 year, 2 months

2
1
0 0

Subject: OSDs added, remapped pgs and objects misplaced cycling up and down

by Chris Dunlop

Hi, ceph-16.2.9 I've added some new osds - some added to existing hosts and some on newly-commissioned hosts. The new osds were added to the data side of an existing EC 8.3 pool. I've been waiting for the system to finish remapping / backfilling for some time. Originally the number of remapped pgs and objects misplaced was steadily decreasing. However in the last few days both remapped pgs and objects misplaced have increased several times. Both then steadily decrease before jumping up again. See below for some details on this. I'm guessing these are associated stats, i.e. the number of objects misplaced reflects the objects in the remapped pgs so it's not surprising they're moving in concert. I thought at first that this cycling might be due to the balancer (which was enabled), but the cycling is continuing after the balancer was disabled, e.g.: $ ts; ceph balancer status 2023-02-09 12:39:14 { "active": true, "last_optimize_duration": "0:00:00.000313", "last_optimize_started": "Thu Feb 9 01:38:55 2023", "mode": "upmap", "optimize_result": "Too many objects (0.078537 > 0.050000) are misplaced; try again later", "plans": [] } $ ts; ceph balancer off 2023-02-09 13:03:27 ceph balancer off 4 days later, the balancer is still inactive, and "last_optimize_started" indicates it hasn't run since before it was turned off: $ ts; ceph balancer status 2023-02-13 10:18:18 { "active": false, "last_optimize_duration": "0:00:00.000420", "last_optimize_started": "Thu Feb 9 02:02:56 2023", "mode": "upmap", "optimize_result": "Too many objects (0.078101 > 0.050000) are misplaced; try again later", "plans": [] } Since the balancer was turned off the cycling continues: "misplaced" gets down to 5% and then rapidly increases to 9%, in conjunction with "starting backfill" messages, before starting to slowly decrease once again. See below for a log extract. Is this "sawtooth" pattern of remapped pgs and misplaced objects a normal consequence of adding OSDs? Will it eventually settle done itself or do I need to "do something"? Is there some way of telling how much work remains until it all settles down? Cheers, Chris ------------------------------------------------------------------------------ # # General health # scrubs disabled until added OSDs complete backfilling etc. # $ ceph -s cluster: id: c6618970-0ce0-4cb2-bc9a-dd5f29b62e24 health: HEALTH_WARN noout,noscrub,nodeep-scrub flag(s) set 5945 pgs not deep-scrubbed in time 5945 pgs not scrubbed in time 5 slow ops, oldest one blocked for 464 sec, mon.k2 has slow ops services: mon: 5 daemons, quorum k2,b2,b4,k1,b5 (age 5d) mgr: b4(active, since 3M), standbys: b5, b2 osd: 82 osds: 82 up (since 2w), 82 in (since 2w); 12 remapped pgs flags noout,noscrub,nodeep-scrub data: pools: 16 pools, 5945 pgs objects: 70.35M objects, 88 TiB usage: 186 TiB used, 242 TiB / 428 TiB avail pgs: 50050192/700705500 objects misplaced (7.143%) 5933 active+clean 10 active+remapped+backfill_wait 2 active+remapped+backfilling io: client: 2.2 MiB/s rd, 1.8 MiB/s wr, 395 op/s rd, 122 op/s wr recovery: 24 MiB/s, 24 objects/s # # remapped pgs steadily decreasing then started periodically increasing # (extract from periodic "ceph -s") # 2023-02-06 00:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 26 remapped pgs 2023-02-06 04:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 25 remapped pgs 2023-02-06 06:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 24 remapped pgs 2023-02-06 09:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 23 remapped pgs 2023-02-06 12:30:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 22 remapped pgs 2023-02-06 14:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 21 remapped pgs 2023-02-06 14:30:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 20 remapped pgs 2023-02-06 20:30:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 19 remapped pgs 2023-02-07 00:00:00 osd: 82 osds: 82 up (since 8d), 82 in (since 7d); 18 remapped pgs 2023-02-07 01:00:00 osd: 82 osds: 82 up (since 8d), 82 in (since 7d); 17 remapped pgs 2023-02-07 11:00:00 osd: 82 osds: 82 up (since 8d), 82 in (since 8d); 15 remapped pgs 2023-02-07 16:30:00 osd: 82 osds: 82 up (since 8d), 82 in (since 8d); 13 remapped pgs 2023-02-07 22:30:00 osd: 82 osds: 82 up (since 9d), 82 in (since 8d); 11 remapped pgs 2023-02-08 15:30:00 osd: 82 osds: 82 up (since 9d), 82 in (since 9d); 9 remapped pgs 2023-02-08 21:30:00 osd: 82 osds: 82 up (since 10d), 82 in (since 9d); 15 remapped pgs <<< increase 2023-02-09 07:30:00 osd: 82 osds: 82 up (since 10d), 82 in (since 9d); 13 remapped pgs 2023-02-09 13:03:27 ceph balancer off 2023-02-09 22:30:00 osd: 82 osds: 82 up (since 11d), 82 in (since 10d); 11 remapped pgs 2023-02-10 14:00:00 osd: 82 osds: 82 up (since 11d), 82 in (since 11d); 9 remapped pgs 2023-02-10 19:00:00 osd: 82 osds: 82 up (since 11d), 82 in (since 11d); 15 remapped pgs <<< increase 2023-02-11 04:30:00 osd: 82 osds: 82 up (since 12d), 82 in (since 12d); 13 remapped pgs 2023-02-11 19:00:00 osd: 82 osds: 82 up (since 12d), 82 in (since 12d); 11 remapped pgs 2023-02-12 12:00:00 osd: 82 osds: 82 up (since 13d), 82 in (since 13d); 9 remapped pgs 2023-02-12 18:00:00 osd: 82 osds: 82 up (since 13d), 82 in (since 13d); 15 remapped pgs <<< increase 2023-02-12 20:30:00 osd: 82 osds: 82 up (since 13d), 82 in (since 13d); 14 remapped pgs 2023-02-13 04:00:00 osd: 82 osds: 82 up (since 2w), 82 in (since 13d); 12 remapped pgs # # objects misplaced increases when the remapped pgs increases # 2023-02-06 00:00:00 pgs: 62898818/685798373 objects misplaced (9.172%) 2023-02-06 04:00:00 pgs: 61995946/686058837 objects misplaced (9.037%) 2023-02-06 06:00:00 pgs: 61090940/686122303 objects misplaced (8.904%) 2023-02-06 09:00:00 pgs: 59285766/686306445 objects misplaced (8.638%) 2023-02-06 12:30:00 pgs: 57626520/686387322 objects misplaced (8.396%) 2023-02-06 14:00:00 pgs: 56813753/686388870 objects misplaced (8.277%) 2023-02-06 14:30:00 pgs: 56578343/686389185 objects misplaced (8.243%) 2023-02-06 20:30:00 pgs: 54255700/686969448 objects misplaced (7.898%) 2023-02-07 00:00:00 pgs: 53089526/687266865 objects misplaced (7.725%) 2023-02-07 01:00:00 pgs: 52810518/687391707 objects misplaced (7.683%) 2023-02-07 11:00:00 pgs: 50169973/688319189 objects misplaced (7.289%) 2023-02-07 16:30:00 pgs: 49312642/689410730 objects misplaced (7.153%) 2023-02-07 22:30:00 pgs: 48244798/689771838 objects misplaced (6.994%) 2023-02-08 15:30:00 pgs: 38058106/691166947 objects misplaced (5.506%) 2023-02-08 21:30:00 pgs: 62851681/691618387 objects misplaced (9.088%) <<< increase 2023-02-09 07:30:00 pgs: 57693559/693214621 objects misplaced (8.323%) 2023-02-09 13:03:27 ceph balancer off 2023-02-09 22:30:00 pgs: 48248474/694618590 objects misplaced (6.946%) 2023-02-10 14:00:00 pgs: 37872501/696136659 objects misplaced (5.440%) 2023-02-10 19:00:00 pgs: 64062339/696645992 objects misplaced (9.196%) <<< increase 2023-02-11 04:30:00 pgs: 52183052/699438360 objects misplaced (7.461%) 2023-02-11 19:00:00 pgs: 49167191/699631520 objects misplaced (7.028%) 2023-02-12 12:00:00 pgs: 39485735/700248360 objects misplaced (5.639%) 2023-02-12 18:00:00 pgs: 63540276/700256808 objects misplaced (9.074%) <<< increase 2023-02-12 20:30:00 pgs: 61766426/700291865 objects misplaced (8.820%) 2023-02-13 04:00:00 pgs: 57622457/700365805 objects misplaced (8.227%) --- annotated log extract --- ## ## "misplaced" decreasing... ## Feb 12 17:50:20 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:18.782191+0000 mgr.b4 (mgr.17297882) 4129637 : cluster [DBG] pgmap v4094135: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 178 op/s; 35014018/700256808 objects misplaced (5.000%); 34 MiB/s, 34 objects/s recovering Feb 12 17:50:20 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:18.782191+0000 mgr.b4 (mgr.17297882) 4129637 : cluster [DBG] pgmap v4094135: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 178 op/s; 35014018/700256808 objects misplaced (5.000%); 34 MiB/s, 34 objects/s recovering Feb 12 17:50:27 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:26.833542+0000 mgr.b4 (mgr.17297882) 4129641 : cluster [DBG] pgmap v4094141: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 2.7 MiB/s rd, 4.9 MiB/s wr, 226 op/s; 35012325/700256808 objects misplaced (5.000%); 37 MiB/s, 36 objects/s recovering Feb 12 17:50:30 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:28.848020+0000 mgr.b4 (mgr.17297882) 4129642 : cluster [DBG] pgmap v4094143: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 436 op/s; 35011830/700256808 objects misplaced (5.000%); 36 MiB/s, 35 objects/s recovering ## ## shortly after, "misplaced" has increased, followed by the "starting backfill" lines ## I'm not sure if the line ordering here is a quirk of journalctl, maybe the "starting backfill" came first which would make some sense ## Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:30.860404+0000 mgr.b4 (mgr.17297882) 4129643 : cluster [DBG] pgmap v4094145: 5945 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5934 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 419 KiB/s rd, 1.2 MiB/s wr, 388 op/s; 44579289/700256808 objects misplaced (6.366%); 31 MiB/s, 30 objects/s recovering Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.031822+0000 osd.64 (osd.64) 362 : cluster [DBG] 33.2cs0 starting backfill to osd.25(4) from (0'0,0'0] MAX to 639282'24881196 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.031951+0000 osd.64 (osd.64) 363 : cluster [DBG] 33.6cs0 starting backfill to osd.25(4) from (0'0,0'0] MAX to 639282'24593871 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.044285+0000 osd.64 (osd.64) 364 : cluster [DBG] 33.6cs0 starting backfill to osd.51(5) from (0'0,0'0] MAX to 639282'24593871 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.044655+0000 osd.64 (osd.64) 365 : cluster [DBG] 33.2cs0 starting backfill to osd.51(5) from (0'0,0'0] MAX to 639282'24881196 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.057888+0000 osd.64 (osd.64) 366 : cluster [DBG] 33.6cs0 starting backfill to osd.63(7) from (0'0,0'0] MAX to 639282'24593871 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.058039+0000 osd.64 (osd.64) 367 : cluster [DBG] 33.2cs0 starting backfill to osd.63(7) from (0'0,0'0] MAX to 639282'24881196 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.077517+0000 osd.64 (osd.64) 368 : cluster [DBG] 33.6cs0 starting backfill to osd.68(2) from (0'0,0'0] MAX to 639282'24593871 ## (more "starting backfill" lines elided here) ## ## "misplaced" continues to increase rapidly to 9%... ## Feb 12 17:50:34 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:32.871242+0000 mgr.b4 (mgr.17297882) 4129644 : cluster [DBG] pgmap v4094148: 5945 pgs: 2 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5932 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 424 KiB/s rd, 1.2 MiB/s wr, 440 op/s; 54133989/700256808 objects misplaced (7.731%); 31 MiB/s, 31 objects/s recovering Feb 12 17:50:36 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:34.886503+0000 mgr.b4 (mgr.17297882) 4129645 : cluster [DBG] pgmap v4094149: 5945 pgs: 2 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5932 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 442 KiB/s rd, 1.1 MiB/s wr, 483 op/s; 54133670/700256808 objects misplaced (7.731%); 41 MiB/s, 41 objects/s recovering Feb 12 17:50:37 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:36.901531+0000 mgr.b4 (mgr.17297882) 4129646 : cluster [DBG] pgmap v4094150: 5945 pgs: 4 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 281 KiB/s rd, 902 KiB/s wr, 324 op/s; 63666090/700256808 objects misplaced (9.092%); 17 MiB/s, 17 objects/s recovering ## ## then starts to fall again... ## Feb 12 17:50:42 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:40.930869+0000 mgr.b4 (mgr.17297882) 4129648 : cluster [DBG] pgmap v4094152: 5945 pgs: 4 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 407 KiB/s rd, 1.4 MiB/s wr, 479 op/s; 63665652/700256808 objects misplaced (9.092%); 28 MiB/s, 28 objects/s recovering Feb 12 17:50:43 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:42.948421+0000 mgr.b4 (mgr.17297882) 4129649 : cluster [DBG] pgmap v4094153: 5945 pgs: 3 active+remapped+backfilling, 12 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 455 KiB/s rd, 1.5 MiB/s wr, 542 op/s; 63665652/700256808 objects misplaced (9.092%); 27 MiB/s, 26 objects/s recovering Feb 12 17:50:46 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:44.961474+0000 mgr.b4 (mgr.17297882) 4129650 : cluster [DBG] pgmap v4094154: 5945 pgs: 3 active+remapped+backfilling, 12 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 436 KiB/s rd, 1.4 MiB/s wr, 518 op/s; 63664579/700256808 objects misplaced (9.092%); 37 MiB/s, 37 objects/s recovering Feb 12 17:50:50 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:48.991308+0000 mgr.b4 (mgr.17297882) 4129652 : cluster [DBG] pgmap v4094156: 5945 pgs: 3 active+remapped+backfilling, 12 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 513 KiB/s rd, 1.6 MiB/s wr, 534 op/s; 63664502/700256808 objects misplaced (9.092%); 33 MiB/s, 33 objects/s recovering ------------------------------------------------------------------------------

1 year, 2 months

2
2
0 0

Quincy: Stuck on image permissions

by hicks＠cgi.cz

Hello guys, could someone help me with this? We've been long-time CEPH users... runing several Mimic + Pacific CEPH clusters. Dozens of disk per cluster, typically. BUT... now I have this brand new Quincy cluster and I'm not able to give CLIENT (Quincy on Rocky 8) rw access to ONE IMAGE on Quincy cluster (cephadm / Rocky 9). I'm using something what worked for us for ages: rbd auth ls: client.xxx key: ... caps: [mon] profile rbd caps: [osd] allow rwx pool prod object_prefix rbd_data.600d1c6723ae; allow rwx pool prod object_prefix rbd_header.600d1c6723ae; allow rx pool prod object_prefix rbd_id.xxx-data rbd info: rbd image 'xxx-data': size 2 TiB in 524288 objects order 22 (4 MiB objects) snapshot_count: 2 id: 600d1c6723ae block_name_prefix: rbd_data.600d1c6723ae format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: rados ls: rbd_data.600d1c6723ae.000000000003958d rbd_header.600d1c6723ae rbd_id.xxx-data BUT... it DOES NOT WORK. When I try it to map on client it says: 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::GetMetadataRequest: 0x7f3a1c001f40 handle_metadata_list: failed to retrieve image metadata: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::RefreshRequest: failed to retrieve pool metadata: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::OpenRequest: failed to refresh image: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::ImageState: 0x555eff78cfc0 failed to open image: (1) Operation not permitted rbd: error opening image xxx-data: (1) Operation not permitted The mapping and access DOES work when I put "osd allow *" into ceph auth. What is the recommended syntax for Quincy? btw: this use case should be mentioned in the manual I think... Thanks!

1 year, 2 months

2
1
0 0

Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1

by duluxoz

Hi All, Sorry if this was mentioned previously (I obviously missed it if it was) but can we upgrade a Ceph Quincy Host/Cluster from Rocky Linux (RHEL) v8.6/8.7 to v9.1 (yet), and if so, what is / where can I find the procedure to do this - ie is there anything "special" that needs to be done because of Ceph, or can we just do a "simple" v8.x +> v9.1 upgrade? Thanks in advance Cheers Dulux-Oz

1 year, 2 months

3
6
0 0

Re: Exit yolo mode by increasing size/min_size does not (really) work

by stefan

ah, thank you so much Eugen! this makes sense! i will report what we will have changed and if it worked or not :) i wish you a nice weekend!

1 year, 2 months

1
0
0 0

Exit yolo mode by increasing size/min_size does not (really) work

by Stefan Pinter

Hi! 😊 It would be very kind of you to help us with that! We have pools in our ceph cluster that are set to replicated size 2 min_size 1. Obviously we want to go to size 3 / min_size 2 but we experience problems with that. USED goes to 100% instantly and MAX AVAIL goes to 0. Write operations seemed to stop. POOLS: NAME ID USED %USED MAX AVAIL OBJECTS Pool1 24 35791G 35.04 66339G 8927762 Pool2 25 11610G 14.89 66339G 3004740 Pool3 26 17557G 100.00 0 2666972 Before the change it was like this: NAME ID USED %USED MAX AVAIL OBJECTS Pool1 24 35791G 35.04 66339G 8927762 Pool2 25 11610G 14.89 66339G 3004740 Pool3 26 17558G 20.93 66339G 2667013 This was quite surprising to us as we’d expect USED to go to something like 30%. Going back to 2/1 also gave us back the 20.93% usage instantly. What’s the matter here? Thank you and best regards Stefan ________________________________ BearingPoint GmbH Sitz: Wien Firmenbuchgericht: Handelsgericht Wien Firmenbuchnummer: FN 175524z The information in this email is confidential and may be legally privileged. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system.

1 year, 2 months

3
5
0 0

issue in connecting Openstack(Kolla-ansible) manila with external ceph (cephadm)

by Haitham Abdulaziz

i deployed kolla-ansible & cephadm on virtual machines (kvm) . My ceph cluster is on 3 vms with 12 vCPU each and 24gb of ram i used cephadm to deploy ceph ceph -s : -------------------------- cluster: id: a0e5ad36-a54c-11ed-9aea-5254008c2a3e health: HEALTH_OK services: mon: 3 daemons, quorum ceph0,ceph1,ceph2 (age 6h) mgr: ceph0.dzutak(active, since 24h), standbys: ceph1.aizuyc mds: 3/3 daemons up, 6 standby osd: 9 osds: 9 up (since 24h), 9 in (since 24h) data: volumes: 3/3 healthy pools: 9 pools, 257 pgs objects: 70 objects, 7.3 KiB usage: 76 MiB used, 780 GiB / 780 GiB avail pgs: 257 active+clean -------------------------- my openstack deployment is AIO on a single node , now i wanna link them together so i started with manila & native cephfs thinking its the easist following this doc : https://docs.openstack.org/manila/latest/admin/cephfs_driver.html#authorizi… i created the user -------------------------- client.manila key: AQC7ot9jfiDsIxAA57fb7S6bVMnr5IadsnukHQ== caps: [mgr] allow rw caps: [mon] allow r caps: [osd] allow rw pool=ganesha_rados_store and created a file system called manila -------------------------- my ceph.conf -------------------------- [global] fsid = a0e5ad36-a54c-11ed-9aea-5254008c2a3e mon_host = [v2:192.168.122.25:3300/0,v1:192.168.122.25:6789/0] [v2:192.168.122.115:3300/0,v1:192.168.122.115:6789/0] [v2:192.168.122.14:3300/0,v1:192.168.122.14:6789/0] -------------------------- i moved the files as to the openstack node and trying to connect them together but it didnt go will , Viewing the logs shows -------------------------- <AIO@cephfsnative1: manila.exception.ShareBackendException: json_command failed - prefix=fs volume ls, argdict={'format': 'json'} - exception message: Bad target type 'mon-mgr'. --------------------------where should i start to fix this issue ?

1 year, 2 months

2
1
0 0

RadosGW - Performance Expectations

by Shawn Weeks

Good morning everyone, been running a small Ceph cluster with Proxmox for a while now and I’ve finally run across an issue I can’t find any information on. I have a 3 node cluster with 9 Samsung PM983 960GB NVME drives running on a dedicated 10gb network. RBD and CephFS performance have been great, most of the time I see over 500MBs writes and a rados benchmark shows 951 MB/s write and 1140 MB/s read bandwidth. The problem I’m seeing is after setting up RadosGW I can only upload to “S3” at around 25MBs with the official AWS CLI. Using s3cmd is slightly better at around 45MB/s. I’m going directly to the RadosGW instance with no load balancers in between and no ssl enabled. Just trying to figure out if this is normal. I’m not expecting it to be as fast as writing directly to a RBD but I was kinda hoping for more than this. So what should I expect in performance from the RadosGW? Here are some rados bench results and my ceph report https://gist.github.com/shawnweeks/f6ef028284b5cdb10d80b8dc0654eec5 https://gist.github.com/shawnweeks/7cfe94c08adbc24f2a3d8077688df438 Thanks Shawn

1 year, 2 months

4
5
0 0

No such file or directory when issuing "rbd du"

by Mehmet

Hello Friends, i have a strange output when issuing following command root@node35:~# rbd du -p cephhdd-001-mypool NAME PROVISIONED USED ... vm-99936587-disk-0@H202302091535 400 GiB 5.2 GiB vm-99936587-disk-0@H202302091635 400 GiB 1.2 GiB vm-99936587-disk-0 400 GiB 732 MiB vm-9999104-cloudinit 4 MiB 4 MiB vm-9999104-disk-0 600 GiB 586 GiB <TOTAL> 49 TiB 44 TiB rbd: du failed: (2) No such file or directory root@node35:~# I do not know why i receive "rbd: du failed: (2) No such file or directory". How can i find the origin for this? My Ceph-Version 17.2.3 installed with "cephadm". Cluster is "HEALTH_OK" with 108 OSDs distributed over 3 Nodes where mgr/mon also resides. Hope you can help Mehmet

1 year, 2 months

2
2
0 0

Yet another question about OSD memory usage ...

by Ulrich Klein

Hi, Yet another question about OSD memory usage ... I have a test cluster running. When I do a ceph orch ps I see for my osd.11: ceph orch ps --refresh NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.11 ceph01 running (2h) 97s ago 2h 23.0G 13.1G 17.2.5 cc65afd6173a 5d1062e8d392 When I chek via top on the machine I see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 39807 ceph 20 0 6254956 3.7g 9228 S 31.2 3.0 846:21.63 /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-lo+ Now, where does ceph orch ps get those 23.0G from, when top just shows 3.7G resident and 6.2G virtual for osd.11? (I do understand that the MEM LIM n the ceph orch ps list is not really the limit) Anyone know where that discrepancy comes from? Ciao, Uli

1 year, 2 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users February 2023