- ceph-users - lists.ceph.io

Re: another osd_pglog memory usage incident
by Dan van der Ster 09 Oct '20

09 Oct '20

On Fri, Oct 9, 2020 at 3:12 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > > > >1. The pg log contains 3000 entries by default (on nautilus). These > >3000 entries can legitimately consume gigabytes of ram for some > >use-cases. (I haven't determined exactly which ops triggered this > >today). > > How can I check how much ram my pg_logs are using? ceph daemon osd.x dump_mempools | jq .mempool.by_pool.osd_pglog > > > > -----Original Message----- > Cc: ceph-users > Subject: [ceph-users] Re: another osd_pglog memory usage incident > > On 09.10.20 13:55, Dan van der Ster wrote: > [...] > > I also noticed a possible relationship with scrubbing -- One week ago > > we increased to osd_max_scrubs=5 to clear out a scrubbing backlog; I > > wonder if the increased read/write ratio somehow led to an exploding > > buffer_anon. Do things stabilize on your side if you temporarily > > disable scrubbing? > > During the worst periods, we had disabled scrubbing. When we re-enabled, > we had our write-job to mitigate the problems. And currently, scrub load > is low. So I cannot tell, but it is very plausible. > > Cheers > Harry > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

1 0

Multisite replication speed
by Nicolas Moal 09 Oct '20

09 Oct '20

Hello everybody, We have two Ceph object clusters replicating over a very long-distance WAN link. Our version of Ceph is 14.2.10. Currently, replication speed seems to be capped around 70 MiB/s even if there's a 10Gb WAN link between the two clusters. The clusters themselves don't seem to suffer from any performance issue. The replication traffic leverages HAProxy VIPs, which means there's a single endpoint (the HAProxy VIP) in the multisite replication configuration. So, my questions are: - Is it possible to improve replication speed by adding more endpoints in the multisite replication configuration? The issue we are facing is that the secondary cluster is way behind the master cluster because of the relatively slow speed. - Is there anything else I can do to optimize replication speed ? Thanks for your comments ! Nicolas

3 4

another osd_pglog memory usage incident
by Dan van der Ster 09 Oct '20

09 Oct '20

Hi all, This morning some osds in our S3 cluster started going OOM, after restarting them I noticed that the osd_pglog is using >1.5GB per osd. (This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all PGs are active+clean). After reading through this list and trying a few things, I'd like to share the following observations for your feedback: 1. The pg log contains 3000 entries by default (on nautilus). These 3000 entries can legitimately consume gigabytes of ram for some use-cases. (I haven't determined exactly which ops triggered this today). 2. The pg log length is decided by the primary osd -- setting osd_max_pg_log_entries/osd_min_pg_log_entries on one single OSD does not have a big effect (because most of the PGs are primaried somewhere else). You need to set it on all the osds for it to be applied to all PGs. 3. We eventually set osd_max_pg_log_entries = 500 everywhere. This decreased the osd_pglog mempool from more than 1.5GB on our largest osds to less that 500MB. 4. The osd_pglog mempool is not accounted for in the osd_memory_target (in nautilus). 5. I have opened a feature request to limit the pg_log length by memory size (https://tracker.ceph.com/issues/47775) This way we could allocate a fraction of memory to the pg log and it would shorten the pglog length (budget) accordingly. 6. Would it be feasible to add an osd option to 'trim pg log at boot' ? This way we could avoid the cumbersome ceph-objectstore-tool trim-pg-log in cases of disaster (osds going oom at boot). For those that had pglog memory usage incidents -- does this match your experience? Thanks! Dan

4 7

Nautilus RGW fails to open Jewel buckets (400 Bad Request)
by Wido den Hollander 09 Oct '20

09 Oct '20

Hi, Most of it is described here: https://tracker.ceph.com/issues/22928 Buckets created under Jewel don't always have the *placement_rule* set in their bucket metadata and this causes Nautilus RGWs to not serve requests for them. Snippet from the metadata: { "key": "bucket.instance:pbx:ams02.446941181.1", "ver": { "tag": "86lc3iVtQpPiJYkh95YCTnhu", "ver": 2 }, "mtime": "2020-10-09 09:12:04.744423Z", "data": { "bucket_info": { "bucket": { "name": "pbx", "marker": "ams02.241978.4", "bucket_id": "ams02.446941181.1", "tenant": "", "explicit_placement": { "data_pool": ".rgw.buckets", "data_extra_pool": "", "index_pool": ".rgw.buckets" } }, "creation_time": "2014-02-16 12:32:15.000000Z", "owner": "vdvm", "flags": 0, "zonegroup": "eu", "placement_rule": "", Notice that *placement_rule* is empty and that this bucket has *explicit_placement* set. There is no way to update the bucket.instance metadata as far as I know, otherwise I could have set a placement rule for the bucket. Earlier on the ML this has been discussed: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ULKK5RU2VXL… People there compiled a manual version of RGW, something I'd rather stay away from. Has anybody seen this and if so: Have you found a solution? The commit that breaks these buckets is this one: https://github.com/ceph/ceph/commit/2a8e8a98d8c56cc374ec671846a20e2b0484bc75 14.2.0 was the first release with that code in there. So two things I'm thinking about and I don't know which one is best: - Update RGW and modify the if-statement added by commit 2a8e8a - Enhance 'bucket check --fix' to update the placement_rule if none is set for a bucket Any hints or suggestions? Wido

1 0

Weird performance issue with long heartbeat and slow ops warnings
by Void Star Nill 09 Oct '20

09 Oct '20

Hello, I have a ceph cluster running 14.2.11. I am running benchmark tests with FIO concurrently on ~2000 volumes of 10G each. During the time initial warm-up FIO creates a 10G file on each volume before it runs the actual read/write I/O operations. During this time, I start seeing the Ceph cluster reporting about 35GiB/s write throughput for a while, but after some time I start seeing "long heartbeat" and "slow ops" warnings and in a few mins the throughput drops to ~1GB/s and stays there until all FIO runs complete. The cluster has 5 monitor nodes and 10 data nodes - each with 10x3.2TB NVME drives. I have setup 3 OSD for each NVME, so there are a total of 300 OSDs. Each server has 200GB uplink and there's no apparent network bottleneck as the network is set up to support over 1Tbps bandwidth. I dont see any CPU or memory issues also on the servers. There is a single manager instance running on one of the mons. The pool is configured for 3 replication factor with min_size of 2. I tried to use pg_num of 8192 and 16384 and saw the issue with both settings. Could you please suggest if this is a known issue or if I can tune any parameters? Long heartbeat ping times on back interface seen, longest is 1202.120 msec Long heartbeat ping times on front interface seen, longest is 1535.191 msec 35 slow ops, oldest one blocked for 122 sec, daemons [osd.135,osd.14,osd.141,osd.143,osd.149,osd.15,osd.151,osd.153,osd.157,osd.162]... have slow ops. Regards, Shridhar

2 2

pg active+clean but can not handle io
by 古轶特 09 Oct '20

09 Oct '20

Hi all: I have a ceph cluster, the version is 12.2.12. this is my ceph osd tree: [root@node-1 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -25 2.78760 root rack-test -26 0.92920 rack rack_1 -37 0.92920 host host_1 12 ssd 0.92920 osd.12 up 1.00000 1.00000 -27 0.92920 rack rack_2 -38 0.92920 host host_2 6 ssd 0.92920 osd.6 up 1.00000 1.00000 -28 0.92920 rack rack_3 -39 0.92920 host host_3 18 ssd 0.92920 osd.18 up 1.00000 1.00000 I have a pool in cluster: pool 14 'gyt-test' replicated size 2 min_size 1 crush_rule 2 object_hash rjenkins pg_num 128 pgp_num 128 last_change 5864 lfor 0/5828 flags hashpspool stripe_width 0 removed_snaps [1~3] crush_rule 2 dump: { "rule_id": 2, "rule_name": "replicated_rule_rack", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -25, "item_name": "rack-test" }, { "op": "chooseleaf_firstn", "num": 0, "type": "rack" }, { "op": "emit" } ] } I have a rbd images in gyt-test pool: [root@node-1 ~]# rbd ls gyt-test gyt-test now, I use fio tool to test this rbd images: [root@node-1 ~]# fio --ioengine=rbd --pool=gyt-test --rbdname=gyt-test --rw=randwrite --bs=4k --numjobs=1 --runtime=120 --iodepth=128 --clientname=admin --direct=1 --name=test --time_based=1 --eta-newline 1 test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=128 fio-3.1 Starting 1 process Jobs: 1 (f=1): [w(1)][2.5%][r=0KiB/s,w=42.2MiB/s][r=0,w=10.8k IOPS][eta 01m:57s] Jobs: 1 (f=1): [w(1)][4.2%][r=0KiB/s,w=57.7MiB/s][r=0,w=14.8k IOPS][eta 01m:55s] Jobs: 1 (f=1): [w(1)][5.8%][r=0KiB/s,w=52.4MiB/s][r=0,w=13.4k IOPS][eta 01m:53s] Jobs: 1 (f=1): [w(1)][7.5%][r=0KiB/s,w=61.1MiB/s][r=0,w=15.6k IOPS][eta 01m:51s] Jobs: 1 (f=1): [w(1)][9.2%][r=0KiB/s,w=30.0MiB/s][r=0,w=7927 IOPS][eta 01m:49s] Jobs: 1 (f=1): [w(1)][10.8%][r=0KiB/s,w=59.1MiB/s][r=0,w=15.1k IOPS][eta 01m:47s] Jobs: 1 (f=1): [w(1)][12.5%][r=0KiB/s,w=51.6MiB/s][r=0,w=13.2k IOPS][eta 01m:45s] Jobs: 1 (f=1): [w(1)][14.2%][r=0KiB/s,w=58.3MiB/s][r=0,w=14.9k IOPS][eta 01m:43s] Jobs: 1 (f=1): [w(1)][15.8%][r=0KiB/s,w=56.1MiB/s][r=0,w=14.4k IOPS][eta 01m:41s] Jobs: 1 (f=1): [w(1)][17.5%][r=0KiB/s,w=44.8MiB/s][r=0,w=11.5k IOPS][eta 01m:39s] This is normal And then, I move host_1 bucket to rack-test: [root@node-1 ~]# ceph osd crush move host_1 root=rack-test moved item id -37 name 'host_1' to location {root=gyt-test} in crush map [root@node-1 ~]# ceph osd tree -25 2.78760 root rack-test -37 0.92920 host host_1 12 ssd 0.92920 osd.12 up 1.00000 1.00000 -26 0 rack rack_1 -27 0.92920 rack rack_2 -38 0.92920 host host_2 6 ssd 0.92920 osd.6 up 1.00000 1.00000 -28 0.92920 rack rack_3 -39 0.92920 host host_3 18 ssd 0.92920 osd.18 up 1.00000 1.00000 use fio tool test gyt-test rbd again: [root@node-1 ~]# fio --ioengine=rbd --pool=gyt-test --rbdname=gyt-test --rw=randwrite --bs=4k --numjobs=1 --runtime=120 --iodepth=64 --clientname=admin --direct=1 --name=test --time_based=1 --eta-newline 1 test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=64 fio-3.1 Starting 1 process Jobs: 1 (f=1): [w(1)][2.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:57s] Jobs: 1 (f=1): [w(1)][4.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:55s] Jobs: 1 (f=1): [w(1)][5.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:53s] Jobs: 1 (f=1): [w(1)][7.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:51s] Jobs: 1 (f=1): [w(1)][9.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:49s] Jobs: 1 (f=1): [w(1)][10.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:47s] Jobs: 1 (f=1): [w(1)][12.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:45s] Jobs: 1 (f=1): [w(1)][14.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:43s] Jobs: 1 (f=1): [w(1)][15.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:41s] Jobs: 1 (f=1): [w(1)][17.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:39s] Jobs: 1 (f=1): [w(1)][19.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:37s] Jobs: 1 (f=1): [w(1)][20.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:35s] Jobs: 1 (f=1): [w(1)][22.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:33s] Jobs: 1 (f=1): [w(1)][24.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:31s] Jobs: 1 (f=1): [w(1)][25.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:29s] Jobs: 1 (f=1): [w(1)][27.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:27s] Jobs: 1 (f=1): [w(1)][29.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:25s] Jobs: 1 (f=1): [w(1)][30.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:23s] Jobs: 1 (f=1): [w(1)][32.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:21s] Jobs: 1 (f=1): [w(1)][34.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:19s] Jobs: 1 (f=1): [w(1)][35.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:17s] Jobs: 1 (f=1): [w(1)][37.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:15s] Jobs: 1 (f=1): [w(1)][39.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:13s] Jobs: 1 (f=1): [w(1)][40.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:11s] rbd gyt-test can not handle io ceph status at this time: [root@node-1 ~]# ceph -s cluster: id: 896eeeed-6f14-4fb0-8d0d-00adda67a36c health: HEALTH_WARN 8782/299948 objects misplaced (2.928%) services: mon: 3 daemons, quorum node-1,node-2,node-3 mgr: node-1(active) osd: 18 osds: 18 up, 18 in; 75 remapped pgs flags nodeep-scrub rbd-mirror: 1 daemon active rgw: 1 daemon active data: pools: 14 pools, 1024 pgs objects: 104.25k objects, 371GiB usage: 641GiB used, 11.5TiB / 12.1TiB avail pgs: 8782/299948 objects misplaced (2.928%) 949 active+clean 75 active+clean+remapped io: client: 106KiB/s rd, 567KiB/s wr, 127op/s rd, 52op/s wr my all pgs is active+clean, remapped pg is already stop recovery, this is no normal. [root@node-1 ~]# ceph pg dump | grep ^14 | grep "\[12" dumped all 14.37 106 0 0 0 0 441110528 1733 1733 active+clean 2020-09-29 00:52:21.808536 6226'33265 6278:34441 [12,18] 12 [12,18] 12 0'0 2020-09-28 11:35:49.973497 0'0 2020-09-28 11:31:12.912584 0 14.1c 93 0 0 0 0 383131648 1512 1512 active+clean 2020-09-29 00:45:18.475265 6226'28692 6278:29883 [12,6] 12 [12,6] 12 0'0 2020-09-28 11:34:55.543085 0'0 2020-09-28 11:31:12.912584 0 14.10 82 0 0 0 0 340963328 1505 1505 active+clean 2020-09-29 00:45:18.474759 6226'26031 6278:27257 [12,6] 12 [12,6] 12 0'0 2020-09-28 11:34:37.400091 0'0 2020-09-28 11:31:12.912584 0 14.2c 85 0 0 0 0 353435648 1677 1677 active+clean 2020-09-29 00:51:24.528368 6226'25262 6278:26344 [12,18] 12 [12,18] 12 0'0 2020-09-28 11:35:28.806418 0'0 2020-09-28 11:31:12.912584 0 14.26 106 0 0 0 0 440807424 1534 1534 active+clean 2020-09-29 00:45:18.474631 6226'33003 6278:34306 [12,6] 12 [12,6] 12 0'0 2020-09-28 11:35:01.590900 0'0 2020-09-28 11:31:12.912584 0 14.6a 78 0 0 0 0 323850240 1671 1671 active+clean 2020-09-29 00:50:57.336812 6226'23954 6278:25009 [12,18] 12 [12,18] 12 0'0 2020-09-28 11:36:06.101612 0'0 2020-09-28 11:31:12.912584 0 these pg primary osd is 12, I feel it unable to understand that osd 12 already move out gyt-test pool crush rule. it seems a BUG?

1 0

Re: el6 / centos6 rpm's for luminous?
by Dan van der Ster 09 Oct '20

09 Oct '20

We had built some rpms locally for ceph-fuse, but AFAIR luminous needs systemd so the server rpms would be difficult. -- dan On Thu, Oct 8, 2020 at 11:12 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > Nobody ever used luminous on el6? > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Wipe an Octopus install
by Samuel Taylor Liston 08 Oct '20

08 Oct '20

Wondering if anyone knows or has put together a way to wipe an Octopus install? I’ve looked for documentation on the process, but if it exists, I haven’t found it yet. I’m going through some test installs - working through the ins and outs of cephadm and containers and would love an easy way to tear things down and start over. In previous releases managed through ceph-deploy there were three very convenient commands that nuked the world. I am looking for something as complete for Octopus. Thanks, Sam Liston (sam.liston(a)utah.edu) ========================================== Center for High Performance Computing - Univ. of Utah 155 S. 1452 E. Rm 405 Salt Lake City, Utah 84112 (801)232-6932 ==========================================

3 6

Fwd: pool pgp_num not updated
by Mac Wynkoop 08 Oct '20

08 Oct '20

Just making sure this makes the list: Mac Wynkoop ---------- Forwarded message --------- From: 胡玮文 <huww98(a)outlook.com> Date: Wed, Oct 7, 2020 at 9:00 PM Subject: Re: pool pgp_num not updated To: Mac Wynkoop <mwynkoop(a)netdepot.com> Hi, You can read about this behavior at https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/ In short, ceph will not increase pgp_num if misplaced > 5% (by default), and once you got misplaced < 5%, it will increase pgp_num gradually, until reaching the value you set. This 5% can be configured by target_max_misplaced_ratio config option. 在 2020年10月8日，03:22，Mac Wynkoop <mwynkoop(a)netdepot.com> 写道： Well, backfilling sure, but will it allow me to actually change the pgp_num as more space frees up? Because the issue is that I cannot modify that value. Thanks, Mac Wynkoop, Senior Datacenter Engineer *NetDepot.com:* Cloud Servers; Delivered Houston | Atlanta | NYC | Colorado Springs 1-844-25-CLOUD Ext 806 On Wed, Oct 7, 2020 at 1:50 PM Eugen Block <eblock(a)nde.ag> wrote: Yes, I think that’s exactly the reason. As soon as the cluster has more space the backfill will continue. Zitat von Mac Wynkoop <mwynkoop(a)netdepot.com>: The cluster is currently in a warn state, here's the scrubbed output of ceph -s: *cluster: id: *redacted* health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 22 nearfull osd(s) 2 pool(s) nearfull Low space hindering backfill (add storage if this doesn't resolve itself): 277 pgs backfill_toofull Degraded data redundancy: 32652738/3651947772 objects degraded (0.894%), 281 pgs degraded, 341 pgs undersized 1214 pgs not deep-scrubbed in time 2647 pgs not scrubbed in time 2 daemons have recently crashed services: mon: 5 daemons, *redacted* (age 44h) mgr: *redacted* osd: 162 osds: 162 up (since 44h), 162 in (since 4d); 971 remapped pgs flags noscrub,nodeep-scrub rgw: 3 daemons active *redacted* tcmu-runner: 18 daemons active *redacted* data: pools: 10 pools, 2648 pgs objects: 409.56M objects, 738 TiB usage: 1.3 PiB used, 580 TiB / 1.8 PiB avail pgs: 32652738/3651947772 objects degraded (0.894%) 517370913/3651947772 objects misplaced (14.167%) 1677 active+clean 477 active+remapped+backfill_wait 100 active+remapped+backfill_wait+backfill_toofull 80 active+undersized+degraded+remapped+backfill_wait 60 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 42 active+undersized+degraded+remapped+backfill_toofull 33 active+undersized+degraded+remapped+backfilling 25 active+remapped+backfilling 25 active+remapped+backfill_toofull 24 active+undersized+remapped+backfilling 23 active+forced_recovery+undersized+degraded+remapped+backfill_wait 19 active+forced_recovery+undersized+degraded+remapped+backfill_wait+backfill_toofull 15 active+undersized+remapped+backfill_wait 14 active+undersized+remapped+backfill_wait+backfill_toofull 12 active+forced_recovery+undersized+degraded+remapped+backfill_toofull 12 active+forced_recovery+undersized+degraded+remapped+backfilling 5 active+undersized+remapped+backfill_toofull 3 active+remapped 1 active+undersized+remapped 1 active+forced_recovery+undersized+remapped+backfilling io: client: 287 MiB/s rd, 40 MiB/s wr, 1.94k op/s rd, 165 op/s wr recovery: 425 MiB/s, 225 objects/s* Now as you can see, we do have a lot of backfill operations going on at the moment. Does that actually prevent Ceph from modifying the pgp_num value of a pool? Thanks, Mac Wynkoop On Wed, Oct 7, 2020 at 8:57 AM Eugen Block <eblock(a)nde.ag> wrote: What is the current cluster status, is it healthy? Maybe increasing pg_num would hit the limit of mon_max_pg_per_osd? Can you share 'ceph -s' output? Zitat von Mac Wynkoop <mwynkoop(a)netdepot.com>: Right, both Norman and I set the pg_num before the pgp_num. For example, here is my current pool settings: *"pool 40 '*redacted*.rgw.buckets.data' erasure size 9 min_size 7 crush_rule 2 object_hash rjenkins pg_num 2048 pgp_num 1024 pgp_num_target 2048 last_change 8458830 lfor 0/0/8445757 flags hashpspool,ec_overwrites,nodelete,backfillfull stripe_width 24576 fast_read 1 application rgw"* So, when I set: "*ceph osd pool set hou-ec-1.rgw.buckets.data pgp_num 2048*" it returns: "*set pool 40 pgp_num to 2048*" But upon checking the pool details again: "*pool 40 '*redacted*.rgw.buckets.data' erasure size 9 min_size 7 crush_rule 2 object_hash rjenkins pg_num 2048 pgp_num 1024 pgp_num_target 2048 last_change 8458870 lfor 0/0/8445757 flags hashpspool,ec_overwrites,nodelete,backfillfull stripe_width 24576 fast_read 1 application rgw*" and the pgp_num value does not increase. Am I just doing something totally wrong? Thanks, Mac Wynkoop On Tue, Oct 6, 2020 at 2:32 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: pg_num and pgp_num need to be the same, not? 3.5.1. Set the Number of PGs To set the number of placement groups in a pool, you must specify the number of placement groups at the time you create the pool. See Create a Pool for details. Once you set placement groups for a pool, you can increase the number of placement groups (but you cannot decrease the number of placement groups). To increase the number of placement groups, execute the following: ceph osd pool set {pool-name} pg_num {pg_num} Once you increase the number of placement groups, you must also increase the number of placement groups for placement (pgp_num) before your cluster will rebalance. The pgp_num should be equal to the pg_num. To increase the number of placement groups for placement, execute the following: ceph osd pool set {pool-name} pgp_num {pgp_num} https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/s… -----Original Message----- To: norman Cc: ceph-users Subject: [ceph-users] Re: pool pgp_num not updated Hi everyone, I'm seeing a similar issue here. Any ideas on this? Mac Wynkoop, On Sun, Sep 6, 2020 at 11:09 PM norman <norman.kern(a)gmx.com> wrote: Hi guys, When I update the pg_num of a pool, I found it not worked(no rebalanced), anyone know the reason? Pool's info: pool 21 'openstack-volumes-rs' replicated size 3 min_size 2 crush_rule 21 object_hash rjenkins pg_num 1024 pgp_num 512 pgp_num_target 1024 autoscale_mode warn last_change 85103 lfor 82044/82044/82044 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~1e6,1e8~300,4e9~18,502~3f,542~11,554~1a,56f~1d7] pool 22 'openstack-vms-rs' replicated size 3 min_size 2 crush_rule 22 object_hash rjenkins pg_num 512 pgp_num 512 pg_num_target 256 pgp_num_target 256 autoscale_mode warn last_change 84769 lfor 0/0/55294 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 application rbd The pgp_num_target is set, but pgp_num not set. I have scale out new OSDs and is backfilling before setting the value, is it the reason? _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Error "Operation not permitted" using rbd pool init command
by floda 08 Oct '20

08 Oct '20

Hi guys, I have stumbled over an error that I cannot solve after more than 2 days trying. Now I turn to you in hope to get some help! :) First of all, I am totally new to Ceph and this is just a test setup for now. I have followed the instructions on the ceph.com homepage as far as I understand. I am about to set up a Ceph storage cluster on three different nodes (running Ubuntu server 20.04). I am using Ceph Octopus. On these 3 nodes I have installed 3 monitors, 3 managers and 12 OSDs (I do know that OSDs and Monitors on the same machines are not recommended but for this test setup I do it anyway). So far is everything working well for me. However, when I try to create an RBD pool it fails with some strange (at least strange for me) error message: 2020-10-08T07:48:36.601+0000 7f8b9b7fe700 -1 librbd::image::GetMetadataRequest: 0x564279b9d670 handle_metadata_list: failed to retrieve image metadata: (1) Operation not permitted 2020-10-08T07:48:36.601+0000 7f8bb6af8380 -1 librbd::PoolMetadata: list: failed listing metadata: (1) Operation not permitted 2020-10-08T07:48:36.601+0000 7f8bb6af8380 -1 librbd::Config: apply_pool_overrides: failed to read pool config overrides: (1) Operation not permitted 2020-10-08T07:48:36.601+0000 7f8b9b7fe700 -1 librbd::image::ValidatePoolRequest: handle_read_rbd_info: failed to read RBD info: (1) Operation not permitted rbd: pool already registered to a different application. I run the commands as the Linux root user and as the Ceph user client.admin (I have turned off apparmor and other hardening things as well). The chep user client.admin has the following setup in its keyring: [client.admin] key = ..... caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" For more information regarding permissions in the system see the attached file permissions.txt that contains the output of the ceph auth list command. I have also attached a file (report.txt) containing the output from the ceph report command. (NOTE: I have removed some information from the files, i.e. keys, fsid, fingerprints, uuid, and replaced them with "..." instead) Can anyone please help me understand why I get this error and what I need to do in order to solve it? Thanks in advanced! Best regards, Fredrik

2 2