November 2020 - ceph-users

by Brent Kennedy

I recently setup a new octopus cluster and was testing the autoscale feature. Used ceph-ansible so its enabled by default. Anyhow, I have three other clusters that are on nautilus, so I wanted to see if it made sense to enable it there on the main pool. Here is a print out of the autoscale status: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE default.rgw.buckets.non-ec 0 2.0 55859G 0.0000 1.0 32 on default.rgw.meta 9298 3.0 55859G 0.0000 1.0 32 on default.rgw.buckets.index 18058M 3.0 55859G 0.0009 1.0 32 on default.rgw.control 0 3.0 55859G 0.0000 1.0 32 on default.rgw.buckets.data 9126G 2.0 55859G 0.3268 1.0 4096 1024 off .rgw.root 3155 3.0 55859G 0.0000 1.0 32 on rbd 155.5G 2.0 55859G 0.0056 1.0 32 on default.rgw.log 374.4k 3.0 55859G 0.0000 1.0 64 on For this entry: default.rgw.buckets.data 9126G 2.0 55859G 0.3268 1.0 4096 1024 off I have it disabled because it showed a warn message, but its recommending a 1024 PG setting. When I use the online ceph calculator at ceph.io, its saying the 4096 setting is correct. So why is autoscaler saying 1024? There are 6 osd servers with 10 OSDs each ( all SSD ). 60 TB total. Pool LS output: pool 1 '.rgw.root' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8800 lfor 0/0/344 flags hashpspool stripe_width 0 application rgw pool 2 'default.rgw.control' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8799 lfor 0/0/346 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.meta' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8798 lfor 0/0/350 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.log' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 8802 lfor 0/0/298 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.buckets.index' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 638 pgp_num 608 pg_num_target 32 pgp_num_target 32 autoscale_mode on last_change 10320 lfor 0/10320/10318 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 7 'default.rgw.buckets.data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 4096 pgp_num 4096 last_change 9467 lfor 0/0/552 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 8 'default.rgw.buckets.non-ec' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8797 lfor 0/0/348 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 9 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8801 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd Regards, -Brent Existing Clusters: Test: Ocotpus 15.2.5 ( all virtual on nvme ) US Production(HDD): Nautilus 14.2.11 with 11 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.11 with 18 osd servers, 3 mons, 4 gateways, 2 iscsi gateways US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(SSD): Octopus 15.2.5 with 5 osd servers, 3 mons, 4 gateways

3 years, 5 months

1
0
0 0

Re: NoSuchKey on key that is visible in s3 list/radosgw bk

by Janek Bevendorff

We are having the exact same problem (also Octopus). The object is listed by s3cmd, but trying to download it results in a 404 error. radosgw-admin object stat shows that the object still exists. Any further ideas how I can restore access to this object? (Sorry if this is a duplicate, but it seems like the mailing list hasn't accepted my original mail). > Mariusz Gronczewski wrote: > > >> Dnia 2020-07-27, o godz. 21:31:33 >> "Robin H. Johnson" <robbat2(a)gentoo.org >> <mailto:robbat2@gentoo.org>> napisał(a): >> >> >>> On Mon, Jul 27, 2020 at 08:02:23PM +0200, Mariusz Gronczewski wrote: >>> >>>> Hi, >>>> I've got a problem on Octopus (15.2.3, debian packages) install, >>>> bucket S3 index shows a file: >>>> s3cmd ls s3://upvid/255/38355 --recursive >>>> 2020-07-27 17:48 50584342 >>>> s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 >>>> radosgw-admin bi list also shows it >>>> { >>>> "type": "plain", >>>> "idx": >>>> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", >>>> "entry": { "name": >>>> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", >>>> "instance": "", "ver": { >>>> "pool": 11, >>>> "epoch": 853842 >>>> }, >>>> "locator": "", >>>> "exists": "true", >>>> "meta": { >>>> "category": 1, >>>> "size": 50584342, >>>> "mtime": "2020-07-27T17:48:27.203008Z", >>>> "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7", >>>> "storage_class": "", >>>> "owner": "filmweb-app", >>>> "owner_display_name": "filmweb app user", >>>> "content_type": "", >>>> "accounted_size": 50584342, >>>> "user_data": "", >>>> "appendable": "false" >>>> }, >>>> "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP", >>>> "flags": 0, >>>> "pending_map": [], >>>> "versioned_epoch": 0 >>>> } >>>> }, >>>> but trying to download it via curl (I've set permissions to public0 >>>> >>> only gets me >>> Does the RADOS object for this still exist? >>> >>> try: >>> radosgw-admin object stat --bucket ... --object >>> '255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' >>> >>> If that doesn't return, then the backing object is gone, and you have >>> a stale index entry that can be cleaned up in most cases with check >>> bucket. >>> For cases where that doesn't fix it, my recommended way to fix it is >>> write a new 0-byte object to the same name, then delete it. >>> >> >> >> >> it does exist: >> >> { >> "name": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", >> "size": 50584342, "policy": { >> "acl": {...}, >> "owner": {...} >> }, >> "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7", >> "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP", >> "manifest": { >> "objs": [], >> "obj_size": 50584342, >> "explicit_objs": "false", >> "head_size": 0, >> "max_head_size": 0, >> "prefix": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2", >> "rules": [ { >> "key": 0, >> "val": { >> "start_part_num": 1, >> "start_ofs": 0, >> "part_size": 8388608, >> "stripe_max_size": 4194304, >> "override_prefix": "" >> } >> }, >> { >> "key": 50331648, >> "val": { >> "start_part_num": 7, >> "start_ofs": 50331648, >> "part_size": 252694, >> "stripe_max_size": 4194304, >> "override_prefix": "" >> } >> } >> ], >> "tail_instance": "", >> "tail_placement": { >> "bucket": { >> "name": "upvid", >> "marker": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "", >> "explicit_placement": { >> "data_pool": "", >> "data_extra_pool": "", >> "index_pool": "" >> } >> }, >> "placement_rule": "default-placement" >> }, >> "begin_iter": { >> "part_ofs": 0, >> "stripe_ofs": 0, >> "ofs": 0, >> "stripe_size": 4194304, >> "cur_part_id": 1, >> "cur_stripe": 0, >> "cur_override_prefix": "", >> "location": { >> "placement_rule": "default-placement", >> "obj": { >> "bucket": { >> "name": "upvid", >> "marker": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "", >> "explicit_placement": { >> "data_pool": "", >> "data_extra_pool": "", >> "index_pool": "" >> } >> }, >> "key": { >> "name": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2.1", >> "instance": "", "ns": "multipart" >> } >> }, >> "raw_obj": { >> "pool": "", >> "oid": "", >> "loc": "" >> }, >> "is_raw": false >> } >> }, >> "end_iter": { >> "part_ofs": 50584342, >> "stripe_ofs": 50584342, >> "ofs": 50584342, >> "stripe_size": 252694, >> "cur_part_id": 8, >> "cur_stripe": 0, >> "cur_override_prefix": "", >> "location": { >> "placement_rule": "default-placement", >> "obj": { >> "bucket": { >> "name": "upvid", >> "marker": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "", >> "explicit_placement": { >> "data_pool": "", >> "data_extra_pool": "", >> "index_pool": "" >> } >> }, >> "key": { >> "name": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2.8", >> "instance": "", "ns": "multipart" >> } >> }, >> "raw_obj": { >> "pool": "", >> "oid": "", >> "loc": "" >> }, >> "is_raw": false >> } >> } >> }, >> "attrs": { >> "user.rgw.pg_ver": "/u0005u", >> "user.rgw.source_zone": "�I}�", >> "user.rgw.tail_tag": >> "88d4f221-0da5-444d-81a8-517771278350.658638.5538809", >> "user.rgw.x-amz-acl": "private", "user.rgw.x-amz-content-sha256": >> "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", >> "user.rgw.x-amz-date": "20200619T194517Z" } >> } >> >> >> >> >> >> -- >> Mariusz Gronczewski, Administrator >> >> Efigence S. A. >> ul. Wołoska 9a, 02-583 Warszawa >> T: [+48] 22 380 13 13 >> NOC: [+48] 22 380 10 20 >> E: admin(a)efigence.com <mailto:admin@efigence.com> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> <mailto:ceph-users-leave@ceph.io>

3 years, 5 months

4
11
0 0

Re: Is there a way to make Cephfs kernel client to write data to ceph osd smoothly with buffer io

by Frank Schilder

Yes, that's right. It would be nice if there was a mount option to have such parameters adjusted on a per-file system basis. I should mention that I observed a significant performance improvement for HDD throughput of the local disk as well when adjusting these parameters for ceph. This is largely due to the "too much memory problem" on big servers. The kernel defaults are suitable for machines with 4-8G of RAM. Any enterprise server will beat that with the consequence of insanely large amounts of dirty buffers, leading to buffer flush panic overloading in particular, network file systems (there is a nice article by SUSE https://www.suse.com/support/kb/doc/?id=000017857). Adjusting these parameters to play nice with ceph might actually improve overall performance as a side effect. I would give it a go. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sage Meng <lkkey80(a)gmail.com> Sent: 12 November 2020 16:00:08 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] Is there a way to make Cephfs kernel client to write data to ceph osd smoothly with buffer io vm.dirty_bytes and vm.dirty_background_bytes are all system-wide control parameters, it will influence all the system jobs by adjusting them. Better to have a Ceph Special way to make the transfer more smoothly. Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> 于2020年11月11日周三下午3:28写道： These kernel parameters influence the flushing of data, and also performance: vm.dirty_bytes vm.dirty_background_bytes Smaller vm.dirty_background_bytes will make the transfer more smooth and the ceph cluster will like that. However, it reduces the chances of merge operations in cache and the ceph cluster will not like that. The tuning is heavily workload dependent. Test with realistic workloads and a reasonably large spectrum of values. I got good results by tuning down vm.dirty_background_bytes just to the point when it reduced client performance of copying large files. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sage Meng <lkkey80(a)gmail.com<mailto:lkkey80@gmail.com>> Sent: 06 November 2020 13:45:53 To: ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Is there a way to make Cephfs kernel client to write data to ceph osd smoothly with buffer io Hi All, Cephfs kernel client is influenced by kernel page cache when we write data to it, outgoing data will be huge when os starts flush page cache. So Is there a way to make Cephfs kernel client to write data to ceph osd smoothly when buffer io is used ? _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

3 years, 5 months

1
0
0 0

Rados Crashing

by Brent Kennedy

We are performing file maintenance( deletes essentially ) and when the process gets to a certain point, all four rados gateways crash with the following: Log output: -5> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj verifying op params -4> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj pre-executing -3> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj executing -2> 2020-10-20 06:09:53.997 7f161758f700 10 monclient: get_auth_request con 0x55d2c02ff800 auth_method 0 -1> 2020-10-20 06:09:54.009 7f1609d74700 5 process_single_shard(): failed to acquire lock on obj_delete_at_hint.0000000079 0> 2020-10-20 06:09:54.035 7f15f1543700 -1 *** Caught signal (Segmentation fault) ** in thread 7f15f1543700 thread_name:civetweb-worker ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) 1: (()+0xf5d0) [0x7f161d3405d0] 2: (()+0x2bec80) [0x55d2bcd1fc80] 3: (std::string::assign(std::string const&)+0x2e) [0x55d2bcd2870e] 4: (rgw_bucket::operator=(rgw_bucket const&)+0x11) [0x55d2bce3e551] 5: (RGWObjManifest::obj_iterator::update_location()+0x184) [0x55d2bced7114] 6: (RGWObjManifest::obj_iterator::operator++()+0x263) [0x55d2bd092793] 7: (RGWRados::update_gc_chain(rgw_obj&, RGWObjManifest&, cls_rgw_obj_chain*)+0x51a) [0x55d2bd0939ea] 8: (RGWRados::Object::complete_atomic_modification()+0x83) [0x55d2bd093c63] 9: (RGWRados::Object::Delete::delete_obj()+0x74d) [0x55d2bd0a87ad] 10: (RGWDeleteObj::execute()+0x915) [0x55d2bd04b6d5] 11: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x915) [0x55d2bcdfbb35] 12: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::string const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x1cd8) [0x55d2bcdfdea8] 13: (RGWCivetWebFrontend::process(mg_connection*)+0x38e) [0x55d2bcd41a1e] 14: (()+0x36bace) [0x55d2bcdccace] 15: (()+0x36d76f) [0x55d2bcdce76f] 16: (()+0x36dc18) [0x55d2bcdcec18] 17: (()+0x7dd5) [0x7f161d338dd5] 18: (clone()+0x6d) [0x7f161c84302d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. My guess is that we need to add more resources to the gateways? They have 2 CPUs and 12GB of memory running as virtual machines on centOS 7.6 . Any thoughts? -Brent

3 years, 5 months

2
2
0 0

Re: Nautilus - osdmap not trimming

by m.sliwinski＠lh.pl

Hi Thanks for the reply. Yeah, i restarted all of the mon servers, in sequence, and yesterday just leader alone without any success. Reports: root@monb01:~# ceph report | grep committed report 4002437698 "monmap_first_committed": 1, "monmap_last_committed": 6, "osdmap_first_committed": 67114, "osdmap_last_committed": 72592, "mdsmap_first_committed": 1, "mdsmap_last_committed": 1, "first_committed": 609225, "last_committed": 609251, "first_committed": 180754137, "last_committed": 180754777, root@monb01:~# root@monb01:~# ceph report | jq .osdmap_clean_epochs report 395175214 { "min_last_epoch_clean": 72592, "last_epoch_clean": { "per_pool": [ { "poolid": 0, "floor": 72592 }, { "poolid": 1, "floor": 72592 }, { "poolid": 2, "floor": 72592 }, { "poolid": 3, "floor": 72592 }, { "poolid": 4, "floor": 72592 }, { "poolid": 5, "floor": 72592 }, { "poolid": 26, "floor": 72592 }, { "poolid": 27, "floor": 72592 }, { "poolid": 28, "floor": 72592 } ] }, "osd_epochs": [ { "id": 0, "epoch": 72592 }, { "id": 1, "epoch": 72592 }, { "id": 2, "epoch": 72592 }, { "id": 3, "epoch": 72592 }, { "id": 4, "epoch": 72592 }, { "id": 5, "epoch": 72592 }, { "id": 6, "epoch": 72592 }, { "id": 7, "epoch": 72592 }, { "id": 8, "epoch": 72592 }, { "id": 9, "epoch": 72592 }, { "id": 10, "epoch": 72592 }, { "id": 11, "epoch": 72592 }, { "id": 12, "epoch": 72592 }, { "id": 13, "epoch": 72592 }, { "id": 14, "epoch": 72592 }, { "id": 15, "epoch": 72592 }, { "id": 16, "epoch": 72592 }, { "id": 17, "epoch": 72592 }, { "id": 18, "epoch": 72592 }, { "id": 19, "epoch": 72592 }, { "id": 20, "epoch": 72592 }, { "id": 21, "epoch": 72592 }, { "id": 22, "epoch": 72592 }, { "id": 23, "epoch": 72592 }, { "id": 24, "epoch": 72592 }, { "id": 25, "epoch": 72592 }, { "id": 26, "epoch": 72592 }, { "id": 27, "epoch": 72592 }, { "id": 28, "epoch": 72592 }, { "id": 29, "epoch": 72592 }, { "id": 30, "epoch": 72592 }, { "id": 31, "epoch": 72592 }, { "id": 32, "epoch": 72592 }, { "id": 33, "epoch": 72592 }, { "id": 34, "epoch": 72592 }, { "id": 35, "epoch": 72592 }, { "id": 36, "epoch": 72592 }, { "id": 37, "epoch": 72592 }, { "id": 38, "epoch": 72592 }, { "id": 39, "epoch": 72592 }, { "id": 40, "epoch": 72592 }, { "id": 41, "epoch": 72592 }, { "id": 42, "epoch": 72592 }, { "id": 43, "epoch": 72592 }, { "id": 44, "epoch": 72592 }, { "id": 45, "epoch": 72592 }, { "id": 46, "epoch": 72592 }, { "id": 47, "epoch": 72592 }, { "id": 48, "epoch": 72592 }, { "id": 49, "epoch": 72592 }, { "id": 50, "epoch": 72592 }, { "id": 51, "epoch": 72592 }, { "id": 52, "epoch": 72592 }, { "id": 53, "epoch": 72592 }, { "id": 54, "epoch": 72592 }, { "id": 55, "epoch": 72592 }, { "id": 56, "epoch": 72592 }, { "id": 57, "epoch": 72592 }, { "id": 58, "epoch": 72592 }, { "id": 59, "epoch": 72592 }, { "id": 60, "epoch": 72592 }, { "id": 61, "epoch": 72592 }, { "id": 62, "epoch": 72592 }, { "id": 63, "epoch": 72592 }, { "id": 64, "epoch": 72592 }, { "id": 65, "epoch": 72592 }, { "id": 66, "epoch": 72592 }, { "id": 67, "epoch": 72592 }, { "id": 68, "epoch": 72592 }, { "id": 69, "epoch": 72592 }, { "id": 70, "epoch": 72592 }, { "id": 71, "epoch": 72592 }, { "id": 72, "epoch": 72592 }, { "id": 73, "epoch": 72592 }, { "id": 74, "epoch": 72592 }, { "id": 75, "epoch": 72592 }, { "id": 76, "epoch": 72592 }, { "id": 77, "epoch": 72592 }, { "id": 78, "epoch": 72592 }, { "id": 79, "epoch": 72592 }, { "id": 80, "epoch": 72592 }, { "id": 81, "epoch": 72592 }, { "id": 83, "epoch": 72592 }, { "id": 84, "epoch": 72592 }, { "id": 85, "epoch": 72592 }, { "id": 86, "epoch": 72592 }, { "id": 87, "epoch": 72592 }, { "id": 88, "epoch": 72592 }, { "id": 89, "epoch": 72592 }, { "id": 90, "epoch": 72592 }, { "id": 91, "epoch": 72592 }, { "id": 92, "epoch": 72592 }, { "id": 93, "epoch": 72592 }, { "id": 94, "epoch": 72592 }, { "id": 95, "epoch": 72592 }, { "id": 96, "epoch": 72592 }, { "id": 97, "epoch": 72592 }, { "id": 98, "epoch": 72592 }, { "id": 99, "epoch": 72592 }, { "id": 100, "epoch": 72592 }, { "id": 101, "epoch": 72592 }, { "id": 102, "epoch": 72592 }, { "id": 103, "epoch": 72592 }, { "id": 104, "epoch": 72592 }, { "id": 105, "epoch": 72592 }, { "id": 106, "epoch": 72592 }, { "id": 107, "epoch": 72592 }, { "id": 108, "epoch": 72592 }, { "id": 109, "epoch": 72592 }, { "id": 110, "epoch": 72592 }, { "id": 111, "epoch": 72592 }, { "id": 112, "epoch": 72592 }, { "id": 113, "epoch": 72592 }, { "id": 114, "epoch": 72592 }, { "id": 115, "epoch": 72592 }, { "id": 116, "epoch": 72592 }, { "id": 117, "epoch": 72592 }, { "id": 118, "epoch": 72592 }, { "id": 119, "epoch": 72592 }, { "id": 120, "epoch": 72592 }, { "id": 121, "epoch": 72592 }, { "id": 122, "epoch": 72592 }, { "id": 123, "epoch": 72592 }, { "id": 124, "epoch": 72592 }, { "id": 125, "epoch": 72592 }, { "id": 126, "epoch": 72592 }, { "id": 127, "epoch": 72592 }, { "id": 128, "epoch": 72592 }, { "id": 129, "epoch": 72592 }, { "id": 130, "epoch": 72592 }, { "id": 131, "epoch": 72592 }, { "id": 132, "epoch": 72592 }, { "id": 133, "epoch": 72592 }, { "id": 134, "epoch": 72592 }, { "id": 135, "epoch": 72592 }, { "id": 136, "epoch": 72592 }, { "id": 137, "epoch": 72592 }, { "id": 138, "epoch": 72592 }, { "id": 139, "epoch": 72592 }, { "id": 140, "epoch": 72592 }, { "id": 141, "epoch": 72592 }, { "id": 142, "epoch": 72592 }, { "id": 143, "epoch": 72592 }, { "id": 144, "epoch": 72592 }, { "id": 145, "epoch": 72592 }, { "id": 146, "epoch": 72592 }, { "id": 147, "epoch": 72592 }, { "id": 148, "epoch": 72592 }, { "id": 149, "epoch": 72592 }, { "id": 150, "epoch": 72592 }, { "id": 151, "epoch": 72592 }, { "id": 152, "epoch": 72592 }, { "id": 153, "epoch": 72592 }, { "id": 154, "epoch": 72592 }, { "id": 155, "epoch": 72592 }, { "id": 156, "epoch": 72592 }, { "id": 157, "epoch": 72592 }, { "id": 158, "epoch": 72592 }, { "id": 159, "epoch": 72592 }, { "id": 160, "epoch": 72592 }, { "id": 161, "epoch": 72592 }, { "id": 162, "epoch": 72592 }, { "id": 163, "epoch": 72592 }, { "id": 164, "epoch": 72592 }, { "id": 165, "epoch": 72592 }, { "id": 166, "epoch": 72592 }, { "id": 167, "epoch": 72592 }, { "id": 168, "epoch": 72592 }, { "id": 169, "epoch": 72592 }, { "id": 170, "epoch": 72592 }, { "id": 171, "epoch": 72592 }, { "id": 172, "epoch": 72592 }, { "id": 173, "epoch": 72592 }, { "id": 174, "epoch": 72592 }, { "id": 175, "epoch": 72592 }, { "id": 176, "epoch": 72592 }, { "id": 177, "epoch": 72592 }, { "id": 178, "epoch": 72592 }, { "id": 179, "epoch": 72592 }, { "id": 180, "epoch": 72592 }, { "id": 181, "epoch": 72592 }, { "id": 182, "epoch": 72592 }, { "id": 183, "epoch": 72592 }, { "id": 184, "epoch": 72592 }, { "id": 185, "epoch": 72592 }, { "id": 186, "epoch": 72592 }, { "id": 187, "epoch": 72592 }, { "id": 188, "epoch": 72592 }, { "id": 189, "epoch": 72592 }, { "id": 190, "epoch": 72592 }, { "id": 191, "epoch": 72592 }, { "id": 192, "epoch": 72592 }, { "id": 193, "epoch": 72592 }, { "id": 194, "epoch": 72592 }, { "id": 195, "epoch": 72592 }, { "id": 196, "epoch": 72592 }, { "id": 197, "epoch": 72592 }, { "id": 198, "epoch": 72592 }, { "id": 199, "epoch": 72592 }, { "id": 200, "epoch": 72592 }, { "id": 201, "epoch": 72592 }, { "id": 202, "epoch": 72592 }, { "id": 203, "epoch": 72592 }, { "id": 204, "epoch": 72592 }, { "id": 205, "epoch": 72592 }, { "id": 206, "epoch": 72592 }, { "id": 207, "epoch": 72592 }, { "id": 208, "epoch": 72592 }, { "id": 209, "epoch": 72592 }, { "id": 210, "epoch": 72592 }, { "id": 211, "epoch": 72592 }, { "id": 212, "epoch": 72592 }, { "id": 213, "epoch": 72592 }, { "id": 214, "epoch": 72592 }, { "id": 215, "epoch": 72592 }, { "id": 216, "epoch": 72592 }, { "id": 217, "epoch": 72592 }, { "id": 218, "epoch": 72592 }, { "id": 219, "epoch": 72592 }, { "id": 220, "epoch": 72592 }, { "id": 221, "epoch": 72592 }, { "id": 222, "epoch": 72592 }, { "id": 223, "epoch": 72592 }, { "id": 224, "epoch": 72592 }, { "id": 225, "epoch": 72592 }, { "id": 226, "epoch": 72592 }, { "id": 227, "epoch": 72592 }, { "id": 228, "epoch": 72592 }, { "id": 229, "epoch": 72592 }, { "id": 230, "epoch": 72592 }, { "id": 231, "epoch": 72592 }, { "id": 232, "epoch": 72592 }, { "id": 233, "epoch": 72592 }, { "id": 234, "epoch": 72592 }, { "id": 235, "epoch": 72592 }, { "id": 236, "epoch": 72592 }, { "id": 237, "epoch": 72592 }, { "id": 238, "epoch": 72592 }, { "id": 239, "epoch": 72592 }, { "id": 240, "epoch": 72592 }, { "id": 241, "epoch": 72592 }, { "id": 242, "epoch": 72592 }, { "id": 243, "epoch": 72592 }, { "id": 244, "epoch": 72592 }, { "id": 245, "epoch": 72592 }, { "id": 246, "epoch": 72592 }, { "id": 247, "epoch": 72592 }, { "id": 248, "epoch": 72592 }, { "id": 249, "epoch": 72592 }, { "id": 250, "epoch": 72592 }, { "id": 251, "epoch": 72592 }, { "id": 252, "epoch": 72592 }, { "id": 253, "epoch": 72592 }, { "id": 254, "epoch": 72592 }, { "id": 255, "epoch": 72592 }, { "id": 256, "epoch": 72592 }, { "id": 257, "epoch": 72592 }, { "id": 258, "epoch": 72592 }, { "id": 259, "epoch": 72592 }, { "id": 260, "epoch": 72592 }, { "id": 261, "epoch": 72592 }, { "id": 262, "epoch": 72592 }, { "id": 263, "epoch": 72592 }, { "id": 264, "epoch": 72592 }, { "id": 265, "epoch": 72592 }, { "id": 266, "epoch": 72592 }, { "id": 267, "epoch": 72592 }, { "id": 268, "epoch": 72592 }, { "id": 269, "epoch": 72592 }, { "id": 270, "epoch": 72592 }, { "id": 271, "epoch": 72592 }, { "id": 272, "epoch": 72592 }, { "id": 273, "epoch": 72592 }, { "id": 274, "epoch": 72592 }, { "id": 275, "epoch": 72592 }, { "id": 276, "epoch": 72592 }, { "id": 277, "epoch": 72592 }, { "id": 278, "epoch": 72592 }, { "id": 279, "epoch": 72592 }, { "id": 280, "epoch": 72592 }, { "id": 281, "epoch": 72592 }, { "id": 282, "epoch": 72592 }, { "id": 283, "epoch": 72592 }, { "id": 284, "epoch": 72592 } ] } root@monb01:~# W dniu 2020-11-12 11:58, Dan van der Ster napisał(a): > I found another possible trimming bug this morning, but I don't expect > it applies to you because you said you restarted the mon leader: > https://tracker.ceph.com/issues/48212 > > Otherwise, couple you please share the output of > > ceph report | grep committed > ceph report | jq .osdmap_clean_epochs > > Thanks, > > Dan > > On Thu, Nov 12, 2020 at 10:56 AM <m.sliwinski(a)lh.pl> wrote: >> >> Hi >> >> Thanks for the response. Our cluster is currently mostly on 14.2.13, >> especially all MONs and MGRs are. >> Some OSDs are still on 14.2.9, but it shouldn't block osdmap trimming >> i >> think, because atm we don't have any down OSDs, i checked for that >> when >> we first noticed the issue. >> I'm working of course on bringing all OSDs to 14.2.13, but it will >> take >> some time as i have to create and test Debian packages for that. >> Could there be any other reason? I found posts about PGs for new pool >> not bein marked as created in MON db while cluster still reporting >> everything as active+clean, but i dont know how to debug that. >> >> -- >> Best regards >> Marcin >> >> >> W dniu 2020-11-11 16:50, Dan van der Ster napisał(a): >> > Hi, >> > >> > v14.2.13 has an important fix in this area: >> > https://tracker.ceph.com/issues/47290 >> > Without this fix, your cluster will not trim if there are any *down* >> > osds in the cluster. >> > >> > On our clusters we are running v14.2.11 patched with commit >> > "mon/OSDMonitor: only take in osd into consideration when trimming >> > osdmaps" -- this trims maps perfectly afaict. >> > >> > I can't vouch for the rest of 14.2.13, so better test that adequately >> > before upgrading. >> > >> > Cheers, Dan >> > >> > >> > On Tue, Nov 10, 2020 at 6:57 PM <m.sliwinski(a)lh.pl> wrote: >> >> >> >> Hi >> >> >> >> We have ceph cluster running on Nautilus, recently upgraded from >> >> Mimic. >> >> When in Mimic we noticed issue with osdmap not trimming, which caused >> >> part of our cluster to crash due to osdmap cache misses. We solved it >> >> by >> >> adding "osd_map_cache_size = 5000" to our ceph.conf >> >> Because we had at that time mixed OSD versions from both Mimic and >> >> Nautilus we decided to finish upgrade, but it didn't solve our >> >> problem. >> >> We have at the moment: "oldest_map": 67114, "newest_map": 72588,and >> >> the >> >> difference is not shrinking even thought cluster is in active+clean >> >> state. Restarting all mon's didn't help. It seems bug is similar to >> >> https://tracker.ceph.com/issues/44184 but there's no solution there. >> >> What else can i check or do? >> >> I don't want do to cangerous things like mon_osd_force_trim_to or >> >> something similar without finding the cause. >> >> >> >> I noticed in MON debug log: >> >> >> >> 2020-11-10 17:11:14.612 7f9592d5b700 10 mon.monb01(a)0(leader).osd >> >> e72571 >> >> should_prune could only prune 4957 epochs (67114..72071), which is >> >> less >> >> than the required minimum (10000) >> >> 2020-11-10 17:11:19.612 7f9592d5b700 10 mon.monb01(a)0(leader).osd >> >> e72571 >> >> should_prune could only prune 4957 epochs (67114..72071), which is >> >> less >> >> than the required minimum (10000) >> >> >> >> So i added config options to reduce those values: >> >> >> >> mon dev mon_debug_block_osdmap_trim false >> >> mon advanced mon_min_osdmap_epochs 100 >> >> mon advanced mon_osdmap_full_prune_min 500 >> >> mon advanced paxos_service_trim_min 10 >> >> >> >> But it didn't help: >> >> >> >> 2020-11-10 18:28:26.165 7f1b700ab700 20 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest osdmap manifest detected in store; reload. >> >> 2020-11-10 18:28:26.169 7f1b700ab700 10 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest store osdmap manifest pinned (67114 .. 72484) >> >> 2020-11-10 18:28:26.169 7f1b700ab700 10 mon.monb01(a)0(leader).osd >> >> e72588 >> >> should_prune not enough epochs to form an interval (last pinned: >> >> 72484, >> >> last to pin: 72488, interval: 10) >> >> >> >> Command "ceph report | jq '.osdmap_manifest' |jq '.pinned_maps[]'" >> >> shows >> >> 67114 on the top, but i'm unable to determine why. >> >> >> >> Same with 'ceph report | jq .osdmap_first_committed': >> >> >> >> root@monb01:/var/log/ceph# ceph report | jq .osdmap_first_committed >> >> report 4073203295 >> >> 67114 >> >> root@monb01:/var/log/ceph# >> >> >> >> When i try to derermine if a certain PG or OSD is keeping it so low i >> >> don't get anything. >> >> >> >> And in MON debug log i get: >> >> >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01@0(leader) e6 >> >> refresh_from_paxos >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 >> >> mon.monb01(a)0(leader).paxosservice(mdsmap 1..1) refresh >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 >> >> mon.monb01(a)0(leader).paxosservice(osdmap 67114..72588) refresh >> >> 2020-11-10 18:42:41.767 7f1b74721700 20 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest osdmap manifest detected in store; reload. >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest store osdmap manifest pinned (67114 .. 72484) >> >> >> >> I also get: >> >> >> >> root@monb01:/var/log/ceph# ceph report |grep "min_last_epoch_clean" >> >> report 2716976759 >> >> "min_last_epoch_clean": 0, >> >> root@monb01:/var/log/ceph# >> >> >> >> >> >> Additional info: >> >> root@monb01:/var/log/ceph# ceph versions >> >> { >> >> "mon": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 3 >> >> }, >> >> "mgr": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 3 >> >> }, >> >> "osd": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 120, >> >> "ceph version 14.2.9 >> >> (581f22da52345dba46ee232b73b990f06029a2a0) >> >> nautilus (stable)": 164 >> >> }, >> >> "mds": {}, >> >> "overall": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 126, >> >> "ceph version 14.2.9 >> >> (581f22da52345dba46ee232b73b990f06029a2a0) >> >> nautilus (stable)": 164 >> >> } >> >> } >> >> >> >> >> >> root@monb01:/var/log/ceph# ceph mon feature ls >> >> >> >> all features >> >> supported: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> persistent: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> on current monmap (epoch 6) >> >> persistent: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> required: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> >> >> >> >> root@monb01:/var/log/ceph# ceph osd dump | grep require >> >> require_min_compat_client luminous >> >> require_osd_release nautilus >> >> >> >> >> >> root@monb01:/var/log/ceph# ceph report | jq >> >> '.osdmap_manifest.pinned_maps | length' >> >> report 1777129876 >> >> 538 >> >> >> >> root@monb01:/var/log/ceph# ceph pg dump -f json | jq .osd_epochs >> >> dumped all >> >> null >> >> >> >> -- >> >> Best regards >> >> Marcin >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 years, 5 months

2
2
0 0

How to run ceph_osd_dump

by Denis Krienbühl

Hi We’ve recently encountered the following errors: [WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 2752.832ms) Slow OSD heartbeats on back from osd.2 [nvme-a] to osd.290 [nvme-c] 2752.832 msec ... Truncated long network list. Use ceph daemon mgr.# dump_osd_network for more information To get more information we wanted to run the dump_osd_network command, but it doesn’t seem to be a valid command: ceph daemon /var/run/ceph/ceph-mgr.$(hostname).asok dump_osd_network 0 no valid command found; 10 closest matches: 0 1 2 abort assert config diff config diff get <var> config get <var> config help [<var>] config set <var> <val>... admin_socket: invalid command Other commands, like ceph daemon dump_cache work, so it seems to hit the right socket. What am I doing wrong? Cheers, Denis

3 years, 5 months

2
2
0 0

How to use ceph-volume to create multiple OSDs per NVMe disk, and with fixed WAL/DB partition on another device?

by victorhooi＠yahoo.com

I'm building a new 4-node Proxmox/Ceph cluster, to hold disk images for our VMs. (Ceph version is 15.2.5). Each node has 6 x NVMe SSDs (4TB), and 1 x Optane drive (960GB). CPU is AMD Rome 7442, so there should be plenty of CPU capacity to spare. My aim is to create 4 x OSDs per NVMe SSD (to make more effective use of the NVMe performance) and use the Optane drive to store the WAL/DB partition for each OSD. (I.e. total of 24 x 35GB WAL/DB partitions). However, I am struggling to get the right ceph-volume command to achieve this. Thanks to a very kind Redditor, I was able to get close: /dev/nvme0n1 is an Optane device (900GB). /dev/nvme2n1 is an Intel NVMe SSD (4TB). ``` # ceph-volume lvm batch --osds-per-device 4 /dev/nvme2n1 --db-devices /dev/nvme0n1 Total OSDs: 4 Solid State VG: Targets: block.db Total size: 893.00 GB Total LVs: 16 Size per LV: 223.25 GB Devices: /dev/nvme0n1 Type Path LV Size % of device ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) ``` This does split up the NVMe disk into 4 OSDs, and creates WAL/DB partition on the Optane drive - however, it creates 4 x 223 GB partitions on the Optane (whereas I want 35GB partitions). Is there any way to specify the WAL/DB partition size in the above? And can it be done, such that you can run successive ceph-volume commands, to add the OSDs and WAL/DB partitions for each NVMe disk? (Or if there's an easier way to achieve the above layout, please let me know). That being said - I also just saw this ceph-users thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/3Y6DEJCF7ZM… It talks there about "osd op num shards" and "osd op num threads per shard" - is there some way to set those, to achieve similar performance to say, 4 x OSDs per NVMe drive, but with only 1 x NVMe? Has anybody done any testing/benchmarking on this they can share?

3 years, 5 months

5
4
0 0

newbie question: direct objects of different sizes to different pools?

by andersnb＠ucar.edu

Hi All, I'm exploring deploying Ceph at my organization for use as an object storage system (using the S3 RGW interface). My users have range of file sizes and I'd like to direct small files to a pool that uses replication and large files to a pool that uses erasure encoding. Is that possible? Thanks! Bill

3 years, 5 months

3
4
0 0

Nautilus - osdmap not trimming

by m.sliwinski＠lh.pl

Hi We have ceph cluster running on Nautilus, recently upgraded from Mimic. When in Mimic we noticed issue with osdmap not trimming, which caused part of our cluster to crash due to osdmap cache misses. We solved it by adding "osd_map_cache_size = 5000" to our ceph.conf Because we had at that time mixed OSD versions from both Mimic and Nautilus we decided to finish upgrade, but it didn't solve our problem. We have at the moment: "oldest_map": 67114, "newest_map": 72588,and the difference is not shrinking even thought cluster is in active+clean state. Restarting all mon's didn't help. It seems bug is similar to https://tracker.ceph.com/issues/44184 but there's no solution there. What else can i check or do? I don't want do to cangerous things like mon_osd_force_trim_to or something similar without finding the cause. I noticed in MON debug log: 2020-11-10 17:11:14.612 7f9592d5b700 10 mon.monb01(a)0(leader).osd e72571 should_prune could only prune 4957 epochs (67114..72071), which is less than the required minimum (10000) 2020-11-10 17:11:19.612 7f9592d5b700 10 mon.monb01(a)0(leader).osd e72571 should_prune could only prune 4957 epochs (67114..72071), which is less than the required minimum (10000) So i added config options to reduce those values: mon dev mon_debug_block_osdmap_trim false mon advanced mon_min_osdmap_epochs 100 mon advanced mon_osdmap_full_prune_min 500 mon advanced paxos_service_trim_min 10 But it didn't help: 2020-11-10 18:28:26.165 7f1b700ab700 20 mon.monb01(a)0(leader).osd e72588 load_osdmap_manifest osdmap manifest detected in store; reload. 2020-11-10 18:28:26.169 7f1b700ab700 10 mon.monb01(a)0(leader).osd e72588 load_osdmap_manifest store osdmap manifest pinned (67114 .. 72484) 2020-11-10 18:28:26.169 7f1b700ab700 10 mon.monb01(a)0(leader).osd e72588 should_prune not enough epochs to form an interval (last pinned: 72484, last to pin: 72488, interval: 10) Command "ceph report | jq '.osdmap_manifest' |jq '.pinned_maps[]'" shows 67114 on the top, but i'm unable to determine why. Same with 'ceph report | jq .osdmap_first_committed': root@monb01:/var/log/ceph# ceph report | jq .osdmap_first_committed report 4073203295 67114 root@monb01:/var/log/ceph# When i try to derermine if a certain PG or OSD is keeping it so low i don't get anything. And in MON debug log i get: 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01@0(leader) e6 refresh_from_paxos 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01(a)0(leader).paxosservice(mdsmap 1..1) refresh 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01(a)0(leader).paxosservice(osdmap 67114..72588) refresh 2020-11-10 18:42:41.767 7f1b74721700 20 mon.monb01(a)0(leader).osd e72588 load_osdmap_manifest osdmap manifest detected in store; reload. 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01(a)0(leader).osd e72588 load_osdmap_manifest store osdmap manifest pinned (67114 .. 72484) I also get: root@monb01:/var/log/ceph# ceph report |grep "min_last_epoch_clean" report 2716976759 "min_last_epoch_clean": 0, root@monb01:/var/log/ceph# Additional info: root@monb01:/var/log/ceph# ceph versions { "mon": { "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)": 120, "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 164 }, "mds": {}, "overall": { "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)": 126, "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 164 } } root@monb01:/var/log/ceph# ceph mon feature ls all features supported: [kraken,luminous,mimic,osdmap-prune,nautilus] persistent: [kraken,luminous,mimic,osdmap-prune,nautilus] on current monmap (epoch 6) persistent: [kraken,luminous,mimic,osdmap-prune,nautilus] required: [kraken,luminous,mimic,osdmap-prune,nautilus] root@monb01:/var/log/ceph# ceph osd dump | grep require require_min_compat_client luminous require_osd_release nautilus root@monb01:/var/log/ceph# ceph report | jq '.osdmap_manifest.pinned_maps | length' report 1777129876 538 root@monb01:/var/log/ceph# ceph pg dump -f json | jq .osd_epochs dumped all null -- Best regards Marcin

3 years, 5 months

2
1
0 0

_get_class not permitted to load rgw_gc

by Dan van der Ster

Hi, We have this "not permitted to load rgw_gc" error on some of our osds. Anyone knows what this is and how to fix it? Nautilus 14.2.11 and CentOS 7 / 8: 2020-11-11 09:48:15.914 7f665c1ea700 0 _get_class not permitted to load rgw_gc 2020-11-11 09:48:15.914 7f665c1ea700 -1 osd.874 163331 class rgw_gc open got (1) Operation not permitted Cheers, Dan

3 years, 5 months

1
3
0 0

2024

2023

2022

2021

2020

2019

ceph-users November 2020