- ceph-users - lists.ceph.io

question about rgw delete speed
by Adrian Nicolae 13 Nov '20

13 Nov '20

Hey guys, I'm in charge of a local cloud-storage service. Our primary object storage is a vendor-based one and I want to replace it in the near future with Ceph with the following setup : - 6 OSD servers with 36 SATA 16TB drives each and 3 big NVME per server (1 big NVME for every 12 drives so I can reserve 300GB NVME storage for every SATA drive), 3 MON, 2 RGW with Epyc 7402p and 128GB RAM. So in the end we'll have ~ 3PB of raw data and 216 SATA drives. Currently we have ~ 100 millions of files on the primary storage with the following distribution : - ~10% = very small files ( less than 1MB - thumbnails, text&office files and so on) - ~60%= small files (between 1MB and 10MB) - 20% = medium files ( between 10MB and 1GB) - 10% = big files (over 1GB). My main concern is the speed of delete operations. We have around 500k-600k delete ops every 24 hours so quite a lot. Our current storage is not deleting all the files fast enough (it's always 1 week-10 days behind) , I guess is not only a software issue and probably the delete speed will get better if we add more drives (we now have 108). What do you think about Ceph delete speed ? I read on other threads that it's not very fast . I wonder if this hw setup can handle our current delete load better than our current storage. On RGW servers I want to use Swift , not S3. And another question : can I start deploying in production directly the latest Ceph version (Octopus) or is it safer to start with Nautilus until Octopus will be more stable ? Any input would be greatly appreciated ! Thanks, Adrian.

4 4

Not able to read file from ceph kernel mount
by Amudhan P 13 Nov '20

13 Nov '20

Hi, I am trying to read file from my ceph kernel mount and file read stays in bytes for very long and I am getting below error msg in dmesg. [ 167.591095] ceph: loaded (mds proto 32) [ 167.600010] libceph: mon0 10.0.103.1:6789 session established [ 167.601167] libceph: client144519 fsid f8bc7682-0d11-11eb-a332- 0cc47a5ec98a [ 272.132787] libceph: osd1 10.0.104.1:6891 socket closed (con state CONNECTING) Ceph cluster status is healthy no error It was working fine until before my entire cluster was down. Using Ceph octopus in debian. Regards Amudhan P

1 3

Cephfs Kernel client not working properly without ceph cluster IP
by Amudhan P 13 Nov '20

13 Nov '20

Hi, I have mounted my cephfs (ceph octopus) thru kernel client in Debian. I get following error in "dmesg" when I try to read any file from my mount. "[ 236.429897] libceph: osd1 10.100.4.1:6891 socket closed (con state CONNECTING)" I use public IP (10.100.3.1) and cluster IP (10.100.4.1) in my ceph cluster. I think public IP is enough to mount the share and work on it but in my case, it needs me to assign public IP also to the client to work properly. Does anyone have experience in this? I have earlier also mailed the ceph-user group but I didn't get any response. So sending again not sure my mail went through. regards Amudhan

4 10

Autoscale - enable or not on main pool?
by Brent Kennedy 13 Nov '20

13 Nov '20

I recently setup a new octopus cluster and was testing the autoscale feature. Used ceph-ansible so its enabled by default. Anyhow, I have three other clusters that are on nautilus, so I wanted to see if it made sense to enable it there on the main pool. Here is a print out of the autoscale status: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE default.rgw.buckets.non-ec 0 2.0 55859G 0.0000 1.0 32 on default.rgw.meta 9298 3.0 55859G 0.0000 1.0 32 on default.rgw.buckets.index 18058M 3.0 55859G 0.0009 1.0 32 on default.rgw.control 0 3.0 55859G 0.0000 1.0 32 on default.rgw.buckets.data 9126G 2.0 55859G 0.3268 1.0 4096 1024 off .rgw.root 3155 3.0 55859G 0.0000 1.0 32 on rbd 155.5G 2.0 55859G 0.0056 1.0 32 on default.rgw.log 374.4k 3.0 55859G 0.0000 1.0 64 on For this entry: default.rgw.buckets.data 9126G 2.0 55859G 0.3268 1.0 4096 1024 off I have it disabled because it showed a warn message, but its recommending a 1024 PG setting. When I use the online ceph calculator at ceph.io, its saying the 4096 setting is correct. So why is autoscaler saying 1024? There are 6 osd servers with 10 OSDs each ( all SSD ). 60 TB total. Pool LS output: pool 1 '.rgw.root' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8800 lfor 0/0/344 flags hashpspool stripe_width 0 application rgw pool 2 'default.rgw.control' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8799 lfor 0/0/346 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.meta' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8798 lfor 0/0/350 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.log' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 8802 lfor 0/0/298 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.buckets.index' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 638 pgp_num 608 pg_num_target 32 pgp_num_target 32 autoscale_mode on last_change 10320 lfor 0/10320/10318 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 7 'default.rgw.buckets.data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 4096 pgp_num 4096 last_change 9467 lfor 0/0/552 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 8 'default.rgw.buckets.non-ec' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8797 lfor 0/0/348 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 9 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8801 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd Regards, -Brent Existing Clusters: Test: Ocotpus 15.2.5 ( all virtual on nvme ) US Production(HDD): Nautilus 14.2.11 with 11 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.11 with 18 osd servers, 3 mons, 4 gateways, 2 iscsi gateways US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(SSD): Octopus 15.2.5 with 5 osd servers, 3 mons, 4 gateways

1 0

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Janek Bevendorff 12 Nov '20

12 Nov '20

We are having the exact same problem (also Octopus). The object is listed by s3cmd, but trying to download it results in a 404 error. radosgw-admin object stat shows that the object still exists. Any further ideas how I can restore access to this object? (Sorry if this is a duplicate, but it seems like the mailing list hasn't accepted my original mail). > Mariusz Gronczewski wrote: > > >> Dnia 2020-07-27, o godz. 21:31:33 >> "Robin H. Johnson" <robbat2(a)gentoo.org >> <mailto:robbat2@gentoo.org>> napisał(a): >> >> >>> On Mon, Jul 27, 2020 at 08:02:23PM +0200, Mariusz Gronczewski wrote: >>> >>>> Hi, >>>> I've got a problem on Octopus (15.2.3, debian packages) install, >>>> bucket S3 index shows a file: >>>> s3cmd ls s3://upvid/255/38355 --recursive >>>> 2020-07-27 17:48 50584342 >>>> s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 >>>> radosgw-admin bi list also shows it >>>> { >>>> "type": "plain", >>>> "idx": >>>> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", >>>> "entry": { "name": >>>> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", >>>> "instance": "", "ver": { >>>> "pool": 11, >>>> "epoch": 853842 >>>> }, >>>> "locator": "", >>>> "exists": "true", >>>> "meta": { >>>> "category": 1, >>>> "size": 50584342, >>>> "mtime": "2020-07-27T17:48:27.203008Z", >>>> "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7", >>>> "storage_class": "", >>>> "owner": "filmweb-app", >>>> "owner_display_name": "filmweb app user", >>>> "content_type": "", >>>> "accounted_size": 50584342, >>>> "user_data": "", >>>> "appendable": "false" >>>> }, >>>> "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP", >>>> "flags": 0, >>>> "pending_map": [], >>>> "versioned_epoch": 0 >>>> } >>>> }, >>>> but trying to download it via curl (I've set permissions to public0 >>>> >>> only gets me >>> Does the RADOS object for this still exist? >>> >>> try: >>> radosgw-admin object stat --bucket ... --object >>> '255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' >>> >>> If that doesn't return, then the backing object is gone, and you have >>> a stale index entry that can be cleaned up in most cases with check >>> bucket. >>> For cases where that doesn't fix it, my recommended way to fix it is >>> write a new 0-byte object to the same name, then delete it. >>> >> >> >> >> it does exist: >> >> { >> "name": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", >> "size": 50584342, "policy": { >> "acl": {...}, >> "owner": {...} >> }, >> "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7", >> "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP", >> "manifest": { >> "objs": [], >> "obj_size": 50584342, >> "explicit_objs": "false", >> "head_size": 0, >> "max_head_size": 0, >> "prefix": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2", >> "rules": [ { >> "key": 0, >> "val": { >> "start_part_num": 1, >> "start_ofs": 0, >> "part_size": 8388608, >> "stripe_max_size": 4194304, >> "override_prefix": "" >> } >> }, >> { >> "key": 50331648, >> "val": { >> "start_part_num": 7, >> "start_ofs": 50331648, >> "part_size": 252694, >> "stripe_max_size": 4194304, >> "override_prefix": "" >> } >> } >> ], >> "tail_instance": "", >> "tail_placement": { >> "bucket": { >> "name": "upvid", >> "marker": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "", >> "explicit_placement": { >> "data_pool": "", >> "data_extra_pool": "", >> "index_pool": "" >> } >> }, >> "placement_rule": "default-placement" >> }, >> "begin_iter": { >> "part_ofs": 0, >> "stripe_ofs": 0, >> "ofs": 0, >> "stripe_size": 4194304, >> "cur_part_id": 1, >> "cur_stripe": 0, >> "cur_override_prefix": "", >> "location": { >> "placement_rule": "default-placement", >> "obj": { >> "bucket": { >> "name": "upvid", >> "marker": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "", >> "explicit_placement": { >> "data_pool": "", >> "data_extra_pool": "", >> "index_pool": "" >> } >> }, >> "key": { >> "name": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2.1", >> "instance": "", "ns": "multipart" >> } >> }, >> "raw_obj": { >> "pool": "", >> "oid": "", >> "loc": "" >> }, >> "is_raw": false >> } >> }, >> "end_iter": { >> "part_ofs": 50584342, >> "stripe_ofs": 50584342, >> "ofs": 50584342, >> "stripe_size": 252694, >> "cur_part_id": 8, >> "cur_stripe": 0, >> "cur_override_prefix": "", >> "location": { >> "placement_rule": "default-placement", >> "obj": { >> "bucket": { >> "name": "upvid", >> "marker": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "", >> "explicit_placement": { >> "data_pool": "", >> "data_extra_pool": "", >> "index_pool": "" >> } >> }, >> "key": { >> "name": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2.8", >> "instance": "", "ns": "multipart" >> } >> }, >> "raw_obj": { >> "pool": "", >> "oid": "", >> "loc": "" >> }, >> "is_raw": false >> } >> } >> }, >> "attrs": { >> "user.rgw.pg_ver": "/u0005u", >> "user.rgw.source_zone": "�I}�", >> "user.rgw.tail_tag": >> "88d4f221-0da5-444d-81a8-517771278350.658638.5538809", >> "user.rgw.x-amz-acl": "private", "user.rgw.x-amz-content-sha256": >> "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", >> "user.rgw.x-amz-date": "20200619T194517Z" } >> } >> >> >> >> >> >> -- >> Mariusz Gronczewski, Administrator >> >> Efigence S. A. >> ul. Wołoska 9a, 02-583 Warszawa >> T: [+48] 22 380 13 13 >> NOC: [+48] 22 380 10 20 >> E: admin(a)efigence.com <mailto:admin@efigence.com> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> <mailto:ceph-users-leave@ceph.io>

4 11

Re: Is there a way to make Cephfs kernel client to write data to ceph osd smoothly with buffer io
by Frank Schilder 12 Nov '20

12 Nov '20

Yes, that's right. It would be nice if there was a mount option to have such parameters adjusted on a per-file system basis. I should mention that I observed a significant performance improvement for HDD throughput of the local disk as well when adjusting these parameters for ceph. This is largely due to the "too much memory problem" on big servers. The kernel defaults are suitable for machines with 4-8G of RAM. Any enterprise server will beat that with the consequence of insanely large amounts of dirty buffers, leading to buffer flush panic overloading in particular, network file systems (there is a nice article by SUSE https://www.suse.com/support/kb/doc/?id=000017857) Adjusting these parameters to play nice with ceph might actually improve overall performance as a side effect. I would give it a go. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sage Meng <lkkey80(a)gmail.com> Sent: 12 November 2020 16:00:08 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] Is there a way to make Cephfs kernel client to write data to ceph osd smoothly with buffer io vm.dirty_bytes and vm.dirty_background_bytes are all system-wide control parameters, it will influence all the system jobs by adjusting them. Better to have a Ceph Special way to make the transfer more smoothly. Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> 于2020年11月11日周三下午3:28写道： These kernel parameters influence the flushing of data, and also performance: vm.dirty_bytes vm.dirty_background_bytes Smaller vm.dirty_background_bytes will make the transfer more smooth and the ceph cluster will like that. However, it reduces the chances of merge operations in cache and the ceph cluster will not like that. The tuning is heavily workload dependent. Test with realistic workloads and a reasonably large spectrum of values. I got good results by tuning down vm.dirty_background_bytes just to the point when it reduced client performance of copying large files. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sage Meng <lkkey80(a)gmail.com<mailto:lkkey80@gmail.com>> Sent: 06 November 2020 13:45:53 To: ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Is there a way to make Cephfs kernel client to write data to ceph osd smoothly with buffer io Hi All, Cephfs kernel client is influenced by kernel page cache when we write data to it, outgoing data will be huge when os starts flush page cache. So Is there a way to make Cephfs kernel client to write data to ceph osd smoothly when buffer io is used ? _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

Rados Crashing
by Brent Kennedy 12 Nov '20

12 Nov '20

We are performing file maintenance( deletes essentially ) and when the process gets to a certain point, all four rados gateways crash with the following: Log output: -5> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj verifying op params -4> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj pre-executing -3> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj executing -2> 2020-10-20 06:09:53.997 7f161758f700 10 monclient: get_auth_request con 0x55d2c02ff800 auth_method 0 -1> 2020-10-20 06:09:54.009 7f1609d74700 5 process_single_shard(): failed to acquire lock on obj_delete_at_hint.0000000079 0> 2020-10-20 06:09:54.035 7f15f1543700 -1 *** Caught signal (Segmentation fault) ** in thread 7f15f1543700 thread_name:civetweb-worker ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) 1: (()+0xf5d0) [0x7f161d3405d0] 2: (()+0x2bec80) [0x55d2bcd1fc80] 3: (std::string::assign(std::string const&)+0x2e) [0x55d2bcd2870e] 4: (rgw_bucket::operator=(rgw_bucket const&)+0x11) [0x55d2bce3e551] 5: (RGWObjManifest::obj_iterator::update_location()+0x184) [0x55d2bced7114] 6: (RGWObjManifest::obj_iterator::operator++()+0x263) [0x55d2bd092793] 7: (RGWRados::update_gc_chain(rgw_obj&, RGWObjManifest&, cls_rgw_obj_chain*)+0x51a) [0x55d2bd0939ea] 8: (RGWRados::Object::complete_atomic_modification()+0x83) [0x55d2bd093c63] 9: (RGWRados::Object::Delete::delete_obj()+0x74d) [0x55d2bd0a87ad] 10: (RGWDeleteObj::execute()+0x915) [0x55d2bd04b6d5] 11: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x915) [0x55d2bcdfbb35] 12: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::string const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x1cd8) [0x55d2bcdfdea8] 13: (RGWCivetWebFrontend::process(mg_connection*)+0x38e) [0x55d2bcd41a1e] 14: (()+0x36bace) [0x55d2bcdccace] 15: (()+0x36d76f) [0x55d2bcdce76f] 16: (()+0x36dc18) [0x55d2bcdcec18] 17: (()+0x7dd5) [0x7f161d338dd5] 18: (clone()+0x6d) [0x7f161c84302d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. My guess is that we need to add more resources to the gateways? They have 2 CPUs and 12GB of memory running as virtual machines on centOS 7.6 . Any thoughts? -Brent

2 2

Re: Nautilus - osdmap not trimming
by m.sliwinski＠lh.pl 12 Nov '20

12 Nov '20

Hi Thanks for the reply. Yeah, i restarted all of the mon servers, in sequence, and yesterday just leader alone without any success. Reports: root@monb01:~# ceph report | grep committed report 4002437698 "monmap_first_committed": 1, "monmap_last_committed": 6, "osdmap_first_committed": 67114, "osdmap_last_committed": 72592, "mdsmap_first_committed": 1, "mdsmap_last_committed": 1, "first_committed": 609225, "last_committed": 609251, "first_committed": 180754137, "last_committed": 180754777, root@monb01:~# root@monb01:~# ceph report | jq .osdmap_clean_epochs report 395175214 { "min_last_epoch_clean": 72592, "last_epoch_clean": { "per_pool": [ { "poolid": 0, "floor": 72592 }, { "poolid": 1, "floor": 72592 }, { "poolid": 2, "floor": 72592 }, { "poolid": 3, "floor": 72592 }, { "poolid": 4, "floor": 72592 }, { "poolid": 5, "floor": 72592 }, { "poolid": 26, "floor": 72592 }, { "poolid": 27, "floor": 72592 }, { "poolid": 28, "floor": 72592 } ] }, "osd_epochs": [ { "id": 0, "epoch": 72592 }, { "id": 1, "epoch": 72592 }, { "id": 2, "epoch": 72592 }, { "id": 3, "epoch": 72592 }, { "id": 4, "epoch": 72592 }, { "id": 5, "epoch": 72592 }, { "id": 6, "epoch": 72592 }, { "id": 7, "epoch": 72592 }, { "id": 8, "epoch": 72592 }, { "id": 9, "epoch": 72592 }, { "id": 10, "epoch": 72592 }, { "id": 11, "epoch": 72592 }, { "id": 12, "epoch": 72592 }, { "id": 13, "epoch": 72592 }, { "id": 14, "epoch": 72592 }, { "id": 15, "epoch": 72592 }, { "id": 16, "epoch": 72592 }, { "id": 17, "epoch": 72592 }, { "id": 18, "epoch": 72592 }, { "id": 19, "epoch": 72592 }, { "id": 20, "epoch": 72592 }, { "id": 21, "epoch": 72592 }, { "id": 22, "epoch": 72592 }, { "id": 23, "epoch": 72592 }, { "id": 24, "epoch": 72592 }, { "id": 25, "epoch": 72592 }, { "id": 26, "epoch": 72592 }, { "id": 27, "epoch": 72592 }, { "id": 28, "epoch": 72592 }, { "id": 29, "epoch": 72592 }, { "id": 30, "epoch": 72592 }, { "id": 31, "epoch": 72592 }, { "id": 32, "epoch": 72592 }, { "id": 33, "epoch": 72592 }, { "id": 34, "epoch": 72592 }, { "id": 35, "epoch": 72592 }, { "id": 36, "epoch": 72592 }, { "id": 37, "epoch": 72592 }, { "id": 38, "epoch": 72592 }, { "id": 39, "epoch": 72592 }, { "id": 40, "epoch": 72592 }, { "id": 41, "epoch": 72592 }, { "id": 42, "epoch": 72592 }, { "id": 43, "epoch": 72592 }, { "id": 44, "epoch": 72592 }, { "id": 45, "epoch": 72592 }, { "id": 46, "epoch": 72592 }, { "id": 47, "epoch": 72592 }, { "id": 48, "epoch": 72592 }, { "id": 49, "epoch": 72592 }, { "id": 50, "epoch": 72592 }, { "id": 51, "epoch": 72592 }, { "id": 52, "epoch": 72592 }, { "id": 53, "epoch": 72592 }, { "id": 54, "epoch": 72592 }, { "id": 55, "epoch": 72592 }, { "id": 56, "epoch": 72592 }, { "id": 57, "epoch": 72592 }, { "id": 58, "epoch": 72592 }, { "id": 59, "epoch": 72592 }, { "id": 60, "epoch": 72592 }, { "id": 61, "epoch": 72592 }, { "id": 62, "epoch": 72592 }, { "id": 63, "epoch": 72592 }, { "id": 64, "epoch": 72592 }, { "id": 65, "epoch": 72592 }, { "id": 66, "epoch": 72592 }, { "id": 67, "epoch": 72592 }, { "id": 68, "epoch": 72592 }, { "id": 69, "epoch": 72592 }, { "id": 70, "epoch": 72592 }, { "id": 71, "epoch": 72592 }, { "id": 72, "epoch": 72592 }, { "id": 73, "epoch": 72592 }, { "id": 74, "epoch": 72592 }, { "id": 75, "epoch": 72592 }, { "id": 76, "epoch": 72592 }, { "id": 77, "epoch": 72592 }, { "id": 78, "epoch": 72592 }, { "id": 79, "epoch": 72592 }, { "id": 80, "epoch": 72592 }, { "id": 81, "epoch": 72592 }, { "id": 83, "epoch": 72592 }, { "id": 84, "epoch": 72592 }, { "id": 85, "epoch": 72592 }, { "id": 86, "epoch": 72592 }, { "id": 87, "epoch": 72592 }, { "id": 88, "epoch": 72592 }, { "id": 89, "epoch": 72592 }, { "id": 90, "epoch": 72592 }, { "id": 91, "epoch": 72592 }, { "id": 92, "epoch": 72592 }, { "id": 93, "epoch": 72592 }, { "id": 94, "epoch": 72592 }, { "id": 95, "epoch": 72592 }, { "id": 96, "epoch": 72592 }, { "id": 97, "epoch": 72592 }, { "id": 98, "epoch": 72592 }, { "id": 99, "epoch": 72592 }, { "id": 100, "epoch": 72592 }, { "id": 101, "epoch": 72592 }, { "id": 102, "epoch": 72592 }, { "id": 103, "epoch": 72592 }, { "id": 104, "epoch": 72592 }, { "id": 105, "epoch": 72592 }, { "id": 106, "epoch": 72592 }, { "id": 107, "epoch": 72592 }, { "id": 108, "epoch": 72592 }, { "id": 109, "epoch": 72592 }, { "id": 110, "epoch": 72592 }, { "id": 111, "epoch": 72592 }, { "id": 112, "epoch": 72592 }, { "id": 113, "epoch": 72592 }, { "id": 114, "epoch": 72592 }, { "id": 115, "epoch": 72592 }, { "id": 116, "epoch": 72592 }, { "id": 117, "epoch": 72592 }, { "id": 118, "epoch": 72592 }, { "id": 119, "epoch": 72592 }, { "id": 120, "epoch": 72592 }, { "id": 121, "epoch": 72592 }, { "id": 122, "epoch": 72592 }, { "id": 123, "epoch": 72592 }, { "id": 124, "epoch": 72592 }, { "id": 125, "epoch": 72592 }, { "id": 126, "epoch": 72592 }, { "id": 127, "epoch": 72592 }, { "id": 128, "epoch": 72592 }, { "id": 129, "epoch": 72592 }, { "id": 130, "epoch": 72592 }, { "id": 131, "epoch": 72592 }, { "id": 132, "epoch": 72592 }, { "id": 133, "epoch": 72592 }, { "id": 134, "epoch": 72592 }, { "id": 135, "epoch": 72592 }, { "id": 136, "epoch": 72592 }, { "id": 137, "epoch": 72592 }, { "id": 138, "epoch": 72592 }, { "id": 139, "epoch": 72592 }, { "id": 140, "epoch": 72592 }, { "id": 141, "epoch": 72592 }, { "id": 142, "epoch": 72592 }, { "id": 143, "epoch": 72592 }, { "id": 144, "epoch": 72592 }, { "id": 145, "epoch": 72592 }, { "id": 146, "epoch": 72592 }, { "id": 147, "epoch": 72592 }, { "id": 148, "epoch": 72592 }, { "id": 149, "epoch": 72592 }, { "id": 150, "epoch": 72592 }, { "id": 151, "epoch": 72592 }, { "id": 152, "epoch": 72592 }, { "id": 153, "epoch": 72592 }, { "id": 154, "epoch": 72592 }, { "id": 155, "epoch": 72592 }, { "id": 156, "epoch": 72592 }, { "id": 157, "epoch": 72592 }, { "id": 158, "epoch": 72592 }, { "id": 159, "epoch": 72592 }, { "id": 160, "epoch": 72592 }, { "id": 161, "epoch": 72592 }, { "id": 162, "epoch": 72592 }, { "id": 163, "epoch": 72592 }, { "id": 164, "epoch": 72592 }, { "id": 165, "epoch": 72592 }, { "id": 166, "epoch": 72592 }, { "id": 167, "epoch": 72592 }, { "id": 168, "epoch": 72592 }, { "id": 169, "epoch": 72592 }, { "id": 170, "epoch": 72592 }, { "id": 171, "epoch": 72592 }, { "id": 172, "epoch": 72592 }, { "id": 173, "epoch": 72592 }, { "id": 174, "epoch": 72592 }, { "id": 175, "epoch": 72592 }, { "id": 176, "epoch": 72592 }, { "id": 177, "epoch": 72592 }, { "id": 178, "epoch": 72592 }, { "id": 179, "epoch": 72592 }, { "id": 180, "epoch": 72592 }, { "id": 181, "epoch": 72592 }, { "id": 182, "epoch": 72592 }, { "id": 183, "epoch": 72592 }, { "id": 184, "epoch": 72592 }, { "id": 185, "epoch": 72592 }, { "id": 186, "epoch": 72592 }, { "id": 187, "epoch": 72592 }, { "id": 188, "epoch": 72592 }, { "id": 189, "epoch": 72592 }, { "id": 190, "epoch": 72592 }, { "id": 191, "epoch": 72592 }, { "id": 192, "epoch": 72592 }, { "id": 193, "epoch": 72592 }, { "id": 194, "epoch": 72592 }, { "id": 195, "epoch": 72592 }, { "id": 196, "epoch": 72592 }, { "id": 197, "epoch": 72592 }, { "id": 198, "epoch": 72592 }, { "id": 199, "epoch": 72592 }, { "id": 200, "epoch": 72592 }, { "id": 201, "epoch": 72592 }, { "id": 202, "epoch": 72592 }, { "id": 203, "epoch": 72592 }, { "id": 204, "epoch": 72592 }, { "id": 205, "epoch": 72592 }, { "id": 206, "epoch": 72592 }, { "id": 207, "epoch": 72592 }, { "id": 208, "epoch": 72592 }, { "id": 209, "epoch": 72592 }, { "id": 210, "epoch": 72592 }, { "id": 211, "epoch": 72592 }, { "id": 212, "epoch": 72592 }, { "id": 213, "epoch": 72592 }, { "id": 214, "epoch": 72592 }, { "id": 215, "epoch": 72592 }, { "id": 216, "epoch": 72592 }, { "id": 217, "epoch": 72592 }, { "id": 218, "epoch": 72592 }, { "id": 219, "epoch": 72592 }, { "id": 220, "epoch": 72592 }, { "id": 221, "epoch": 72592 }, { "id": 222, "epoch": 72592 }, { "id": 223, "epoch": 72592 }, { "id": 224, "epoch": 72592 }, { "id": 225, "epoch": 72592 }, { "id": 226, "epoch": 72592 }, { "id": 227, "epoch": 72592 }, { "id": 228, "epoch": 72592 }, { "id": 229, "epoch": 72592 }, { "id": 230, "epoch": 72592 }, { "id": 231, "epoch": 72592 }, { "id": 232, "epoch": 72592 }, { "id": 233, "epoch": 72592 }, { "id": 234, "epoch": 72592 }, { "id": 235, "epoch": 72592 }, { "id": 236, "epoch": 72592 }, { "id": 237, "epoch": 72592 }, { "id": 238, "epoch": 72592 }, { "id": 239, "epoch": 72592 }, { "id": 240, "epoch": 72592 }, { "id": 241, "epoch": 72592 }, { "id": 242, "epoch": 72592 }, { "id": 243, "epoch": 72592 }, { "id": 244, "epoch": 72592 }, { "id": 245, "epoch": 72592 }, { "id": 246, "epoch": 72592 }, { "id": 247, "epoch": 72592 }, { "id": 248, "epoch": 72592 }, { "id": 249, "epoch": 72592 }, { "id": 250, "epoch": 72592 }, { "id": 251, "epoch": 72592 }, { "id": 252, "epoch": 72592 }, { "id": 253, "epoch": 72592 }, { "id": 254, "epoch": 72592 }, { "id": 255, "epoch": 72592 }, { "id": 256, "epoch": 72592 }, { "id": 257, "epoch": 72592 }, { "id": 258, "epoch": 72592 }, { "id": 259, "epoch": 72592 }, { "id": 260, "epoch": 72592 }, { "id": 261, "epoch": 72592 }, { "id": 262, "epoch": 72592 }, { "id": 263, "epoch": 72592 }, { "id": 264, "epoch": 72592 }, { "id": 265, "epoch": 72592 }, { "id": 266, "epoch": 72592 }, { "id": 267, "epoch": 72592 }, { "id": 268, "epoch": 72592 }, { "id": 269, "epoch": 72592 }, { "id": 270, "epoch": 72592 }, { "id": 271, "epoch": 72592 }, { "id": 272, "epoch": 72592 }, { "id": 273, "epoch": 72592 }, { "id": 274, "epoch": 72592 }, { "id": 275, "epoch": 72592 }, { "id": 276, "epoch": 72592 }, { "id": 277, "epoch": 72592 }, { "id": 278, "epoch": 72592 }, { "id": 279, "epoch": 72592 }, { "id": 280, "epoch": 72592 }, { "id": 281, "epoch": 72592 }, { "id": 282, "epoch": 72592 }, { "id": 283, "epoch": 72592 }, { "id": 284, "epoch": 72592 } ] } root@monb01:~# W dniu 2020-11-12 11:58, Dan van der Ster napisał(a): > I found another possible trimming bug this morning, but I don't expect > it applies to you because you said you restarted the mon leader: > https://tracker.ceph.com/issues/48212 > > Otherwise, couple you please share the output of > > ceph report | grep committed > ceph report | jq .osdmap_clean_epochs > > Thanks, > > Dan > > On Thu, Nov 12, 2020 at 10:56 AM <m.sliwinski(a)lh.pl> wrote: >> >> Hi >> >> Thanks for the response. Our cluster is currently mostly on 14.2.13, >> especially all MONs and MGRs are. >> Some OSDs are still on 14.2.9, but it shouldn't block osdmap trimming >> i >> think, because atm we don't have any down OSDs, i checked for that >> when >> we first noticed the issue. >> I'm working of course on bringing all OSDs to 14.2.13, but it will >> take >> some time as i have to create and test Debian packages for that. >> Could there be any other reason? I found posts about PGs for new pool >> not bein marked as created in MON db while cluster still reporting >> everything as active+clean, but i dont know how to debug that. >> >> -- >> Best regards >> Marcin >> >> >> W dniu 2020-11-11 16:50, Dan van der Ster napisał(a): >> > Hi, >> > >> > v14.2.13 has an important fix in this area: >> > https://tracker.ceph.com/issues/47290 >> > Without this fix, your cluster will not trim if there are any *down* >> > osds in the cluster. >> > >> > On our clusters we are running v14.2.11 patched with commit >> > "mon/OSDMonitor: only take in osd into consideration when trimming >> > osdmaps" -- this trims maps perfectly afaict. >> > >> > I can't vouch for the rest of 14.2.13, so better test that adequately >> > before upgrading. >> > >> > Cheers, Dan >> > >> > >> > On Tue, Nov 10, 2020 at 6:57 PM <m.sliwinski(a)lh.pl> wrote: >> >> >> >> Hi >> >> >> >> We have ceph cluster running on Nautilus, recently upgraded from >> >> Mimic. >> >> When in Mimic we noticed issue with osdmap not trimming, which caused >> >> part of our cluster to crash due to osdmap cache misses. We solved it >> >> by >> >> adding "osd_map_cache_size = 5000" to our ceph.conf >> >> Because we had at that time mixed OSD versions from both Mimic and >> >> Nautilus we decided to finish upgrade, but it didn't solve our >> >> problem. >> >> We have at the moment: "oldest_map": 67114, "newest_map": 72588,and >> >> the >> >> difference is not shrinking even thought cluster is in active+clean >> >> state. Restarting all mon's didn't help. It seems bug is similar to >> >> https://tracker.ceph.com/issues/44184 but there's no solution there. >> >> What else can i check or do? >> >> I don't want do to cangerous things like mon_osd_force_trim_to or >> >> something similar without finding the cause. >> >> >> >> I noticed in MON debug log: >> >> >> >> 2020-11-10 17:11:14.612 7f9592d5b700 10 mon.monb01(a)0(leader).osd >> >> e72571 >> >> should_prune could only prune 4957 epochs (67114..72071), which is >> >> less >> >> than the required minimum (10000) >> >> 2020-11-10 17:11:19.612 7f9592d5b700 10 mon.monb01(a)0(leader).osd >> >> e72571 >> >> should_prune could only prune 4957 epochs (67114..72071), which is >> >> less >> >> than the required minimum (10000) >> >> >> >> So i added config options to reduce those values: >> >> >> >> mon dev mon_debug_block_osdmap_trim false >> >> mon advanced mon_min_osdmap_epochs 100 >> >> mon advanced mon_osdmap_full_prune_min 500 >> >> mon advanced paxos_service_trim_min 10 >> >> >> >> But it didn't help: >> >> >> >> 2020-11-10 18:28:26.165 7f1b700ab700 20 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest osdmap manifest detected in store; reload. >> >> 2020-11-10 18:28:26.169 7f1b700ab700 10 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest store osdmap manifest pinned (67114 .. 72484) >> >> 2020-11-10 18:28:26.169 7f1b700ab700 10 mon.monb01(a)0(leader).osd >> >> e72588 >> >> should_prune not enough epochs to form an interval (last pinned: >> >> 72484, >> >> last to pin: 72488, interval: 10) >> >> >> >> Command "ceph report | jq '.osdmap_manifest' |jq '.pinned_maps[]'" >> >> shows >> >> 67114 on the top, but i'm unable to determine why. >> >> >> >> Same with 'ceph report | jq .osdmap_first_committed': >> >> >> >> root@monb01:/var/log/ceph# ceph report | jq .osdmap_first_committed >> >> report 4073203295 >> >> 67114 >> >> root@monb01:/var/log/ceph# >> >> >> >> When i try to derermine if a certain PG or OSD is keeping it so low i >> >> don't get anything. >> >> >> >> And in MON debug log i get: >> >> >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01@0(leader) e6 >> >> refresh_from_paxos >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 >> >> mon.monb01(a)0(leader).paxosservice(mdsmap 1..1) refresh >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 >> >> mon.monb01(a)0(leader).paxosservice(osdmap 67114..72588) refresh >> >> 2020-11-10 18:42:41.767 7f1b74721700 20 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest osdmap manifest detected in store; reload. >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest store osdmap manifest pinned (67114 .. 72484) >> >> >> >> I also get: >> >> >> >> root@monb01:/var/log/ceph# ceph report |grep "min_last_epoch_clean" >> >> report 2716976759 >> >> "min_last_epoch_clean": 0, >> >> root@monb01:/var/log/ceph# >> >> >> >> >> >> Additional info: >> >> root@monb01:/var/log/ceph# ceph versions >> >> { >> >> "mon": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 3 >> >> }, >> >> "mgr": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 3 >> >> }, >> >> "osd": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 120, >> >> "ceph version 14.2.9 >> >> (581f22da52345dba46ee232b73b990f06029a2a0) >> >> nautilus (stable)": 164 >> >> }, >> >> "mds": {}, >> >> "overall": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 126, >> >> "ceph version 14.2.9 >> >> (581f22da52345dba46ee232b73b990f06029a2a0) >> >> nautilus (stable)": 164 >> >> } >> >> } >> >> >> >> >> >> root@monb01:/var/log/ceph# ceph mon feature ls >> >> >> >> all features >> >> supported: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> persistent: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> on current monmap (epoch 6) >> >> persistent: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> required: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> >> >> >> >> root@monb01:/var/log/ceph# ceph osd dump | grep require >> >> require_min_compat_client luminous >> >> require_osd_release nautilus >> >> >> >> >> >> root@monb01:/var/log/ceph# ceph report | jq >> >> '.osdmap_manifest.pinned_maps | length' >> >> report 1777129876 >> >> 538 >> >> >> >> root@monb01:/var/log/ceph# ceph pg dump -f json | jq .osd_epochs >> >> dumped all >> >> null >> >> >> >> -- >> >> Best regards >> >> Marcin >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

How to run ceph_osd_dump
by Denis Krienbühl 12 Nov '20

12 Nov '20

Hi We’ve recently encountered the following errors: [WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 2752.832ms) Slow OSD heartbeats on back from osd.2 [nvme-a] to osd.290 [nvme-c] 2752.832 msec ... Truncated long network list. Use ceph daemon mgr.# dump_osd_network for more information To get more information we wanted to run the dump_osd_network command, but it doesn’t seem to be a valid command: ceph daemon /var/run/ceph/ceph-mgr.$(hostname).asok dump_osd_network 0 no valid command found; 10 closest matches: 0 1 2 abort assert config diff config diff get <var> config get <var> config help [<var>] config set <var> <val>... admin_socket: invalid command Other commands, like ceph daemon dump_cache work, so it seems to hit the right socket. What am I doing wrong? Cheers, Denis

2 2

How to use ceph-volume to create multiple OSDs per NVMe disk, and with fixed WAL/DB partition on another device?
by victorhooi＠yahoo.com 12 Nov '20

12 Nov '20

I'm building a new 4-node Proxmox/Ceph cluster, to hold disk images for our VMs. (Ceph version is 15.2.5). Each node has 6 x NVMe SSDs (4TB), and 1 x Optane drive (960GB). CPU is AMD Rome 7442, so there should be plenty of CPU capacity to spare. My aim is to create 4 x OSDs per NVMe SSD (to make more effective use of the NVMe performance) and use the Optane drive to store the WAL/DB partition for each OSD. (I.e. total of 24 x 35GB WAL/DB partitions). However, I am struggling to get the right ceph-volume command to achieve this. Thanks to a very kind Redditor, I was able to get close: /dev/nvme0n1 is an Optane device (900GB). /dev/nvme2n1 is an Intel NVMe SSD (4TB). ``` # ceph-volume lvm batch --osds-per-device 4 /dev/nvme2n1 --db-devices /dev/nvme0n1 Total OSDs: 4 Solid State VG: Targets: block.db Total size: 893.00 GB Total LVs: 16 Size per LV: 223.25 GB Devices: /dev/nvme0n1 Type Path LV Size % of device ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) ``` This does split up the NVMe disk into 4 OSDs, and creates WAL/DB partition on the Optane drive - however, it creates 4 x 223 GB partitions on the Optane (whereas I want 35GB partitions). Is there any way to specify the WAL/DB partition size in the above? And can it be done, such that you can run successive ceph-volume commands, to add the OSDs and WAL/DB partitions for each NVMe disk? (Or if there's an easier way to achieve the above layout, please let me know). That being said - I also just saw this ceph-users thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/3Y6DEJCF7ZM… It talks there about "osd op num shards" and "osd op num threads per shard" - is there some way to set those, to achieve similar performance to say, 4 x OSDs per NVMe drive, but with only 1 x NVMe? Has anybody done any testing/benchmarking on this they can share?

5 4