- ceph-users - lists.ceph.io

Resolving LARGE_OMAP_OBJECTS
by Drew Weaver 30 Mar '21

30 Mar '21

Howdy, the dashboard on our cluster keeps showing LARGE_OMAP_OBJECTS. I went through this document https://www.suse.com/support/kb/doc/?id=000019698 I've found that we have a total of 5 buckets, each one is owned by a different user. From what I have read on this issue it seems to flip flop between this is an actual problem that will cause real world issues to "we just raised the limit in the next version". Does anyone have any expertise on whether this is an actual problem or if we should just tune the numbers and how do you determine that? One other quick question: is there a way to add usage information for buckets into mgr for version 14? Thanks, -Drew

3 8

OSD Crash During Deep-Scrub
by Dave Hall 29 Mar '21

29 Mar '21

Hello, A while back, I was having an issue with an OSD repeatedly crashing. I ultimately reweighted it to zero and then marked out 'Out'. Since I found that the logs for thoses crashes match https://tracker.ceph.com/issues/46490 . Since the OSD is in a 'Safe-to-Destroy' state, I'm wondering the best course of action - should I just mark it back in? Or should I destroy and rebuild it. If clearing it in the way I have, in combination with updating to 14.2.16, will prevent it from misbehaving, why go through the trouble of destroying and rebuilding? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office)

2 1

Re: Nautilus - PG Autoscaler Gobal vs Pool Setting
by Dave Hall 29 Mar '21

29 Mar '21

All, In looking at the options for setting the default pg autoscale option, I notice that there is a global option setting and a per-pool option setting. It seems that the options at the pool level are off, warn, and on. The same, I assume for the global setting. Is there a way to get rid of the per-pool setting and set the pool to honor the global setting? I think I'm looking for 'off, warn, on, or global'. It seems that once the per-pool option is set for all of one's pools, the global value is irrelevant. This also implies that in a circumstance where one would want to temporarily suspend autoscaling it would be required to modify the setting for each pool and then to modify it back afterward. Thoughts? Thanks -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu On Mon, Mar 29, 2021 at 1:44 PM Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > Yes the PG autoscalar has a way of reducing PG count way too far. There’s > a claim that it’s better in Pacific, but I tend to recommend disabling it > and calculating / setting pg_num manually. > > > On Mar 29, 2021, at 9:06 AM, Dave Hall <kdhall(a)binghamton.edu> wrote: > > > > Eugen, > > > > I didn't really think my cluster was eating itself, but I also didn't > want > > to be in denial. > > > > Regarding the autoscaler, I really thought that it only went up - I > didn't > > expect that it would decrease the number of PGs. Plus, I thought I had > it > > turned off. I see now that it's off globally but enabled for this > > particular pool. Also, I see that the target PG count is lower than the > > current. > > > > I guess you learn something new every day. > > > > -Dave > > > > -- > > Dave Hall > > Binghamton University > > kdhall(a)binghamton.edu > > 607-760-2328 (Cell) > > 607-777-4641 (Office) > > > > > > On Mon, Mar 29, 2021 at 7:52 AM Eugen Block <eblock(a)nde.ag> wrote: > > > >> Hi, > >> > >> that sounds like the pg_autoscaler is doing its work. Check with: > >> > >> ceph osd pool autoscale-status > >> > >> I don't think ceph is eating itself or that you're losing data. ;-) > >> > >> > >> Zitat von Dave Hall <kdhall(a)binghamton.edu>: > >> > >>> Hello, > >>> > >>> About 3 weeks ago I added a node and increased the number of OSDs in my > >>> cluster from 24 to 32, and then marked one old OSD down because it was > >>> frequently crashing. . > >>> > >>> After adding the new OSDs the PG count jumped fairly dramatically, but > >> ever > >>> since, amidst a continuous low level of rebalancing, the number of PGs > >> has > >>> gradually decreased to less by 25% from it's max value. Although I > don't > >>> have specific notes, my perception is that the current number of PGs is > >>> actually lower than it was before I added OSDs. > >>> > >>> So what's going on here? It is possible to imagine that my cluster is > >>> slowly eating itself, and that I'm about to lose 200TB of data. It's > also > >>> possible to imagine that this is all due to the gradual optimization of > >> the > >>> pools. > >>> > >>> Note that the primary pool is an EC 8 + 2 containing about 124TB. > >>> > >>> Thanks. > >>> > >>> -Dave > >>> > >>> -- > >>> Dave Hall > >>> Binghamton University > >>> kdhall(a)binghamton.edu > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

Nautilus - PG count decreasing after adding OSDs
by Dave Hall 29 Mar '21

29 Mar '21

Hello, About 3 weeks ago I added a node and increased the number of OSDs in my cluster from 24 to 32, and then marked one old OSD down because it was frequently crashing. . After adding the new OSDs the PG count jumped fairly dramatically, but ever since, amidst a continuous low level of rebalancing, the number of PGs has gradually decreased to less by 25% from it's max value. Although I don't have specific notes, my perception is that the current number of PGs is actually lower than it was before I added OSDs. So what's going on here? It is possible to imagine that my cluster is slowly eating itself, and that I'm about to lose 200TB of data. It's also possible to imagine that this is all due to the gradual optimization of the pools. Note that the primary pool is an EC 8 + 2 containing about 124TB. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 2

Re: Cluster suspends when Add Mon or stop and start after a while.
by Frank Schilder 29 Mar '21

29 Mar '21

Please use the correct list: ceph-users(a)ceph.io Probably same problem I had. Try reducing mon_sync_max_payload_size=4096 and start a new MON. Should just take a few seconds to boot up. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder Sent: 29 March 2021 16:58:19 To: by morphin; ceph-users(a)ceph.com Subject: Re: [ceph-users] Cluster suspends when Add Mon or stop and start after a while. Probably same problem I had. Try reducing mon_sync_max_payload_size=4096 and start a new MON. Should just take a few seconds to boot up. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: by morphin <morphinwithyou(a)gmail.com> Sent: 28 March 2021 11:25:02 To: ceph-users(a)ceph.com Subject: [ceph-users] Cluster suspends when Add Mon or stop and start after a while. Hello! I have a cluster with Datacenter crushmap (A+B).(9+9 = 18 servers) The cluster started with v12.2.0 Luminous 4 years ago. All these years I upgraded the Cluster Luminous > Mimic > v14.2.16 Nautilus. Now I have a weird issue. When I add a mon or shutdown a while and start it up again, all the cluster suspends, ceph -s do not respond and other two monitors starting election while booting mon is syncing. (logs below) 2021-03-28 00:18:23.482 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:23.782 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:24.292 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:26.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3919 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:29.782 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7899) init, last seen epoch 7899, mid-election, bumping 2021-03-28 00:18:29.812 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:31.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3951 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:31.872 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:32.072 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:32.482 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:33.282 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:34.812 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7901) init, last seen epoch 7901, mid-election, bumping 2021-03-28 00:18:34.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:34.872 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:35.072 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:35.492 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:36.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3989 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:36.292 7fe2f2e0f700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:39.842 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7903) init, last seen epoch 7903, mid-election, bumping 2021-03-28 00:18:39.872 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:40.872 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.082 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4027 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:41.492 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.812 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:42.312 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:43.882 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:44.082 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:44.492 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:45.302 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:46.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4062 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:47.812 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7905) init, last seen epoch 7905, mid-election, bumping 2021-03-28 00:18:47.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:51.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4091 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:52.842 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7907) init, last seen epoch 7907, mid-election, bumping 2021-03-28 00:18:52.872 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:52.882 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:53.082 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:53.492 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:54.302 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:56.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4131 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:57.872 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7909) init, last seen epoch 7909, mid-election, bumping 2021-03-28 00:18:57.902 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:19:21.851 7f9127602700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:25.061 7f912ce0d700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:31.481 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:44.291 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:59.311 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:20:14.321 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:20:54.481 7f9128604700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #1158. Immutable memtables: 0. 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.490450) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 5] Flushing memtable with next log file: 1158 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054490507, "job": 5, "event": "flush_started", "num_memtables": 1, "num_entries": 391495, "num_deletes": 0, "total_data_size": 26756127, "memory_usage": 32861536, "flush_reason": "Write Buffer Full"} 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 5] Level-0 flush table #1159: started 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054615407, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 1159, "file_size": 14913671, "table_properties": {"data_size": 13733422, "index_size": 200663, "filter_size": 978757, "raw_key_size": 18143237, "raw_average_key_size": 46, "raw_value_size": 7829900, "raw_average_value_size": 20, "num_data_blocks": 3398, "num_entries": 391495, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 5] Level-0 flush table #1159: 14913671 bytes OK 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615451) [db/memtable_list.cc:380] [default] Level-0 commit table #1159 started 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615892) [db/memtable_list.cc:428] [default] Level-0 commit table #1159: memtable #1 done 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615900) EVENT_LOG_v1 {"time_micros": 1616880054615897, "job": 5, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [4, 0, 0, 0, 0, 0, 2], "immutable_memtables": 0} 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615925) [db/db_impl_compaction_flush.cc:201] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[4 0 0 0 0 0 2] max score 1.00 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: [db/db_impl_files.cc:356] [JOB 5] Try to delete WAL files size 24016022, prev total WAL file size 24798914, number of live WAL files 2. 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 6] Compacting 4@0 + 2@6 files to L6, score 1.00 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 5 Base level 0, inputs: [1159(14MB) 1155(25MB) 1149(591KB) 1157(24MB)], [1146(68MB) 1147(53MB)] 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054616197, "job": 6, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [1159, 1155, 1149, 1157], "files_L6": [1146, 1147], "score": 1, "input_data_size": 195345305} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 6] Generated table #1160: 547031 keys, 40744575 bytes 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055675107, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 1160, "file_size": 40744575, "table_properties": {"data_size": 39048586, "index_size": 327543, "filter_size": 1367621, "raw_key_size": 25391768, "raw_average_key_size": 46, "raw_value_size": 30761471, "raw_average_value_size": 56, "num_data_blocks": 6004, "num_entries": 547031, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 6] Compacted 4@0 + 2@6 files to L6 => 40744575 bytes 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: (Original Log Time 2021/03/28-00:20:55.676525) [db/compaction_job.cc:768] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 184.5 rd, 38.5 wr, level 6, files in(4, 2) out(1) MB in(64.7, 121.6) out(38.9), read-write-amplify(3.5) write-amplify(0.6) OK, records in: 3571913, records dropped: 3024882 output_compression: NoCompression 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: (Original Log Time 2021/03/28-00:20:55.676541) EVENT_LOG_v1 {"time_micros": 1616880055676534, "job": 6, "event": "compaction_finished", "compaction_time_micros": 1058962, "compaction_time_cpu_micros": 1047128, "output_level": 6, "num_output_files": 1, "total_output_size": 40744575, "num_input_records": 3571913, "num_output_records": 547031, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055677454, "job": 6, "event": "table_file_deletion", "file_number": 1159} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055678431, "job": 6, "event": "table_file_deletion", "file_number": 1157} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055679410, "job": 6, "event": "table_file_deletion", "file_number": 1155} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055679506, "job": 6, "event": "table_file_deletion", "file_number": 1149} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055681388, "job": 6, "event": "table_file_deletion", "file_number": 1147} 2021-03-28 00:20:55.681 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055684258, "job": 6, "event": "table_file_deletion", "file_number": 1146} As you can see in the leader mon switchs to election mod and gives these logs until the sync is finished. mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id The booting mon only gives this: mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id The booting mon prepares its *.stt tables and I watched the var/lib/ceph/mon/*/*/*.log file and I saw these logs: !osd_snapremoved_epoch_1_0013c460,�!osd_snapremoved_epoch_1_0013c462.� I guess after all these upgrades Ceph missed some variable or something else. The problem is huge because I can not bootstrap a mon without stopping entire cluster. It takes 30min to boot up... I tried to remove and add new mons but nothing changed. I've a backup service and it takes snapshots and backup to blockstorage and deletes the snapshot. It works for 3 years now and the snapremoved log in th mon log under "var/lib/ceph/mon/*/*/*.log" makes me wonder, what if all these snapshots makes huge bottleneck to the mon bootup? What should I do? _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

bucket index and WAL/DB
by Marcelo 29 Mar '21

29 Mar '21

Hello everybody. I searched in several places and I couldn't find any information about what the best bucket index and WAL / DB organization would be. I have several hosts consisting of 12 HDDs and 2 NVMes, and currently one of the NVMes serves as WAL / DB for the 10 OSDs and the other NVMe is partitioned in two, serving as 2 OSDs to serve the S3 index pool. I saw in ceph-ansible a playbook (infrastructure-playbooks / lv-create.yml) that creates a division where we have an OSD living with a journal on the same NVMe. The problem is that in lv-vars.yaml used by lv-create.yml it is said that this only applies to the filestore. Is this correct or can I use this same structure with bluestore? Thank you all, Marcelo.

2 4

Nautilus: Reduce the number of managers
by Dave Hall 29 Mar '21

29 Mar '21

Hello, We are in the process of bringing new hardware online that will allow us to get all of the MGRs, MONs, MDSs, etc. off of our OSD nodes and onto dedicated management nodes. I've created MGRs and MONs on the new nodes, and I found procedures for disabling the MONs from the OSD nodes. Now I'm looking for the correct procedure to remove the MGRs from the OSD nodes. I haven't found any reference to this in the documentation. Is it as simple as stopping and disabling the systemd service/target? Or are there Ceph commands? Do I need to clean up /var/lib/ceph/mgr? Same questions about MDS in the near term, but I haven't searched the docs yet. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 1

memory consumption by osd
by Tony Liu 29 Mar '21

29 Mar '21

Hi, Here is a snippet from top on a node with 10 OSDs. =========================== MiB Mem : 257280.1 total, 2070.1 free, 31881.7 used, 223328.3 buff/cache MiB Swap: 128000.0 total, 126754.7 free, 1245.3 used. 221608.0 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30492 167 20 0 4483384 2.9g 16696 S 6.0 1.2 707:05.25 ceph-osd 35396 167 20 0 4444952 2.8g 16468 S 5.0 1.1 815:58.52 ceph-osd 33488 167 20 0 4161872 2.8g 16580 S 4.7 1.1 496:07.94 ceph-osd 36371 167 20 0 4387792 3.0g 16748 S 4.3 1.2 762:37.64 ceph-osd 39185 167 20 0 5108244 3.1g 16576 S 4.0 1.2 998:06.73 ceph-osd 38729 167 20 0 4748292 2.8g 16580 S 3.3 1.1 895:03.67 ceph-osd 34439 167 20 0 4492312 2.8g 16796 S 2.0 1.1 921:55.50 ceph-osd 31473 167 20 0 4314500 2.9g 16684 S 1.3 1.2 680:48.09 ceph-osd 32495 167 20 0 4294196 2.8g 16552 S 1.0 1.1 545:14.53 ceph-osd 37230 167 20 0 4586020 2.7g 16620 S 1.0 1.1 844:12.23 ceph-osd =========================== Does it look OK with 2GB free? I can't tell how that 220GB is used for buffer/cache. Is that used by OSDs? Is it controlled by configuration or auto scaling based on physical memory? Any clarifications would be helpful. Thanks! Tony

5 7

Re: How to clear Health Warning status?
by jinguk.kwon＠ungleich.ch 29 Mar '21

29 Mar '21

Hello there, Thank you for your response. There is no error at syslog, dmesg, or SMART. # ceph health detail HEALTH_WARN Too many repaired reads on 2 OSDs OSD_TOO_MANY_REPAIRS Too many repaired reads on 2 OSDs osd.29 had 38 reads repaired osd.16 had 17 reads repaired How can i clear this waning ? My ceph is version 14.2.9(clear_shards_repaired is not supported.) /dev/sdh1 on /var/lib/ceph/osd/ceph-16 type xfs (rw,relatime,attr2,inode64,noquota) # cat dmesg | grep sdh [ 12.990728] sd 5:2:3:0: [sdh] 19531825152 512-byte logical blocks: (10.0 TB/9.09 TiB) [ 12.990728] sd 5:2:3:0: [sdh] Write Protect is off [ 12.990728] sd 5:2:3:0: [sdh] Mode Sense: 1f 00 00 08 [ 12.990728] sd 5:2:3:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 13.016616] sdh: sdh1 sdh2 [ 13.017780] sd 5:2:3:0: [sdh] Attached SCSI disk # ceph tell osd.29 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 6.464404, "bytes_per_sec": 166100668.21318716, "iops": 39.60148530320815 } # ceph tell osd.16 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 9.6168945000000008, "bytes_per_sec": 111651617.26584397, "iops": 26.619819942914003 } Thank you > On 26 Mar 2021, at 16:04, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > Did you look at syslog, dmesg, or SMART? Mostly likely the drives are failing. > > >> On Mar 25, 2021, at 9:55 PM, jinguk.kwon(a)ungleich.ch wrote: >> >> Hello there, >> >> Thank you for advanced. >> My ceph is ceph version 14.2.9 >> I have a repair issue too. >> >> ceph health detail >> HEALTH_WARN Too many repaired reads on 2 OSDs >> OSD_TOO_MANY_REPAIRS Too many repaired reads on 2 OSDs >> osd.29 had 38 reads repaired >> osd.16 had 17 reads repaired >> >> ~# ceph tell osd.16 bench >> { >> "bytes_written": 1073741824, >> "blocksize": 4194304, >> "elapsed_sec": 7.1486738159999996, >> "bytes_per_sec": 150201541.10217974, >> "iops": 35.81083800844663 >> } >> ~# ceph tell osd.29 bench >> { >> "bytes_written": 1073741824, >> "blocksize": 4194304, >> "elapsed_sec": 6.9244327500000002, >> "bytes_per_sec": 155065672.9246161, >> "iops": 36.970537406114602 >> } >> >> But it looks like those osds are ok. how can i clear this warning ? >> >> Best regards >> JG >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Do I need to update ceph.conf and restart each OSD after adding more MONs?
by Tony Liu 29 Mar '21

29 Mar '21

Hi, Do I need to update ceph.conf and restart each OSD after adding more MONs? This is with 15.2.8 deployed by cephadm. When adding MON, "mon_host" should be updated accordingly. Given [1], is that update "the monitor cluster’s centralized configuration database" or "runtime overrides set by an administrator"? [1] https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#config-sourc… Thanks! Tony

3 6