March 2021 - ceph-users - lists.ceph.io

by jinguk.kwon＠ungleich.ch

Hello there, Thank you for your response. There is no error at syslog, dmesg, or SMART. # ceph health detail HEALTH_WARN Too many repaired reads on 2 OSDs OSD_TOO_MANY_REPAIRS Too many repaired reads on 2 OSDs osd.29 had 38 reads repaired osd.16 had 17 reads repaired How can i clear this waning ? My ceph is version 14.2.9(clear_shards_repaired is not supported.) /dev/sdh1 on /var/lib/ceph/osd/ceph-16 type xfs (rw,relatime,attr2,inode64,noquota) # cat dmesg | grep sdh [ 12.990728] sd 5:2:3:0: [sdh] 19531825152 512-byte logical blocks: (10.0 TB/9.09 TiB) [ 12.990728] sd 5:2:3:0: [sdh] Write Protect is off [ 12.990728] sd 5:2:3:0: [sdh] Mode Sense: 1f 00 00 08 [ 12.990728] sd 5:2:3:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 13.016616] sdh: sdh1 sdh2 [ 13.017780] sd 5:2:3:0: [sdh] Attached SCSI disk # ceph tell osd.29 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 6.464404, "bytes_per_sec": 166100668.21318716, "iops": 39.60148530320815 } # ceph tell osd.16 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 9.6168945000000008, "bytes_per_sec": 111651617.26584397, "iops": 26.619819942914003 } Thank you > On 26 Mar 2021, at 16:04, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > Did you look at syslog, dmesg, or SMART? Mostly likely the drives are failing. > > >> On Mar 25, 2021, at 9:55 PM, jinguk.kwon(a)ungleich.ch wrote: >> >> Hello there, >> >> Thank you for advanced. >> My ceph is ceph version 14.2.9 >> I have a repair issue too. >> >> ceph health detail >> HEALTH_WARN Too many repaired reads on 2 OSDs >> OSD_TOO_MANY_REPAIRS Too many repaired reads on 2 OSDs >> osd.29 had 38 reads repaired >> osd.16 had 17 reads repaired >> >> ~# ceph tell osd.16 bench >> { >> "bytes_written": 1073741824, >> "blocksize": 4194304, >> "elapsed_sec": 7.1486738159999996, >> "bytes_per_sec": 150201541.10217974, >> "iops": 35.81083800844663 >> } >> ~# ceph tell osd.29 bench >> { >> "bytes_written": 1073741824, >> "blocksize": 4194304, >> "elapsed_sec": 6.9244327500000002, >> "bytes_per_sec": 155065672.9246161, >> "iops": 36.970537406114602 >> } >> >> But it looks like those osds are ok. how can i clear this warning ? >> >> Best regards >> JG >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

3 years, 1 month

2
1
0 0

Do I need to update ceph.conf and restart each OSD after adding more MONs?

by Tony Liu

Hi, Do I need to update ceph.conf and restart each OSD after adding more MONs? This is with 15.2.8 deployed by cephadm. When adding MON, "mon_host" should be updated accordingly. Given [1], is that update "the monitor cluster’s centralized configuration database" or "runtime overrides set by an administrator"? [1] https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#config-sourc… Thanks! Tony

3 years, 1 month

3
6
0 0

Re: [ Failed ] Upgrade path for Ceph Ansible from Octopus to Pacific

by Lokendra Rathour

Request the moderators to approve the same. It is since long and the solution to the issue is not found yet. -Lokendra On Tue, 23 Mar 2021, 10:17 Lokendra Rathour, <lokendrarathour(a)gmail.com> wrote: > Hi Team, > I am trying to upgrade my existing Ceph Cluster (using Ceph-ansible) from > current release Octopus to pacific for which I am using a rolling upgrade. > Facing various issues in getting it done, please note as below and suggest: > > issue 1: when updating the all.yml file with ceph release number to 16 and > and ceph release as Pacific: > > *TASK [ceph-validate : validate ceph_repository_community] > ************************************************************************** > > > > > > > > *task path: > /home/ansible/ceph-ansible/roles/ceph-validate/tasks/main.yml:20Tuesday 23 > March 2021 10:00:09 +0530 (0:00:00.141) 0:01:06.028 *********fatal: > [cephnode1]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'fatal: [cephnode2]: FAILED! => changed=false > msg: ceph_stable_release must be either 'nautilus' or 'octopus'fatal: > [cephnode3]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'* > > *Issue 2: by keeping the ceph_stable_release to octopus and changing the > ceph release number to 16 it gives error as :* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *fatal: [cephnode1 -> cephnode1]: FAILED! => changed=true cmd: - ceph - > --cluster - ceph - osd - require-osd-release - pacific delta: > '0:00:00.478257' end: '2021-03-22 19:54:22.892994' invocation: > module_args: _raw_params: ceph --cluster ceph osd require-osd-release > pacific _uses_shell: false argv: null chdir: null > creates: null executable: null removes: null stdin: null > stdin_add_newline: true strip_empty_ends: true warn: true msg: > non-zero return code rc: 22 start: '2021-03-22 19:54:22.414737' stderr: > |- Invalid command: pacific not in luminous|mimic|nautilus|octopus > osd require-osd-release luminous|mimic|nautilus|octopus > [--yes-i-really-mean-it] : set the minimum allowed OSD release to > participate in the cluster Error EINVAL: invalid command stderr_lines: > <omitted> stdout: '' stdout_lines: <omitted>* > > Problem Statement : > > Not able to upgrade to the upper version of Ceph-ansible from octopus to > Pacific. Please suggest/Support. > > -- > ~ Lokendra > skype: lokendrarathour > > >

3 years, 1 month

1
0
0 0

Re: [ Failed ] Upgrade path for Ceph Ansible from Octopus to Pacific

by Lokendra Rathour

HI Robert, Just checked your email on ceph users list. I will try to look deep into the question. For now i have a qery related to the uphrade itself. Is it possible for you to send any link /documents that you are following to upgrade ceph. I am trying to upgrade ceph cluster using ceph ansible on centos 7 from nautilus to octopus but have many el7 related upgrade issue. And similarly i am trying to do it in centos8 but no luck. I have tried posting my query to community (refer mail.in the trail) but it is yet to be posted after moderators approval. Any support would be appreciated thank you once again for your help. -Lokendra On Tue, 23 Mar 2021, 10:17 Lokendra Rathour, <lokendrarathour(a)gmail.com> wrote: > Hi Team, > I am trying to upgrade my existing Ceph Cluster (using Ceph-ansible) from > current release Octopus to pacific for which I am using a rolling upgrade. > Facing various issues in getting it done, please note as below and suggest: > > issue 1: when updating the all.yml file with ceph release number to 16 and > and ceph release as Pacific: > > *TASK [ceph-validate : validate ceph_repository_community] > ************************************************************************** > > > > > > > > *task path: > /home/ansible/ceph-ansible/roles/ceph-validate/tasks/main.yml:20Tuesday 23 > March 2021 10:00:09 +0530 (0:00:00.141) 0:01:06.028 *********fatal: > [cephnode1]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'fatal: [cephnode2]: FAILED! => changed=false > msg: ceph_stable_release must be either 'nautilus' or 'octopus'fatal: > [cephnode3]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'* > > *Issue 2: by keeping the ceph_stable_release to octopus and changing the > ceph release number to 16 it gives error as :* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *fatal: [cephnode1 -> cephnode1]: FAILED! => changed=true cmd: - ceph - > --cluster - ceph - osd - require-osd-release - pacific delta: > '0:00:00.478257' end: '2021-03-22 19:54:22.892994' invocation: > module_args: _raw_params: ceph --cluster ceph osd require-osd-release > pacific _uses_shell: false argv: null chdir: null > creates: null executable: null removes: null stdin: null > stdin_add_newline: true strip_empty_ends: true warn: true msg: > non-zero return code rc: 22 start: '2021-03-22 19:54:22.414737' stderr: > |- Invalid command: pacific not in luminous|mimic|nautilus|octopus > osd require-osd-release luminous|mimic|nautilus|octopus > [--yes-i-really-mean-it] : set the minimum allowed OSD release to > participate in the cluster Error EINVAL: invalid command stderr_lines: > <omitted> stdout: '' stdout_lines: <omitted>* > > Problem Statement : > > Not able to upgrade to the upper version of Ceph-ansible from octopus to > Pacific. Please suggest/Support. > > -- > ~ Lokendra > skype: lokendrarathour > > >

3 years, 1 month

1
0
0 0

Cluster suspends when Add Mon or stop and start after a while.

by by morphin

Hello! I have a cluster with Datacenter crushmap (A+B).(9+9 = 18 servers) The cluster started with v12.2.0 Luminous 4 years ago. All these years I upgraded the Cluster Luminous > Mimic > v14.2.16 Nautilus. Now I have a weird issue. When I add a mon or shutdown a while and start it up again, all the cluster suspends, ceph -s do not respond and other two monitors starting election while booting mon is syncing. (logs below) 2021-03-28 00:18:23.482 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:23.782 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:24.292 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:26.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3919 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:29.782 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7899) init, last seen epoch 7899, mid-election, bumping 2021-03-28 00:18:29.812 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:31.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3951 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:31.872 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:32.072 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:32.482 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:33.282 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:34.812 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7901) init, last seen epoch 7901, mid-election, bumping 2021-03-28 00:18:34.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:34.872 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:35.072 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:35.492 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:36.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3989 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:36.292 7fe2f2e0f700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:39.842 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7903) init, last seen epoch 7903, mid-election, bumping 2021-03-28 00:18:39.872 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:40.872 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.082 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4027 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:41.492 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.812 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:42.312 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:43.882 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:44.082 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:44.492 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:45.302 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:46.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4062 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:47.812 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7905) init, last seen epoch 7905, mid-election, bumping 2021-03-28 00:18:47.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:51.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4091 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:52.842 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7907) init, last seen epoch 7907, mid-election, bumping 2021-03-28 00:18:52.872 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:52.882 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:53.082 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:53.492 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:54.302 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:56.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4131 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:57.872 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7909) init, last seen epoch 7909, mid-election, bumping 2021-03-28 00:18:57.902 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:19:21.851 7f9127602700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:25.061 7f912ce0d700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:31.481 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:44.291 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:59.311 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:20:14.321 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:20:54.481 7f9128604700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #1158. Immutable memtables: 0. 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.490450) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 5] Flushing memtable with next log file: 1158 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054490507, "job": 5, "event": "flush_started", "num_memtables": 1, "num_entries": 391495, "num_deletes": 0, "total_data_size": 26756127, "memory_usage": 32861536, "flush_reason": "Write Buffer Full"} 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 5] Level-0 flush table #1159: started 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054615407, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 1159, "file_size": 14913671, "table_properties": {"data_size": 13733422, "index_size": 200663, "filter_size": 978757, "raw_key_size": 18143237, "raw_average_key_size": 46, "raw_value_size": 7829900, "raw_average_value_size": 20, "num_data_blocks": 3398, "num_entries": 391495, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 5] Level-0 flush table #1159: 14913671 bytes OK 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615451) [db/memtable_list.cc:380] [default] Level-0 commit table #1159 started 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615892) [db/memtable_list.cc:428] [default] Level-0 commit table #1159: memtable #1 done 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615900) EVENT_LOG_v1 {"time_micros": 1616880054615897, "job": 5, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [4, 0, 0, 0, 0, 0, 2], "immutable_memtables": 0} 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615925) [db/db_impl_compaction_flush.cc:201] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[4 0 0 0 0 0 2] max score 1.00 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: [db/db_impl_files.cc:356] [JOB 5] Try to delete WAL files size 24016022, prev total WAL file size 24798914, number of live WAL files 2. 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 6] Compacting 4@0 + 2@6 files to L6, score 1.00 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 5 Base level 0, inputs: [1159(14MB) 1155(25MB) 1149(591KB) 1157(24MB)], [1146(68MB) 1147(53MB)] 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054616197, "job": 6, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [1159, 1155, 1149, 1157], "files_L6": [1146, 1147], "score": 1, "input_data_size": 195345305} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 6] Generated table #1160: 547031 keys, 40744575 bytes 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055675107, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 1160, "file_size": 40744575, "table_properties": {"data_size": 39048586, "index_size": 327543, "filter_size": 1367621, "raw_key_size": 25391768, "raw_average_key_size": 46, "raw_value_size": 30761471, "raw_average_value_size": 56, "num_data_blocks": 6004, "num_entries": 547031, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 6] Compacted 4@0 + 2@6 files to L6 => 40744575 bytes 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: (Original Log Time 2021/03/28-00:20:55.676525) [db/compaction_job.cc:768] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 184.5 rd, 38.5 wr, level 6, files in(4, 2) out(1) MB in(64.7, 121.6) out(38.9), read-write-amplify(3.5) write-amplify(0.6) OK, records in: 3571913, records dropped: 3024882 output_compression: NoCompression 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: (Original Log Time 2021/03/28-00:20:55.676541) EVENT_LOG_v1 {"time_micros": 1616880055676534, "job": 6, "event": "compaction_finished", "compaction_time_micros": 1058962, "compaction_time_cpu_micros": 1047128, "output_level": 6, "num_output_files": 1, "total_output_size": 40744575, "num_input_records": 3571913, "num_output_records": 547031, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055677454, "job": 6, "event": "table_file_deletion", "file_number": 1159} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055678431, "job": 6, "event": "table_file_deletion", "file_number": 1157} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055679410, "job": 6, "event": "table_file_deletion", "file_number": 1155} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055679506, "job": 6, "event": "table_file_deletion", "file_number": 1149} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055681388, "job": 6, "event": "table_file_deletion", "file_number": 1147} 2021-03-28 00:20:55.681 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055684258, "job": 6, "event": "table_file_deletion", "file_number": 1146} As you can see in the leader mon switchs to election mod and gives these logs until the sync is finished. mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id The booting mon only gives this: mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id The booting mon prepares its *.stt tables and I watched the var/lib/ceph/mon/*/*/*.log file and I saw these logs: !osd_snapremoved_epoch_1_0013c460,�!osd_snapremoved_epoch_1_0013c462.� I guess after all these upgrades Ceph missed some variable or something else. The problem is huge because I can not bootstrap a mon without stopping entire cluster. It takes 30min to boot up... I tried to remove and add new mons but nothing changed. I've a backup service and it takes snapshots and backup to blockstorage and deletes the snapshot. It works for 3 years now and the snapremoved log in th mon log under "var/lib/ceph/mon/*/*/*.log" makes me wonder, what if all these snapshots makes huge bottleneck to the mon bootup? What should I do?

3 years, 1 month

1
0
0 0

Can I create 8+2 Erasure coding pool on 5 node?

by by morphin

Hello. I have 5 node Cluster in A datacenter. Also I have same 5 node in B datacenter. They're gonna be 10 node 8+2 EC cluster for backup but I need to add the 5 node later. I have to sync my S3 data with multisite on the 5 node cluster in A datacenter and move them to the B and add the other 5 node to the same cluster. The question is: Can I create 8+2 ec pool on 5 node cluster and add the 5 node later? How can I rebalance the data after that? Or is there any better solution in my case? what should I do?

3 years, 1 month

5
5
0 0

haproxy rewrite for s3 subdomain

by Marc

Has anyone adapted their haproxy config for rewriting new s3 subdomain urls(?) to old path?

3 years, 1 month

1
0
0 0

Cephfs metadata and MDS on same node

by Jesper Lykkegaard Karlsen

Dear Ceph’ers I am about to upgrade MDS nodes for Cephfs in the Ceph cluster (erasure code 8+3 ) I am administrating. Since they will get plenty of memory and CPU cores, I was wondering if it would be a good idea to move metadata OSDs (NVMe's currently on OSD nodes together with cephfs_data ODS (HDD)) to the MDS nodes? Configured as: 4 x MDS with each a metadata OSD and configured with 4 x replication so each metadata OSD would have a complete copy of metadata. I know MDS, stores al lot of metadata in RAM, but if metadata OSDs were on MDS nodes, would that not bring down latency? Anyway, I am just asking for your opinion on this? Pros and cons or even better somebody who actually have tried this? Best regards, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: jelka(a)mbg.au.dk<mailto:jelka@mbg.au.dk> Tlf: +45 50906203

3 years, 1 month

4
3
0 0

OpenSSL security update for Octopus container?

by Robert Sander

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

3 years, 1 month

1
0
0 0

Possible to update from luminous 12.2.8 to nautilus latest?

by Szabo, Istvan (Agoda)

Hi, Is it possible to do a big jump or needs to go slower to luminous latest, then mimic latest, then nautilus latest? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 years, 1 month

2
1
0 0

2024

2023

2022

2021

2020

2019

ceph-users March 2021