- ceph-users - lists.ceph.io

Re: [ Failed ] Upgrade path for Ceph Ansible from Octopus to Pacific
by Lokendra Rathour 28 Mar '21

28 Mar '21

Request the moderators to approve the same. It is since long and the solution to the issue is not found yet. -Lokendra On Tue, 23 Mar 2021, 10:17 Lokendra Rathour, <lokendrarathour(a)gmail.com> wrote: > Hi Team, > I am trying to upgrade my existing Ceph Cluster (using Ceph-ansible) from > current release Octopus to pacific for which I am using a rolling upgrade. > Facing various issues in getting it done, please note as below and suggest: > > issue 1: when updating the all.yml file with ceph release number to 16 and > and ceph release as Pacific: > > *TASK [ceph-validate : validate ceph_repository_community] > ************************************************************************** > > > > > > > > *task path: > /home/ansible/ceph-ansible/roles/ceph-validate/tasks/main.yml:20Tuesday 23 > March 2021 10:00:09 +0530 (0:00:00.141) 0:01:06.028 *********fatal: > [cephnode1]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'fatal: [cephnode2]: FAILED! => changed=false > msg: ceph_stable_release must be either 'nautilus' or 'octopus'fatal: > [cephnode3]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'* > > *Issue 2: by keeping the ceph_stable_release to octopus and changing the > ceph release number to 16 it gives error as :* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *fatal: [cephnode1 -> cephnode1]: FAILED! => changed=true cmd: - ceph - > --cluster - ceph - osd - require-osd-release - pacific delta: > '0:00:00.478257' end: '2021-03-22 19:54:22.892994' invocation: > module_args: _raw_params: ceph --cluster ceph osd require-osd-release > pacific _uses_shell: false argv: null chdir: null > creates: null executable: null removes: null stdin: null > stdin_add_newline: true strip_empty_ends: true warn: true msg: > non-zero return code rc: 22 start: '2021-03-22 19:54:22.414737' stderr: > |- Invalid command: pacific not in luminous|mimic|nautilus|octopus > osd require-osd-release luminous|mimic|nautilus|octopus > [--yes-i-really-mean-it] : set the minimum allowed OSD release to > participate in the cluster Error EINVAL: invalid command stderr_lines: > <omitted> stdout: '' stdout_lines: <omitted>* > > Problem Statement : > > Not able to upgrade to the upper version of Ceph-ansible from octopus to > Pacific. Please suggest/Support. > > -- > ~ Lokendra > skype: lokendrarathour > > >

1 0

Re: [ Failed ] Upgrade path for Ceph Ansible from Octopus to Pacific
by Lokendra Rathour 28 Mar '21

28 Mar '21

HI Robert, Just checked your email on ceph users list. I will try to look deep into the question. For now i have a qery related to the uphrade itself. Is it possible for you to send any link /documents that you are following to upgrade ceph. I am trying to upgrade ceph cluster using ceph ansible on centos 7 from nautilus to octopus but have many el7 related upgrade issue. And similarly i am trying to do it in centos8 but no luck. I have tried posting my query to community (refer mail.in the trail) but it is yet to be posted after moderators approval. Any support would be appreciated thank you once again for your help. -Lokendra On Tue, 23 Mar 2021, 10:17 Lokendra Rathour, <lokendrarathour(a)gmail.com> wrote: > Hi Team, > I am trying to upgrade my existing Ceph Cluster (using Ceph-ansible) from > current release Octopus to pacific for which I am using a rolling upgrade. > Facing various issues in getting it done, please note as below and suggest: > > issue 1: when updating the all.yml file with ceph release number to 16 and > and ceph release as Pacific: > > *TASK [ceph-validate : validate ceph_repository_community] > ************************************************************************** > > > > > > > > *task path: > /home/ansible/ceph-ansible/roles/ceph-validate/tasks/main.yml:20Tuesday 23 > March 2021 10:00:09 +0530 (0:00:00.141) 0:01:06.028 *********fatal: > [cephnode1]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'fatal: [cephnode2]: FAILED! => changed=false > msg: ceph_stable_release must be either 'nautilus' or 'octopus'fatal: > [cephnode3]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'* > > *Issue 2: by keeping the ceph_stable_release to octopus and changing the > ceph release number to 16 it gives error as :* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *fatal: [cephnode1 -> cephnode1]: FAILED! => changed=true cmd: - ceph - > --cluster - ceph - osd - require-osd-release - pacific delta: > '0:00:00.478257' end: '2021-03-22 19:54:22.892994' invocation: > module_args: _raw_params: ceph --cluster ceph osd require-osd-release > pacific _uses_shell: false argv: null chdir: null > creates: null executable: null removes: null stdin: null > stdin_add_newline: true strip_empty_ends: true warn: true msg: > non-zero return code rc: 22 start: '2021-03-22 19:54:22.414737' stderr: > |- Invalid command: pacific not in luminous|mimic|nautilus|octopus > osd require-osd-release luminous|mimic|nautilus|octopus > [--yes-i-really-mean-it] : set the minimum allowed OSD release to > participate in the cluster Error EINVAL: invalid command stderr_lines: > <omitted> stdout: '' stdout_lines: <omitted>* > > Problem Statement : > > Not able to upgrade to the upper version of Ceph-ansible from octopus to > Pacific. Please suggest/Support. > > -- > ~ Lokendra > skype: lokendrarathour > > >

1 0

Cluster suspends when Add Mon or stop and start after a while.
by by morphin 28 Mar '21

28 Mar '21

Hello! I have a cluster with Datacenter crushmap (A+B).(9+9 = 18 servers) The cluster started with v12.2.0 Luminous 4 years ago. All these years I upgraded the Cluster Luminous > Mimic > v14.2.16 Nautilus. Now I have a weird issue. When I add a mon or shutdown a while and start it up again, all the cluster suspends, ceph -s do not respond and other two monitors starting election while booting mon is syncing. (logs below) 2021-03-28 00:18:23.482 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:23.782 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:24.292 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:26.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3919 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:29.782 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7899) init, last seen epoch 7899, mid-election, bumping 2021-03-28 00:18:29.812 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:31.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3951 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:31.872 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:32.072 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:32.482 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:33.282 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:34.812 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7901) init, last seen epoch 7901, mid-election, bumping 2021-03-28 00:18:34.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:34.872 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:35.072 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:35.492 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:36.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3989 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:36.292 7fe2f2e0f700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:39.842 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7903) init, last seen epoch 7903, mid-election, bumping 2021-03-28 00:18:39.872 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:40.872 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.082 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4027 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:41.492 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.812 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:42.312 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:43.882 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:44.082 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:44.492 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:45.302 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:46.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4062 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:47.812 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7905) init, last seen epoch 7905, mid-election, bumping 2021-03-28 00:18:47.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:51.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4091 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:52.842 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7907) init, last seen epoch 7907, mid-election, bumping 2021-03-28 00:18:52.872 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:52.882 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:53.082 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:53.492 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:54.302 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:56.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4131 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:57.872 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7909) init, last seen epoch 7909, mid-election, bumping 2021-03-28 00:18:57.902 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:19:21.851 7f9127602700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:25.061 7f912ce0d700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:31.481 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:44.291 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:59.311 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:20:14.321 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:20:54.481 7f9128604700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #1158. Immutable memtables: 0. 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.490450) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 5] Flushing memtable with next log file: 1158 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054490507, "job": 5, "event": "flush_started", "num_memtables": 1, "num_entries": 391495, "num_deletes": 0, "total_data_size": 26756127, "memory_usage": 32861536, "flush_reason": "Write Buffer Full"} 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 5] Level-0 flush table #1159: started 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054615407, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 1159, "file_size": 14913671, "table_properties": {"data_size": 13733422, "index_size": 200663, "filter_size": 978757, "raw_key_size": 18143237, "raw_average_key_size": 46, "raw_value_size": 7829900, "raw_average_value_size": 20, "num_data_blocks": 3398, "num_entries": 391495, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 5] Level-0 flush table #1159: 14913671 bytes OK 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615451) [db/memtable_list.cc:380] [default] Level-0 commit table #1159 started 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615892) [db/memtable_list.cc:428] [default] Level-0 commit table #1159: memtable #1 done 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615900) EVENT_LOG_v1 {"time_micros": 1616880054615897, "job": 5, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [4, 0, 0, 0, 0, 0, 2], "immutable_memtables": 0} 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615925) [db/db_impl_compaction_flush.cc:201] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[4 0 0 0 0 0 2] max score 1.00 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: [db/db_impl_files.cc:356] [JOB 5] Try to delete WAL files size 24016022, prev total WAL file size 24798914, number of live WAL files 2. 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 6] Compacting 4@0 + 2@6 files to L6, score 1.00 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 5 Base level 0, inputs: [1159(14MB) 1155(25MB) 1149(591KB) 1157(24MB)], [1146(68MB) 1147(53MB)] 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054616197, "job": 6, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [1159, 1155, 1149, 1157], "files_L6": [1146, 1147], "score": 1, "input_data_size": 195345305} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 6] Generated table #1160: 547031 keys, 40744575 bytes 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055675107, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 1160, "file_size": 40744575, "table_properties": {"data_size": 39048586, "index_size": 327543, "filter_size": 1367621, "raw_key_size": 25391768, "raw_average_key_size": 46, "raw_value_size": 30761471, "raw_average_value_size": 56, "num_data_blocks": 6004, "num_entries": 547031, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 6] Compacted 4@0 + 2@6 files to L6 => 40744575 bytes 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: (Original Log Time 2021/03/28-00:20:55.676525) [db/compaction_job.cc:768] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 184.5 rd, 38.5 wr, level 6, files in(4, 2) out(1) MB in(64.7, 121.6) out(38.9), read-write-amplify(3.5) write-amplify(0.6) OK, records in: 3571913, records dropped: 3024882 output_compression: NoCompression 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: (Original Log Time 2021/03/28-00:20:55.676541) EVENT_LOG_v1 {"time_micros": 1616880055676534, "job": 6, "event": "compaction_finished", "compaction_time_micros": 1058962, "compaction_time_cpu_micros": 1047128, "output_level": 6, "num_output_files": 1, "total_output_size": 40744575, "num_input_records": 3571913, "num_output_records": 547031, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055677454, "job": 6, "event": "table_file_deletion", "file_number": 1159} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055678431, "job": 6, "event": "table_file_deletion", "file_number": 1157} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055679410, "job": 6, "event": "table_file_deletion", "file_number": 1155} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055679506, "job": 6, "event": "table_file_deletion", "file_number": 1149} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055681388, "job": 6, "event": "table_file_deletion", "file_number": 1147} 2021-03-28 00:20:55.681 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055684258, "job": 6, "event": "table_file_deletion", "file_number": 1146} As you can see in the leader mon switchs to election mod and gives these logs until the sync is finished. mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id The booting mon only gives this: mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id The booting mon prepares its *.stt tables and I watched the var/lib/ceph/mon/*/*/*.log file and I saw these logs: !osd_snapremoved_epoch_1_0013c460,�!osd_snapremoved_epoch_1_0013c462.� I guess after all these upgrades Ceph missed some variable or something else. The problem is huge because I can not bootstrap a mon without stopping entire cluster. It takes 30min to boot up... I tried to remove and add new mons but nothing changed. I've a backup service and it takes snapshots and backup to blockstorage and deletes the snapshot. It works for 3 years now and the snapremoved log in th mon log under "var/lib/ceph/mon/*/*/*.log" makes me wonder, what if all these snapshots makes huge bottleneck to the mon bootup? What should I do?

1 0

Can I create 8+2 Erasure coding pool on 5 node?
by by morphin 27 Mar '21

27 Mar '21

Hello. I have 5 node Cluster in A datacenter. Also I have same 5 node in B datacenter. They're gonna be 10 node 8+2 EC cluster for backup but I need to add the 5 node later. I have to sync my S3 data with multisite on the 5 node cluster in A datacenter and move them to the B and add the other 5 node to the same cluster. The question is: Can I create 8+2 ec pool on 5 node cluster and add the 5 node later? How can I rebalance the data after that? Or is there any better solution in my case? what should I do?

5 5

haproxy rewrite for s3 subdomain
by Marc 27 Mar '21

27 Mar '21

Has anyone adapted their haproxy config for rewriting new s3 subdomain urls(?) to old path?

1 0

Cephfs metadata and MDS on same node
by Jesper Lykkegaard Karlsen 26 Mar '21

26 Mar '21

Dear Ceph’ers I am about to upgrade MDS nodes for Cephfs in the Ceph cluster (erasure code 8+3 ) I am administrating. Since they will get plenty of memory and CPU cores, I was wondering if it would be a good idea to move metadata OSDs (NVMe's currently on OSD nodes together with cephfs_data ODS (HDD)) to the MDS nodes? Configured as: 4 x MDS with each a metadata OSD and configured with 4 x replication so each metadata OSD would have a complete copy of metadata. I know MDS, stores al lot of metadata in RAM, but if metadata OSDs were on MDS nodes, would that not bring down latency? Anyway, I am just asking for your opinion on this? Pros and cons or even better somebody who actually have tried this? Best regards, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: jelka(a)mbg.au.dk<mailto:jelka@mbg.au.dk> Tlf: +45 50906203

4 3

Possible to update from luminous 12.2.8 to nautilus latest?
by Szabo, Istvan (Agoda) 26 Mar '21

26 Mar '21

Hi, Is it possible to do a big jump or needs to go slower to luminous latest, then mimic latest, then nautilus latest? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Re: How to clear Health Warning status?
by jinguk.kwon＠ungleich.ch 26 Mar '21

26 Mar '21

Hello there, Thank you for advanced. My ceph is ceph version 14.2.9 I have a repair issue too. ceph health detail HEALTH_WARN Too many repaired reads on 2 OSDs OSD_TOO_MANY_REPAIRS Too many repaired reads on 2 OSDs osd.29 had 38 reads repaired osd.16 had 17 reads repaired ~# ceph tell osd.16 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 7.1486738159999996, "bytes_per_sec": 150201541.10217974, "iops": 35.81083800844663 } ~# ceph tell osd.29 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 6.9244327500000002, "bytes_per_sec": 155065672.9246161, "iops": 36.970537406114602 } But it looks like those osds are ok. how can i clear this warning ? Best regards JG

1 0

How ceph sees when the pool is getting full?
by Szabo, Istvan (Agoda) 25 Mar '21

25 Mar '21

Hi, I want to set alert on the user's pool before it got's full but in nautilus I still haven't found the way which is the value of their data usage based on ceph detail df? POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR k8s-dbss-w-mdb 16 1.8 TiB 488.61k 2.6 TiB 2.87 45 TiB N/A 1.8 TiB 488.61k 0 B 0 B This output is still confusing :/ ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

ceph octopus mysterious OSD crash
by Philip Brown 25 Mar '21

25 Mar '21

I've been banging on my ceph octopus test cluster for a few days now. 8 nodes. each node has 2 SSDs and 8 HDDs. They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db partition. service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0 things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were "down". I went to check the logs, with journalctl -u ceph-xxxx(a)osd.xxx all it showed were a bunch of generic debug info, and the fact that it stopped. and various automatic attempts to restart. but no indication of what was wrong, and why the restarts KEEP failing. sample output: systemd[1]: Stopped Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00. systemd[1]: Starting Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00... bash[9340]: ceph-e51eb2fa-7f82-11eb-94d5-78e3b5148f00-osd.33-activate bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. podman[9369]: 2021-03-07 16:00:15.543010794 -0800 PST m=+0.318475882 container create podman[9369]: 2021-03-07 16:00:15.73461926 -0800 PST m=+0.510084288 container init ..... bash[1611473]: --> ceph-volume lvm activate successful for osd ID: 33 podman[1611501]: 2021-03-18 10:23:02.564242824 -0700 PDT m=+1.379793448 container died bash[1611473]: ceph-xx-xx-xx-xx-osd.33 bash[1611473]: WARNING: The same type, major and minor should not be used for multiple devices. (repeat, repeat...) podman[1611615]: 2021-03-18 10:23:03.530992487 -0700 PDT m=+0.333130660 container create .... systemd[1]: Started Ceph osd.33 for xx-xx-xx-xx systemd[1]: ceph-xx-xx-xx-xx(a)osd.33.service: main process exited, code=exited, status=1/FAILURE bash[1611797]: ceph-xx-xx-xx-xx-osd.33-deactivate and eventually it just gives up. smartctl -a doesnt show any errors on the HDD dmesg doesnt show anything. So... what do I do? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

6 21