December 2020 - ceph-users

MON: global_init: error reading config file.

by Oscar Segarra

Hi, In my environment I have a single node and I'm trying to run a ceph monitor as a docker container using a kv store. Version: Octopus (stable-5.0) 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: STAYALIVE: container will not die if a command fails., 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'auth' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'global' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'mon' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'mds' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'osd' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'client' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: Adding Mon Host - vdicnode01., 0.0.0.0, vdicnode01, true, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: Configuration found for cluster ceph. Writing to disk., true, 2020-12-12T00:24:29+01:00 vdicnode01.local /tmp/gztmpGeLTQkuFB/confd[139]: INFO Backend set to etcd, 2020-12-12T00:24:29+01:00 vdicnode01.local /tmp/gztmpGeLTQkuFB/confd[139]: INFO Starting confd, 2020-12-12T00:24:29+01:00 vdicnode01.local /tmp/gztmpGeLTQkuFB/confd[139]: INFO Backend source(s) set to http://127.0.0.1:2379, 2020-12-12 00:24:29 /opt/ceph-container/bin/entrypoint.sh: Adding bootstrap keyrings., 2020-12-12 00:24:30 /opt/ceph-container/bin/entrypoint.sh: Adding mon/admin Keyrings., 2020-12-12 00:24:30 /opt/ceph-container/bin/entrypoint.sh: Trying to get the most recent monmap..., Error initializing cluster client: InvalidArgumentError('RADOS invalid argument (error calling conf_read_file)',), 2020-12-12 00:24:30 /opt/ceph-container/bin/entrypoint.sh: Peers not found, using initial monmap., 2020-12-12 00:24:30 /opt/ceph-container/bin/entrypoint.sh: Removing lock for vdicnode01., PrevNode.Value: vdicnode01, importing contents of /var/lib/ceph/bootstrap-osd/ceph.keyring into /etc/ceph/ceph.mon.keyring, importing contents of /var/lib/ceph/bootstrap-mds/ceph.keyring into /etc/ceph/ceph.mon.keyring, importing contents of /var/lib/ceph/bootstrap-rgw/ceph.keyring into /etc/ceph/ceph.mon.keyring, importing contents of /var/lib/ceph/bootstrap-rbd-mirror/ceph.keyring into /etc/ceph/ceph.mon.keyring, importing contents of /etc/ceph/ceph.client.admin.keyring into /etc/ceph/ceph.mon.keyring, global_init: error reading config file., And container dies (even the stayalive flag). docker service create -d --network=host \ --mount type=bind,source=/var/lib/ceph,destination=/var/lib/ceph \ --mount type=bind,source=/var/log/ceph,destination=/var/log/ceph \ --mount type=bind,source=/etc/localtime,destination=/etc/localtime,readonly \ -e KV_TYPE=etcd \ -e KV_IP=127.0.0.1 \ -e KV_PORT=2379 \ -e MON_IP=0.0.0.0 \ -e CEPH_PUBLIC_NETWORK=192.168.100.0/24 \ -e CEPH_CLUSTER_NETWORK=192.168.100.0/24 \ -e DEBUG=stayalive \ --name=vdicube_ceph_mon \ --mode=global \ --constraint node.labels.mon==1 \ ceph/daemon:v5.0.6-stable-5.0-octopus-centos-8 mon Anyone has experimented the same issue? Thanks a lot. Ó

3 years, 4 months

1
0
0 0

Incomplete PG due to primary OSD crashing during EC backfill - get_hash_info: Mismatch of total_chunk_size 0

by Byrne, Thomas (STFC,RAL,SC)

Hi all, Got an odd issue that I'm not sure how to solve on our Nautilus 14.2.9 EC cluster. The primary OSD of an EC 8+3 PG died this morning with a very sad disk (thousands of pending sectors). After the down out interval a new 'up' primary was assigned and the backfill started. Twenty minutes later the acting primary (not the new 'up' primary) started crashing with a "get_hash_info: Mismatch of total_chunk_size 0" error (see log below) This crash always happens at the same object, with different acting primaries, and with a different new 'up' primary. I can't see anything in the logs that points to a particular OSD being the issue, so I suspect there is a corrupted object in the PG that is causing issues, but I'm not sure how to dig into this further. The PG is currently active (but degraded), but only whilst nobackfill or noout are set (+turning the new OSD off), and if the flags are unset the backfill will eventually crash enough OSDs to render the PG incomplete, which is not ideal. I would appreciate being able to resolve this so I can go back to letting Ceph deal with down OSDs itself :) Does anyone have some pointers on how to dig into or resolve this? Happy to create a tracker ticket and post more logs if this looks like a bug. Thanks, Tom OSD log with debug_osd=20 (preamble cut from subsequent lines in an attempt to improve readability...): 2020-12-10 15:14:16.130 7fc0a1575700 10 osd.1708 pg_epoch: 1162259 pg[11.214s1( v 1162255'714638 (1162110'711564,1162255'714638] local-lis/les=1162253/1162254 n=133385 ec=1069520/992 lis/c 1162253/1125301 les/c/f 1162254/1125302/257760 1162252/1162253/1162253) [2449,1708,2099,1346,4309,777,5098,4501,4134,217,4643]/[2147483647,1708,2099,1346,4309,777,5098,4501,4134,217,4643]p1708(1) backfill=[2449(0)] r=1 lpr=1162253 pi=[1125301,1162253)/3 rops=1 crt=1162255'714638 lcod 1162254'714637 mlcod 1162254'714637 active+undersized+degraded+remapped+backfilling mbc={}] run_recovery_op: starting RecoveryOp(hoid=11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head v=1162125'713150 missing_on=2449(0) missing_on_shards=0 recovery_info=ObjectRecoveryInfo(11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head@1162125'713150, size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}) recovery_progress=ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false) obc refcount=3 state=IDLE waiting_on_pushes= extent_requested=0,0) continue_recovery_op: continuing RecoveryOp(hoid=11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head v=1162125'713150 missing_on=2449(0) missing_on_shards=0 recovery_info=ObjectRecoveryInfo(11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head@1162125'713150, size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}) recovery_progress=ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false) obc refcount=4 state=IDLE waiting_on_pushes= extent_requested=0,0) get_hash_info: Getting attr on 11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head get_hash_info: not in cache 11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head get_hash_info: found on disk, size 524288 get_hash_info: Mismatch of total_chunk_size 0 2020-12-10 15:14:16.136 7fc0a1575700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/osd/ECBackend.cc: In function 'void ECBackend::continue_recovery_op(ECBackend::RecoveryOp&, RecoveryMessages*)' thread 7fc0a1575700 time 2020-12-10 15:14:16.132060 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/osd/ECBackend.cc: 585: FAILED ceph_assert(op.hinfo) ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55e1569acf7d] 2: (()+0x4cb145) [0x55e1569ad145] 3: (ECBackend::continue_recovery_op(ECBackend::RecoveryOp&, RecoveryMessages*)+0x1764) [0x55e156da5834] 4: (ECBackend::run_recovery_op(PGBackend::RecoveryHandle*, int)+0x65b) [0x55e156da6c6b] 5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x1491) [0x55e156c26681] 6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x114c) [0x55e156c29f7c] 7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x2ff) [0x55e156a8b32f] 8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0x55e156d1aa19] 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55e156aa6b4f] 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x55e15704b216] 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55e15704dd30] 12: (()+0x7ea5) [0x7fc0c2d5dea5] 13: (clone()+0x6d) [0x7fc0c1c208dd] 2020-12-10 15:14:16.143 7fc0a1575700 -1 *** Caught signal (Aborted) ** This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

3 years, 4 months

1
2
0 0

Scrubbing - osd down

by Miroslav Boháč

Hi, I have a problem with crashing OSD daemons in our Ceph 15.2.6 cluster . The problem was temporarily resolved by disabling scrub and deep-scrub. All PGs are active+clean. After a few days I tried to enable scrubbing again, but the problem persists. OSD with high latencies, PG laggy, osd not responding, OSD was marked down and the cluster is not usable. Problem appeared after failover when 10 OSD was marked as out. I will appreciate any help or advice, how to resolve this problem. Regards, Miroslav 2020-12-10T21:39:57.721883+0100 osd.1 (osd.1) 5 : cluster [DBG] 18.11 deep-scrub starts 2020-12-10T21:39:57.880861+0100 osd.1 (osd.1) 6 : cluster [DBG] 18.11 deep-scrub ok 2020-12-10T21:39:58.713422+0100 osd.1 (osd.1) 7 : cluster [DBG] 43.5 scrub starts 2020-12-10T21:39:58.719372+0100 osd.1 (osd.1) 8 : cluster [DBG] 43.5 scrub ok 2020-12-10T21:39:59.296962+0100 mgr.pve2-prg2a (mgr.91575377) 118746 : cluster [DBG] pgmap v119000: 1737 pgs: 3 active+clean+laggy, 3 active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 117 KiB/s rd, 2.5 MiB/s wr, 269 op/s 2020-12-10T21:40:00.088421+0100 osd.29 (osd.29) 74 : cluster [DBG] 1.13b deep-scrub starts 2020-12-10T21:40:01.300373+0100 mgr.pve2-prg2a (mgr.91575377) 118747 : cluster [DBG] pgmap v119001: 1737 pgs: 3 active+clean+laggy, 3 active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 202 op/s 2020-12-10T21:40:02.681058+0100 osd.34 (osd.34) 13 : cluster [DBG] 1.a2 deep-scrub ok 2020-12-10T21:40:03.304009+0100 mgr.pve2-prg2a (mgr.91575377) 118749 : cluster [DBG] pgmap v119002: 1737 pgs: 3 active+clean+laggy, 3 active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 198 op/s 2020-12-10T21:40:05.316233+0100 mgr.pve2-prg2a (mgr.91575377) 118750 : cluster [DBG] pgmap v119003: 1737 pgs: 6 active+clean+laggy, 3 active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 249 op/s 2020-12-10T21:40:07.319643+0100 mgr.pve2-prg2a (mgr.91575377) 118751 : cluster [DBG] pgmap v119004: 1737 pgs: 6 active+clean+laggy, 3 active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 212 op/s 2020-12-10T21:40:15.523134+0100 mon.pve1-prg2a (mon.0) 125943 : cluster [DBG] osd.4 reported failed by osd.24 2020-12-10T21:40:15.523325+0100 mon.pve1-prg2a (mon.0) 125944 : cluster [DBG] osd.39 reported failed by osd.24 2020-12-10T21:40:16.112299+0100 mon.pve1-prg2a (mon.0) 125946 : cluster [WRN] Health check failed: 0 slow ops, oldest one blocked for 32 sec, osd.8 has slow ops (SLOW_OPS) 2020-12-10T21:40:16.202867+0100 mon.pve1-prg2a (mon.0) 125947 : cluster [DBG] osd.4 reported failed by osd.34 2020-12-10T21:40:16.202986+0100 mon.pve1-prg2a (mon.0) 125948 : cluster [INF] osd.4 failed (root=default,host=pve1-prg2a) (2 reporters from different host after 24.000267 >= grace 22.361677) 2020-12-10T21:40:16.373925+0100 mon.pve1-prg2a (mon.0) 125949 : cluster [DBG] osd.39 reported failed by osd.6 2020-12-10T21:40:16.865608+0100 mon.pve1-prg2a (mon.0) 125951 : cluster [DBG] osd.39 reported failed by osd.8 2020-12-10T21:40:17.125917+0100 mon.pve1-prg2a (mon.0) 125952 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2020-12-10T21:40:17.139006+0100 mon.pve1-prg2a (mon.0) 125953 : cluster [DBG] osdmap e12819: 40 total, 39 up, 30 in 2020-12-10T21:40:17.140248+0100 mon.pve1-prg2a (mon.0) 125954 : cluster [DBG] osd.39 reported failed by osd.21 2020-12-10T21:40:17.344244+0100 mgr.pve2-prg2a (mgr.91575377) 118757 : cluster [DBG] pgmap v119012: 1737 pgs: 9 peering, 61 stale+active+clean, 1 active+clean+scrubbing+deep, 7 active+clean+laggy, 1659 active+clean; 3.3 TiB data, 7.8 TiB used, 14 TiB / 22 TiB avail; 44 KiB/s rd, 2.5 MiB/s wr, 107 op/s 2020-12-10T21:40:17.378069+0100 mon.pve1-prg2a (mon.0) 125955 : cluster [DBG] osd.39 reported failed by osd.26 2020-12-10T21:40:17.424429+0100 mon.pve1-prg2a (mon.0) 125956 : cluster [DBG] osd.39 reported failed by osd.18 2020-12-10T21:40:17.829447+0100 mon.pve1-prg2a (mon.0) 125957 : cluster [DBG] osd.39 reported failed by osd.36 2020-12-10T21:40:17.847373+0100 mon.pve1-prg2a (mon.0) 125958 : cluster [DBG] osd.39 reported failed by osd.1 2020-12-10T21:40:17.858371+0100 mon.pve1-prg2a (mon.0) 125959 : cluster [DBG] osd.39 reported failed by osd.17 2020-12-10T21:40:17.915755+0100 mon.pve1-prg2a (mon.0) 125960 : cluster [DBG] osd.39 reported failed by osd.28 2020-12-10T21:40:24.151192+0100 mon.pve1-prg2a (mon.0) 125986 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2020-12-10T21:40:24.608038+0100 mon.pve1-prg2a (mon.0) 125987 : cluster [WRN] Health check update: 0 slow ops, oldest one blocked for 37 sec, osd.8 has slow ops (SLOW_OPS) 2020-12-10T21:40:25.375322+0100 mgr.pve2-prg2a (mgr.91575377) 118761 : cluster [DBG] pgmap v119017: 1737 pgs: 1 active+clean+scrubbing+deep, 6 active+undersized, 40 active+undersized+wait, 13 active+undersized+degraded, 101 active+undersized+degraded+wait, 3 stale+active+clean, 2 active+clean+laggy, 1571 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 4.0 MiB/s wr, 179 op/s; 74043/2676420 objects degraded (2.766%); 622 KiB/s, 0 objects/s recovering 2020-12-10T21:40:27.379311+0100 mgr.pve2-prg2a (mgr.91575377) 118762 : cluster [DBG] pgmap v119018: 1737 pgs: 1 active+clean+scrubbing+deep, 6 active+undersized, 40 active+undersized+wait, 13 active+undersized+degraded, 101 active+undersized+degraded+wait, 3 stale+active+clean, 2 active+clean+laggy, 1571 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 3.6 MiB/s wr, 154 op/s; 74043/2676420 objects degraded (2.766%); 510 KiB/s, 0 objects/s recovering 2020-12-10T21:40:29.391905+0100 mgr.pve2-prg2a (mgr.91575377) 118763 : cluster [DBG] pgmap v119019: 1737 pgs: 1 active+clean+scrubbing+deep, 11 active+undersized, 35 active+undersized+wait, 21 active+undersized+degraded, 93 active+undersized+degraded+wait, 3 stale+active+clean, 2 active+clean+laggy, 1571 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 3.9 MiB/s wr, 170 op/s; 74043/2676420 objects degraded (2.766%); 456 KiB/s, 0 objects/s recovering 2020-12-10T21:40:29.609801+0100 mon.pve1-prg2a (mon.0) 125991 : cluster [WRN] Health check update: Degraded data redundancy: 74043/2676420 objects degraded (2.766%), 114 pgs degraded (PG_DEGRADED) 2020-12-10T21:40:29.609845+0100 mon.pve1-prg2a (mon.0) 125992 : cluster [WRN] Health check update: 0 slow ops, oldest one blocked for 42 sec, osd.8 has slow ops (SLOW_OPS) 2020-12-10T21:40:31.395715+0100 mgr.pve2-prg2a (mgr.91575377) 118764 : cluster [DBG] pgmap v119020: 1737 pgs: 1 active+clean+scrubbing+deep, 11 active+undersized, 35 active+undersized+wait, 21 active+undersized+degraded, 93 active+undersized+degraded+wait, 3 stale+active+clean, 2 active+clean+laggy, 1571 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 154 KiB/s rd, 2.1 MiB/s wr, 151 op/s; 74043/2676420 objects degraded (2.766%); 425 KiB/s, 0 objects/s recovering 2020-12-10T21:40:32.797773+0100 osd.8 (osd.8) 19 : cluster [WRN] slow request osd_op(client.91721949.0:169935 1.3b 1:dc12cc94:::rbd_header.0373d6a5f7de48:head [watch ping cookie 139741691437056] snapc 0=[] ondisk+write+known_if_redirected e12816) initiated 2020-12-10T21:39:59.813054+0100 currently delayed 2020-12-10T21:40:33.399277+0100 mgr.pve2-prg2a (mgr.91575377) 118765 : cluster [DBG] pgmap v119021: 1737 pgs: 1 active+clean+scrubbing+deep, 13 active+undersized, 33 active+undersized+wait, 28 active+undersized+degraded, 86 active+undersized+degraded+wait, 3 stale+active+clean, 2 active+clean+laggy, 1571 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 156 KiB/s rd, 1.6 MiB/s wr, 144 op/s; 74043/2676420 objects degraded (2.766%) 2020-12-10T21:40:33.841597+0100 osd.8 (osd.8) 20 : cluster [WRN] slow request osd_op(client.91721949.0:169935 1.3b 1:dc12cc94:::rbd_header.0373d6a5f7de48:head [watch ping cookie 139741691437056] snapc 0=[] ondisk+write+known_if_redirected e12816) initiated 2020-12-10T21:39:59.813054+0100 currently delayed 2020-12-10T21:40:34.611650+0100 mon.pve1-prg2a (mon.0) 125996 : cluster [WRN] Health check update: 0 slow ops, oldest one blocked for 47 sec, daemons [osd.22,osd.8] have slow ops. (SLOW_OPS) 2020-12-10T21:40:34.612166+0100 mon.pve1-prg2a (mon.0) 125997 : cluster [INF] osd.39 failed (root=default,host=pve5-prg2a) (5 reporters from different host after 39.438374 >= grace 37.721344) 2020-12-10T21:40:34.615286+0100 mon.pve1-prg2a (mon.0) 125998 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2020-12-10T21:40:34.621801+0100 mon.pve1-prg2a (mon.0) 125999 : cluster [DBG] osdmap e12821: 40 total, 38 up, 30 in 2020-12-10T21:40:34.880626+0100 osd.8 (osd.8) 21 : cluster [WRN] slow request osd_op(client.91721949.0:169935 1.3b 1:dc12cc94:::rbd_header.0373d6a5f7de48:head [watch ping cookie 139741691437056] snapc 0=[] ondisk+write+known_if_redirected e12816) initiated 2020-12-10T21:39:59.813054+0100 currently delayed 8/0 sis=12819) [28,35] r=-1 lpr=12819 pi=[12507,12819)/1 crt=12818'17226265 lcod 12818'17226264 mlcod 0'0 unknown NOTIFY mbc={}] exit Start 0.000020 0 0.000000 -823> 2020-12-10T21:40:44.079+0100 7f5310751700 5 osd.4 pg_epoch: 12822 pg[1.cc( v 12818'17226265 (12812'17224439,12818'17226265] local-lis/les=12507/12508 n=1566 ec=171/171 lis/c=12507/12507 les/c/f=12508/12508/0 sis=12819) [28,35] r=-1 lpr=12819 pi=[12507,12819)/1 crt=12818'17226265 lcod 12818'17226264 mlcod 0'0 unknown NOTIFY mbc={}] enter Started/Stray -822> 2020-12-10T21:40:44.191+0100 7f5324f7a700 5 prioritycache tune_memory target: 4294967296 mapped: 3494428672 unmapped: 383787008 heap: 3878215680 old mem: 2845415832 new mem: 2845415832 -821> 2020-12-10T21:40:44.311+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f530ef4e700' had timed out after 15 -820> 2020-12-10T21:40:44.311+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f5312f56700' had timed out after 15 -819> 2020-12-10T21:40:44.311+0100 7f532ca01700 1 osd.4 12822 is_healthy false -- internal heartbeat failed -818> 2020-12-10T21:40:44.311+0100 7f532ca01700 1 osd.4 12822 not healthy; waiting to boot -817> 2020-12-10T21:40:44.311+0100 7f532ca01700 1 osd.4 12822 tick checking mon for new map -816> 2020-12-10T21:40:44.311+0100 7f530e74d700 5 osd.4 12822 heartbeat osd_stat(store_statfs(0x7c5f5b0000/0x2e36c0000/0xba40000000, data 0x3af1c2f453/0x3afd380000, compress 0x0/0x0/0x0, omap 0x30256c, meta 0x2e33bda94), peers [3,5,6,7,8,9,13,20,28,36] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,22]) -815> 2020-12-10T21:40:44.403+0100 7f5323337700 10 monclient: tick -814> 2020-12-10T21:40:44.403+0100 7f5323337700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2020-12-10T21:40:14.409098+0100) -813> 2020-12-10T21:40:44.403+0100 7f5323337700 10 log_client log_queue is 2 last_log 7 sent 5 num 2 unsent 2 sending 2 -812> 2020-12-10T21:40:44.403+0100 7f5323337700 10 log_client will send 2020-12-10T21:40:44.063147+0100 osd.4 (osd.4) 6 : cluster [WRN] Monitor daemon marked osd.4 down, but it is still running -811> 2020-12-10T21:40:44.403+0100 7f5323337700 10 log_client will send 2020-12-10T21:40:44.063157+0100 osd.4 (osd.4) 7 : cluster [DBG] map e12822 wrongly marked me down at e12819 -810> 2020-12-10T21:40:44.403+0100 7f5323337700 10 monclient: _send_mon_message to mon.pve1-prg2a at v2:10.104.200.11:3300/0 -809> 2020-12-10T21:40:45.191+0100 7f5324f7a700 5 prioritycache tune_memory target: 4294967296 mapped: 3494428672 unmapped: 383787008 heap: 3878215680 old mem: 2845415832 new mem: 2845415832 -808> 2020-12-10T21:40:45.351+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f530ef4e700' had timed out after 15 -807> 2020-12-10T21:40:45.351+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f5312f56700' had timed out after 15 -3> 2020-12-10T21:42:10.466+0100 7f5324f7a700 5 prioritycache tune_memory target: 4294967296 mapped: 3510689792 unmapped: 367525888 heap: 3878215680 old mem: 2845415832 new mem: 2845415832 -2> 2020-12-10T21:42:10.694+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f530ef4e700' had timed out after 15 -1> 2020-12-10T21:42:10.694+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f530ef4e700' had suicide timed out after 150

3 years, 4 months

2
3
0 0

Ceph benchmark tool (cbt)

by Seena Fallah

Hi all, I want to benchmark my production cluster with cbt. I read a bit of the code and I see something strange in it, for example, it's going to create ceph-osd by it selves ( https://github.com/ceph/cbt/blob/master/cluster/ceph.py#L373) and also shutdown the whole cluster!! ( https://github.com/ceph/cbt/blob/master/cluster/ceph.py#L212) Is there any configuration to not do harmful things with the cluster and for example just test the read_ahead_kb or simply stop some OSDs and other things that can be reverted and not get the cluster fully down?! Thanks.

3 years, 4 months

3
2
0 0

All ceph commands hangs - bad magic number in monitor log

by Evrard Van Espen - Weather-Measures

Hello all, We have a production cluster running since weeks deployed on centos8 with |cephadm| (3 nodes with 6x12TB HDD each, one |mon| and one |mgr| on each node too). Today, the "master" node (the from on which I run all the setup commands) crashed so I rebooted the server. Now, all ceph commands hangs (|ceph -s| for ex), when I do |cephadm shell| it asks me to specify the FSID (showing me 3 fsid) and in the logs of the unit |ceph-<FSID>@mon.ceph-n-01.service| I get this line looping : |Dec 10 16:13:07 ceph-n-01 bash[36847]: debug 2020-12-10T16:13:07.099+0000 7f9251890700 0 cephx: verify_authorizer could not decrypt ticket info: error: bad magic in decode_decrypt, 924161695680779075 != 18374858748799134293 Dec 10 16:13:10 ceph-n-01 bash[36847]: debug 2020-12-10T16:13:10.288+0000 7f9251890700 1 mon.ceph-n-01@0(probing) e3 handle_auth_request failed to assign global_id | Thanks for all your help :)

3 years, 4 months

1
0
0 0

removing index for non-existent buckets

by Christopher Durham

Hi, I am uisng 15.2.7 on CentOS 8.1. I have a number of old buckets that are listed with # radosgw-admin metadata list bucket.instance but are not listed with: # radosgw-admin bucket list Lets say that one of them is: 'old-bucket' and its instance is 'c100feda-5e16-48a4-b908-7be61aa877ef.123.1' The bucket/instance is empty: # radosgw-admin bucket list --bucket old-bucket --bucket-id c100feda-5e16-48a4-b908-7be61aa877ef.123.1[] Again, if I ist the index pool I see the .dir object: # rados -p <indexpoool> ls | grep c100feda-5e16-48a4-b908-7be61aa877ef.dir.c100feda-5e16-48a4-b908-7be61aa877ef.123.1.N (an N for each shard) I THINK I can delete these old indices with the following two commands # radosgw-admin bi purge --bucket old-bucket --bucket-id c100feda-5e16-48a4-b908-7be61aa877ef.123.1# radosgw-admin metadata rm bucket.instance:old-bucket:c100feda-5e16-48a4-b908-7be61aa877ef.123.1 Am I correct? What if I have another zone in a multi site configuration? Will this suffice to remove the indices from both sites? I thought I better check herebefore doing this and causing myself grief. Thanks! -Chris

3 years, 4 months

1
0
0 0

Running Mons on msgrv2/3300 only.

by Wesley Dillingham

We rebuilt all of our mons in one cluster such that they bind only to port 3300 with msgrv2. Previous to this we were binding to both 6789 and 3300. All of our server and client components are sufficiently new (14.2.x) and we haven’t observed any disruption but I am inquiring if this may be problematic for any unforeseen reason. We don’t intend to have any older clients connecting. https://docs.ceph.com/en/latest/rados/configuration/msgr2/ doesn’t mention much about running with only v2 so I just want to make sure we aren’t setting ourselves up for trouble. Thanks. -- Respectfully, Wes Dillingham Site Reliability Engineer IV Storage Engineering / Ceph wdillingham(at)godaddy.com

3 years, 4 months

3
2
0 0

CentOS

by DHilsbos＠performair.com

All; As you may or may not know; this morning RedHat announced the end of CentOS as a rebuild distribution[1]. "CentOS" will be retired in favor of the recently announced "CentOS Stream." Can Ceph be installed on CentOS Stream? Since CentOS Stream is currently at 8, the question really is: Can Ceph Octopus be installed on CentOS Stream 8? How about Nautilus? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com [1: https://blog.centos.org/2020/12/future-is-centos-stream/]

3 years, 4 months

4
6
0 0

DocuBetter Meeting cancelled this week.

by John Zachary Dover

The DocuBetter meeting scheduled for one hour from the time of this email is cancelled. Send documentation concerns to me at zac.dover(a)gmail.com. Zac Dover Upstream Docs Ceph

3 years, 4 months

1
0
0 0

How to copy an OSD from one failing disk to another one

by huxiaoyu＠horebdata.cn

Hi, dear cephers, On one ceph i have a failing disk, whose SMART information signals an impending failure but still availble for reads and writes. I am setting up a new disk on the same node to replace it. What is the best procedure to migrate data (or COPY ) from the failing OSD to the new one? Is there any stardard method to copy the OSD from one to another? best regards, samuel huxiaoyu(a)horebdata.cn

3 years, 4 months

3
2
0 0

2024

2023

2022

2021

2020

2019

ceph-users December 2020