December 2020 - ceph-users

Anonymous access to grafana
by Alessandro Piazza 12 Dec '20

12 Dec '20

Dear all, It seems that by default the grafana web page embedded inside the ceph dashboard is publicly available in read-only mode. More specifically the grafana configuration inside the docker running the grafana instance has the following configuration file (/usr/share/ceph/mgr/cephadm/templates/services/grafana/grafana.ini.j2) [auth.anonymous] enabled = true org_name = 'Main Org.' org_role = 'Viewer' Do you think that this might be a security concern? Is there a way to enforce authentication also for the read-only mode? I wasn't able to find any documentation on how to configure grafana. The only thing I found which might be related to this issue is the following: https://tracker.ceph.com/issues/45372. Regards, Alessandro Piazza

1 0

MON: global_init: error reading config file.
by Oscar Segarra 11 Dec '20

11 Dec '20

Hi, In my environment I have a single node and I'm trying to run a ceph monitor as a docker container using a kv store. Version: Octopus (stable-5.0) 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: STAYALIVE: container will not die if a command fails., 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'auth' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'global' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'mon' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'mds' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'osd' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: 'client' key already exists, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: Adding Mon Host - vdicnode01., 0.0.0.0, vdicnode01, true, 2020-12-12 00:24:28 /opt/ceph-container/bin/entrypoint.sh: Configuration found for cluster ceph. Writing to disk., true, 2020-12-12T00:24:29+01:00 vdicnode01.local /tmp/gztmpGeLTQkuFB/confd[139]: INFO Backend set to etcd, 2020-12-12T00:24:29+01:00 vdicnode01.local /tmp/gztmpGeLTQkuFB/confd[139]: INFO Starting confd, 2020-12-12T00:24:29+01:00 vdicnode01.local /tmp/gztmpGeLTQkuFB/confd[139]: INFO Backend source(s) set to http://127.0.0.1:2379, 2020-12-12 00:24:29 /opt/ceph-container/bin/entrypoint.sh: Adding bootstrap keyrings., 2020-12-12 00:24:30 /opt/ceph-container/bin/entrypoint.sh: Adding mon/admin Keyrings., 2020-12-12 00:24:30 /opt/ceph-container/bin/entrypoint.sh: Trying to get the most recent monmap..., Error initializing cluster client: InvalidArgumentError('RADOS invalid argument (error calling conf_read_file)',), 2020-12-12 00:24:30 /opt/ceph-container/bin/entrypoint.sh: Peers not found, using initial monmap., 2020-12-12 00:24:30 /opt/ceph-container/bin/entrypoint.sh: Removing lock for vdicnode01., PrevNode.Value: vdicnode01, importing contents of /var/lib/ceph/bootstrap-osd/ceph.keyring into /etc/ceph/ceph.mon.keyring, importing contents of /var/lib/ceph/bootstrap-mds/ceph.keyring into /etc/ceph/ceph.mon.keyring, importing contents of /var/lib/ceph/bootstrap-rgw/ceph.keyring into /etc/ceph/ceph.mon.keyring, importing contents of /var/lib/ceph/bootstrap-rbd-mirror/ceph.keyring into /etc/ceph/ceph.mon.keyring, importing contents of /etc/ceph/ceph.client.admin.keyring into /etc/ceph/ceph.mon.keyring, global_init: error reading config file., And container dies (even the stayalive flag). docker service create -d --network=host \ --mount type=bind,source=/var/lib/ceph,destination=/var/lib/ceph \ --mount type=bind,source=/var/log/ceph,destination=/var/log/ceph \ --mount type=bind,source=/etc/localtime,destination=/etc/localtime,readonly \ -e KV_TYPE=etcd \ -e KV_IP=127.0.0.1 \ -e KV_PORT=2379 \ -e MON_IP=0.0.0.0 \ -e CEPH_PUBLIC_NETWORK=192.168.100.0/24 \ -e CEPH_CLUSTER_NETWORK=192.168.100.0/24 \ -e DEBUG=stayalive \ --name=vdicube_ceph_mon \ --mode=global \ --constraint node.labels.mon==1 \ ceph/daemon:v5.0.6-stable-5.0-octopus-centos-8 mon Anyone has experimented the same issue? Thanks a lot. Ó

1 0

Incomplete PG due to primary OSD crashing during EC backfill - get_hash_info: Mismatch of total_chunk_size 0
by Byrne, Thomas (STFC,RAL,SC) 11 Dec '20

11 Dec '20

Hi all, Got an odd issue that I'm not sure how to solve on our Nautilus 14.2.9 EC cluster. The primary OSD of an EC 8+3 PG died this morning with a very sad disk (thousands of pending sectors). After the down out interval a new 'up' primary was assigned and the backfill started. Twenty minutes later the acting primary (not the new 'up' primary) started crashing with a "get_hash_info: Mismatch of total_chunk_size 0" error (see log below) This crash always happens at the same object, with different acting primaries, and with a different new 'up' primary. I can't see anything in the logs that points to a particular OSD being the issue, so I suspect there is a corrupted object in the PG that is causing issues, but I'm not sure how to dig into this further. The PG is currently active (but degraded), but only whilst nobackfill or noout are set (+turning the new OSD off), and if the flags are unset the backfill will eventually crash enough OSDs to render the PG incomplete, which is not ideal. I would appreciate being able to resolve this so I can go back to letting Ceph deal with down OSDs itself :) Does anyone have some pointers on how to dig into or resolve this? Happy to create a tracker ticket and post more logs if this looks like a bug. Thanks, Tom OSD log with debug_osd=20 (preamble cut from subsequent lines in an attempt to improve readability...): 2020-12-10 15:14:16.130 7fc0a1575700 10 osd.1708 pg_epoch: 1162259 pg[11.214s1( v 1162255'714638 (1162110'711564,1162255'714638] local-lis/les=1162253/1162254 n=133385 ec=1069520/992 lis/c 1162253/1125301 les/c/f 1162254/1125302/257760 1162252/1162253/1162253) [2449,1708,2099,1346,4309,777,5098,4501,4134,217,4643]/[2147483647,1708,2099,1346,4309,777,5098,4501,4134,217,4643]p1708(1) backfill=[2449(0)] r=1 lpr=1162253 pi=[1125301,1162253)/3 rops=1 crt=1162255'714638 lcod 1162254'714637 mlcod 1162254'714637 active+undersized+degraded+remapped+backfilling mbc={}] run_recovery_op: starting RecoveryOp(hoid=11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head v=1162125'713150 missing_on=2449(0) missing_on_shards=0 recovery_info=ObjectRecoveryInfo(11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head@1162125'713150, size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}) recovery_progress=ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false) obc refcount=3 state=IDLE waiting_on_pushes= extent_requested=0,0) continue_recovery_op: continuing RecoveryOp(hoid=11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head v=1162125'713150 missing_on=2449(0) missing_on_shards=0 recovery_info=ObjectRecoveryInfo(11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head@1162125'713150, size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}) recovery_progress=ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false) obc refcount=4 state=IDLE waiting_on_pushes= extent_requested=0,0) get_hash_info: Getting attr on 11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head get_hash_info: not in cache 11:28447b4a:::962de230-ed6c-44f2-ab02-788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dIApqLNDPp.22:head get_hash_info: found on disk, size 524288 get_hash_info: Mismatch of total_chunk_size 0 2020-12-10 15:14:16.136 7fc0a1575700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/osd/ECBackend.cc: In function 'void ECBackend::continue_recovery_op(ECBackend::RecoveryOp&, RecoveryMessages*)' thread 7fc0a1575700 time 2020-12-10 15:14:16.132060 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/osd/ECBackend.cc: 585: FAILED ceph_assert(op.hinfo) ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55e1569acf7d] 2: (()+0x4cb145) [0x55e1569ad145] 3: (ECBackend::continue_recovery_op(ECBackend::RecoveryOp&, RecoveryMessages*)+0x1764) [0x55e156da5834] 4: (ECBackend::run_recovery_op(PGBackend::RecoveryHandle*, int)+0x65b) [0x55e156da6c6b] 5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x1491) [0x55e156c26681] 6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x114c) [0x55e156c29f7c] 7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x2ff) [0x55e156a8b32f] 8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0x55e156d1aa19] 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55e156aa6b4f] 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x55e15704b216] 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55e15704dd30] 12: (()+0x7ea5) [0x7fc0c2d5dea5] 13: (clone()+0x6d) [0x7fc0c1c208dd] 2020-12-10 15:14:16.143 7fc0a1575700 -1 *** Caught signal (Aborted) ** This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

1 2

Scrubbing - osd down
by Miroslav Boháč 11 Dec '20

11 Dec '20

Hi, I have a problem with crashing OSD daemons in our Ceph 15.2.6 cluster . The problem was temporarily resolved by disabling scrub and deep-scrub. All PGs are active+clean. After a few days I tried to enable scrubbing again, but the problem persists. OSD with high latencies, PG laggy, osd not responding, OSD was marked down and the cluster is not usable. Problem appeared after failover when 10 OSD was marked as out. I will appreciate any help or advice, how to resolve this problem. Regards, Miroslav 2020-12-10T21:39:57.721883+0100 osd.1 (osd.1) 5 : cluster [DBG] 18.11 deep-scrub starts 2020-12-10T21:39:57.880861+0100 osd.1 (osd.1) 6 : cluster [DBG] 18.11 deep-scrub ok 2020-12-10T21:39:58.713422+0100 osd.1 (osd.1) 7 : cluster [DBG] 43.5 scrub starts 2020-12-10T21:39:58.719372+0100 osd.1 (osd.1) 8 : cluster [DBG] 43.5 scrub ok 2020-12-10T21:39:59.296962+0100 mgr.pve2-prg2a (mgr.91575377) 118746 : cluster [DBG] pgmap v119000: 1737 pgs: 3 active+clean+laggy, 3 active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 117 KiB/s rd, 2.5 MiB/s wr, 269 op/s 2020-12-10T21:40:00.088421+0100 osd.29 (osd.29) 74 : cluster [DBG] 1.13b deep-scrub starts 2020-12-10T21:40:01.300373+0100 mgr.pve2-prg2a (mgr.91575377) 118747 : cluster [DBG] pgmap v119001: 1737 pgs: 3 active+clean+laggy, 3 active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 202 op/s 2020-12-10T21:40:02.681058+0100 osd.34 (osd.34) 13 : cluster [DBG] 1.a2 deep-scrub ok 2020-12-10T21:40:03.304009+0100 mgr.pve2-prg2a (mgr.91575377) 118749 : cluster [DBG] pgmap v119002: 1737 pgs: 3 active+clean+laggy, 3 active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 198 op/s 2020-12-10T21:40:05.316233+0100 mgr.pve2-prg2a (mgr.91575377) 118750 : cluster [DBG] pgmap v119003: 1737 pgs: 6 active+clean+laggy, 3 active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 249 op/s 2020-12-10T21:40:07.319643+0100 mgr.pve2-prg2a (mgr.91575377) 118751 : cluster [DBG] pgmap v119004: 1737 pgs: 6 active+clean+laggy, 3 active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 212 op/s 2020-12-10T21:40:15.523134+0100 mon.pve1-prg2a (mon.0) 125943 : cluster [DBG] osd.4 reported failed by osd.24 2020-12-10T21:40:15.523325+0100 mon.pve1-prg2a (mon.0) 125944 : cluster [DBG] osd.39 reported failed by osd.24 2020-12-10T21:40:16.112299+0100 mon.pve1-prg2a (mon.0) 125946 : cluster [WRN] Health check failed: 0 slow ops, oldest one blocked for 32 sec, osd.8 has slow ops (SLOW_OPS) 2020-12-10T21:40:16.202867+0100 mon.pve1-prg2a (mon.0) 125947 : cluster [DBG] osd.4 reported failed by osd.34 2020-12-10T21:40:16.202986+0100 mon.pve1-prg2a (mon.0) 125948 : cluster [INF] osd.4 failed (root=default,host=pve1-prg2a) (2 reporters from different host after 24.000267 >= grace 22.361677) 2020-12-10T21:40:16.373925+0100 mon.pve1-prg2a (mon.0) 125949 : cluster [DBG] osd.39 reported failed by osd.6 2020-12-10T21:40:16.865608+0100 mon.pve1-prg2a (mon.0) 125951 : cluster [DBG] osd.39 reported failed by osd.8 2020-12-10T21:40:17.125917+0100 mon.pve1-prg2a (mon.0) 125952 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2020-12-10T21:40:17.139006+0100 mon.pve1-prg2a (mon.0) 125953 : cluster [DBG] osdmap e12819: 40 total, 39 up, 30 in 2020-12-10T21:40:17.140248+0100 mon.pve1-prg2a (mon.0) 125954 : cluster [DBG] osd.39 reported failed by osd.21 2020-12-10T21:40:17.344244+0100 mgr.pve2-prg2a (mgr.91575377) 118757 : cluster [DBG] pgmap v119012: 1737 pgs: 9 peering, 61 stale+active+clean, 1 active+clean+scrubbing+deep, 7 active+clean+laggy, 1659 active+clean; 3.3 TiB data, 7.8 TiB used, 14 TiB / 22 TiB avail; 44 KiB/s rd, 2.5 MiB/s wr, 107 op/s 2020-12-10T21:40:17.378069+0100 mon.pve1-prg2a (mon.0) 125955 : cluster [DBG] osd.39 reported failed by osd.26 2020-12-10T21:40:17.424429+0100 mon.pve1-prg2a (mon.0) 125956 : cluster [DBG] osd.39 reported failed by osd.18 2020-12-10T21:40:17.829447+0100 mon.pve1-prg2a (mon.0) 125957 : cluster [DBG] osd.39 reported failed by osd.36 2020-12-10T21:40:17.847373+0100 mon.pve1-prg2a (mon.0) 125958 : cluster [DBG] osd.39 reported failed by osd.1 2020-12-10T21:40:17.858371+0100 mon.pve1-prg2a (mon.0) 125959 : cluster [DBG] osd.39 reported failed by osd.17 2020-12-10T21:40:17.915755+0100 mon.pve1-prg2a (mon.0) 125960 : cluster [DBG] osd.39 reported failed by osd.28 2020-12-10T21:40:24.151192+0100 mon.pve1-prg2a (mon.0) 125986 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2020-12-10T21:40:24.608038+0100 mon.pve1-prg2a (mon.0) 125987 : cluster [WRN] Health check update: 0 slow ops, oldest one blocked for 37 sec, osd.8 has slow ops (SLOW_OPS) 2020-12-10T21:40:25.375322+0100 mgr.pve2-prg2a (mgr.91575377) 118761 : cluster [DBG] pgmap v119017: 1737 pgs: 1 active+clean+scrubbing+deep, 6 active+undersized, 40 active+undersized+wait, 13 active+undersized+degraded, 101 active+undersized+degraded+wait, 3 stale+active+clean, 2 active+clean+laggy, 1571 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 4.0 MiB/s wr, 179 op/s; 74043/2676420 objects degraded (2.766%); 622 KiB/s, 0 objects/s recovering 2020-12-10T21:40:27.379311+0100 mgr.pve2-prg2a (mgr.91575377) 118762 : cluster [DBG] pgmap v119018: 1737 pgs: 1 active+clean+scrubbing+deep, 6 active+undersized, 40 active+undersized+wait, 13 active+undersized+degraded, 101 active+undersized+degraded+wait, 3 stale+active+clean, 2 active+clean+laggy, 1571 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 3.6 MiB/s wr, 154 op/s; 74043/2676420 objects degraded (2.766%); 510 KiB/s, 0 objects/s recovering 2020-12-10T21:40:29.391905+0100 mgr.pve2-prg2a (mgr.91575377) 118763 : cluster [DBG] pgmap v119019: 1737 pgs: 1 active+clean+scrubbing+deep, 11 active+undersized, 35 active+undersized+wait, 21 active+undersized+degraded, 93 active+undersized+degraded+wait, 3 stale+active+clean, 2 active+clean+laggy, 1571 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 3.9 MiB/s wr, 170 op/s; 74043/2676420 objects degraded (2.766%); 456 KiB/s, 0 objects/s recovering 2020-12-10T21:40:29.609801+0100 mon.pve1-prg2a (mon.0) 125991 : cluster [WRN] Health check update: Degraded data redundancy: 74043/2676420 objects degraded (2.766%), 114 pgs degraded (PG_DEGRADED) 2020-12-10T21:40:29.609845+0100 mon.pve1-prg2a (mon.0) 125992 : cluster [WRN] Health check update: 0 slow ops, oldest one blocked for 42 sec, osd.8 has slow ops (SLOW_OPS) 2020-12-10T21:40:31.395715+0100 mgr.pve2-prg2a (mgr.91575377) 118764 : cluster [DBG] pgmap v119020: 1737 pgs: 1 active+clean+scrubbing+deep, 11 active+undersized, 35 active+undersized+wait, 21 active+undersized+degraded, 93 active+undersized+degraded+wait, 3 stale+active+clean, 2 active+clean+laggy, 1571 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 154 KiB/s rd, 2.1 MiB/s wr, 151 op/s; 74043/2676420 objects degraded (2.766%); 425 KiB/s, 0 objects/s recovering 2020-12-10T21:40:32.797773+0100 osd.8 (osd.8) 19 : cluster [WRN] slow request osd_op(client.91721949.0:169935 1.3b 1:dc12cc94:::rbd_header.0373d6a5f7de48:head [watch ping cookie 139741691437056] snapc 0=[] ondisk+write+known_if_redirected e12816) initiated 2020-12-10T21:39:59.813054+0100 currently delayed 2020-12-10T21:40:33.399277+0100 mgr.pve2-prg2a (mgr.91575377) 118765 : cluster [DBG] pgmap v119021: 1737 pgs: 1 active+clean+scrubbing+deep, 13 active+undersized, 33 active+undersized+wait, 28 active+undersized+degraded, 86 active+undersized+degraded+wait, 3 stale+active+clean, 2 active+clean+laggy, 1571 active+clean; 3.3 TiB data, 7.8 TiB used, 21 TiB / 29 TiB avail; 156 KiB/s rd, 1.6 MiB/s wr, 144 op/s; 74043/2676420 objects degraded (2.766%) 2020-12-10T21:40:33.841597+0100 osd.8 (osd.8) 20 : cluster [WRN] slow request osd_op(client.91721949.0:169935 1.3b 1:dc12cc94:::rbd_header.0373d6a5f7de48:head [watch ping cookie 139741691437056] snapc 0=[] ondisk+write+known_if_redirected e12816) initiated 2020-12-10T21:39:59.813054+0100 currently delayed 2020-12-10T21:40:34.611650+0100 mon.pve1-prg2a (mon.0) 125996 : cluster [WRN] Health check update: 0 slow ops, oldest one blocked for 47 sec, daemons [osd.22,osd.8] have slow ops. (SLOW_OPS) 2020-12-10T21:40:34.612166+0100 mon.pve1-prg2a (mon.0) 125997 : cluster [INF] osd.39 failed (root=default,host=pve5-prg2a) (5 reporters from different host after 39.438374 >= grace 37.721344) 2020-12-10T21:40:34.615286+0100 mon.pve1-prg2a (mon.0) 125998 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2020-12-10T21:40:34.621801+0100 mon.pve1-prg2a (mon.0) 125999 : cluster [DBG] osdmap e12821: 40 total, 38 up, 30 in 2020-12-10T21:40:34.880626+0100 osd.8 (osd.8) 21 : cluster [WRN] slow request osd_op(client.91721949.0:169935 1.3b 1:dc12cc94:::rbd_header.0373d6a5f7de48:head [watch ping cookie 139741691437056] snapc 0=[] ondisk+write+known_if_redirected e12816) initiated 2020-12-10T21:39:59.813054+0100 currently delayed 8/0 sis=12819) [28,35] r=-1 lpr=12819 pi=[12507,12819)/1 crt=12818'17226265 lcod 12818'17226264 mlcod 0'0 unknown NOTIFY mbc={}] exit Start 0.000020 0 0.000000 -823> 2020-12-10T21:40:44.079+0100 7f5310751700 5 osd.4 pg_epoch: 12822 pg[1.cc( v 12818'17226265 (12812'17224439,12818'17226265] local-lis/les=12507/12508 n=1566 ec=171/171 lis/c=12507/12507 les/c/f=12508/12508/0 sis=12819) [28,35] r=-1 lpr=12819 pi=[12507,12819)/1 crt=12818'17226265 lcod 12818'17226264 mlcod 0'0 unknown NOTIFY mbc={}] enter Started/Stray -822> 2020-12-10T21:40:44.191+0100 7f5324f7a700 5 prioritycache tune_memory target: 4294967296 mapped: 3494428672 unmapped: 383787008 heap: 3878215680 old mem: 2845415832 new mem: 2845415832 -821> 2020-12-10T21:40:44.311+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f530ef4e700' had timed out after 15 -820> 2020-12-10T21:40:44.311+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f5312f56700' had timed out after 15 -819> 2020-12-10T21:40:44.311+0100 7f532ca01700 1 osd.4 12822 is_healthy false -- internal heartbeat failed -818> 2020-12-10T21:40:44.311+0100 7f532ca01700 1 osd.4 12822 not healthy; waiting to boot -817> 2020-12-10T21:40:44.311+0100 7f532ca01700 1 osd.4 12822 tick checking mon for new map -816> 2020-12-10T21:40:44.311+0100 7f530e74d700 5 osd.4 12822 heartbeat osd_stat(store_statfs(0x7c5f5b0000/0x2e36c0000/0xba40000000, data 0x3af1c2f453/0x3afd380000, compress 0x0/0x0/0x0, omap 0x30256c, meta 0x2e33bda94), peers [3,5,6,7,8,9,13,20,28,36] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,22]) -815> 2020-12-10T21:40:44.403+0100 7f5323337700 10 monclient: tick -814> 2020-12-10T21:40:44.403+0100 7f5323337700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2020-12-10T21:40:14.409098+0100) -813> 2020-12-10T21:40:44.403+0100 7f5323337700 10 log_client log_queue is 2 last_log 7 sent 5 num 2 unsent 2 sending 2 -812> 2020-12-10T21:40:44.403+0100 7f5323337700 10 log_client will send 2020-12-10T21:40:44.063147+0100 osd.4 (osd.4) 6 : cluster [WRN] Monitor daemon marked osd.4 down, but it is still running -811> 2020-12-10T21:40:44.403+0100 7f5323337700 10 log_client will send 2020-12-10T21:40:44.063157+0100 osd.4 (osd.4) 7 : cluster [DBG] map e12822 wrongly marked me down at e12819 -810> 2020-12-10T21:40:44.403+0100 7f5323337700 10 monclient: _send_mon_message to mon.pve1-prg2a at v2:10.104.200.11:3300/0 -809> 2020-12-10T21:40:45.191+0100 7f5324f7a700 5 prioritycache tune_memory target: 4294967296 mapped: 3494428672 unmapped: 383787008 heap: 3878215680 old mem: 2845415832 new mem: 2845415832 -808> 2020-12-10T21:40:45.351+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f530ef4e700' had timed out after 15 -807> 2020-12-10T21:40:45.351+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f5312f56700' had timed out after 15 -3> 2020-12-10T21:42:10.466+0100 7f5324f7a700 5 prioritycache tune_memory target: 4294967296 mapped: 3510689792 unmapped: 367525888 heap: 3878215680 old mem: 2845415832 new mem: 2845415832 -2> 2020-12-10T21:42:10.694+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f530ef4e700' had timed out after 15 -1> 2020-12-10T21:42:10.694+0100 7f532ca01700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f530ef4e700' had suicide timed out after 150

2 3

Ceph benchmark tool (cbt)
by Seena Fallah 11 Dec '20

11 Dec '20

Hi all, I want to benchmark my production cluster with cbt. I read a bit of the code and I see something strange in it, for example, it's going to create ceph-osd by it selves ( https://github.com/ceph/cbt/blob/master/cluster/ceph.py#L373) and also shutdown the whole cluster!! ( https://github.com/ceph/cbt/blob/master/cluster/ceph.py#L212) Is there any configuration to not do harmful things with the cluster and for example just test the read_ahead_kb or simply stop some OSDs and other things that can be reverted and not get the cluster fully down?! Thanks.

3 2

All ceph commands hangs - bad magic number in monitor log
by Evrard Van Espen - Weather-Measures 11 Dec '20

11 Dec '20

Hello all, We have a production cluster running since weeks deployed on centos8 with |cephadm| (3 nodes with 6x12TB HDD each, one |mon| and one |mgr| on each node too). Today, the "master" node (the from on which I run all the setup commands) crashed so I rebooted the server. Now, all ceph commands hangs (|ceph -s| for ex), when I do |cephadm shell| it asks me to specify the FSID (showing me 3 fsid) and in the logs of the unit |ceph-<FSID>@mon.ceph-n-01.service| I get this line looping : |Dec 10 16:13:07 ceph-n-01 bash[36847]: debug 2020-12-10T16:13:07.099+0000 7f9251890700 0 cephx: verify_authorizer could not decrypt ticket info: error: bad magic in decode_decrypt, 924161695680779075 != 18374858748799134293 Dec 10 16:13:10 ceph-n-01 bash[36847]: debug 2020-12-10T16:13:10.288+0000 7f9251890700 1 mon.ceph-n-01@0(probing) e3 handle_auth_request failed to assign global_id | Thanks for all your help :)

1 0

removing index for non-existent buckets
by Christopher Durham 10 Dec '20

10 Dec '20

Hi, I am uisng 15.2.7 on CentOS 8.1. I have a number of old buckets that are listed with # radosgw-admin metadata list bucket.instance but are not listed with: # radosgw-admin bucket list Lets say that one of them is: 'old-bucket' and its instance is 'c100feda-5e16-48a4-b908-7be61aa877ef.123.1' The bucket/instance is empty: # radosgw-admin bucket list --bucket old-bucket --bucket-id c100feda-5e16-48a4-b908-7be61aa877ef.123.1[] Again, if I ist the index pool I see the .dir object: # rados -p <indexpoool> ls | grep c100feda-5e16-48a4-b908-7be61aa877ef.dir.c100feda-5e16-48a4-b908-7be61aa877ef.123.1.N (an N for each shard) I THINK I can delete these old indices with the following two commands # radosgw-admin bi purge --bucket old-bucket --bucket-id c100feda-5e16-48a4-b908-7be61aa877ef.123.1# radosgw-admin metadata rm bucket.instance:old-bucket:c100feda-5e16-48a4-b908-7be61aa877ef.123.1 Am I correct? What if I have another zone in a multi site configuration? Will this suffice to remove the indices from both sites? I thought I better check herebefore doing this and causing myself grief. Thanks! -Chris

1 0

Running Mons on msgrv2/3300 only.
by Wesley Dillingham 10 Dec '20

10 Dec '20

We rebuilt all of our mons in one cluster such that they bind only to port 3300 with msgrv2. Previous to this we were binding to both 6789 and 3300. All of our server and client components are sufficiently new (14.2.x) and we haven’t observed any disruption but I am inquiring if this may be problematic for any unforeseen reason. We don’t intend to have any older clients connecting. https://docs.ceph.com/en/latest/rados/configuration/msgr2/ doesn’t mention much about running with only v2 so I just want to make sure we aren’t setting ourselves up for trouble. Thanks. -- Respectfully, Wes Dillingham Site Reliability Engineer IV Storage Engineering / Ceph wdillingham(at)godaddy.com

3 2

CentOS
by DHilsbos＠performair.com 09 Dec '20

09 Dec '20

All; As you may or may not know; this morning RedHat announced the end of CentOS as a rebuild distribution[1]. "CentOS" will be retired in favor of the recently announced "CentOS Stream." Can Ceph be installed on CentOS Stream? Since CentOS Stream is currently at 8, the question really is: Can Ceph Octopus be installed on CentOS Stream 8? How about Nautilus? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com [1: https://blog.centos.org/2020/12/future-is-centos-stream/]

4 6

DocuBetter Meeting cancelled this week.
by John Zachary Dover 09 Dec '20

09 Dec '20

The DocuBetter meeting scheduled for one hour from the time of this email is cancelled. Send documentation concerns to me at zac.dover(a)gmail.com. Zac Dover Upstream Docs Ceph

1 0

How to copy an OSD from one failing disk to another one
by huxiaoyu＠horebdata.cn 08 Dec '20

08 Dec '20

Hi, dear cephers, On one ceph i have a failing disk, whose SMART information signals an impending failure but still availble for reads and writes. I am setting up a new disk on the same node to replace it. What is the best procedure to migrate data (or COPY ) from the failing OSD to the new one? Is there any stardard method to copy the OSD from one to another? best regards, samuel huxiaoyu(a)horebdata.cn

3 2

Announcing go-ceph v0.7.0
by John Mulligan 08 Dec '20

08 Dec '20

I'm happy to announce another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.7.0 Changes in the release are detailed in the link above. The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. Specific questions, comments, bugs etc are best directed at our github issues tracker. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

1 0

Ceph on vector machines
by Bobby 08 Dec '20

08 Dec '20

Hi all, Just out of curiosity.....Considering vector machines are being used in HPC applications to accelerate certain kernels, do you think there are some workloads in Ceph that could be good candidates to be offloaded and accelerated on vector machines ? Thanks in advance..... BR

1 0

rgw index shard much larger than others
by Dan van der Ster 08 Dec '20

08 Dec '20

Dear friends, Running 14.2.11, we have one particularly large bucket with a very strange distribution of objects among the shards. The bucket has 512 shards, and most shards have ~75k entries, but shard 0 has 1.75M entries: # rados -p default.rgw.buckets.index listomapkeys .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.0 | wc -l 1752085 # rados -p default.rgw.buckets.index listomapkeys .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.1 | wc -l 78388 # rados -p default.rgw.buckets.index listomapkeys .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.2 | wc -l 78764 We had resharded this bucket (manually) from 32 up to 512 shards just before upgrading from 12.2.12 to 14.2.11 a couple weeks ago. Any idea why shard .0 is getting such an imbalance of entries? Should we manually reshard this bucket again? Thanks! Dan

3 6

Re: How to copy an OSD from one failing disk to another one
by Janne Johansson 08 Dec '20

08 Dec '20

"ceph osd set norebalance" "ceph osd set nobackfill" Add new OSD, set osd weight to 0 on old OSD unset the norebalance and nobackfill options, and the cluster will do it all for you. Den tis 8 dec. 2020 kl 13:13 skrev huxiaoyu(a)horebdata.cn < huxiaoyu(a)horebdata.cn>: > Hi, dear cephers, > > On one ceph i have a failing disk, whose SMART information signals an > impending failure but still availble for reads and writes. I am setting up > a new disk on the same node to replace it. > What is the best procedure to migrate data (or COPY ) from the failing OSD > to the new one? > > Is there any stardard method to copy the OSD from one to another? > > best regards, > > samuel > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- May the most significant bit of your life be positive.

2 1

CfP Software Defined Storage devroom
by Jan Fajerski 08 Dec '20

08 Dec '20

FOSDEM is a free software event that offers open source communities a place to meet, share ideas and collaborate. It is well known for being highly developer-oriented and in the past brought together 8000+ participants from all over the world. It's home is in the city of Brussels (Belgium). FOSDEM 2021 will take place as an online event during the weekend of February 6./7. 2021. More details about the event can be found at http://fosdem.org/ ** Call For Participation The Software Defined Storage devroom will go into it's fifth round for talks around Open Source Software Defined Storage projects, management tools and real world deployments. Presentation topics could include but are not limited too: - Your work on a SDS project like Ceph, Gluster, OpenEBS, CORTX or Longhorn - Your work on or with SDS related projects like OpenStack SWIFT or Container Storage Interface - Management tools for SDS deployments - Monitoring tools for SDS clusters ** Important dates: - Dec 27th 2020: submission deadline for talk proposals - Dec 31st 2020: announcement of the final schedule - Feb 6th 2021: Software Defined Storage dev room Talk proposals will be reviewed by a steering committee: - Niels de Vos (OpenShift Container Storage Developer - Red Hat) - Jan Fajerski (Ceph Developer - SUSE) - TBD Use the FOSDEM 'pentabarf' tool to submit your proposal: https://penta.fosdem.org/submission/FOSDEM21 - If necessary, create a Pentabarf account and activate it. Please reuse your account from previous years if you have already created it. https://penta.fosdem.org/user/new_account/FOSDEM21 - In the "Person" section, provide First name, Last name (in the "General" tab), Email (in the "Contact" tab) and Bio ("Abstract" field in the "Description" tab). - Submit a proposal by clicking on "Create event". - If you plan to register your proposal in several tracks to increase your chances, don't! Register your talk once, in the most accurate track. - Presentations have to be pre-recorded before the event and will be streamed on the event weekend. - Important! Select the "Software Defined Storage devroom" track (on the "General" tab). - Provide the title of your talk ("Event title" in the "General" tab). - Provide a description of the subject of the talk and the intended audience (in the "Abstract" field of the "Description" tab) - Provide a rough outline of the talk or goals of the session (a short list of bullet points covering topics that will be discussed) in the "Full description" field in the "Description" tab - Provide an expected length of your talk in the "Duration" field. We suggest a length between 15 and 45 minutes. ** For accepted talks Once your proposal is accepted we will assign you a volunteer deputy who will help you to produce the talk recording. The volunteer will also try to ensure the recording is of good quality, help with uploading it to the system, broadcasting it during the event and moderate the Q&A session after the broadcast. Please note that as a presenter you're expected to be available online during and especially after the broadcast of you talk. The schedule will be available under https://fosdem.org/2021/schedule/track/software_defined_storage/ Hope to hear from you soon! And please forward this announcement. If you have any further questions, please write to the mailing list at storage-devroom(a)lists.fosdem.org and we will try to answer as soon as possible. Thanks!

1 0

MDS lost, Filesystem degraded and wont mount
by Anton Aleksandrov 08 Dec '20

08 Dec '20

Hello community, we are on ceph 13.2.8 - today something happenned with one MDS and cephs status tells, that filesystem is degraded. It won't mount either. I have take server with MDS, that was not working down. There are 2 more MDS servers, but they stay in "rejoin" state. Also only 1 is shown in "services", even though there are 2. Both running MDS servers have these lines in their logs: heartbeat_map is_healthy 'MDSRank' had timed out after 15 mds.beacon.mds2 Skipping beacon heartbeat to monitors (last acked 28.8979s ago); MDS internal heartbeat is not healthy! On one of MDS nodes I enabled more detailed debug, so I am getting there also: mds.beacon.mds3 Sending beacon up:standby seq 178 mds.beacon.mds3 received beacon reply up:standby seq 178 rtt 0.000999968 Makes no sense and too much stress in my head... Anyone could help please? Anton.

4 14

Garbage Collection on Luminous
by Priya Sehgal 08 Dec '20

08 Dec '20

Hi, I have a production cluster and it is experiencing lot of DELETES since many months. However, with the default gc configs - I did not see the cluster space utilization going down. Moreover, the gc list has more than 4 million objects. I tried increasing the gc configs on 4 rados gateways and fired manual gc using the command: *sudo radosgw-admin gc process* The changed GC configs are as follows: rgw gc max objs = 100 rgw gc obj min wait = 3600 rgw gc processor max time = 900 rgw gc processor period = 3600 rgw gc max concurrent io = 20 rgw gc max trim chunk = 64 *Questions* 1. Since the cluster is active cluster and having both PUT and DELETE requests - I am unable to determine whether manual GC is indeed helping. I did see an increase in free space one day - but from the past few days I do not see any increase in free space. How do I determine whether GC is really running? *sudo radosgw-admin gc list | grep oid | wc -l * This command the list is always increasing and I do see very old objects when I grep for oid. 2. Does GC have some problems running automatically on Luminous? It is enabled. 3. Are the above configs fine or can they be made more aggressive? 4. Is there a faster way to claim space which is pending GC without impacting much performance? Thanks and Regards, Priya -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

2 1

Re: guest fstrim not showing free space
by John Petrini 07 Dec '20

07 Dec '20

What driver did you use to mount the volumes? I believe only virtio-scsi supports fstrim commands.

2 2

Re: guest fstrim not showing free space
by Nathan Fish 07 Dec '20

07 Dec '20

Is the VM's / ext4? On Sun., Dec. 6, 2020, 12:57 p.m. Marc Roos, <M.Roos(a)f1-outsourcing.eu> wrote: > > > I have a 74GB vm with 34466MB free space. But when I do fstrim / 'rbd > du' shows still 60GB used. > When I fill the 34GB of space with an image, delete it and do again the > fstrim 'rbd du' still shows 59GB used. > > Is this normal? Or should I be able to get it to ~30GB used? > > > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 2

Ceph in FIPS Validated Environment
by Van Alstyne, Kenneth 07 Dec '20

07 Dec '20

All: I recently was tasked with building and implementing Ceph in an environment where FIPS cryptography is strictly enforced. As such, I ran into several issues regarding Ceph's use of low-level cryptographic functions since those are strictly forbidden when OpenSSL is in FIPS mode. The obvious solution is to migrate away from the low level crypto functions and over to OpenSSL's EVP API, which I wrongly assumed would be a huge undertaking. As it turns out, low level crypto functions are only used in a handful of places and the work to migrate away has already been completed in the following PRs: https://github.com/ceph/ceph/pull/23260 https://github.com/ceph/ceph/pull/32675 The latter looks like will be merged in for the Pacific release, but the former appears to have been abandoned. The perception is that these pulls are only related to performance improvements, but they also solve the corner case of running Ceph in a FIPS-enforced environment. Anecdotally, I rebased the two pulls on the latest stable Octopus release, 15.2.7, and have a cluster up and running with no issues as far as I can tell in a FIPS-enforced environment. Are there any thoughts about reopening PR#23260 and updating both PRs to notate that they also resolve FIPS compatibility issues? Thanks, -- Kenneth Van Alstyne Systems Architect M: 804.240.2327 14291 Park Meadow Drive, Chantilly, VA 20151 perspecta

1 0

Re: set rbd metadata 'conf_rbd_qos_bps_limit', make 'mkfs.xfs /dev/nbdX ' blocked
by 912273695＠qq.com 07 Dec '20

07 Dec '20

Hi，Jason： As discussed last time，After setting conf_rbd_qos_bps_limit, the speed of discard is also limited， which can make operation such as mkfs.xfs very slow, though we can add -K option to solve this problem, but we can't make sure that other operation or application never call discard interface, so if the discard operation can be not limited by conf_rbd_qos_bps_limit option, and if so, is there any other risk? Thanks. Beset Regards， LiuGangbiao

1 0

dashboard 500 internal error when listing buckets
by levin ng 07 Dec '20

07 Dec '20

Hi all, I just have done 15.2.7 installation with having 3 mon , 4 osd and 3 rgw, however after enabling rgw dashboard, the buckets page popup 500 internal error, but users and daemons are listed fine in the dashboard, radosgw-admin can list out everything without problem as well. I've tired to turn on rgw and mgr debug log the logs just shown as below, I would appreciate if you have any idea to fix this problem. Thank you! mgr log: debug 2020-12-07T07:48:32.761+0000 7f057a17d700 0 [dashboard ERROR request] [172.25.88.12:44024] [GET] [500] [0.050s] [admin] [513.0B] /api/rgw/bucket/test91 debug 2020-12-07T07:48:32.761+0000 7f057a17d700 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "0b4cfe92-c25d-474f-ad51-05d441c260df"} rgw log: debug 2020-12-07T09:09:20.725+0000 7f2d16bdf700 1 ====== starting new request req=0x7f2d76d913a0 ===== debug 2020-12-07T09:09:20.725+0000 7f2d16bdf700 2 req 4313 0s initializing for trans_id = tx0000000000000000010d9-005fcdf140-40bb4-cast debug 2020-12-07T09:09:20.725+0000 7f2d16bdf700 2 req 4313 0s getting op 0 debug 2020-12-07T09:09:20.725+0000 7f2d16bdf700 2 req 4313 0s s3:list_bucket verifying requester debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 2 req 4313 0.030002579s s3:list_bucket normalizing buckets and tenants debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 2 req 4313 0.030002579s s3:list_bucket init permissions debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 2 req 4313 0.030002579s s3:list_bucket recalculating target debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 2 req 4313 0.030002579s s3:list_bucket reading permissions debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 2 req 4313 0.030002579s s3:list_bucket init op debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 2 req 4313 0.030002579s s3:list_bucket verifying op mask debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 2 req 4313 0.030002579s s3:list_bucket verifying op permissions debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 5 req 4313 0.030002579s s3:list_bucket Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=test91-tsa, acct_name=test91-tsa, perm_mask=15, is_admin=0) mask=49 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 5 Searching permissions for uid=test91-tsa debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 5 Found permission: 15 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 5 Searching permissions for uid=test91-tsa$test91-tsa debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 5 Permissions for user not found debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 5 Searching permissions for group=1 mask=49 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 5 Permissions for group not found debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 5 Searching permissions for group=2 mask=49 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 5 Permissions for group not found debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 5 req 4313 0.030002579s s3:list_bucket -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=test91-tsa, acct_name=test91-tsa, perm_mask=15, is_admin=0), owner=test91-tsa, perm=1 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 2 req 4313 0.030002579s s3:list_bucket verifying op params debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 2 req 4313 0.030002579s s3:list_bucket pre-executing debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 2 req 4313 0.030002579s s3:list_bucket executing debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 1 -- 172.25.88.14:0/3384359887 --> [v2: 172.25.88.20:6800/2627145462,v1:172.25.88.20:6801/2627145462] -- osd_op(unknown.0.0:36867 15.5 15:a0395ab0:::.dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected e1209) v8 -- 0x55b1258ffb80 con 0x55b125517800 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 1 -- 172.25.88.14:0/3384359887 --> [v2: 172.25.88.19:6800/4042281875,v1:172.25.88.19:6801/4042281875] -- osd_op(unknown.0.0:36868 15.4 15:2069ca14:::.dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected e1209) v8 -- 0x55b126b0c780 con 0x55b125516400 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 1 -- 172.25.88.14:0/3384359887 --> [v2: 172.25.88.17:6800/1165805227,v1:172.25.88.17:6801/1165805227] -- osd_op(unknown.0.0:36869 15.2 15:40c91296:::.dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected e1209) v8 -- 0x55b129b04f00 con 0x55b1246e4800 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 1 -- 172.25.88.14:0/3384359887 --> [v2: 172.25.88.17:6800/1165805227,v1:172.25.88.17:6801/1165805227] -- osd_op(unknown.0.0:36870 15.2 15:58086687:::.dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected e1209) v8 -- 0x55b129b05680 con 0x55b1246e4800 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 1 -- 172.25.88.14:0/3384359887 --> [v2: 172.25.88.17:6800/1165805227,v1:172.25.88.17:6801/1165805227] -- osd_op(unknown.0.0:36871 15.7 15:e9aebc19:::.dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected e1209) v8 -- 0x55b127f5f180 con 0x55b1246e4800 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 1 -- 172.25.88.14:0/3384359887 --> [v2: 172.25.88.17:6800/1165805227,v1:172.25.88.17:6801/1165805227] -- osd_op(unknown.0.0:36872 15.7 15:e0023539:::.dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected e1209) v8 -- 0x55b1279f6f00 con 0x55b1246e4800 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 1 -- 172.25.88.14:0/3384359887 --> [v2: 172.25.88.19:6800/4042281875,v1:172.25.88.19:6801/4042281875] -- osd_op(unknown.0.0:36873 15.1 15:9600557f:::.dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected e1209) v8 -- 0x55b12b8ec280 con 0x55b125516400 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 1 -- 172.25.88.14:0/3384359887 --> [v2: 172.25.88.19:6800/4042281875,v1:172.25.88.19:6801/4042281875] -- osd_op(unknown.0.0:36874 15.4 15:2984dfaf:::.dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected e1209) v8 -- 0x55b12a320500 con 0x55b125516400 debug 2020-12-07T09:09:20.755+0000 7f2d16bdf700 1 -- 172.25.88.14:0/3384359887 --> [v2: 172.25.88.20:6800/2627145462,v1:172.25.88.20:6801/2627145462] -- osd_op(unknown.0.0:36875 15.5 15:b8046cce:::.dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected e1209) v8 -- 0x55b129b20280 con 0x55b125517800 debug 2020-12-07T09:09:20.756+0000 7f2d65652700 1 -- 172.25.88.14:0/3384359887 <== osd.0 v2:172.25.88.17:6800/1165805227 7141 ==== osd_op_reply(36872 .dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.5 [call out=1507b] v0'0 uv369 ondisk = 0) v8 ==== 196+0+1507 (crc 0 0 0) 0x55b12b719b00 con 0x55b1246e4800 debug 2020-12-07T09:09:20.756+0000 7f2d65652700 1 -- 172.25.88.14:0/3384359887 <== osd.0 v2:172.25.88.17:6800/1165805227 7142 ==== osd_op_reply(36870 .dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.3 [call out=2033b] v0'0 uv293 ondisk = 0) v8 ==== 196+0+2033 (crc 0 0 0) 0x55b12b719b00 con 0x55b1246e4800 debug 2020-12-07T09:09:20.756+0000 7f2d65652700 1 -- 172.25.88.14:0/3384359887 <== osd.0 v2:172.25.88.17:6800/1165805227 7143 ==== osd_op_reply(36876 .dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.9 [call out=1626b] v0'0 uv371 ondisk = 0) v8 ==== 196+0+1626 (crc 0 0 0) 0x55b12b719b00 con 0x55b1246e4800 debug 2020-12-07T09:09:20.756+0000 7f2d65652700 1 -- 172.25.88.14:0/3384359887 <== osd.3 v2:172.25.88.20:6800/2627145462 6963 ==== osd_op_reply(36867 .dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.0 [call out=642b] v0'0 uv211 ondisk = 0) v8 ==== 196+0+642 (crc 0 0 0) 0x55b12b719b00 con 0x55b125517800 debug 2020-12-07T09:09:20.756+0000 7f2d65652700 1 -- 172.25.88.14:0/3384359887 <== osd.3 v2:172.25.88.20:6800/2627145462 6964 ==== osd_op_reply(36875 .dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.8 [call out=1610b] v0'0 uv213 ondisk = 0) v8 ==== 196+0+1610 (crc 0 0 0) 0x55b12b719b00 con 0x55b125517800 debug 2020-12-07T09:09:20.756+0000 7f2d64650700 1 -- 172.25.88.14:0/3384359887 <== osd.2 v2:172.25.88.19:6800/4042281875 10713 ==== osd_op_reply(36868 .dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.1 [call out=2931b] v0'0 uv199 ondisk = 0) v8 ==== 196+0+2931 (crc 0 0 0) 0x55b12b462fc0 con 0x55b125516400 debug 2020-12-07T09:09:20.757+0000 7f2d64650700 1 -- 172.25.88.14:0/3384359887 <== osd.2 v2:172.25.88.19:6800/4042281875 10714 ==== osd_op_reply(36873 .dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.6 [call out=149b] v0'0 uv213 ondisk = 0) v8 ==== 196+0+149 (crc 0 0 0) 0x55b12b462fc0 con 0x55b125516400 debug 2020-12-07T09:09:20.757+0000 7f2d64650700 1 -- 172.25.88.14:0/3384359887 <== osd.2 v2:172.25.88.19:6800/4042281875 10715 ==== osd_op_reply(36874 .dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.7 [call out=110b] v0'0 uv169 ondisk = 0) v8 ==== 196+0+110 (crc 0 0 0) 0x55b12b462fc0 con 0x55b125516400 debug 2020-12-07T09:09:20.757+0000 7f2d64650700 1 -- 172.25.88.14:0/3384359887 <== osd.2 v2:172.25.88.19:6800/4042281875 10716 ==== osd_op_reply(36877 .dir.4f3ad19b-c138-4c21-a860-df0df8841e27.179340.6.10 [call out=1052b] v0'0 uv215 ondisk = 0) v8 ==== 197+0+1052 (crc 0 0 0) 0x55b12b462fc0 con 0x55b125516400 debug 2020-12-07T09:09:20.757+0000 7f2d16bdf700 2 req 4313 0.032002751s s3:list_bucket completing debug 2020-12-07T09:09:20.757+0000 7f2d06bbf700 2 req 4313 0.032002751s s3:list_bucket op status=0 debug 2020-12-07T09:09:20.757+0000 7f2d06bbf700 2 req 4313 0.032002751s s3:list_bucket http status=200 debug 2020-12-07T09:09:20.757+0000 7f2d06bbf700 1 ====== req done req=0x7f2d76d913a0 op status=0 http_status=200 latency=0.032002751s ======

1 0

guest fstrim not showing free space
by Marc Roos 06 Dec '20

06 Dec '20

I have a 74GB vm with 34466MB free space. But when I do fstrim / 'rbd du' shows still 60GB used. When I fill the 34GB of space with an image, delete it and do again the fstrim 'rbd du' still shows 59GB used. Is this normal? Or should I be able to get it to ~30GB used?

2 2

ceph daemon mgr.# dump_osd_network: no valid command found
by Frank Schilder 06 Dec '20

06 Dec '20

Dear cephers, I see "Long heartbeat ping times on back interface seen" in ceph status and ceph health detail says that I should "Use ceph daemon mgr.# dump_osd_network for more information". I tries, but it seems this command was removed during upgrade from mimic 13.2.8 to 13.2.10: [root@ceph-01 ~]# ceph daemon mgr.ceph-01 dump_osd_network no valid command found; 10 closest matches: log flush log dump git_version get_command_descriptions kick_stale_sessions help config unset <var> config show dump_mempools dump_cache admin_socket: invalid command Has this been replaced by some other command? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 4

Increase number of objects in flight during recovery
by Frank Schilder 06 Dec '20

06 Dec '20

Hi all, I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect. Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge: NAME ID USED %USED MAX AVAIL OBJECTS con-fs2-meta1 12 216 MiB 0.02 933 GiB 13311115 con-fs2-meta2 13 0 B 0 933 GiB 118389897 con-fs2-data 14 698 TiB 72.15 270 TiB 286826739 Unfortunately, there were no recommendations on dimensioning PG numbers for this pool, so I used the same for con-fs2-meta1, and con-fs2-meta2. In hindsight, this was potentially a bad idea, the meta2 pool should have a much higher PG count or a much more aggressive recovery policy. I now need to rebalance PGs on meta2 and it is going way too slow compared with the performance of the SSDs it is located on. In a way, I would like to keep the PG count where it is, but increase the recovery rate for this pool by a factor of 10. Please let me know what options I have. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

5 9

ceph-volume / ecnrypted OSD issues with functionalities
by Panayiotis Gotsis 04 Dec '20

04 Dec '20

Hello, I have made some tests with creating OSDs and I have found out that there are big issues with the ceph-volume functionality. 1) If using dmcrypt and separate data and db block devices, ceph-volume creates cryprodevs/PVs/VGs/LVs for both devices. This might seem as normal, until one considers the possibility that a single SSD will support multiple OSDs and the whole concept of being able to resize bluestore DB partitions gets very complicated like this; I can have a full SSD encrypted, create a PV/VG on top and provide the LVs myself without an extra layer of encryption. 2) If I try to circumvent this limitation by asking ceph-volume to use already encrypted devices for both the data and the bluestore db, the OSDs are not auto scanned at startup, ceph-volume simple scan complaints that they are not real devices and nothing starts. Am I missing something?

1 1

block.db/block.wal device performance dropped after upgrade to 14.2.10
by Vladimir Prokofev 04 Dec '20

04 Dec '20

Good day, cephers! We've recently upgraded our cluster from 14.2.8 to 14.2.10 release, also performing full system packages upgrade(Ubuntu 18.04 LTS). After that performance significantly dropped, main reason beeing that journal SSDs are now have no merges, huge queues, and increased latency. There's a few screenshots in attachments. This is for an SSD journal that supports block.db/block.wal for 3 spinning OSDs, and it looks like this for all our SSD block.db/wal devices across all nodes. Any ideas what may cause that? Maybe I've missed something important in release notes?

6 20

High read throughput on BlueFS
by Seena Fallah 04 Dec '20

04 Dec '20

Hi all, When my cluster gets into a recovery state (adding new node) I see a huge read throughput on its disks and it affects the latency! Disks are SSD and they don't have a separate WAL/DB. I'm using nautilus 14.2.14 and bluefs_buffered_io is false by default. When this throughput came on my disk it will get too much high latency. After I turned on bluefs_buffered_io another huge throughput around 1.2GB/s came in and it against affect my latency but much less than the previous one! (Graphs are attached and bluefs_buffered_io was turned on with ceph tell injectargs at 13:41 also I have restarted the OSD at 13:16 because it doesn't get better at the moment) I have four questions: 1. What are they? I see the recovery speed is 20MB/s and client io on that OSD is 10MB/s so what is this high throughput for?! 2. How can I control this throughput? Because my disks don't support this much throughput! 3. I see a common issue here https://tracker.ceph.com/issues/36482 that I think it's similar to my case. It was discussed about read_ahead, well should I change the read_ahead_kb config of my disk to support this type of request? I'm using the default value in ubuntu (128) 4. Is there any tuning required that can help to turn off the bluefs_buffered_io again? Configs I used for recovery: osd max backfills = 1 osd recovery max active = 1 osd recovery op priority = 1 osd recovery priority = 1 osd recovery sleep ssd = 0.2 My OSD memory target is around 6GB. Thanks.

1 2

bucket radoslist stuck in a loop while listing objects
by James, GleSYS 04 Dec '20

04 Dec '20

Hi, I recently attempted to run the ‘rgw-orphan-list’ tool against our cluster (octopus 15.2.7) to identify any orphans and noticed that the 'radosgw-admin bucket radoslist’ command appeared to be stuck in a loop. I saw in the 'radosgw-admin-XXXXXX.intermediate’ output file the same sequence of objects looping repeatedly, and the command would not progress. There is a tracker item that appears to be for this issue (https://tracker.ceph.com/issues/47074) but I believe it is still unverified whether this is a bug or not. I ran the command 'radosgw-admin bucket radoslist --bucket=<bucketname> --debug-rgw=20 > radoslist-out.txt 2> radoslist-err.txt’ in order to identify what was happening as per the tracker, but I would appreciate some help in analysing the logs. In the radoslist-err.txt file, I see entries similar to the following, which just repeat until I cancel the command: 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=345930 stripe_ofs=345930 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=360383 stripe_ofs=360383 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=302704 stripe_ofs=302704 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=381794 stripe_ofs=381794 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=334269 stripe_ofs=334269 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=378182 stripe_ofs=378182 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=325041 stripe_ofs=325041 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=295118 stripe_ofs=295118 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=315823 stripe_ofs=315823 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=346112 stripe_ofs=346112 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=303128 stripe_ofs=303128 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=364011 stripe_ofs=364011 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): result: ofs=331137 stripe_ofs=331137 part_ofs=0 rule->part_size=0 2020-12-04T12:37:22.227+0100 7fc405f79080 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 Just before these entries begin, I see the following: 2020-12-04T12:37:22.203+0100 7fc405f79080 20 RGWRados::Bucket::List::list_objects_ordered INFO end of outer loop, truncated=1, count=4, attempt=9 2020-12-04T12:37:22.203+0100 7fc405f79080 20 RGWRadosList::do_incomplete_multipart processing incomplete multipart entry RGWMultipartUploadEntry{ obj.key="_multipart_/halon_2_settings_2020-08-28.json.2~TD-WUPWhOicUKkz4NTxZcxuEl3b_IqC.meta" mp=RGWMPObj:{ prefix="/halon_2_settings_2020-08-28.json.2~TD-WUPWhOicUKkz4NTxZcxuEl3b_IqC", meta="/halon_2_settings_2020-08-28.json.2~TD-WUPWhOicUKkz4NTxZcxuEl3b_IqC.meta" } } 2020-12-04T12:37:22.203+0100 7fc405f79080 20 RGWRadosList::do_incomplete_multipart processing incomplete multipart entry RGWMultipartUploadEntry{ obj.key="_multipart_gl-events_0_2020-08-28.json.2~OQWHwayxwHwUMLteZWQZWRnsdgT5Cl2.meta" mp=RGWMPObj:{ prefix="gl-events_0_2020-08-28.json.2~OQWHwayxwHwUMLteZWQZWRnsdgT5Cl2", meta="gl-events_0_2020-08-28.json.2~OQWHwayxwHwUMLteZWQZWRnsdgT5Cl2.meta" } } 2020-12-04T12:37:22.203+0100 7fc405f79080 20 RGWRadosList::do_incomplete_multipart processing incomplete multipart entry RGWMultipartUploadEntry{ obj.key="_multipart_gl-events_0_2020-08-28.json.2~zz-6rqcuHGmSPYTY2IRueMP5HIFOL6t.meta" mp=RGWMPObj:{ prefix="gl-events_0_2020-08-28.json.2~zz-6rqcuHGmSPYTY2IRueMP5HIFOL6t", meta="gl-events_0_2020-08-28.json.2~zz-6rqcuHGmSPYTY2IRueMP5HIFOL6t.meta" } } 2020-12-04T12:37:22.203+0100 7fc405f79080 20 RGWRadosList::do_incomplete_multipart processing incomplete multipart entry RGWMultipartUploadEntry{ obj.key="_multipart_gl-events_1_2020-08-28.json.2~-WCk4AJE1k7or-KN6qTePxfIvHL1NLf.meta" mp=RGWMPObj:{ prefix="gl-events_1_2020-08-28.json.2~-WCk4AJE1k7or-KN6qTePxfIvHL1NLf", meta="gl-events_1_2020-08-28.json.2~-WCk4AJE1k7or-KN6qTePxfIvHL1NLf.meta" } } Is anyone able to interpret these logs and see an explanation as to why the command appears to be looping over the same objects? Thanks, James.

1 0

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

PG_DAMAGED
by Szabo, Istvan (Agoda) 04 Dec '20

04 Dec '20

Hi, Not sure is it related to my 15.2.7 update, but today I got many time this issue: 2020-12-04T15:14:23.910799+0700 osd.40 (osd.40) 11 : cluster [DBG] 11.2 deep-scrub starts 2020-12-04T15:14:23.947255+0700 osd.40 (osd.40) 12 : cluster [ERR] 11.2 soid 11:434f049b:::.dir.75333f99-93d0-4238-91a4-ba833a0edd24.1744118.372.1:head : omap_digest 0x48532c00 != omap_digest 0x8a18f5d7 from shard 40 2020-12-04T15:14:23.977138+0700 mgr.hk-cephmon-2s02 (mgr.2120884) 4330 : cluster [DBG] pgmap v4338: 209 pgs: 209 active+clean; 2.8 GiB data, 21 TiB used, 513 TiB / 534 TiB avail; 32 KiB/s rd, 32 op/s 2020-12-04T15:14:24.030888+0700 osd.40 (osd.40) 13 : cluster [ERR] 11.2 soid 11:4b86603b:::.dir.75333f99-93d0-4238-91a4-ba833a0edd24.1744118.197.3:head : omap_digest 0xcb62779b != omap_digest 0xefef7471 from shard 40 2020-12-04T15:14:24.229000+0700 osd.40 (osd.40) 14 : cluster [ERR] 11.2 deep-scrub 0 missing, 2 inconsistent objects 2020-12-04T15:14:24.229003+0700 osd.40 (osd.40) 15 : cluster [ERR] 11.2 deep-scrub 2 errors 2020-12-04T15:14:25.978189+0700 mgr.hk-cephmon-2s02 (mgr.2120884) 4331 : cluster [DBG] pgmap v4339: 209 pgs: 1 active+clean+scrubbing+deep, 208 active+clean; 2.8 GiB data, 21 TiB used, 513 TiB / 534 TiB avail; 55 KiB/s rd, 0 B/s wr, 61 op/s 2020-12-04T15:14:27.978588+0700 mgr.hk-cephmon-2s02 (mgr.2120884) 4332 : cluster [DBG] pgmap v4340: 209 pgs: 1 active+clean+scrubbing+deep, 208 active+clean; 2.8 GiB data, 21 TiB used, 513 TiB / 534 TiB avail; 43 KiB/s rd, 0 B/s wr, 49 op/s 2020-12-04T15:14:30.293180+0700 mon.hk-cephmon-2s01 (mon.0) 4475 : cluster [ERR] Health check failed: 2 scrub errors (OSD_SCRUB_ERRORS) 2020-12-04T15:14:30.293196+0700 mon.hk-cephmon-2s01 (mon.0) 4476 : cluster [ERR] Health check failed: Possible data damage: 1 pg inconsistent (PG_DAMAGED) I had to repair pg and it worked fine, but not sure where this come from. I have this in the log only :/ Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

4 6

Whether removing device_health_metrics pool is ok or not
by Satoru Takeuchi 04 Dec '20

04 Dec '20

Hi, Could you tell me whether it's ok to remove device_health_metrics pool after disabling device monitoring feature? I don't use device monitoring feature because I capture hardware information from other way. However, after disabling this feature, device_health_metrics pool stll exists. I don't want to concern HEALTH_WARN caused by problems in PGs of this pool. As a result of reading the source code of device monitoring module, it seems to be safe to remove this pool. Is my understanding correct? Thanks, Satoru

2 2

add server in crush map before osd
by Francois Legrand 03 Dec '20

03 Dec '20

Hello, I have a ceph nautilus cluster. The crushmap is organized with 2 rooms, servers in these rooms and osd in these servers, I have a crush rule to replicate data over the servers in different rooms. Now, I want to add a new server in one of the rooms. My point is that I would like to specify the room of this new server BEFORE creating osd in this server (so the data added to the osd will be directly at the right location). My problem is that it seems that servers appears in the crushmap only when they have osds... and when you create a first osd, the server is inserted in the crushmap under the default bucket (so not in a room and then the first data stored in this osd will not be at the correct location). I could move it after (if I do it rapidly, there will be no that much data to move after), but I was wondering if there is a way to either define the position of a server in the crushmap hierarchy before creating osd or eventually to specify the room when creating the first osd ? F.

7 7

Many ceph commands hang. broken mgr?
by Paul Mezzanini 03 Dec '20

03 Dec '20

Ever since we jumped from 14.2.9 to .12 (and beyond) a lot of the ceph commands just hang. The mgr daemon also just stops responding to our Prometheus scrapes occasionally. A daemon restart and it wakes back up. I have nothing pointing to these being related but it feels that way. I also tried to get device health monitoring with smart up and running around that upgrade time. It never seemed to be able to pull in and report on the health across the drives. I did see the osd process firing off smartctl on occasion though so it was trying to do something. Again, I have nothing pointing to this being related but it feels like it may be. Some commands that currently hang: ceph osd pool autoscale-status ceph balancer * ceph iostat (oddly, this spit out a line of all 0 stats once and then hung) ceph fs status toggling ceph device monitoring on or off and a lot of the device health stuff too Mgr logs on disk show flavors of this: 2020-11-24 13:05:07.883 7f19e2c40700 0 log_channel(audit) log [DBG] : from='mon.0 -' entity='mon.' cmd=[{,",p,r,e,f,i,x,",:, ,",o,s,d, ,p,e,r,f,",,, ,",f,o,r,m,a,t,",:, ,",j,s,o,n,",}]: dispatch 2020-11-24 13:05:07.895 7f19e2c40700 0 log_channel(audit) log [DBG] : from='mon.0 -' entity='mon.' cmd=[{,",p,r,e,f,i,x,",:, ,",o,s,d, ,p,o,o,l, ,s,t,a,t,s,",,, ,",f,o,r,m,a,t,",:, ,",j,s,o,n,",}]: dispatch 2020-11-24 13:05:08.567 7f19e1c3e700 0 log_channel(cluster) log [DBG] : pgmap v587: 17149 pgs: 1 active+remapped+backfill_wait, 2 active+clean+scrubbing, 55 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 17082 active+clean; 2.1 PiB data, 3.5 PiB used, 2.9 PiB / 6.4 PiB avail; 108 MiB/s rd, 53 MiB/s wr, 1.20k op/s; 7525420/9900121381 objects misplaced (0.076%); 99 MiB/s, 40 objects/s recovering ceph status: cluster: id: 971a5242-f00d-421e-9bf4-5a716fcc843a health: HEALTH_WARN 1 nearfull osd(s) 1 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon-01,ceph-mon-03,ceph-mon-02 (age 4h) mgr: ceph-mon-01(active, since 97s), standbys: ceph-mon-03, ceph-mon-02 mds: cephfs:1 {0=ceph-mds-02=up:active} 3 up:standby osd: 843 osds: 843 up (since 13d), 843 in (since 2w); 10 remapped pgs rgw: 1 daemon active (ceph-rgw-01) task status: scrub status: mds.ceph-mds-02: idle data: pools: 16 pools, 17149 pgs objects: 1.61G objects, 2.1 PiB usage: 3.5 PiB used, 2.9 PiB / 6.4 PiB avail pgs: 6482000/9900825469 objects misplaced (0.065%) 17080 active+clean 54 active+clean+scrubbing+deep 9 active+remapped+backfilling 5 active+clean+scrubbing 1 active+remapped+backfill_wait io: client: 877 MiB/s rd, 1.8 GiB/s wr, 1.91k op/s rd, 3.33k op/s wr recovery: 136 MiB/s, 55 objects/s ceph config dump: WHO MASK LEVEL OPTION VALUE RO global advanced cluster_network 192.168.42.0/24 * global advanced mon_max_pg_per_osd 400 global advanced mon_pg_warn_max_object_skew -1.000000 global dev mon_warn_on_pool_pg_num_not_power_of_two false global advanced osd_max_backfills 2 global advanced osd_max_scrubs 4 global advanced osd_scrub_during_recovery false global advanced public_network 1xx.xx.171.0/24 10.16.171.0/24 * mon advanced mon_allow_pool_delete true mgr advanced mgr/balancer/mode none mgr advanced mgr/devicehealth/enable_monitoring false osd advanced bluestore_compression_mode passive osd advanced osd_deep_scrub_large_omap_object_key_threshold 2000000 osd advanced osd_op_queue_cut_off high * osd advanced osd_scrub_load_threshold 5.000000 mds advanced mds_beacon_grace 300.000000 mds basic mds_cache_memory_limit 16384000000 mds advanced mds_log_max_segments 256 client advanced rbd_default_features 5 client.libvirt advanced admin_socket /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok * client.libvirt basic log_file /var/log/ceph/qemu-guest-$pid.log * /etc/ceph/ceph.conf is the stub file with fsid and the mons listed. Yes I have a drive that just started to tickle the full warn limit. That's what pulled me back into the "I should fix this" mode. I'm manually adjusting the weight on that one for the time being along with slowly lowering pg_num on an oversized pool. The cluster still has this issue when in health_ok. I'm free to do a lot of debugging and poking around even though this is our production cluster. The only service I refuse to play around with is the MDS. That one bites back. Does anyone have more ideas on where to look to try and figure out what's going on? -- Paul Mezzanini Sr Systems Administrator / Engineer, Research Computing Information & Technology Services Finance & Administration Rochester Institute of Technology o:(585) 475-3245 | pfmeec(a)rit.edu CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information. ------------------------

2 7

add OSDs to cluster
by mj 03 Dec '20

03 Dec '20

Hi, We are wondering why adding an OSD to a healthy cluster results in a (very small percentage of) "Degraded data redundancy". (0.020%) We understand a large percentage of misplaced objects (7.622%) But since we're adding an OSD to a HEALTH_OK cluster, there should really not be any degraded data redundancy, right..? MJ

2 1

slow down keys/s in recovery
by Seena Fallah 03 Dec '20

03 Dec '20

Hi all, Is there any configuration to slow down keys/s in recovery mode? Thanks.

4 17

Re: Monitors not starting, getting "e3 handle_auth_request failed to assign global_id"
by Hoan Nguyen Van 03 Dec '20

03 Dec '20

I have same issue. My cluster version is 14.2.1, i never meet it before. I see some information from this tracker : https://tracker.ceph.com/issues/48033

2 1

How to create single OSD with SSD db device with cephadm
by 胡玮文 03 Dec '20

03 Dec '20

Hi all, I’m new to ceph. I recently deployed a ceph cluster with cephadm. Now I want to add a single new OSD daemon with a db device on SSD. But I can’t find any documentation about this. I have tried: 1. Using web dashboard. This requires at least one filter to proceed (type, vendor, model or size). But I just want to select the block device manually. 2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based. 3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot specify my SSD db device in this way. 4. On the target host, run ‘cephadm shell’ then ceph-volume prepare and activate. But ceph-volume seems can’t create systemd service outside the container like ‘ceph orch’ does. 5. On the target host, run ‘cephadm ceph-volume’, but it requires a json config file, I can’t figure out what is that. Any help is appreciated. Thanks.

2 4

replace osd with Octopus
by Tony Liu 02 Dec '20

02 Dec '20

Hi, I did some search about replacing osd, and found some different steps, probably for different release? Is there recommended process to replace an osd with Octopus? Two cases here: 1) replace HDD whose WAL and DB are on a SSD. 1-1) failed disk is replaced by the same model. 1-2) working disk is replaced by bigger one. 2) replace the SSD holding WAL and DB for multiple HDDs. Thanks! Tony

4 13

Ceph-ansible vs. Cephadm - Nautilus to Octopus and beyond
by Dave Hall 02 Dec '20

02 Dec '20

Hello. The topic of Ceph-Ansible hasn't appeared in the list for a few months, but there's been a lot of talk about Cephadm. So what are the pros and cons? Is Cephadm good enough to put Ceph-Ansible out of business, or will it still be viable beyond Pacific? For my part the Ansible approach seems more straightforward since one can just lay out a cluster in a flat file and press 'go'. BTW, I currently have a non-containerized cluster running Nautilus, but I'll eventually have to move to Octopus. Further, I'm not (yet) convinced that the container-based approach is better than bare-metal. I think I saw that Cephadm will only deploy container-based clusters. Is this a hint that bare-metal is going away in the long run? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

1 0

Upgrade to 15.2.7 fails on mixed x86_64/arm64 cluster
by Bryan Stillwell 02 Dec '20

02 Dec '20

I tried upgrading my home cluster to 15.2.7 (from 15.2.5) today and it appears to be entering a loop when trying to match docker images for ceph:v15.2.7: 2020-12-01T16:47:26.761950-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mgr daemons... 2020-12-01T16:47:26.769581-0700 mgr.aladdin.liknom [INF] Upgrade: All mgr daemons are up to date. 2020-12-01T16:47:26.770096-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mon daemons... 2020-12-01T16:47:28.800426-0700 mgr.aladdin.liknom [INF] Upgrade: All mon daemons are up to date. 2020-12-01T16:47:28.800878-0700 mgr.aladdin.liknom [INF] Upgrade: Checking crash daemons... 2020-12-01T16:47:28.851819-0700 mgr.aladdin.liknom [INF] Upgrade: Setting container_image for all crash... 2020-12-01T16:47:28.855595-0700 mgr.aladdin.liknom [INF] Upgrade: All crash daemons are up to date. 2020-12-01T16:47:28.856283-0700 mgr.aladdin.liknom [INF] Upgrade: Checking osd daemons... 2020-12-01T16:47:31.348345-0700 mgr.aladdin.liknom [INF] Upgrade: Pulling docker.io/ceph/ceph:v15.2.7 on mandalaybay 2020-12-01T16:47:35.311065-0700 mgr.aladdin.liknom [INF] Upgrade: image docker.io/ceph/ceph:v15.2.7 pull on mandalaybay got new image 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959 (not 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e), restarting 2020-12-01T16:47:35.534893-0700 mgr.aladdin.liknom [INF] Upgrade: Target is docker.io/ceph/ceph:v15.2.7 with id 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959 2020-12-01T16:47:35.546444-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mgr daemons... 2020-12-01T16:47:35.547185-0700 mgr.aladdin.liknom [INF] Upgrade: Need to upgrade myself (mgr.aladdin.liknom) 2020-12-01T16:47:37.506337-0700 mgr.aladdin.liknom [INF] Upgrade: Pulling docker.io/ceph/ceph:v15.2.7 on ether 2020-12-01T16:47:40.770290-0700 mgr.aladdin.liknom [INF] Upgrade: image docker.io/ceph/ceph:v15.2.7 pull on ether got new image 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e (not 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959), restarting 2020-12-01T16:47:41.172402-0700 mgr.aladdin.liknom [INF] Upgrade: Target is docker.io/ceph/ceph:v15.2.7 with id 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e 2020-12-01T16:47:41.226550-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mgr daemons... 2020-12-01T16:47:41.230932-0700 mgr.aladdin.liknom [INF] Upgrade: All mgr daemons are up to date. 2020-12-01T16:47:41.231887-0700 mgr.aladdin.liknom [INF] Upgrade: Checking mon daemons... 2020-12-01T16:47:43.179844-0700 mgr.aladdin.liknom [INF] Upgrade: All mon daemons are up to date. 2020-12-01T16:47:43.180305-0700 mgr.aladdin.liknom [INF] Upgrade: Checking crash daemons... 2020-12-01T16:47:43.187481-0700 mgr.aladdin.liknom [INF] Upgrade: Setting container_image for all crash... 2020-12-01T16:47:43.191821-0700 mgr.aladdin.liknom [INF] Upgrade: All crash daemons are up to date. 2020-12-01T16:47:43.192290-0700 mgr.aladdin.liknom [INF] Upgrade: Checking osd daemons... 2020-12-01T16:47:45.692126-0700 mgr.aladdin.liknom [INF] Upgrade: Pulling docker.io/ceph/ceph:v15.2.7 on mandalaybay 2020-12-01T16:47:50.679789-0700 mgr.aladdin.liknom [INF] Upgrade: image docker.io/ceph/ceph:v15.2.7 pull on mandalaybay got new image 9a0677fecc08d155a8e643b37c6e97d45c04747d9cb9455cafe0a7590d00b959 (not 2bc420ddb175bd1cf9031387948a8812d1bda9ef1180e429b4704e3c06bb943e), restarting The machines 'ether' and 'aladdin' are x86_64 machines, but 'mandalaybay' is a raspberry pi 4 (arm64). Is there a way to bypass this check to allow me to finish upgrading the cluster? Thanks, Bryan

2 1

Determine effective min_alloc_size for a specific OSD
by 胡玮文 02 Dec '20

02 Dec '20

Hi all, I’ve read from this mail list that too high bluestore_min_alloc_size will result in too much space wasted if I have many small objects. But too low bluestore_min_alloc_size will reduce performance. I’ve also read that this config can’t be changed after OSD creation. Now I want to tune this config myself. I change bluestore_min_alloc_size_hdd from default 64k to 32k, then deployed some more OSDs with cephadm. I assume old OSD will continue to use the old setting, and new OSDs will use the new setting. But how can I verify that? To be specific, how can I query the effective min_alloc_size for a specific OSD? I’ve tried to use “ceph daemon osd.X ...” but cannot find the appropriate command. I also searched the OSD logs. Thanks.

4 4

ceph in docker the log_file config is empty
by goodluck 02 Dec '20

02 Dec '20

when I use kolla to deploy ceph in docker. I found there is no ceph logs output with osds and mons. But the mgr have the ceph logs output in the logfile. While I found the ceph log_file is empty even I set the log_file config in ceph.conf. ceph daemon /var/run/ceph/ceph-osd.0.asok config show|grep log_file "log_file": "", "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", "mon_cluster_log_file_level": "debug", cat /etc/ceph/ceph.conf [global] log file = /var/log/kolla/ceph/$cluster-$name.log log to syslog = false err to syslog = false log to stderr = false err to stderr = false fsid = 3722db7f-9831-473f-a591-a98bd76464d8 mon initial members = mon host = mon addr = auth cluster required = cephx auth service required = cephx auth client required = cephx #setuser match path = /var/lib/ceph/$type/$cluster-$id osd crush update on start = false osd objectstore = filestore bluestore = false [mon] mon compact on start = true mon cluster log file = /var/log/kolla/ceph/$cluster.log [ { "Id": "c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df", "Created": "2020-12-01T06:42:06.300762742Z", "Path": "dumb-init", "Args": [ "--single-child", "--", "kolla_start" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 51905, "ExitCode": 0, "Error": "", "StartedAt": "2020-12-01T06:48:05.787234081Z", "FinishedAt": "2020-12-01T06:48:05.485792802Z" }, "Image": "sha256:973a3e0d341100d5a53f64ef7f9b440aadfce01a9ca52e9637678361aa7fb1ba", "ResolvConfPath": "/var/lib/docker/containers/c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df/resolv.conf", "HostnamePath": "/var/lib/docker/containers/c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df/hostname", "HostsPath": "/var/lib/docker/containers/c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df/hosts", "LogPath": "/var/lib/docker/containers/c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df/c6d9fdcd6c974e2a95527542792ea44d59218373f1b4694af5d0c4626f1779df-json.log", "Name": "/ceph_osd_0", "RestartCount": 0, "Driver": "btrfs", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "unconfined", "ExecIDs": null, "HostConfig": { "Binds": [ "/etc/localtime:/etc/localtime:ro", "/etc/kolla/ceph-osd/:/var/lib/kolla/config_files/:ro", "/var/lib/ceph/osd/d4e7ed3e-f997-463b-9350-ffa9f5c1b429:/var/lib/ceph/osd/ceph-0:rw", "kolla_logs:/var/log/kolla/:rw", "/var/run/:/var/run/:rw", "/dev/:/dev/:rw" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": { "max-file": "5", "max-size": "50m" } }, "NetworkMode": "host", "PortBindings": {}, "RestartPolicy": { "Name": "unless-stopped", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Capabilities": null, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "private", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": true, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": [ "label=disable" ], "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": [], "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": null, "ReadonlyPaths": null }, "GraphDriver": { "Data": null, "Name": "btrfs" }, "Mounts": [ { "Type": "volume", "Name": "kolla_logs", "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "RW": true, "Propagation": "" }, { "Type": "bind", "Source": "/var/run", "Destination": "/var/run", "Mode": "rw", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/dev", "Destination": "/dev", "Mode": "rw", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/localtime", "Destination": "/etc/localtime", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/kolla/ceph-osd", "Destination": "/var/lib/kolla/config_files", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/ceph/osd/d4e7ed3e-f997-463b-9350-ffa9f5c1b429", "Destination": "/var/lib/ceph/osd/ceph-0", "Mode": "rw", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "compute-taishan-01", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "OSD_BS_FSUUID=d4e7ed3e-f997-463b-9350-ffa9f5c1b429", "OSD_STORETYPE=filestore", "OSD_ID=0", "KOLLA_SERVICE_NAME=ceph-osd-0", "KOLLA_INSTALL_TYPE=source", "KOLLA_INSTALL_METATYPE=mixed", "KOLLA_BASE_ARCH=aarch64", "DEBIAN_FRONTEND=noninteractive", "TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728", "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS", "JOURNAL_PARTITION=/dev/disk/by-partuuid/80664f67-6196-496c-81f2-02d974edbe52", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "LANG=en_US.UTF-8", "KOLLA_BASE_DISTRO=ubuntu", "KOLLA_DISTRO_PYTHON_VERSION=3.6", "PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id un)@$(hostname -s) $(pwd)]$ " ], "Cmd": [ "kolla_start" ], "Image": "10.12.254.254:4020/cszt-v4-anke/ubuntu-source-ceph-osd:4.0.1", "Volumes": { "/dev/": {}, "/etc/localtime": {}, "/var/lib/ceph/osd/ceph-0": {}, "/var/lib/kolla/config_files/": {}, "/var/log/kolla/": {}, "/var/run/": {} }, "WorkingDir": "", "Entrypoint": [ "dumb-init", "--single-child", "-" ], "OnBuild": null, "Labels": { "build-date": "20201019", "kolla_version": "0.1.0", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "ceph-osd" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "5bdcea4f0b2cc2f54e4d6c4abad0ca60ef080c74e7419ac75cde837025aedffe", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "/var/run/docker/netns/default", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "host": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "b5bfa3839097ea5932eaef6cd2b05ed2190eeec845f765f5ead543247787f860", "EndpointID": "67275cf88430263a8e49e5be9109a9ca2fedb3b99b348840072b54e52d2bdb57", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "", "DriverOpts": null }

2 2

OSD Metadata Imbalance
by Paul Kramme 01 Dec '20

01 Dec '20

Hello, my cluster is currently showing a metadata imbalance. Normally, all OSDs have around 23GB metadata (META column), but 4 OSDs out of 56 have 34 GB metadata. Compacting reduces the data for some OSDs, but not for others. OSDs where the compaction worked quickly grow to back to 34GB. Our cluster configuration: * 8 nodes, each with 6 HDDs OSDs and 1 SSD used for blockdb and WAL * k=4 m=2 EC * v14.2.14 Normal OSD: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 40 hdd 11.09470 1.00000 11 TiB 8.6 TiB 8.4 TiB 1.3 GiB 23 GiB 2.5 TiB 77.15 1.01 130 up Big OSD: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 11.09499 1.00000 11 TiB 8.6 TiB 8.4 TiB 1.8 GiB 30 GiB 2.5 TiB 77.59 1.02 130 up There are 56 OSDs in the cluster, 4 OSDs of which are bigger. These OSDs are all in different hosts. Why is that? Is that dangerous or could lead to problems such as performance degrades? Thanks, Paul

1 0

osd out cant' bring it back online
by Oliver Weinmann 01 Dec '20

01 Dec '20

Hi, I'm still evaluating ceph 15.2.5 in a lab so the problem is not really hurting me, but I want to understand it and hopefully fix it. It is a good practice. To test the resilience of the cluster I try to break it by doing all kinds of things. Today I powered off (clean shutdown) one osd node and powered it back on. Last time I tried this there was no problem getting it back online. After a few minutes the cluster health was back to ok. This time it stayed degraded forever. I checked and noticed that the service osd.0 on the osd node was failing. So i used google and there people recommended to simply delete the osd and re-create it. I tried it and still can't get the osd back in service. First I removed the osd: [root@gedasvl02 ~]# ceph osd out 0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 osd.0 is already out. [root@gedasvl02 ~]# ceph auth del 0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 Error EINVAL: bad entity name [root@gedasvl02 ~]# ceph auth del osd.0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 updated [root@gedasvl02 ~]# ceph osd rm 0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 removed osd.0 [root@gedasvl02 ~]# ceph osd tree INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.43658 root default -7 0.21829 host gedaopl01 2 ssd 0.21829 osd.2 up 1.00000 1.00000 -3 0 host gedaopl02 -5 0.21829 host gedaopl03 3 ssd 0.21829 osd.3 up 1.00000 1.00000 Looks ok it's gone... Then i zapped it: [root@gedasvl02 ~]# ceph orch device zap gedaopl02 /dev/sdb --force INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 INFO:cephadm:/usr/bin/podman:stderr WARNING: The same type, major and minor should not be used for multiple devices. INFO:cephadm:/usr/bin/podman:stderr --> Zapping: /dev/sdb INFO:cephadm:/usr/bin/podman:stderr --> Zapping lvm member /dev/sdb. lv_path is /dev/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a/osd-block-3a79800d-2a19-45d8-a850-82c6a8113323 INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a/osd-block-3a79800d-2a19-45d8-a850-82c6a8113323 bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/podman:stderr 10+0 records out INFO:cephadm:/usr/bin/podman:stderr 10485760 bytes (10 MB, 10 MiB) copied, 0.0314447 s, 333 MB/s INFO:cephadm:/usr/bin/podman:stderr stderr: INFO:cephadm:/usr/bin/podman:stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-3bf1bb28-0858-4464-a848-d7f56319b40a INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/sbin/vgremove -v -f ceph-3bf1bb28-0858-4464-a848-d7f56319b40a INFO:cephadm:/usr/bin/podman:stderr stderr: Removing ceph--3bf1bb28--0858--4464--a848--d7f56319b40a-osd--block--3a79800d--2a19--45d8--a850--82c6a8113323 (253:0) INFO:cephadm:/usr/bin/podman:stderr stderr: Archiving volume group "ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" metadata (seqno 5). INFO:cephadm:/usr/bin/podman:stderr stderr: Releasing logical volume "osd-block-3a79800d-2a19-45d8-a850-82c6a8113323" INFO:cephadm:/usr/bin/podman:stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" (seqno 6). INFO:cephadm:/usr/bin/podman:stderr stdout: Logical volume "osd-block-3a79800d-2a19-45d8-a850-82c6a8113323" successfully removed INFO:cephadm:/usr/bin/podman:stderr stderr: Removing physical volume "/dev/sdb" from volume group "ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" INFO:cephadm:/usr/bin/podman:stderr stdout: Volume group "ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" successfully removed INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/podman:stderr 10+0 records out INFO:cephadm:/usr/bin/podman:stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0355641 s, 295 MB/s INFO:cephadm:/usr/bin/podman:stderr --> Zapping successful for: <Raw Device: /dev/sdb> And re-added it: [root@gedasvl02 ~]# ceph orch daemon add osd gedaopl02:/dev/sdb INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 Created osd(s) 0 on host 'gedaopl02' But the osd is still out... [root@gedasvl02 ~]# ceph osd tree INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.43658 root default -7 0.21829 host gedaopl01 2 ssd 0.21829 osd.2 up 1.00000 1.00000 -3 0 host gedaopl02 -5 0.21829 host gedaopl03 3 ssd 0.21829 osd.3 up 1.00000 1.00000 0 0 osd.0 down 0 1.00000 Looking at the cluster log in the webui i see the following error: Failed to apply osd.dashboard-admin-1606745745154 spec DriveGroupSpec(name=dashboard-admin-1606745745154->placement=PlacementSpec(host_pattern='*'), service_id='dashboard-admin-1606745745154', service_type='osd', data_devices=DeviceSelection(size='223.6GB', rotational=False, all=False), osd_id_claims={}, unmanaged=False, filter_logic='AND', preview_only=False): No filters applied Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 2108, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/module.py", line 2005, in _apply_service self.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 43, in create_from_spec ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 127, in prepare_drivegroup drive_selection = DriveSelection(drive_group, inventory_for_host) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 32, in __init__ self._data = self.assign_devices(self.spec.data_devices) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 138, in assign_devices if not all(m.compare(disk) for m in FilterGenerator(device_filter)): File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 138, in <genexpr> if not all(m.compare(disk) for m in FilterGenerator(device_filter)): File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/matchers.py", line 410, in compare raise Exception("No filters applied") Exception: No filters applied I have another error "pgs undersized", maybe this is also causing trouble? [root@gedasvl02 ~]# ceph -s INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: d0920c36-2368-11eb-a5de-005056b703af health: HEALTH_WARN Degraded data redundancy: 13142/39426 objects degraded (33.333%), 176 pgs degraded, 225 pgs undersized services: mon: 1 daemons, quorum gedasvl02 (age 2w) mgr: gedasvl02.vqswxg(active, since 2w), standbys: gedaopl02.yrwzqh mds: cephfs:1 {0=cephfs.gedaopl01.zjuhem=up:active} 1 up:standby osd: 3 osds: 2 up (since 4d), 2 in (since 94m) task status: scrub status: mds.cephfs.gedaopl01.zjuhem: idle data: pools: 7 pools, 225 pgs objects: 13.14k objects, 77 GiB usage: 148 GiB used, 299 GiB / 447 GiB avail pgs: 13142/39426 objects degraded (33.333%) 176 active+undersized+degraded 49 active+undersized io: client: 0 B/s rd, 6.1 KiB/s wr, 0 op/s rd, 0 op/s wr Best Regards, Oliver

2 5

v15.2.7 Octopus released
by David Galloway 01 Dec '20

01 Dec '20

This is the 7th backport release in the Octopus series. This release fixes a serious bug in RGW that has been shown to cause data loss when a read of a large RGW object (i.e., one with at least one tail segment) takes longer than one half the time specified in the configuration option `rgw_gc_obj_min_wait`. The bug causes the tail segments of that read object to be added to the RGW garbage collection queue, which will in turn cause them to be deleted after a period of time. Changelog --------- * rgw: during GC defer, prevent new GC enqueue Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.7.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 88e41c6c49beb18add4fdb6b4326ca466d931db8

1 0