- ceph-users - lists.ceph.io

Re: MDS: obscene buffer_anon memory use when scanning lots of files
by John Madden 15 Apr '20

15 Apr '20

Upgraded to 14.2.7, doesn't appear to have affected the behavior. As requested: ~$ ceph tell mds.mds1 heap stats 2020-02-10 16:52:44.313 7fbda2cae700 0 client.59208005 ms_handle_reset on v2:x.x.x.x:6800/3372494505 2020-02-10 16:52:44.337 7fbda3cb0700 0 client.59249562 ms_handle_reset on v2:x.x.x.x:6800/3372494505 mds.mds1 tcmalloc heap stats:------------------------------------------------ MALLOC: 50000388656 (47684.1 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 174879528 ( 166.8 MiB) Bytes in central cache freelist MALLOC: + 14511680 ( 13.8 MiB) Bytes in transfer cache freelist MALLOC: + 14089320 ( 13.4 MiB) Bytes in thread cache freelists MALLOC: + 90534048 ( 86.3 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 50294403232 (47964.5 MiB) Actual memory used (physical + swap) MALLOC: + 50987008 ( 48.6 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 50345390240 (48013.1 MiB) Virtual address space used MALLOC: MALLOC: 260018 Spans in use MALLOC: 20 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. ~$ ceph tell mds.mds1 heap release 2020-02-10 16:52:47.205 7f037eff5700 0 client.59249625 ms_handle_reset on v2:x.x.x.x:6800/3372494505 2020-02-10 16:52:47.237 7f037fff7700 0 client.59249634 ms_handle_reset on v2:x.x.x.x:6800/3372494505 mds.mds1 releasing free RAM back to system. The pools over 15 minutes or so: ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 2045, "bytes": 3069493686 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 2445, "bytes": 3111162538 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 7850, "bytes": 7658678767 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 12274, "bytes": 11436728978 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 13747, "bytes": 11539478519 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 14615, "bytes": 13859676992 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 23267, "bytes": 22290063830 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 44944, "bytes": 40726959425 } And one about a minute after the heap release showing continued growth: ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 50694, "bytes": 47343942094 } This is on a single active MDS with 2 standbys, scan for about a million files with about 20 parallel threads on two clients, open and read each if it exists. On Wed, Jan 22, 2020 at 8:25 AM John Madden <jmadden.com(a)gmail.com> wrote: > > > Couldn't John confirm that this is the issue by checking the heap stats and triggering the release via > > > > ceph tell mds.mds1 heap stats > > ceph tell mds.mds1 heap release > > > > (this would be much less disruptive than restarting the MDS) > > That was my first thought as well, but `release` doesn't appear to do > anything in this case. > > John

3 8

telemetry.ceph.com certificate expired
by Eneko Lacunza 15 Apr '20

15 Apr '20

Hi all, We're receiving a certificate error for telemetry module: Module 'telemetry' has failed: HTTPSConnectionPool(host='telemetry.ceph.com', port=443): Max retries exceeded with url: /report (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)); Seems certificate expired yesterday (14th april). Cheers Eneko -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es

2 1

v14.2.9 Nautilus released
by Abhishek 15 Apr '20

15 Apr '20

This is the ninth bugfix release of Nautilus. This release fixes a couple of security issues in RGW & Messenger V2. We recommend all users to upgrade to this release. The official release blog entry is at https://ceph.io/releases/v14-2-9-nautilus-released/ Notable Changes: - CVE-2020-1759: Fixed nonce reuse in msgr V2 secure mode - CVE-2020-1760: Fixed XSS due to RGW GetObject header-splitting Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.9.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 581f22da52345dba46ee232b73b990f06029a2a0 Best, Abhishek

1 0

radosgw garbage collection seems stuck and mannual gc process didn't work
by 346415320＠qq.com 15 Apr '20

15 Apr '20

Ceph Version : Mimic 13.2.4 The cluster has been running steadily for more than a year, recently I found cluster usage grows faster than usual .And we figured out the problem is garbage collection. 'radosgw-admin gc list ' has millions of objects to gc. the earliest tag time is 2019-09 , but 99% of them are from 2020-03 to now `ceph df` GLOBAL: SIZE AVAIL RAW USED %RAW USED 1.7 PiB 1.1 PiB 602 TiB 35.22 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS .rgw.root 10 1.2 KiB 0 421 TiB 4 default.rgw.control 11 0 B 0 421 TiB 8 default.rgw.data.root 12 0 B 0 421 TiB 0 default.rgw.gc 13 0 B 0 421 TiB 0 default.rgw.log 14 4.8 GiB 0 421 TiB 6414 default.rgw.intent-log 15 0 B 0 421 TiB 0 default.rgw.meta 16 110 KiB 0 421 TiB 463 default.rgw.usage 17 0 B 0 421 TiB 0 default.rgw.users.keys 18 0 B 0 421 TiB 0 default.rgw.users.email 19 0 B 0 421 TiB 0 default.rgw.users.swift 20 0 B 0 421 TiB 0 default.rgw.users.uid 21 0 B 0 421 TiB 0 default.rgw.buckets.extra 22 0 B 0 421 TiB 0 default.rgw.buckets.index 23 0 B 0 421 TiB 118720 default.rgw.buckets.data 24 263 TiB 38.41 421 TiB 138902771 default.rgw.buckets.non-ec 25 0 B 0 421 TiB 16678 however we counted each bucket usage by ' radosgw-admin bucket stats ' ,it should cost only 160TiB , about 80TiB are in GC list former gc config setting before we find gc problem: rgw_gc_max_objs = 32 rgw_gc_obj_min_wait = 3600 rgw_gc_processor_period = 3600 rgw_gc_processor_max_time = 3600 yesterday we adjust our setting and restart rgw: rgw_gc_max_objs = 1024 rgw_gc_obj_min_wait = 300 rgw_gc_processor_period = 600 rgw_gc_processor_max_time = 600 rgw_gc_max_concurrent_io = 40 rgw_gc_max_trim_chunk = 1024 today we use ' rados -p default.rgw.log listomapkeys gc.$i --cluster ceph -N gc | wc -l ' (i from 0 to 1023) well , only gc.0 to gc.511 has data here are some result sorted -time 14:43 result: …… 36 gc_202004111443/gc.502.tag 38 gc_202004111443/gc.501.tag 40 gc_202004111443/gc.136.tag 46 gc_202004111443/gc.511.tag 212 gc_202004111443/gc.9.tag 218 gc_202004111443/gc.24.tag 21976 gc_202004111443/gc.13.tag 42956 gc_202004111443/gc.26.tag 71772 gc_202004111443/gc.25.tag 85766 gc_202004111443/gc.6.tag 104504 gc_202004111443/gc.7.tag 105444 gc_202004111443/gc.10.tag 106114 gc_202004111443/gc.3.tag 126860 gc_202004111443/gc.31.tag 127352 gc_202004111443/gc.23.tag 147942 gc_202004111443/gc.27.tag 148046 gc_202004111443/gc.15.tag 167116 gc_202004111443/gc.28.tag 167932 gc_202004111443/gc.21.tag 187986 gc_202004111443/gc.5.tag 188312 gc_202004111443/gc.22.tag 209084 gc_202004111443/gc.30.tag 209152 gc_202004111443/gc.18.tag 209702 gc_202004111443/gc.19.tag 231100 gc_202004111443/gc.8.tag 249622 gc_202004111443/gc.14.tag 251092 gc_202004111443/gc.2.tag 251366 gc_202004111443/gc.12.tag 251802 gc_202004111443/gc.0.tag 252158 gc_202004111443/gc.11.tag 272114 gc_202004111443/gc.1.tag 291518 gc_202004111443/gc.20.tag 293646 gc_202004111443/gc.16.tag 312998 gc_202004111443/gc.17.tag 352984 gc_202004111443/gc.29.tag 488232 gc_202004111443/gc.4.tag 5935806 total -time 16:53 result: …… 28 gc_202004111653/gc.324.tag 28 gc_202004111653/gc.414.tag 30 gc_202004111653/gc.350.tag 30 gc_202004111653/gc.456.tag 204 gc_202004111653/gc.9.tag 208 gc_202004111653/gc.24.tag 21986 gc_202004111653/gc.13.tag 42964 gc_202004111653/gc.26.tag 71780 gc_202004111653/gc.25.tag 85778 gc_202004111653/gc.6.tag 104512 gc_202004111653/gc.7.tag 105452 gc_202004111653/gc.10.tag 106122 gc_202004111653/gc.3.tag 126866 gc_202004111653/gc.31.tag 127372 gc_202004111653/gc.23.tag 147944 gc_202004111653/gc.27.tag 148058 gc_202004111653/gc.15.tag 167124 gc_202004111653/gc.28.tag 167936 gc_202004111653/gc.21.tag 187992 gc_202004111653/gc.5.tag 188320 gc_202004111653/gc.22.tag 209090 gc_202004111653/gc.30.tag 209170 gc_202004111653/gc.18.tag 209704 gc_202004111653/gc.19.tag 231108 gc_202004111653/gc.8.tag 249632 gc_202004111653/gc.14.tag 251096 gc_202004111653/gc.2.tag 251376 gc_202004111653/gc.12.tag 251806 gc_202004111653/gc.0.tag 252170 gc_202004111653/gc.11.tag 272118 gc_202004111653/gc.1.tag 291526 gc_202004111653/gc.20.tag 293656 gc_202004111653/gc.16.tag 313000 gc_202004111653/gc.17.tag 352988 gc_202004111653/gc.29.tag 488238 gc_202004111653/gc.4.tag 5932714 total -time 18:55 result: …… 22 gc_202004111855/gc.187.tag 22 gc_202004111855/gc.331.tag 24 gc_202004111855/gc.163.tag 24 gc_202004111855/gc.83.tag 194 gc_202004111855/gc.9.tag 208 gc_202004111855/gc.24.tag 21994 gc_202004111855/gc.13.tag 42966 gc_202004111855/gc.26.tag 71796 gc_202004111855/gc.25.tag 85788 gc_202004111855/gc.6.tag 104522 gc_202004111855/gc.7.tag 105458 gc_202004111855/gc.10.tag 106126 gc_202004111855/gc.3.tag 126872 gc_202004111855/gc.31.tag 127378 gc_202004111855/gc.23.tag 147954 gc_202004111855/gc.27.tag 148066 gc_202004111855/gc.15.tag 167136 gc_202004111855/gc.28.tag 167940 gc_202004111855/gc.21.tag 187996 gc_202004111855/gc.5.tag 188326 gc_202004111855/gc.22.tag 209092 gc_202004111855/gc.30.tag 209178 gc_202004111855/gc.18.tag 209720 gc_202004111855/gc.19.tag 231126 gc_202004111855/gc.8.tag 249646 gc_202004111855/gc.14.tag 251106 gc_202004111855/gc.2.tag 251378 gc_202004111855/gc.12.tag 251816 gc_202004111855/gc.0.tag 252176 gc_202004111855/gc.11.tag 272134 gc_202004111855/gc.1.tag 291528 gc_202004111855/gc.20.tag 293662 gc_202004111855/gc.16.tag 313010 gc_202004111855/gc.17.tag 352996 gc_202004111855/gc.29.tag 488250 gc_202004111855/gc.4.tag 5931710 total -time 22:54 result: …… 32 gc_202004112254/gc.191.tag 32 gc_202004112254/gc.196.tag 32 gc_202004112254/gc.486.tag 34 gc_202004112254/gc.382.tag 208 gc_202004112254/gc.9.tag 210 gc_202004112254/gc.24.tag 22012 gc_202004112254/gc.13.tag 43008 gc_202004112254/gc.26.tag 71828 gc_202004112254/gc.25.tag 85814 gc_202004112254/gc.6.tag 104544 gc_202004112254/gc.7.tag 105468 gc_202004112254/gc.10.tag 106146 gc_202004112254/gc.3.tag 126882 gc_202004112254/gc.31.tag 127396 gc_202004112254/gc.23.tag 147980 gc_202004112254/gc.27.tag 148088 gc_202004112254/gc.15.tag 167146 gc_202004112254/gc.28.tag 167960 gc_202004112254/gc.21.tag 188006 gc_202004112254/gc.5.tag 188352 gc_202004112254/gc.22.tag 209114 gc_202004112254/gc.30.tag 209204 gc_202004112254/gc.18.tag 209758 gc_202004112254/gc.19.tag 231148 gc_202004112254/gc.8.tag 249668 gc_202004112254/gc.14.tag 251126 gc_202004112254/gc.2.tag 251406 gc_202004112254/gc.12.tag 251850 gc_202004112254/gc.0.tag 252186 gc_202004112254/gc.11.tag 272154 gc_202004112254/gc.1.tag 291552 gc_202004112254/gc.20.tag 293684 gc_202004112254/gc.16.tag 313038 gc_202004112254/gc.17.tag 353014 gc_202004112254/gc.29.tag 488274 gc_202004112254/gc.4.tag 5935738 total it seems gc.0-gc.31 keeps on increasing. but gc.32 -gc.511 works fine. maybe some objects in gc.0-gc.31 stuck gc process, so gc thread on gc.0 - gc.31 cannot proceed after setting debug_rgw=20 , we find only a few process period (every 10 minutes as set) can log like '2020-04-11 23:02:18.724 7f7d1e61b700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.985987.4__multipart_playback/2009550000004491/6b97a8a61424129d996d7a13769e8c66.ts.2~KnoW62bPMfmWBWtH0THWWzV3urFJglW.1' most of process period only shows: '2020-04-11 23:32:14.814 7f7d1de1a700 2 object expiration: start 2020-04-11 23:32:14.814 7f7d1de1a700 20 proceeding shard = obj_delete_at_hint.0000000000 2020-04-11 23:32:14.815 7f7d1de1a700 20 proceeding shard = obj_delete_at_hint.0000000001 …… 2020-04-11 23:22:14.811 7f7d1de1a700 20 proceeding shard = obj_delete_at_hint.0000000126 2020-04-11 23:22:14.813 7f7d1de1a700 2 object expiration: stop 2020-04-11 23:22:21.364 7f7d1e61b700 10 RGWGC::process() failed to acquire lock on gc.4 2020-04-11 23:22:31.074 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:22:53.074 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:23:15.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:23:37.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:23:59.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:24:21.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:24:43.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:25:05.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:25:27.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:25:49.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:26:11.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:26:33.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:26:55.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:27:17.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:27:39.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:28:01.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:28:23.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:28:45.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:29:07.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:29:29.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:29:51.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:30:13.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:30:35.077 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:30:57.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:31:19.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:31:41.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:32:03.077 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:32:14.814 7f7d1de1a700 2 object expiration: start 2020-04-11 23:32:14.814 7f7d1de1a700 20 proceeding shard = obj_delete_at_hint.0000000000 2020-04-11 23:32:14.815 7f7d1de1a700 20 proceeding shard = obj_delete_at_hint.0000000001 ……' and sometimes gc removes a few objects like: '2020-04-10 12:01:45.141 7f2acdd38700 2 object expiration: stop 2020-04-10 12:01:45.886 7f2ace539700 10 RGWGC::process() failed to acquire lock on gc.8 2020-04-10 12:01:45.911 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.985987.4__multipart_playback/2009510000009744/8d5cb6722d478fd2395af1bc25b73880.ts.2~rDnSwcUmF-MKFwneDyMQRiI4tXuw6K7.1 2020-04-10 12:01:45.911 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.985987.4__multipart_playback/2009510000009744/8d5cb6722d478fd2395af1bc25b73880.ts.2~astXMDje7aBnOLzHG7qIdofTrpHTsUB.1 2020-04-10 12:01:45.911 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.985987.4__multipart_playback/2009550000003154/39abc9fe1bf0f289c6e02132cbc2d96f.ts.2~YW4Uv7qhbKqZoru7wIGvJXZAuytD7s6.1 2020-04-10 12:01:45.911 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.985987.4__multipart_playback/2009550000004519/296b0c23da59ff9eba8a522fc5febd11.ts.2~bL7CZCB-mKP7w2RHFomEsgBzAH1rZt9.1 2020-04-10 12:01:45.912 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.2190746.7__shadow_.VYxHLF5VGGXMFsYy-CNiMOfDVI-NTfQ_1 2020-04-10 12:01:45.912 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.2190746.7__shadow_.VYxHLF5VGGXMFsYy-CNiMOfDVI-NTfQ_2 2020-04-10 12:01:45.912 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.2190746.7__shadow_.VYxHLF5VGGXMFsYy-CNiMOfDVI-NTfQ_3 2020-04-10 12:01:45.912 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.2190746.7__shadow_.VYxHLF5VGGXMFsYy-CNiMOfDVI-NTfQ_4 2020-04-10 12:01:45.912 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.2190746.7__shadow_.VYxHLF5VGGXMFsYy-CNiMOfDVI-NTfQ_5 2020-04-10 12:01:45.914 7f2ace539700 10 RGWGC::process() failed to acquire lock on gc.10 2020-04-10 12:01:45.915 7f2ace539700 10 RGWGC::process() failed to acquire lock on gc.11 2020-04-10 12:02:02.638 7f2acf53b700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-10 12:02:24.638 7f2acf53b700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-10 12:02:46.638 7f2acf53b700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-10 12:03:08.638 7f2acf53b700 2 RGWDataChangesLog::ChangesRenewThread: start ……' we scan those logs on 5 gc node( no other workload like object read/write or lifecycle, only for gc). all is the same. the "removing xxx" log appears only a few times within a few seconds in whole day's work so we suppose some objs in gc.0-gc.31 stuck gc process mannual ' radosgw-admin gc process ' not work , log is the same as above Has anyone encountered the same problem? what can i do to make gc thread proceed on gc.0-gc.31 or get everything back to normal

3 4

CephFS and Samba/CIFS permissions (xattr)
by Victor Rodriguez 15 Apr '20

15 Apr '20

Hello, I have a CephFS running on v14.2.8 correctly. I also have a VM which runs Samba as AD controller and fileserver (Zentyal). My plan was to mount a CephFS path on that VM and make Samba share those files to a Windows network. But I cant make the shares work as Samba is asking to mount the CephFS resource with "user_xattr" mount option, which the kernel driver doesnt support. Besides that, looks like I cant set CIFS permissions on directories/files in CephFS because extended attributes are not supported. Is that the expected behavior or am I overlooking something? Looking for information about this issue, I have read about ceph-vfs and Samba CTDB. Are those options really necessary to get Samba over CephFS with CIFS/AD domain permissions? Thanks a lot. Victor.

1 0

Announcing go-ceph v0.3.0
by John Mulligan 15 Apr '20

15 Apr '20

I'm happy to announce the another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.3.0 The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild and the ceph-csi project is starting to make use of this library. Specific questions, comments, bugs etc are best directed at our github issues tracker. --- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

Fwd: Question on rbd maps
by Void Star Nill 15 Apr '20

15 Apr '20

Hello, Is there a way to find out all the clients where the volumes are mapped from a central point? We have a large fleet of machines that use ceph rbd volumes. For some maintenance purposes, we need to find out if a volume is mapped anywhere before acting on it. Right now we go and query each client machines with `rbd showmapped` commands. Is there a variant of this CLI that lists all mappings on a single node (ceph mon for instance)? Thanks, Shridhar

3 6

Fwd: question on rbd locks
by Void Star Nill 15 Apr '20

15 Apr '20

Hello All, Is there a way to specify that a lock (shared or exclusive) on an rbd volume be released if the client machine becomes unreachable or irresponsive? In one of our clusters, we use rbd locks on volumes to make sure provide a kind of shared or exclusive access - to make sure there are no writers when someone is reading and there are no readers when someone is writing. However, we often run into issues when one of the machines gets into kernel panic or something and the whole pipeline gets stalled. So is there a way to tell ceph to release the lock if the client becomes unavailable? Thanks, Shridhar

4 7

ceph-mgr with large connections in CLOSE_WAIT state
by Void Star Nill 15 Apr '20

15 Apr '20

Hi, I am seeing a large number of connections from ceph-mgr are stuck in CLOSE_WAIT state with data stuck in the receive queue. Looks like ceph-mgr process is not reading the data completely off the socket buffers and terminating the connections properly. I also notice that the access to dashboard plugin stops working after couple of days and I need to restart ceph-mgr for it to start working again. I haven't noticed any impact of volume I/O operations though. We are running nautilus version (14.2.4-1xenial) Is this a known issue? $ sudo netstat -anp|grep CLOSE_WAIT | wc -l 906 $ sudo netstat -anp|grep CLOSE_WAIT tcp6 77 0 x.x.x.x:8443 x.x.12.195:48624 CLOSE_WAIT 161716/ceph-mgr tcp6 210 0 x.x.x.x:8443 x.x.12.209:59220 CLOSE_WAIT 161716/ceph-mgr tcp6 2443 0 x.x.x.x:8443 x.x.0.103:65276 CLOSE_WAIT 161716/ceph-mgr tcp6 159 0 x.x.x.x:8443 x.x.12.202:60150 CLOSE_WAIT 161716/ceph-mgr tcp6 1 0 x.x.x.x:8443 x.x.12.161:51472 CLOSE_WAIT 161716/ceph-mgr tcp6 283 0 x.x.x.x:8443 x.x.12.202:44900 CLOSE_WAIT 161716/ceph-mgr tcp6 32 0 x.x.x.x:8443 x.x.12.207:58202 CLOSE_WAIT 161716/ceph-mgr tcp6 32 0 x.x.x.x:8443 x.x.12.200:52162 CLOSE_WAIT 161716/ceph-mgr tcp6 32 0 x.x.x.x:8443 x.x.12.167:43148 CLOSE_WAIT 161716/ceph-mgr tcp6 305 0 x.x.x.x:8443 x.x.12.167:60102 CLOSE_WAIT 161716/ceph-mgr tcp6 79 0 x.x.x.x:8443 x.x.12.195:48504 CLOSE_WAIT 161716/ceph-mgr tcp6 69 0 x.x.x.x:8443 x.x.12.167:50362 CLOSE_WAIT 161716/ceph-mgr tcp6 38 0 x.x.x.x:8443 x.x.12.167:36840 CLOSE_WAIT 161716/ceph-mgr tcp6 76 0 x.x.x.x:8443 x.x.12.209:54166 CLOSE_WAIT 161716/ceph-mgr tcp6 68 0 x.x.x.x:8443 x.x.12.207:34312 CLOSE_WAIT 161716/ceph-mgr tcp6 209 0 x.x.x.x:8443 x.x.12.209:52642 CLOSE_WAIT 161716/ceph-mgr tcp6 220 0 x.x.x.x:8443 x.x.12.202:41156 CLOSE_WAIT 161716/ceph-mgr tcp6 202 0 x.x.x.x:8443 x.x.12.209:55448 CLOSE_WAIT 161716/ceph-mgr tcp6 193 0 x.x.x.x:8443 x.x.12.207:56296 CLOSE_WAIT 161716/ceph-mgr tcp6 672 0 x.x.x.x:8443 x.x.12.207:35516 CLOSE_WAIT 161716/ceph-mgr tcp6 1186 0 x.x.x.x:8443 x.x.28.4:54074 CLOSE_WAIT 161716/ceph-mgr tcp6 163 0 x.x.x.x:8443 x.x.12.209:54458 CLOSE_WAIT 161716/ceph-mgr tcp6 214 0 x.x.x.x:8443 x.x.12.167:54912 CLOSE_WAIT 161716/ceph-mgr tcp6 129 0 x.x.x.x:8443 x.x.12.200:42494 CLOSE_WAIT 161716/ceph-mgr tcp6 86 0 x.x.x.x:8443 x.x.12.195:58618 CLOSE_WAIT 161716/ceph-mgr tcp6 103 0 x.x.x.x:8443 x.x.12.209:45408 CLOSE_WAIT 161716/ceph-mgr tcp6 77 0 x.x.x.x:8443 x.x.12.202:60792 CLOSE_WAIT 161716/ceph-mgr tcp6 189 0 x.x.x.x:8443 x.x.12.161:58640 CLOSE_WAIT 161716/ceph-mgr tcp6 68 0 x.x.x.x:8443 x.x.12.167:42004 CLOSE_WAIT 161716/ceph-mgr tcp6 1 0 x.x.x.x:8443 x.x.12.195:36348 CLOSE_WAIT 161716/ceph-mgr Thanks, Shridhar

1 1

PGs unknown (osd down) after conversion to cephadm
by Dr. Marco Savoca 14 Apr '20

14 Apr '20

2 1