December 2020 - ceph-users

Re: Monitors not starting, getting "e3 handle_auth_request failed to assign global_id"

by Hoan Nguyen Van

I found a merge request, ceph mon has new optíon :mon_sync_max_payload_keys https://github.com/ceph/ceph/commit/d6037b7f484e13cfc9136e63e4cf7fac6ad6896… My value of options mon_sync_max_payload_size is 4096. If mon_sync_max_payload_keys options is default, ceph mon may be not sync because slow opt. What do you think?

3 years, 4 months

1
0
0 0

Slow Replication on Campus

by Vikas Rana

Hi Friends, We have 2 Ceph clusters on campus and we setup the second cluster as the DR solution. The images on the DR side are always behind the master. Ceph Version : 12.2.11 VMWARE_LUN0: global_id: 23460954-6986-4961-9579-0f2a1e58e2b2 state: up+replaying description: replaying, master_position=[object_number=2632711, tag_tid=24, entry_tid=1967382595], mirror_position=[object_number=1452837, tag_tid=24, entry_tid=456440697], entries_behind_master=1510941898 last_update: 2020-11-30 14:13:38 VMWARE_LUN1: global_id: cb579579-13b0-4522-b65f-c64ec44cbfaf state: up+replaying description: replaying, master_position=[object_number=1883943, tag_tid=28, entry_tid=1028822927], mirror_position=[object_number=1359161, tag_tid=28, entry_tid=358296085], entries_behind_master=670526842 last_update: 2020-11-30 14:13:33 Any suggestion on tuning or any parameters we can set on RBD-mirror to speed up the replication. Both cluster have very little activity. Appreciate your help. Thanks, -Vikas

3 years, 4 months

2
1
0 0

iscsi and iser

by Marc Boisis

Hi, I would like to know if you support iser in gwcli like the traditional targetcli or if this is planned in a future version of ceph ? Thanks Marc

3 years, 4 months

2
1
0 0

Ceph 15.2.4 segfault, msgr-worker

by Ivan Kurnosov

Hi Team, this night I have caught the following segfault. Nothing else looks suspicious (but I'm a quite newbie in ceph management, so perhaps just don't know where to look at). I could not google any similar segfault from anybody else. Was it a known problem fixed in later versions? The cluster has been running for quite a while now (several months) and this has happened for the first time. ``` debug 0> 2020-12-02T10:31:09.295+0000 7f3e61943700 -1 *** Caught signal (Segmentation fault) ** in thread 7f3e61943700 thread_name:msgr-worker-1 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) 1: (()+0x12dd0) [0x7f3e65933dd0] 2: (std::_Rb_tree<boost::intrusive_ptr<AsyncConnection>, boost::intrusive_ptr<AsyncConnection>, std::_Identity<boost::intrusive_ptr<AsyncConnection> >, std::less<boost::intrusive_ptr<AsyncConnection> >, std::allocator<boost::intrusive_ptr<AsyncConnection> > >::find(boost::intrusive_ptr<AsyncConnection> const&) const+0x2c) [0x55cee407ca7c] 3: (AsyncConnection::_stop()+0xab) [0x55cee40767eb] 4: (ProtocolV2::stop()+0x8f) [0x55cee40a189f] 5: (ProtocolV2::_fault()+0x133) [0x55cee40a1b03] 6: (ProtocolV2::handle_read_frame_preamble_main(std::unique_ptr<ceph::buffer::v15_2_0::ptr_node, ceph::buffer::v15_2_0::ptr_node::disposer>&&, int)+0x551) [0x55cee40a63d1] 7: (ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x3c) [0x55cee40a273c] 8: (AsyncConnection::process()+0x8a9) [0x55cee4079ab9] 9: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xcb7) [0x55cee3eceb67] 10: (()+0xdb914c) [0x55cee3ed414c] 11: (()+0xc2b73) [0x7f3e64f83b73] 12: (()+0x82de) [0x7f3e659292de] 13: (clone()+0x43) [0x7f3e64660e83] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_rwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7f3e3d88b700 / osd_srv_heartbt 7f3e41893700 / tp_osd_tp 7f3e4589b700 / tp_osd_tp 7f3e4f8af700 / rocksdb:dump_st 7f3e52a8c700 / safe_timer 7f3e53a8e700 / ms_dispatch 7f3e566b7700 / bstore_mempool 7f3e5d0ca700 / safe_timer 7f3e61943700 / msgr-worker-1 7f3e62144700 / msgr-worker-0 max_recent 10000 max_new 1000 log_file /var/lib/ceph/crash/2020-12-02T10:31:09.301492Z_84e8430f-30fd-469f-8e22-c2e1ccc675da/log --- end dump of recent events --- reraise_fatal: default handler for signal 11 didn't terminate the process? ``` -- With best regards, Ivan Kurnosov

3 years, 4 months

2
1
0 0

Removing an applied service set

by Michael Wodniok

Hi, we created multiple CephFS, this invloved deploying mutliple mds-services using `ceph orch apply mds [...]`. Worked like a charm. Now the filesystem has been removed and the leftovers of the filesystem should also be removed, but I can't delete the services as cephadm/orchestration module is recreating them. What is the "official" way to delete this applied service set? Setting placement size to 0 is not possible in Ceph 15. Kind regards, Michael

3 years, 4 months

2
2
0 0

Re: pool nearfull, 300GB rbd image occupies 11TB!

by mk

In fact journaling was enabled, is it enough to disable feature and pool shrinks automatically again? Or still any additional actions are required? — Max > On 13. Dec 2020, at 15:53, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > rbd status > rbd info > > If the ‘journaling’ flag is enabled, use ‘rbd feature’ to remove it from the image. > > > >> On Dec 13, 2020, at 6:22 AM, mk <mk(a)pop.de> wrote: >> >> Yes, few months ago I had enabled mirroring for few weeks and disabled again. Is there any additional actions has to be taken regarding journalling also ?? >> fyi. I also have copied rbd image into one newly created pool but after few weeks new pool grew up again to 11TB which is current state >> -- >> BR >> Max >> >>> On 13. Dec 2020, at 14:34, Jason Dillaman <jdillama(a)redhat.com> wrote: >>> >>>> On Sun, Dec 13, 2020 at 6:03 AM mk <mk(a)pop.de> wrote: >>>> >>>> rados ls -p ssdshop >>>> outputs 20MB of lines without any bench prefix >>>> ... >>>> rbd_data.d4993cc3c89825.00000000000074ec >>>> rbd_data.d4993cc3c89825.0000000000001634 >>>> journal_data.83.d4993cc3c89825.333485 >>>> journal_data.83.d4993cc3c89825.380648 >>>> journal_data.83.d4993cc3c89825.503838 >>>> ... >>> >>> If you have journal objects indexed at 333485 and 503838, that's >>> nearly 2TiBs of data right there. Sounds like you enabled journaling >>> and mirroring but perhaps never turned it off when you stopped using >>> it? >>> >>>>> On 13. Dec 2020, at 11:05, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: >>>>> >>>>> Any chance you might have orphaned `rados bench` objects ? This happens more than one might think. >>>>> >>>>> `rados ls > /tmp/out` >>>>> >>>>> Inspect the result. You should see a few administrative objects, some header and data objects for the RBD volume. If you see a zillion with names like `bench*` there’s your culprit. Those can be cleaned up. >>>>> >>>>> >>>>> >>>>>> On Dec 12, 2020, at 11:42 PM, mk <mk(a)pop.de> wrote: >>>>>> >>>>>> Hi folks, >>>>>> my cluster shows strange behavior, the only ssd pool on cluster with repsize 3 and pg/pgp size 512 >>>>>> which contains 300GB rbd image and only one snapshot occupies 11TB space! >>>>>> >>>>>> I have tried objectmap check / rebuild, fstrim etc. which couldn’t solve that problem, any help would be appreciated. >>>>>> >>>>>> >>>>>> ceph version 14.2.7 nautilus (stable) >>>>>> >>>>>> >>>>>> ceph df >>>>>> ------- >>>>>> RAW STORAGE: >>>>>> CLASS SIZE AVAIL USED RAW USED %RAW USED >>>>>> hdd 107 TiB 68 TiB 39 TiB 39 TiB 36.45 >>>>>> ssd 21 TiB 11 TiB 11 TiB 11 TiB 50.78 >>>>>> TOTAL 128 TiB 78 TiB 50 TiB 50 TiB 38.84 >>>>>> >>>>>> POOLS: >>>>>> POOL ID STORED OBJECTS USED %USED MAX AVAIL >>>>>> ssdshop 83 3.5 TiB 517.72k 11 TiB 96.70 124 GiB >>>>>> >>>>>> >>>>>> rados df >>>>>> -------- >>>>>> POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR >>>>>> ssdshop 11 TiB 537040 28316 1611120 0 0 0 11482773 15 GiB 44189589 854 GiB 0 B 0 B >>>>>> >>>>>> >>>>>> >>>>>> rbd du -p ssdshop >>>>>> ----------------- >>>>>> NAME PROVISIONED USED >>>>>> shp-de-300gb.rbd@snap_2020-12-12_20:30:00 300 GiB 289 GiB >>>>>> shp-de-300gb.rbd 300 GiB 109 GiB >>>>>> <TOTAL> 300 GiB 398 GiB >>>>>> >>>>>> >>>>>> crush_rule >>>>>> ----------- >>>>>> rule ssd { >>>>>> id 3 >>>>>> type replicated >>>>>> min_size 1 >>>>>> max_size 10 >>>>>> step take dc1 class ssd >>>>>> step chooseleaf firstn 2 type rack >>>>>> step emit >>>>>> step take dc2 class ssd >>>>>> step chooseleaf firstn -1 type rack >>>>>> step emit >>>>>> } >>>>>> >>>>>> BR >>>>>> Max >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> >>> -- >>> Jason >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

3 years, 4 months

2
2
0 0

pool nearfull, 300GB rbd image occupies 11TB!

by mk

Hi folks, my cluster shows strange behavior, the only ssd pool on cluster with repsize 3 and pg/pgp size 512 which contains 300GB rbd image and only one snapshot occupies 11TB space! I have tried objectmap check / rebuild, fstrim etc. which couldn’t solve that problem, any help would be appreciated. ceph version 14.2.7 nautilus (stable) ceph df ------- RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 107 TiB 68 TiB 39 TiB 39 TiB 36.45 ssd 21 TiB 11 TiB 11 TiB 11 TiB 50.78 TOTAL 128 TiB 78 TiB 50 TiB 50 TiB 38.84 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL ssdshop 83 3.5 TiB 517.72k 11 TiB 96.70 124 GiB rados df -------- POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR ssdshop 11 TiB 537040 28316 1611120 0 0 0 11482773 15 GiB 44189589 854 GiB 0 B 0 B rbd du -p ssdshop ----------------- NAME PROVISIONED USED shp-de-300gb.rbd@snap_2020-12-12_20:30:00 300 GiB 289 GiB shp-de-300gb.rbd 300 GiB 109 GiB <TOTAL> 300 GiB 398 GiB crush_rule ----------- rule ssd { id 3 type replicated min_size 1 max_size 10 step take dc1 class ssd step chooseleaf firstn 2 type rack step emit step take dc2 class ssd step chooseleaf firstn -1 type rack step emit } BR Max

3 years, 4 months

3
4
0 0

pool nearfull, 300GB rbd image occupies 11TB!

by mk

Hi folks, my cluster shows strange behavior, the only ssd pool on cluster with repsize 3 and pg/pgp size 512 which contains 300GB rbd image and only one snapshot occupies 11TB space! I have tried objectmap check / rebuild, fstrim etc. which couldn’t solve that problem, any help would be appreciated. ceph version 14.2.7 nautilus (stable) ceph df ------- RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 107 TiB 68 TiB 39 TiB 39 TiB 36.45 ssd 21 TiB 11 TiB 11 TiB 11 TiB 50.78 TOTAL 128 TiB 78 TiB 50 TiB 50 TiB 38.84 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL ssdshop 83 3.5 TiB 517.72k 11 TiB 96.70 124 GiB rados df -------- POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR ssdshop 11 TiB 537040 28316 1611120 0 0 0 11482773 15 GiB 44189589 854 GiB 0 B 0 B rbd du -p ssdshop ----------------- NAME PROVISIONED USED shp-de-300gb.rbd@snap_2020-12-12_20:30:00 300 GiB 289 GiB shp-de-300gb.rbd 300 GiB 109 GiB <TOTAL> 300 GiB 398 GiB crush_rule ----------- rule ssd { id 3 type replicated min_size 1 max_size 10 step take dc1 class ssd step chooseleaf firstn 2 type rack step emit step take dc2 class ssd step chooseleaf firstn -1 type rack step emit } BR Max

3 years, 4 months

1
0
0 0

Third nautilus OSD dead in 11 days - FAILED is_valid_io(off, len)

by Jonas Jelten

Hi! Yesterday a third OSD died with a failed assertion, and it can no longer boot. It's the third OSD within 11 days. There's already a tracker issue: https://tracker.ceph.com/issues/48276 2020-12-11 20:06:51.839 7fe2b5ffd700 -1 /build/ceph-14.2.13/src/os/bluestore/KernelDevice.cc: In function 'virtual int KernelDevice::aio_write(uint64_t, ceph::bufferlist&, IOContext*, bool, int)' thread 7fe2b5ffd /build/ceph-14.2.13/src/os/bluestore/KernelDevice.cc: 864: FAILED ceph_assert(is_valid_io(off, len)) ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55ff9fac9eea] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55ff9faca0c5] 3: (KernelDevice::aio_write(unsigned long, ceph::buffer::v14_2_0::list&, IOContext*, bool, int)+0x19e9) [0x55ffa01272a9] 4: (BlueStore::_deferred_submit_unlock(BlueStore::OpSequencer*)+0x621) [0x55ff9ffe87f1] 5: (BlueStore::deferred_try_submit()+0x3e0) [0x55ff9ffe9da0] 6: (BlueStore::_kv_finalize_thread()+0x6c6) [0x55ffa002fcc6] 7: (BlueStore::KVFinalizeThread::entry()+0xd) [0x55ffa006066d] 8: (()+0x76db) [0x7fe2c61456db] 9: (clone()+0x3f) [0x7fe2c4ee571f] When the OSD tries to boot (now with 14.2.15), it fails again: -4> 2020-12-11 20:07:05.160 7ff2d15e7c00 1 bluestore(/var/lib/ceph/osd/ceph-147) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2 -3> 2020-12-11 20:07:05.160 7ff2d15e7c00 1 bluestore(/var/lib/ceph/osd/ceph-147) _upgrade_super from 2, latest 2 -2> 2020-12-11 20:07:05.160 7ff2d15e7c00 1 bluestore(/var/lib/ceph/osd/ceph-147) _upgrade_super done -1> 2020-12-11 20:07:05.184 7ff2d15e7c00 -1 /build/ceph-14.2.15/src/os/bluestore/KernelDevice.cc: In function 'virtual int KernelDevice::aio_write(uint64_t, ceph::bufferlist&, IOContext*, bool, int)' thread 7ff2d15e7c00 time 2020-12-11 20:07:05.186142 /build/ceph-14.2.15/src/os/bluestore/KernelDevice.cc: 888: FAILED ceph_assert(is_valid_io(off, len)) ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x564ad1323fba] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x564ad1324195] 3: (KernelDevice::aio_write(unsigned long, ceph::buffer::v14_2_0::list&, IOContext*, bool, int)+0x16da) [0x564ad198154a] 4: (BlueStore::_deferred_submit_unlock(BlueStore::OpSequencer*)+0x621) [0x564ad1842911] 5: (BlueStore::deferred_try_submit()+0x3e0) [0x564ad1843ec0] 6: (BlueStore::_osr_drain_all()+0x1a5) [0x564ad1845255] 7: (BlueStore::_deferred_replay()+0x1c7) [0x564ad188b297] 8: (BlueStore::_mount(bool, bool)+0x83e) [0x564ad1892e1e] 9: (OSD::init()+0x3f3) [0x564ad13d3db3] 10: (main()+0x5214) [0x564ad132ccf4] 11: (__libc_start_main()+0xe7) [0x7ff2cde60bf7] 12: (_start()+0x2a) [0x564ad135e72a] Anyone else with these symptoms? Any ideas how we can track it down? -- Jonas

3 years, 4 months

2
2
0 0

Anonymous access to grafana

by Alessandro Piazza

Dear all, It seems that by default the grafana web page embedded inside the ceph dashboard is publicly available in read-only mode. More specifically the grafana configuration inside the docker running the grafana instance has the following configuration file (/usr/share/ceph/mgr/cephadm/templates/services/grafana/grafana.ini.j2) [auth.anonymous] enabled = true org_name = 'Main Org.' org_role = 'Viewer' Do you think that this might be a security concern? Is there a way to enforce authentication also for the read-only mode? I wasn't able to find any documentation on how to configure grafana. The only thing I found which might be related to this issue is the following: https://tracker.ceph.com/issues/45372. Regards, Alessandro Piazza

3 years, 4 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users December 2020