- ceph-users - lists.ceph.io

Experience with messenger v2 in Nautilus
by Stefan Kooman 03 Jan '20

03 Jan '20

Hi, I'm wondering how many of are using messenger v2 in Nautilus after upgrading from a previous release (Luminous / Mimic). Does it work for you? Or why did you not enable it (yet)? Thanks, Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

3 2

Mimic 13.2.8 deep scrub error: "size 333447168 > 134217728 is too large"
by Robert Sander 03 Jan '20

03 Jan '20

Hi, when Ceph is running a deep scrub it sometimes complains about an object being too large: Dez 17 07:11:44 al23 ceph-osd[3339]: 2019-12-17 07:11:44.066 7f8d38064700 0 log_channel(cluster) log [DBG] : 8.2d scrub starts Dez 17 07:11:44 al23 ceph-osd[3339]: 2019-12-17 07:11:44.090 7f8d38064700 -1 log_channel(cluster) log [ERR] : 8.2d soid 8:b431ae8d:::gparted-live-0.31.0-1-amd64.iso:head : size 333447168 > 134217728 is too large Dez 17 07:11:44 al23 ceph-osd[3339]: 2019-12-17 07:11:44.102 7f8d38064700 -1 log_channel(cluster) log [ERR] : 8.2d scrub 0 missing, 1 inconsistent objects All OSDs are BlueStore. What is happening here? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 2

Access ceph cluster health from REST API
by Vikram Giriraj 03 Jan '20

03 Jan '20

Hi, I am trying to get ceph cluster health and cluster status via REST api. My cluster is running the latest nautilus release(v14). I found that the "ceph-rest-api" is deprecated ( https://docs.ceph.com/docs/mimic/releases/luminous/ ) and now there is ceph RESTful module. But cluster health information is not available from the ceph Restful module. Could someone help me with this. Regards, -- Vikram Giriraj Software Engineer,

1 0

rgw multisite debugging
by Frank R 02 Jan '20

02 Jan '20

Hi all, This is a self inflicted issue but I am wondering if there is a way to recover. Setup: - RGW Multisite with 1 realm, 1 zonegroup and 2 zones. - Metadata and data replication enable (data bi-directional). - Only the master side is currently used by clients. - Ceph 12.2.12 My mistake was to delete all the logs in the rgw.log pool on the master end and some on the secondary side. Data replication is surprisingly working fine but metadata replication is not. When I run "metadata sync init" from the secondary side I see: ----- # radosgw-admin metadata sync init ERROR: sync.init_sync_status() returned ret=-5 2020-01-02 16:10:30.215774 7f26e9544dc0 0 meta sync: ERROR: failed to fetch mdlog info 2020-01-02 16:10:30.215797 7f26e9544dc0 -1 meta sync: ERROR: fail to fetch master log info (r=-5) ----- With rgw-debug=20 and debug-ms=1 I see: ----- received header:HTTP/1.1 500 Internal Server Error ----- On the master side I can run the following commands successfully: # radosgw-admin mdlog status # radosgw-admin mdlog list From the debug session on the master side I see: ----- cache get: name=default.rgw.log++meta.history : hit (requested=0x1, cached=0x17) failed to decode the mdlog history: buffer::end_of_buffer failed to read mdlog history: (5) Input/output error ----- I can view the meta.history log though: # radosgw-admin log show --object=meta.history { "bucket_id": "", "bucket_owner": "", "bucket": "", "log_entries": [ { "bucket": "", "time": "0.000000", "time_local": "0.000000", "remote_addr": "", "user": "", "operation": "", "uri": "", "http_status": "", "error_code": "", "bytes_sent": 7738151095376705906, "bytes_received": 4856413621961454149, "object_size": 7595168404649370735, "total_time": 0, "user_agent": "", "referrer": "" } ], "log_sum": { "bytes_sent": 7738151095376705906, "bytes_received": 4856413621961454149, "total_time": 0, "total_entries": 1 } } ----- Any ideas on how to further troubleshoot the issue? thx Frank

1 0

Mimic downgrade (13.2.8 -> 13.2.6) failed assert in combination with bitmap allocator
by Stefan Kooman 27 Dec '19

27 Dec '19

Hi, We have seen several issues (mailed about that earlier to this list) after the upgrade to Mimic 13.2.8. We decided to downgrade the OSD servewrs to 13.2.6 to check if issues disappear. However we ran into issues with that ... We use bluestore allocator since Luminous 12.2.12 to combat latency issues on the OSDs. We also used that succesfully on Mimic 13.2.6. bluestore_allocator = bitmap bluefs_allocator = bitmap When downgrading to 13.2.6 we hit the following assert: 2019-12-27 14:14:16.409 7f2ed2dcce00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3.5 TiB 2019-12-27 14:14:16.409 7f2ed2dcce00 1 bluefs mount 2019-12-27 14:14:16.413 7f2ed2dcce00 -1 /build/ceph-13.2.6/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7f2ed2dcce00 time 2019-12-27 14:14:16.414793 /build/ceph-13.2.6/src/os/bluestore/fastbmap_allocator_impl.h: 749: FAILED assert(available >= allocated) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f2eca0a497e] 2: (()+0x2fab07) [0x7f2eca0a4b07] 3: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x44d) [0xc91dbd] 4: (BlueFS::mount()+0x260) [0xc6e6c0] 5: (BlueStore::_open_db(bool, bool)+0x17cd) [0xb8f50d] 6: (BlueStore::_mount(bool, bool)+0x4b7) [0xbbfb77] 7: (OSD::init()+0x295) [0x761fc5] 8: (main()+0x367b) [0x64f23b] 9: (__libc_start_main()+0xf0) [0x7f2ec7c3f830] 10: (_start()+0x29) [0x718929] We upgraded the node back to 13.2.8 again which started without issues. We did do a "downgrade test" on a test cluster ... that cluster did not suffer from this issue. It turned out that the cluster was not using the bitmap allocator ... after enabling the bitmap allocator there on a 13.2.6 node (that has been previviously downgraded but had never run with the bitmap allocator) and restarting the node this came online just fine. However, an upgrade to 13.2.8 with bitmap allocator enabled, and a downgrade again to 13.2.6 would trigger the same assert. Switching back to default (stupid allocator) again would work (initially) for 2 out of 3 OSDs. One would fail right away with rocksdb corruption: 2019-12-27 15:10:50.945 7fc77fbcbe00 20 osd.6 1952 register_pg 2.16 0x990c800 2019-12-27 15:10:50.945 7fc77fbcbe00 10 osd.6:2._attach_pg 2.16 0x990c800 2019-12-27 15:10:50.945 7fc77fbcbe00 10 osd.6 1952 pgid 2.0 coll 2.0_head 2019-12-27 15:10:50.945 7fc77fbcbe00 10 osd.6 1952 _make_pg 2.0 2019-12-27 15:10:50.945 7fc77fbcbe00 5 osd.6 pg_epoch: 1952 pg[2.0(unlocked)] enter Initial 2019-12-27 15:10:50.945 7fc77fbcbe00 20 osd.6 pg_epoch: 1952 pg[2.0(unlocked)] enter NotTrimming 2019-12-27 15:10:50.945 7fc77fbcbe00 -1 abort: Corruption: block checksum mismatch: expected 1122551773, got 2333355710 in db/000397.sst offset 57741 size 4044 2019-12-27 15:10:50.949 7fc77fbcbe00 -1 *** Caught signal (Aborted) ** in thread 7fc77fbcbe00 thread_name:ceph-osd ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0x11390) [0x7fc775520390] 2: (gsignal()+0x38) [0x7fc774a53428] 3: (abort()+0x16a) [0x7fc774a5502a] 4: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::list*)+0x4a8) [0xbff498] 5: (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >*)+0x201) [0xb852e1] 6: (PG::read_info(ObjectStore*, spg_t, coll_t const&, pg_info_t&, PastIntervals&, unsigned char&)+0x16b) [0x7ecc8b] 7: (PG::read_state(ObjectStore*)+0x56) [0x81aff6] 8: (OSD::load_pgs()+0x566) [0x759516] 9: (OSD::init()+0xcd3) [0x762a03] 10: (main()+0x367b) [0x64f23b] 11: (__libc_start_main()+0xf0) [0x7fc774a3e830] 12: (_start()+0x29) [0x718929] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. And after a restart we got rocksdb messages like this: ... 2019-12-27 15:11:32.322 7fd1c0fbbe00 4 rocksdb: [/build/ceph-13.2.6/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000402 ... ... -352> 2019-12-27 15:11:11.598 7fa0f0558e00 -1 abort: Corruption: Bad table magic number: expected 9863518390377041911, found 11124 in db/000397.sst ... After we set osd.6 out ... osd.7 crashed after a while (while backfilling) and would fail to restart again with the following message: 2019-12-27 15:27:10.833 7f1bb4701e00 4 rocksdb: [/build/ceph-13.2.6/src/rocksdb/db/db_impl.cc:252] Shutdown: canceling all background work 2019-12-27 15:27:10.833 7f1bb4701e00 4 rocksdb: [/build/ceph-13.2.6/src/rocksdb/db/db_impl.cc:397] Shutdown complete 2019-12-27 15:27:10.833 7f1bb4701e00 -1 rocksdb: Corruption: CURRENT file does not end with newline 2019-12-27 15:27:10.833 7f1bb4701e00 -1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db erroring opening db: 2019-12-27 15:27:10.833 7f1bb4701e00 1 bluefs umount 2019-12-27 15:27:10.833 7f1bb4701e00 1 stupidalloc 0x0x325aee0 shutdown 2019-12-27 15:27:10.833 7f1bb4701e00 1 bdev(0x380a380 /var/lib/ceph/osd/ceph-7/block) close 2019-12-27 15:27:11.093 7f1bb4701e00 1 bdev(0x380a000 /var/lib/ceph/osd/ceph-7/block) close 2019-12-27 15:27:11.345 7f1bb4701e00 -1 osd.7 0 OSD:init: unable to mount object store 2019-12-27 15:27:11.345 7f1bb4701e00 -1 ESC[0;31m ** ERROR: osd init failed: (5) Input/output errorESC[0m A restart / reboot of the node would not help. For those of you still running 13.2.6 ... I would not recommend upgrading to 13.2.8 (at least not for storage nodes ... mon / mds still seem to work fine). Does bitmap allocator modify the OSD on disk data in some way? Are you supposed to be able to switch between different allocators? Thanks, Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 0

rgw - ERROR: failed to fetch mdlog info
by Frank R 27 Dec '19

27 Dec '19

I am seeing the following errors on an RGW multisite slave: 1. ERROR: failed to fetch mdlog info 2. failed to fetch local sync status: (5) Input/output error Data seems to be replicating but not metadata. Does anyone have any ideas on what may be wrong? ----- # radosgw-admin sync status realm 8f7fd3fd-f72d-411d-b06b-7b4b579f5f2f (prod) zonegroup 60a2cb75-6978-46a3-b830-061c8be9dc75 (prod) zone ffce148e-3b24-462d-98bf-8c212de31de5 (us-east-1) 2019-12-27 12:29:13.329597 7f71c4ec9dc0 0 meta sync: ERROR: failed to fetch mdlog info metadata sync syncing full sync: 0/64 shards failed to fetch local sync status: (5) Input/output error data sync source: 7fe96e52-d6f7-4ad6-b66e-ecbbbffbc18e (us-east-2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 12 shards behind shards: [29,31,33,45,46,48,54,76,87,113,120,127] oldest incremental change not applied: 2019-12-27 12:28:58.0.107159s 23 shards are recovering recovering shards: [1,2,24,26,29,33,35,37,40,41,42,46,48,51,54,66,76,95,100,101,122,123,127] -----

1 0

s3curl putuserpolicy get 405
by 黄明友 26 Dec '19

26 Dec '19

hi,all I follow the https://docs.ceph.com/docs/master/radosgw/STSLite/ ; run s3curl.pl --debug --id admin -- -s -v -X POST "http://localhost:8000/?Action=PutUserPolicy&PolicyName=Policy1&UserName=TES…"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Deny\",\"Action\":\"s3:*\",\"Resource\":\[\"*\"\],\"Condition\":\{\"BoolIfExists\":\{\"sts:authentication\":\"false\"\}\}\},\{\"Effect\":\"Allow\",\"Action\":\"sts:GetSessionToken\",\"Resource\":\"*\",\"Condition\":\{\"BoolIfExists\":\{\"sts:authentication\":\"false\"\}\}\}\]\}&Version=2010-05-08" get error like this, <?xml version="1.0" encoding="UTF-8"?><Error><Code>MethodNotAllowed</Code><RequestId>tx000000000000000001273-005e031dca-e11e-wx6</RequestId><HostId>e11e-wx6-wx</HostId></Error> the ceph verion is ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) 黄明友 IT基础架构部经理 V.Photos 云摄影移动电话: +86 13540630430 客服电话：400 - 806 - 5775 电子邮件: hmy(a)v.photos 官方网址: www.v.photos 上海黄浦区中山东二路88号外滩SOHO3Q F栋 2层北京朝阳区光华路9号光华路SOHO二期南二门SOHO3Q 1层广州天河区林和中路136号天誉花园二期3Wcoffice 天誉青创社区深圳南山区蛇口网谷科技大厦二期A座102网谷双创街 1层成都成华区建设路世贸广场 7层

2 2

Re: Ceph and centos 8
by 林浩 26 Dec '19

26 Dec '19

Hi I can't find el8 rpm package here. http://download.ceph.com/rpm-hammer/ Might I know when the rpm package will be ready for centos8 ?

1 0

rgw logs
by Frank R 25 Dec '19

25 Dec '19

I have about 1TB of data in the pool default.rgw.logs. What logs are stored in this pool and can they be safely deleted? thx Frank

2 1

Ubuntu Bionic arm64 repo missing packages
by Bryan Stillwell 20 Dec '19

20 Dec '19

I was going to try adding an OSD to my home cluster using one of the 4GB Raspberry Pis today, but it appears that the Ubuntu Bionic arm64 repo is missing a bunch of packages: $ sudo grep ^Package: /var/lib/apt/lists/download.ceph.com_debian-nautilus_dists_bionic_main_binary-arm64_Packages Package: ceph-deploy Package: ceph-mgr-dashboard Package: ceph-mgr-diskprediction-cloud Package: ceph-mgr-diskprediction-local Package: ceph-mgr-k8sevents Package: ceph-mgr-rook Package: ceph-mgr-ssh Package: cephfs-shell Package: libcephfs-java Package: python-ceph-argparse Package: python3-ceph-argparse It looks like the packages were built, but for some reason the Packages file is missing them: http://download.ceph.com/debian-nautilus/pool/main/c/ceph/ Could someone look into why the repo isn't finding these packages? Thanks, Bryan

1 1