- ceph-users - lists.ceph.io

Pacific: RadosGW crashing on multipart uploads.
by Chu, Vincent 01 Jul '21

01 Jul '21

Hi, I'm running into an issue with RadosGW where multipart uploads crash, but only on buckets with a hyphen, period or underscore in the bucket name and with a bucket policy applied. We've tested this in pacific 16.2.3 and pacific 16.2.4. Anyone run into this before? ubuntu@ubuntu:~/ubuntu$ aws --endpoint http://placeholder.com:7480 s3 cp ubuntu.iso s3://bucket.test upload failed: ./ubuntu.iso to s3://bucket.test/ubuntu.iso Connection was closed before we received a valid response from endpoint URL: "http://placeholder.com:7480/bucket.test/ubuntu.iso?uploads". Here is the crash log. -12> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 1 ====== starting new request req=0x7fadf8998620 ===== -11> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s initializing for trans_id = tx000000000000000000963-0060db861a-17e77ee-default -10> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s getting op 4 -9> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s s3:init_multipart verifying requester -8> 2021-06-29T20:44:10.948+0000 7fae1f4ec700 2 req 2403 0.008000608s s3:init_multipart normalizing buckets and tenants -7> 2021-06-29T20:44:10.948+0000 7fae1f4ec700 2 req 2403 0.008000608s s3:init_multipart init permissions -6> 2021-06-29T20:44:10.954+0000 7faedf66c700 0 Supplied principal is discarded: arn:aws:iam::default:user -5> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart recalculating target -4> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart reading permissions -3> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart init op -2> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart verifying op mask -1> 2021-06-29T20:44:10.955+0000 7faedf66c700 2 req 2403 0.015001140s s3:init_multipart verifying op permissions 0> 2021-06-29T20:44:10.964+0000 7faedf66c700 -1 *** Caught signal (Segmentation fault) ** in thread 7faedf66c700 thread_name:radosgw ceph version 16.2.3 (381b476cb3900f9a92eb95d03b4850b953cfd79a) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7faf2dd05b20] 2: (rgw_bucket::rgw_bucket(rgw_bucket const&)+0x23) [0x7faf38b4d083] 3: (rgw::sal::RGWObject::get_obj() const+0x20) [0x7faf38b7bcf0] 4: (RGWInitMultipart::verify_permission(optional_yield)+0x6c) [0x7faf38e6608c] 5: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x86a) [0x7faf38b2db1a] 6: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x26dd) [0x7faf38b3232d] 7: /lib64/libradosgw.so.2(+0x4a1c0b) [0x7faf38a83c0b] 8: /lib64/libradosgw.so.2(+0x4a36a4) [0x7faf38a856a4] 9: /lib64/libradosgw.so.2(+0x4a390e) [0x7faf38a8590e] 10: make_fcontext() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -- Vincent Chu A-4: Advanced Research in Cyber Systems Los Alamos National Laboratory

2 1

cephadm dashboard errors
by Anthony Palermo 01 Jul '21

01 Jul '21

I am having issues accessing the dashboard [dashboard WARNING request] [::ffff:10.2.16.6:49696] [GET] [401] [0.045s] [463.0B] [d4a77c44-f9ee-45bc-8f90-870362e131bc] /api/mgr/module/telemetry Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.373+0000 7f536f15b700 0 [dashboard INFO request] [::ffff:10.2.16.6:49695] [GET] [200] [0.019s] [29.0B] [d3ecd267-948d-4dfa-887b-ccef517ad91b] /ui-api/langs Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.385+0000 7f536c956700 0 [dashboard ERROR exception] Internal Server Error Jun 30 20:51:16 cfnode01 bash[2655]: Traceback (most recent call last): Jun 30 20:51:16 cfnode01 bash[2655]: File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 46, in dashboard_exception_handler Jun 30 20:51:16 cfnode01 bash[2655]: return handler(*args, **kwargs) Jun 30 20:51:16 cfnode01 bash[2655]: File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ Jun 30 20:51:16 cfnode01 bash[2655]: return self.callable(*self.args, **self.kwargs) Jun 30 20:51:16 cfnode01 bash[2655]: File "/usr/share/ceph/mgr/dashboard/controllers/home.py", line 135, in __call__ Jun 30 20:51:16 cfnode01 bash[2655]: return serve_file(full_path) Jun 30 20:51:16 cfnode01 bash[2655]: File "/lib/python3.6/site-packages/cherrypy/lib/static.py", line 77, in serve_file Jun 30 20:51:16 cfnode01 bash[2655]: cptools.validate_since() Jun 30 20:51:16 cfnode01 bash[2655]: File "/lib/python3.6/site-packages/cherrypy/lib/cptools.py", line 116, in validate_since Jun 30 20:51:16 cfnode01 bash[2655]: raise cherrypy.HTTPRedirect([], 304) Jun 30 20:51:16 cfnode01 bash[2655]: cherrypy._cperror.HTTPRedirect: ([], 304) Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.385+0000 7f536d157700 0 [dashboard WARNING request] [::ffff:10.2.16.6:49699] [GET] [401] [0.022s] [462.0B] [a59575fa-5f58-4db1-85aa-4f155e9ea639] /api/summary Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.385+0000 7f536a952700 0 [dashboard WARNING request] [::ffff:10.2.16.6:49697] [GET] [401] [0.007s] [463.0B] [f5eb4e44-060e-451e-a379-8990f8871ef8] /api/feature_toggles Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.385+0000 7f536c956700 0 [dashboard INFO request] [::ffff:10.2.16.6:49694] [GET] [304] [0.017s] [0.0B] [d90f8f90-2f65-401e-8bf3-ad6c1c62a368] /Ceph_Logo.487a0001b327fa7f5232.svg Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.457+0000 7f536e95a700 0 [dashboard INFO request] [::ffff:10.2.16.6:49695] [POST] [200] [0.005s] [40.0B] [9058a1ae-5f0a-47d1-b5fd-bcd646fdac48] /api/auth/check Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.461+0000 7f536e95a700 0 [dashboard ERROR frontend.error] (http://10.1.50.20:8080/#/login) Http failure response for http://10.1.50.20:8080/ui-api/standard_settings: 401 Unauthorized Jun 30 20:51:16 cfnode01 bash[2655]: None Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.465+0000 7f536e159700 0 [dashboard INFO request] [::ffff:10.2.16.6:49698] [GET] [200] [0.020s] [29.0B] [ae97598b-fc78-4510-88d1-d79d6251d6a2] /ui-api/langs Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.465+0000 7f536e95a700 0 [dashboard INFO request] [::ffff:10.2.16.6:49696] [POST] [200] [0.009s] [24.0B] [ad9954d6-7743-4215-9a4a-df3e908d84da] /ui-api/logging/js-error

1 0

Unprotect snapshot: device or resource busy
by Jan Kasprzak 01 Jul '21

01 Jul '21

Hello, Ceph users, How can I figure out why it is not possible to unprotect a snapshot in a RBD image? I use this RBD pool for OpenNebula, and somehow there is a snapshot in one image, which OpenNebula does not see. So I wanted to delete the snapshot: # rbd info one/one-1312 rbd image 'one-1312': size 8 GiB in 2048 objects order 22 (4 MiB objects) snapshot_count: 1 id: 6732dccd50fa75 block_name_prefix: rbd_data.6732dccd50fa75 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Wed Jun 30 10:41:59 2021 access_timestamp: Wed Jun 30 16:48:30 2021 modify_timestamp: Wed Jun 30 15:52:18 2021 # rbd snap ls one/one-1312 SNAPID NAME SIZE PROTECTED TIMESTAMP 1727 snap 8 GiB yes Wed Jun 30 16:11:39 2021 # rbd snap rm one/one-1312@snap Removing snap: 0% complete...failed. rbd: snapshot 'snap' is protected from removal. 2021-07-01 08:33:41.489 7f79c6ffd700 -1 librbd::Operations: snapshot is protected # rbd snap unprotect one/one-1312@snap 2021-07-01 08:28:40.747 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [68ba8e7bace188] in pool 'one' 2021-07-01 08:28:40.749 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy 2021-07-01 08:28:40.749 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: 0x56522f10e830 should_complete_error: ret_val=-16 rbd: unprotecting snap failed: 2021-07-01 08:28:40.751 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: 0x56522f10e830 should_complete_error: ret_val=-16 As far as I can see neither the snapshot nor the RBD image itself is used by a running qemu in my cluster. How can I delete the snapshot or debug the problem further? This is Ceph 14.2.21 on CentOS (mons are CentOS 8 now, and OSDs are CentOS 7). Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 6

ceph tcp fastopen
by Seena Fallah 01 Jul '21

01 Jul '21

Hi, Is ceph using TCP_FASTOPEN for its sockets? If not why? Thanks.

1 0

Having issues to start more than 24 OSDs per host
by Jan.Jansen＠gdata.de 01 Jul '21

01 Jul '21

Hello We did try to use Cephadm with Podman to start 44 OSDs per host which consistently stop after adding 24 OSDs per host. We did look into the cephadm.log on the problematic host and saw that the command `cephadm ceph-volume lvm list --format json` did stuck. We were the output of the command wasn't complete. Therefore, we tried to use compacted JSON and we could increase the number to 36 OSDs per host. If you need more information just ask. Podman version: 3.2.1 Ceph version: 16.2.4 OS version: Suse Leap 15.3 Greetings, Jan

3 3

cephfs forward scrubbing docs
by Dan van der Ster 01 Jul '21

01 Jul '21

Hi, Today while debugging something we had a few questions that might lead to improving the cephfs forward scrub docs: https://docs.ceph.com/en/latest/cephfs/scrub/ tldr: 1. Should we document which sorts of issues that the forward scrub is able to fix? 2. Can we make it more visible (in docs) that scrubbing is not supported with multi-mds? 3. Isn't the new `ceph -s` scrub task status misleading with multi-mds? Details here: 1) We found a CephFS directory with a number of zero sized files: # ls -l ... -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58 upload_fc501199e3e7abe6b574101cf34aeefb.png -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23 upload_fce4f55348185fefa0abdd8d11095ba8.gif -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54 upload_fd95b8358851f0dac22fb775046a6163.png ... The user claims that those files were non-zero sized last week. The sequence of zero sized files includes *all* files written between Nov 2 and 9. The user claims that his client was running out of memory, but this is now fixed. So I suspect that his ceph client (kernel 3.10.0-1127.19.1.el7.x86_64) was not behaving well. Anyway, I noticed that even though the dentries list 0 bytes, the underlying rados objects have data, and the data looks good. E.g. # rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx 200212e68b5.00000000 # file 200212e68b5.00000000 200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA, non-interlaced So I managed to recover the files doing something like this (using an input file mapping inode to filename) [see PS 0]. But I'm wondering if a forward scrub is able to fix this sort of problem directly? Should we document which sorts of issues that the forward scrub is able to fix? I anyway tried to scrub it, which led to: # ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx recursive repair Scrub is not currently supported for multiple active MDS. Please reduce max_mds to 1 and then scrub. So ... 2) Shouldn't we update the doc to mention loud and clear that scrub is not currently supported for multiple active MDS? 3) I was somehow surprised by this, because I had thought that the new `ceph -s` multi-mds scrub status implied that multi-mds scrubbing was now working: task status: scrub status: mds.x: idle mds.y: idle mds.z: idle Is it worth reporting this task status for cephfs if we can't even scrub them? Thanks!! Dan [0] mkdir -p recovered while read -r a b; do for i in {0..9} do echo "rados stat --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i $(printf "%x" $a).0000000$i done echo cat $(printf "%x" $a).* ">" $(printf "%x" $a) echo mv $(printf "%x" $a) recovered/$b done < inones_fnames.txt

2 2

Semantics of cephfs-mirror
by Manuel Holtgrewe 30 Jun '21

30 Jun '21

Dear all, I'm sorry if I'm asking for the obvious or missing a previous discussion of this but I could not find the answer to my question online. I'd be happy to be pointed to the right direction only. The cephfs-mirror tool in pacific looks extremely promising. How does it work exactly? Is it based on files and (recursive) ctime or rather based on object information? Does it handle incremental changes (only) between snapshots? There is an issue related to this that mentions recursive ctime. But that would mean that users could "rsync -a" data to the file system and this would not get synchronized. I have good experience with ZFS which is able to identify changes between two snapshots A and B and then only transfer these changes (using a sub-file level, on the ZFS equivalent of blocks to my understanding) to another server with the same file system that is in the exact state as snapshot A. Does cephfs-mirror work the same? Best wishes, Manuel

2 1

Re: upgrading from Nautilus on CentOS7 to Octopus on Ubuntu 20.04.2
by Radoslav Milanov 30 Jun '21

30 Jun '21

Sounds exactly like testing to me... On 30.6.2021 г. 17:46 ч., Teoman Onay wrote: > What do you mean by different? > > RHEL is a supported product while CentOS is not. You get bug/security > fixes sooner on RHEL than on CentOS as depending on their severity > level they are released during the z-stream releases. > > Stream is one minor release ahead than RHEL which means it already > contains part of the fixes which will be released a few months later > in RHEL. It could be considered even more stable as it already > contains part of the fixes. > > > > > On Wed, 30 Jun 2021, 15:15 Radoslav Milanov, > <radoslav.milanov(a)gmail.com <mailto:radoslav.milanov@gmail.com>> wrote: > > If stream is so great why is RHEL different ? > > On 30.6.2021 г. 03:49 ч., Teoman Onay wrote: > >> > >>> For similar reasons, CentOS 8 stream, as opposed to every > other CentOS > >> released before, is very experimental. I would never go in > production with > >> CentOS 8 stream. > >> > >> > > Experimental?? Looks like you still don't understand what CentOS > stream is. > > If you have some time just read this: > > > https://www.linkedin.com/pulse/why-you-should-have-already-been-centos-stre… > <https://www.linkedin.com/pulse/why-you-should-have-already-been-centos-stre…> > > > > He summarized quite well what CentOS stream is. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > <mailto:ceph-users@ceph.io> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > <mailto:ceph-users-leave@ceph.io> >

1 0

v14.2.22 Nautilus released
by David Galloway 30 Jun '21

30 Jun '21

We're happy to announce the 22nd and likely final backport release in the Nautilus series. Ultimately, we recommend all users upgrade to newer Ceph releases. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2021/v14-2-22-nautilus-released Notable Changes --------------- * This release sets `bluefs_buffered_io` to true by default to improve performance for metadata heavy workloads. Enabling this option has been reported to occasionally cause excessive kernel swapping under certain workloads. Currently, the most consistent performing combination is to enable bluefs_buffered_io and disable system level swap. * The default value of `bluestore_cache_trim_max_skip_pinned` has been increased to 1000 to control memory growth due to onodes. * Several other bug fixes in BlueStore, including a fix for an unexpected ENOSPC bug in Avl/Hybrid allocators. * The trimming logic in the monitor has been made dynamic, with the introduction of `paxos_service_trim_max_multiplier`, a factor by which `paxos_service_trim_max` is multiplied to make trimming faster, when required. Setting it to 0 disables the upper bound check for trimming and makes the monitors trim at the maximum rate. * A `--max <n>` option is available with the `osd ok-to-stop` command to provide up to N OSDs that can be stopped together without making PGs unavailable. * OSD: the option `osd_fast_shutdown_notify_mon` has been introduced to allow the OSD to notify the monitor it is shutting down even if `osd_fast_shutdown` is enabled. This helps with the monitor logs on larger clusters, that may get many 'osd.X reported immediately failed by osd.Y' messages, and confuse tools. * A long-standing bug that prevented 32-bit and 64-bit client/server interoperability under msgr v2 has been fixed. In particular, mixing armv7l (armhf) and x86_64 or aarch64 servers in the same cluster now works. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-14.2.22.tar.gz * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: ca74598065096e6fcbd8433c8779a2be0c889351

1 0

Ceph connect to openstack
by Szabo, Istvan (Agoda) 30 Jun '21

30 Jun '21

Hi, Is there any proper documentation how to connect ceph with openstack? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 3