- ceph-users - lists.ceph.io

Announcing go-ceph v0.5.0
by John Mulligan 11 Aug '20

11 Aug '20

I'm happy to announce the another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.5.0 The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. Specific questions, comments, bugs etc are best directed at our github issues tracker. --- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

pgs not deep scrubbed in time - false warning?
by Dirk Sarpe 11 Aug '20

11 Aug '20

Hi, since some time (I think upgrade to nautilus) we get X pgs not deep scrubbed in time I deep-scrubbed the pgs when the error occurred and expected the cluster to recover over time, but no such luck. The warning comes up again and again. In our spinning rust cluster we allow deep scrubbing only from 19:00 to 06:00 and changed the deep scrub interval to 28 days (detailed osd scrub config below). Looking at the pgs which are supposedly not deep scrubbed in time, reveals that their 28 day period is not over yet, example: # date && ceph health detail | awk '$1=="pg" { print $0 }' Di 11. Aug 09:32:17 CEST 2020 pg 0.787 not deep-scrubbed since 2020-07-30 03:08:24.899264 pg 0.70c not deep-scrubbed since 2020-07-30 02:45:08.989329 pg 0.6c1 not deep-scrubbed since 2020-07-30 03:01:15.199496 pg 13.3 not deep-scrubbed since 2020-07-30 03:29:54.536825 pg 0.d9 not deep-scrubbed since 2020-07-30 03:12:34.503586 pg 0.41a not deep-scrubbed since 2020-07-30 03:01:23.514582 pg 0.490 not deep-scrubbed since 2020-07-30 03:05:45.616100 I wonder if I have missed or messed up some parameter that could cause this. # ceph daemon osd.40 config show | grep scrub "mds_max_scrub_ops_in_progress": "5", "mon_scrub_inject_crc_mismatch": "0.000000", "mon_scrub_inject_missing_keys": "0.000000", "mon_scrub_interval": "86400", "mon_scrub_max_keys": "100", "mon_scrub_timeout": "300", "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", "mon_warn_pg_not_scrubbed_ratio": "0.500000", "osd_debug_deep_scrub_sleep": "0.000000", "osd_deep_scrub_interval": "2419200.000000", "osd_deep_scrub_keys": "1024", "osd_deep_scrub_large_omap_object_key_threshold": "200000", "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", "osd_deep_scrub_randomize_ratio": "0.150000", "osd_deep_scrub_stride": "1048576", "osd_deep_scrub_update_digest_min_age": "7200", "osd_max_scrubs": "1", "osd_op_queue_mclock_scrub_lim": "0.001000", "osd_op_queue_mclock_scrub_res": "0.000000", "osd_op_queue_mclock_scrub_wgt": "1.000000", "osd_requested_scrub_priority": "120", "osd_scrub_auto_repair": "false", "osd_scrub_auto_repair_num_errors": "5", "osd_scrub_backoff_ratio": "0.660000", "osd_scrub_begin_hour": "19", "osd_scrub_begin_week_day": "0", "osd_scrub_chunk_max": "25", "osd_scrub_chunk_min": "5", "osd_scrub_cost": "52428800", "osd_scrub_during_recovery": "false", "osd_scrub_end_hour": "6", "osd_scrub_end_week_day": "7", "osd_scrub_interval_randomize_ratio": "0.500000", "osd_scrub_invalid_stats": "true", "osd_scrub_load_threshold": "0.500000", "osd_scrub_max_interval": "604800.000000", "osd_scrub_max_preemptions": "5", "osd_scrub_min_interval": "172800.000000", "osd_scrub_priority": "5", "osd_scrub_sleep": "0.100000", ceph version 14.2.10 My ad hoc helper script #!/bin/bash # gently deep scrub all pgs which failed to deep scrub in set period ceph health detail | awk '$1=="pg" { print $2 } ' | while read -r pg do ceph pg deep-scrub "$pg" sleep 60 done Cheers, Dirk

1 1

Get instant help for Cash App Refund issues.
by mary smith 11 Aug '20

11 Aug '20

Do you Cash App Refund on your cash app balance or bank account? If yes then connect with the technical team as they ensure quick tech support services, You’ll be guided with precise solutions for the refund process. Just call the technical experts and seek their assistance on this. You can call them at any time of the day for support. https://www.customercare-email.com/blog/how-to-refund-a-payment-on-your-cas…

1 0

Fix Cash App transfer error? Contact Cash.app/help for help.
by mary smith 11 Aug '20

11 Aug '20

One of the app’s primary function is transfer and making payments. But if there’s a tech error that’s not encouraging you to do that, then you can watch some tech videos on Youtube for assistance or you can call the Cash.app/help number for the assistance and relay your problem to the customer care rep to get a solution to fix the issue. https://www.cash-app-customer-service.com/

1 0

Deleterious effects OSD queue
by João Victor Mafra 10 Aug '20

10 Aug '20

Hi folks, Regarding OSD operation queues (shards), I found this in the documentation (https://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/) "A lower number of shards will increase the impact of the mClock queues, but may have other deleterious effects". What are these deleterious effects? Why is it so bad to use just one queue for example? Is there any kind of bottleneck or costly procedures in these queues?

1 0

EntityAddress format in ceph ssd blacklist commands
by Void Star Nill 10 Aug '20

10 Aug '20

Hi, I want to understand the <EntityAddress> format for `ceph osd blacklist` commands. The documentation just says it's the address. But I am not sure if it can just be the host IP address or anything else. What does *:0/* *3710147553* represent in the following output? $ ceph osd blacklist ls listed 1 entries127.0.0.1:0/3710147553 2018-03-19 11:32:24.716146 $ ceph osd blacklist rm 127.0.0.1:0/3710147553 un-blacklisting 127.0.0.1:0/3710147553 Regards, Shridhar

2 4

ceph rbd iscsi gwcli Non-existent images
by Steven Vacaroaia 10 Aug '20

10 Aug '20

Hi, I would appreciate any help/hints to solve this issue iscis (gwcli) cannot see the images anymore This configuration worked fine for many months What changed was that ceph is "nearly full" I am in the process of cleaning it up ( by deleting objects from one of the pools) and I do see reads and writes on the cluster as well as images info so not sure what gwcli does not like ( targetcli ls not working either - just froze ) Below some info ceph version ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) gwcli --version gwcli - 2.7 ceph osd dump | grep ratio full_ratio 0.96 backfillfull_ratio 0.92 nearfull_ratio 0.9 [root@osd02 ~]# rbd -p rbd info rep01 rbd image 'rep01': size 7 TiB in 1835008 objects order 22 (4 MiB objects) id: 15b366b8b4567 block_name_prefix: rbd_data.15b366b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Nov 1 15:57:52 2018 [root@osd02 ~]# rbd -p rbd info vmware01 rbd image 'vmware01': size 6 TiB in 1572864 objects order 22 (4 MiB objects) id: 16d3f6b8b4567 block_name_prefix: rbd_data.16d3f6b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Nov 29 13:56:28 2018 [root@osd02 ~]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 33 TiB 7.5 TiB 25 TiB 77.16 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cephfs_metadata 22 173 MiB 0.01 1.4 TiB 469 cephfs_data 23 1.7 TiB 69.78 775 GiB 486232 rbd 24 11 TiB 93.74 775 GiB 2974077 [root@osd02 ~]# ceph health detail HEALTH_ERR 2 nearfull osd(s); 2 pool(s) nearfull; Module 'prometheus' has failed: IOError("Port 9283 not free on '10.10.35.20'",) OSD_NEARFULL 2 nearfull osd(s) osd.12 is near full osd.17 is near full POOL_NEARFULL 2 pool(s) nearfull pool 'cephfs_data' is nearfull pool 'rbd' is nearfull gwcli /iscsi-target...nner-21faa413> info Client Iqn .. iqn.1998-01.com.vmware:banner-21faa413 Ip Address .. Alias .. Logged In .. Auth - chap .. cephuser/PASSWORD Group Name .. Luns - rbd.rep01 .. lun_id=0 - rbd.vmware01 .. lun_id=1 osd02 journal: client update failed on iqn.1998-01.com.vmware:banner-21faa413 : Non-existent images ['rbd.vmware01'] requested for iqn.1998-01.com.vmware:banner-21faa413 Aug 7 14:15:39 osd02 journal: 127.0.0.1 - - [07/Aug/2020 14:15:39] "PUT /api/_clientlun/iqn.1998-01.com.vmware:banner-21faa413 HTTP/1.1" 500 - Aug 7 14:15:39 osd02 journal: _clientlun change on 127.0.0.1 failed with 500 Aug 7 14:15:39 osd02 journal: 127.0.0.1 - - [07/Aug/2020 14:15:39] "DELETE /api/clientlun/iqn.1998-01.com.vmware:banner-21faa413 HTTP/1.1" 500 -

2 3

Re: RGW unable to delete a bucket
by Matt Benjamin 10 Aug '20

10 Aug '20

Hi Folks, I don't know of a downstream issue that looks like this, and we've upstreamed every fix for bucket listing and cleanup we have. We are pursuing a space leak believed to arise in "radosgw-admin bucket rm --purge-objects" but not a non-terminating listing. The only upstream release not planned to get a backport of orphans list tools is Luminous. I thought backport to Octopus was already done by the backport team? regards, Matt On Thu, Aug 6, 2020 at 2:40 PM EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > > You'r not the only one affected by this issue > > As far as i know several huge companies hitted this bug too, but private patches or tools are not public released. > > This is caused for the a resharding process during upload in previous versions. > > Workarround for us.: > > - Delete objects of the bucket at rados level. > - Delete the index file of the bucket. > > Pray to god to not happen again. > > Still pending backporting to Nautilus of the new experimental tool to find orphans in RGW > > Maybe @Matt Benjamin can give us and ETA for get ready that tool backported... > > Regards > > > > -----Mensaje original----- > De: Andrei Mikhailovsky <andrei(a)arhont.com> > Enviado el: jueves, 6 de agosto de 2020 13:55 > Para: ceph-users <ceph-users(a)ceph.io> > Asunto: [ceph-users] Re: RGW unable to delete a bucket > > BUMP... > > > ----- Original Message ----- > > From: "Andrei Mikhailovsky" <andrei(a)arhont.com> > > To: "ceph-users" <ceph-users(a)ceph.io> > > Sent: Tuesday, 4 August, 2020 17:16:28 > > Subject: [ceph-users] RGW unable to delete a bucket > > > Hi > > > > I am trying to delete a bucket using the following command: > > > > # radosgw-admin bucket rm --bucket=<bucket-name> --purge-objects > > > > However, in console I get the following messages. About 100+ of those > > messages per second. > > > > 2020-08-04T17:11:06.411+0100 7fe64cacf080 1 > > RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket > > listing requires read #1 > > > > > > The command has been running for about 35 days days and it still > > hasn't finished. The size of the bucket is under 1TB for sure. Probably around 500GB. > > > > I have recently removed about a dozen of old buckets without any > > issues. It's this particular bucket that is being very stubborn. > > > > Anything I can do to remove it, including it's objects and any orphans > > it might have? > > > > > > Thanks > > > > Andrei > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309

2 1

DocuBetter Meeting this week -- 12 Aug 2020 0830 PDT
by John Zachary Dover 10 Aug '20

10 Aug '20

There is a general documentation meeting called the "DocuBetter Meeting", and it is held every two weeks. The next DocuBetter Meeting will be on 12 Aug 2020 at 0830 PDT, and will run for thirty minutes. Everyone with a documentation-related request or complaint is invited. The meeting will be held here: https://bluejeans.com/908675367 This meeting will cover the reorganization of the Ceph website as well as Zac's recently-developed workflow that aims to make it possible to move the good ideas from stale or poorly-formed documentation PRs into the documentation more quickly. Send documentation-related requests and complaints to me by replying to this email and CCing me at zac.dover(a)gmail.com. The next DocuBetter meeting is scheduled for: 12 Aug 2020 0830 PDT 12 Aug 2020 1630 UTC 13 Aug 2020 0230 AEST Etherpad: https://pad.ceph.com/p/Ceph_Documentation Zac's docs whiteboard: https://pad.ceph.com/p/docs_whiteboard Meeting: https://bluejeans.com/908675367

1 0

RGW 14.2.10 Regresion? ordered bucket listing requires read #1
by EDH - Manuel Rios 10 Aug '20

10 Aug '20

Hi, We got our cluster updated to the lastest versión 14.2.10 Checking rgw logs after 14.2.10 upgrade 2020-08-10 10:21:49.186 7f74cd7db700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.188 7f75eca19700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.189 7f7479f34700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.193 7f74d77ef700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.196 7f753c0b8700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.197 7f74cd7db700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.197 7f745d6fb700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 Looks like this -> https://tracker.ceph.com/issues/42358 Regards Manuel

1 0