- ceph-users - lists.ceph.io

Re: rgw recovering shards
by Konstantin Shalygin 30 Oct '19

30 Oct '19

On 10/27/19 6:01 AM, Frank R wrote: > I hate to be a pain but I have one more question. > > After I run > > radosgw-admin reshard stale-instances rm > > if I run > > radosgw-admin reshard stale-instances list > > some new entries appear for a bucket that no longer exists. Is there a > way to cancel the operation on the old bucket? > `radosgw-admin reshard stale-instances rm` should fix all your issues, if this not you should debug this. k

2 3

Dirlisting hangs with cephfs
by Kári Bertilsson 30 Oct '19

30 Oct '19

This seems to happen mostly when listing folders containing 10k+ folders. The dirlisting hangs indefinitely or until i restart the active MDS and then the hanging "ls" command will finish running. Every time restarting the active MDS fixes the problem for a while.

4 14

Lower mem radosgw config?
by Dan van der Ster 30 Oct '19

30 Oct '19

Hi all, Does anyone have a good config for lower memory radosgw machines? We have 16GB VMs and our radosgw's go OOM when we have lots of parallel clients (e.g. I see around 500 objecter_ops via the rgw asok). Maybe lowering rgw_thread_pool_size from 512 would help? (This is running latest luminous). Thanks, Dan

2 1

ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12
by Marc Roos 30 Oct '19

30 Oct '19

I am getting these since Nautilus upgrade [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 ffff911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d9 ffff9117b63e3900 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de ffff9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859da ffff911993e84700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e6 ffff911993e85500 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 ffff911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859db ffff911993e85800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859df ffff9119697ca700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 ffff91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 ffff9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 ffff911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d9 ffff9117b63e3900 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de ffff9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859da ffff911993e84700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e6 ffff911993e85500 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 ffff911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859db ffff911993e85800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859df ffff9119697ca700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 ffff91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 ffff9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 ffff911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de ffff9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 ffff911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 ffff91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 ffff9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 ffff911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de ffff9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 ffff911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 ffff91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 ffff9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 ffff911d3eef5a00 fail -12

1 0

V/v Log IP clinet in rados gateway log
by tuan dung 30 Oct '19

30 Oct '19

Hi all, I want to log IP client to rados gateway log to check information about loadbalancing and other things. I using LB before rados gateway nodes, what need to be done for configuration in rados gateway? thank you very much. Br, ---------------------------------------------- Dương Tuấn Dũng Email: dungdt.aicgroup(a)gmail.com ĐT: 0986153686

2 1

pg stays in unknown states for a long time
by 潘东元 30 Oct '19

30 Oct '19

I have 104 pg stays in unknown states for a long time [root@node-1 /]# ceph -s cluster: id: 653c6c1a-607e-4a62-bb92-dfe2f0d7afb6 health: HEALTH_ERR 1 osds down Reduced data availability: 104 pgs inactive 24 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,9,10 14 stuck requests are blocked > 4096 sec. Implicated osds 5,6 services: mon: 3 daemons, quorum node-1,node-2,node-3 mgr: node-1(active), standbys: node-2, node-3 osd: 12 osds: 11 up, 12 in flags nodeep-scrub rbd-mirror: 1 daemon active data: pools: 7 pools, 360 pgs objects: 1.80k objects, 3.91GiB usage: 17.6GiB used, 7.96TiB / 7.98TiB avail pgs: 28.889% pgs unknown 256 active+clean 104 unknown io: client: 1.56MiB/s wr, 0op/s rd, 83op/s wr [root@node-1 /]# ceph health detail HEALTH_ERR Reduced data availability: 104 pgs inactive; 30 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,8,9,10; 14 stuck requests are blocked > 4096 sec. Implicated osds 5,6 PG_AVAILABILITY Reduced data availability: 104 pgs inactive pg 1.0 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.1 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.2 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.3 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.4 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.5 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.6 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.7 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 2.0 is stuck inactive for 2857.069686, current state unknown, last acting [] ...... [root@node-1 /]# ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 3.1d unknown [] -1 [] -1 3.1c unknown [] -1 [] -1 3.1b unknown [] -1 [] -1 3.1a unknown [] -1 [] -1 3.19 unknown [] -1 [] -1 ...... my pool size = 3 [root@node-1 /]# ceph pg 3.1d query Error ENOENT: i don't have pgid 3.1d

1 0

Several ceph osd commands hang
by Thomas Schneider 30 Oct '19

30 Oct '19

Hi, in my unhealthy cluster I cannot run several ceph osd command because they hang, e.g. ceph osd df ceph osd pg dump Also, ceph balancer status hangs. How can I fix this issue? THX

3 5

After delete 8.5M Objects in a bucket still 500K left
by EDH - Manuel Rios Fernandez 29 Oct '19

29 Oct '19

Hi Ceph's! We started deteling a bucket several days ago. Total size 47TB / 8.5M objects. Now we see the cli bucket rm stucked and by console drop this messages. [root@ceph-rgw03 ~]# 2019-10-28 13:55:43.880 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 1000 incomplete multipart uploads 2019-10-28 13:56:24.021 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 2000 incomplete multipart uploads 2019-10-28 13:57:04.726 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 3000 incomplete multipart uploads 2019-10-28 13:57:45.424 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 4000 incomplete multipart uploads 2019-10-28 13:58:25.905 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 5000 incomplete multipart uploads 2019-10-28 13:59:06.898 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 6000 incomplete multipart uploads 2019-10-28 13:59:47.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 7000 incomplete multipart uploads 2019-10-28 14:00:42.102 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 8000 incomplete multipart uploads 2019-10-28 14:01:23.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 9000 incomplete multipart uploads 2019-10-28 14:02:06.028 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 10000 incomplete multipart uploads 2019-10-28 14:02:48.648 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 11000 incomplete multipart uploads 2019-10-28 14:03:29.807 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 12000 incomplete multipart uploads 2019-10-28 14:04:11.180 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 13000 incomplete multipart uploads 2019-10-28 14:04:52.396 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 14000 incomplete multipart uploads 2019-10-28 14:05:33.050 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 15000 incomplete multipart uploads 2019-10-28 14:06:13.652 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 16000 incomplete multipart uploads 2019-10-28 14:06:54.806 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 17000 incomplete multipart uploads 2019-10-28 14:07:35.867 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 18000 incomplete multipart uploads 2019-10-28 14:08:16.886 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 19000 incomplete multipart uploads 2019-10-28 14:08:57.711 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 20000 incomplete multipart uploads 2019-10-28 14:09:38.032 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 21000 incomplete multipart uploads 2019-10-28 14:10:18.377 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 22000 incomplete multipart uploads 2019-10-28 14:10:58.833 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 23000 incomplete multipart uploads 2019-10-28 14:11:39.078 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 24000 incomplete multipart uploads 2019-10-28 14:12:24.731 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 25000 incomplete multipart uploads 2019-10-28 14:13:12.176 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 26000 incomplete multipart uploads Bucket stats show 500K objects left. Looks like bucket rm is trying to abort all incompleted mutipart. But in bucket stats this operation is not reflected removing objects from stats. May be wait to get up 500K or it's a bug? Regards Manuel

2 2

Compression on existing RGW buckets
by Bryan Stillwell 29 Oct '19

29 Oct '19

I'm wondering if it's possible to enable compression on existing RGW buckets? The cluster is running Luminous 12.2.12 with FileStore as the backend (no BlueStore compression then). We have a cluster that recently started to rapidly fill up with compressible content (qcow2 images) and I would like to enable compression for new uploads to slow the growth. The documentation seems to imply that changing zone placement rules can only be done at creation time. Is there something I'm missing that would allow me to enable compression on a per-bucket or even a per-user basis after a cluster has been used for quite a while? Thanks, Bryan

3 3

Bogus Entries in RGW Usage Log / Large omap object in rgw.log pool
by David Monschein 29 Oct '19

29 Oct '19

Hi All, Running an object storage cluster, originally deployed with Nautilus 14.2.1 and now running 14.2.4. Last week I was alerted to a new warning from my object storage cluster: [root@ceph1 ~]# ceph health detail HEALTH_WARN 1 large omap objects LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'default.rgw.log' Search the cluster log for 'Large omap object found' for more details. I looked into this and found the object and pool in question (default.rgw.log): [root@ceph1 /var/log/ceph]# grep -R -i 'Large omap object found' . ./ceph.log:2019-10-24 12:21:26.984802 osd.194 (osd.194) 715 : cluster [WRN] Large omap object found. Object: 5:0fbdcb32:usage::usage.17:head Key count: 702330 Size (bytes): 92881228 [root@ceph1 ~]# ceph --format=json pg ls-by-pool default.rgw.log | jq '.[]' | egrep '(pgid|num_large_omap_objects)' | grep -v '"num_large_omap_objects": 0,' | grep -B1 num_large_omap_objects "pgid": "5.70", "num_large_omap_objects": 1, While I was investigating, I noticed an enormous amount of entries in the RGW usage log: [root@ceph ~]# radosgw-admin usage show | grep -c bucket 223326 There are entries for over 223k buckets! This was pretty scary to see, considering we only have maybe 500 legitimate buckets in this fairly new cluster. Almost all of the entries in the usage log are bogus entries from anonymous users. It looks like someone/something was scanning, looking for vulnerabilities, etc. Here are a few example entries, notice none of the operations were successful: <-----SNIP-----> { "bucket": "pk1914.php", "time": "2019-07-26 21:00:00.000000Z", "epoch": 1564174800, "owner": "anonymous", "categories": [ { "category": "post_obj", "bytes_sent": 586, "bytes_received": 0, "ops": 2, "successful_ops": 0 } ] }, { "bucket": "plus", "time": "2019-07-26 21:00:00.000000Z", "epoch": 1564174800, "owner": "anonymous", "categories": [ { "category": "post_obj", "bytes_sent": 6314, "bytes_received": 0, "ops": 22, "successful_ops": 0 } ] }, { "bucket": "pma.php", "time": "2019-07-26 21:00:00.000000Z", "epoch": 1564174800, "owner": "anonymous", "categories": [ { "category": "post_obj", "bytes_sent": 580, "bytes_received": 0, "ops": 2, "successful_ops": 0 } ] <-----SNIP-----> I suspect that the large omap warning from Ceph is related to the 223k+ entries in the RGW usage log. I have two questions: 1) How can I remove these bogus bucket entries from the RGW usage log? When I issue $(radosgw-admin usage trim), it only resets the stats to 0, but does not actually remove the bogus bucket entries. 2) Is it possible to prevent Ceph from logging usage statistics for buckets that do not exist? Some more output if it's useful:## ceph --format=json pg ls-by-pool default.rgw.log | jq '.[]' <-----SNIP-----> { "pgid": "5.70", "version": "4712'5226850", "reported_seq": "5636679", "reported_epoch": "4712", "state": "active+clean", "last_fresh": "2019-10-24 14:40:59.287019", "last_change": "2019-10-24 12:21:26.984997", "last_active": "2019-10-24 14:40:59.287019", "last_peered": "2019-10-24 14:40:59.287019", "last_clean": "2019-10-24 14:40:59.287019", "last_became_active": "2019-10-18 16:02:06.007865", "last_became_peered": "2019-10-18 16:02:06.007865", "last_unstale": "2019-10-24 14:40:59.287019", "last_undegraded": "2019-10-24 14:40:59.287019", "last_fullsized": "2019-10-24 14:40:59.287019", "mapping_epoch": 4672, "log_start": "4712'5223781", "ondisk_log_start": "4712'5223781", "created": 1105, "last_epoch_clean": 4673, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "4712'5223077", "last_scrub_stamp": "2019-10-24 12:21:26.984947", "last_deep_scrub": "4712'5223077", "last_deep_scrub_stamp": "2019-10-24 12:21:26.984947", "last_clean_scrub_stamp": "2019-10-24 12:21:26.984947", "log_size": 3069, "ondisk_log_size": 3069, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 2, "num_object_clones": 0, "num_object_copies": 6, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 2, "num_whiteouts": 0, "num_read": 6140544, "num_read_kb": 6769195, "num_write": 10664318, "num_write_kb": 9859326, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 8, "num_bytes_recovered": 0, "num_keys_recovered": 2666590, "num_objects_omap": 1, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 1, "num_objects_manifest": 0, "num_omap_bytes": 92881228, "num_omap_keys": 702330, "num_objects_repaired": 0 }, "up": [ 194, 79, 51 ], "acting": [ 194, 79, 51 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 194, "acting_primary": 194, "purged_snaps": [] }, I appreciate any assistance. Thank you. - Dave Monschein

2 2