- ceph-users - lists.ceph.io

by Herbert Alexander Faleiros

Hi, is there any way to fix it instead a reboot? [128632.995249] block nbd0: Possible stuck request 00000000b14a04af: control (read@2097152,4096B). Runtime 9540 seconds [128663.718993] block nbd0: Possible stuck request 00000000b14a04af: control (read@2097152,4096B). Runtime 9570 seconds [128694.434774] block nbd0: Possible stuck request 00000000b14a04af: control (read@2097152,4096B). Runtime 9600 seconds [128725.154515] block nbd0: Possible stuck request 00000000b14a04af: control (read@2097152,4096B). Runtime 9630 seconds # ceph -v ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable) # rbd-nbd list-mapped # # uname -r 5.4.52-050452-generic Thanks, -- Herbert

3 years, 9 months

3
6
0 0

Fwd: BlueFS assertion ceph_assert(h->file->fnode.ino != 1)

by Aleksei Zakharov

3 years, 9 months

2
1
0 0

Push config to all hosts

by Cem Zafer

Hi, What is the best method(s) to push ceph.conf to all hosts in octopus (15.x)? Thanks.

3 years, 9 months

2
1
0 0

please help me fix iSCSI Targets not available

by David Thuong

iSCSI Targets not available Please consult the documentation on how to configure and enable the iSCSI Targets management functionality. Available information: There are no gateways defined any idea to enable it. tks so much

3 years, 9 months

2
1
0 0

cache tier dirty status

by Budai Laszlo

Hello all, is there a way to interrogate a cache tier pool about the number of dirty objects/bytes that it contains? Thank you, Laszlo

3 years, 9 months

1
0
0 0

Reinitialize rgw garbage collector

by Michael Bisig

Hi all, I have a question about the garbage collector within RGWs. We run Nautilus 14.2.8 and we have 32 garbage objects in the gc pool with totally 39 GB of garbage that needs to be processed. When we run, radosgw-admin gc process --include-all objects are processed but most of them won't be deleted. This can be checked by using --debug-rgw=5 in the command and stat the objects which are mentioned that they have been processed. Also the monitoring doesn't show that a huge amount of objects are deleted by the gc. So, I assume that it doesn't actually delete the objects. It might be due to a renewed time stamp? (not sure about this) Is there anybody who had similar issues with removing a large amount of garbage and is there a way to let the gc delete the objects? Most of the objects within the gc list are __multipart__ objects. Are they processed differently than single part objects? E.g. collect all the multiparts before the deletion actually happens or how is this implemented? The garbage is still increasing and the gc cannot process things what scares us a bit. Also, we cannot bypass the gc because the bucket is still in use. I also thought about reinitializing the GC in order to get an up to date list of garbage. (some entries show with `radosgw-admin gc list --include-all` are over a month old) Is there a way to make this happen and how save is it? I thought about exporting the omapobjects from the gc pool (as a backup) and delete the objects within the pool (or rename the pool). I appreciate any input and thank you in advance. Regards, Michael

3 years, 9 months

1
0
0 0

Ceph-deploy on rhel.

by sathvik vutukuri

Hi, Did any one installed ceph-deploy on rhel7 with rados gate way. I see there are no rpms available for rhel7 in ceph-deploy in download.ceph.com for nautilis , luminous, octopus versions. Is ceph-deploy go to for rhel7???

3 years, 9 months

3
9
0 0

large omap objects

by Zhenshi Zhou

Hi all, I have a cluster providing object storage. The cluster has worked well until someone saves flink checkpoints in the 'flink' bucket. I checked its behavior and I find that the flink saves the current checkpoint data and delete the former ones frequently. I suppose that it makes the bucket index get large omap object. I got '1 large omap objects' warning message these days. And after I check the cluster status and logs, of course, all of those large omap object points to exactly the same one index. The warning message: *cluster [WRN] Large omap object found. Object: 17:2f908b17:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.166289.1:head PG: 17.e8d109f4 (17.74) Key count: 568681 Size (bytes): 149443581* I did 'bilog trim' and 'pg deep-scrub' and the cluster became health again. However, I cannot do this all the time. Is there a way to solve this issue permanently? Thanks

3 years, 9 months

1
0
0 0

OSD_SCRUB_ERRORS 1 scrub errors

by Abhimnyu Dhobale

Dear Team, Ceph is showing below errors frequently on different Disks. every time after pg repair it is resolving. # ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 1.574 is active+clean+inconsistent, acting [19,25,2] # cat /var/log/ceph/ceph-osd.19.log | grep error 2020-07-12 11:42:11.824 7f864e0b2700 -1 log_channel(cluster) log [ERR] : 1.574 shard 25 soid 1:2ea0a7a3:::rbd_data.515c96b8b4567.0000000000007a7c:head : candidate had a read error 2020-07-12 11:42:15.035 7f86520ba700 -1 log_channel(cluster) log [ERR] : 1.574 deep-scrub 1 errors # ceph --version ceph version 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable) # rados -p vpsacephcl01 list-inconsistent-obj 1.3c9 --format=json-pretty { "epoch": 845, "inconsistents": [ { "object": { "name": "rbd_data.515c96b8b4567.000000000000c377", "nspace": "", "locator": "", "snap": "head", "version": 21101 }, "errors": [], "union_shard_errors": [ "read_error" ], "selected_object_info": { "oid": { "oid": "rbd_data.515c96b8b4567.000000000000c377", "key": "", "snapid": -2, "hash": 867656649, "max": 0, "pool": 1, "namespace": "" }, "version": "853'21101", "prior_version": "853'21100", "last_reqid": "client.2317742.0:24909022", "user_version": 21101, "size": 4194304, "mtime": "2020-07-16 21:02:20.564245", "local_mtime": "2020-07-16 21:02:20.572003", "lost": 0, "flags": [ "dirty", "omap_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xffffffff", "omap_digest": "0xffffffff", "expected_object_size": 4194304, "expected_write_size": 4194304, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 5, "primary": false, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x8ebd7de4" }, { "osd": 19, "primary": true, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x8ebd7de4" }, { "osd": 24, "primary": false, "errors": [ "read_error" ], "size": 4194304 } ] } ] } -- Thanks & Regards Abhimnyu Dhobale

3 years, 9 months

3
2
0 0

journal based mirroring works but snapshot based not

by yveskretzschmar＠web.de

Hi, i have a working journal based mirror setup initially created with nautilus. I recently upgraded to octopus (15.2.4) to use snapshot based mirroring. After that I disabled mirroring for the first image and reenabled it snapshot based. The remote Image is created but it never starts syncing. Network usage is only about some kb/s. And after disabling mirroring on the image the remote images doesn't get deleted. The same is with a new test pool and new images. I also remove the peer from the pools and disabled mirroring on the pools and recreated it from scratch, because I've found this: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LRBDKQ2YXQN… bit it didn't resolved it. Mirroring daemon shows the follwing after enabling snapshot based mirroring on an image: 2020-07-24T13:45:40.253+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images: peer_uuid=3d2164b5-96ca-4ad6-9fe3-4eb1f6c02f3e, added_count=1, removed_count=0 2020-07-24T13:45:40.253+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images_added: peer_uuid=3d2164b5-96ca-4ad6-9fe3-4eb1f6c02f3e, global_image_ids=[d7f7d308-d18b-488f-9dcc-87f8e77f05cc] 2020-07-24T13:45:40.253+0200 7f33dae1d700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 add_image: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 start_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=MAP_UPDATE 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 execute_policy_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, policy_action=MAP 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 map: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_image_mapping: updates=[{global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477}], removes=[] 2020-07-24T13:45:41.258+0200 7f33dae1d700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 finish_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=MAP_UPDATE, r=0 2020-07-24T13:45:41.294+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:41.402+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:41.502+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:41.606+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:45:42.258+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 start_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=ACQUIRE 2020-07-24T13:45:42.258+0200 7f33da61c700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 notify_listener_acquire_release_images: acquire=[{global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477}], release=[] 2020-07-24T13:45:42.258+0200 7f33da61c700 5 rbd::mirror::NamespaceReplayer: 0x55cfcc4c6000 handle_acquire_image: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477 2020-07-24T13:45:42.258+0200 7f33dae1d700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 finish_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=ACQUIRE, r=0 2020-07-24T13:45:42.306+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images: peer_uuid=, added_count=1, removed_count=0 2020-07-24T13:45:42.306+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images_added: peer_uuid=, global_image_ids=[d7f7d308-d18b-488f-9dcc-87f8e77f05cc] 2020-07-24T13:45:42.330+0200 7f33a3daf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 init: failed to retrieve mirror peer uuid from remote pool 2020-07-24T13:45:42.330+0200 7f33dae1d700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 get_replay_status: replay not running 2020-07-24T13:45:46.298+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:46.402+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:46.506+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:46.606+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:45:51.298+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:51.406+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:51.506+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:51.610+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:45:52.330+0200 7f33dae1d700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 get_replay_status: replay not running 2020-07-24T13:45:56.298+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:56.406+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:56.506+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:56.610+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:46:01.303+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:46:01.407+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:46:01.511+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:46:01.611+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:46:02.331+0200 7f33dae1d700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 get_replay_status: replay not running Regards, Yves

3 years, 9 months

3
8
0 0

2024

2023

2022

2021

2020

2019

ceph-users