- ceph-users - lists.ceph.io

large omap objects
by Zhenshi Zhou 27 Jul '20

27 Jul '20

Hi all, I have a cluster providing object storage. The cluster has worked well until someone saves flink checkpoints in the 'flink' bucket. I checked its behavior and I find that the flink saves the current checkpoint data and delete the former ones frequently. I suppose that it makes the bucket index get large omap object. I got '1 large omap objects' warning message these days. And after I check the cluster status and logs, of course, all of those large omap object points to exactly the same one index. The warning message: *cluster [WRN] Large omap object found. Object: 17:2f908b17:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.166289.1:head PG: 17.e8d109f4 (17.74) Key count: 568681 Size (bytes): 149443581* I did 'bilog trim' and 'pg deep-scrub' and the cluster became health again. However, I cannot do this all the time. Is there a way to solve this issue permanently? Thanks

1 0

OSD_SCRUB_ERRORS 1 scrub errors
by Abhimnyu Dhobale 25 Jul '20

25 Jul '20

Dear Team, Ceph is showing below errors frequently on different Disks. every time after pg repair it is resolving. # ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 1.574 is active+clean+inconsistent, acting [19,25,2] # cat /var/log/ceph/ceph-osd.19.log | grep error 2020-07-12 11:42:11.824 7f864e0b2700 -1 log_channel(cluster) log [ERR] : 1.574 shard 25 soid 1:2ea0a7a3:::rbd_data.515c96b8b4567.0000000000007a7c:head : candidate had a read error 2020-07-12 11:42:15.035 7f86520ba700 -1 log_channel(cluster) log [ERR] : 1.574 deep-scrub 1 errors # ceph --version ceph version 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable) # rados -p vpsacephcl01 list-inconsistent-obj 1.3c9 --format=json-pretty { "epoch": 845, "inconsistents": [ { "object": { "name": "rbd_data.515c96b8b4567.000000000000c377", "nspace": "", "locator": "", "snap": "head", "version": 21101 }, "errors": [], "union_shard_errors": [ "read_error" ], "selected_object_info": { "oid": { "oid": "rbd_data.515c96b8b4567.000000000000c377", "key": "", "snapid": -2, "hash": 867656649, "max": 0, "pool": 1, "namespace": "" }, "version": "853'21101", "prior_version": "853'21100", "last_reqid": "client.2317742.0:24909022", "user_version": 21101, "size": 4194304, "mtime": "2020-07-16 21:02:20.564245", "local_mtime": "2020-07-16 21:02:20.572003", "lost": 0, "flags": [ "dirty", "omap_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xffffffff", "omap_digest": "0xffffffff", "expected_object_size": 4194304, "expected_write_size": 4194304, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 5, "primary": false, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x8ebd7de4" }, { "osd": 19, "primary": true, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x8ebd7de4" }, { "osd": 24, "primary": false, "errors": [ "read_error" ], "size": 4194304 } ] } ] } -- Thanks & Regards Abhimnyu Dhobale

3 2

journal based mirroring works but snapshot based not
by yveskretzschmar＠web.de 24 Jul '20

24 Jul '20

Hi, i have a working journal based mirror setup initially created with nautilus. I recently upgraded to octopus (15.2.4) to use snapshot based mirroring. After that I disabled mirroring for the first image and reenabled it snapshot based. The remote Image is created but it never starts syncing. Network usage is only about some kb/s. And after disabling mirroring on the image the remote images doesn't get deleted. The same is with a new test pool and new images. I also remove the peer from the pools and disabled mirroring on the pools and recreated it from scratch, because I've found this: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LRBDKQ2YXQN… bit it didn't resolved it. Mirroring daemon shows the follwing after enabling snapshot based mirroring on an image: 2020-07-24T13:45:40.253+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images: peer_uuid=3d2164b5-96ca-4ad6-9fe3-4eb1f6c02f3e, added_count=1, removed_count=0 2020-07-24T13:45:40.253+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images_added: peer_uuid=3d2164b5-96ca-4ad6-9fe3-4eb1f6c02f3e, global_image_ids=[d7f7d308-d18b-488f-9dcc-87f8e77f05cc] 2020-07-24T13:45:40.253+0200 7f33dae1d700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 add_image: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 start_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=MAP_UPDATE 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 execute_policy_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, policy_action=MAP 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 map: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_image_mapping: updates=[{global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477}], removes=[] 2020-07-24T13:45:41.258+0200 7f33dae1d700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 finish_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=MAP_UPDATE, r=0 2020-07-24T13:45:41.294+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:41.402+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:41.502+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:41.606+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:45:42.258+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 start_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=ACQUIRE 2020-07-24T13:45:42.258+0200 7f33da61c700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 notify_listener_acquire_release_images: acquire=[{global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477}], release=[] 2020-07-24T13:45:42.258+0200 7f33da61c700 5 rbd::mirror::NamespaceReplayer: 0x55cfcc4c6000 handle_acquire_image: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477 2020-07-24T13:45:42.258+0200 7f33dae1d700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 finish_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=ACQUIRE, r=0 2020-07-24T13:45:42.306+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images: peer_uuid=, added_count=1, removed_count=0 2020-07-24T13:45:42.306+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images_added: peer_uuid=, global_image_ids=[d7f7d308-d18b-488f-9dcc-87f8e77f05cc] 2020-07-24T13:45:42.330+0200 7f33a3daf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 init: failed to retrieve mirror peer uuid from remote pool 2020-07-24T13:45:42.330+0200 7f33dae1d700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 get_replay_status: replay not running 2020-07-24T13:45:46.298+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:46.402+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:46.506+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:46.606+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:45:51.298+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:51.406+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:51.506+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:51.610+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:45:52.330+0200 7f33dae1d700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 get_replay_status: replay not running 2020-07-24T13:45:56.298+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:56.406+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:56.506+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:56.610+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:46:01.303+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:46:01.407+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:46:01.511+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:46:01.611+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:46:02.331+0200 7f33dae1d700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 get_replay_status: replay not running Regards, Yves

3 8

ceph orch apply [osd, mon] -i YAML file not found
by Hayashida, Mami 24 Jul '20

24 Jul '20

I am exploring the new cephadm tool by deploying a tiny test cluster using OpenStack VMs, but have encountered the following issue: ceph orch apply osd -i <yaml file> command fails with "No such file or directory: <yaml file>". (Monitor setup with a yaml file resulted in the same error) ``` root@octopus-mon-1:~# ls -lh total 12K drwxr-xr-x 2 root root 4.0K Jul 23 14:20 Playbooks -rw-r--r-- 1 root root 99 Jul 23 16:16 mon_spec.yml -rw-r--r-- 1 root root 127 Jul 23 16:03 osd_spec.yml root@octopus-mon-1:~# cat osd_spec.yml --- service_type: osd service_id: default_drive_group placement: host_pattern: 'octopus-osd-[1-3]' data_devices: all: true root@octopus-mon-1:~# ceph orch apply osd -i /root/osd_spec.yml --dry-run --verbose INFO:cephadm:Inferring fsid 9e87977c-cd16-11ea-b571-fa163ec9a1ad INFO:cephadm:Inferring config /var/lib/ceph/9e87977c-cd16-11ea-b571-fa163ec9a1ad/mon.octopus-mon-1/config INFO:cephadm:Using recent ceph image ceph/ceph:v15 parsed_args: Namespace(admin_socket=None, block=False, cephconf=None, client_id=None, client_name=None, cluster=None, cluster_timeout=None, completion=False, help=False, input_file='/root/osd_spec.yml', output_file=None, output_format=None, period=1, setgroup=None, setuser=None, status=False, verbose=True, version=False, watch=False, watch_channel=None, watch_debug=False, watch_error=False, watch_info=False, watch_sec=False, watch_warn=False), childargs: ['orch', 'apply', 'osd', '--dry-run'] Can't open input file /root/osd_spec.yml: [Errno 2] No such file or directory: '/root/osd_spec.yml' root@octopus-mon-1:~# ceph orch --version INFO:cephadm:Inferring fsid 9e87977c-cd16-11ea-b571-fa163ec9a1ad INFO:cephadm:Inferring config /var/lib/ceph/9e87977c-cd16-11ea-b571-fa163ec9a1ad/mon.octopus-mon-1/config INFO:cephadm:Using recent ceph image ceph/ceph:v15 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) *Mami Hayashida* *Research Computing Associate* Univ. of Kentucky ITS Research Computing Infrastructure

2 3

ceph-volume on FreeBSD: ImportError: cannot import name 'devices' from 'ceph_volume_zfs'
by Ronny Forberger 24 Jul '20

24 Jul '20

1 0

weight_set array in Ceph CRUSH
by Bobby 24 Jul '20

24 Jul '20

Hi, I have a confusion regarding a struct data type in Ceph CRUSH source code. Header fie in here https://github.com/ceph/ceph/blob/master/src/crush/crush.h If you see below. there is a struct data type namely _weight_set_. What I have understood going through different CRUSH maps, this _weight_set_ array should be a 2D array, am I right? struct crush_choose_arg { __s32 *ids; /*!< values to use instead of items */ __u32 ids_size; /*!< size of the __ids__ array */ struct crush_weight_set *weight_set; /*!< weight replacements for a given position */ __u32 weight_set_positions; }; BR Bobby

1 0

rbd map image with journaling
by Herbert Alexander Faleiros 24 Jul '20

24 Jul '20

Hi, is there any way to do that without disabling journaling? # rbd map image@snap rbd: sysfs write failed RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable image@snap journaling". In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (6) No such device or address # ceph -v ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable) # uname -r 5.4.52-050452-generic Thanks, -- Herbert

3 3

IPv6 connectivity gone for Ceph Telemetry
by Wido den Hollander 24 Jul '20

24 Jul '20

Hi, I was just checking on a few (13) IPv6-only Ceph clusters and I noticed that they couldn't send their Telemetry data anymore: telemetry.ceph.com has address 8.43.84.137 This server used to have Dual-Stack connectivity while it was still hosted at OVH. It seemed to have moved to Red Hat, but lost IPv6 connectivity. How can we get this back? Wido

2 1

Re: journal based mirroring works but snapshot based not
by Yves Kretzschmar-Schwipper 24 Jul '20

24 Jul '20

1 0

What affection to vm with Monitor down
by 展荣臻（信泰） 24 Jul '20

24 Jul '20

Hi,everyone Our production ceph cluster the version is hammer have 3 monitors and 300+ osds.Monitor daemon runs on host diffrent from osd daemon. openstack vm runs on rbd.Now we need to maintain monitor host so must shutdown monitor host a moment, one by one. But keep two monitor host online all the time. What is the impact of monitor down on vm? How to do reduce the impact?

2 1