- ceph-users - lists.ceph.io

Possibly unused client
by Alexander E. Patrakov 16 Dec '20

16 Dec '20

Hello, While working with a customer, I went through the output of "ceph auth list", and found a client entry that nobody can tell what it is used for. There is a strong suspicion that it is an unused left-over from old times, but again, nobody is sure. How can I confirm that it was not used for, say, the past week? Or, what logs should I turn on so that if it is used during the next week, it is mentioned there? -- Alexander E. Patrakov CV: http://pc.cd/PLz7

2 2

ceph stuck removing image from trash
by Andre Gebers 16 Dec '20

16 Dec '20

Hi, I'm running a 15.2.4 test cluster in a rook-ceph environment. The cluster is reporting HEALTH_OK but it seems it is stuck removing an image. Last section of 'ceph status' output: progress: Removing image replicapool/43def5e07bf47 from trash (6h) [............................] (remaining: 32y) This is now going for a couple of weeks and I was wondering if there is a way to speed it up? The cluster doesn't seem to be doing much judging from the system load. I've created this largish image to test what is possible with the setup but how do I get it out of the trash now? # rbd info --image-id 43def5e07bf47 -p replicapool rbd image 'csi-vol-cfaa1b00-1711-11eb-b9c9-2aa51e1e24e5': size 1 EiB in 274877906944 objects order 22 (4 MiB objects) snapshot_count: 0 id: 43def5e07bf47 block_name_prefix: rbd_data.43def5e07bf47 format: 2 features: layering op_features: flags: create_timestamp: Sun Oct 25 22:31:23 2020 access_timestamp: Sun Oct 25 22:31:23 2020 modify_timestamp: Sun Oct 25 22:31:23 2020 Any pointers how to resolve this issue are much appreciated. Regards Andre

4 3

issue on adding SSD to SATA cluster for db/wal
by Zhenshi Zhou 16 Dec '20

16 Dec '20

Hi all, I have a 14.2.15 cluster with all SATA OSDs. Now we plan to add SSDs in the cluster for db/wal usage. I checked the docs and found a command 'ceph-bluestore-tool' can deal with the issue. I added db/wal to the osd in my test environment but in the end it still get the warning message. "osd.0 spilled over 64 KiB metadata from 'db' device (7 MiB used of 8.0 GiB) to slow device" my procedure : sdd is the new disk for db/wal. sgdisk --new=1:0:+8GB --change-name=1:bluestore_block_db_0 --partition-guid=1:$(uuidgen) --mbrtogpt -- /dev/sdd sgdisk --new=2:0:+1GB --change-name=2:bluestore_block_wal_0 --partition-guid=2:$(uuidgen) --mbrtogpt -- /dev/sdd systemctl stop ceph-osd@0 CEPH_ARGS="--bluestore-block-db-size 8589934592" ceph-bluestore-tool bluefs-bdev-new-db --path /var/lib/ceph/osd/ceph-0 --dev-target /dev/sdd1 ceph-bluestore-tool bluefs-bdev-new-wal --path /var/lib/ceph/osd/ceph-0/ --dev-target /dev/sdd2 ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-0/ systemctl start ceph-osd@0 ceph tell osd.0 compact The warning message tells that there is some metadata still in the slow device. How can I deal with this issue? Thanks

2 2

Whether read I/O is accpted when the number of replica is under pool's min_size
by Satoru Takeuchi 15 Dec '20

15 Dec '20

Hi, Could you tell me whether read I/O is acxepted when the number of replicas is under pool's min_size? I read the official document and found that there is a difference of the effect of pool's min_size between pool's document and the pool's configuration document. Pool's document: https://docs.ceph.com/en/latest/rados/operations/pools/ > To set a minimum number of required replicas for I/O, you should use the min_size setting. Pool's configuration document. https://docs.ceph.com/en/latest/rados/configuration/pool-pg-config-ref/ > osd pool default min size Description Sets the minimum number of written replicas for objects in the pool in order to acknowledge a write operation to the client. The former says that min_size is related to both read/write I/O. On the other hands, the latter says it's only about write I/O. So which is correct description? As a result of my verification, the former looks correct. I created a RGW object in a pool having 3 replicas and its min_size is 2. Both read and write aren't accepted when 2 replicas are gone. Thanks, Satoru

2 2

multiple OSD crash, unfound objects
by Michael Thomas 15 Dec '20

15 Dec '20

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects. The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem). I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg…. I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects. I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below). The guide also says "Sometimes it simply takes some time for the cluster to query possible locations." I'm not sure how long "some time" might take, but it hasn't changed after several hours. My questions are: * Is there a way to force the cluster to query the possible locations sooner? * Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes? --Mike ceph -s: cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops. services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs task status: scrub status: mds.ceph4: idle data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................]

4 34

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

Weird ceph df
by Szabo, Istvan (Agoda) 15 Dec '20

15 Dec '20

Hi, It is a nautilus 14.2.13 ceph. The quota on the pool is 745GiB, how can be the stored data 788GiB? (2 replicas pool). Based on the used column it means just 334GiB is used because the pool has 2 replicas only. I don't understand. POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR k8s-dbss-w-mdc 12 788 GiB 202.42k 668 GiB 0.75 43 TiB N/A 745 GiB 202.42k 0 B 0 B Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Re: Monitors not starting, getting "e3 handle_auth_request failed to assign global_id"
by Hoan Nguyen Van 14 Dec '20

14 Dec '20

I found a merge request, ceph mon has new optíon :mon_sync_max_payload_keys https://github.com/ceph/ceph/commit/d6037b7f484e13cfc9136e63e4cf7fac6ad6896… My value of options mon_sync_max_payload_size is 4096. If mon_sync_max_payload_keys options is default, ceph mon may be not sync because slow opt. What do you think?

1 0

Slow Replication on Campus
by Vikas Rana 14 Dec '20

14 Dec '20

Hi Friends, We have 2 Ceph clusters on campus and we setup the second cluster as the DR solution. The images on the DR side are always behind the master. Ceph Version : 12.2.11 VMWARE_LUN0: global_id: 23460954-6986-4961-9579-0f2a1e58e2b2 state: up+replaying description: replaying, master_position=[object_number=2632711, tag_tid=24, entry_tid=1967382595], mirror_position=[object_number=1452837, tag_tid=24, entry_tid=456440697], entries_behind_master=1510941898 last_update: 2020-11-30 14:13:38 VMWARE_LUN1: global_id: cb579579-13b0-4522-b65f-c64ec44cbfaf state: up+replaying description: replaying, master_position=[object_number=1883943, tag_tid=28, entry_tid=1028822927], mirror_position=[object_number=1359161, tag_tid=28, entry_tid=358296085], entries_behind_master=670526842 last_update: 2020-11-30 14:13:33 Any suggestion on tuning or any parameters we can set on RBD-mirror to speed up the replication. Both cluster have very little activity. Appreciate your help. Thanks, -Vikas

2 1

iscsi and iser
by Marc Boisis 14 Dec '20

14 Dec '20

Hi, I would like to know if you support iser in gwcli like the traditional targetcli or if this is planned in a future version of ceph ? Thanks Marc

2 1