- ceph-users - lists.ceph.io

Multiple CephFS mounts and FSCache
by caskd 07 Aug '23

07 Aug '23

Hello everyone, I've been trying to use CephFS and mix it together with fscache, however i was never able to have multiple mounts with fscache enabled. Is this a known intentional limitation or a bug? It would be possible to work around it by mounting the root of the filesystem and using bind mounts but i have separate volumes that need to be separately mounted. How to replicate: mount -t ceph -o fsc admin@.filesystem1=/path1 /tmp/one # Succeeds mount -t ceph -o fsc admin@.filesystem1=/path2 /tmp/two # Fails complaining about no mds being available The alternative of using no fscache works just fine: mount -t ceph admin@.filesystem1=/path1 /tmp/one # Succeeds mount -t ceph admin@.filesystem1=/path2 /tmp/two # Succeeds Versions: - ceph quincy 17.2.6 - linux 6.4.6 - cachefilesd 0.10.10 -- Alex D. RedXen System & Infrastructure Administration https://redxen.eu/

2 3

64k buckets for 1 user
by Szabo, Istvan (Agoda) 07 Aug '23

07 Aug '23

Hi, We are in a transition where I'd like to ask my user who stores 2B objects in 1 bucket to split it some way. Thinking for the future we identified to make it future proof and don't store huge amount of objects in 1 bucket, we would need to create 65xxx buckets. Is there anybody aware of any issue with this amount of buckets please? I guess better to split to multiple buckets rather than have gigantic bucket. Thank you the advises ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Natuilus: Taking out OSDs that are 'Failure Pending'
by Dave Hall 04 Aug '23

04 Aug '23

Hello. It's been a while. I have a Nautilus cluster with 72 x 12GB HDD OSDs (BlueStore) and mostly of EC 8+2 pools/PGs. It's been working great - some nodes went nearly 900 days without a reboot. As of yesterday I found that I have 3 OSDs with a Smart status of 'Pending Failure'. New drives are ordered and will be here next week. There is a procedure in the documentation for replacing an OSD, but I can't do that directly until I receive the drives. My inclination is to mark these 3 OSDs 'OUT' before they crash completely, but I want to confirm my understanding of Ceph's response to this. Mainly, given my EC pools (or replicated pools for that matter), if I mark all 3 OSD out all at once will I risk data loss? If I have it right, marking an OSD out will simply cause Ceph to move all of the PG shards from that OSD to other OSDs, so no major risk of data loss. However, if it would be better to do them one per day or something, I'd rather be safe. I also assume that I should wait for the rebalance to complete before I initiate the replacement procedure. Your thoughts? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

4 4

snapshot timestamp
by Tony Liu 04 Aug '23

04 Aug '23

Hi, We know snapshot is on a point of time. Is this point of time tracked internally by some sort of sequence number, or the timestamp showed by "snap ls", or something else? I noticed that when "deep cp", the timestamps of all snapshot are changed to copy-time. Say I create a snapshot at 1PM and make a copy at 3PM, the timestamp of snapshot in the copy is 3PM. If I rollback the copy to this snapshot, I'd assume it will actually bring me back to the state of 1PM. Is that correct? If the above is true, I won't be able to rely on timestamp to track snapshots. Say I create a snapshot every hour and make a backup by copy at the end of the day. Then the original image is damaged and backup is used to restore the work. On this backup image, how do I know which snapshot was on 1PM, which was on 2PM, etc.? Any advices to track snapshots properly in such case? I can definitely build something else to help on this, but I'd like to know how much Ceph can support it. Thanks! Tony

2 2

What's the max of snap ID?
by Tony Liu 04 Aug '23

04 Aug '23

Hi, There is a snap ID for each snapshot. How is this ID allocated, sequentially? Did some tests, it seems this ID is per pool, starting from 4 and always going up. Is that correct? What's the max of this ID? What's going to happen when ID reaches the max, going back to start from 4 again? Thanks! Tony

3 3

cephfs mount problem - client session lacks required features
by Götz Reinicke 04 Aug '23

04 Aug '23

Hi, During the upgrade from centos7/nautilus to ubuntu 18/nautilus (still updating the MONs) I got a cephfs client who refuses or is refused to mount the ceph fs again. The clients says: mount error 13 = Permission denied The cephmds log: lacks required features 0x0000000000001000 client supports 0x00000000000000ff The mds/mon is still centos7/nautilus. The Clients centos7 as well. Any ideas ? Thx for suggestions and hints . Best Götz

2 2

question about OSD onode hits ratio
by Ben 04 Aug '23

04 Aug '23

Hi, We have a cluster running for a while. From grafana ceph dashboard, I saw OSD onode hits ratio 92% when cluster was just up and running. After couple month, it says now 70%. This is not a good trend I think. Just wondering what should be done to stop this trend. Many thank, Ben

2 1

Ceph Quincy and liburing.so.2 on Rocky Linux 9
by dobrie2＠gmu.edu 04 Aug '23

04 Aug '23

I've been digging and I can't see that this has come up anywhere. I'm trying to update a client from Pacific 17.2.3-2 to 17.2.6-4 and I'm getting the error Error: Problem: cannot install the best update candidate for package ceph-base-2:17.2.3-2.el9s.x86_64 - nothing provides liburing.so.2()(64bit) needed by ceph-base-2:17.2.6-4.el9s.x86_64 - nothing provides liburing.so.2(LIBURING_2.0)(64bit) needed by ceph-base-2:17.2.6-4.el9s.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) Did Ceph Pacific switch to requiring liburing 2? Rocky 9 only provides 0.7-7. CentOS stream seems to have 1.0.7-3 (at least back to when I set up that repo on Foreman; I don't remember if I'm keeping it up-to-date). Can I/should I just do --nobest when updating? I could probably build it from a source RPM from another RH-based distro, but I'd rather keep it clean with the same distro.

6 6

MDS nodes blocklisted
by Nathan Harper 04 Aug '23

04 Aug '23

Hi, We're having sporadic problems with a CephFS filesystem where MDSs end up on the OSD blocklist. We're still digging around looking for a cause (Ceph related or other infrastructure cause). The cluster isn't massive (68 OSDs spread over 34 hosts), each host is a VM, with MGR/MON/MDS on non-OSD hosts. Running Ceph 16.2.10 Any suggestions for debugging this further?

3 2

cephfs snapshot mirror peer_bootstrap import hung
by Adiga, Anantha 04 Aug '23

04 Aug '23

Hi Could you please provide guidance on how to diagnose this issue: In this case, there are two Ceph clusters: cluster A, 4 nodes and cluster B, 3 node, in different locations. Both are already running RGW multi-site, A is master. Cephfs snapshot mirroring is being configured on the clusters. Cluster A is the primary, cluster B is the peer. Cephfs snapshot mirroring is being configured. The bootstrap import step on the primary node hangs. On the target cluster : --------------------------- "version": "16.2.5", "release": "pacific", "release_type": "stable" root@cr21meg16ba0101:/# ceph fs snapshot mirror peer_bootstrap create cephfs client.mirror_remote flex2-site {"token": "eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd1h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTUuNzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6MzMwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0="} root@cr21meg16ba0101:/var/run/ceph# On the source cluster: ---------------------------- "version": "17.2.6", "release": "quincy", "release_type": "stable" root@fl31ca104ja0201:/# ceph -s cluster: id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e health: HEALTH_OK services: mon: 3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 111m) mgr: fl31ca104ja0201.nwpqlh(active, since 11h), standbys: fl31ca104ja0203, fl31ca104ja0202 mds: 1/1 daemons up, 2 standby osd: 44 osds: 44 up (since 111m), 44 in (since 4w) cephfs-mirror: 1 daemon active (1 hosts) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 25 pools, 769 pgs objects: 614.40k objects, 1.9 TiB usage: 2.8 TiB used, 292 TiB / 295 TiB avail pgs: 769 active+clean root@fl31ca104ja0302:/# ceph mgr module enable mirroring module 'mirroring' is already enabled root@fl31ca104ja0302:/# ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd1h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTUuNzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6MzMwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0= root@fl31ca104ja0302:/var/run/ceph# ceph --admin-daemon /var/run/ceph/ceph-client.cephfs-mirror.fl31ca104ja0302.sypagt.7.94083135960976.asok status { "metadata": { "ceph_sha1": "d7ff0d10654d2280e08f1ab989c7cdf3064446a5", "ceph_version": "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)", "entity_id": "cephfs-mirror.fl31ca104ja0302.sypagt", "hostname": "fl31ca104ja0302", "pid": "7", "root": "/" }, "dentry_count": 0, "dentry_pinned_count": 0, "id": 5194553, "inst": { "name": { "type": "client", "num": 5194553 }, "addr": { "type": "v1", "addr": "10.45.129.5:0", "nonce": 2497002034 } }, "addr": { "type": "v1", "addr": "10.45.129.5:0", "nonce": 2497002034 }, "inst_str": "client.5194553 10.45.129.5:0/2497002034", "addr_str": "10.45.129.5:0/2497002034", "inode_count": 1, "mds_epoch": 118, "osd_epoch": 6266, "osd_epoch_barrier": 0, "blocklisted": false, "fs_name": "cephfs" } root@fl31ca104ja0302:/home/general# docker logs ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-cephfs-mirror-fl31ca104ja0302-sypagt --tail 10 debug 2023-08-03T05:24:27.413+0000 7f8eb6fc0280 0 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable), process cephfs-mirror, pid 7 debug 2023-08-03T05:24:27.413+0000 7f8eb6fc0280 0 pidfile_write: ignore empty --pid-file debug 2023-08-03T05:24:27.445+0000 7f8eb6fc0280 1 mgrc service_daemon_register cephfs-mirror.5184622 metadata {arch=x86_64,ceph_release=quincy,ceph_version=ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable),ceph_version_short=17.2.6,container_hostname=fl31ca104ja0302,container_image=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e,cpu=Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz,distro=centos,distro_description=CentOS Stream 8,distro_version=8,hostname=fl31ca104ja0302,id=fl31ca104ja0302.sypagt,instance_id=5184622,kernel_description=#82-Ubuntu SMP Tue Jun 6 23:10:23 UTC 2023,kernel_version=5.15.0-75-generic,mem_swap_kb=8388604,mem_total_kb=527946928,os=Linux} debug 2023-08-03T05:27:10.419+0000 7f8ea1b2c700 0 client.5194553 ms_handle_reset on v2:10.45.128.141:3300/0 debug 2023-08-03T05:50:10.917+0000 7f8ea1b2c700 0 client.5194553 ms_handle_reset on v2:10.45.128.139:3300/0 Thank you, Anantha

2 5