Hello Eugenio, All previous "it just hangs" issues that I have seen previously were down to some network problem. Please check that you can ping all OSDs, MDSs, and MONs from the client. Please retest using large pings (ping -M dont -s 8972 192.168.12.34). Please inspect firewalls. If multiple network cards are used, make sure that cables are not accidentally swapped. On Wed, Sep 4, 2024 at 6:50 PM Eugenio Tampieri <eugenio.tampieri@readydigital.it> wrote:
Has it worked before or did it just stop working at some point? What's the exact command that fails (and error message if there is)?
It was working using the NFS gateway, I never tried with the Ceph FUSE mount. The command is ceph-fuse --id migration /mnt/repo. No error message, it just hangs.
For the "too many PGs per OSD" I suppose I have to add some other OSDs, right?
Either that or reduce the number of PGs. If you had only a few pools I'd suggest to leave it to the autoscaler, but not for 13 pools. You can paste 'ceph osd df' and 'ceph osd pool ls detail' if you need more input for that.
I already have the autoscaler enabled. Here is the output you asked for --- ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2 hdd 0.90970 1.00000 932 GiB 332 GiB 330 GiB 1.7 MiB 1.4 GiB 600 GiB 35.63 0.88 329 up 4 hdd 0.90970 1.00000 932 GiB 400 GiB 399 GiB 1.6 MiB 1.5 GiB 531 GiB 42.94 1.07 331 up 3 hdd 0.45479 1.00000 466 GiB 203 GiB 202 GiB 1.0 MiB 988 MiB 263 GiB 43.57 1.08 206 up 5 hdd 0.93149 1.00000 932 GiB 379 GiB 378 GiB 1.6 MiB 909 MiB 552 GiB 40.69 1.01 321 up TOTAL 3.2 TiB 1.3 TiB 1.3 TiB 5.9 MiB 4.8 GiB 1.9 TiB 40.30 MIN/MAX VAR: 0.88/1.08 STDDEV: 3.15 --- ceph osd pool ls detail pool 1 '.mgr' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 24150 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr pool 2 'kubernetes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 24150 lfor 0/0/92 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 24150 lfor 0/0/123 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 24150 lfor 0/0/132 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 24150 lfor 0/0/132 flags hashpspool stripe_width 0 application rgw pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 24150 lfor 0/0/134 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw pool 7 'repo_data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 30692 lfor 0/30692/30690 flags hashpspool stripe_width 0 application cephfs pool 8 'repo_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 24150 lfor 0/0/150 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 9 '.nfs' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 24150 lfor 0/0/169 flags hashpspool stripe_width 0 application nfs pool 11 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 24150 lfor 0/0/592 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw pool 12 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 24150 lfor 0/0/592 flags hashpspool stripe_width 0 application rgw pool 13 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 24150 lfor 0/0/644 flags hashpspool stripe_width 0 application rgw pool 19 'kubernetes-lan' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 24150 lfor 0/0/15682 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd --- Regards
Zitat von Eugenio Tampieri <eugenio.tampieri@readydigital.it>:
Hi Eugen, Sorry, but I had some trouble when I signed up and then I was away so I missed your reply.
ceph auth export client.migration [client.migration] key = redacted caps mds = "allow rw fsname=repo" caps mon = "allow r fsname=repo" caps osd = "allow rw tag cephfs data=repo"
For the "too many PGs per OSD" I suppose I have to add some other OSDs, right?
Thanks,
Eugenio
-----Messaggio originale----- Da: Eugen Block <eblock@nde.ag> Inviato: mercoledì 4 settembre 2024 10:07 A: ceph-users@ceph.io Oggetto: [ceph-users] Re: CephFS troubleshooting
Hi, I already responded to your first attempt:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/GS7KJ RJP7BAOF66KJM255G27TJ4KG656/
Please provide the requested details.
Zitat von Eugenio Tampieri <eugenio.tampieri@readydigital.it>:
Hello, I'm writing to troubleshoot an otherwise functional Ceph quincy cluster that has issues with cephfs. I cannot mount it with ceph-fuse (it gets stuck), and if I mount it with NFS I can list the directories but I cannot read or write anything. Here's the output of ceph -s cluster: id: 3b92e270-1dd6-11ee-a738-000c2937f0ec health: HEALTH_WARN mon ceph-storage-a is low on available space 1 daemons have recently crashed too many PGs per OSD (328 > max 250)
services: mon: 5 daemons, quorum ceph-mon-a,ceph-storage-a,ceph-mon-b,ceph-storage-c,ceph-storage-d (age 105m) mgr: ceph-storage-a.ioenwq(active, since 106m), standbys: ceph-mon-a.tiosea mds: 1/1 daemons up, 2 standby osd: 4 osds: 4 up (since 104m), 4 in (since 24h) rbd-mirror: 2 daemons active (2 hosts) rgw: 2 daemons active (2 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 13 pools, 481 pgs objects: 231.83k objects, 648 GiB usage: 1.3 TiB used, 1.8 TiB / 3.1 TiB avail pgs: 481 active+clean
io: client: 1.5 KiB/s rd, 8.6 KiB/s wr, 1 op/s rd, 0 op/s wr Best regards,
Eugenio Tampieri _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov