- ceph-users - lists.ceph.io

Prevent users to create buckets
by sinan＠turka.nl 18 Apr '24

18 Apr '24

Hello, I am using Ceph RGW for S3. Is it possible to create (sub)users that cannot create/delete buckets and are limited to specific buckets? At the end, I want to create 3 separate users and for each user I want to create a bucket. The users should only have access to their own bucket and should not be able to create new or delete buckets. One approach could be to limit the max_buckets to 1 so the user cannot create new buckets, but it will still have access to other buckets and will able to delete buckets. Any advice here? Thanks! Sinan

4 3

crushmap history
by Blair Bethwaite 18 Apr '24

18 Apr '24

Hi all, Do the Mons store any crushmap history, and if so how does one get at it please? I ask because we've recently encountered an issue in a medium scale (~5PB raw) EC based RGW focused cluster where "something" happened, which we still don't know, that suddenly caused us to see 94% of objects (5.4 billion of them) misplaced. We've tracked down the first log message of that pgmap state change: Mar 29 10:30:31 mon1 bash\[5804\]: debug 2024-03-29T10:30:31.152+0000 7f3b6e378700 0 log\_channel(cluster) log \[DBG\] : pgmap v44327: 2273 pgs: 225 active+clean, 2038 active+remapped+backfill\_wait, 10 active+remapped+backfilling; 1.6 PiB data, 2.1 PiB used, 2.2 PiB / 4.3 PiB avail; 5426274136/5752755429 objects misplaced (94.325%); 248 MiB/s, 109 objects/s recovering This appears to have been preceded (aside from a single HTTP HEAD request coming into RGW) by a 5 minute gap in logs where either journald couldn't keep up with debug messages or the Mons were stuck. The last log before that occurs seems to be a compaction event kicking off: mon1 bash\[25927\]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Mar 29 10:24:14 mon1 bash\[25927\]: \*\* Compaction Stats \[L\] \*\* Mar 29 10:24:14 mon1 bash\[25927\]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Mar 29 10:24:14 mon1 bash\[25927\]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Mar 29 10:24:14 mon1 bash\[25927\]: Low 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 116.0 11.4 0.02 0.01 7 0.003 490 462 Mar 29 10:24:14 mon1 bash\[25927\]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.9 1.23 1.20 28 0.044 0 0 Mar 29 10:24:14 mon1 bash\[25927\]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 16.4 0.00 0.00 1 0.001 0 0 We're left wondering what the heck has happened to cause such a huge redistribution of data in the cluster when we've not made any corresponding changes, so wanting to see if there's any breadcrumbs we can find. Appreciate any pointers! -- Cheers, ~Blairo

2 2

Client kernel crashes on cephfs access
by Marc Ruhmann 18 Apr '24

18 Apr '24

Hi everyone, I would like to ask for help regarding client kernel crashes that happen on cephfs access. We have been struggling with this for over a month now with over 100 crashes on 7 hosts during that time. Our cluster runs version 18.2.1. Our clients run CentOS Stream. On CentOS Stream 9 the problem started with kernel version 5.14.0-425.el9. Version 5.14.0-419.el9 is the last one without problems. It also occurred on CentOS Stream 8, starting with version 4.18.0-546.el8 (4.18.0-544.el8 being the last good one). The problem presents itself by the client kernel crashing, forcing a reboot of the machine. Apparently it is triggered by a certain level of IO on the cephfs mount. It works perfectly fine when we rollback to the last good kernel version. The exact call trace in vmcore-dmesg.txt differs between occurrences. Here are two typical examples: ``` [ 8641.382499] list_del corruption. next->prev should be ffff88bd0a4d4c80, but was ffff88bcefdfd280 [ 8641.382521] ------------[ cut here ]------------ [ 8641.382521] kernel BUG at lib/list_debug.c:54! [ 8641.382528] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 8641.382591] CPU: 2 PID: 83929 Comm: kworker/2:0 Kdump: loaded Not tainted 5.14.0-432.el9.x86_64 #1 [ 8641.382610] Hardware name: oVirt RHEL/RHEL-AV, BIOS edk2-20230524-4.el9_3 05/24/2023 [ 8641.382624] Workqueue: ceph-cap ceph_cap_unlink_work [ceph] [ 8641.382662] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47 [ 8641.382681] Code: c7 c7 78 42 d8 b1 e8 f9 87 fe ff 0f 0b 48 89 fe 48 c7 c7 08 43 d8 b1 e8 e8 87 fe ff 0f 0b 48 c7 c7 b8 43 d8 b1 e8 da 87 fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 78 43 d8 b1 e8 c6 87 fe ff 0f 0b [ 8641.382711] RSP: 0018:ffff95a000d6be60 EFLAGS: 00010246 [ 8641.382722] RAX: 0000000000000054 RBX: ffff88bced76dc00 RCX: 0000000000000000 [ 8641.382734] RDX: 0000000000000000 RSI: ffff88c02eea0840 RDI: ffff88c02eea0840 [ 8641.382746] RBP: ffff88bd0a4d4c80 R08: 80000000ffff8434 R09: 0000000000ffff10 [ 8641.382758] R10: 000000000000000f R11: 000000000000000f R12: ffff88c02eeb2800 [ 8641.382779] R13: ffff88bcc4610258 R14: ffff88bcc46101b8 R15: ffff88bcc46101c8 [ 8641.382793] FS: 0000000000000000(0000) GS:ffff88c02ee80000(0000) knlGS:0000000000000000 [ 8641.382809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8641.382819] CR2: 00007f35cee8a000 CR3: 0000000105708004 CR4: 00000000007706e0 [ 8641.382832] PKRU: 55555554 [ 8641.382838] Call Trace: [ 8641.382844] <TASK> [ 8641.382850] ? show_trace_log_lvl+0x1c4/0x2df [ 8641.382860] ? show_trace_log_lvl+0x1c4/0x2df [ 8641.382870] ? ceph_cap_unlink_work+0x3f/0x140 [ceph] [ 8641.382893] ? __die_body.cold+0x8/0xd [ 8641.382902] ? die+0x2b/0x50 [ 8641.382911] ? do_trap+0xce/0x120 [ 8641.382919] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382930] ? do_error_trap+0x65/0x80 [ 8641.382938] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382948] ? exc_invalid_op+0x4e/0x70 [ 8641.382958] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382975] ? asm_exc_invalid_op+0x16/0x20 [ 8641.382988] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382998] ceph_cap_unlink_work+0x3f/0x140 [ceph] [ 8641.383021] process_one_work+0x1e2/0x3b0 [ 8641.383032] ? __pfx_worker_thread+0x10/0x10 [ 8641.383043] worker_thread+0x50/0x3a0 [ 8641.383051] ? __pfx_worker_thread+0x10/0x10 [ 8641.383061] kthread+0xdd/0x100 [ 8641.383069] ? __pfx_kthread+0x10/0x10 [ 8641.383078] ret_from_fork+0x29/0x50 [ 8641.383090] </TASK> [ 8641.383095] Modules linked in: tls ceph libceph dns_resolver fscache netfs nft_counter ipt_REJECT xt_owner xt_conntrack nft_compat nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables libcrc32c nfnetlink vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_common nfit virtio_gpu iTCO_wdt iTCO_vendor_support libnvdimm lpc_ich virtio_dma_buf drm_shmem_helper drm_kms_helper i2c_i801 rapl syscopyarea sysfillrect sysimgblt virtio_balloon fb_sys_fops i2c_smbus pcspkr joydev fuse drm ext4 mbcache jbd2 sr_mod cdrom sd_mod ahci t10_pi sg libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net virtio_console virtio_scsi net_failover failover serio_raw ``` ``` [ 3538.365469] list_del corruption. next->prev should be ffff8d2b75997c80, but was ffff8d2afcfaae80 [ 3538.365488] ------------[ cut here ]------------ [ 3538.365488] kernel BUG at lib/list_debug.c:54! [ 3538.365493] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 3538.365553] CPU: 0 PID: 910 Comm: php-fpm Kdump: loaded Not tainted 5.14.0-432.el9.x86_64 #1 [ 3538.365569] Hardware name: oVirt RHEL/RHEL-AV, BIOS edk2-20230524-4.el9_3 05/24/2023 [ 3538.365582] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47 [ 3538.365612] Code: c7 c7 78 42 38 8e e8 f9 87 fe ff 0f 0b 48 89 fe 48 c7 c7 08 43 38 8e e8 e8 87 fe ff 0f 0b 48 c7 c7 b8 43 38 8e e8 da 87 fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 78 43 38 8e e8 c6 87 fe ff 0f 0b [ 3538.365641] RSP: 0018:ffffae870073fda0 EFLAGS: 00010246 [ 3538.365652] RAX: 0000000000000054 RBX: ffff8d2b75997800 RCX: 0000000000000000 [ 3538.365668] RDX: 0000000000000000 RSI: ffff8d2e2ee20840 RDI: ffff8d2e2ee20840 [ 3538.365681] RBP: ffff8d2b75997ab8 R08: 80000000ffff842f R09: 0000000000ffff10 [ 3538.365693] R10: 000000000000000f R11: 000000000000000f R12: 00000000ffffc032 [ 3538.365705] R13: ffff8d2b75997c80 R14: ffff8d2ac480b800 R15: ffff8d2ac480b9c8 [ 3538.365717] FS: 00007f9be42097c0(0000) GS:ffff8d2e2ee00000(0000) knlGS:0000000000000000 [ 3538.365733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3538.365744] CR2: 00007fa97dc5398c CR3: 0000000104248004 CR4: 00000000007706f0 [ 3538.365756] PKRU: 55555554 [ 3538.365761] Call Trace: [ 3538.365768] <TASK> [ 3538.365774] ? show_trace_log_lvl+0x1c4/0x2df [ 3538.365785] ? show_trace_log_lvl+0x1c4/0x2df [ 3538.365796] ? ceph_drop_caps_for_unlink+0xb8/0x170 [ceph] [ 3538.365828] ? __die_body.cold+0x8/0xd [ 3538.365836] ? die+0x2b/0x50 [ 3538.365845] ? do_trap+0xce/0x120 [ 3538.365853] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365863] ? do_error_trap+0x65/0x80 [ 3538.365871] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365881] ? exc_invalid_op+0x4e/0x70 [ 3538.365891] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365901] ? asm_exc_invalid_op+0x16/0x20 [ 3538.365912] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365923] ceph_drop_caps_for_unlink+0xb8/0x170 [ceph] [ 3538.365947] ceph_unlink+0xed/0x450 [ceph] [ 3538.365970] vfs_unlink+0x114/0x290 [ 3538.365980] do_unlinkat+0x1af/0x2e0 [ 3538.365990] __x64_sys_unlink+0x3e/0x60 [ 3538.365999] do_syscall_64+0x59/0x90 [ 3538.366008] ? syscall_exit_to_user_mode+0x22/0x40 [ 3538.366018] ? do_syscall_64+0x69/0x90 [ 3538.366027] ? do_syscall_64+0x69/0x90 [ 3538.366035] entry_SYSCALL_64_after_hwframe+0x72/0xdc [ 3538.366046] RIP: 0033:0x7f9be40ff27b [ 3538.366069] Code: f0 ff ff 73 01 c3 48 8b 0d a2 ab 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 75 ab 0f 00 f7 d8 64 89 01 48 [ 3538.367031] RSP: 002b:00007ffd8640de58 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3538.367576] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 00007f9be40ff27b [ 3538.368116] RDX: 0000000000000007 RSI: 0000000000000001 RDI: 00007f9bdd4af698 [ 3538.368646] RBP: 00007f9bdd4af698 R08: 00000000ffffffc9 R09: 0000000000000038 [ 3538.369156] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 3538.369671] R13: 00007f9bdd4af698 R14: 0000000000000001 R15: 00007f9be3c15290 [ 3538.370182] </TASK> [ 3538.370682] Modules linked in: ceph libceph dns_resolver fscache netfs nft_counter ipt_REJECT xt_owner xt_conntrack nft_compat nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables libcrc32c nfnetlink vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency_common virtio_gpu virtio_dma_buf drm_shmem_helper isst_if_common drm_kms_helper nfit syscopyarea sysfillrect sysimgblt fb_sys_fops libnvdimm i2c_i801 iTCO_wdt iTCO_vendor_support lpc_ich i2c_smbus virtio_balloon rapl joydev pcspkr drm fuse ext4 mbcache jbd2 sr_mod cdrom sg ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net virtio_blk virtio_console net_failover virtio_scsi failover serio_raw ``` I checked the changelogs of the kernel versions and spotted these three commits that were backported: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… The first one adds changes that look related. Does anybody have experienced this as well or know something about this? Thanks and best regards, Marc

6 7

Ceph Community Management Update
by Josh Durgin 17 Apr '24

17 Apr '24

Hi everyone, I’d like to extend a warm thank you to Mike Perez for his years of service as community manager for Ceph. He is changing focuses now to engineering. The Ceph Foundation board decided to use services from the Linux Foundation to fulfill some community management responsibilities, rather than rely on a single member organization employing a community manager. The Linux Foundation will assist with Ceph Foundation membership and governance matters. Please welcome Noah Lehman (cc’d) as our social media and marketing point person - for anything related to this area, including the Ceph YouTube channel, please reach out to him. Ceph days will continue to be organized and funded by organizations around the world, with the help of the Ceph Ambassadors ( https://ceph.io/en/community/ambassadors/) Gaurav Sitlani (cc’d) will help organize the ambassadors going forward. For other matters, please contact council(a)ceph.io and we’ll direct the matter to the appropriate people. Thanks, Neha Ojha, Dan van der Ster, Josh Durgin Ceph Executive Council

2 1

cephadm custom jinja2 service templates
by Stolte, Felix 17 Apr '24

17 Apr '24

Hi folks, I would like to use a custom jina2 template for an ingress service for rendering the keepalived and haproxy config. Can someone tell me how to override the default templates? Best regards Felix --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschaeftsfuehrung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Karsten Beneke (stellv. Vorsitzender), Dr. Ir. Pieter Jansens, Prof. Dr. Frauke Melchior --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

2 1

(deep-)scrubs blocked by backfill
by Frank Schilder 17 Apr '24

17 Apr '24

Hi all, I have a technical question about scrub scheduling. I replaced a disk and it is back-filling slowly. We have set osd_scrub_during_recovery = true and still observe that scrub times continuously increase (number of PGs not scrubbed in time is continuously increasing). Investigating the situation it looks like any OSD that has a PG in states "backfill_wait" or "backfilling" is preventing scrubs to be scheduled on PGs it is a member of. However, it seems it is not quite like that. On the one hand I have never seen a PG in a state like "active+scrubbing+remapped+backfilling", so backfilling PGs at least never seem to scrub. On the other hand, it seems like more PGs are scrubbed than would be eligible if *all* OSDs with a remapped PG on it would refuse scrubs. It looks like something in between "only OSDs with a backfilling PG block requests for scrub reservations" and "all OSDs with a PG in states backfilling or backfill_wait block requests for scrub reservations". Does the position in the backfill reservation queue play a role? If anyone has insight into how scrub reservations are granted and when not in the situation of an OSD backfilling that would be great. My naive interpretation of "osd_scrub_during_recovery = true" was that scrubs proceed as if no backfill was going on. This, however, is clearly not the case. Having an answer to my question above would help me a lot to get an idea when things will go back to normal. Thanks a lot and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Performance of volume size, not a block size
by Mitsumasa KONDO 17 Apr '24

17 Apr '24

Hi, In AWS EBS gp3, AWS says that small volume size cannot achieve best performance. I think it's a feature or tendency of general distributed storages including Ceph. Is that right in Ceph block storage? I read many docs on ceph community. I never heard of Ceph storage. https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html Regard, -- Mitsumasa KONDO

4 6

feature_map differs across mon_status
by Joel Davidow 17 Apr '24

17 Apr '24

Just curious why the feature_map portions differ in the return of mon_status across a cluster. Below is an example from each of five mons in a healthy 16.2.10 cephadm cluster: root@mon.d:~# ceph tell mon.a mon_status | jq .feature_map { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 42 } ], "client": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "mgr": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ] } root@mon.d:~# ceph tell mon.b mon_status | jq .feature_map { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 36 } ], "client": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 2 } ] } root@mon.d:~# ceph tell mon.c mon_status | jq .feature_map { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 81 } ], "client": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 4 } ], "mgr": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 2 } ] } root@mon.d:~# ceph tell mon.d mon_status | jq .feature_map { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 112 } ], "client": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 7 } ] } root@mon.d:~# ceph tell mon.e mon_status | jq .feature_map { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 88 } ], "client": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 4 } ] }

2 1

cephFS on CentOS7
by Dario Graña 17 Apr '24

17 Apr '24

Hello everyone! We deployed a platform with Ceph Quincy and now we need to give access to some old nodes with CentOS7 until 30/07/2024. I found two approaches, the first one, deploying Ganesha NFS and bringing access through the NFS protocol. The second one is to use an older cephfs client, specifically the Octopus client. I would like to know if there is a third option and which the community would recommend. Thanks in advance. Regards! -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

Working ceph cluster reports large amount of pgs in state unknown/undersized and objects degraded
by Tobias Langner 17 Apr '24

17 Apr '24

We operate a tiny ceph cluster across three machines, each running two OSDs and one of each mds, mgr, and mon. The cluster serves one main erasure-coded (2+1) storage pool and a few other management-related pools. The cluster has been running smoothly for several months. A few weeks ago we noticed a health warning reporting backfillfull/nearfull osds and pools. Here is the output of `ceph -s` at this point (extraced from logs): -------------------------------------------------------------------------------- cluster: health: HEALTH_WARN 1 backfillfull osd(s) 2 nearfull osd(s) Reduced data availability: 163 pgs inactive, 1 pg peering Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull Degraded data redundancy: 1486709/10911157 objects degraded (13.626%), 68 pgs degraded, 68 pgs undersized 162 pgs not scrubbed in time 6 pool(s) backfillfull services: mon: 3 daemons, quorum mon.101,mon.102,mon.100 (age 5m) mgr: mgr-102(active, since 54m), standbys: mgr-101, mgr-100 mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 6 osds: 6 up (since 4m), 6 in (since 2w); 7 remapped pgs data: volumes: 1/1 healthy pools: 6 pools, 338 pgs objects: 3.64M objects, 14 TiB usage: 13 TiB used, 1.7 TiB / 15 TiB avail pgs: 47.929% pgs unknown 0.296% pgs not active 1486709/10911157 objects degraded (13.626%) 52771/10911157 objects misplaced (0.484%) 162 unknown 106 active+clean 67 active+undersized+degraded 1 active+undersized+degraded+remapped+backfill_toofull 1 remapped+peering 1 active+remapped+backfill_toofull -------------------------------------------------------------------------------- I now see the large amount of pgs in state unknown and the fact that a significant fraction of objects is degraded despite all osds being up, but we didn't notice this back then. Because the cluster continued to act fine from an FS access perspective, we didn't really notice the developing problem and did not intervene. From then one, things have mostly gone downwards. Now we `ceph -s` reports the following: -------------------------------------------------------------------------------- cluster: health: HEALTH_WARN noout flag(s) set Reduced data availability: 117 pgs inactive Degraded data redundancy: 2095625/12121767 objects degraded (17.288%), 114 pgs degraded, 114 pgs undersized 117 pgs not scrubbed in time services: mon: 3 daemons, quorum mon.101,mon.102,mon.100 (age 15h) mgr: mgr-102(active, since 7d), standbys: mgr-100, mgr-101 mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 6 osds: 6 up (since 55m), 6 in (since 3w) flags noout data: volumes: 1/1 healthy pools: 6 pools, 338 pgs objects: 4.04M objects, 15 TiB usage: 12 TiB used, 2.8 TiB / 15 TiB avail pgs: 34.615% pgs unknown 2095625/12121767 objects degraded (17.288%) 117 unknown 114 active+undersized+degraded 107 active+clean -------------------------------------------------------------------------------- Note in particular the still very large number of pgs in state unknown, which hasn't changed in days. Same goes for the degraded pgs. Also, the cluster should have around 37TiB storage available but now it only reports 15 TiB. We did a bit of digging around but couldn't really get to the bottom of the unknown pgs and how we can recover from that. One other data point is that the command `ceph osd df tree` gets stuck on two of the three machines and one the one where it returns something, it looks like this: -------------------------------------------------------------------------------- ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 47.67506 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root default -13 18.26408 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - datacenter dc.100 -5 18.26408 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host osd-100 3 hdd 10.91409 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 91 up osd.3 5 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 48 up osd.5 -9 14.69998 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - datacenter dc.101 -7 14.69998 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host osd-101 0 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 83 up osd.0 1 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 86 up osd.1 -11 14.71100 - 15 TiB 12 TiB 12 TiB 77 MiB 21 GiB 2.6 TiB 82.00 1.00 - datacenter dc.102 -17 7.35550 - 7.4 TiB 6.3 TiB 6.2 TiB 16 MiB 11 GiB 1.1 TiB 85.16 1.04 - host osdroid-102-1 4 hdd 7.35550 1.00000 7.4 TiB 6.3 TiB 6.2 TiB 16 MiB 11 GiB 1.1 TiB 85.16 1.04 114 up osd.4 -15 7.35550 - 7.4 TiB 5.8 TiB 5.7 TiB 61 MiB 10 GiB 1.6 TiB 78.83 0.96 - host osdroid-102-2 2 hdd 7.35550 1.00000 7.4 TiB 5.8 TiB 5.7 TiB 61 MiB 10 GiB 1.6 TiB 78.83 0.96 107 up osd.2 TOTAL 15 TiB 12 TiB 12 TiB 77 MiB 21 GiB 2.6 TiB 82.00 MIN/MAX VAR: 0/1.04 STDDEV: 66.97 -------------------------------------------------------------------------------- The odd part here is that for some reason only osd.2 and osd.4 seem to contribute size to the cluster. Interestingly, accessing content from the storage pool works mostly without issues, which shouldn't work if 4 out of 6 OSDs weren't properly up. Any ideas what we can do to get our cluster back into a sane state? I'm happy to provide more logs or command output, please let me know. Thanks!

1 0