- ceph-users - lists.ceph.io

Full cluster outage when ECONNREFUSED is triggered
by Denis Krienbühl 24 Nov '23

24 Nov '23

Hi We’ve recently had a serious outage at work, after a host had a network problem: - We rebooted a single host in a cluster of fifteen hosts across three racks. - The single host had a bad network configuration after booting, causing it to send some packets to the wrong network. - One network still worked and offered a connection to the mons. - The other network connection was bad. Packets were refused, not dropped. - Due to osd_fast_fail_on_connection_refused=true, the broken host forced the mons to take all other OSDs down (immediate failure). - Only after shutting down the faulty host, was it possible to start the shut down OSDs, to restore the cluster. We have since solved the problem by removing the default route that caused the packets to end up in the wrong network, where they were summarily rejected by a firewall. That is, we made sure that packets would be dropped in the future, not rejected. Still, I figured I’ll send this experience of ours to this mailing list, as this seems to be something others might encounter as well. In the following PR, that introduced osd_fast_fail_on_connection_refused, there’s this description: > This changeset adds additional handler (handle_refused()) to the dispatchers > and code that detects when connection attempt fails with ECONNREFUSED error > (connection refused) which is a clear indication that host is alive, but > daemon isn't, so daemons can instantly mark the other side as undoubtly > downed without the need for grace timer. And this comment: > As for flapping, we discussed it on ceph-devel ml > and came to conclusion that it requires either broken firewall or network > configuration to cause this, and these are more serious issues that should > be resolved first before worrying about OSDs flapping (either way, flapping > OSDs could be good for getting someone's attention). https://github.com/ceph/ceph/pull/8558https://github.com/ceph/ceph/pull/8558 It has left us wondering if these are the right assumptions. An ECONNREFUSED condition can bring down a whole cluster, and I wonder if there should be some kind of safe-guard to ensure that this is avoided. One badly configured host should generally not be able do that, and if the packets are dropped, instead of refused, the cluster notices that the OSD down reports come only from one host, and acts accordingly. What do you think? Does this warrant a change in Ceph? I’m happy to provide details and create a ticket. Cheers, Denis

4 9

[CEPH] Ceph multi nodes failed
by Nguyễn Hữu Khôi 24 Nov '23

24 Nov '23

Hello guys. I see many docs and threads talking about osd failed. I have a question: how many nodes in a cluster can be failed. I am using ec 8 + 2(10 osd nodes) and when I shutdown 2 nodes then my cluster crashes, It cannot write anymore. Thank you. Regards Nguyen Huu Khoi

4 6

Object size
by Miroslav Svoboda 24 Nov '23

24 Nov '23

Hi, please, is it better to reduce the default object size from 4MB to some smaller value for the rbd image where there will be a lot of small mail and webhosting files? Thanks Svoboda Miroslav

2 1

CEPH Daemon container CentOS Stream 8 over CentOS Stream 9 host
by Gaël THEROND 24 Nov '23

24 Nov '23

Hi team, I’m experimenting a bit CentOS Stream 9 on our infrastructure as we’re migrating away from CentOS Stream 8. As our deployment model is an hyperconverged one, I have CEPH and OPENSTACK running on the same hosts (OSDs+NOVA/CINDER). That prohibits me to let CEPH nodes running on CentOS Stream 8. However, I’ve noticed an issue related to LVM, which is a bit annoying, when running CentOS Stream 8 based ceph/daemon container over a CentOS Stream 9 node. Indeed, when the ceph-volume lvm batch is processed and perform the lvm subcommand process call, the pvcreate do not persist anymore on the host and so consequently, when you reboot, all ceph related disks disappear as LVM can’t find their PVID or device references. Even a pvscan doesn’t find them. The only workaround that we found out right now is to pvcreate the host disks, pvremove them, dd them and then run the installation process again, from my understanding it kinda warmup the host lvm system and consequently let it keep up with the ceph-volume run later on. For additional information: We deploy the cluster using ceph-ansible stable-6.0 branch (pacific). The container come from quay.io and is 6.0.11 CentOS Stream 8 release. This release is working perfectly on a CentOS Stream 8 based host. One weird thing that we catched is that on a CentOS Stream 8 host using this CentOS Stream 8 based image, lvm/dm are creating new dm block devices named after ceph vg/lv where on a CentOS Stream 9 based host, lvm/dm create the new vg/lv as symlink to dm-xY dm block devices. And last question, is there any planned build of the ceph/daemon:6.0.11 or greater image based on CentOS Stream 9 ? Thanks a lot!

1 0

[MDS] mds stuck in laggy state, CephFS unusable
by kvesligaj＠cs.hr 24 Nov '23

24 Nov '23

Hi, we're having a peculiar issue which we found out during HA/DR testing in our Ceph cluster. Basic info about cluster: Version: Quincy (17.2.6) 5 nodes configured in stretch cluster (2 DCs with one arbiter node which is also admin node for the cluster) On every node beside the admin node we have OSD and MON services. We have 3 MGR instances in cluster. Specific thing that we wanted to test is multiple CephFS with each having multiple MDS (with HA in mind). We deployed MDS on every node, increased max_mds to 2 for every CephFS and other two MDS-es are in standby-replay mode (they are automatically configured during CephFS creation to follow specific CephFS - join_fscid). We did multiple tests and when we have only one CephFS it behaves as expected (two MDS are in up:active state and clients can connect and interact with CephFS as if nothing has happened). When we test with multiple CephFS (two for example) and we shutdown two nodes one of MDS is stuck in up:active laggy state and when this happens the CephFS for which this happens is unusable, client hangs and it is stuck like that until we power on other DC. This happens even when there are no clients connected to this specific CephFS. We can provide additional logs and do any tests necessary. We checked the usual culprits and our nodes don't show any excessive CPU or memory usage. We would appreciate any help.

1 0

CephFS - MDS removed from map - filesystem keeps to be stopped
by Denis Polom 24 Nov '23

24 Nov '23

Hi running Ceph Pacific 16.2.13. we had full CephFS filesystem and after adding new HW we tried to start it but our MDS daemons are pushed to be standby and are removed from MDS map. Filesystem was broken, so we repaired it with: # ceph fs fail cephfs # cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary # cephfs-journal-tool --rank=cephfs:0 journal reset then I started ceph-mds service and marked rank as repaired mds after some time has switched to standby. Log is bellow. I would appreciate any help to resolve this situation. Thank you. from log: 2023-11-22T14:11:49.212+0100 7f5dc155e700 1 mds.0.9604 handle_mds_map i am now mds.0.9604 2023-11-22T14:11:49.212+0100 7f5dc155e700 1 mds.0.9604 handle_mds_map state change up:rejoin --> up:active 2023-11-22T14:11:49.212+0100 7f5dc155e700 1 mds.0.9604 recovery_done -- successful recovery! 2023-11-22T14:11:49.212+0100 7f5dc155e700 1 mds.0.9604 active_start 2023-11-22T14:11:49.216+0100 7f5dc155e700 1 mds.0.9604 cluster recovered. 2023-11-22T14:11:49.216+0100 7f5dc3d63700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.8.127:0/2123529386 conn(0x55a60627a800 0x55a606e5b000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0). handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4564700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.6.88:0/1899426587 conn(0x55a60627ac00 0x55a6070d0000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).h andle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4564700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.4.216:0/2058542052 conn(0x55a6070c9800 0x55a6070d1800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0). handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc3d63700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.4.220:0/1549374180 conn(0x55a60708d000 0x55a6070d0800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0). handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4d65700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.8.180:0/270666178 conn(0x55a60703a000 0x55a6070cf800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).h andle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4d65700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.8.178:0/3673271488 conn(0x55a6070c9400 0x55a6070d1000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0). handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4d65700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.4.167:0/2667964940 conn(0x55a6070c9c00 0x55a607112000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0). handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc3d63700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.6.70:0/3181830075 conn(0x55a607116000 0x55a607112800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).h andle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc4564700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.245.6.72:0/3744737352 conn(0x55a60627a800 0x55a606e5b000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).h andle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.216+0100 7f5dc3d63700 0 --1- [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] >> v1:10.244.18.140:0/1607447464 conn(0x55a60627ac00 0x55a6070d0000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0) .handle_connect_message_2 accept peer reset, then tried to connect to us, replacing 2023-11-22T14:11:49.220+0100 7f5dc155e700 1 mds.mds1 Updating MDS map to version 9608 from mon.1 2023-11-22T14:11:49.220+0100 7f5dc155e700 1 mds.0.9604 handle_mds_map i am now mds.0.9604 2023-11-22T14:11:49.220+0100 7f5dc155e700 1 mds.0.9604 handle_mds_map state change up:active --> up:stopping 2023-11-22T14:11:52.412+0100 7f5dc3562700 1 mds.mds1 asok_command: client ls {prefix=client ls} (starting...) 2023-11-22T14:11:57.412+0100 7f5dc3562700 1 mds.mds1 asok_command: client ls {prefix=client ls} (starting...) 2023-11-22T14:12:02.416+0100 7f5dc3562700 1 mds.mds1 asok_command: client ls {prefix=client ls} (starting...) 2023-11-22T14:12:07.420+0100 7f5dc3562700 1 mds.mds1 asok_command: client ls {prefix=client ls} (starting...) 2023-11-22T14:12:12.420+0100 7f5dc3562700 1 mds.mds1 asok_command: client ls {prefix=client ls} (starting...) 2023-11-22T14:12:13.552+0100 7f5dc155e700 1 mds.mds1 Updating MDS map to version 9609 from mon.1 2023-11-22T14:12:13.552+0100 7f5dc155e700 1 mds.mds1 Map removed me [mds.mds1{0:5320528} state up:stopping seq 67 addr [v2:10.245.4.103:6800/1548097835,v1:10.245.4.103:6801/1548097835] compat {c=[1],r=[1],i=[7ff]}] from cluster; respawnin g! See cluster/monitor logs for details. 2023-11-22T14:12:13.552+0100 7f5dc155e700 1 mds.mds1 respawn!

2 1

CLT Meeting minutes 2023-11-23
by Nizamudeen A 23 Nov '23

23 Nov '23

Hello, etherpad history lost need a way to recover from DB or find another way to back things up discuss the quincy/dashboard-v3 backports? was tabled from 11/1 https://github.com/ceph/ceph/pull/54252 agreement is to not backport breaking features to stable branches. 18.2.1 LRC upgrade affected by https://tracker.ceph.com/issues/62570#note-4 https://ceph-storage.slack.com/archives/C1HFJ4VTN/p1700575544548809 Figure out the reproducer and add tests https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… is the trigger on the kernel client side? Clients with that patch should work with the server-side code that broke older ones Suggestion to introduce a matrix of pre-built "older" kernels into the fs suite gibba vs LRC upgrades LRC shouldn't be updated often, it should act more like a production cluster RCs for reef, quincy and pacific for next week when there is more time to discuss Regards, -- Nizamudeen A Software Engineer Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

3 2

Erasure vs replica
by Albert Shih 23 Nov '23

23 Nov '23

Hi everyone, I just like to know what's your opinion about the reliability of erasure coding. Of course I can understand if we want the «best of the best of the best» ;-) I can choose the replica method. I heard in many location “replica” are more reliable, “replica” are more efficient etc... Yes...well since 25 years I'm using raid (5, 6, lvm, raidz1, raidz2, etc.) I never loose data only once when a firmware bug in some xxxxx card crash the raid volume. Now 25 years later lot of people recommend to use replica so if I buy XTo I'm only going to have X/3 To (vs raidz2 where I loose 2 disks over 9-12 disks). So my question are : Anyone use in large scale erasure coding for critical (same level as raidz1/raid5 ou raidz2/raid6) ? Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 23 nov. 2023 14:51:28 CET

4 3

cephadm vs ceph.conf
by Albert Shih 23 Nov '23

23 Nov '23

Hi everyone. Still me with my newbie question....sorry. I'm using cephadm to deploy my ceph cluster, but when I search in the documentation «docs.ceph.com» I see in some place like https://docs.ceph.com/en/latest/rados/configuration/pool-pg-config-ref/ to change something in the /etc/ceph/ceph.conf. How that is taking account by cephadm ? I see in the docker container they have a overlay for /etc/ceph/ceph.conf. Should I modify the ceph.conf (vi/emacs) directly ? Should I modify the ceph.conf (vi/emacs) directly and restart something ? Should I use some cephadm shell and don't manually touche ceph.conf ? And what about the future ? Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 23 nov. 2023 15:21:47 CET

4 6

How to use hardware
by Albert Shih 23 Nov '23

23 Nov '23

Hi everyone, In the purpose to deploy a medium size of ceph cluster (300 OSD) we have 6 bare-metal server for the OSD, and 5 bare-metal server for the service (MDS, Mon, etc.) Those 5 bare-metal server have each 48 cores and 256 Gb. What would be the smartest way to use those 5 server, I see two way : first : Server 1 : MDS,MON, grafana, prometheus, webui Server 2: MON Server 3: MON Server 4 : MDS Server 5 : MDS so 3 MDS, 3 MON. and we can loose 2 servers. Second KVM on each server Server 1 : 3 VM : One for grafana & CIe, and 1 MDS, 2 MON other server : 1 MDS, 1 MON in total : 5 MDS, 5 MON and we can loose 4 servers. So on paper it's seem the second are smarter, but it's also more complex, so my question are «is it worth the complexity to have 5 MDS/MON for 300 OSD». Important : The main goal of this ceph cluster are not to get the maximum I/O speed, I would not say the speed is not a factor, but it's not the main point. Regards. -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 17 nov. 2023 10:49:27 CET

5 8