I found a warning in Red Hat documentation regarding OSD and RBD: Ceph Block Devices must be deployed on separate nodes from the Ceph Monitor and OSD nodes. Running kernel clients and kernel server daemons on the same node can lead to kernel deadlocks. [1] It was hard to tell from the documentation if this is still true as some Rook documentation [2] mentioned a solution. Is having an RBD on the same node as the OSD's still a no-no or is it OK with the conditions listed in the Rook docs (i.e. patched kernel + no XFS)? TIA! --chuck [1] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/in... [2] https://rook.io/docs/rook/v1.6/ceph-common-issues.html#a-worker-node-using-r...
I prefer to run separately. I collocate only rgw,mgr,mon. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- On 2021. May 25., at 7:52, Tuffli, Chuck <chuck.tuffli@hpe.com> wrote: I found a warning in Red Hat documentation regarding OSD and RBD: Ceph Block Devices must be deployed on separate nodes from the Ceph Monitor and OSD nodes. Running kernel clients and kernel server daemons on the same node can lead to kernel deadlocks. [1] It was hard to tell from the documentation if this is still true as some Rook documentation [2] mentioned a solution. Is having an RBD on the same node as the OSD's still a no-no or is it OK with the conditions listed in the Rook docs (i.e. patched kernel + no XFS)? TIA! --chuck [1] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/in... [2] https://rook.io/docs/rook/v1.6/ceph-common-issues.html#a-worker-node-using-r... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Den tis 25 maj 2021 kl 02:51 skrev Tuffli, Chuck <chuck.tuffli@hpe.com>:
I found a warning in Red Hat documentation regarding OSD and RBD: Ceph Block Devices must be deployed on separate nodes from the Ceph Monitor and OSD nodes. Running kernel clients and kernel server daemons on the same node can lead to kernel deadlocks. [1] It was hard to tell from the documentation if this is still true as some Rook documentation [2] mentioned a solution. Is having an RBD on the same node as the OSD's still a no-no or is it OK with the conditions listed in the Rook docs (i.e. patched kernel + no XFS)? TIA!
I think it warns against a machine hosting data on an OSD while at the same time mounting an RBD which has parts served by the OSD it serves. If you end up with a situation where both parts require lots of memory (during OSD recovery for instance) and the RBD (or the filesystem on top of it) also needs memory, stuff can wedge so both suffer. The same would go for running a mon on the same machine as the OSD, if the OSD memory requirements pushes the machine into swap for instance, then OSD recovery would be slower but probably be "more or less ok", but if the mon process gets swapped out at the same time, it will most certainly not be a good member of the mon quorum and that will potentially lead to even worse issues than a slow OSD recovering. If all things are perfect, you have endless ram and cpu and iops and so on then this might not apply to you, but if you are in the lower 50% of machine resources like half the ceph admins out here, then it might just be a good idea to consider the above warning. -- May the most significant bit of your life be positive.
participants (3)
-
Janne Johansson
-
Szabo, Istvan (Agoda)
-
Tuffli, Chuck