November 2023 - ceph-users

by Szabo, Istvan (Agoda)

Hi Developers, What is the status of the deduplication for objectsore? I see it under the dev area only since octopus even with the latest release. https://docs.ceph.com/en/octopus/dev/deduplication/ Is it something that can be used in production? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 months, 3 weeks

1
0
0 0

Ceph/daemon container lvm tools don’t work

by Gaël THEROND

Hi team, I’m experimenting a bit CentOS Stream 9 on our infrastructure as we’re migrating away from CentOS Stream 8. As our deployment model is an hyperconverged one, I have CEPH and OPENSTACK running on the same hosts (OSDs+NOVA/CINDER). That prohibits me to let CEPH nodes running on CentOS Stream 8. However, I’ve noticed an issue related to LVM, which is a bit annoying, when running CentOS Stream 8 based ceph/daemon container over a CentOS Stream 9 node. Indeed, when the ceph-volume lvm batch is processed and perform the lvm subcommand process call, the pvcreate do not persist anymore on the host and so consequently, when you reboot, all ceph related disks disappear as LVM can’t find their PVID or device references. Even a pvscan doesn’t find them. The only workaround that we found out right now is to pvcreate the host disks, pvremove them, dd them and then run the installation process again, from my understanding it kinda warmup the host lvm system and consequently let it keep up with the ceph-volume run later on. For additional information: We deploy the cluster using ceph-ansible stable-6.0 branch (pacific). The container come from quay.io and is 6.0.11 CentOS Stream 8 release. This release is working perfectly on a CentOS Stream 8 based host. One weird thing that we catched is that on a CentOS Stream 8 host using this CentOS Stream 8 based image, lvm/dm are creating new dm block devices named after ceph vg/lv where on a CentOS Stream 9 based host, lvm/dm create the new vg/lv as symlink to dm-xY dm block devices. And last question, is there any planned build of the ceph/daemon:6.0.11 or greater image based on CentOS Stream 9 ? Thanks a lot! PS: I did forget the [ceph-users] tag at first post and don’t know if it’s something important for the mailing distribution.

5 months, 3 weeks

1
0
0 0

Where is a simple getting started guide for a very basic cluster?

by Leo28C

I'm pulling my hair trying to get a simple cluster going. I first tried Gluster but I have an old system that can't handle the latest version, so I resorted to Ceph. However, I can't get my cluster to work. I tried to find tutorials but everything uses tools on top of Ceph, whereas I'm trying to use it barebones. (I don't trust nfs or samba for this purpose) The three systems I'm trying to cluster together have a variety of Linux distros, and one of them is discontinued, so I can't use a lot of newer stuff. I need to make do with the commands `ceph`, `ceph-mon`, and `ceph-volume`. I do *not* have access to `ceph-disk` which I ran into a lot online. ChatGPT guided me up to a certain point before she started making stuff up, so I need to start from scratch. I just need to share a directory (or rather a block device in the case of Ceph) across these three systems that already have a secure connection to one another. Can somebody please help me out? Thanks in advance

5 months, 3 weeks

3
2
0 0

About lifecycle of RGW

by VÔ VI

Hi community, I'm running a large s3 running with ingress and backend rgw, total traffic of my cluster ~30Gbps real time traffic. I' am have a problem with lifecycle in rgw, object can't delete with following log Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 1 lifecycle: ERROR: publishing notification failed, with error: -2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067328.134916_15b007878cf011ee960ba1c41ff4eaf9_3007777.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067328.134916_15b007878cf011ee960ba1c41ff4eaf9_3007777.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 1 lifecycle: ERROR: publishing notification failed, with error: -2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067327.693377_14a3adc68cf011ee960ba1c41ff4eaf9_3007677.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067327.693377_14a3adc68cf011ee960ba1c41ff4eaf9_3007677.test (2) No such file or directory wp_thrd: 4, 2 My cluster setting rgw_lc_max_worker = 6 rgw_lc_debug_interval=60 (setting during testing) I am testing bucket lifecycle expire every 5 minute and 1 minute radosgw-lc list of bucket is processing "bucket": ":ec3204cam05:f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2", "started": "Mon, 27 Nov 2023 07:29:58 GMT", "status": "PROCESSING" s3cmd getlifecycle s3://ec3204cam04 <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>Expire after 5 day</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>5</Days> </Expiration> </Rule> </LifecycleConfiguration> s3cmd getlifecycle s3://ec3204cam05 <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>Expire after 1 day</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>1</Days> </Expiration> </Rule> </LifecycleConfiguration> I don't know why the rgw show log can't delete object with no such file, how i can find the problem, I hope someone can help me Thanks

5 months, 3 weeks

1
0
0 0

Re: openstack Vm shutoff by itself

by AJ_ sunny

++adding @ceph-users-confirm+4555fdc6282a38c849f4d27a40339f1b7e4bde74@ceph.io <ceph-users-confirm+4555fdc6282a38c849f4d27a40339f1b7e4bde74(a)ceph.io> ++Adding dev(a)ceph.io Thanks,&, Regards Arihant Jain On Mon, 27 Nov, 2023, 7:48 am AJ_ sunny, <jains8550(a)gmail.com> wrote: > Hi team, > > After doing above changes I am still getting the issue in which machine > continuously went shutdown > > In nova-compute logs I am getting only this footprint > > Logs:- > 2023-10-16 08:48:10.971 7 WARNING nova.compute.manager > [req-c7b731db-2b61-400e-917f-8645c9984696 f226d81a45dd46488fb2e19515 848 > 316d215042914de190f5f9e1c8466bf0 default default] [instance: > 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] Received unexpected - vent > network-vif-plugged-f191f6c8-dff5-4c1b-94b3-8d91aa6ff5ac for instance with > vm_state active and task_state None. 2023-10-21 22:42:44.589 7 INFO > nova.compute.manager [-] [instance: 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] > VM Stopped (Lifecyc Event) > > 2023-10-21 22:42:44.683 7 INFO nova.compute.manager > [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] [instance: 4b04d3f1- > fbd-4b63-b693-a0ef316ecff3] During _sync_instance_power_state the DB > power_state (1) does not match the vm_power_state from ti e hypervisor (4). > Updating power_state in the DB to match the hypervisor. > > 2023-10-21 22:42:44.811 7 WARNING nova.compute.manager > [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d ----] [instance: 4b04d3f > 1-1fbd-4b63-b693-a0ef316ecff3] Instance shutdown by itself. Calling the > stop API. Current vm_state: active, current task_state : None, original DB > power_state: 1, current VM power_state: 4 2023-10-21 22:42:44.977 7 INFO > nova.compute.manager [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] > [instance: 4b04d3f1-1 > > fbd-4b63-b693-a0ef316ecff3] Instance is already powered off in the > hypervisor when stop is called. > > > And in this architecture we are using ceph is the backend storage for > Nova,glance & cinder > When machine auto goes down and if i try to start the machine it will go > in error i.e. in Vm console is show I/O ERROR during boot so first we need > to rebuild the volume from ceph side then I have to start the machine > Rbd object-map rebuild<volume-id> > Openstack server start <server-id> > > So this issue is showing two faces one from ceph side and another from > nova-compute log > can someone please help me out to fix out this issue asap > > Thanks & Regards > Arihant Jain > > On Tue, 24 Oct, 2023, 4:56 pm , <smooney(a)redhat.com> wrote: > >> On Tue, 2023-10-24 at 10:11 +0530, AJ_ sunny wrote: >> > Hi team, >> > >> > Vm is not shutting off by owner from inside its automatically went to >> > shutdown i.e. libvirt lifecycle stop event triggering >> > In my nova.conf configuration I am using ram_allocation_ratio = 1.5 >> > And previously I tried to set in nova.conf >> > Sync_power_state_interval = -1 but still facing the same problem >> > OOM might be causing this issue >> > Can you please give me some idea to fix this issue if OOM is the cause >> the general answer is swap. >> >> nova should alwasy be deployed with swap even if you do not have over >> commit enabled. >> there are a few reason for this the first being python allocates memory >> diffently if >> any swap is aviable, even 1G is enough to have it not try to commit all >> memory. so >> when swap is aviable the nova/neutron agents will use much less resident >> memeory even with >> out usign any of the swap space. >> >> we have some docs about this downstream >> >> https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17… >> >> if you are being ultra conservative we recommend allocating (ram * >> allocation ratio) in swap so in your case allcoate >> 1.5 times your ram as swap. we woudl expect the actul useage of swap to >> be a small fraction of that however so we >> also provide a formula for >> >> overcommit_ratio = NovaRAMAllocationRatio - 1 >> Minimum swap size (MB) = (total_RAM * overcommit_ratio) + >> RHEL_min_swap >> Recommended swap size (MB) = total_RAM * (overcommit_ratio + >> percentage_of_RAM_to_use_for_swap) >> >> so say your host had 64G of ram with an allocation ratio of 1.5 and a min >> swap percentaiong of 25% >> the conserviver swap recommentation would be >> >> (64*(0.5+0.25)) + disto_min_swap >> (64*0.75) + 4G = 52G of recommended swap >> >> if your wondering why we add a min swap precentage and disto min swap its >> basically to acocund for the >> Qemu and host OS memory overhead as well as the memory used by the >> nova/neutron agents and libvirt/ovs >> >> >> if your not using memory over commit my general recommdation is if you >> have less then 64G of ram allcoate 16G if you >> have more then 256G of ram allocate 64G and you should be fine. when you >> do use memofy over commit you must >> have at least enouch swap to account for the qemu overhead of all >> instance + the over committed memory. >> >> >> the other common cause of OOM errors is if you are using numa affinity >> and the guest dont request >> hw:mem_page_size=<something> without setting a mem_page_size request we >> dont do numa aware memory placement. the kernel >> OOM system works >> on a per numa node basis, numa affintiy does not supprot memory over >> commit either so that is likly not your issue. >> i jsut said i woudl mention it to cover all basis. >> >> regards >> sean >> >> >> >> > >> > >> > Thanks & Regards >> > Arihant Jain >> > >> > On Mon, 23 Oct, 2023, 11:29 pm , <smooney(a)redhat.com> wrote: >> > >> > > On Mon, 2023-10-23 at 13:19 -0400, Jonathan Proulx wrote: >> > > > >> > > > I've seen similar log traces with overcommitted memory when the >> > > > hypervisor runs out of physical memory and OOM killer gets the VM >> > > > process. >> > > > >> > > > This is an unusuall configuration (I think) but if the VM owner >> claims >> > > > they didn't power down the VM internally you might look at the local >> > > > hypevisor logs to see if the VM process crashed or was killed for >> some >> > > > other reason. >> > > yep OOM events are one common causes fo this. >> > > >> > > nova is bacialy just saying "hay you said this vm should be active >> but its >> > > not, im going to update the db to reflect >> > > reality." you can turn that off with >> > > >> > > >> https://docs.openstack.org/nova/latest/configuration/config.html#workaround… >> > > or >> > > >> > > >> https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.sy… >> > > either disabel the sync via setign the interval to -1 >> > > or disable haneling the virt lifecycle events. >> > > >> > > i would recommend the sync_power_state_interval approach but again if >> vms >> > > are stopping >> > > and you dont know why you likely should discover why rahter then just >> > > turning if the update of the nova db to reflect >> > > the actual sate. >> > > >> > > > >> > > > -Jon >> > > > >> > > > On Mon, Oct 23, 2023 at 02:02:26PM +0100, smooney(a)redhat.com wrote: >> > > > :On Mon, 2023-10-23 at 17:45 +0530, AJ_ sunny wrote: >> > > > :> Hi team, >> > > > :> >> > > > :> I am using openstack kolla ansible on wallaby version and >> currently I >> > > am >> > > > :> facing issue with virtual machine, vm is shutoff by itself and >> and >> > > from log >> > > > :> it seems libvirt lifecycle stop event triggering again and again >> > > > :> >> > > > :> Logs:- >> > > > :> 2023-10-16 08:48:10.971 7 WARNING nova.compute.manager >> > > > :> [req-c7b731db-2b61-400e-917f-8645c9984696 >> f226d81a45dd46488fb2e19515 >> > > 848 >> > > > :> 316d215042914de190f5f9e1c8466bf0 default default] [instance: >> > > > :> 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] Received unexpected - vent >> > > > :> network-vif-plugged-f191f6c8-dff5-4c1b-94b3-8d91aa6ff5ac for >> instance >> > > with >> > > > :> vm_state active and task_state None. 2023-10-21 22:42:44.589 7 >> INFO >> > > > :> nova.compute.manager [-] [instance: >> > > 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] >> > > > :> VM Stopped (Lifecyc Event) >> > > > :> >> > > > :> 2023-10-21 22:42:44.683 7 INFO nova.compute.manager >> > > > :> [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] [instance: 4b04d3f1- >> > > > :> fbd-4b63-b693-a0ef316ecff3] During _sync_instance_power_state >> the DB >> > > > :> power_state (1) does not match the vm_power_state from ti e >> > > hypervisor (4). >> > > > :> Updating power_state in the DB to match the hypervisor. >> > > > :> >> > > > :> 2023-10-21 22:42:44.811 7 WARNING nova.compute.manager >> > > > :> [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d ----] [instance: >> 4b04d3f >> > > > :> 1-1fbd-4b63-b693-a0ef316ecff3] Instance shutdown by itself. >> Calling >> > > the >> > > > :> stop API. Current vm_state: active, current task_state : None, >> > > original DB >> > > > :> power_state: 1, current VM power_state: 4 2023-10-21 >> 22:42:44.977 7 >> > > INFO >> > > > :> nova.compute.manager [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] >> > > > :> [instance: 4b04d3f1-1 >> > > > :> >> > > > :> fbd-4b63-b693-a0ef316ecff3] Instance is already powered off in >> the >> > > > :> hypervisor when stop is called. >> > > > : >> > > > :that sounds like the guest os shutdown the vm. >> > > > :i.e. somethign in the guest ran sudo poweroff >> > > > :then nova detected teh vm was stoped by the user and updated its >> db to >> > > match >> > > > :that. >> > > > : >> > > > :that is the expected beahvior wehn you have the power sync enabled. >> > > > :it is enabled by default. >> > > > :> >> > > > :> >> > > > :> Thanks & Regards >> > > > :> Arihant Jain >> > > > :> +91 8299719369 >> > > > : >> > > > >> > > >> > > >> >>

5 months, 3 weeks

1
1
0 0

Quincy patch upgrade

by Ben

Dear Cephers, With improvement on dmclock in 17.2.7, we are considering upgrade from 17.2.5 to 17.2.7. seeing this which is worrysome: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/SG7CKALU3AI… any suggestions for an easy upgrade? Regards, Ben

5 months, 3 weeks

1
0
0 0

Full cluster outage when ECONNREFUSED is triggered

by Denis Krienbühl

Hi We’ve recently had a serious outage at work, after a host had a network problem: - We rebooted a single host in a cluster of fifteen hosts across three racks. - The single host had a bad network configuration after booting, causing it to send some packets to the wrong network. - One network still worked and offered a connection to the mons. - The other network connection was bad. Packets were refused, not dropped. - Due to osd_fast_fail_on_connection_refused=true, the broken host forced the mons to take all other OSDs down (immediate failure). - Only after shutting down the faulty host, was it possible to start the shut down OSDs, to restore the cluster. We have since solved the problem by removing the default route that caused the packets to end up in the wrong network, where they were summarily rejected by a firewall. That is, we made sure that packets would be dropped in the future, not rejected. Still, I figured I’ll send this experience of ours to this mailing list, as this seems to be something others might encounter as well. In the following PR, that introduced osd_fast_fail_on_connection_refused, there’s this description: > This changeset adds additional handler (handle_refused()) to the dispatchers > and code that detects when connection attempt fails with ECONNREFUSED error > (connection refused) which is a clear indication that host is alive, but > daemon isn't, so daemons can instantly mark the other side as undoubtly > downed without the need for grace timer. And this comment: > As for flapping, we discussed it on ceph-devel ml > and came to conclusion that it requires either broken firewall or network > configuration to cause this, and these are more serious issues that should > be resolved first before worrying about OSDs flapping (either way, flapping > OSDs could be good for getting someone's attention). https://github.com/ceph/ceph/pull/8558https://github.com/ceph/ceph/pull/8558 It has left us wondering if these are the right assumptions. An ECONNREFUSED condition can bring down a whole cluster, and I wonder if there should be some kind of safe-guard to ensure that this is avoided. One badly configured host should generally not be able do that, and if the packets are dropped, instead of refused, the cluster notices that the OSD down reports come only from one host, and acts accordingly. What do you think? Does this warrant a change in Ceph? I’m happy to provide details and create a ticket. Cheers, Denis

5 months, 3 weeks

4
9
0 0

[CEPH] Ceph multi nodes failed

by Nguyễn Hữu Khôi

Hello guys. I see many docs and threads talking about osd failed. I have a question: how many nodes in a cluster can be failed. I am using ec 8 + 2(10 osd nodes) and when I shutdown 2 nodes then my cluster crashes, It cannot write anymore. Thank you. Regards Nguyen Huu Khoi

5 months, 3 weeks

4
6
0 0

Object size

by Miroslav Svoboda

Hi, please, is it better to reduce the default object size from 4MB to some smaller value for the rbd image where there will be a lot of small mail and webhosting files? Thanks Svoboda Miroslav

5 months, 3 weeks

2
1
0 0

CEPH Daemon container CentOS Stream 8 over CentOS Stream 9 host

by Gaël THEROND

Hi team, I’m experimenting a bit CentOS Stream 9 on our infrastructure as we’re migrating away from CentOS Stream 8. As our deployment model is an hyperconverged one, I have CEPH and OPENSTACK running on the same hosts (OSDs+NOVA/CINDER). That prohibits me to let CEPH nodes running on CentOS Stream 8. However, I’ve noticed an issue related to LVM, which is a bit annoying, when running CentOS Stream 8 based ceph/daemon container over a CentOS Stream 9 node. Indeed, when the ceph-volume lvm batch is processed and perform the lvm subcommand process call, the pvcreate do not persist anymore on the host and so consequently, when you reboot, all ceph related disks disappear as LVM can’t find their PVID or device references. Even a pvscan doesn’t find them. The only workaround that we found out right now is to pvcreate the host disks, pvremove them, dd them and then run the installation process again, from my understanding it kinda warmup the host lvm system and consequently let it keep up with the ceph-volume run later on. For additional information: We deploy the cluster using ceph-ansible stable-6.0 branch (pacific). The container come from quay.io and is 6.0.11 CentOS Stream 8 release. This release is working perfectly on a CentOS Stream 8 based host. One weird thing that we catched is that on a CentOS Stream 8 host using this CentOS Stream 8 based image, lvm/dm are creating new dm block devices named after ceph vg/lv where on a CentOS Stream 9 based host, lvm/dm create the new vg/lv as symlink to dm-xY dm block devices. And last question, is there any planned build of the ceph/daemon:6.0.11 or greater image based on CentOS Stream 9 ? Thanks a lot!

5 months, 3 weeks

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users November 2023