- ceph-users - lists.ceph.io

OSDs failing to start due to crc32 and osdmap error
by Denis Polom 27 Nov '23

27 Nov '23

Hi we have issue to start some OSDs on one node on our Ceph Quincy 17.2.7 cluster. Some OSDs on that node are running fine, but some failing to start. Looks like crc32 checksum error, and failing to get OSD map. I found a some discussions on that but nothing helped. I've also tried to insert current OSD map but that ends with error: # CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap osdmap (#-1:20684533:::osdmap.931991:0#) does not exist. Log is bellow Any ideas please? Thank you From log file: 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public interface 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0# 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load OSD map for epoch 927580, got 0 bytes /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time 2023-11-27T16:01:51.443522+0100 /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x561ad07d2624] 2: ceph-osd(+0xc2e836) [0x561ad07d2836] 3: (OSD::init()+0x4026) [0x561ad08e5a86] 4: main() 5: __libc_start_main() 6: _start() *** Caught signal (Aborted) ** in thread 7f3f17aa13c0 thread_name:ceph-osd 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1 /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time 2023-11-27T16:01:51.443522+0100 /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x561ad07d2624] 2: ceph-osd(+0xc2e836) [0x561ad07d2836] 3: (OSD::init()+0x4026) [0x561ad08e5a86] 4: main() 5: __libc_start_main() 6: _start() ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x561ad07d268c] 5: ceph-osd(+0xc2e836) [0x561ad07d2836] 6: (OSD::init()+0x4026) [0x561ad08e5a86] 7: main() 8: __libc_start_main() 9: _start() 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal (Aborted) ** in thread 7f3f17aa13c0 thread_name:ceph-osd ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x561ad07d268c] 5: ceph-osd(+0xc2e836) [0x561ad07d2836] 6: (OSD::init()+0x4026) [0x561ad08e5a86] 7: main() 8: __libc_start_main() 9: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -558> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public interface -5> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0# -2> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load OSD map for epoch 927580, got 0 bytes -1> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1 /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time 2023-11-27T16:01:51.443522+0100 /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x561ad07d2624] 2: ceph-osd(+0xc2e836) [0x561ad07d2836] 3: (OSD::init()+0x4026) [0x561ad08e5a86] 4: main() 5: __libc_start_main() 6: _start() 0> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal (Aborted) ** in thread 7f3f17aa13c0 thread_name:ceph-osd ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x561ad07d268c] 5: ceph-osd(+0xc2e836) [0x561ad07d2836] 6: (OSD::init()+0x4026) [0x561ad08e5a86] 7: main() 8: __libc_start_main() 9: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -562> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public interface -9> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0# -6> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load OSD map for epoch 927580, got 0 bytes -5> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1 /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time 2023-11-27T16:01:51.443522+0100 /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x561ad07d2624] 2: ceph-osd(+0xc2e836) [0x561ad07d2836] 3: (OSD::init()+0x4026) [0x561ad08e5a86] 4: main() 5: __libc_start_main() 6: _start() -4> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal (Aborted) ** in thread 7f3f17aa13c0 thread_name:ceph-osd ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x561ad07d268c] 5: ceph-osd(+0xc2e836) [0x561ad07d2836] 6: (OSD::init()+0x4026) [0x561ad08e5a86] 7: main() 8: __libc_start_main() 9: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Aborted

3 8

understand "extent"
by Tony Liu 27 Nov '23

27 Nov '23

Hi, The context is RBD on bluestore. I did check extent on Wiki. I see "extent" when talking about snapshot and export/import. For example, when create a snapshot, we mark extents. When there is write to marked extents, we will make a copy. I also know that user data on block device maps to objects. How "extent" and "object" are related? Can I say extent is a set of continuous objects (with default tripe settings)? Thanks! Tony

2 1

About number of osd node can be failed with erasure code 3+2
by tranphong079＠gmail.com 27 Nov '23

27 Nov '23

Hi Groups, Recently I was setting up a ceph cluster with 10 nodes 144 osd, and I use S3 for it with pool erasure code EC3+2 on it. I have a question, how many osd nodes can fail with erasure code 3+2 with cluster working normal (read, write)? and can i choose better erasure code ec7+3, 8+2 etc..? With the erasure code algorithm, it only ensures no data loss, but does not guarantee that the cluster operates normally and does not block IO when osd nodes down. Is that right? Thanks to the community.

2 1

Issue with CephFS (mds stuck in clientreplay status) since upgrade to 18.2.0.
by Lo Re Giuseppe 27 Nov '23

27 Nov '23

Hi, We have upgraded one ceph cluster from 17.2.7 to 18.2.0. Since then we are having CephFS issues. For example this morning: “”” [root@naret-monitor01 ~]# ceph -s cluster: id: 63334166-d991-11eb-99de-40a6b72108d0 health: HEALTH_WARN 1 filesystem is degraded 3 clients failing to advance oldest client/flush tid 3 MDSs report slow requests 6 pgs not scrubbed in time 29 daemons have recently crashed … “”” The ceph orch, ceph crash and ceph fs status commands were hanging. After a “ceph mgr fail” those commands started to respond. Then I have noticed that there was one mds with most of the slow operations, “”” [WRN] MDS_SLOW_REQUEST: 3 MDSs report slow requests mds.cephfs.naret-monitor01.nuakzo(mds.0): 18 slow requests are blocked > 30 secs mds.cephfs.naret-monitor01.uvevbf(mds.1): 1683 slow requests are blocked > 30 secs mds.cephfs.naret-monitor02.exceuo(mds.2): 1 slow requests are blocked > 30 secs “”” Then I tried to restart it with “”” [root@naret-monitor01 ~]# ceph orch daemon restart mds.cephfs.naret-monitor01.uvevbf Scheduled to restart mds.cephfs.naret-monitor01.uvevbf on host 'naret-monitor01' “”” After the cephfs entered into this situation: “”” [root@naret-monitor01 ~]# ceph fs status cephfs - 198 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.naret-monitor01.nuakzo Reqs: 0 /s 17.2k 16.2k 1892 14.3k 1 active cephfs.naret-monitor02.ztdghf Reqs: 0 /s 28.1k 10.3k 752 6881 2 clientreplay cephfs.naret-monitor02.exceuo 63.0k 6491 541 66 3 active cephfs.naret-monitor03.lqppte Reqs: 0 /s 16.7k 13.4k 8233 990 POOL TYPE USED AVAIL cephfs.cephfs.meta metadata 5888M 18.5T cephfs.cephfs.data data 119G 215T cephfs.cephfs.data.e_4_2 data 2289G 3241T cephfs.cephfs.data.e_8_3 data 9997G 470T STANDBY MDS cephfs.naret-monitor03.eflouf cephfs.naret-monitor01.uvevbf MDS version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) “”” The file system is totally unresponsive (we can mount it on client nodes but any operations like a simple ls hangs). During the night we had a lot of mds crashes, I can share the content. Does anybody have an idea on how to tackle this problem? Best, Giuseppe

3 3

Does cephfs ensure close-to-open consistency after enabling lazyio?
by Jianjun Zheng 27 Nov '23

27 Nov '23

2 1

import/export with --export-format 2
by Tony Liu 27 Nov '23

27 Nov '23

Hi, src-image is 1GB (provisioned size). I did the following 3 tests. 1. rbd export src-image - | rbd import - dst-image 2. rbd export --export-format 2 src-image - | rbd import --export-format 2 - dst-image 3. rbd export --export-format 2 src-image - | rbd import - dst-image With #1 and #2, dst-image size (rbd info) is the same as src-image, which is expected. With #3, dst-image size (rbd info) is close to used size (rbd du), not the provisioned size of src-image. I'm not sure if this image is actually useable when write into it. The questions is that, is #3 not supposed to be used at all? I checked doc, didn't see something like "--export-format 2 has to be used for importing the image which is exported with --export-format 2 option". Any comments? Thanks! Tony

2 1

Experience with deduplication
by Szabo, Istvan (Agoda) 27 Nov '23

27 Nov '23

Hi Developers, What is the status of the deduplication for objectsore? I see it under the dev area only since octopus even with the latest release. https://docs.ceph.com/en/octopus/dev/deduplication/ Is it something that can be used in production? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Ceph/daemon container lvm tools don’t work
by Gaël THEROND 27 Nov '23

27 Nov '23

Hi team, I’m experimenting a bit CentOS Stream 9 on our infrastructure as we’re migrating away from CentOS Stream 8. As our deployment model is an hyperconverged one, I have CEPH and OPENSTACK running on the same hosts (OSDs+NOVA/CINDER). That prohibits me to let CEPH nodes running on CentOS Stream 8. However, I’ve noticed an issue related to LVM, which is a bit annoying, when running CentOS Stream 8 based ceph/daemon container over a CentOS Stream 9 node. Indeed, when the ceph-volume lvm batch is processed and perform the lvm subcommand process call, the pvcreate do not persist anymore on the host and so consequently, when you reboot, all ceph related disks disappear as LVM can’t find their PVID or device references. Even a pvscan doesn’t find them. The only workaround that we found out right now is to pvcreate the host disks, pvremove them, dd them and then run the installation process again, from my understanding it kinda warmup the host lvm system and consequently let it keep up with the ceph-volume run later on. For additional information: We deploy the cluster using ceph-ansible stable-6.0 branch (pacific). The container come from quay.io and is 6.0.11 CentOS Stream 8 release. This release is working perfectly on a CentOS Stream 8 based host. One weird thing that we catched is that on a CentOS Stream 8 host using this CentOS Stream 8 based image, lvm/dm are creating new dm block devices named after ceph vg/lv where on a CentOS Stream 9 based host, lvm/dm create the new vg/lv as symlink to dm-xY dm block devices. And last question, is there any planned build of the ceph/daemon:6.0.11 or greater image based on CentOS Stream 9 ? Thanks a lot! PS: I did forget the [ceph-users] tag at first post and don’t know if it’s something important for the mailing distribution.

1 0

Where is a simple getting started guide for a very basic cluster?
by Leo28C 27 Nov '23

27 Nov '23

I'm pulling my hair trying to get a simple cluster going. I first tried Gluster but I have an old system that can't handle the latest version, so I resorted to Ceph. However, I can't get my cluster to work. I tried to find tutorials but everything uses tools on top of Ceph, whereas I'm trying to use it barebones. (I don't trust nfs or samba for this purpose) The three systems I'm trying to cluster together have a variety of Linux distros, and one of them is discontinued, so I can't use a lot of newer stuff. I need to make do with the commands `ceph`, `ceph-mon`, and `ceph-volume`. I do *not* have access to `ceph-disk` which I ran into a lot online. ChatGPT guided me up to a certain point before she started making stuff up, so I need to start from scratch. I just need to share a directory (or rather a block device in the case of Ceph) across these three systems that already have a secure connection to one another. Can somebody please help me out? Thanks in advance

3 2

About lifecycle of RGW
by VÔ VI 27 Nov '23

27 Nov '23

Hi community, I'm running a large s3 running with ingress and backend rgw, total traffic of my cluster ~30Gbps real time traffic. I' am have a problem with lifecycle in rgw, object can't delete with following log Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 1 lifecycle: ERROR: publishing notification failed, with error: -2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067328.134916_15b007878cf011ee960ba1c41ff4eaf9_3007777.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067328.134916_15b007878cf011ee960ba1c41ff4eaf9_3007777.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 1 lifecycle: ERROR: publishing notification failed, with error: -2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067327.693377_14a3adc68cf011ee960ba1c41ff4eaf9_3007677.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067327.693377_14a3adc68cf011ee960ba1c41ff4eaf9_3007677.test (2) No such file or directory wp_thrd: 4, 2 My cluster setting rgw_lc_max_worker = 6 rgw_lc_debug_interval=60 (setting during testing) I am testing bucket lifecycle expire every 5 minute and 1 minute radosgw-lc list of bucket is processing "bucket": ":ec3204cam05:f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2", "started": "Mon, 27 Nov 2023 07:29:58 GMT", "status": "PROCESSING" s3cmd getlifecycle s3://ec3204cam04 <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>Expire after 5 day</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>5</Days> </Expiration> </Rule> </LifecycleConfiguration> s3cmd getlifecycle s3://ec3204cam05 <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>Expire after 1 day</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>1</Days> </Expiration> </Rule> </LifecycleConfiguration> I don't know why the rgw show log can't delete object with no such file, how i can find the problem, I hope someone can help me Thanks

1 0