December 2023 - Dev - lists.ceph.io

Restricted access to /teuthology on teuthology.front.sepia.ceph.com

by Patrick Donnelly

Hi folks, As many of you are aware, RAM/CPU is particularly scarce on teuthology.front.sepia.ceph.com which means that any log viewing of QA results can cause swapping and general slowness. At today's Ceph Infrastructure Weekly call [1], I proposed only allowing root, www-data, and teuthworker access to /teuthology so that others would be forced to look at logs on other beefier development machines like senta [2], vossi [3], your personal workstation [4], or even some temporary locked node [5]. The current plan is to proceed barring some reasonable justification not to. There are a few to-do items to make it happen laid out in the minutes. Until then, admins of teuthology would appreciate it if you already start moving your log viewing to other machines without waiting for a technical barrier to be set up. [1] https://pad.ceph.com/p/ceph-infra-weekly [2] https://wiki.sepia.ceph.com/doku.php?id=hardware:senta [3] https://wiki.sepia.ceph.com/doku.php?id=hardware:vossi [4] https://wiki.sepia.ceph.com/doku.php?id=services:cephfs [5] https://wiki.sepia.ceph.com/doku.php?id=testnodeaccess -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 days, 13 hours

1
1
0 0

arrow submodule version

by Casey Bodley

hey Gal and Eric, in today's standup, we discussed the version of our apache arrow submodule. it's currently pinned at 6.0.1, which was tagged in nov. 2021. the centos9 builds are using the system package libarrow-devel-9.0.0. arrow's upstream recently tagged an 11.0.0 release as far as i know, there still aren't any system packages for ubuntu, so we're likely to be stuck with the submodule for quite a while. how do guys want to handle these updates? is it worth trying to update before the reef release?

3 months, 1 week

4
7
0 0

CLT Meeting Minutes 2023-12-20

by Laura Flores

(Zac Dover) "make check" fails now even for some docs builds. For example: https://github.com/ceph/ceph/pull/54970, which is a simple edit of ReStructured Text in doc/radosgw/compression.rst. Greg Farnum and Dan Mick have already done preliminary investigation of this matter here: https://ceph-storage.slack.com/archives/C1HFJ4VTN/p1703048785756359. - Follow Slack thread for updates; we'll continue looking into it Still 38 PRs to scrub for 16.2.15: https://github.com/ceph/ceph/pulls?q=is%3Aopen+is%3Apr+milestone%3Apacific - Looking for PRs that are necessary in the release, as well as non-trivial PRs that have been open for a while -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

5 months

1
0
0 0

Fwd: Can not activate some OSDs after upgrade (bad crc on label)

by Huseyin Cotuk

Hi Cephers, Any idea about this? Best regards, Huseyin Cotuk hcotuk(a)gmail.com Begin forwarded message: > > From: Huseyin Cotuk <hcotuk(a)gmail.com> > Subject: Can not activate some OSDs after upgrade (bad crc on label) > Date: 19 December 2023 at 14:09:20 GMT+3 > To: ceph-users(a)ceph.io > > Hello Cephers, > > I have two identical Ceph clusters with 32 OSDs each, running radosgw with EC. They were running Octopus on Ubuntu 20.04. > > On one of these clusters, I have upgraded OS to Ubuntu 22.04 and Ceph version is upgraded to Quincy 17.2.6. This cluster completed the process without any issue and it works as expected. > > On the second cluster, I followed the same procedure and upgraded the cluster. After upgrade 9 of 32 OSDs can not be activated. AFAIU, the label of these OSDs can not be read. ceph-volume lvm activate {osd.id} {osd_fsid} command fails as below: > > stderr: failed to read label for /dev/ceph-block-13/block-13: (5) Input/output error > stderr: 2023-12-19T11:46:25.310+0300 7f088cd7ea80 -1 bluestore(/dev/ceph-block-13/block-13) _read_bdev_label bad crc on label, expected 2340927273 != actual 2067505886 > > All ceph-bluestore-tool and ceph-object-storetool commands fail with the same message, so I can not try repair, fsck or migrate. > > # ceph-bluestore-tool repair --deep yes --path /var/lib/ceph/osd/ceph-13/ > failed to load os-type: (2) No such file or directory > 2023-12-19T13:57:06.551+0300 7f39b1635a80 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label bad crc on label, expected 2340927273 != actual 2067505886 > > I also tried show label with bluestore-tool without success. > > # ceph-bluestore-tool show-label --dev /dev/ceph-block-13/block-13 > unable to read label for /dev/ceph-block-13/block-13: (5) Input/output error > 2023-12-19T14:01:19.668+0300 7fdcdd111a80 -1 bluestore(/dev/ceph-block-13/block-13) _read_bdev_label bad crc on label, expected 2340927273 != actual 2067505886 > > I can get the information including osd_fsif, block_uuid of all failed OSDs via ceph-volume lvm list like below. > > ====== osd.13 ====== > > [block] /dev/ceph-block-13/block-13 > > block device /dev/ceph-block-13/block-13 > block uuid jFaTba-ln5r-muQd-7Ef9-3tWe-JwvO-qW9nqi > cephx lockbox secret > cluster fsid 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f > cluster name ceph > crush device class None > encrypted 0 > osd fsid c9ee3ef6-73d7-4029-9cd6-086cc95d2f27 > osd id 13 > osdspec affinity > type block > vdo 0 > devices /dev/mapper/mpathb > > All vgs and lvs look healthy. > > # lvdisplay ceph-block-13/block-13 > --- Logical volume --- > LV Path /dev/ceph-block-13/block-13 > LV Name block-13 > VG Name ceph-block-13 > LV UUID jFaTba-ln5r-muQd-7Ef9-3tWe-JwvO-qW9nqi > LV Write Access read/write > LV Creation host, time ank-backup01, 2023-11-29 10:41:53 +0300 > LV Status available > # open 0 > LV Size <7.28 TiB > Current LE 1907721 > Segments 1 > Allocation inherit > Read ahead sectors auto > - currently set to 256 > Block device 253:33 > > This is a single node cluster running only radosgw. The environment is as follows: > > # ceph -v > ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) > > # lsb_release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 22.04.3 LTS > Release: 22.04 > Codename: jammy > > # ceph osd crush rule dump > [ > { > "rule_id": 0, > "rule_name": "osd_replicated_rule", > "type": 1, > "steps": [ > { > "op": "take", > "item": -2, > "item_name": "default~hdd" > }, > { > "op": "choose_firstn", > "num": 0, > "type": "osd" > }, > { > "op": "emit" > } > ] > }, > { > "rule_id": 2, > "rule_name": "default.rgw.buckets.data", > "type": 3, > "steps": [ > { > "op": "set_chooseleaf_tries", > "num": 5 > }, > { > "op": "set_choose_tries", > "num": 100 > }, > { > "op": "take", > "item": -1, > "item_name": "default" > }, > { > "op": "choose_indep", > "num": 0, > "type": "osd" > }, > { > "op": "emit" > } > ] > } > ] > > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 226.29962 root default > -3 226.29962 host ank-backup01 > 0 hdd 7.29999 osd.0 up 1.00000 1.00000 > 1 hdd 7.29999 osd.1 up 1.00000 1.00000 > 2 hdd 7.29999 osd.2 up 1.00000 1.00000 > 3 hdd 7.29999 osd.3 up 1.00000 1.00000 > 4 hdd 7.29999 osd.4 up 1.00000 1.00000 > 5 hdd 7.29999 osd.5 up 1.00000 1.00000 > 6 hdd 7.29999 osd.6 up 1.00000 1.00000 > 7 hdd 7.29999 osd.7 up 1.00000 1.00000 > 8 hdd 7.29999 osd.8 up 1.00000 1.00000 > 9 hdd 7.29999 osd.9 up 1.00000 1.00000 > 10 hdd 7.29999 osd.10 up 1.00000 1.00000 > 11 hdd 7.29999 osd.11 up 1.00000 1.00000 > 12 hdd 7.29999 osd.12 down 0 1.00000 > 13 hdd 7.29999 osd.13 down 0 1.00000 > 14 hdd 7.29999 osd.14 down 0 1.00000 > 15 hdd 7.29999 osd.15 down 0 1.00000 > 16 hdd 7.29999 osd.16 down 0 1.00000 > 17 hdd 7.29999 osd.17 down 0 1.00000 > 18 hdd 7.29999 osd.18 down 0 1.00000 > 19 hdd 7.29999 osd.19 down 0 1.00000 > 20 hdd 7.29999 osd.20 down 0 1.00000 > 21 hdd 7.29999 osd.21 up 1.00000 1.00000 > 22 hdd 7.29999 osd.22 up 1.00000 1.00000 > 23 hdd 7.29999 osd.23 up 1.00000 1.00000 > 24 hdd 7.29999 osd.24 up 1.00000 1.00000 > 25 hdd 7.29999 osd.25 up 1.00000 1.00000 > 26 hdd 7.29999 osd.26 up 1.00000 1.00000 > 27 hdd 7.29999 osd.27 up 1.00000 1.00000 > 28 hdd 7.29999 osd.28 up 1.00000 1.00000 > 29 hdd 7.29999 osd.29 up 1.00000 1.00000 > 30 hdd 7.29999 osd.30 up 1.00000 1.00000 > 31 hdd 7.29999 osd.31 up 1.00000 1.00000 > > Does anybody have any idea why the labels of these OSDs can not be read? Any help would be appreciated. > > Best Regards, > Huseyin Cotuk > hcotuk(a)gmail.com > > > >

5 months

2
1
0 0

No User + Dev Monthly Meetup this week - Happy Holidays!

by Laura Flores

Hi all, A quick reminder that the User + Dev Monthly Meetup that was scheduled for this week December 21 is cancelled due to the holidays. The User + Dev Monthly Meetup will resume in the new year on January 18. If you have a topic you'd like to present at an upcoming meetup, you're welcome to submit it here: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… Wishing everyone a happy holiday season! Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

5 months

1
0
0 0

v18.2.1 Reef released

by Yuri Weinstein

We're happy to announce the 1st backport release in the Reef series. This is the first backport release in the Reef series, and the first with Debian packages, for Debian Bookworm. We recommend all users update to this release. https://ceph.io/en/news/blog/2023/v18-2-1-reef-released/ Notable Changes --------------- * RGW: S3 multipart uploads using Server-Side Encryption now replicate correctly in multi-site. Previously, the replicas of such objects were corrupted on decryption. A new tool, ``radosgw-admin bucket resync encrypted multipart``, can be used to identify these original multipart uploads. The ``LastModified`` timestamp of any identified object is incremented by 1ns to cause peer zones to replicate it again. For multi-site deployments that make any use of Server-Side Encryption, we recommended running this command against every bucket in every zone after all zones have upgraded. * CEPHFS: MDS evicts clients which are not advancing their request tids which causes a large buildup of session metadata resulting in the MDS going read-only due to the RADOS operation exceeding the size threshold. `mds_session_metadata_threshold` config controls the maximum size that a (encoded) session metadata can grow. * RGW: New tools have been added to radosgw-admin for identifying and correcting issues with versioned bucket indexes. Historical bugs with the versioned bucket index transaction workflow made it possible for the index to accumulate extraneous "book-keeping" olh entries and plain placeholder entries. In some specific scenarios where clients made concurrent requests referencing the same object key, it was likely that a lot of extra index entries would accumulate. When a significant number of these entries are present in a single bucket index shard, they can cause high bucket listing latencies and lifecycle processing failures. To check whether a versioned bucket has unnecessary olh entries, users can now run ``radosgw-admin bucket check olh``. If the ``--fix`` flag is used, the extra entries will be safely removed. A distinct issue from the one described thus far, it is also possible that some versioned buckets are maintaining extra unlinked objects that are not listable from the S3/ Swift APIs. These extra objects are typically a result of PUT requests that exited abnormally, in the middle of a bucket index transaction - so the client would not have received a successful response. Bugs in prior releases made these unlinked objects easy to reproduce with any PUT request that was made on a bucket that was actively resharding. Besides the extra space that these hidden, unlinked objects consume, there can be another side effect in certain scenarios, caused by the nature of the failure mode that produced them, where a client of a bucket that was a victim of this bug may find the object associated with the key to7fe91d5d5842e04be3b4f514d6dd990c54b29c76 be in an inconsistent state. To check whether a versioned bucket has unlinked entries, users can now run ``radosgw-admin bucket check unlinked``. If the ``--fix`` flag is used, the unlinked objects will be safely removed. Finally, a third issue made it possible for versioned bucket index stats to be accounted inaccurately. The tooling for recalculating versioned bucket stats also had a bug, and was not previously capable of fixing these inaccuracies. This release resolves those issues and users can now expect that the existing ``radosgw-admin bucket check`` command will produce correct results. We recommend that users with versioned buckets, especially those that existed on prior releases, use these new tools to check whether their buckets are affected and to clean them up accordingly. * mgr/snap-schedule: For clusters with multiple CephFS file systems, all the snap-schedule commands now expect the '--fs' argument. * RADOS: A POOL_APP_NOT_ENABLED health warning will now be reported if the application is not enabled for the pool irrespective of whether the pool is in use or not. Always add ``application`` label to a pool to avoid reporting of POOL_APP_NOT_ENABLED health warning for that pool. The user might temporarilty mute this warning using ``ceph health mute POOL_APP_NOT_ENABLED``. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.2.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 7fe91d5d5842e04be3b4f514d6dd990c54b29c76

5 months

1
0
0 0

pacific 16.2.15

by Yuri Weinstein

Hello We are finalizing the remaining issues for the last pacific release 16.2.15. Dev leads, please reply with issues that you plan to add. TIA

5 months, 1 week

2
1
0 0

Ceph Leadership Team meeting notes 2023-12-13

by Neha Ojha

Highlights from this week's CLT meeting Release updates: 18.2.1 - QE validation done, gibba lab upgrade complete without issue, LRC ran into issues with unexpected cephadm exception and one monitor down - being investigated with no known impact on services yet 16.2.15 - https://github.com/ceph/ceph/milestone/17 in progress General updates: - Squid kickoff PR(https://github.com/ceph/ceph/pull/53191) going through QE validation across suites, will be merged before the holidays - MDSMap decode issue (18.2.0/1 user-space cephfs clients broken with squid+) has been discovered, CephFS team will work on a plan of action to address it (Venky to send a detailed email to the dev list about the issue) - FYI, a minor change to vstart is being merged https://github.com/ceph/ceph/pull/53393 - Ceph Days NYC being planned by Bloomberg - it will be nice to announce the Squid release at the event (current ETA April 2024) Thanks, Neha

5 months, 1 week

1
0
0 0

mds.0.journaler.pq(ro) _finish_read got error -2

by Eugen Block

Hi, I'm trying to help someone with a broken CephFS. We managed to recover basic ceph functionality but the CephFS is still inaccessible (currently read-only). We went through the disaster recovery steps but to no avail. Here's a snippet from the startup logs: ---snip--- mds.0.41 Booting: 2: waiting for purge queue recovered mds.0.journaler.pq(ro) _finish_probe_end write_pos = 14797504512 (header had 14789452521). recovered. mds.0.purge_queue operator(): open complete mds.0.purge_queue operator(): recovering write_pos monclient: get_auth_request con 0x55c280bc5c00 auth_method 0 monclient: get_auth_request con 0x55c280ee0c00 auth_method 0 mds.0.journaler.pq(ro) _finish_read got error -2 mds.0.purge_queue _recover: Error -2 recovering write_pos mds.0.purge_queue _go_readonly: going readonly because internal IO failed: No such file or directory mds.0.journaler.pq(ro) set_readonly mds.0.41 unhandled write error (2) No such file or directory, force readonly... mds.0.cache force file system read-only force file system read-only ---snip--- I've added the dev mailing list, maybe someone can give some advice how to continue from here (we could try to recover with an empty metadata pool). Or is this FS lost? Thanks! Eugen

5 months, 1 week

2
2
0 0

Announcing go-ceph v0.25.0

by Anoop C S

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.25.0 More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S

5 months, 1 week

1
0
0 0

2024

2023

2022

2021

2020

2019

Dev December 2023