Reef release candidate - v18.1.2
Hi everyone, This is the second release candidate for Reef. The Reef release comes with a new RockDB version (7.9.2) [0], which incorporates several performance improvements and features. Our internal testing doesn't show any side effects from the new version, but we are very eager to hear community feedback on it. This is the first release to have the ability to tune RockDB settings per column family [1], which allows for more granular tunings to be applied to different kinds of data stored in RocksDB. A new set of settings has been used in Reef to optimize performance for most kinds of workloads with a slight penalty in some cases, outweighed by large improvements in use cases such as RGW, in terms of compactions and write amplification. We would highly encourage community members to give these a try against their performance benchmarks and use cases. The detailed list of changes in terms of RockDB and BlueStore can be found in https://pad.ceph.com/p/reef-rc-relnotes. If any of our community members would like to help us with performance investigations or regression testing of the Reef release candidate, please feel free to provide feedback via email or in https://pad.ceph.com/p/reef_scale_testing. For more active discussions, please use the #ceph-at-scale slack channel in ceph-storage.slack.com. This RC has gone thru partial testing due to issues we are experiencing in the sepia lab. Please try it out and report any issues you encounter. Happy testing! Thanks, YuriW
On 6/30/23 18:36, Yuri Weinstein wrote:
This RC has gone thru partial testing due to issues we are experiencing in the sepia lab. Please try it out and report any issues you encounter. Happy testing!
I tested the RC (v18.1.2) this afternoon. I tried out the new "read balancer". I hit asserts after applying the "ceph osd pg-upmap-primary $pd-id" commands on all affected OSDs. I posted (or tried at least) a ceph crash report: "2023-07-10T14:40:33.087472Z_c279b11c-7a69-488f-bd19-ed11cc4b0553" "assert_condition": "pg_upmap_primaries.empty()", "assert_file": "/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4795-g2f6e4f7d/rpm/el8/BUILD/ceph-18.0.0-4795-g2f6e4f7d/src/osd/OSDMap.cc", "assert_func": "void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const", "assert_line": 3251, "assert_msg": "/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4795-g2f6e4f7d/rpm/el8/BUILD/ceph-18.0.0-4795-g2f6e4f7d/src/osd/OSDMap.cc: In function 'void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const' thread 7f9c4ca18700 time 2023-07-10T14:40:33.074045+0000\n/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4795-g2f6e4f7d/rpm/el8/BUILD/ceph-18.0.0-4795-g2f6e4f7d/src/osd/OSDMap.cc: 3251: FAILED ceph_assert(pg_upmap_primaries.empty())\n", Shall I create a tracker issue for this? Gr. Stefan
Hi Stefan, Yes, please create a tracker. I will take a look at the issue, Thanks, Laura Flores On Mon, Jul 10, 2023 at 10:50 AM Stefan Kooman <stefan@bit.nl> wrote:
On 6/30/23 18:36, Yuri Weinstein wrote:
This RC has gone thru partial testing due to issues we are experiencing in the sepia lab. Please try it out and report any issues you encounter. Happy testing!
I tested the RC (v18.1.2) this afternoon. I tried out the new "read balancer". I hit asserts after applying the "ceph osd pg-upmap-primary $pd-id" commands on all affected OSDs. I posted (or tried at least) a ceph crash report: "2023-07-10T14:40:33.087472Z_c279b11c-7a69-488f-bd19-ed11cc4b0553"
"assert_condition": "pg_upmap_primaries.empty()", "assert_file":
"/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4795-g2f6e4f7d/rpm/el8/BUILD/ceph-18.0.0-4795-g2f6e4f7d/src/osd/OSDMap.cc", "assert_func": "void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const", "assert_line": 3251, "assert_msg": "/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4795-g2f6e4f7d/rpm/el8/BUILD/ceph-18.0.0-4795-g2f6e4f7d/src/osd/OSDMap.cc:
In function 'void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const' thread 7f9c4ca18700 time 2023-07-10T14:40:33.074045+0000\n/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4795-g2f6e4f7d/rpm/el8/BUILD/ceph-18.0.0-4795-g2f6e4f7d/src/osd/OSDMap.cc:
3251: FAILED ceph_assert(pg_upmap_primaries.empty())\n",
Shall I create a tracker issue for this?
Gr. Stefan
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
On 6/30/23 18:36, Yuri Weinstein wrote:
This RC has gone thru partial testing due to issues we are experiencing in the sepia lab. Please try it out and report any issues you encounter. Happy testing!
If I install cephadm from package, 18.1.2 on ubuntu focal in my case, cepadm usages the ceph-ci/ceph:main container images: "Pulling container image quay.ceph.io/ceph-ci/ceph:main". And these container images are out of date (18.0.0-4869-g05e449f9 (05e449f9a2a65c297f31628af8f01f63cf36f261) reef (dev)": 1). AFAIK there is no way to tell cephadm bootstrap to use a specific version. Although the help mentions "--allow-mismatched-release", so it might be possible? Gr. Stefan
Can you elaborate on how you installed cephadm? When I pull from quay.io/ceph/ceph:v18.1.2, I see the version v18.1.2 podman run -it quay.io/ceph/ceph:v18.1.2 Trying to pull quay.io/ceph/ceph:v18.1.2... Getting image source signatures Copying blob f3a0532868dc done Copying blob 9ba8dbcf96c4 done Copying config 3b66ad272b done Writing manifest to image destination Storing signatures [root@66c274be11ab /]# ceph --version ceph version 18.1.2 (a5c951305c2409669162c235d81981bdc60dd9e7) reef (rc) On Wed, Jul 12, 2023 at 2:06 PM Stefan Kooman <stefan@bit.nl> wrote:
On 6/30/23 18:36, Yuri Weinstein wrote:
This RC has gone thru partial testing due to issues we are experiencing in the sepia lab. Please try it out and report any issues you encounter. Happy testing!
If I install cephadm from package, 18.1.2 on ubuntu focal in my case, cepadm usages the ceph-ci/ceph:main container images: "Pulling container image quay.ceph.io/ceph-ci/ceph:main". And these container images are out of date (18.0.0-4869-g05e449f9 (05e449f9a2a65c297f31628af8f01f63cf36f261) reef (dev)": 1).
AFAIK there is no way to tell cephadm bootstrap to use a specific version. Although the help mentions "--allow-mismatched-release", so it might be possible?
Gr. Stefan
On 7/12/23 23:21, Yuri Weinstein wrote:
Can you elaborate on how you installed cephadm?
Add ceph repo (mirror): cat /etc/apt/sources.list.d/ceph.list deb http://ceph.download.bit.nl/debian-18.1.2 focal main wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add - apt update apt install cephadm It install cephadm 18.1.2 cephadm bootstrap --mon-ip $ip Then it pulls "quay.ceph.io/ceph-ci/ceph main" Instead of 18.1.2 container image. Gr. Stefan
When I pull from quay.io/ceph/ceph:v18.1.2, I see the version v18.1.2
podman run -it quay.io/ceph/ceph:v18.1.2 Trying to pull quay.io/ceph/ceph:v18.1.2... Getting image source signatures Copying blob f3a0532868dc done Copying blob 9ba8dbcf96c4 done Copying config 3b66ad272b done Writing manifest to image destination Storing signatures [root@66c274be11ab /]# ceph --version ceph version 18.1.2 (a5c951305c2409669162c235d81981bdc60dd9e7) reef (rc)
On Wed, Jul 12, 2023 at 2:06 PM Stefan Kooman <stefan@bit.nl> wrote:
On 6/30/23 18:36, Yuri Weinstein wrote:
This RC has gone thru partial testing due to issues we are experiencing in the sepia lab. Please try it out and report any issues you encounter. Happy testing!
If I install cephadm from package, 18.1.2 on ubuntu focal in my case, cepadm usages the ceph-ci/ceph:main container images: "Pulling container image quay.ceph.io/ceph-ci/ceph:main". And these container images are out of date (18.0.0-4869-g05e449f9 (05e449f9a2a65c297f31628af8f01f63cf36f261) reef (dev)": 1).
AFAIK there is no way to tell cephadm bootstrap to use a specific version. Although the help mentions "--allow-mismatched-release", so it might be possible?
Gr. Stefan
Thanks for the report - this is being fixed in https://github.com/ceph/ceph/pull/52343 On Wed, Jul 12, 2023 at 2:53 PM Stefan Kooman <stefan@bit.nl> wrote:
On 7/12/23 23:21, Yuri Weinstein wrote:
Can you elaborate on how you installed cephadm?
Add ceph repo (mirror): cat /etc/apt/sources.list.d/ceph.list deb http://ceph.download.bit.nl/debian-18.1.2 focal main
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add - apt update apt install cephadm
It install cephadm 18.1.2
cephadm bootstrap --mon-ip $ip
Then it pulls "quay.ceph.io/ceph-ci/ceph main"
Instead of 18.1.2 container image.
Gr. Stefan
When I pull from quay.io/ceph/ceph:v18.1.2, I see the version v18.1.2
podman run -it quay.io/ceph/ceph:v18.1.2 Trying to pull quay.io/ceph/ceph:v18.1.2... Getting image source signatures Copying blob f3a0532868dc done Copying blob 9ba8dbcf96c4 done Copying config 3b66ad272b done Writing manifest to image destination Storing signatures [root@66c274be11ab /]# ceph --version ceph version 18.1.2 (a5c951305c2409669162c235d81981bdc60dd9e7) reef (rc)
On Wed, Jul 12, 2023 at 2:06 PM Stefan Kooman <stefan@bit.nl> wrote:
On 6/30/23 18:36, Yuri Weinstein wrote:
This RC has gone thru partial testing due to issues we are experiencing in the sepia lab. Please try it out and report any issues you encounter. Happy testing!
If I install cephadm from package, 18.1.2 on ubuntu focal in my case, cepadm usages the ceph-ci/ceph:main container images: "Pulling container image quay.ceph.io/ceph-ci/ceph:main". And these container images are out of date (18.0.0-4869-g05e449f9 (05e449f9a2a65c297f31628af8f01f63cf36f261) reef (dev)": 1).
AFAIK there is no way to tell cephadm bootstrap to use a specific version. Although the help mentions "--allow-mismatched-release", so it might be possible?
Gr. Stefan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Josh Durgin
-
Laura Flores
-
Stefan Kooman
-
Yuri Weinstein