Hi,
Today we had a big issue with slow ops on the nvme drives which holding the index pool.
Why the nvme shows full if on ceph is barely utilized? Which one I should belive?
When I check the ceph osd df it shows 10% usage of the osds (1x 2TB nvme drive has 4x osds on it):
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
195 nvme 0.43660 1.00000 447 GiB 47 GiB 161 MiB 46 GiB 656 MiB 400 GiB 10.47 0.21 64 up
252 nvme 0.43660 1.00000 447 GiB 46 GiB 161 MiB 45 GiB 845 MiB 401 GiB 10.35 0.21 64 up
253 nvme 0.43660 1.00000 447 GiB 46 GiB 229 MiB 45 GiB 662 MiB 401 GiB 10.26 0.21 66 up
254 nvme 0.43660 1.00000 447 GiB 46 GiB 161 MiB 44 GiB 1.3 GiB 401 GiB 10.26 0.21 65 up
255 nvme 0.43660 1.00000 447 GiB 47 GiB 161 MiB 46 GiB 1.2 GiB 400 GiB 10.58 0.21 64 up
288 nvme 0.43660 1.00000 447 GiB 46 GiB 161 MiB 44 GiB 1.2 GiB 401 GiB 10.25 0.21 64 up
289 nvme 0.43660 1.00000 447 GiB 46 GiB 161 MiB 45 GiB 641 MiB 401 GiB 10.33 0.21 64 up
290 nvme 0.43660 1.00000 447 GiB 45 GiB 229 MiB 44 GiB 668 MiB 402 GiB 10.14 0.21 65 up
However in nvme list it says full:
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 90D0A00XTXTR KCD6XLUL1T92 1 1.92 TB / 1.92 TB 512 B + 0 B GPK6
/dev/nvme1n1 60P0A003TXTR KCD6XLUL1T92 1 1.92 TB / 1.92 TB 512 B + 0 B GPK6
With some other node the test was like:
* if none of the disk full, no slow ops.
* If 1x disk full and the other not, has slow ops but not too much
* if none of the disk full, no slow ops.
The full disks are very highly utilized during recovery and they are holding back the operations from the other nvmes.
What's the reason that even if the pgs are the same in the cluster +/-1 regarding space they are not equally utilized.
Thank you
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Hey Cephers,
Hope you're all doing well! I'm in a bit of a pickle and could really use
some of your power.
Here's the scoop:
I have a setup with around 10 HDDs. and 2 NVME's (+uninteresting boot
disks)
My initial goal was to configure part of the HDDs (6 out of 7TB) into an
md0 or similar device to be used as a DB device. (the rest would be nice to
use as an nvme osd)
I made some clumsy attempts to set them up "right"
While the OSDs are getting deployed, they are not being shown back to the
dashboard.
The specific error when running `ceph orch device ls`: 'Insufficient space
(<10 extents) on vgs, LVM detected, locked.'
Given these, I have a few questions:
Are there specific configurations or steps that I might be missing when
setting up the DB device with multiple HDDs?
(rel; I currently try things like this
https://paste.openstack.org/show/bdPHXQ0BMypWnZTYosT2/ )
Could the error message indicate a particular issue with my current setup
or approach?
If anyone has successfully configured a similar setup, could you please
share your insights or steps taken?
Thanks a bunch!
Cheers,
The Upstream Documentation Team is writing a Beginner's Guide. If you're reading this email, then you are invited to contribute to it.
I have a firm idea of what I want the Beginner's Guide to contain, but it is important to me to consult the Ceph community during the early stages of the composition of the Beginner's Guide, to make sure that we create a Beginner's Guide that is useful.
The first section of the Beginner's Guide will be a brief overview of the components of a Ceph cluster. This section will be brief enough to be read in just a couple of minutes.
The second section of the Beginner's Guide will be a set of reference architectures. This section will describe the differences between the different kinds of clusters. For example, a CephFS cluster requires Monitors, Managers, OSDs, and also Metadata Servers. An object storage cluster requires Monitors, Managers, OSDs, and a RADOS Gateway. The second section of the Beginner's Guide will describe the various kinds of storage clusters and the pieces that each of the kinds requires. After reading this section, a beginner will know for example which kind of cluster requires a metadata server (it's a CephFS cluster).
I am considering writing a third section of the Beginner's Guide that addresses early misconceptions about Ceph. This section would consist of things that people wish they had known when they started using Ceph. The User+Dev Monthly Meetup of November 2023 generated a beefy list of these things, and here are some of them: different storage types (block, file, object) are all built on RADOS, but they don't interact with each other--you can't treat your S3 bucket the way you treat a CephFS directory; there is a difference between a RADOS object and an S3 object. This section would also answer some Ceph-beginner questions: just what is the CRUSH map and what does it do?; which components of a Ceph cluster can run in VMs and which must run on bare metal?; what is the difference between consumer grade hardware and enterprise grade hardware?
I expect to have finished writing the Beginner's Guide by Cephalocon 2024.
Please tell me if there is something that you'd like to see in a Ceph Beginner's Guide that I haven't mentioned.
Zac Dover
Upstream DocumentationCeph Foundation
Hi,
I'm trying to build a DEBUG version of Ceph Reef on a virtual Ubuntu-LTS
22.04 running on Lima by following the README on Ceph's github repo. The
build failed and the last CMake error was ""g++-11: error: unrecognized
command-line option '-Wimplicit-const-int-float-conversion'". Does anyone
know what I can do to fix the compilation error? I could try different gcc
versions, but I'd assume Ceph's build scripts would install and verify all
the dependencies. Thanks,
The system configuration is as follows:
> uname -a
Linux lima-ceph-dev 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49
UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
I followed the instructions on the README in Ceph's github repo:
https://github.com/ceph/ceph, and the command ./do_cmake.sh failed at step
[137/2150] that builds frontend dashboard with the error message "ninja:
build stopped: subcommand failed." The last error logged in the file
CMakeError.log has to do with "g++-11: error: unrecognized command-line
option '-Wimplicit-const-int-float-conversion'".
Below the last error message on the CMakeError.log:
Performing C++ SOURCE FILE Test
COMPILER_SUPPORTS_WARN_IMPLICIT_CONST_INT_FLOAT_CONVERSION failed with the
following output:
Change Dir: /home/dyuan.linux/ceph/build/CMakeFiles/CMakeTmp
Run Build Command(s):/usr/bin/ninja cmTC_bab6d && [1/2] Building CXX object
CMakeFiles/cmTC_bab6d.dir/src.cxx.o
FAILED: CMakeFiles/cmTC_bab6d.dir/src.cxx.o
/usr/bin/g++-11
-DCOMPILER_SUPPORTS_WARN_IMPLICIT_CONST_INT_FLOAT_CONVERSION -fPIE
-Wimplicit-const-int-float-conversion -std=c++20 -o
CMakeFiles/cmTC_bab6d.dir/src.cxx.o -c
/home/dyuan.linux/ceph/build/CMakeFiles/CMakeTmp/src.cxx
g++-11: error: unrecognized command-line option
'-Wimplicit-const-int-float-conversion'
ninja: build stopped: subcommand failed.
I think I ran in to this before but I forget the fix:
HEALTH_WARN 1 stray host(s) with 1 daemon(s) not managed by cephadm
[WRN] CEPHADM_STRAY_HOST: 1 stray host(s) with 1 daemon(s) not managed by cephadm
stray host cn06.ceph.fu.intra has 1 stray daemons: ['mon.cn03']
Pacific 16.2.11
How do I clear this?
Thanks
-jeremy
Hi community,
I have a problem with my osd, i want to show dump_historic_ops of osd
I follow the guide:
https://www.ibm.com/docs/en/storage-fusion/2.6?topic=alerts-cephosdslowops
But when i run command
ceph daemon osd.8 dump_historic_ops show the error, the command run on node
with osd.8
Can't get admin socket path: unable to get conf option admin_socket for
osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid
types are: auth, mon, osd, mds, mgr, client\n"
I am running ceph cluster reef version by cephadmin install
What should I do?
Thank you.
Hi community,
My cluster running with 10 nodes and 2 nodes goes down, sometimes the log
shows the slow ops, what is the root cause?
My osd is HDD and block.db and wal is 500GB SSD per osd.
Health check update: 13 slow ops, oldest one blocked for 167 sec, osd.10
has slow ops (SLOW_OPS)
Thanks to the community.
Dear Ceph users,
I am replacing some small disks on one of my hosts with bigger ones. I
delete the OSD from the web UI, preserving the ID for replacement, then
after the rebalancing is finished I change the disk and the cluster
automatically re-creates the OSD with the same ID. Then I adjust the
CRUSH weight.
Everything works fine except for the handling of the device ID of some
of the the new disks. As you can see below there are 5 IDs associated to
2 devices and 2 OSDs, while these are actually different disks since
OSDs see different and corrects sizes.
[ceph: root@bofur /]# ceph device ls-by-host romolo
DEVICE DEV DAEMONS
EXPECTED FAILURE
AMCC_9650SE-16M_DISK_82723576349B5E000984 sdc osd.42
AMCC_9650SE-16M_DISK_83214021349B63000A50 sdd osd.56
AMCC_9650SE-16M_DISK_83450671349B680004B3 sdf osd.68
AMCC_9650SE-16M_DISK_83471183349B680021DA sde osd.65
AMCC_9650SE-16M_DISK_9QG58JCX349B590000EE sdb osd.13
AMCC_9650SE-16M_DISK_AF248795608D6A000016 sdq osd.62
AMCC_9650SE-16M_DISK_J0210858000000000000 sdi sdn osd.105 osd.20
AMCC_9650SE-16M_DISK_J0210926000000000000 sdg sdl osd.36 osd.5
AMCC_9650SE-16M_DISK_N0ECFHAL000000000000 sdj sdo osd.25 osd.60
AMCC_9650SE-16M_DISK_N0R5P9WT000000000000 sdk sdp osd.51 osd.70
AMCC_9650SE-16M_DISK_PBGDG6EE000000000000 sdh sdm osd.45 osd.9
SanDisk_SSD_PLUS_21089P443002 sda mon.romolo
I really don't understand what happened, if I did something wrong, or
how to fix this.
Any help is greatly appreciated.
Nicola
