- ceph-users - lists.ceph.io

v17.2.7 Quincy released
by Yuri Weinstein 12 Nov '23

12 Nov '23

We're happy to announce the 7th backport release in the Quincy series. https://ceph.io/en/news/blog/2023/v17-2-7-quincy-released/ Notable Changes --------------- * `ceph mgr dump` command now displays the name of the Manager module that registered a RADOS client in the `name` field added to elements of the `active_clients` array. Previously, only the address of a module's RADOS client was shown in the `active_clients` array. * mClock Scheduler: The mClock scheduler (default scheduler in Quincy) has undergone significant usability and design improvements to address the slow backfill issue. Some important changes are: * The 'balanced' profile is set as the default mClock profile because it represents a compromise between prioritizing client IO or recovery IO. Users can then choose either the 'high_client_ops' profile to prioritize client IO or the 'high_recovery_ops' profile to prioritize recovery IO. * QoS parameters including reservation and limit are now specified in terms of a fraction (range: 0.0 to 1.0) of the OSD's IOPS capacity. * The cost parameters (osd_mclock_cost_per_io_usec_* and osd_mclock_cost_per_byte_usec_*) have been removed. The cost of an operation is now determined using the random IOPS and maximum sequential bandwidth capability of the OSD's underlying device. * Degraded object recovery is given higher priority when compared to misplaced object recovery because degraded objects present a data safety issue not present with objects that are merely misplaced. Therefore, backfilling operations with the 'balanced' and 'high_client_ops' mClock profiles may progress slower than what was seen with the 'WeightedPriorityQueue' (WPQ) scheduler. * The QoS allocations in all mClock profiles are optimized based on the above fixes and enhancements. * For more detailed information see: https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ * RGW: S3 multipart uploads using Server-Side Encryption now replicate correctly in multi-site. Previously, the replicas of such objects were corrupted on decryption. A new tool, ``radosgw-admin bucket resync encrypted multipart``, can be used to identify these original multipart uploads. The ``LastModified`` timestamp of any identified object is incremented by 1 nanosecond to cause peer zones to replicate it again. For multi-site deployments that make any use of Server-Side Encryption, we recommended running this command against every bucket in every zone after all zones have upgraded. * CephFS: MDS evicts clients which are not advancing their request tids which causes a large buildup of session metadata resulting in the MDS going read-only due to the RADOS operation exceeding the size threshold. `mds_session_metadata_threshold` config controls the maximum size that a (encoded) session metadata can grow. * CephFS: After recovering a Ceph File System post following the disaster recovery procedure, the recovered files under `lost+found` directory can now be deleted. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: b12291d110049b2f35e32e0de30d70e9a4c060d2

2 1

check_memory_usage() recreation in OSD:tick()
by Suyash Dongre 12 Nov '23

12 Nov '23

Hi guys!This is a function for MDS (src/mds/MDCache.cc) that prints out the rss and other memory stats every couple seconds at debug level 2.I am trying to create a similar function for printing memory and other stats for OSD. (will be adding the code in src/osd/OSD:tick() function),can anyone provide a basic guidelines about what all classes, functions I will need to use and what all will I require for achieving so?Any help would be extremely valuable, Thanks for your time! void MDCache::check_memory_usage() { static MemoryModel mm(g_ceph_context); static MemoryModel::snap last; mm.sample(&last); static MemoryModel::snap baseline = last; // check client caps ceph_assert(CInode::count() == inode_map.size() + snap_inode_map.size() + num_shadow_inodes); double caps_per_inode = 0.0; if (CInode::count()) caps_per_inode = (double)Capability::count() / (double)CInode::count(); dout(2) << "Memory usage: " << " total " << last.get_total() << ", rss " << last.get_rss() << ", heap " << last.get_heap() << ", baseline " << baseline.get_heap() << ", " << num_inodes_with_caps << " / " << CInode::count() << " inodes have caps" << ", " << Capability::count() << " caps, " << caps_per_inode << " caps per inode" << dendl; mds->update_mlogger(); mds->mlogger->set(l_mdm_rss, last.get_rss()); mds->mlogger->set(l_mdm_heap, last.get_heap()); }

1 0

one cephfs volume becomes very slow
by Ben 10 Nov '23

10 Nov '23

Dear cephers, we have a cephfs volume, that will be mounted by many clients with concurrent read/write capability. From time to time, maybe when concurrency goes as high as 100 clients' access, accessing it will become very slow to be useful at all. the cluster has multiple active mds. All disks are hdd. Any ideas to improve this? here is one of mds log during the slow time, others are simillar: {"log":"debug 2023-11-08T07:26:00.114+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for \u003e 5.662970 secs\n","stream":"stderr","time":"2023-11-08T07:26:00.121996282Z"} {"log":"debug 2023-11-08T07:26:00.114+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 5.662970 seconds old, received at 2023-11-08T07:25:54.458863+0000: peer_request:client.12917739:8654334 currently dispatched\n","stream":"stderr","time":"2023-11-08T07:26:00.122016551Z"} {"log":"debug 2023-11-08T07:29:54.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for \u003e 11.900602 secs\n","stream":"stderr","time":"2023-11-08T07:29:54.124567293Z"} {"log":"debug 2023-11-08T07:29:54.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 11.900601 seconds old, received at 2023-11-08T07:29:42.223813+0000: client_request(client.27494331:18564666 getattr pAsLsXsFs #0x70001830366 2023-11-08T07:29:42.219416+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting\n","stream":"stderr","time":"2023-11-08T07:29:54.124589613Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 5 slow requests, 5 included below; oldest blocked for \u003e 17.900670 secs\n","stream":"stderr","time":"2023-11-08T07:30:00.124691442Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 17.900670 seconds old, received at 2023-11-08T07:29:42.223813+0000: client_request(client.27494331:18564666 getattr pAsLsXsFs #0x70001830366 2023-11-08T07:29:42.219416+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting\n","stream":"stderr","time":"2023-11-08T07:30:00.124726772Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 6.649942 seconds old, received at 2023-11-08T07:29:53.474541+0000: client_request(mds.1:305661 rename #0x70001851b32/91e670f9004ddb237a353b2a9ddc063208f5 #0x649/800019f1da7 caller_uid=0, caller_gid=0{}) currently failed to acquire_locks\n","stream":"stderr","time":"2023-11-08T07:30:00.124731626Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 6.649864 seconds old, received at 2023-11-08T07:29:53.474619+0000: client_request(mds.1:305662 rename #0x70001851b32/91e670f9004ddb237a353b2a9ddc063208f5 #0x649/800019f1da7 caller_uid=0, caller_gid=0{}) currently requesting remote authpins\n","stream":"stderr","time":"2023-11-08T07:30:00.124734415Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 6.649719 seconds old, received at 2023-11-08T07:29:53.474764+0000: client_request(client.27497255:25173 getattr pAsLsXsFs #0x800019f1da7 2023-11-08T07:29:53.473182+0000 caller_uid=0, caller_gid=0{}) currently requesting remote authpins\n","stream":"stderr","time":"2023-11-08T07:30:00.124736973Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 6.648454 seconds old, received at 2023-11-08T07:29:53.476029+0000: client_request(mds.1:305663 rename #0x70001851b32/91e670f9004ddb237a353b2a9ddc063208f5 #0x649/800019f1da7 caller_uid=0, caller_gid=0{}) currently requesting remote authpins\n","stream":"stderr","time":"2023-11-08T07:30:00.124739607Z"} {"log":"debug 2023-11-08T07:43:30.127+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 2 slow requests, 2 included below; oldest blocked for \u003e 5.206645 secs\n","stream":"stderr","time":"2023-11-08T07:43:30.133682292Z"} {"log":"debug 2023-11-08T07:43:30.127+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 5.206644 seconds old, received at 2023-11-08T07:43:24.926862+0000: client_request(client.27430891:5371608 mkdir #0x700018317cd/13 2023-11-08T07:43:24.924423+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply\n","stream":"stderr","time":"2023-11-08T07:43:30.133708161Z"} {"log":"debug 2023-11-08T07:43:30.127+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 5.206209 seconds old, received at 2023-11-08T07:43:24.927297+0000: client_request(client.27430891:5371609 create #0x2000d8a535a/2d69440b170ada3dd3670b5f3e2ebe4d75eeb274ff2a8238a23162e95601 2023-11-08T07:43:24.924423+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply\n","stream":"stderr","time":"2023-11-08T07:43:30.133711838Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 5 slow requests, 5 included below; oldest blocked for \u003e 7.332759 secs\n","stream":"stderr","time":"2023-11-08T07:50:18.138195062Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 7.318159 seconds old, received at 2023-11-08T07:50:10.819829+0000: client_request(client.27041435:5802294 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.815700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:18.138221837Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 7.305700 seconds old, received at 2023-11-08T07:50:10.832287+0000: client_request(client.27041435:5802298 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.827700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:18.138225662Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 6.977220 seconds old, received at 2023-11-08T07:50:11.160767+0000: client_request(client.27041435:5802299 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:11.155701+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:18.138228595Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 7.332758 seconds old, received at 2023-11-08T07:50:10.805229+0000: client_request(client.27041435:5802289 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.799700+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting\n","stream":"stderr","time":"2023-11-08T07:50:18.138231253Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 7.327715 seconds old, received at 2023-11-08T07:50:10.810272+0000: client_request(client.27041435:5802290 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.803700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:18.138234251Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 5 slow requests, 5 included below; oldest blocked for \u003e 13.332825 secs\n","stream":"stderr","time":"2023-11-08T07:50:24.138201305Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 13.318225 seconds old, received at 2023-11-08T07:50:10.819829+0000: client_request(client.27041435:5802294 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.815700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:24.138221584Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 13.305766 seconds old, received at 2023-11-08T07:50:10.832287+0000: client_request(client.27041435:5802298 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.827700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:24.13822536Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 12.977286 seconds old, received at 2023-11-08T07:50:11.160767+0000: client_request(client.27041435:5802299 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:11.155701+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:24.138242952Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 13.332825 seconds old, received at 2023-11-08T07:50:10.805229+0000: client_request(client.27041435:5802289 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.799700+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting\n","stream":"stderr","time":"2023-11-08T07:50:24.138245661Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 13.327781 seconds old, received at 2023-11-08T07:50:10.810272+0000: client_request(client.27041435:5802290 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.803700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:24.13824822Z"} {"log":"debug 2023-11-08T07:53:54.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for \u003e 5.641663 secs\n","stream":"stderr","time":"2023-11-08T07:53:54.140575892Z"} {"log":"debug 2023-11-08T07:53:54.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 5.641662 seconds old, received at 2023-11-08T07:53:48.498707+0000: client_request(client.12072730:30349041 rename #0x70001831e1d/ec603fcfb5f191c00efd5f87d2fb96cbd04a40928b9d1f11744d7ef6d742 #0x7000182f285/e9e5e277 2023-11-08T07:53:48.497133+0000 caller_uid=0, caller_gid=0{}) currently acquired locks\n","stream":"stderr","time":"2023-11-08T07:53:54.140612721Z"}

2 3

Permanent KeyError: 'TYPE' ->17.2.7: return self.blkid_api['TYPE'] == 'part'
by Harry G Coin 10 Nov '23

10 Nov '23

These repeat for every host, only after upgrading from prev release Quincy to 17.2.7. As a result, the cluster is always warned, never indicates healthy. root@noc1:~# ceph health detail HEALTH_WARN failed to probe daemons or devices [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host sysmon1 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/mon.sysmon1/config Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha2 56:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e -e NODE_NAME=sysmon1 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/4067126d-01cb-40af-824a-881c130140f8:/var /run/ceph:z -v /var/log/ceph/4067126d-01cb-40af-824a-881c130140f8:/var/log/ceph:z -v /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lv m -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpl1e27bun:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e inventory --format=json-pretty --filter-for-batch /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr Traceback (most recent call last): /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module> /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/docker: stderr self.main(self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/docker: stderr return f(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/inventory/main.py", line 60, in main /usr/bin/docker: stderr list_all=self.args.list_all)) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 50, in __init__ /usr/bin/docker: stderr sys_info.devices.keys()] /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 49, in <listcomp> /usr/bin/docker: stderr all_devices_vgs=all_devices_vgs) for k in /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 147, in __init__ /usr/bin/docker: stderr self.available_lvm, self.rejected_reasons_lvm = self._check_lvm_reject_reasons() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 646, in _check_lvm_reject_reasons /usr/bin/docker: stderr rejected.extend(self._check_generic_reject_reasons()) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 601, in _check_generic_reject_reasons /usr/bin/docker: stderr if self.is_acceptable_device: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 502, in is_acceptable_device /usr/bin/docker: stderr return self.is_device or self.is_partition or self.is_lv /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 482, in is_partition /usr/bin/docker: stderr return self.blkid_api['TYPE'] == 'part' /usr/bin/docker: stderr KeyError: 'TYPE' Traceback (most recent call last): File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9679, in <module> main() File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9667, in main r = ctx.func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2116, in _infer_config return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2032, in _infer_fsid return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2144, in _infer_image return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2019, in _validate_fsid return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 6272, in command_ceph_volume out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 1807, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@ sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e -e NODE_NAME=sysmon1 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/4067126d-01cb-40af-824a-881c130140f8: /var/run/ceph:z -v /var/log/ceph/4067126d-01cb-40af-824a-881c130140f8:/var/log/ceph:z -v /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/ru n/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpl1e27bun:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e inventory --format=json-pretty --filter-for-b atch host noc3 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/mon.noc3/config Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha2 56:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e -e NODE_NAME=noc3 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/4067126d-01cb-40af-824a-881c130140f8:/var/ru n/ceph:z -v /var/log/ceph/4067126d-01cb-40af-824a-881c130140f8:/var/log/ceph:z -v /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm - v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpjjmsgag7:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e inventory --format=json-pretty --filter-for-batch /usr/bin/docker: stderr Traceback (most recent call last): /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module> /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/docker: stderr self.main(self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/docker: stderr return f(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/inventory/main.py", line 60, in main /usr/bin/docker: stderr list_all=self.args.list_all)) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 50, in __init__ /usr/bin/docker: stderr sys_info.devices.keys()] /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 49, in <listcomp> /usr/bin/docker: stderr all_devices_vgs=all_devices_vgs) for k in /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 147, in __init__ /usr/bin/docker: stderr self.available_lvm, self.rejected_reasons_lvm = self._check_lvm_reject_reasons() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 646, in _check_lvm_reject_reasons /usr/bin/docker: stderr rejected.extend(self._check_generic_reject_reasons()) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 601, in _check_generic_reject_reasons /usr/bin/docker: stderr if self.is_acceptable_device: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 502, in is_acceptable_device /usr/bin/docker: stderr return self.is_device or self.is_partition or self.is_lv /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 482, in is_partition /usr/bin/docker: stderr return self.blkid_api['TYPE'] == 'part' /usr/bin/docker: stderr KeyError: 'TYPE' Traceback (most recent call last): File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9679, in <module> main() File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9667, in main r = ctx.func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2116, in _infer_config return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2032, in _infer_fsid return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2144, in _infer_image return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2019, in _validate_fsid return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 6272, in command_ceph_volume out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 1807, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@ sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e -e NODE_NAME=noc3 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/4067126d-01cb-40af-824a-881c130140f8:/va r/run/ceph:z -v /var/log/ceph/4067126d-01cb-40af-824a-881c130140f8:/var/log/ceph:z -v /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/l vm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpjjmsgag7:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e inventory --format=json-pretty --filter-for-batc h ....

2 2

Redeploy ceph orch OSDs after reboot, but don't mark as 'unmanaged'
by Janek Bevendorff 10 Nov '23

10 Nov '23

Hi, We have our cluster RAM-booted, so we start from a clean slate after every reboot. That means I need to redeploy all OSD daemons as well. At the moment, I run cephadm deploy via Salt on the rebooted node, which brings the deployed OSDs back up, but the problem with this is that the deployed OSD shows up as 'unmanaged' in ceph orch ps afterwards. I could simply skip the cephadm call and wait for the Ceph orchestrator to reconcile and auto-activate the disks, but that can take up to 15 minutes, which is unacceptable. Running ceph cephadm osd activate is not an option either, since I don't have the admin keyring deployed on the OSD hosts (I could do that, but I don't want to). How can I manually activate the OSDs after a reboot and hand over control to the Ceph orchestrator afterwards? I checked the deployments in /var/lib/ceph/<FSID>, but the only difference I found between my manual cephadm deployment and what ceph orch does is that the device links to /dev/mapper/ceph--... instead of /dev/ceph-... Any hints appreciated! Janek

2 6

IO stalls when primary OSD device blocks in 17.2.6
by Daniel Schreiber 10 Nov '23

10 Nov '23

Dear cephers, we are sometimes observing stalling IO on our ceph 17.2.6 cluster when the backing device for the primary OSD of a PG fails and seems to block read IO to objects from that pg. If I set the OSD with the broken device to down, the IO continues. Setting the OSD to down is not sufficient. The cluster is running on Debian 11, the pool is an erasure coded cephfs data pool. The OSD has a HDD data device and an SSD db device. The data devices is the one which failed and was blocking IO. The OSD was reporting slow ops and short time after that smartd notified about unreadable sectors. Has anyone seen such behaviour? Are there some tweaks that I missed? Kind regards, Daniel -- Daniel Schreiber Facharbeitsgruppe Systemsoftware Universitaetsrechenzentrum Technische Universität Chemnitz Straße der Nationen 62 (Raum B303) 09111 Chemnitz Germany Tel: +49 371 531 35444 Fax: +49 371 531 835444

2 1

Re: Reinitialize rgw garbage collector
by Pierre GINDRAUD 10 Nov '23

10 Nov '23

Hello Michael, Did you receive any help on this ? We are issuing the same problem without solution for now. Regard -- Pierre

1 0

Help needed with Grafana password
by Sake Ceph 10 Nov '23

10 Nov '23

2 10

mds hit find_exports balancer runs too long
by zxcs 10 Nov '23

10 Nov '23

Hi, Experts, we have a CephFS cluster running with 16.2.*, and enable multi active mds, found somehow mds complain some info as below: mds.*.bal find_exports balancer runs too long and we already set below config mds_bal_interval = 30 mds_bal_sample_interval = 12 and then we can see slow mds request from `ceph -s`. our question is : why ceph mds complain this and how can we prevent this problem from happening again? Thanks a ton, xiong

1 0

Ceph Dashboard - Community News Sticker [Feedback]
by Nizamudeen A 10 Nov '23

10 Nov '23

Hello, We wanted to get some feedback on one of the features that we are planning to bring in for upcoming releases. On the Ceph GUI, we thought it could be interesting to show information regarding the community events, ceph release information (Release notes and changelogs) and maybe even notify about new blog post releases and also inform regarding the community group meetings. There would be options to subscribe to the events that you want to get notified. Before proceeding with its implementation, we thought it'd be good to get some community feedback around it. So please let us know what you think (the goods and the bads). Regards, -- Nizamudeen A Software Engineer Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

7 7