- ceph-users - lists.ceph.io

Ambigous mds behind on trimming and slowps issue on ceph 17.2.5 with rook 1.10.8 operator
by Akash Warkhade 14 Feb '24

14 Feb '24

Guys, We were facing cephFs volume mount issue and ceph status it was showing mds slow requests Mds behind on trimming After restarting mds pods it was resolved But wanted to know Root caus of this It was started after 2 hours of one of the active mds was crashed So does that an active mds crash can cause this issue ? Please provide your inputs anyone

1 0

Unable to add OSD after removing completely
by salam＠dcl-online.com 14 Feb '24

14 Feb '24

Hello, I have a Ceph cluster created by orchestrator Cephadm. It consists of 3 Dell PowerEdge R730XD servers. This cluster's hard drives used as OSD were configured as RAID 0. The configuration summery is as the following: ceph-node1 (mgr, mon) Public network: 172.16.7.11/22 Cluster network: 10.10.10.11/24, 10.10.10.14/24 ceph-node2 (mgr, mon) Public network: 172.16.7.12/22 Cluster network: 10.10.10.12/24, 10.10.10.15/24 ceph-node3 (mon) Public network: 172.16.7.13/22 Cluster network: 10.10.10.13/24, 10.10.10.16/24 Recently I removed all OSDs from node3 with the following set of commands sudo ceph osd out osd.3 sudo systemctl stop ceph(a)osd.3.service sudo ceph osd rm osd.3 sudo ceph osd crush rm osd.3 sudo ceph auth del osd.3 After this, I configured all OSD hard drives as non-RAIN from the server settings and tried to add the hard drives as OSD again. First I used the following command to add them automatically ceph orch apply osd --all-available-devices --unmanaged=false But this was generating the following error in my Ceph GUI console CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices I am also unable to add the hard drives manually with the following command sudo ceph orch daemon add osd ceph-node3:/dev/sdb Can anyone please help me with this issue? I really appreciate any help you can provide.

2 3

RECENT_CRASH: x daemons have recently crashed
by Jaemin Joo 14 Feb '24

14 Feb '24

Hi everyone, I'd like to ask you why this message happened. There are no symptoms with a warning message. After archiving the message, It happened again. It will be very helpful if you give me the hint about what part I should look at. [log]----------------------------------------------- root@ceph-01:~# ceph crash ls-newID ENTITY NEW 2024-02-09T12:14:38.115931Z_a2a9e502-7582-4fb6-8187-f1e8c74291d9 client.rgw.archive.rnd-ceph-01.mbbqyj * 2024-02-10T10:29:08.507254Z_fbfce5ae-34b4-48fe-8a41-73ea489f3b4c client.rgw.archive.rnd-ceph-01.mbbqyj * 2024-02-11T08:43:18.430257Z_eeab3b6b-3241-400a-ab31-00b5fe3cc1a3 client.rgw.archive.rnd-ceph-01.mbbqyj * 2024-02-12T06:57:32.793665Z_39241466-5024-43ec-8e60-859760abaa69 client.rgw.archive.rnd-ceph-01.mbbqyj * 2024-02-13T05:12:17.654693Z_0ec8d096-9ddd-4cb6-a9fa-3f67d07b78ad client.rgw.archive.rnd-ceph-01.mbbqyj * root@ceph-01:~# ceph crash info 2024-02-09T12:14:38.115931Z_a2a9e502-7582-4fb6-8187-f1e8c74291d9 { "backtrace": [ "/lib64/libpthread.so.0(+0x12ce0) [0x7fbe48b1dce0]", "gsignal()", "abort()", "/lib64/libc.so.6(+0x21cd9) [0x7fbe47141cd9]", "/lib64/libc.so.6(+0x473f6) [0x7fbe471673f6]", "(boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>::allocate()+0x17d) [0x7fbe4b7cac4d]", "/lib64/libradosgw.so.2(+0x64e7e9) [0x7fbe4b79f7e9]", "/lib64/libradosgw.so.2(+0x64e98b) [0x7fbe4b79f98b]", "(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u> const, void>::operator()()+0x8d) [0x7fbe4b7cbd9d]","(void boost::asio::io_context::basic_executor_type<std::allocator<void>,0u>::execute<boost::asio::detail::strand_executor_service::allocator_binder<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u> const, void>, std::allocator<void> > >(boost::asio::detail::strand_executor_service::allocator_binder<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u> const, void>, std::allocator<void> >&&) const+0x12a) [0x7fbe4b7cbf4a]", "/lib64/libradosgw.so.2(+0x653f34) [0x7fbe4b7a4f34]", "/lib64/libradosgw.so.2(+0x658f19) [0x7fbe4b7a9f19]", "(boost::asio::detail::scheduler::run(boost::system::error_code&)+0x538) [0x7fbe4b7baf48]", "/lib64/libradosgw.so.2(+0x64d109) [0x7fbe4b79e109]", "/lib64/libstdc++.so.6(+0xc2ba3) [0x7fbe47b42ba3]", "/lib64/libpthread.so.0(+0x81ca) [0x7fbe48b131ca]", "clone()" ], "ceph_version": "17.2.3", "crash_id": "2024-02-09T12:14:38.115931Z_a2a9e502-7582-4fb6-8187-f1e8c74291d9", "entity_name": "client.rgw.archive.rnd-ceph-01.mbbqyj", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "radosgw", "stack_sig": "5cd72ee2dcf4cbea3fcb0f28778bedd6859eb402e1482d1ef750cc75d05cd47b", "timestamp": "2024-02-09T12:14:38.115931Z", "utsname_hostname": "rnd-ceph-01", "utsname_machine": "x86_64", "utsname_release": "5.4.0-153-generic", "utsname_sysname": "Linux", "utsname_version": "#170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023" }

1 0

Help with setting-up Influx MGR module: ERROR - queue is full
by Fulvio Galeazzi 13 Feb '24

13 Feb '24

Hi there! Has anyone any experience with the Influx Ceph mgr module? I am using 17.2.7 on CentOS8-Stream, I configured one of my clusters, I test with "ceph influx send" (whereas official doc https://docs.ceph.com/en/quincy/mgr/influx/ mentions the non-existing "ceph influx self-test") but nothing goes to the influx databases. Here is my config (password not shown): mgr advanced mgr/influx/database cephct * mgr advanced mgr/influx/hostname influxdb-dev.cloud.garr.it * mgr advanced mgr/influx/interval 300 * mgr advanced mgr/influx/password **** * mgr advanced mgr/influx/ssl false * mgr advanced mgr/influx/username cephctusr * mgr advanced mgr/influx/verify_ssl false * After enabling the module, in the MGR/MON logs I see, after a while: 2024-02-13T09:06:41.283+0100 7f5be9fff700 0 [influx ERROR root] Queue is full, failed to add chunk and "ceph health detail" shows: WRN] MGR_INFLUX_QUEUE_FULL: Failed to chunk to InfluxDB Queue Queue is full. InfluxDB might be slow with processing data (edited) (I searched a bit for "failed to chunk" but found nothing) MGR coexist with MON, and I verified (by installing influxdb by hand) that from the MON the command influx -database cephct -username cephctusr -password '****' -host influxdb-dev.cloud.garr.it indeed works. Hmm, actually while making my tests, at some point something arrived to the influxDB server, but only for 5 minutes or so, yesterday morning: it is practically impossible for me now to reconstruct what the configuration was at the time... may be during one server reboot? In any case, only the following measurements ceph_pg_summary_osd ceph_pg_summary_pool were populated, and they do not contain terribly exciting metrics, only status of PGs for each pool and number of PG per OSD. I guess the interesting metrics reported in the documentation (latency, bytes, operations...) should end up into some other measurement. I am not particularly fond of Influx, just seeking for "something"(Influx? Telegraf?) to store metrics and eventually plot to Grafana, to replace the current Zabbix-based solution. I experimented with Prometheus with some satisfaction, some time ago, although it requires a scraper which I'd be happy to avoid, especially given the point below. An additional constraint is that I have at least 3 distinct Ceph production clusters to monitor, so I'd need a way to differentiate them in a simple manner. How are you dealing with these matters, namely storing configuration and metrics "somewhere"? Thanks a lot! (for your patience in reading this, at least) Fulvio -- Fulvio Galeazzi GARR-Net Department tel.: +39-334-6533-250 skype: fgaleazzi70

1 0

Announcing go-ceph v0.26.0
by Sven Anderson 13 Feb '24

13 Feb '24

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.26.0 <https://github.com/ceph/go-ceph/releases/tag/v0.23.0> The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 0

Accumulation of removed_snaps_queue After Deleting Snapshots in Ceph RBD
by localhost Liam 13 Feb '24

13 Feb '24

Hello, I'm encountering an issue with Ceph when using it as the backend storage for OpenStack Cinder. Specifically, after deleting RBD snapshots through Cinder, I've noticed a significant increase in the removed_snaps_queue entries within the corresponding Ceph pool. It seems to affect the pool's performance and space efficiency. I understand that snapshot deletion in Cinder is an asynchronous operation, and Ceph itself uses a lazy deletion mechanism to handle snapshot removal. However, even after allowing sufficient time, the entries in removed_snaps_queue do not decrease as expected. I have several questions for the community: Are there recommended methods or best practices for managing or reducing entries in removed_snaps_queue? Is there any tool or command that can safely clear these residual snapshot entries without affecting the integrity of active snapshots and data? Is this issue known, and are there any bug reports or plans for fixes related to it? Thank you very much for your assistance!

2 3

Increase number of PGs
by Murilo Morais 12 Feb '24

12 Feb '24

Good morning and happy holidays everyone! Guys, what would be the best strategy to increase the number of PGs in a POOL that is already in production?

2 2

Installing ceph s3.
by Albert Shih 12 Feb '24

12 Feb '24

Hi everyone. I couldn't find a documentation about how to install a S3/Swift API (as I understand it's RadosGW) on quincy. I can find some documentation on octupus (https://docs.ceph.com/en/octopus/install/ceph-deploy/install-ceph-gateway/) Very strangely when I go https://docs.ceph.com/en/quincy/radosgw/ I can see a lot of very detailed documentation about each component, but cannot find a more global documentation. Any new documentation somewhere ? I think it's not a good idea to use the one on octopus... Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: lun. 12 févr. 2024 18:08:59 CET

2 2

RGW core dump at start
by Vladimir Sigunov 10 Feb '24

10 Feb '24

Hello Community! I would appreciate any help/suggestions with the massive RGWs outage we are facing. The cluster's overall status is acceptable (HEALTH_WARN because of some pgs not scrubbed in time), and the cluster is operational. However, all RGWs fail to start with a core dump. The only issue I see at the moment is the RGW GC queue (radosgs-admin gc list) that contains 600K records. I believe this could be the root cause of the issue. When I pause OSD iops (ceph osd pause), all RGWs starting with no issues. There are no large OMAPs or any other warnings in ceph -s output. I would appreciate any help or suggestions you can provide. Sincerely, Vladimir.

2 2

Ceph Storage || Deploy/Install/Bootstrap a Ceph Cluster || Cephadm Orchestrator CLI method
by ankit＠eurus.net 09 Feb '24

09 Feb '24

Hi Guys, I am newbie and trying to install Ceph Storage cluster and following this https://docs.ceph.com/en/latest/cephadm/install/#cephadm-deploying-new-clus… ============================================================= OS - Ubuntu 22.04.3 LTS (Jammy Jellyfish) 4 node Cluster - mon1,mgr1,2 OSD nodes mon1 node can ssh all nodes via root to sudo ceph-user and ceph-user to ceph-user on other nodes basic requirements are done like podman, python3, systemd,ntp, lvm. =================================================================== cephadm bootstrap --mon-ip 192.168.2.125 - after running this i am getting following error. ceph-user@mon1:~$ sudo cephadm bootstrap --mon-ip 192.168.2.125 Creating directory /etc/ceph for ceph.conf Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chrony.service is enabled and running Repeating the final host check... podman (/usr/bin/podman) version 3.4.4 is present systemctl is present lvcreate is present Unit chrony.service is enabled and running Host looks OK Cluster fsid: 90813682-c656-11ee-9ca3-0800274ff361 Verifying IP 192.168.2.125 port 3300 ... Verifying IP 192.168.2.125 port 6789 ... Mon IP `192.168.2.125` is in CIDR network `192.168.2.0/24` Mon IP `192.168.2.125` is in CIDR network `192.168.2.0/24` Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Pulling container image quay.io/ceph/ceph:v17... Ceph version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting mon public_network to 192.168.2.0/24 Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 9283 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr not available, waiting (3/15)... mgr not available, waiting (4/15)... mgr not available, waiting (5/15)... mgr not available, waiting (6/15)... mgr not available, waiting (7/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Generating ssh key... Wrote public SSH key to /etc/ceph/ceph.pub Adding key to root@localhost authorized_keys... Adding host mon1... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Deploying ceph-exporter service with default placement... Non-zero exit code 22 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=mon1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/90813682-c656-11ee-9ca3-0800274ff361:/var/log/ceph:z -v /tmp/ceph-tmpnjonhex7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3gil6lbb:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch apply ceph-exporter /usr/bin/ceph: stderr Error EINVAL: Usage: /usr/bin/ceph: stderr ceph orch apply -i <yaml spec> [--dry-run] /usr/bin/ceph: stderr ceph orch apply <service_type> [--placement=<placement_string>] [--unmanaged] /usr/bin/ceph: stderr Traceback (most recent call last): File "/usr/sbin/cephadm", line 9653, in <module> main() File "/usr/sbin/cephadm", line 9641, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2205, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 5774, in command_bootstrap prepare_ssh(ctx, cli, wait_for_mgr_restart) File "/usr/sbin/cephadm", line 5275, in prepare_ssh cli(['orch', 'apply', t]) File "/usr/sbin/cephadm", line 5708, in cli return CephContainer( File "/usr/sbin/cephadm", line 4144, in run out, _, _ = call_throws(self.ctx, self.run_cmd(), File "/usr/sbin/cephadm", line 1853, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=mon1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/90813682-c656-11ee-9ca3-0800274ff361:/var/log/ceph:z -v /tmp/ceph-tmpnjonhex7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3gil6lbb:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch apply ceph-exporter What i am doing wrong or missing? Please help. Many Thanks AS

4 5