- ceph-users - lists.ceph.io

Quincy: failure to enable mgr rgw module if not --force
by Michel Jouvin 25 Oct '23

25 Oct '23

Hi, I'm trying to use the rgw mgr module to configure RGWs. Unfortunately it is not present in 'ceph mgr module ls' list and any attempt to enable it suggests that one mgr doesn't support it and that --force should be added. Adding --force effectively enabled it. It is strange as it is a brand new cluster, created in Quincy, using cephadm. Why this need for --force? And it seems that even if the module is listed as enabled, the 'ceph rgw' command is not recognized and the help is not available for the rgw subcommand? What are we doing wrong? Cheers, Michel

3 3

Dashboard crash with rook/reef and external prometheus
by r-ceph＠rich0.org 24 Oct '23

24 Oct '23

I'm fairly new to the community so I figured I'd ask about this here before creating an issue - I'm not sure how supported this config is. I am running rook v1.12.6 and ceph 18.2.0. I've enabled the dashboard in the CRD and it has been working for a while. However, the charts are empty. I do have Prometheus+Grafana running on my cluster, and I can access many of the ceph metrics from there. With the upgrade to reef I noticed that many of the qunicy dashboard elements have been replaced by charts, so I wanted to get those working. I discovered that if I run ceph dashboard set-prometheus-api-host <url> the charts immediately are populated (including historical data). However, when I do this I rapidly start getting ceph health alerts due to a crashing mgr module. If I set the prometheus api host url to '' the crashes stop accumulating, though this disables the charts. I am running the prometheus-community/prometheus-25.2.0 chart. Various ceph grafana dashboards that I've found published work fine. The following are relevant dumps. Please let me know if you have any ideas, or if I should go ahead and create an issue for this... mgr console output during crash: debug 2023-10-24T15:11:23.498+0000 7fc81fa5d700 -1 mgr.server reply reply (2) No such file or directory This Orchestrator does not support `orch prometheus access info` debug 2023-10-24T15:11:23.502+0000 7fc7ea3f3700 0 [dashboard INFO request] [::ffff:10.1.0.106:49760] [GET] [200] [0.012s] [admin] [101.0B] /api/health/get_cluster_capacity debug 2023-10-24T15:11:23.502+0000 7fc813985700 0 [stats WARNING root] cmdtag not found in client metadata debug 2023-10-24T15:11:23.502+0000 7fc813985700 0 [stats WARNING root] cmdtag not found in client metadata debug 2023-10-24T15:11:23.502+0000 7fc7e83ef700 0 [dashboard INFO request] [::ffff:10.1.0.106:5580] [GET] [200] [0.011s] [admin] [73.0B] /api/osd/settings debug 2023-10-24T15:11:23.506+0000 7fc85411a700 0 log_channel(audit) log [DBG] : from='mon.2 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": "json"}]: dispatch debug 2023-10-24T15:11:23.506+0000 7fc813985700 0 [stats WARNING root] cmdtag not found in client metadata debug 2023-10-24T15:11:23.506+0000 7fc7e9bf2700 0 [dashboard INFO request] [::ffff:10.1.0.106:20241] [GET] [200] [0.014s] [admin] [34.0B] /api/prometheus/rules debug 2023-10-24T15:11:23.630+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:23.734+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:23.802+0000 7fc86511c700 0 log_channel(cluster) log [DBG] : pgmap v126: 617 pgs: 53 active+remapped+backfill_wait, 2 active+remapped+backfilling, 562 active+clean; 34 TiB data, 68 TiB used, 64 TiB / 132 TiB avail; 2.4 MiB/s rd, 93 KiB/s wr, 21 op/s; 1213586/22505781 objects misplaced (5.392%) debug 2023-10-24T15:11:23.862+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:23.962+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.058+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.158+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.270+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.546+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.654+0000 7fc7ebbf6700 0 [dashboard INFO orchestrator] is orchestrator available: True, debug 2023-10-24T15:11:24.654+0000 7fc7ebbf6700 0 [dashboard INFO request] [::ffff:10.1.0.106:13711] [GET] [200] [1.170s] [admin] [3.2K] /api/health/minimal debug 2023-10-24T15:11:25.802+0000 7fc86511c700 0 log_channel(cluster) log [DBG] : pgmap v127: 617 pgs: 53 active+remapped+backfill_wait, 2 active+remapped+backfilling, 562 active+clean; 34 TiB data, 68 TiB used, 64 TiB / 132 TiB avail; 1.1 MiB/s rd, 53 KiB/s wr, 17 op/s; 1213586/22505781 objects misplaced (5.392%) debug 2023-10-24T15:11:27.802+0000 7fc86511c700 0 log_channel(cluster) log [DBG] : pgmap v128: 617 pgs: 53 active+remapped+backfill_wait, 2 active+remapped+backfilling, 562 active+clean; 34 TiB data, 68 TiB used, 64 TiB / 132 TiB avail; 1.8 MiB/s rd, 58 KiB/s wr, 18 op/s; 1213586/22505781 objects misplaced (5.392%) debug 2023-10-24T15:11:28.494+0000 7fc813985700 0 [stats WARNING root] cmdtag not found in client metadata debug 2023-10-24T15:11:28.498+0000 7fc7eb3f5700 0 [dashboard INFO request] [::ffff:10.1.0.106:20241] [GET] [200] [0.011s] [admin] [73.0B] /api/osd/settings debug 2023-10-24T15:11:28.498+0000 7fc85411a700 0 log_channel(audit) log [DBG] : from='mon.2 -' entity='mon.' cmd=[{"prefix": "orch prometheus access info"}]: dispatch debug 2023-10-24T15:11:28.502+0000 7fc7ec3f7700 0 [dashboard INFO request] [::ffff:10.1.0.106:5580] [GET] [200] [0.006s] [admin] [102.0B] /api/health/get_cluster_capacity debug 2023-10-24T15:11:28.502+0000 7fc7e93f1700 0 [dashboard INFO request] [::ffff:10.1.0.106:44009] [GET] [200] [0.005s] [admin] [22.0B] /api/prometheus/notifications debug 2023-10-24T15:11:28.502+0000 7fc81fa5d700 -1 Remote method threw exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 675, in get_prometheus_access_info raise NotImplementedError() NotImplementedError ceph crash info dump: { "backtrace": [ " File \"/usr/share/ceph/mgr/orchestrator/_interface.py\", line 675, in get_prometheus_access_info\n raise NotImplementedError()", "NotImplementedError" ], "ceph_version": "18.2.0", "crash_id": "2023-10-24T14:59:52.921427Z_08e06575-0431-47fe-afc5-be8e4a7d1144", "entity_name": "mgr.a", "mgr_module": "rook", "mgr_module_caller": "ActivePyModule::dispatch_remote get_prometheus_access_info", "mgr_python_exception": "NotImplementedError", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mgr", "stack_sig": "bbf52dcdbbe54d67edf59ebdb5d201fffd921db5a9dd4431964c2aaac2250c7e", "timestamp": "2023-10-24T14:59:52.921427Z", "utsname_hostname": "k8s4", "utsname_machine": "x86_64", "utsname_release": "5.15.0-87-generic", "utsname_sysname": "Linux", "utsname_version": "#97-Ubuntu SMP Mon Oct 2 21:09:21 UTC 2023" } -- Rich

1 0

traffic by IP address / bucket / user
by Boris Behrens 24 Oct '23

24 Oct '23

Hi, did someone have a solution ready to monitor traffic by IP address? Cheers Boris

2 1

Modify user op status=-125
by mahnoosh shahidi 24 Oct '23

24 Oct '23

Hi all, I couldn't understand what does the status -125 mean from the docs. I'm getting 500 response status code when I call rgw admin APIs and the only log in the rgw log files is as follows. s3:get_obj recalculating target initializing for trans_id = tx00000aa90f570fb8281cf-006537bf9e-84395fa-default s3:get_obj reading permissions getting op 1 s3:put_obj verifying requester s3:put_obj normalizing buckets and tenants s3:put_obj init permissions s3:put_obj recalculating target s3:put_obj reading permissions s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions s3:put_obj verifying op params s3:put_obj pre-executing s3:put_obj executing :modify_user completing WARNING: set_req_state_err err_no=125 resorting to 500 :modify_user op status=-125 :modify_user http status=500 ====== req done req=0x7f3f85a78620 op status=-125 http_status=500 latency=0.076000459s ====== Can anyone explain what this error means and why it's happening? Best Regards, Mahnoosh

2 4

RadosGW load balancing with Kubernetes + ceph orch
by Janek Bevendorff 23 Oct '23

23 Oct '23

Hey all, My Ceph cluster is managed mostly by cephadm / ceph orch to avoid circular dependencies between in our infrastructure deployment. Our RadosGW endpoints, however, are managed by Kubernetes, since it provides proper load balancing and service health checks. This leaves me in the unsatisfactory situation that Ceph complains about 'stray' RGW daemons in the cluster. The only two solutions to this that I found were a) to turn of the warning, which applies to all daemons and not just the RGWs (not pretty!), or b) to move the deployment out of Kubernetes. For the latter, I could define external Endpoints in Kubernetes, so that I still have load balancing, but then I don't have proper health checks any more. Meaning, if one of the RGW endpoints goes down, requests to our S3 endpoint will intermittently time out in round-robin fashion (not pretty at all!). Can you think of a better option to solve this? I would already be satisfied with turning off the warning for RGW daemons only, but there doesn't seem to be a config option for that. Thanks Janek

1 0

Ceph orch OSD redeployment after boot on stateless RAM root
by Janek Bevendorff 23 Oct '23

23 Oct '23

Hi, I recently moved from a manual Ceph deployment using Saltstack to a hybrid of Saltstack and cephadm / ceph orch. We are provisioning our Ceph hosts using a stateless PXE RAM root, so I definitely need Saltstack to bootstrap at least the Ceph APT repository and the MON/MGR deployment. After that, ceph orch can take over and deploy the remaining daemons. The MONs/MGRs are deployed after each reboot with cephadm deploy --name mon.{{ ceph.node_id }} --fsid {{ ceph.conf.global.fsid }} --config /etc/ceph/ceph.conf cephadm deploy --name mgr.{{ ceph.node_id }} --fsid {{ ceph.conf.global.fsid }} --config /etc/ceph/ceph.conf (the MON store is provided in /var/lib/ceph/{{ ceph.conf.global.fsid }}/mon.{{ ceph.node_id }}). Since cephadm ceph-volume lvm activate --all is broken (see https://tracker.ceph.com/issues/55395) I am activating each OSD individually like this: cephadm deploy --name osd.{{ osd_id }} --fsid {{ ceph.conf.global.fsid }} --osd-fsid {{ osd_fsid }} --config /etc/ceph/ceph.conf Now my question: Is there a better way to do this and can ceph orch take care of this in the same way it deploys my MDS? All OSDs are listed as <unmanaged> in ceph orch ls (I think this is by design?) and I cannot find a way to activate them automatically via ceph orch when the host boots up. I tried ceph cephadm osd activate HOSTNAME, but all I get is "Created no osd(s) on host HOSTNAME; already created?" The docs only talk about how I can create new OSDs, but not how I can automatically redeploy existing OSDs after a fresh boot. It seems like it is generally assumed that OSD deployments are persistent and next time the host boots, systemd simply activates the existing units. I'd be glad about any hints! Janek

1 0

ATTN: DOCS rgw bucket pubsub notification.
by Artem Torubarov 22 Oct '23

22 Oct '23

Hi, The latest documentation <https://docs.ceph.com/en/latest/radosgw/s3-notification-compatibility/#supp…> says that: "pulling and acking of events stored in Ceph (as an internal destination)." is a valid source for s3 notifications. If i got it right, pull-based subscription was possible only with radosgw PubSub sync module, which was removed in reef release <https://docs.ceph.com/en/latest/releases/reef/#:~:text=The%20pubsub%20funct…> . Suggested changes: Currently, we support: HTTP/S, Kafka and AMQP. And also support pulling and acking of events stored in Ceph (as an internal destination). Best regards, Artem

3 2

Ceph 16.2.14: OSDs randomly crash in bstore_kv_sync
by Zakhar Kirpichenko 20 Oct '23

20 Oct '23

Hi, After upgrading to Ceph 16.2.14 we had several OSD crashes in bstore_kv_sync thread: 1. "assert_thread_name": "bstore_kv_sync", 2. "backtrace": [ 3. "/lib64/libpthread.so.0(+0x12cf0) [0x7ff2f6750cf0]", 4. "gsignal()", 5. "abort()", 6. "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x564dc5f87d0b]", 7. "/usr/bin/ceph-osd(+0x584ed4) [0x564dc5f87ed4]", 8. "(RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t const&)+0x15e) [0x564dc6604a9e]", 9. "(BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x77d) [0x564dc66951cd]", 10. "(BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x90) [0x564dc6695670]", 11. "(BlueFS::fsync(BlueFS::FileWriter*)+0x18b) [0x564dc66b1a6b]", 12. "(BlueRocksWritableFile::Sync()+0x18) [0x564dc66c1768]", 13. "(rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x564dc6b6496f]", 14. "(rocksdb::WritableFileWriter::SyncInternal(bool)+0x402) [0x564dc6c761c2]", 15. "(rocksdb::WritableFileWriter::Sync(bool)+0x88) [0x564dc6c77808]", 16. "(rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x309) [0x564dc6b780c9]", 17. "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x2629) [0x564dc6b80c69]", 18. "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x564dc6b80e61]", 19. "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x564dc6b1f644]", 20. "(RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x564dc6b2004a]", 21. "(BlueStore::_kv_sync_thread()+0x30d8) [0x564dc6602ec8]", 22. "(BlueStore::KVSyncThread::entry()+0x11) [0x564dc662ab61]", 23. "/lib64/libpthread.so.0(+0x81ca) [0x7ff2f67461ca]", 24. "clone()" 25. ], I am attaching two instances of crash info for further reference: https://pastebin.com/E6myaHNU OSD configuration is rather simple and close to default: osd.6 dev bluestore_cache_size_hdd 4294967296 osd.6 dev bluestore_cache_size_ssd 4294967296 osd advanced debug_rocksdb 1/5 osd advanced osd_max_backfills 2 osd basic osd_memory_target 17179869184 osd advanced osd_recovery_max_active 2 osd advanced osd_scrub_sleep 0.100000 osd advanced rbd_balance_parent_reads false debug_rocksdb is a recent change, otherwise this configuration has been running without issues for months. The crashes happened on two different hosts with identical hardware, the hosts and storage (NVME DB/WAL, HDD block) don't exhibit any issues. We have not experienced such crashes with Ceph < 16.2.14. Is this a known issue, or should I open a bug report? Best regards, Zakhar

3 13

fixing future rctime
by MARTEL Arnaud 20 Oct '23

20 Oct '23

Hi all, I have some troubles with my backup script because there are few files, in a deep sub-directory, with a creation/modification date in the future (for example: 2040-02-06 18:00:00). As my script uses the ceph.dir.rctime extended attribute to identify the files and directories to backup, it now browses and sync a lot of unchanged sub-directories… I tried a lot of things, including remove and recreate the files so they have now the current datetime, but rctime is never updated. Even when I remove the last directory (ie: the one where the files are located), rctime is not updated for the parent directories. Has someone a trick to reset rctime to current datetime (or any other solution to remove this inconsistent value of rctime) with quincy (17.2.6) ? Regards, Arnaud

2 2

Specify priority for active MGR and MDS
by Nicolas FONTAINE 19 Oct '23

19 Oct '23

Hi everyone, Is there a way to specify which MGR and which MDS should be the active one? Thanks, Nicolas.

3 2