- ceph-users - lists.ceph.io

Re: Performance improvement suggestion
by quaglio＠bol.com.br 01 Feb '24

01 Feb '24

1 1

Syslog server log naming
by Torkil Svensgaard 01 Feb '24

01 Feb '24

We have ceph (currently 18.2.0) log to an rsyslog server with the following file name format: template (name="DynaFile" type="string" string="/tank/syslog/%fromhost-ip%/%hostname%/%programname%.log") Around May 25th this year something changed so instead of getting the usual program log names we are also getting a lot of diffent logs with weird names. Here's an ls excerpt: " ... -rw-------. 1 root root 517K Feb 1 05:42 hardcore_raman.log -rw-------. 1 root root 198K Feb 1 05:42 sleepy_moser.log -rw-------. 1 root root 203K Feb 1 05:42 friendly_gagarin.log -rw-------. 1 root root 1.1K Feb 1 05:42 goofy_hypatia.log -rw-------. 1 root root 164K Feb 1 05:42 kind_chebyshev.log -rw-------. 1 root root 11K Feb 1 05:42 magical_archimedes.log -rw-------. 1 root root 373K Feb 1 05:42 busy_bardeen.log -rw-------. 1 root root 178K Feb 1 05:42 trusting_euler.log -rw-------. 1 root root 526K Feb 1 05:42 inspiring_golick.log -rw-------. 1 root root 369K Feb 1 06:12 condescending_ganguly.log -rw-------. 1 root root 191K Feb 1 06:12 mystifying_torvalds.log -rw-------. 1 root root 475K Feb 1 06:12 charming_nash.log -rw-------. 1 root root 168K Feb 1 06:12 zealous_sinoussi.log -rw-------. 1 root root 325K Feb 1 06:12 amazing_booth.log -rw-------. 1 root root 516K Feb 1 06:12 great_ardinghelli.log -rw-------. 1 root root 313K Feb 1 06:12 magical_bell.log -rw-------. 1 root root 22K Feb 1 06:12 nifty_swartz.log -rw-------. 1 root root 426 Feb 1 06:12 upbeat_beaver.log -rw-------. 1 root root 166K Feb 1 06:13 funny_lederberg.log -rw-------. 1 root root 164K Feb 1 06:13 frosty_murdock.log -rw-------. 1 root root 374K Feb 1 06:13 elastic_banach.log -rw-------. 1 root root 308K Feb 1 06:13 inspiring_cohen.log -rw-------. 1 root root 176K Feb 1 06:13 angry_wu.log -rw-------. 1 root root 662 Feb 1 06:42 admiring_kalam.log -rw-------. 1 root root 3.1K Feb 1 06:43 thirsty_colden.log -rw-------. 1 root root 4.5M Feb 1 07:01 run-parts.log -rw-------. 1 root root 16M Feb 1 07:01 CROND.log -rw-------. 1 root root 109M Feb 1 07:06 python3.log -rw-------. 1 root root 3.4M Feb 1 07:29 systemd-journald.log -rw-------. 1 root root 596M Feb 1 07:34 sudo.log -rw-------. 1 root root 549 Feb 1 07:44 interesting_rosalind.log -rw-------. 1 root root 342K Feb 1 07:45 beautiful_hamilton.log -rw-------. 1 root root 348K Feb 1 07:45 cool_ride.log -rw-------. 1 root root 15G Feb 1 07:45 conmon.log -rw-------. 1 root root 395K Feb 1 07:45 compassionate_satoshi.log -rw-------. 1 root root 11K Feb 1 07:45 hardcore_noether.log -rw-------. 1 root root 223K Feb 1 07:45 wizardly_johnson.log -rw-------. 1 root root 270M Feb 1 07:49 sshd.log -rw-------. 1 root root 111M Feb 1 07:49 systemd-logind.log -rw-------. 1 root root 1.6G Feb 1 07:50 systemd.log -rw-------. 1 root root 119M Feb 1 07:54 rsyslogd.log -rw-------. 1 root root 94G Feb 1 07:55 ceph-8ee2d228-ed21-4580-8bbf-064.log -rw-------. 1 root root 1.1G Feb 1 07:56 podman.log -rw-------. 1 root root 1.8G Feb 1 07:58 ceph-mgr.log -rw-------. 1 root root 213G Feb 1 07:58 ceph-osd.log -rw-------. 1 root root 48G Feb 1 07:58 ceph-mon.log " Those are container names or something like that? The file content seems to be assorted bits from the ceph disk tool: " # cat goofy_hypatia.log Jun 7 04:12:43 dopey goofy_hypatia[3224681]: 167 167 Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> passed data devices: 22 physical, 0 LVM Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> relative data size: 1.0 Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> All data devices are unavailable Jun 24 09:00:08 dopey goofy_hypatia[2319188]: [] Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> passed data devices: 22 physical, 0 LVM Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> relative data size: 1.0 Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> All data devices are unavailable 2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> passed data devices: 2 physical, 0 LVM 2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> relative data size: 1.0 2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> All data devices are unavailable 2023-12-26T09:38:59+01:00 dopey goofy_hypatia[893177]: 167 167 2023-12-26T10:41:14+01:00 dopey goofy_hypatia[1057467]: 167 167 2024-01-05T21:00:37+01:00 dopey goofy_hypatia[3395314]: {} 2024-02-01T05:42:17+01:00 dopey goofy_hypatia[845150]: 167 167 " Anyone else had this issue? Suggestions on how to get a real program name instead? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 3

how can install latest dev release?
by garcetto 01 Feb '24

01 Feb '24

good morning, how can i install latest dev release using cephadm? thank you.

3 4

Merging two ceph clusters
by Nico Schottelius 01 Feb '24

01 Feb '24

Good morning, in the spirit of the previous thread, I am wondering if anyone ever succeeded in merging two separate ceph clusters into one? Background from my side: we are running multiple ceph clusters in k8s/rook, but we still have some Nautilus/Devuan based clusters that are hard-to-impossible to upgrade. One of the ideas was to shift over the existing workload of the older clusters into the newer clusters. In the best case the whole process would be possible while keeping everything running in the new and old cluster. One clear challenge is that the two clusters obviously have a different FSID. The pool (names) in the two clusters differ, but the pool IDs are overlapping. If anyone has ever done this or has ideas on how to accomplish this, I would be very interested in hearing your opinion. Greetings from the Swiss mountains, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Changing A Ceph Cluster's Front- And/Or Back-End Networks IP Address(es)
by duluxoz 01 Feb '24

01 Feb '24

Hi All, Quick Q: How easy/hard is it to change the IP networks of: 1) A Ceph Cluster's "Front-End" Network? 2) A Ceph Cluster's "Back-End" Network? Is it a "simply" matter of: a) Placing the Nodes in maintenance mode b) Changing a config file (I assume it's /etc/ceph/ceph.conf) on each Node c) Rebooting the Nodes d) Taking each Node out of Maintenance Mode Thanks in advance Cheers Dulux-Oz

2 1

S3 object appears in ListObject but 404 when issuing a GET
by Mathias Chapelain 31 Jan '24

31 Jan '24

Hello, Recently we got a problem from an internal customer on our S3. Our setup consist of roughly 10 servers with 140 OSDs. Our 3 RGWs are collocated with monitors on dedicated servers in a HA setup with HAProxy in front. We are running 16.2.14 on Podman with Cephadm. Our S3 is constantly having a traffic of 500 req/s average per RGW instance. The problem is described in this issue: https://tracker.ceph.com/issues/63935. Basically this customer is having a Grafana Mimir instance pushing to our S3 and during a compaction process it does a special pattern like this: ``` 29/Dec/2023:17:13:28.961 rgw-frontend~ rgw-backend/server-mon-01-rgw0 0/0/0/127/127 200 228 - - ---- 132/132/70/67/0 0/0 "PUT /1234/object HTTP/1.1" 29/Dec/2023:17:13:29.101 rgw-frontend~ rgw-backend/server-mon-01-rgw0 0/0/0/1/1 200 381 - - ---- 132/132/76/71/0 0/0 "GET /1234/object HTTP/1.1" 29/Dec/2023:17:13:29.121 rgw-frontend~ rgw-backend/server-mon-01-rgw0 0/0/0/1/1 200 381 - - ---- 132/132/71/59/0 0/0 "GET /1234/object HTTP/1.1" 29/Dec/2023:17:13:29.137 rgw-frontend~ rgw-backend/server-mon-03-rgw0 0/0/0/4/4 204 153 - - ---- 132/132/71/6/0 0/0 "DELETE /1234/object HTTP/1.1" 29/Dec/2023:19:03:21.671 rgw-frontend~ rgw-backend/server-mon-03-rgw0 0/0/0/1/1 404 472 - - ---- 55/55/26/0/0 0/0 "GET /1234/object HTTP/1.1" ``` It is doing PUT, GET and DELETE in the same second. Afterwards the customer can see the deleted object when doing a ListObjects in the bucket but if he tries to access it then RGW returns a 404. After looking in Ceph, it appears the object has a bucket index entry but the associated RADOS object does not exist anymore. The bucket does not have versioning or object locking. Did someone encounter something similar? Thank you! Regards, -- Mathias Chapelain Storage Engineer Proton AG

1 0

Pacific: Drain hosts does not remove mgr daemon
by Mevludin Blazevic 31 Jan '24

31 Jan '24

Hi all, after performing "ceph orch host drain" on one of our host with only the mgr container left, I encounter that another mgr daemon is indeed deployed on another host, but the "old" does not get removed from the drain command. The same happens if I edit the mgr service via UI to define different hosts for the daemon and again the old mgr daemons are not getting removed. Any recommendations? I am using a setup with podman and RHEL. Best, Mevludin

2 1

Help on rgw metrics (was rgw_user_counters_cache)
by garcetto 31 Jan '24

31 Jan '24

good morning, i was struggling trying to understand why i cannot find this setting on my reef version, is it because is only on latest dev ceph version and not before? https://docs.ceph.com/en/*latest* /radosgw/metrics/#user-bucket-counter-caches Reef gives 404.... https://docs.ceph.com/en/reef/radosgw/metrics/ thank you!

2 1

NFS HA - "virtual_ip": null after upgrade to reef
by Torkil Svensgaard 31 Jan '24

31 Jan '24

Hi Last week we created an NFS service like this: " ceph nfs cluster create jumbo "ceph-flash1,ceph-flash2,ceph-flash3" --ingress --virtual_ip 172.21.15.74/22 --ingress-mode keepalive-only " Worked like a charm. Yesterday we upgraded from 17.2.7 to 18.20.0 and the NFS virtual IP seems to have gone missing in the process: " # ceph nfs cluster info jumbo { "jumbo": { "backend": [ { "hostname": "ceph-flash1", "ip": "172.21.15.148", "port": 2049 } ], "virtual_ip": null } } " Service spec: " service_type: nfs service_id: jumbo service_name: nfs.jumbo placement: count: 1 hosts: - ceph-flash1 - ceph-flash2 - ceph-flash3 spec: port: 2049 virtual_ip: 172.21.15.74 " I've tried restarting the nfs.jumbo service which didn't help. Suggestions? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 7

how to avoid pglogs dups bug in Pacific
by ADRIAN NICOLAE 31 Jan '24

31 Jan '24

Hi, I'm running Pacific 16.2.4 and I want to start a manual pg split process on the data pool (from 2048 to 4096). I'm reluctant to upgrade to 16.2.14/15 at this point. Can I avoid the dups bug (https://tracker.ceph.com/issues/53729) if I will increase the pgs slowly with 32 or 64pgs at every increment instead of moving directly to 4096 ? I don't have the autoscaler enabled. Thanks.

1 0