Hi,
If I'm correct in a S3 installation it's good practice to have a HA proxy,
I also read somewhere the cephadm tool can deploy the HA Proxy.
But is it a good practice to use cephadm to deploy the HA Proxy or it's
better do deploy it manually on a other server (who does only that).
Regards
--
Albert SHIH 🦫 🐸
France
Heure locale/Local time:
mer. 27 mars 2024 09:18:04 CET
Hi everyone!
My Ceph cluster (17.2.6) has a CephFS volume which is showing 41TB usage
for the data pool, but there are only 5.5TB of files in it. There are
fewer than 100 files on the filesystem in total, so where is all that
space going?
How can I analyze my cephfs to understand what is using that space, and
if possible, how can I reclaim that space?
Thank you.
--
Regards,
Thorne Lawler - Senior System Administrator
*DDNS* | ABN 76 088 607 265
First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172
P +61 499 449 170
_DDNS
/_*Please note:* The information contained in this email message and any
attached files may be confidential information, and may also be the
subject of legal professional privilege. _If you are not the intended
recipient any use, disclosure or copying of this email is unauthorised.
_If you received this email in error, please notify Discount Domain Name
Services Pty Ltd on 03 9815 6868 to report this matter and delete all
copies of this transmission together with any attachments. /
If the documentation is to be believed, it's just install the zabbix
sender, then;
ceph mgr module enable zabbix
ceph zabbix config-set zabbix_host my-zabbix-server
(Optional) Set the identifier to the fsid.
And poof. I should now have a discovered entity on my zabbix server to add
templates to.
However, this has not worked yet on either of my ceph clusters (one RHEL,
one proxmox).
Reference: https://docs.ceph.com/en/latest/mgr/zabbix/
On Reddit advice, I installed the Ceph templates for Zabbix.
https://raw.githubusercontent.com/ceph/ceph/master/src/pybind/mgr/zabbix/za…
Still no dice. No traffic at all seems to be generated, that I've seen
from packet traces,
... OK.
I su'ed to the ceph user on both clusters, and ran zabbix_send:
zabbix_sender -v -z 10.0.0.1 -s "$my_fsid" -k ceph.osd_avg_pgs -o 1
Response from "10.0.0.1:10051": "processed: 1; failed: 0; total: 1; seconds
spent: 0.000042"
sent: 1; skipped: 0; total: 1
As the ceph user, ceph zabbix send/discovery still fail.
I am officially stumped.
Any ideas as to which tree I should be barking up?
Thanks in advance!
hi there, need some help please.
we are planning to replace our rbd-mirror setup and go to stretch mode.
the goal is, to have the cluster in 2 fire compartment server rooms.
start was a default proxmox/ceph setup.
now, i followed the howto from:
https://docs.ceph.com/en/latest/rados/operations/stretch-mode/
with the crushmap i ended in an error:
in rule 'stretch_rule' item 'dc1' not defined (dc1,2,3 is like site1,2,3
in the docu).
my osd tree looks like
pve-test02-01:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 3.80951 root default
-15 0.06349 host pve-test01-01
0 ssd 0.06349 osd.0 up 1.00000 1.00000
-19 0.06349 host pve-test01-02
1 ssd 0.06349 osd.1 up 1.00000 1.00000
-17 0.06349 host pve-test01-03
2 ssd 0.06349 osd.2 up 1.00000 1.00000
...
i think there is something missing.
should i need adding the buckets manually?
...
for disaster failover, any ideas are welcome.
man thanks for help and kind regards,
ronny
We transfer data (300 million small files) using rsync between cephfs
from version 12.2.13 to 18.2.1. After about the same time (about 7 hours
in this case), copying stops for a minute
```
health:HEALTH_WARN
1 clients failing to advance oldest client/flush tid
1 MDSs report slow metadata IOs
1 MDSs behind on trimming
```
The only interesting messages in the logs are:
```
ceph-mds[640738]: mds.beacon.cephfs.X missed beacon ack from the monitors
```
I watched debugging on the client (destination) via watch -n1
/sys/kernel/debug/ceph/451eea44-d7a0-11ee-9117-b496914b4c02.client32497
```
item total
------------------------------------------
opened files / total inodes 1 / 866757
pinned i_caps / total inodes 866757 / 866757
opened inodes / total inodes 1 / 866757
item total avg_lat(us) min_lat(us) max_lat(us) stdev(us)
--------------------------------------------------
---------------------------------
read 0 0 0 0 0
write 5129252 194689 9080 54693415 1023
metadata 29045143 670 161 87794124 369
item total miss hit
-------------------------------------------------
d_lease 5361 3026 315488442
caps 866757 11 483177478
```
During the copy, "pinned i_caps / total inodes" are gradually increased
until it reaches the value "mds_max_caps_per_client" (default: 1Mi).
Then "pinned i_caps / total inodes" begins to decrease to almost 0, at
which time HEALTH_WARN appears and transfer stops. "op/s wr" increases
from 200 to 1.5k. Then total inodes begin to increase again along with
the resumption of copying and the cluster goes into the HEALTHY state.
Mount options:
/mnt/cephfs-old 10.77.12.90:6789,10.77.12.91:6789,10.77.12.92:6789:/
ceph rw,noatime,nodiratime,name=admin,secret=<hidden>,acl
/mnt/cephfs-new 10.77.12.139:6789,10.77.12.140:6789,10.77.12.141:6789:/
ceph rw,noatime,nodiratime,name=admin,secret=<hidden>,acl,caps_max=10000
Client properties on the MDS server (removed unnecessary):
ceph daemon mds.cephfs.X client ls 32497
[
{
...
"id": 32497,
"state": "open",
"num_leases": 0,
"num_caps": 980679,
"request_load_avg": 7913,
"requests_in_flight": 466,
"num_completed_flushes": 464,
"recall_caps": {
"value": 0,
"halflife": 60
},
"release_caps": {
"value": 1732.2552002208533,
"halflife": 60
},
"recall_caps_throttle": {
"value": 0,
"halflife": 1.35000000000000001
},
"recall_caps_throttle2o": {
"value": 0,
"halflife": 0.5
},
"session_cache_liveness": {
"value": 42186.620275326415,
"halflife": 300
},
"cap_acquisition": {
"value": 0,
"halflife": 30
},
...
}
]
ceph daemonperf mds.cephfs.X
```
-------------------------------------------mds------
------------------------------------- --mds_cache--- ------mds_log
------------ -mds_mem- -------mds_server------- mds_ -----objecter------
purg
req rlat slr fwd inos caps exi imi hifc crev cgra ctru cfsa cfa hcc hccd
hccr prcr|stry recy recd|subm evts segs repl|ino dn |hcr hcs hsr cre cat
|sess|actv rd wr rdwr|purg|
114 0 0 0 1.9M 438k 0 0 0 0 253 0 0 0 0 0 0 59 | 0 0 0 |128 123k 129 0
|1.3M 1.9M|114 0 0 0 0 | 3 | 0 0 440 0 | 0
101 0 0 0 1.9M 438k 0 0 0 0 0 0 0 0 0 0 0 53 | 0 0 0 |106 123k 129 0
|1.3M 1.9M|101 0 0 0 0 | 3 | 0 0 0 0 | 0
...
``` - from this output it is clear that the client does not send cap
release at all (column hccr, decoding of the columns "ceph daemon mds.X
perf schema")
Then I was able to find the right query to google similar problems:
https://www.spinics.net/lists/ceph-users/msg50573.htmlhttps://ceph-users.ceph.narkive.com/mcyPtEyz/rsync-kernel-client-cepfs-mkst…https://www.spinics.net/lists/ceph-users/msg50158.htmlhttps://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/B7K6B5VXM3I…
It turns out that the problem is in rsync, in the way it works?
The only "solution" is to do it on the client according to a schedule
(or upon reaching a certain number of open caps) “echo 2 >
/proc/sys/vm/drop_caches”. After this command, the cephfs client
releases the cached caps. And if there were a lot of them, then MDS
becomes slow again.
We also tried to mount cephfs with the option "caps_max=10000" so that
the client would do a forced release when the specified value is
reached, but this did not help.
We can limit mds_max_caps_per_client (not tested), but this also affects
all clients at once.
The command "ceph daemon mds.cephfs.X cache drop" (with or without an
additional parameter) does not help
Tested on Linux kernels (client side): 5.10 and 6.1
Did I understand everything correctly? is this the expected behavior
when running rsync?
And one more problem (I don’t know if it’s related or not), when rsync
finishes copying, all caps are freed except the last two (pinned i_caps
/ total inodes 2 / 2)
At this moment a warning appears (or remains after releasing a large
number of caps): 1 clients failing to advance oldest client/flush tid
But then it doesn't disappear. I waited 12 hours.
Warning disappears only after executing the "sync" command on the
client. and in the client metrics you can see "pinned i_caps / total
inodes 1 / 1"
Note: running "echo 2 > /proc/sys/vm/drop_caches" does not help in this
case.
Hi,
We are trying to deploy Ceph Reef 18.2.1 using cephadm on mixed architecture hosts using x86_64 for the mons and aarch64 for the OSDs.
During deployment we use the following config for the bootstrap process, where $REPOSITORY is our docker repo.
[global]
container_image = $REPOSITORY/ceph/ceph:v18.2.1
[mgr]
mgr/cephadm/container_image_base = $REPOSITORY/ceph/ceph:v18.2.1
mgr/cephadm/container_image_prometheus = $REPOSITORY/ceph/prometheus:v2.33.4
mgr/cephadm/container_image_node_exporter = $REPOSITORY/ceph/node-exporter:v1.3.1
mgr/cephadm/container_image_grafana = $REPOSITORY/ceph/ceph-grafana:8.3.5
mgr/cephadm/container_image_alertmanager = $REPOSITORY/ceph/alertmanager:v0.23.0
[osd]
container_image = $REPOSITORY/ceph/ceph:v18.2.1
Once the bootstrap process is complete, if we do a ceph config dump, the container image for global and osd changes from version tag to sha reference, meaning that when deploying the containers on the OSDs they try using the amd64 container image and not the aarch64 image and fail deployment.
Is there a config setting we are missing or a workaround for this?
Thanks
Iain
Iain Stott
OpenStack Engineer
Iain.Stott(a)thg.com
[THG Ingenuity Logo]<https://www.thg.com>
www.thg.com<https://www.thg.com/>
[LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en>
Hi Y'all,
I'm seeing this warning via 'ceph -s' (this is on Reef):
# ceph -s
cluster:
id: 58bde08a-d7ed-11ee-9098-506b4b4da440
health: HEALTH_WARN
3 clients failing to advance oldest client/flush tid
1 MDSs report slow requests
1 MDSs behind on trimming
services:
mon: 5 daemons, quorum
pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 3d)
mgr: pr-md-01.jemmdf(active, since 3w), standbys: pr-md-02.emffhz
mds: 1/1 daemons up, 1 standby
osd: 46 osds: 46 up (since 3d), 46 in (since 2w)
data:
volumes: 1/1 healthy
pools: 4 pools, 1313 pgs
objects: 258.13M objects, 454 TiB
usage: 688 TiB used, 441 TiB / 1.1 PiB avail
pgs: 1303 active+clean
8 active+clean+scrubbing
2 active+clean+scrubbing+deep
io:
client: 131 MiB/s rd, 111 MiB/s wr, 41 op/s rd, 613 op/s wr
I googled around and looked at the docs and it seems like this isn't a
critical problem, but I couldn't find a clear path to resolution. Does
anyone have any advice on what I can do to resolve the health issues up top?
My CephFS filesystem is incredibly busy so I have a feeling that has
some impact here, but not 100% sure...
Thanks as always for the help!
cheers,
erich
Hi,
On my active mgr (only one) on my cluster I got repeatedly something like
Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists
Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists
Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists
Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists
Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists
Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists
Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists
Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists
Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists
Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists
I check the dashboard (first hit with google on that message) and that seem OK, no config about it listen address.
Those messages come each ~ 1 hour.
I notice those message appear each time just after
Mar 26 08:50:04 cthulhu2 ceph-mgr[2843]: [volumes INFO volumes.module] Starting _cmd_fs_subvolume_ls(prefix:fs subvolume ls, target:['mon-mgr', ''], vol_name:cephfs) < ""
but I don't known if that's related.
Any clue ?
Regards
--
Albert SHIH 🦫 🐸
France
Heure locale/Local time:
mar. 26 mars 2024 10:52:53 CET
