Please help collecting stats of Ceph monitor disk writes
Hi! Further to my thread "Ceph 16.2.x mon compactions, disk writes" ( https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XGCI2LFW5RH3...) where we have established that Ceph monitors indeed write considerable amounts of data to disks, I would like to request fellow Ceph users to provide feedback and help gather some statistics regarding whether this happens on all clusters or on some specific subset of clusters. The procedure is rather simple and won't take much of your time. If you are willing to help, please follow this procedure: --------- 1. Install iotop and run the following command on any of your monitor nodes: iotop -ao -bn 2 -d 300 2>&1 | grep -E "TID|ceph-mon" This will collect a 5-minute disk I/O statistics and produce an output containing the stats for Ceph monitor threads running on the node: TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 4854 be/4 167 8.62 M 2.27 G 0.00 % 0.72 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [rocksdb:low0] 4919 be/4 167 0.00 B 39.43 M 0.00 % 0.02 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [ms_dispatch] 4855 be/4 167 8.00 K 19.55 M 0.00 % 0.00 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [rocksdb:high0] We're particularly interested in the amount of written data. --------- 2. Optional: collect the number of "manual compaction" events from the monitor. This step will depend on how your monitor runs. My cluster is managed by cephadm and monitors run in docker containers, thus I can do something like this, where MYMONCONTAINERID is the container ID of Ceph monitor: # date; d=$(date +'%Y-%m-%d'); docker logs MYMONCONTAINERID 2>&1 | grep $d | grep -ci "manual compaction from" Fri 13 Oct 2023 06:29:39 AM UTC 580 Alternatively, I could run the command against the log file MYMONLOGFILE, whose location I obtained with docker inspect: # date; d=$(date +'%Y-%m-%d'); grep $d MYMONLOGFILE | grep -ci "manual compaction from" Fri 13 Oct 2023 06:35:27 AM UTC 588 If you run monitors with podman or without containerization, please get this information the way that is most convenient in your setup. --------- 3. Optional: collect the monitor store.db size. Usually the monitor store.db is available at /var/lib/ceph/FSID/mon.NAME/store.db/, for example: # du -hs /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/ 642M /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/ --------- 4. Optional: collect Ceph cluster version and status. For example: root@ceph01:/# ceph version; ceph -s ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_OK services: mon: 5 daemons, quorum ceph01,ceph03,ceph04,ceph05,ceph02 (age 2w) mgr: ceph01.vankui(active, since 13d), standbys: ceph02.shsinf osd: 96 osds: 96 up (since 2w), 95 in (since 3w) data: pools: 10 pools, 2400 pgs objects: 6.30M objects, 16 TiB usage: 61 TiB used, 716 TiB / 777 TiB avail pgs: 2396 active+clean 3 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 71 MiB/s rd, 60 MiB/s wr, 2.94k op/s rd, 2.56k op/s wr --------- 5. Reply to this thread and submit the collected information. For example: 1) iotop results: ... Paste data obtained in step 1) 2) manual compactions: ... Paste data obtained in step 2), or put "N/A" 3) monitor store.db size: ... Paste data obtained in step 3), or put "N/A" 4) cluster version and status: ... Paste data obtained in step 4), or put "N/A" ------------- I would very much appreciate your effort and help with gathering these stats. Please don't hesitate to contact me with any questions or concerns. Best regards, Zakhar
Hi, this is on our nautilus cluster, not sure if it is relevant, however here are the results: 1) iotop results: TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 1801 be/4 ceph 0.00 B 108.10 M 0.00 % 0.02 % ceph-mon -f --cluster ceph --id cephmon-01 --setuser ceph --setgroup ceph [rocksdb:low0] 1840 be/4 ceph 0.00 B 3.95 M 0.00 % 0.00 % ceph-mon -f --cluster ceph --id cephmon-01 --setuser ceph --setgroup ceph [fn_monstore] 1859 be/4 ceph 0.00 B 3.89 M 0.00 % 0.00 % ceph-mon -f --cluster ceph --id cephmon-01 --setuser ceph --setgroup ceph [safe_timer] 1802 be/4 ceph 0.00 B 2.82 M 0.00 % 0.00 % ceph-mon -f --cluster ceph --id cephmon-01 --setuser ceph --setgroup ceph [rocksdb:high0] 1742 be/4 ceph 0.00 B 56.00 K 0.00 % 0.00 % ceph-mon -f --cluster ceph --id cephmon-01 --setuser ceph --setgroup ceph [log] 2) manual compactions: Fri Oct 13 10:11:59 CEST 2023 127 3) monitor store.db size: 165M /var/lib/ceph/mon/ceph-cephmon-01/store.db/ 4) cluster version and status: ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) cluster: id: 43587260-09d9-4f3b-b118-365b5fc8ab64 health: HEALTH_WARN 1 clients failing to respond to cache pressure services: mon: 3 daemons, quorum cephmon-01,cephmon-02,cephmon-03 (age 6M) mgr: cephmon-02(active, since 15M), standbys: cephmon-03, cephmon-01 mds: cephfs:1 {0=cephmds-02=up:active} 1 up:standby-replay 1 up:standby osd: 240 osds: 240 up (since 6d), 240 in (since 6d) data: pools: 3 pools, 5184 pgs objects: 276.38M objects, 719 TiB usage: 1.1 PiB used, 498 TiB / 1.6 PiB avail pgs: 5175 active+clean 9 active+clean+scrubbing+deep io: client: 15 MiB/s rd, 1.8 MiB/s wr, 50 op/s rd, 3 op/s wr On 10/13/23 08:58, Zakhar Kirpichenko wrote:
Hi!
Further to my thread "Ceph 16.2.x mon compactions, disk writes" ( https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XGCI2LFW5RH3...) where we have established that Ceph monitors indeed write considerable amounts of data to disks, I would like to request fellow Ceph users to provide feedback and help gather some statistics regarding whether this happens on all clusters or on some specific subset of clusters.
The procedure is rather simple and won't take much of your time.
If you are willing to help, please follow this procedure:
---------
1. Install iotop and run the following command on any of your monitor nodes:
iotop -ao -bn 2 -d 300 2>&1 | grep -E "TID|ceph-mon"
This will collect a 5-minute disk I/O statistics and produce an output containing the stats for Ceph monitor threads running on the node:
TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 4854 be/4 167 8.62 M 2.27 G 0.00 % 0.72 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [rocksdb:low0] 4919 be/4 167 0.00 B 39.43 M 0.00 % 0.02 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [ms_dispatch] 4855 be/4 167 8.00 K 19.55 M 0.00 % 0.00 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [rocksdb:high0]
We're particularly interested in the amount of written data.
---------
2. Optional: collect the number of "manual compaction" events from the monitor.
This step will depend on how your monitor runs. My cluster is managed by cephadm and monitors run in docker containers, thus I can do something like this, where MYMONCONTAINERID is the container ID of Ceph monitor:
# date; d=$(date +'%Y-%m-%d'); docker logs MYMONCONTAINERID 2>&1 | grep $d | grep -ci "manual compaction from" Fri 13 Oct 2023 06:29:39 AM UTC 580
Alternatively, I could run the command against the log file MYMONLOGFILE, whose location I obtained with docker inspect:
# date; d=$(date +'%Y-%m-%d'); grep $d MYMONLOGFILE | grep -ci "manual compaction from" Fri 13 Oct 2023 06:35:27 AM UTC 588
If you run monitors with podman or without containerization, please get this information the way that is most convenient in your setup.
---------
3. Optional: collect the monitor store.db size.
Usually the monitor store.db is available at /var/lib/ceph/FSID/mon.NAME/store.db/, for example:
# du -hs /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/ 642M /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/
---------
4. Optional: collect Ceph cluster version and status.
For example:
root@ceph01:/# ceph version; ceph -s ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_OK
services: mon: 5 daemons, quorum ceph01,ceph03,ceph04,ceph05,ceph02 (age 2w) mgr: ceph01.vankui(active, since 13d), standbys: ceph02.shsinf osd: 96 osds: 96 up (since 2w), 95 in (since 3w)
data: pools: 10 pools, 2400 pgs objects: 6.30M objects, 16 TiB usage: 61 TiB used, 716 TiB / 777 TiB avail pgs: 2396 active+clean 3 active+clean+scrubbing+deep 1 active+clean+scrubbing
io: client: 71 MiB/s rd, 60 MiB/s wr, 2.94k op/s rd, 2.56k op/s wr
---------
5. Reply to this thread and submit the collected information.
For example:
1) iotop results: ... Paste data obtained in step 1)
2) manual compactions: ... Paste data obtained in step 2), or put "N/A"
3) monitor store.db size: ... Paste data obtained in step 3), or put "N/A"
4) cluster version and status: ... Paste data obtained in step 4), or put "N/A"
-------------
I would very much appreciate your effort and help with gathering these stats. Please don't hesitate to contact me with any questions or concerns.
Best regards,
Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _________________________________________________________ D i e t m a r R i e d e r, Mag.Dr. Head of Bioinformatics Core Facility Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 | Mobile: +43 676 8716 72402 Email: dietmar.rieder@i-med.ac.at Web: http://www.icbi.at
Hi, thanks for looking into this: our system disks also wear out too quickly! Here are the numbers on our small cluster. Best, 1) iotop results: TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 6426 be/4 ceph 0.00 B 590.00 K ?unavailable? ceph-mon -f --cluster ceph --id hpc1a --setuser ceph --setgroup ceph [log] 6813 be/4 ceph 275.49 M 275.93 M ?unavailable? ceph-mon -f --cluster ceph --id hpc1a --setuser ceph --setgroup ceph [rocksdb:low0] 6814 be/4 ceph 145.00 K 30.93 M ?unavailable? ceph-mon -f --cluster ceph --id hpc1a --setuser ceph --setgroup ceph [rocksdb:high0] 7087 be/4 ceph 25.87 M 14.54 M ?unavailable? ceph-mon -f --cluster ceph --id hpc1a --setuser ceph --setgroup ceph [fn_monstore] 7094 be/4 ceph 12.02 M 7.46 M ?unavailable? ceph-mon -f --cluster ceph --id hpc1a --setuser ceph --setgroup ceph [safe_timer] 7099 be/4 ceph 33.00 K 74.00 K ?unavailable? ceph-mon -f --cluster ceph --id hpc1a --setuser ceph --setgroup ceph [ms_dispatch] 2) manual compactions: Fri 13 Oct 2023 11:26:07 AM CEST 523 3) monitor store.db size: 8.0M /var/lib/ceph/mon/ceph-hpc1a/store.db/ 4) cluster version and status: ceph version 16.2.13 (b81a1d7f978c8d41cf452da7af14e190542d2ee2) pacific (stable) cluster: id: b351decf-4168-45ec-b8de-372051cf634a health: HEALTH_OK services: mon: 5 daemons, quorum hpc1a,hpc1b,hpc2c,hpcg2,hpc2d (age 7d) mgr: hpc1a(active, since 9d), standbys: hpc2c, hpcg2 mds: 9/9 daemons up, 4 standby osd: 13 osds: 13 up (since 11d), 13 in (since 7w) data: volumes: 9/9 healthy pools: 21 pools, 475 pgs objects: 27.53M objects, 13 TiB usage: 34 TiB used, 24 TiB / 57 TiB avail pgs: 475 active+clean io: client: 4.9 MiB/s rd, 61 MiB/s wr, 12 op/s rd, 993 op/s wr
Here is some data from a small, very lightly loaded cluster. It is manually deployed on debian11, with the mon store on an SSD: 1) iotop results: TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 1923 be/4 ceph 0.00 B 104.00 K ?unavailable? ceph-mon -f --cluster ceph --id mon1 --setuser ceph --setgroup ceph [log] 1951 be/4 ceph 0.00 B 264.82 M ?unavailable? ceph-mon -f --cluster ceph --id mon1 --setuser ceph --setgroup ceph [rocksdb:low0] 1952 be/4 ceph 0.00 B 7.25 M ?unavailable? ceph-mon -f --cluster ceph --id mon1 --setuser ceph --setgroup ceph [rocksdb:high0] 2148 be/4 ceph 0.00 B 6.51 M ?unavailable? ceph-mon -f --cluster ceph --id mon1 --setuser ceph --setgroup ceph [fn_monstore] 2155 be/4 ceph 0.00 B 3.62 M ?unavailable? ceph-mon -f --cluster ceph --id mon1 --setuser ceph --setgroup ceph [safe_timer] 2160 be/4 ceph 0.00 B 248.00 K ?unavailable? ceph-mon -f --cluster ceph --id mon1 --setuser ceph --setgroup ceph [ms_dispatch] 2) manual compactions: Fri 13 Oct 11:23:48 BST 2023 3780 3) monitor store.db size: 136M /var/lib/ceph/mon/ceph-mon1/store.db/ 4) cluster version and status: root@mon1:~# ceph version; ceph -s ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) cluster: id: 9208361c-5b68-41ed-8155-cc246a3fe538 health: HEALTH_OK services: mon: 3 daemons, quorum mon1,mon2,mon3 (age 5d) mgr: mon3(active, since 5d), standbys: mon1, mon2 mds: 1/1 daemons up, 2 standby osd: 18 osds: 18 up (since 5d), 18 in (since 5d) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 14 pools, 745 pgs objects: 3.32M objects, 7.2 TiB usage: 21 TiB used, 24 TiB / 46 TiB avail pgs: 745 active+clean io: client: 0 B/s rd, 432 KiB/s wr, 0 op/s rd, 5 op/s wr On 13/10/2023 07:58, Zakhar Kirpichenko wrote:
Hi!
Further to my thread "Ceph 16.2.x mon compactions, disk writes" ( https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XGCI2LFW5RH3...) where we have established that Ceph monitors indeed write considerable amounts of data to disks, I would like to request fellow Ceph users to provide feedback and help gather some statistics regarding whether this happens on all clusters or on some specific subset of clusters.
The procedure is rather simple and won't take much of your time.
If you are willing to help, please follow this procedure:
---------
1. Install iotop and run the following command on any of your monitor nodes:
iotop -ao -bn 2 -d 300 2>&1 | grep -E "TID|ceph-mon"
This will collect a 5-minute disk I/O statistics and produce an output containing the stats for Ceph monitor threads running on the node:
TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 4854 be/4 167 8.62 M 2.27 G 0.00 % 0.72 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [rocksdb:low0] 4919 be/4 167 0.00 B 39.43 M 0.00 % 0.02 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [ms_dispatch] 4855 be/4 167 8.00 K 19.55 M 0.00 % 0.00 % ceph-mon -n mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true [rocksdb:high0]
We're particularly interested in the amount of written data.
---------
2. Optional: collect the number of "manual compaction" events from the monitor.
This step will depend on how your monitor runs. My cluster is managed by cephadm and monitors run in docker containers, thus I can do something like this, where MYMONCONTAINERID is the container ID of Ceph monitor:
# date; d=$(date +'%Y-%m-%d'); docker logs MYMONCONTAINERID 2>&1 | grep $d | grep -ci "manual compaction from" Fri 13 Oct 2023 06:29:39 AM UTC 580
Alternatively, I could run the command against the log file MYMONLOGFILE, whose location I obtained with docker inspect:
# date; d=$(date +'%Y-%m-%d'); grep $d MYMONLOGFILE | grep -ci "manual compaction from" Fri 13 Oct 2023 06:35:27 AM UTC 588
If you run monitors with podman or without containerization, please get this information the way that is most convenient in your setup.
---------
3. Optional: collect the monitor store.db size.
Usually the monitor store.db is available at /var/lib/ceph/FSID/mon.NAME/store.db/, for example:
# du -hs /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/ 642M /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/
---------
4. Optional: collect Ceph cluster version and status.
For example:
root@ceph01:/# ceph version; ceph -s ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_OK
services: mon: 5 daemons, quorum ceph01,ceph03,ceph04,ceph05,ceph02 (age 2w) mgr: ceph01.vankui(active, since 13d), standbys: ceph02.shsinf osd: 96 osds: 96 up (since 2w), 95 in (since 3w)
data: pools: 10 pools, 2400 pgs objects: 6.30M objects, 16 TiB usage: 61 TiB used, 716 TiB / 777 TiB avail pgs: 2396 active+clean 3 active+clean+scrubbing+deep 1 active+clean+scrubbing
io: client: 71 MiB/s rd, 60 MiB/s wr, 2.94k op/s rd, 2.56k op/s wr
---------
5. Reply to this thread and submit the collected information.
For example:
1) iotop results: ... Paste data obtained in step 1)
2) manual compactions: ... Paste data obtained in step 2), or put "N/A"
3) monitor store.db size: ... Paste data obtained in step 3), or put "N/A"
4) cluster version and status: ... Paste data obtained in step 4), or put "N/A"
-------------
I would very much appreciate your effort and help with gathering these stats. Please don't hesitate to contact me with any questions or concerns.
Best regards,
Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Zakhar, I'm pretty sure you wanted the #manual compactions for an entire day, not from whenever the log starts to current time, which is most often not 23:59. You need to get the date from the previous day and make sure the log contains a full 00:00-23:59 window. 1) iotop results: TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 2256 be/4 ceph 0.00 B 17.48 M 0.00 % 0.80 % ceph-mon --cluster ceph --setuser ceph --setgroup ceph --foreground -i ceph-01 --mon-data /var/lib/ceph/mon/ceph-ceph-01 --public-addr 192.168.32.65 [safe_timer] 2230 be/4 ceph 0.00 B 1514.19 M 0.00 % 0.37 % ceph-mon --cluster ceph --setuser ceph --setgroup ceph --foreground -i ceph-01 --mon-data /var/lib/ceph/mon/ceph-ceph-01 --public-addr 192.168.32.65 [rocksdb:low0] 2250 be/4 ceph 0.00 B 36.23 M 0.00 % 0.15 % ceph-mon --cluster ceph --setuser ceph --setgroup ceph --foreground -i ceph-01 --mon-data /var/lib/ceph/mon/ceph-ceph-01 --public-addr 192.168.32.65 [fn_monstore] 2231 be/4 ceph 0.00 B 50.52 M 0.00 % 0.02 % ceph-mon --cluster ceph --setuser ceph --setgroup ceph --foreground -i ceph-01 --mon-data /var/lib/ceph/mon/ceph-ceph-01 --public-addr 192.168.32.65 [rocksdb:high0] 2225 be/4 ceph 0.00 B 120.00 K 0.00 % 0.00 % ceph-mon --cluster ceph --setuser ceph --setgroup ceph --foreground -i ceph-01 --mon-data /var/lib/ceph/mon/ceph-ceph-01 --public-addr 192.168.32.65 [log] 2) manual compactions (over a full 24h window): 1882 3) monitor store.db size: 616M 4) cluster version and status: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) cluster: id: xxx health: HEALTH_WARN 1 large omap objects services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 7w) mgr: ceph-25(active, since 4w), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1282 up (since 27h), 1282 in (since 3w) task status: data: pools: 14 pools, 25065 pgs objects: 2.18G objects, 3.9 PiB usage: 4.8 PiB used, 8.3 PiB / 13 PiB avail pgs: 25037 active+clean 26 active+clean+scrubbing+deep 2 active+clean+scrubbing io: client: 1.7 GiB/s rd, 1013 MiB/s wr, 3.02k op/s rd, 1.78k op/s wr Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Thank you, Frank. Tbh, I think it doesn't matter if the number of manual compactions is for 24h or for a smaller period, as long as it's over a reasonable period of time, so that an average number of compactions per hour can be calculated. /Z On Fri, 13 Oct 2023 at 16:01, Frank Schilder <frans@dtu.dk> wrote:
Hi Zakhar,
I'm pretty sure you wanted the #manual compactions for an entire day, not from whenever the log starts to current time, which is most often not 23:59. You need to get the date from the previous day and make sure the log contains a full 00:00-23:59 window.
1) iotop results: TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 2256 be/4 ceph 0.00 B 17.48 M 0.00 % 0.80 % ceph-mon --cluster ceph --setuser ceph --setgroup ceph --foreground -i ceph-01 --mon-data /var/lib/ceph/mon/ceph-ceph-01 --public-addr 192.168.32.65 [safe_timer] 2230 be/4 ceph 0.00 B 1514.19 M 0.00 % 0.37 % ceph-mon --cluster ceph --setuser ceph --setgroup ceph --foreground -i ceph-01 --mon-data /var/lib/ceph/mon/ceph-ceph-01 --public-addr 192.168.32.65 [rocksdb:low0] 2250 be/4 ceph 0.00 B 36.23 M 0.00 % 0.15 % ceph-mon --cluster ceph --setuser ceph --setgroup ceph --foreground -i ceph-01 --mon-data /var/lib/ceph/mon/ceph-ceph-01 --public-addr 192.168.32.65 [fn_monstore] 2231 be/4 ceph 0.00 B 50.52 M 0.00 % 0.02 % ceph-mon --cluster ceph --setuser ceph --setgroup ceph --foreground -i ceph-01 --mon-data /var/lib/ceph/mon/ceph-ceph-01 --public-addr 192.168.32.65 [rocksdb:high0] 2225 be/4 ceph 0.00 B 120.00 K 0.00 % 0.00 % ceph-mon --cluster ceph --setuser ceph --setgroup ceph --foreground -i ceph-01 --mon-data /var/lib/ceph/mon/ceph-ceph-01 --public-addr 192.168.32.65 [log]
2) manual compactions (over a full 24h window): 1882
3) monitor store.db size: 616M
4) cluster version and status:
ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)
cluster: id: xxx health: HEALTH_WARN 1 large omap objects
services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 7w) mgr: ceph-25(active, since 4w), standbys: ceph-26, ceph-01, ceph-03, ceph-02 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1282 up (since 27h), 1282 in (since 3w)
task status:
data: pools: 14 pools, 25065 pgs objects: 2.18G objects, 3.9 PiB usage: 4.8 PiB used, 8.3 PiB / 13 PiB avail pgs: 25037 active+clean 26 active+clean+scrubbing+deep 2 active+clean+scrubbing
io: client: 1.7 GiB/s rd, 1013 MiB/s wr, 3.02k op/s rd, 1.78k op/s wr
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
so that an average number of compactions per hour can be calculated
For this you need the time window length for the counts. That's not automatic with the command you sent. For example, my MON log file started at 4am in the morning, there was a logrotate. So the assumption that the current time and count will give a correct average for the day starting at 00:00 failed in my situation. To get a correct count for today I would have had to combine 2 log files and send you the current time and count: # grep -i "manual compaction from" /var/log/ceph/ceph-mon.ceph-01.log-20231013 /var/log/ceph/ceph-mon.ceph-01.log | grep "2023-10-13" | wc -l 1226 # grep -i "manual compaction from" /var/log/ceph/ceph-mon.ceph-01.log | grep "2023-10-13" | wc -l 912 # date Fri Oct 13 16:07:57 CEST 2023 Apart from that, I agree that a shorter time window will do. Its the start time of the log that's missing. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
participants (5)
-
Chris Palmer
-
Dietmar Rieder
-
Eric Le Lay
-
Frank Schilder
-
Zakhar Kirpichenko