3 Feb
2020
3 Feb
'20
7:38 p.m.
Hello,
I have this message on my new ceph cluster in Nautilus. I have a cephfs with a copy of
~100TB in progress.
/var/log/ceph/artemis.log:2020-02-03 16:22:49.970437
osd.66 (osd.66) 1137 : cluster [WRN] Large omap object found. Object:
8:579bf162:::mds3_openfiles.0:head PG: 8.468fd9ea (8.2a) Key count: 206548 Size (bytes):
6691941
/var/log/ceph/artemis-osd.66.log:2020-02-03
16:22:49.966 7fe77af62700 0 log_channel(cluster) log [WRN] : Large omap object found.
Object: 8:579bf162:::mds3_openfiles.0:head PG: 8.468fd9ea (8.2a) Key count: 206548 Size
(bytes): 6691941
I found this thread about a similar issue in the archives of the list
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/JUFYDCQ2AHF…
But I'm not sure what I can do in my situation, can I increase
osd_deep_scrub_large_omap_object_key_threshold or it's a bad idea?
Thanks for your help.
Here some useful (I guess) information:
Filesystem Size Used Avail
Use% Mounted on
10.90.37.4,10.90.37.6,10.90.37.8:/ 329T 32T 297T 10% /artemis
artemis@icitsrv5:~$ ceph -s
cluster:
id: 815ea021-7839-4a63-9dc1-14f8c5feecc6
health: HEALTH_WARN
1 large omap objects
services:
mon: 3 daemons, quorum iccluster003,iccluster005,iccluster007 (age 2w)
mgr: iccluster021(active, since 7h), standbys: iccluster009, iccluster023
mds: cephfs:5 5 up:active
osd: 120 osds: 120 up (since 5d), 120 in (since 5d)
rgw: 8 daemons active (iccluster003.rgw0, iccluster005.rgw0, iccluster007.rgw0,
iccluster013.rgw0, iccluster015.rgw0, iccluster019.rgw0, iccluster021.rgw0,
iccluster023.rgw0)
data:
pools: 10 pools, 2161 pgs
objects: 72.02M objects, 125 TiB
usage: 188 TiB used, 475 TiB / 662 TiB avail
pgs: 2157 active+clean
4 active+clean+scrubbing+deep
io:
client: 31 KiB/s rd, 803 KiB/s wr, 31 op/s rd, 184 op/s wr
artemis@icitsrv5:~$ ceph health detail
HEALTH_WARN 1 large omap objects
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool 'cephfs_metadata'
Search the cluster log for 'Large omap object found' for more details.
artemis@icitsrv5:~$ ceph fs status
cephfs - 3 clients
======
+------+--------+--------------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+--------------+---------------+-------+-------+
| 0 | active | iccluster015 | Reqs: 0 /s | 251k | 251k |
| 1 | active | iccluster001 | Reqs: 3 /s | 20.2k | 19.1k |
| 2 | active | iccluster017 | Reqs: 1 /s | 116k | 112k |
| 3 | active | iccluster019 | Reqs: 0 /s | 263k | 263k |
| 4 | active | iccluster013 | Reqs: 123 /s | 16.3k | 16.3k |
+------+--------+--------------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 13.9G | 135T |
| cephfs_data | data | 51.3T | 296T |
+-----------------+----------+-------+-------+
+-------------+
| Standby MDS |
+-------------+
+-------------+
MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus
(stable)
root@iccluster019:~# ceph --cluster artemis daemon osd.13 config show | grep large_omap
"osd_deep_scrub_large_omap_object_key_threshold": "200000",
"osd_deep_scrub_large_omap_object_value_sum_threshold":
"1073741824",
artemis@icitsrv5:~$ rados -p cephfs_metadata listxattr
mds3_openfiles.0
artemis@icitsrv5:~$ rados -p cephfs_metadata getomapheader mds3_openfiles.0
header (42 bytes) :
00000000 13 00 00 00 63 65 70 68 20 66 73 20 76 6f 6c 75 |....ceph fs volu|
00000010 6d 65 20 76 30 31 31 01 01 0d 00 00 00 14 63 00 |me v011.......c.|
00000020 00 00 00 00 00 01 00 00 00 00 |..........|
0000002a
Best regards,
--
Yoann Moulin
EPFL IC-IT