ceph-users
Threads by month
- ----- 2026 -----
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- 43 participants
- 9162 discussions
Ceph 16.2.14: how to set mon_rocksdb_options to enable RocksDB compression?
by Zakhar Kirpichenko 17 Oct '23
by Zakhar Kirpichenko 17 Oct '23
17 Oct '23
Hi,
I'm still trying to fight large Ceph monitor writes. One option I
considered is enabling RocksDB compression, as our nodes have more than
sufficient RAM and CPU. Unfortunately, monitors seem to completely ignore
the compression setting:
I tried:
- setting ceph config set mon.ceph05 mon_rocksdb_options
"write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
restarting the test monitor. The monitor started with no RocksDB
compression:
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Compression
algorithms supported:
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
kZSTDNotFinalCompression supported: 0
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
kXpressCompression supported: 0
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
kLZ4HCCompression supported: 1
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
kLZ4Compression supported: 1
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
kBZip2Compression supported: 0
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
kZlibCompression supported: 1
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
kSnappyCompression supported: 1
...
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
Options.compression: NoCompression
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
Options.bottommost_compression: Disabled
- setting ceph config set mon mon_rocksdb_options
"write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
restarting the test monitor. The monitor started with no RocksDB
compression, the same way as above.
In each case config options were correctly set and readable with config
get. I also found a suggestion in ceph-users (
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJM232IHN7…)
to set compression in a similar manner. Unfortunately, these options appear
to be ignored.
How can I enable RocksDB compression in Ceph monitors?
I would very much appreciate your advices and comments.
Best regards,
Zakhar
3
8
Hi,
For some reason, I need to recalculate the bucket stats. Is this possible?
Thanks
1
0
NFS - HA and Ingress: [ https://docs.ceph.com/en/latest/mgr/nfs/#ingress ]
Referring to Note#2, is NFS high-availability functionality considered complete (and stable)?
1
0
Hello.
It's been a while. For the past couple years I've had a cluster running
Nautilus on Debian 10 using the Debian Ceph packages, and deployed with
Ceph-Ansible. It's not a huge cluster - 10 OSD nodes with 80 x 12TB HDD
OSDs, plus 3 management nodes, and about 40% full at the moment - but it is
a critical resource for one of our researchers.
Back then I had some misgivings about non-Debian packages and also about
containerized Ceph. I don't know if my feelings about these things have
changed that much, but it's time to upgrade, and, with the advent of
cephadm it looks like it's just better to stay mainstream.
So I'm looking for advice on how to get from where I'm at to at least
Pacific or Quincy.
I've read a little in the last couple days. I've seen various opinions on
(not) skipping releases and on when to switch to cephadm. I'm also
concerned about cleaning up those old Debian packages - will there be a
point where I can 'apt-get purge' them without harming the cluster.
One particular thing: The upgrade instructions in various places on
docs.ceph.com say something like
Upgrade monitors by installing the new packages and restarting the monitor
daemons.
To me this is kind of vague. Perhaps there is a different concept fo
'packages' within the cephadm environment. I could really use some
clarification on this.
I'd also consider decommissioning a few nodes, setting up a new cluster on
fresh Debian installs. and migrating the data and remaining nodes. This
would be a long and painful process - decommission a node, move it, move
some data, decommission another node - and I don't know what effect it
would have on external references to our object store.
Please advise.
Thanks.
-Dave
--
Dave Hall
Binghamton University
kdhall(a)binghamton.edu
3
3
Hi
We have kerberos working with bare metal kernel NFS exporting RBDs. I
can see in the ceph documentation[1] that nfs-ganesha should work with
kerberos but I'm having little luck getting it to work.
This bit from the container log seems to suggest that some plumbing is
missing?
"
13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 :
ganesha.nfsd-2[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT
:Callback creds directory (/var/run/ganesha) already exists
13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 :
ganesha.nfsd-2[main] find_keytab_entry :NFS CB :WARN :Configuration file
does not specify default realm while getting default realm name
13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 :
ganesha.nfsd-2[main] gssd_refresh_krb5_machine_credential :NFS CB :CRIT
:ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry
found in keytab /etc/krb5.keytab for connection with host localhost
13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 :
ganesha.nfsd-2[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN
:gssd_refresh_krb5_machine_credential failed (-1765328160:0)
"
Thoughts?
Mvh.
Torkil
[1] https://docs.ceph.com/en/quincy/mgr/nfs/#create-cephfs-export
--
Torkil Svensgaard
Systems Administrator
Danish Research Centre for Magnetic Resonance DRCMR, Section 714
Copenhagen University Hospital Amager and Hvidovre
Kettegaard Allé 30, 2650 Hvidovre, Denmark
2
5
13 Oct '23
Hi!
Further to my thread "Ceph 16.2.x mon compactions, disk writes" (
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XGCI2LFW5RH…)
where we have established that Ceph monitors indeed write considerable
amounts of data to disks, I would like to request fellow Ceph users to
provide feedback and help gather some statistics regarding whether this
happens on all clusters or on some specific subset of clusters.
The procedure is rather simple and won't take much of your time.
If you are willing to help, please follow this procedure:
---------
1. Install iotop and run the following command on any of your monitor nodes:
iotop -ao -bn 2 -d 300 2>&1 | grep -E "TID|ceph-mon"
This will collect a 5-minute disk I/O statistics and produce an output
containing the stats for Ceph monitor threads running on the node:
TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
4854 be/4 167 8.62 M 2.27 G 0.00 % 0.72 % ceph-mon -n
mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
--default-mon-cluster-log-to-file=false
--default-mon-cluster-log-to-stderr=true [rocksdb:low0]
4919 be/4 167 0.00 B 39.43 M 0.00 % 0.02 % ceph-mon -n
mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
--default-mon-cluster-log-to-file=false
--default-mon-cluster-log-to-stderr=true [ms_dispatch]
4855 be/4 167 8.00 K 19.55 M 0.00 % 0.00 % ceph-mon -n
mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
--default-mon-cluster-log-to-file=false
--default-mon-cluster-log-to-stderr=true [rocksdb:high0]
We're particularly interested in the amount of written data.
---------
2. Optional: collect the number of "manual compaction" events from the
monitor.
This step will depend on how your monitor runs. My cluster is managed by
cephadm and monitors run in docker containers, thus I can do something like
this, where MYMONCONTAINERID is the container ID of Ceph monitor:
# date; d=$(date +'%Y-%m-%d'); docker logs MYMONCONTAINERID 2>&1 | grep $d
| grep -ci "manual compaction from"
Fri 13 Oct 2023 06:29:39 AM UTC
580
Alternatively, I could run the command against the log file MYMONLOGFILE,
whose location I obtained with docker inspect:
# date; d=$(date +'%Y-%m-%d'); grep $d MYMONLOGFILE | grep -ci "manual
compaction from"
Fri 13 Oct 2023 06:35:27 AM UTC
588
If you run monitors with podman or without containerization, please get
this information the way that is most convenient in your setup.
---------
3. Optional: collect the monitor store.db size.
Usually the monitor store.db is available at
/var/lib/ceph/FSID/mon.NAME/store.db/, for example:
# du -hs
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/
642M
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/
---------
4. Optional: collect Ceph cluster version and status.
For example:
root@ceph01:/# ceph version; ceph -s
ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific
(stable)
cluster:
id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86
health: HEALTH_OK
services:
mon: 5 daemons, quorum ceph01,ceph03,ceph04,ceph05,ceph02 (age 2w)
mgr: ceph01.vankui(active, since 13d), standbys: ceph02.shsinf
osd: 96 osds: 96 up (since 2w), 95 in (since 3w)
data:
pools: 10 pools, 2400 pgs
objects: 6.30M objects, 16 TiB
usage: 61 TiB used, 716 TiB / 777 TiB avail
pgs: 2396 active+clean
3 active+clean+scrubbing+deep
1 active+clean+scrubbing
io:
client: 71 MiB/s rd, 60 MiB/s wr, 2.94k op/s rd, 2.56k op/s wr
---------
5. Reply to this thread and submit the collected information.
For example:
1) iotop results:
... Paste data obtained in step 1)
2) manual compactions:
... Paste data obtained in step 2), or put "N/A"
3) monitor store.db size:
... Paste data obtained in step 3), or put "N/A"
4) cluster version and status:
... Paste data obtained in step 4), or put "N/A"
-------------
I would very much appreciate your effort and help with gathering these
stats. Please don't hesitate to contact me with any questions or concerns.
Best regards,
Zakhar
5
6
Hi,
while writing a response to [1] I tried to convert an existing
directory within a single cephfs into a subvolume. According to [2]
that should be possible, I'm just wondering how to confirm that it
actually worked. Because setting the xattr works fine, the directory
just doesn't show up in the subvolume ls command. This is what I tried
(in Reef and Pacific):
# one "regular" subvolume already exists
$ ceph fs subvolume ls cephfs
[
{
"name": "subvol1"
}
]
# mounted / and created new subdir
$ mkdir /mnt/volumes/subvol2
$ setfattr -n ceph.dir.subvolume -v 1 /mnt/volumes/subvol2
# still only one subvolume
$ ceph fs subvolume ls cephfs
[
{
"name": "subvol1"
}
]
I also tried it directly underneath /mnt:
$ mkdir /mnt/subvol2
$ setfattr -n ceph.dir.subvolume -v 1 /mnt/subvol2
But still no subvolume2 available. What am I missing here?
Thanks
Eugen
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/G4ZWGGUPPFQ…
[2] https://www.spinics.net/lists/ceph-users/msg72341.html
3
7
Hi,
I've seen this issue mentioned in the past, but with older releases. So
I'm wondering if anybody has any pointers.
The Ceph cluster is running Pacific 16.2.13 on Ubuntu 20.04. Almost all
clients are working fine, with the exception of our backup server. This
is using the kernel CephFS client on Ubuntu 22.04 with kernel 6.2.0 [1]
(so I suspect a newer Ceph version?).
The backup server has multiple (12) CephFS mount points. One of them,
the busiest, regularly causes this error on the cluster:
HEALTH_WARN 1 clients failing to respond to capability release
[WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release
mds.mds-server(mds.0): Client backupserver:cephfs-backupserver failing to respond to capability release client_id: 521306112
And occasionally, which may be unrelated, but occurs at the same time:
[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
mds.mds-server(mds.0): 1 slow requests are blocked > 30 secs
The second one clears itself, but the first sticks until I can unmount
the filesystem on the client after the backup completes.
It appears that whilst it's in this stuck state there may be one or more
directory trees that are inaccessible to all clients. The backup server
is walking the whole tree but never gets stuck itself, so either the
inaccessible directory entry is caused after it has gone past, or it's
not affected. Maybe the backup server is holding a directory when it
shouldn't?
It may be that an upgrade to Quincy resolves this, since it's more
likely to be inline with the kernel client version wise, but I don't
want to knee-jerk upgrade just to try and fix this problem.
Thanks for any advice.
Tim.
[1] The reason for the newer kernel is that the backup performance from
CephFS was terrible with older kernels. This newer kernel does at least
resolve that issue.
4
5
Dear cephers,
with one osd down(200GB/9.1TB data), rebalance takes 3 hours still in
progress. Client bandwidth can go as high as 200MB/s. With little client
request throughput, recovery goes at couple MB/s. I wonder if there is
configuration to polish for improvement. It runs with quincy 17.2.5,
deployed by cephadm. The slowness can do harm in peak hours of usage.
Best wishes,
Ben
-------------------------------------------------
volumes: 1/1 healthy
pools: 8 pools, 209 pgs
objects: 93.04M objects, 4.8 TiB
usage: 15 TiB used, 467 TiB / 482 TiB avail
pgs: 1206837/279121971 objects degraded (0.432%)
208 active+clean
1 active+undersized+degraded+remapped+backfilling
io:
client: 80 KiB/s rd, 420 KiB/s wr, 12 op/s rd, 29 op/s wr
recovery: 6.2 MiB/s, 113 objects/s
2
2
Hello All,
Greetings. We've a Ceph Cluster with the version
*ceph version 14.2.16-402-g7d47dbaf4d
(7d47dbaf4d0960a2e910628360ae36def84ed913) nautilus (stable)
===================================
Issues: 1 pg in inconsistent state and does not recover.
# ceph -s
cluster:
id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794
health: HEALTH_ERR
2 large omap objects
1 pools have many more objects per pg than average
159224 scrub errors
Possible data damage: 1 pg inconsistent
2 pgs not deep-scrubbed in time
2 pgs not scrubbed in time
# ceph health detail
HEALTH_ERR 2 large omap objects; 1 pools have many more objects per pg than average; 159224 scrub errors; Possible data damage: 1 pg inconsistent; 2 pgs not deep-scrubbed in time; 2 pgs not scrubbed in time
LARGE_OMAP_OBJECTS 2 large omap objects
2 large objects found in pool 'default.rgw.log'
Search the cluster log for 'Large omap object found' for more details.
MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average
pool iscsi-images objects per pg (541376) is more than 14.9829 times cluster average (36133)
OSD_SCRUB_ERRORS 159224 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
pg 15.f4f is active+clean+inconsistent, acting [238,106,402,266,374,498,590,627,684,73,66]
PG_NOT_DEEP_SCRUBBED 2 pgs not deep-scrubbed in time
pg 1.5c not deep-scrubbed since 2021-04-05 23:20:13.714446
pg 1.55 not deep-scrubbed since 2021-04-11 07:12:37.185074
PG_NOT_SCRUBBED 2 pgs not scrubbed in time
pg 1.5c not scrubbed since 2023-07-10 21:15:50.352848
pg 1.55 not scrubbed since 2023-06-24 10:02:10.038311
======================================
We have implemented below command to resolve it
1. We have ran pg repair command "ceph pg repair 15.f4f
2. We have restarted associated OSDs that is mapped to pg 15.f4f
3. We tuned osd_max_scrubs value and set it to 9.
4. We have done scrub and deep scrub by ceph pg scrub 15.4f4 & ceph pg deep-scrub 15.f4f
5. We also tried to ceph-objectstore-tool command to fix it
==============================================
We have checked the logs of the primary OSD of the respective inconsistent PG and found the below errors.
[ERR] : 15.f4fs0 shard 402(2) 15:f2f3fff4:::94a51ddb-a94f-47bc-9068-509e8c09af9a.7862003.20_c%2f4%2fd61%2f885%2f49627697%2f192_1.ts:head : missing
/var/log/ceph/ceph-osd.238.log:339:2023-10-06 00:37:06.410 7f65024cb700 -1 log_channel(cluster) log [ERR] : 15.f4fs0 shard 266(3) 15:f2f00002:::94a51ddb-a94f-47bc-9068-509e8c09af9a.11432468.3_TN8QHE_04.20.2020_08.41%2fCV_MAGNETIC%2fV_274396%2fCHUNK_2440801%2fSFILE_CONTAINER_031.FOLDER%2f3:head : missing
/var/log/ceph/ceph-osd.238.log:340:2023-10-06 00:37:06.410 7f65024cb700 -1 log_channel(cluster) log [ERR] : 15.f4fs0 shard 402(2) 15:f2f00002:::94a51ddb-a94f-47bc-9068-509e8c09af9a.11432468.3_TN8QHE_04.20.2020_08.41%2fCV_MAGNETIC%2fV_274396%2fCHUNK_2440801%2fSFILE_CONTAINER_031.FOLDER%2f3:head : missing
/var/log/ceph/ceph-osd.238.log:341:2023-10-06 00:37:06.410 7f65024cb700 -1 log_channel(cluster) log [ERR] : 15.f4fs0 shard 590(6) 15:f2f00002:::94a51ddb-a94f-47bc-9068-509e8c09af9a.11432468.3_TN8QHE_04.20.2020_08.41%2fCV_MAGNETIC%2fV_274396%2fCHUNK_2440801%2fSFILE_CONTAINER_031.FOLDER%2f3:head : missing
===============================
and also we noticed that the no. of scrub errors in ceph health status are matching with the ERR log entries in the primary OSD logs of the inconsistent PG as below
grep -Hn 'ERR' /var/log/ceph/ceph-osd.238.log|wc -l
159226
================================
Ceph is cleaning the scrub errors but rate of scrub repair is very slow (avg of 200 scrub errors per day) ,we want to increase the rate of scrub error repair to finish the cleanup of pending 159224 scrub errors.
#ceph pg 15.f4f query
{
"state": "active+clean+inconsistent",
"snap_trimq": "[]",
"snap_trimq_len": 0,
"epoch": 409009,
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting_recovery_backfill": [
"66(10)",
"73(9)",
"106(1)",
"238(0)",
"266(3)",
"374(4)",
"402(2)",
"498(5)",
"590(6)",
"627(7)",
"684(8)"
],
"info": {
"pgid": "15.f4fs0",
"last_update": "409009'7998",
"last_complete": "409009'7998",
"log_tail": "382701'4900",
"last_user_version": 592883,
"last_backfill": "MAX",
"last_backfill_bitwise": 0,
"purged_snaps": [],
"history": {
"epoch_created": 19813,
"epoch_pool_created": 16141,
"last_epoch_started": 407097,
"last_interval_started": 407096,
"last_epoch_clean": 407097,
"last_interval_clean": 407096,
"last_epoch_split": 19849,
"last_epoch_marked_full": 0,
"same_up_since": 407096,
"same_interval_since": 407096,
"same_primary_since": 407038,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935"
},
"stats": {
"version": "409009'7998",
"reported_seq": "379048",
"reported_epoch": "409009",
"state": "active+clean+inconsistent",
"last_fresh": "2023-10-06 15:49:09.174662",
"last_change": "2023-10-06 00:53:53.705308",
"last_active": "2023-10-06 15:49:09.174662",
"last_peered": "2023-10-06 15:49:09.174662",
"last_clean": "2023-10-06 15:49:09.174662",
"last_became_active": "2023-10-03 19:55:56.742034",
"last_became_peered": "2023-10-03 19:55:56.742034",
"last_unstale": "2023-10-06 15:49:09.174662",
"last_undegraded": "2023-10-06 15:49:09.174662",
"last_fullsized": "2023-10-06 15:49:09.174662",
"mapping_epoch": 407096,
"log_start": "382701'4900",
"ondisk_log_start": "382701'4900",
"created": 19813,
"last_epoch_clean": 407097,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935",
"log_size": 3098,
"ondisk_log_size": 3098,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 45852508474,
"num_objects": 40234,
"num_object_clones": 0,
"num_object_copies": 442574,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 40234,
"num_whiteouts": 0,
"num_read": 23940,
"num_read_kb": 2900611,
"num_write": 34950,
"num_write_kb": 2440513,
"num_scrub_errors": 159224,
"num_shallow_scrub_errors": 159224,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 86741,
"num_bytes_recovered": 97127524421,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 42593
},
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 238,
"acting_primary": 238,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 407097,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "66(10)",
"pgid": "15.f4fs10",
"last_update": "409009'7998",
"last_complete": "407068'7791",
"log_tail": "378965'4700",
"last_user_version": 592676,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": [],
"history": {
"epoch_created": 19813,
"epoch_pool_created": 16141,
"last_epoch_started": 407097,
"last_interval_started": 407096,
"last_epoch_clean": 407097,
"last_interval_clean": 407096,
"last_epoch_split": 19849,
"last_epoch_marked_full": 0,
"same_up_since": 407096,
"same_interval_since": 407096,
"same_primary_since": 407038,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935"
},
"stats": {
"version": "407068'7791",
"reported_seq": "376441",
"reported_epoch": "407068",
"state": "active+clean+inconsistent",
"last_fresh": "2023-10-03 19:07:04.483241",
"last_change": "2023-10-03 18:02:27.182058",
"last_active": "2023-10-03 19:07:04.483241",
"last_peered": "2023-10-03 19:07:04.483241",
"last_clean": "2023-10-03 19:07:04.483241",
"last_became_active": "2023-10-03 18:02:27.181798",
"last_became_peered": "2023-10-03 18:02:27.181798",
"last_unstale": "2023-10-03 19:07:04.483241",
"last_undegraded": "2023-10-03 19:07:04.483241",
"last_fullsized": "2023-10-03 19:07:04.483241",
"mapping_epoch": 407096,
"log_start": "378965'4700",
"ondisk_log_start": "378965'4700",
"created": 19813,
"last_epoch_clean": 407068,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "407037'7783",
"last_scrub_stamp": "2023-10-03 17:14:13.232978",
"last_deep_scrub": "406839'7692",
"last_deep_scrub_stamp": "2023-09-28 14:48:47.098935",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935",
"log_size": 3091,
"ondisk_log_size": 3091,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 45930070381,
"num_objects": 40379,
"num_object_clones": 0,
"num_object_copies": 444169,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 40379,
"num_whiteouts": 0,
"num_read": 23277,
"num_read_kb": 2877425,
"num_write": 34413,
"num_write_kb": 2409735,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 86740,
"num_bytes_recovered": 97127524421,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 42593
},
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 238,
"acting_primary": 238,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 407097,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "73(9)",
"pgid": "15.f4fs9",
"last_update": "409009'7998",
"last_complete": "409009'7998",
"log_tail": "378965'4700",
"last_user_version": 592677,
"last_backfill": "MAX",
"last_backfill_bitwise": 0,
"purged_snaps": [],
"history": {
"epoch_created": 19813,
"epoch_pool_created": 16141,
"last_epoch_started": 407097,
"last_interval_started": 407096,
"last_epoch_clean": 407097,
"last_interval_clean": 407096,
"last_epoch_split": 19849,
"last_epoch_marked_full": 0,
"same_up_since": 407096,
"same_interval_since": 407096,
"same_primary_since": 407038,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935"
},
"stats": {
"version": "407095'7792",
"reported_seq": "376677",
"reported_epoch": "407095",
"state": "active+undersized+degraded+inconsistent",
"last_fresh": "2023-10-03 19:54:53.799024",
"last_change": "2023-10-03 19:53:22.457623",
"last_active": "2023-10-03 19:54:53.799024",
"last_peered": "2023-10-03 19:54:53.799024",
"last_clean": "2023-10-03 19:49:47.048960",
"last_became_active": "2023-10-03 19:53:22.457623",
"last_became_peered": "2023-10-03 19:53:22.457623",
"last_unstale": "2023-10-03 19:54:53.799024",
"last_undegraded": "2023-10-03 19:53:22.379335",
"last_fullsized": "2023-10-03 19:53:22.379134",
"mapping_epoch": 407096,
"log_start": "378965'4700",
"ondisk_log_start": "378965'4700",
"created": 19813,
"last_epoch_clean": 407093,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "407037'7783",
"last_scrub_stamp": "2023-10-03 17:14:13.232978",
"last_deep_scrub": "406839'7692",
"last_deep_scrub_stamp": "2023-09-28 14:48:47.098935",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935",
"log_size": 3092,
"ondisk_log_size": 3092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 45929876177,
"num_objects": 40378,
"num_object_clones": 0,
"num_object_copies": 444158,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 40378,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 40378,
"num_whiteouts": 0,
"num_read": 23280,
"num_read_kb": 2877429,
"num_write": 34414,
"num_write_kb": 2409735,
"num_scrub_errors": 159632,
"num_shallow_scrub_errors": 159632,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 86740,
"num_bytes_recovered": 97127524421,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 42593
},
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"avail_no_missing": [
"238(0)",
"73(9)",
"106(1)",
"266(3)",
"374(4)",
"402(2)",
"498(5)",
"590(6)",
"627(7)",
"684(8)"
],
"object_location_counts": [
{
"shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)",
"objects": 40378
}
],
"blocked_by": [],
"up_primary": 238,
"acting_primary": 238,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 407097,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "106(1)",
"pgid": "15.f4fs1",
"last_update": "409009'7998",
"last_complete": "409009'7998",
"log_tail": "378965'4700",
"last_user_version": 592677,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": [],
"history": {
"epoch_created": 19813,
"epoch_pool_created": 16141,
"last_epoch_started": 407097,
"last_interval_started": 407096,
"last_epoch_clean": 407097,
"last_interval_clean": 407096,
"last_epoch_split": 19849,
"last_epoch_marked_full": 0,
"same_up_since": 407096,
"same_interval_since": 407096,
"same_primary_since": 407038,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935"
},
"stats": {
"version": "407095'7792",
"reported_seq": "376677",
"reported_epoch": "407095",
"state": "active+undersized+degraded+inconsistent",
"last_fresh": "2023-10-03 19:54:53.799024",
"last_change": "2023-10-03 19:53:22.457623",
"last_active": "2023-10-03 19:54:53.799024",
"last_peered": "2023-10-03 19:54:53.799024",
"last_clean": "2023-10-03 19:49:47.048960",
"last_became_active": "2023-10-03 19:53:22.457623",
"last_became_peered": "2023-10-03 19:53:22.457623",
"last_unstale": "2023-10-03 19:54:53.799024",
"last_undegraded": "2023-10-03 19:53:22.379335",
"last_fullsized": "2023-10-03 19:53:22.379134",
"mapping_epoch": 407096,
"log_start": "378965'4700",
"ondisk_log_start": "378965'4700",
"created": 19813,
"last_epoch_clean": 407093,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "407037'7783",
"last_scrub_stamp": "2023-10-03 17:14:13.232978",
"last_deep_scrub": "406839'7692",
"last_deep_scrub_stamp": "2023-09-28 14:48:47.098935",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935",
"log_size": 3092,
"ondisk_log_size": 3092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 45929876177,
"num_objects": 40378,
"num_object_clones": 0,
"num_object_copies": 444158,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 40378,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 40378,
"num_whiteouts": 0,
"num_read": 23280,
"num_read_kb": 2877429,
"num_write": 34414,
"num_write_kb": 2409735,
"num_scrub_errors": 159632,
"num_shallow_scrub_errors": 159632,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 86740,
"num_bytes_recovered": 97127524421,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 42593
},
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"avail_no_missing": [
"238(0)",
"73(9)",
"106(1)",
"266(3)",
"374(4)",
"402(2)",
"498(5)",
"590(6)",
"627(7)",
"684(8)"
],
"object_location_counts": [
{
"shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)",
"objects": 40378
}
],
"blocked_by": [],
"up_primary": 238,
"acting_primary": 238,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 407097,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "266(3)",
"pgid": "15.f4fs3",
"last_update": "409009'7998",
"last_complete": "409009'7998",
"log_tail": "378965'4700",
"last_user_version": 592677,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": [],
"history": {
"epoch_created": 19813,
"epoch_pool_created": 16141,
"last_epoch_started": 407097,
"last_interval_started": 407096,
"last_epoch_clean": 407097,
"last_interval_clean": 407096,
"last_epoch_split": 19849,
"last_epoch_marked_full": 0,
"same_up_since": 407096,
"same_interval_since": 407096,
"same_primary_since": 407038,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935"
},
"stats": {
"version": "407095'7792",
"reported_seq": "376677",
"reported_epoch": "407095",
"state": "active+undersized+degraded+inconsistent",
"last_fresh": "2023-10-03 19:54:53.799024",
"last_change": "2023-10-03 19:53:22.457623",
"last_active": "2023-10-03 19:54:53.799024",
"last_peered": "2023-10-03 19:54:53.799024",
"last_clean": "2023-10-03 19:49:47.048960",
"last_became_active": "2023-10-03 19:53:22.457623",
"last_became_peered": "2023-10-03 19:53:22.457623",
"last_unstale": "2023-10-03 19:54:53.799024",
"last_undegraded": "2023-10-03 19:53:22.379335",
"last_fullsized": "2023-10-03 19:53:22.379134",
"mapping_epoch": 407096,
"log_start": "378965'4700",
"ondisk_log_start": "378965'4700",
"created": 19813,
"last_epoch_clean": 407093,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "407037'7783",
"last_scrub_stamp": "2023-10-03 17:14:13.232978",
"last_deep_scrub": "406839'7692",
"last_deep_scrub_stamp": "2023-09-28 14:48:47.098935",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935",
"log_size": 3092,
"ondisk_log_size": 3092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 45929876177,
"num_objects": 40378,
"num_object_clones": 0,
"num_object_copies": 444158,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 40378,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 40378,
"num_whiteouts": 0,
"num_read": 23280,
"num_read_kb": 2877429,
"num_write": 34414,
"num_write_kb": 2409735,
"num_scrub_errors": 159632,
"num_shallow_scrub_errors": 159632,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 86740,
"num_bytes_recovered": 97127524421,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 42593
},
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"avail_no_missing": [
"238(0)",
"73(9)",
"106(1)",
"266(3)",
"374(4)",
"402(2)",
"498(5)",
"590(6)",
"627(7)",
"684(8)"
],
"object_location_counts": [
{
"shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)",
"objects": 40378
}
],
"blocked_by": [],
"up_primary": 238,
"acting_primary": 238,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 407097,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "374(4)",
"pgid": "15.f4fs4",
"last_update": "409009'7998",
"last_complete": "409009'7998",
"log_tail": "378965'4700",
"last_user_version": 592677,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": [],
"history": {
"epoch_created": 19813,
"epoch_pool_created": 16141,
"last_epoch_started": 407097,
"last_interval_started": 407096,
"last_epoch_clean": 407097,
"last_interval_clean": 407096,
"last_epoch_split": 19849,
"last_epoch_marked_full": 0,
"same_up_since": 407096,
"same_interval_since": 407096,
"same_primary_since": 407038,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935"
},
"stats": {
"version": "407095'7792",
"reported_seq": "376677",
"reported_epoch": "407095",
"state": "active+undersized+degraded+inconsistent",
"last_fresh": "2023-10-03 19:54:53.799024",
"last_change": "2023-10-03 19:53:22.457623",
"last_active": "2023-10-03 19:54:53.799024",
"last_peered": "2023-10-03 19:54:53.799024",
"last_clean": "2023-10-03 19:49:47.048960",
"last_became_active": "2023-10-03 19:53:22.457623",
"last_became_peered": "2023-10-03 19:53:22.457623",
"last_unstale": "2023-10-03 19:54:53.799024",
"last_undegraded": "2023-10-03 19:53:22.379335",
"last_fullsized": "2023-10-03 19:53:22.379134",
"mapping_epoch": 407096,
"log_start": "378965'4700",
"ondisk_log_start": "378965'4700",
"created": 19813,
"last_epoch_clean": 407093,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "407037'7783",
"last_scrub_stamp": "2023-10-03 17:14:13.232978",
"last_deep_scrub": "406839'7692",
"last_deep_scrub_stamp": "2023-09-28 14:48:47.098935",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935",
"log_size": 3092,
"ondisk_log_size": 3092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 45929876177,
"num_objects": 40378,
"num_object_clones": 0,
"num_object_copies": 444158,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 40378,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 40378,
"num_whiteouts": 0,
"num_read": 23280,
"num_read_kb": 2877429,
"num_write": 34414,
"num_write_kb": 2409735,
"num_scrub_errors": 159632,
"num_shallow_scrub_errors": 159632,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 86740,
"num_bytes_recovered": 97127524421,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 42593
},
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"avail_no_missing": [
"238(0)",
"73(9)",
"106(1)",
"266(3)",
"374(4)",
"402(2)",
"498(5)",
"590(6)",
"627(7)",
"684(8)"
],
"object_location_counts": [
{
"shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)",
"objects": 40378
}
],
"blocked_by": [],
"up_primary": 238,
"acting_primary": 238,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 407097,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "402(2)",
"pgid": "15.f4fs2",
"last_update": "409009'7998",
"last_complete": "409009'7998",
"log_tail": "378965'4700",
"last_user_version": 592677,
"last_backfill": "MAX",
"last_backfill_bitwise": 0,
"purged_snaps": [],
"history": {
"epoch_created": 19813,
"epoch_pool_created": 16141,
"last_epoch_started": 407097,
"last_interval_started": 407096,
"last_epoch_clean": 407097,
"last_interval_clean": 407096,
"last_epoch_split": 19849,
"last_epoch_marked_full": 0,
"same_up_since": 407096,
"same_interval_since": 407096,
"same_primary_since": 407038,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935"
},
"stats": {
"version": "407095'7792",
"reported_seq": "376677",
"reported_epoch": "407095",
"state": "active+undersized+degraded+inconsistent",
"last_fresh": "2023-10-03 19:54:53.799024",
"last_change": "2023-10-03 19:53:22.457623",
"last_active": "2023-10-03 19:54:53.799024",
"last_peered": "2023-10-03 19:54:53.799024",
"last_clean": "2023-10-03 19:49:47.048960",
"last_became_active": "2023-10-03 19:53:22.457623",
"last_became_peered": "2023-10-03 19:53:22.457623",
"last_unstale": "2023-10-03 19:54:53.799024",
"last_undegraded": "2023-10-03 19:53:22.379335",
"last_fullsized": "2023-10-03 19:53:22.379134",
"mapping_epoch": 407096,
"log_start": "378965'4700",
"ondisk_log_start": "378965'4700",
"created": 19813,
"last_epoch_clean": 407093,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "407037'7783",
"last_scrub_stamp": "2023-10-03 17:14:13.232978",
"last_deep_scrub": "406839'7692",
"last_deep_scrub_stamp": "2023-09-28 14:48:47.098935",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935",
"log_size": 3092,
"ondisk_log_size": 3092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 45929876177,
"num_objects": 40378,
"num_object_clones": 0,
"num_object_copies": 444158,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 40378,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 40378,
"num_whiteouts": 0,
"num_read": 23280,
"num_read_kb": 2877429,
"num_write": 34414,
"num_write_kb": 2409735,
"num_scrub_errors": 159632,
"num_shallow_scrub_errors": 159632,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 86740,
"num_bytes_recovered": 97127524421,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 42593
},
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"avail_no_missing": [
"238(0)",
"73(9)",
"106(1)",
"266(3)",
"374(4)",
"402(2)",
"498(5)",
"590(6)",
"627(7)",
"684(8)"
],
"object_location_counts": [
{
"shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)",
"objects": 40378
}
],
"blocked_by": [],
"up_primary": 238,
"acting_primary": 238,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 407097,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "498(5)",
"pgid": "15.f4fs5",
"last_update": "409009'7998",
"last_complete": "409009'7998",
"log_tail": "378965'4700",
"last_user_version": 592677,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": [],
"history": {
"epoch_created": 19813,
"epoch_pool_created": 16141,
"last_epoch_started": 407097,
"last_interval_started": 407096,
"last_epoch_clean": 407097,
"last_interval_clean": 407096,
"last_epoch_split": 19849,
"last_epoch_marked_full": 0,
"same_up_since": 407096,
"same_interval_since": 407096,
"same_primary_since": 407038,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935"
},
"stats": {
"version": "407095'7792",
"reported_seq": "376677",
"reported_epoch": "407095",
"state": "active+undersized+degraded+inconsistent",
"last_fresh": "2023-10-03 19:54:53.799024",
"last_change": "2023-10-03 19:53:22.457623",
"last_active": "2023-10-03 19:54:53.799024",
"last_peered": "2023-10-03 19:54:53.799024",
"last_clean": "2023-10-03 19:49:47.048960",
"last_became_active": "2023-10-03 19:53:22.457623",
"last_became_peered": "2023-10-03 19:53:22.457623",
"last_unstale": "2023-10-03 19:54:53.799024",
"last_undegraded": "2023-10-03 19:53:22.379335",
"last_fullsized": "2023-10-03 19:53:22.379134",
"mapping_epoch": 407096,
"log_start": "378965'4700",
"ondisk_log_start": "378965'4700",
"created": 19813,
"last_epoch_clean": 407093,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "407037'7783",
"last_scrub_stamp": "2023-10-03 17:14:13.232978",
"last_deep_scrub": "406839'7692",
"last_deep_scrub_stamp": "2023-09-28 14:48:47.098935",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935",
"log_size": 3092,
"ondisk_log_size": 3092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 45929876177,
"num_objects": 40378,
"num_object_clones": 0,
"num_object_copies": 444158,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 40378,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 40378,
"num_whiteouts": 0,
"num_read": 23280,
"num_read_kb": 2877429,
"num_write": 34414,
"num_write_kb": 2409735,
"num_scrub_errors": 159632,
"num_shallow_scrub_errors": 159632,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 86740,
"num_bytes_recovered": 97127524421,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 42593
},
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"avail_no_missing": [
"238(0)",
"73(9)",
"106(1)",
"266(3)",
"374(4)",
"402(2)",
"498(5)",
"590(6)",
"627(7)",
"684(8)"
],
"object_location_counts": [
{
"shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)",
"objects": 40378
}
],
"blocked_by": [],
"up_primary": 238,
"acting_primary": 238,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 407097,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "590(6)",
"pgid": "15.f4fs6",
"last_update": "409009'7998",
"last_complete": "409009'7998",
"log_tail": "378965'4700",
"last_user_version": 592677,
"last_backfill": "MAX",
"last_backfill_bitwise": 0,
"purged_snaps": [],
"history": {
"epoch_created": 19813,
"epoch_pool_created": 16141,
"last_epoch_started": 407097,
"last_interval_started": 407096,
"last_epoch_clean": 407097,
"last_interval_clean": 407096,
"last_epoch_split": 19849,
"last_epoch_marked_full": 0,
"same_up_since": 407096,
"same_interval_since": 407096,
"same_primary_since": 407038,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935"
},
"stats": {
"version": "407095'7792",
"reported_seq": "376677",
"reported_epoch": "407095",
"state": "active+undersized+degraded+inconsistent",
"last_fresh": "2023-10-03 19:54:53.799024",
"last_change": "2023-10-03 19:53:22.457623",
"last_active": "2023-10-03 19:54:53.799024",
"last_peered": "2023-10-03 19:54:53.799024",
"last_clean": "2023-10-03 19:49:47.048960",
"last_became_active": "2023-10-03 19:53:22.457623",
"last_became_peered": "2023-10-03 19:53:22.457623",
"last_unstale": "2023-10-03 19:54:53.799024",
"last_undegraded": "2023-10-03 19:53:22.379335",
"last_fullsized": "2023-10-03 19:53:22.379134",
"mapping_epoch": 407096,
"log_start": "378965'4700",
"ondisk_log_start": "378965'4700",
"created": 19813,
"last_epoch_clean": 407093,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "407037'7783",
"last_scrub_stamp": "2023-10-03 17:14:13.232978",
"last_deep_scrub": "406839'7692",
"last_deep_scrub_stamp": "2023-09-28 14:48:47.098935",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935",
"log_size": 3092,
"ondisk_log_size": 3092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 45929876177,
"num_objects": 40378,
"num_object_clones": 0,
"num_object_copies": 444158,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 40378,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 40378,
"num_whiteouts": 0,
"num_read": 23280,
"num_read_kb": 2877429,
"num_write": 34414,
"num_write_kb": 2409735,
"num_scrub_errors": 159632,
"num_shallow_scrub_errors": 159632,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 86740,
"num_bytes_recovered": 97127524421,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 42593
},
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"avail_no_missing": [
"238(0)",
"73(9)",
"106(1)",
"266(3)",
"374(4)",
"402(2)",
"498(5)",
"590(6)",
"627(7)",
"684(8)"
],
"object_location_counts": [
{
"shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)",
"objects": 40378
}
],
"blocked_by": [],
"up_primary": 238,
"acting_primary": 238,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 407097,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "627(7)",
"pgid": "15.f4fs7",
"last_update": "409009'7998",
"last_complete": "409009'7998",
"log_tail": "378965'4700",
"last_user_version": 592677,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": [],
"history": {
"epoch_created": 19813,
"epoch_pool_created": 16141,
"last_epoch_started": 407097,
"last_interval_started": 407096,
"last_epoch_clean": 407097,
"last_interval_clean": 407096,
"last_epoch_split": 19849,
"last_epoch_marked_full": 0,
"same_up_since": 407096,
"same_interval_since": 407096,
"same_primary_since": 407038,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935"
},
"stats": {
"version": "407095'7792",
"reported_seq": "376677",
"reported_epoch": "407095",
"state": "active+undersized+degraded+inconsistent",
"last_fresh": "2023-10-03 19:54:53.799024",
"last_change": "2023-10-03 19:53:22.457623",
"last_active": "2023-10-03 19:54:53.799024",
"last_peered": "2023-10-03 19:54:53.799024",
"last_clean": "2023-10-03 19:49:47.048960",
"last_became_active": "2023-10-03 19:53:22.457623",
"last_became_peered": "2023-10-03 19:53:22.457623",
"last_unstale": "2023-10-03 19:54:53.799024",
"last_undegraded": "2023-10-03 19:53:22.379335",
"last_fullsized": "2023-10-03 19:53:22.379134",
"mapping_epoch": 407096,
"log_start": "378965'4700",
"ondisk_log_start": "378965'4700",
"created": 19813,
"last_epoch_clean": 407093,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "407037'7783",
"last_scrub_stamp": "2023-10-03 17:14:13.232978",
"last_deep_scrub": "406839'7692",
"last_deep_scrub_stamp": "2023-09-28 14:48:47.098935",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935",
"log_size": 3092,
"ondisk_log_size": 3092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 45929876177,
"num_objects": 40378,
"num_object_clones": 0,
"num_object_copies": 444158,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 40378,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 40378,
"num_whiteouts": 0,
"num_read": 23280,
"num_read_kb": 2877429,
"num_write": 34414,
"num_write_kb": 2409735,
"num_scrub_errors": 159632,
"num_shallow_scrub_errors": 159632,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 86740,
"num_bytes_recovered": 97127524421,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 42593
},
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"avail_no_missing": [
"238(0)",
"73(9)",
"106(1)",
"266(3)",
"374(4)",
"402(2)",
"498(5)",
"590(6)",
"627(7)",
"684(8)"
],
"object_location_counts": [
{
"shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)",
"objects": 40378
}
],
"blocked_by": [],
"up_primary": 238,
"acting_primary": 238,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 407097,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "684(8)",
"pgid": "15.f4fs8",
"last_update": "409009'7998",
"last_complete": "409009'7998",
"log_tail": "378965'4700",
"last_user_version": 592677,
"last_backfill": "MAX",
"last_backfill_bitwise": 0,
"purged_snaps": [],
"history": {
"epoch_created": 19813,
"epoch_pool_created": 16141,
"last_epoch_started": 407097,
"last_interval_started": 407096,
"last_epoch_clean": 407097,
"last_interval_clean": 407096,
"last_epoch_split": 19849,
"last_epoch_marked_full": 0,
"same_up_since": 407096,
"same_interval_since": 407096,
"same_primary_since": 407038,
"last_scrub": "408987'7946",
"last_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_deep_scrub": "408987'7946",
"last_deep_scrub_stamp": "2023-10-06 00:53:53.705241",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935"
},
"stats": {
"version": "407095'7792",
"reported_seq": "376677",
"reported_epoch": "407095",
"state": "active+undersized+degraded+inconsistent",
"last_fresh": "2023-10-03 19:54:53.799024",
"last_change": "2023-10-03 19:53:22.457623",
"last_active": "2023-10-03 19:54:53.799024",
"last_peered": "2023-10-03 19:54:53.799024",
"last_clean": "2023-10-03 19:49:47.048960",
"last_became_active": "2023-10-03 19:53:22.457623",
"last_became_peered": "2023-10-03 19:53:22.457623",
"last_unstale": "2023-10-03 19:54:53.799024",
"last_undegraded": "2023-10-03 19:53:22.379335",
"last_fullsized": "2023-10-03 19:53:22.379134",
"mapping_epoch": 407096,
"log_start": "378965'4700",
"ondisk_log_start": "378965'4700",
"created": 19813,
"last_epoch_clean": 407093,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "407037'7783",
"last_scrub_stamp": "2023-10-03 17:14:13.232978",
"last_deep_scrub": "406839'7692",
"last_deep_scrub_stamp": "2023-09-28 14:48:47.098935",
"last_clean_scrub_stamp": "2023-09-28 14:48:47.098935",
"log_size": 3092,
"ondisk_log_size": 3092,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 45929876177,
"num_objects": 40378,
"num_object_clones": 0,
"num_object_copies": 444158,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 40378,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 40378,
"num_whiteouts": 0,
"num_read": 23280,
"num_read_kb": 2877429,
"num_write": 34414,
"num_write_kb": 2409735,
"num_scrub_errors": 159632,
"num_shallow_scrub_errors": 159632,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 86740,
"num_bytes_recovered": 97127524421,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 42593
},
"up": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"acting": [
238,
106,
402,
266,
374,
498,
590,
627,
684,
73,
66
],
"avail_no_missing": [
"238(0)",
"73(9)",
"106(1)",
"266(3)",
"374(4)",
"402(2)",
"498(5)",
"590(6)",
"627(7)",
"684(8)"
],
"object_location_counts": [
{
"shards": "73(9),106(1),238(0),266(3),374(4),402(2),498(5),590(6),627(7),684(8)",
"objects": 40378
}
],
"blocked_by": [],
"up_primary": 238,
"acting_primary": 238,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 407097,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2023-10-03 19:55:56.164849",
"might_have_unfound": [
{
"osd": "66(10)",
"status": "already probed"
},
{
"osd": "73(9)",
"status": "already probed"
},
{
"osd": "106(1)",
"status": "already probed"
},
{
"osd": "266(3)",
"status": "already probed"
},
{
"osd": "374(4)",
"status": "already probed"
},
{
"osd": "402(2)",
"status": "already probed"
},
{
"osd": "498(5)",
"status": "already probed"
},
{
"osd": "590(6)",
"status": "already probed"
},
{
"osd": "627(7)",
"status": "already probed"
},
{
"osd": "684(8)",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"recovery_ops": [],
"read_ops": []
}
},
"scrub": {
"scrubber.epoch_start": "407096",
"scrubber.active": false,
"scrubber.state": "INACTIVE",
"scrubber.start": "MIN",
"scrubber.end": "MIN",
"scrubber.max_end": "MIN",
"scrubber.subset_last_update": "0'0",
"scrubber.deep": false,
"scrubber.waiting_on_whom": []
}
},
{
"name": "Started",
"enter_time": "2023-10-03 19:55:55.247621"
}
],
"agent_state": {}
}
node1:~ #
3
5