Hello everyone, short update. We added 3 SSD OSDs to the cluster to have some more space and more IO. After all objects were in their new place, it took ~1 week that the OSDs reduced their size. The size of the metadata pool did not change in this time but the OSDs removed multiple TB of occupied space. Now everything looks fine. Let's hope this will stay like this. Best regards, Lars [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel@ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai On Wed, Jul 3, 2024 at 2:06 PM Lars Köppel <lars.koeppel@ariadne.ai> wrote:
Hello everyone,
we reduced the load slightly for the last 2 weeks and the fast growth stopped. But after a short burst of heavy load 1 OSD had filled up to the max again. So I had to zapp it. At the moment it is rebuilding the drive.
After ejecting all clients and reducing the load, the growth is minimal. We ordered some parts to add some more SSDs to this pool. I will update when they have arrived and the balancing is done.
Another thing I have observed is that the limits for the scrubs are not respected. We only have 75 OSDs and set osd_max_scrubs to 1 but there are often more than 100 scrubs ongoing. Has there been anything changed since 17.2.6?
Best regards, Lars
[image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel@ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai
On Thu, Jun 13, 2024 at 3:24 PM Lars Köppel <lars.koeppel@ariadne.ai> wrote:
ok. Thank you for your help. We will try this and report back in a few days.
[image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel@ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai
On Thu, Jun 13, 2024 at 3:01 PM Eugen Block <eblock@nde.ag> wrote:
I'm quite sure that this could result in the impact you're seeing. To confirm that suspicion you could stop deleting and wait a couple of days to see if the usage stabilizes. And if it does, maybe delete less files at once or so to see how far you can tweak it. That would be my approach.
Zitat von Lars Köppel <lars.koeppel@ariadne.ai>:
We have been using snapshots for a long time. The only change in usage is that we are currently deleting many small files from the system. Because this is slow (~150 requests/s) this is running for the last few weeks. Could such a load result in a problem with the MDS?
I have to ask for permission to order more drives. This could take some time.
[image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel@ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai
On Thu, Jun 13, 2024 at 12:55 PM Eugen Block <eblock@nde.ag> wrote:
Downgrading isn't supported, I don't think that would be a good idea. I also don't see anything obvious standing out in the pg output. Any chance you can add more OSDs to the metadata pool to see if it stops at some point? Did the cluster usage change in any way? For example cephfs snapshots which haven't been used before or something like that?
Zitat von Lars Köppel <lars.koeppel@ariadne.ai>:
I updated from 17.2.6 to 17.2.7 and a few hours later to 18.2.2. Would it be an option to go back to 17.2.6?
[image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel@ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai
On Wed, Jun 12, 2024 at 5:30 PM Eugen Block <eblock@nde.ag> wrote:
> Which version did you upgrade from to 18.2.2? > I can’t pin it down to a specific issue, but somewhere in the back of > my mind is something related to a new omap format or something. But > I’m really not sure at all. > > Zitat von Lars Köppel <lars.koeppel@ariadne.ai>: > > > I am happy to help you with as much information as possible. I probably > > just don't know where to look for it. > > Below are the requested information. The cluster is rebuilding the > > zapped OSD at the moment. This will probably take the next few days. > > > > > > sudo ceph pg ls-by-pool metadata > > PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* > > OMAP_KEYS* LOG LOG_DUPS STATE > > SINCE VERSION REPORTED UP ACTING > > SCRUB_STAMP DEEP_SCRUB_STAMP > > LAST_SCRUB_DURATION SCRUB_SCHEDULING > > 10.0 5217325 4994695 0 0 4194304 5880891340 > > 9393865 1885 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'180849582 79875:391519635 [0,1,2]p0 [1,2]p1 > > 2024-06-11T09:08:09.829362+0000 2024-05-28T05:52:59.321589+0000 > > 627 periodic scrub scheduled @ 2024-06-17T08:21:31.808348+0000 > > 10.1 5214785 5193424 0 0 0 5843682713 > > 9410150 1912 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180914288 79875:342746928 [2,1,0]p2 [2,1]p2 > > 2024-06-01T15:56:28.927288+0000 2024-05-27T03:31:37.682966+0000 > > 966 queued for scrub > > 10.2 5218432 5187168 0 0 0 6402011266 > > 9812513 1874 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180970531 79875:341340204 [0,1,2]p0 [1,2]p1 > > 2024-06-11T13:40:58.994256+0000 2024-06-11T13:40:58.994256+0000 > > 1942 periodic scrub scheduled @ 2024-06-17T06:07:15.329675+0000 > > 10.3 5217413 5217413 0 0 8388788 5766005023 > > 9271787 1923 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181012233 79875:388295881 [1,0,2]p1 [1,2]p1 > > 2024-06-12T00:35:56.965547+0000 2024-05-23T19:54:56.121729+0000 > > 492 periodic scrub scheduled @ 2024-06-18T06:39:31.103864+0000 > > 10.4 5220069 5220069 0 0 12583466 6027548724 > > 9537290 1959 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181576075 79875:405295868 [1,2,0]p1 [1,2]p1 > > 2024-06-11T17:47:22.923514+0000 2024-05-31T02:06:55.339574+0000 > > 581 periodic scrub scheduled @ 2024-06-17T00:59:37.214420+0000 > > 10.5 5216162 5211999 0 0 4194304 5941347251 > > 9542764 1930 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180455793 79875:338418517 [2,1,0]p2 [2,1]p2 > > 2024-06-11T22:50:16.170708+0000 2024-05-30T23:49:54.316379+0000 > > 528 periodic scrub scheduled @ 2024-06-17T04:39:25.905185+0000 > > 10.6 5216100 4980459 0 0 4521984 6428088514 > > 9850762 1911 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'184045876 79875:396809795 [0,2,1]p0 [1,2]p1 > > 2024-06-11T22:24:05.102716+0000 2024-06-11T22:24:05.102716+0000 > > 1082 periodic scrub scheduled @ 2024-06-17T07:58:44.289885+0000 > > 10.7 5218232 5218232 0 0 4194304 6377065363 > > 9849360 1919 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182672562 79875:342449062 [1,0,2]p1 [1,2]p1 > > 2024-06-11T06:22:15.689422+0000 2024-06-11T06:22:15.689422+0000 > > 8225 periodic scrub scheduled @ 2024-06-17T13:05:59.225052+0000 > > 10.8 5219620 5182816 0 0 0 6167304290 > > 9691796 1896 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179628377 79875:378022884 [2,1,0]p2 [2,1]p2 > > 2024-06-11T22:06:01.386763+0000 2024-06-11T22:06:01.386763+0000 > > 1286 periodic scrub scheduled @ 2024-06-17T07:54:54.133093+0000 > > 10.9 5219448 5164591 0 0 8388698 5796048346 > > 9338312 1868 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181739392 79875:387412389 [2,1,0]p2 [2,1]p2 > > 2024-06-12T05:21:00.586747+0000 2024-05-26T11:10:59.780673+0000 > > 539 periodic scrub scheduled @ 2024-06-18T15:32:59.155092+0000 > > 10.a 5219861 5163635 0 0 12582912 5841839055 > > 9387200 1916 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180205688 79875:379381294 [1,2,0]p1 [1,2]p1 > > 2024-06-11T12:35:05.571200+0000 2024-05-22T11:07:16.041773+0000 > > 1093 periodic deep scrub scheduled @ > 2024-06-17T05:21:40.136463+0000 > > 10.b 5217949 5217949 0 0 16777216 5935863260 > > 9462127 1881 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181655745 79875:343806807 [0,1,2]p0 [1,2]p1 > > 2024-06-11T22:41:28.976920+0000 2024-05-26T08:43:29.217457+0000 > > 520 periodic scrub scheduled @ 2024-06-17T17:44:32.764093+0000 > > 10.c 5221697 5217118 0 0 4194304 6015217841 > > 9574445 1928 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180892826 79875:341490398 [2,1,0]p2 [2,1]p2 > > 2024-06-11T09:20:58.443473+0000 2024-05-30T00:13:50.306507+0000 > > 768 periodic scrub scheduled @ 2024-06-16T19:41:21.977436+0000 > > 10.d 5217727 4908764 0 0 0 5825598519 > > 9349877 1930 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'179050347 79875:387455993 [0,1,2]p0 [1,2]p1 > > 2024-06-02T07:27:27.873631+0000 2024-05-20T03:58:30.225170+0000 > > 952 queued for deep scrub > > 10.e 5215040 5210572 0 0 4194304 5790634469 > > 9327651 1905 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180049044 79875:377196119 [2,1,0]p2 [2,1]p2 > > 2024-06-12T05:03:08.521423+0000 2024-05-22T13:47:46.753131+0000 > > 527 periodic scrub scheduled @ 2024-06-17T17:30:36.079456+0000 > > 10.f 5217202 4885952 0 0 8388630 6005274626 > > 9589178 1917 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'181953069 79875:396402453 [0,2,1]p0 [1,2]p1 > > 2024-06-11T12:20:49.661741+0000 2024-06-11T12:20:49.661741+0000 > > 3949 periodic scrub scheduled @ 2024-06-17T18:37:10.974112+0000 > > 10.10 5214985 5157391 0 0 4194394 6275024632 > > 9721505 1922 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181801708 79875:395055895 [2,0,1]p2 [2,1]p2 > > 2024-06-11T01:49:47.831191+0000 2024-06-11T01:49:47.831191+0000 > > 9579 periodic scrub scheduled @ 2024-06-16T11:40:28.305905+0000 > > 10.11 5219236 4983134 0 0 4194304 5787580073 > > 9316893 1901 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'181011816 79875:343488767 [1,0,2]p1 [1,2]p1 > > 2024-06-03T00:06:14.862209+0000 2024-05-27T00:05:56.468361+0000 > > 960 queued for scrub > > 10.12 5214103 5182770 0 0 90 5970750188 > > 9505613 1891 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180210613 79875:341042616 [0,1,2]p0 [1,2]p1 > > 2024-06-12T06:39:48.252419+0000 2024-05-28T01:32:03.381577+0000 > > 527 periodic scrub scheduled @ 2024-06-18T11:23:33.994438+0000 > > 10.13 5214580 5214580 0 0 8388642 5793432968 > > 9335940 1957 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182394614 79875:393423214 [1,2,0]p1 [1,2]p1 > > 2024-06-10T17:52:04.690203+0000 2024-05-22T03:39:02.453336+0000 > > 1151 periodic deep scrub scheduled @ > 2024-06-16T00:28:37.682765+0000 > > 10.14 5218591 5218591 0 0 5053325 6046268958 > > 9582481 1915 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'184576050 79875:425213800 [2,0,1]p2 [2,1]p2 > > 2024-06-03T05:36:07.656965+0000 2024-05-28T22:46:16.814318+0000 > > 985 queued for scrub > > 10.15 5215919 4907184 0 0 11939335 5752284246 > > 9285870 1889 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'179713036 79875:337938973 [0,2,1]p0 [1,2]p1 > > 2024-06-11T14:06:50.138992+0000 2024-05-24T07:52:22.522455+0000 > > 786 periodic scrub scheduled @ 2024-06-17T13:00:48.869790+0000 > > 10.16 5221091 5221091 0 0 8388608 5966308565 > > 9487816 1916 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'183445984 79875:399413009 [0,2,1]p0 [1,2]p1 > > 2024-06-03T02:01:53.649417+0000 2024-06-03T02:01:53.649417+0000 > > 6935 queued for scrub > > 10.17 5218069 5194732 0 0 4194304 6231638001 > > 9735915 1890 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182390128 79875:340719915 [2,0,1]p2 [2,1]p2 > > 2024-06-12T06:31:01.523916+0000 2024-06-12T06:31:01.523916+0000 > > 1272 periodic scrub scheduled @ 2024-06-17T16:33:45.723673+0000 > > 10.18 5214253 5159897 0 0 0 6128506469 > > 9673261 1886 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179754208 79875:389150273 [2,1,0]p2 [2,1]p2 > > 2024-06-12T02:53:31.027154+0000 2024-06-12T02:53:31.027154+0000 > > 1308 periodic scrub scheduled @ 2024-06-17T22:19:41.359070+0000 > > 10.19 5220131 5183342 0 0 0 5869775360 > > 9460583 1953 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181750923 79875:393162989 [2,0,1]p2 [2,1]p2 > > 2024-06-11T19:21:24.106504+0000 2024-05-22T15:42:01.721091+0000 > > 522 periodic scrub scheduled @ 2024-06-17T20:09:55.173174+0000 > > 10.1a 5217968 5186700 0 0 4194304 6183652625 > > 9733847 1896 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181056240 79875:394241128 [0,2,1]p0 [1,2]p1 > > 2024-06-02T15:37:49.730965+0000 2024-06-02T15:37:49.730965+0000 > > 7347 queued for scrub > > 10.1b 5221259 5161879 0 0 0 5989030303 > > 9561035 1875 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181052305 79875:343144823 [0,1,2]p0 [1,2]p1 > > 2024-06-12T06:01:07.213894+0000 2024-06-02T00:23:51.063901+0000 > > 543 periodic scrub scheduled @ 2024-06-17T15:51:21.700275+0000 > > 10.1c 5217034 5217034 0 0 554 5979861913 > > 9504442 1875 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180913313 79875:341328275 [1,2,0]p1 [1,2]p1 > > 2024-06-01T20:02:47.025129+0000 2024-06-01T20:02:47.025129+0000 > > 6568 queued for scrub > > 10.1d 5219691 4997027 0 0 12582912 6081321658 > > 9599164 1918 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'179638837 79875:391808225 [0,2,1]p0 [1,2]p1 > > 2024-06-11T20:27:02.506715+0000 2024-05-31T22:58:06.578167+0000 > > 517 periodic scrub scheduled @ 2024-06-16T21:05:59.442200+0000 > > 10.1e 5213369 5208776 0 0 4194304 6295629499 > > 9765317 1941 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181145210 79875:391326487 [2,1,0]p2 [2,1]p2 > > 2024-06-12T07:49:28.432084+0000 2024-06-12T07:49:28.432084+0000 > > 1990 periodic scrub scheduled @ 2024-06-18T01:04:07.677468+0000 > > 10.1f 5218399 5000094 0 0 8388608 5821063844 > > 9421222 1896 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'180960296 79875:400635456 [0,2,1]p0 [1,2]p1 > > 2024-06-11T13:53:40.278050+0000 2024-05-22T19:39:02.345663+0000 > > 761 periodic scrub scheduled @ 2024-06-16T22:24:39.094932+0000 > > 10.20 5216437 5185379 0 0 0 6280994122 > > 9719574 1945 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181610117 79875:391874431 [0,1,2]p0 [1,2]p1 > > 2024-06-11T12:58:57.754648+0000 2024-06-11T12:58:57.754648+0000 > > 2288 periodic scrub scheduled @ 2024-06-17T23:33:35.937406+0000 > > 10.21 5217349 5217349 0 0 8388698 6348931429 > > 9822197 1867 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181724357 79875:343883762 [0,1,2]p0 [1,2]p1 > > 2024-06-11T18:28:34.044546+0000 2024-06-11T18:28:34.044546+0000 > > 1817 periodic scrub scheduled @ 2024-06-17T12:18:47.312213+0000 > > 10.22 5215544 4943475 0 0 4194304 5982815232 > > 9482159 1899 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'180230331 79875:340839853 [2,0,1]p2 [2,1]p2 > > 2024-06-11T15:21:21.994674+0000 2024-05-30T08:01:03.301650+0000 > > 649 periodic scrub scheduled @ 2024-06-17T01:33:20.228420+0000 > > 10.23 5217758 5217758 0 0 146 5804086623 > > 9336298 1920 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180948432 79875:388158699 [1,2,0]p1 [1,2]p1 > > 2024-06-03T05:19:32.311846+0000 2024-05-27T08:13:15.250058+0000 > > 971 queued for scrub > > 10.24 5221505 5221505 0 0 0 6360995644 > > 9816333 1925 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182198758 79875:406847098 [0,2,1]p0 [1,2]p1 > > 2024-06-12T04:00:44.315682+0000 2024-06-12T04:00:44.315682+0000 > > 2046 periodic scrub scheduled @ 2024-06-18T01:00:22.708069+0000 > > 10.25 5214911 5214911 0 0 12583002 5726320657 > > 9264352 1910 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180132024 79875:338167995 [1,2,0]p1 [1,2]p1 > > 2024-06-11T14:43:47.076906+0000 2024-05-23T12:16:34.271856+0000 > > 715 periodic scrub scheduled @ 2024-06-17T14:42:04.951046+0000 > > 10.26 5217588 4973667 0 0 0 6040775867 > > 9598406 1927 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'188144536 79875:400277134 [2,0,1]p2 [2,1]p2 > > 2024-06-02T06:55:09.016528+0000 2024-06-02T06:55:09.016528+0000 > > 7047 queued for scrub > > 10.27 5218490 5218490 0 0 8388630 6265942186 > > 9818746 1941 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182400328 79875:342052414 [1,2,0]p1 [1,2]p1 > > 2024-06-12T01:01:06.674323+0000 2024-06-12T01:01:06.674323+0000 > > 1500 periodic scrub scheduled @ 2024-06-18T09:04:18.289464+0000 > > 10.28 5221153 5221153 0 0 0 6049095371 > > 9597182 1913 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179438777 79875:378265375 [1,2,0]p1 [1,2]p1 > > 2024-06-12T04:09:58.896192+0000 2024-06-01T04:32:46.011974+0000 > > 553 periodic scrub scheduled @ 2024-06-18T00:53:21.846375+0000 > > 10.29 5214066 5214066 0 0 4194304 5767555623 > > 9316899 1937 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181568997 79875:387683159 [0,2,1]p0 [1,2]p1 > > 2024-06-10T22:48:57.757493+0000 2024-05-21T07:14:53.348341+0000 > > 938 periodic deep scrub scheduled @ > 2024-06-16T06:24:55.108695+0000 > > 10.2a 5216314 4977811 0 0 4194304 6012567810 > > 9563804 1951 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'181140734 79875:381017643 [2,1,0]p2 [2,1]p2 > > 2024-06-11T09:57:29.097708+0000 2024-05-30T03:47:22.798044+0000 > > 1169 periodic scrub scheduled @ 2024-06-16T10:08:48.053531+0000 > > 10.2b 5219620 5219620 0 0 8388608 5829917912 > > 9412426 1889 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180465782 79875:342263856 [1,0,2]p1 [1,2]p1 > > 2024-06-12T08:30:08.941782+0000 2024-05-24T22:00:29.535205+0000 > > 636 periodic scrub scheduled @ 2024-06-17T09:00:09.155675+0000 > > 10.2c 5214087 5214087 0 0 0 5874182922 > > 9375708 1871 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180616041 79875:341244736 [1,2,0]p1 [1,2]p1 > > 2024-06-10T23:10:08.184410+0000 2024-05-27T17:41:07.583255+0000 > > 1268 periodic scrub scheduled @ 2024-06-16T00:28:52.710385+0000 > > 10.2d 5213968 4984822 0 0 4337824 5835754026 > > 9417033 1947 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'178824607 79875:386427308 [2,0,1]p2 [2,1]p2 > > 2024-06-12T09:27:43.774331+0000 2024-05-27T05:33:45.293266+0000 > > 494 periodic scrub scheduled @ 2024-06-18T10:19:18.219366+0000 > > 10.2e 5216263 4974456 0 0 12582912 6025714323 > > 9553880 1940 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'180639631 79875:378284872 [1,0,2]p1 [1,2]p1 > > 2024-06-12T01:19:42.477063+0000 2024-05-31T08:42:41.293335+0000 > > 526 periodic scrub scheduled @ 2024-06-17T09:53:24.981953+0000 > > 10.2f 5220057 5220057 0 0 13839831 5804027574 > > 9370862 1882 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180961915 79875:394513378 [1,0,2]p1 [1,2]p1 > > 2024-06-11T19:58:05.145147+0000 2024-05-24T20:18:32.301831+0000 > > 535 periodic scrub scheduled @ 2024-06-18T03:16:19.773648+0000 > > 10.30 5217090 4955469 0 0 12582934 6438521431 > > 9924825 1869 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'181488488 79875:404940886 [2,1,0]p2 [2,1]p2 > > 2024-06-12T07:16:18.095439+0000 2024-06-12T07:16:18.095439+0000 > > 2190 periodic scrub scheduled @ 2024-06-18T10:42:55.574570+0000 > > 10.31 5218176 5218176 0 0 4194304 6215016361 > > 9752698 1897 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181622879 79875:343554151 [1,2,0]p1 [1,2]p1 > > 2024-06-12T09:00:15.941065+0000 2024-06-12T09:00:15.941065+0000 > > 1806 periodic scrub scheduled @ 2024-06-18T02:46:00.756777+0000 > > 10.32 5223171 4704732 0 0 8388608 6205669256 > > 9800320 1920 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180911263 79875:342754128 [0,2,1]p0 [1,2]p1 > > 2024-06-11T19:49:09.010583+0000 2024-06-11T19:49:09.010583+0000 > > 1664 periodic scrub scheduled @ 2024-06-17T07:07:21.062789+0000 > > 10.33 5215341 4525338 0 0 4194304 6066965588 > > 9635403 1888 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181467497 79875:402958597 [1,2,0]p1 [1,2]p1 > > 2024-06-12T01:59:23.933458+0000 2024-05-31T17:01:38.620377+0000 > > 512 periodic scrub scheduled @ 2024-06-17T22:07:45.026120+0000 > > 10.34 5217342 5217342 0 0 8388608 5999920961 > > 9570527 1956 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182013407 79875:419219483 [1,0,2]p1 [1,2]p1 > > 2024-06-12T09:10:17.390128+0000 2024-06-01T15:40:23.241769+0000 > > 601 periodic scrub scheduled @ 2024-06-18T09:56:03.061859+0000 > > 10.35 5222114 5222114 0 0 8388608 6021420075 > > 9571300 1952 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180948342 79875:338598678 [1,2,0]p1 [1,2]p1 > > 2024-06-11T22:59:17.017317+0000 2024-05-30T15:59:15.553266+0000 > > 539 periodic scrub scheduled @ 2024-06-17T02:41:31.718321+0000 > > 10.36 5223657 5223657 0 0 4331271 5844578100 > > 9347405 1903 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'183363403 79875:408484683 [0,2,1]p0 [1,2]p1 > > 2024-06-11T18:37:46.653097+0000 2024-05-25T14:13:51.648712+0000 > > 553 periodic scrub scheduled @ 2024-06-17T12:58:00.134657+0000 > > 10.37 5219248 5219248 0 0 4194304 6072826085 > > 9579049 1936 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182091890 79875:341545087 [1,0,2]p1 [1,2]p1 > > 2024-06-11T14:19:40.744743+0000 2024-05-29T18:40:12.602405+0000 > > 771 periodic scrub scheduled @ 2024-06-16T18:24:26.104867+0000 > > 10.38 5214022 5214022 0 0 4194858 5780805541 > > 9299038 1951 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'178809936 79875:390113817 [1,0,2]p1 [1,2]p1 > > 2024-06-02T00:55:22.046433+0000 2024-05-25T16:49:10.279003+0000 > > 931 queued for scrub > > 10.39 5220692 4542094 0 0 0 6401799077 > > 9864164 1872 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'182881827 79875:402437400 [2,0,1]p2 [2,1]p2 > > 2024-06-12T01:42:35.239155+0000 2024-06-12T01:42:35.239155+0000 > > 1370 periodic scrub scheduled @ 2024-06-17T10:51:19.580502+0000 > > 10.3a 5215648 4552651 0 0 0 6194465524 > > 9713558 1898 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181458295 79875:394916652 [2,1,0]p2 [2,1]p2 > > 2024-06-11T21:19:29.020889+0000 2024-06-11T21:19:29.020889+0000 > > 1227 periodic scrub scheduled @ 2024-06-17T07:19:46.503819+0000 > > 10.3b 5218220 5218220 0 0 8388608 5951035110 > > 9490698 1878 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180815984 79875:342438023 [2,1,0]p2 [2,1]p2 > > 2024-06-03T05:03:19.483049+0000 2024-05-28T18:50:55.369009+0000 > > 1004 queued for scrub > > 10.3c 5221615 5221615 0 0 12582912 6260894058 > > 9727358 1938 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181561627 79875:342153262 [1,2,0]p1 [1,2]p1 > > 2024-06-11T19:12:06.206008+0000 2024-06-11T19:12:06.206008+0000 > > 1471 periodic scrub scheduled @ 2024-06-17T02:09:19.786893+0000 > > 10.3d 5213771 5213771 0 0 0 6215975311 > > 9699615 1946 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179354493 79875:399858588 [1,2,0]p1 [1,2]p1 > > 2024-06-11T04:05:10.877703+0000 2024-06-11T04:05:10.877703+0000 > > 8123 periodic scrub scheduled @ 2024-06-17T08:03:22.770035+0000 > > 10.3e 5219446 4659577 0 0 4194304 5852588566 > > 9370152 1933 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180035882 79875:391595513 [0,1,2]p0 [1,2]p1 > > 2024-06-11T20:35:43.624903+0000 2024-05-22T21:40:55.606650+0000 > > 520 periodic scrub scheduled @ 2024-06-17T00:54:06.347642+0000 > > 10.3f 5219729 4889443 0 0 16777216 5925457842 > > 9484458 1948 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'181330738 79875:409113809 [0,2,1]p0 [1,2]p1 > > 2024-06-12T05:12:01.852109+0000 2024-05-27T20:03:22.704122+0000 > > 533 periodic scrub scheduled @ 2024-06-17T13:07:19.370647+0000 > > 10.40 5222241 4445164 0 0 0 6312545191 > > 9773697 1873 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181341418 79875:393186178 [2,1,0]p2 [2,1]p2 > > 2024-06-10T20:00:37.561045+0000 2024-06-10T20:00:37.561045+0000 > > 7712 periodic scrub scheduled @ 2024-06-16T05:59:45.317316+0000 > > 10.41 5215195 5183508 0 0 4194326 6000770355 > > 9529537 1938 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181494334 79875:343712275 [0,2,1]p0 [1,2]p1 > > 2024-06-12T04:18:58.693597+0000 2024-06-01T11:05:59.370690+0000 > > 539 periodic scrub scheduled @ 2024-06-17T15:34:34.997281+0000 > > 10.42 5219564 4681323 0 0 12582912 5822745109 > > 9331625 1916 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179811520 79875:340958258 [1,2,0]p1 [1,2]p1 > > 2024-06-02T19:58:06.439770+0000 2024-05-21T18:28:28.794738+0000 > > 11044 queued for deep scrub > > 10.43 5219412 4599458 0 0 594 5737440613 > > 9239869 1950 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180634536 79875:388879991 [1,0,2]p1 [1,2]p1 > > 2024-06-01T18:13:17.068416+0000 2024-05-20T16:32:20.302230+0000 > > 1002 queued for deep scrub > > 10.44 5216008 5216008 0 0 4194304 6270195298 > > 9786806 1944 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182088681 79875:406175480 [1,0,2]p1 [1,2]p1 > > 2024-06-12T00:19:01.240405+0000 2024-06-12T00:19:01.240405+0000 > > 1593 periodic scrub scheduled @ 2024-06-17T12:19:53.888923+0000 > > 10.45 5216238 4694313 0 0 0 5714807184 > > 9223666 1944 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179709878 79875:338307727 [1,0,2]p1 [1,2]p1 > > 2024-06-02T00:39:50.657116+0000 2024-05-21T02:52:14.376959+0000 > > 958 queued for deep scrub > > 10.46 5217950 5217950 0 0 9729074 5945415128 > > 9499020 1915 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'183726716 79875:395024265 [1,2,0]p1 [1,2]p1 > > 2024-06-11T14:56:28.875949+0000 2024-05-29T04:19:53.409918+0000 > > 724 periodic scrub scheduled @ 2024-06-16T17:42:09.428674+0000 > > 10.47 5219125 5219125 0 0 4194304 5775128028 > > 9299099 1864 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181328922 79875:340510799 [1,0,2]p1 [1,2]p1 > > 2024-06-11T17:58:16.753327+0000 2024-05-23T14:24:25.876796+0000 > > 652 periodic scrub scheduled @ 2024-06-17T20:23:31.713805+0000 > > 10.48 5216471 5185269 0 0 4194326 6352053972 > > 9843580 1929 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179547710 79875:378125137 [0,1,2]p0 [1,2]p1 > > 2024-06-11T23:22:17.435614+0000 2024-06-11T23:22:17.435614+0000 > > 1379 periodic scrub scheduled @ 2024-06-17T19:10:56.456654+0000 > > 10.49 5217119 4580870 0 0 4194304 5971558944 > > 9526047 1879 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'182489297 79875:389298429 [2,0,1]p2 [2,1]p2 > > 2024-06-11T08:57:42.791265+0000 2024-05-29T07:13:57.867295+0000 > > 959 periodic scrub scheduled @ 2024-06-17T06:44:34.852894+0000 > > 10.4a 5217577 4909174 0 0 4194304 5960086599 > > 9511657 1910 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'180982438 79875:381432369 [0,1,2]p0 [1,2]p1 > > 2024-06-12T02:31:42.576663+0000 2024-06-01T09:08:07.860775+0000 > > 480 periodic scrub scheduled @ 2024-06-18T01:57:37.976185+0000 > > 10.4b 5220270 5220270 0 0 159808 6313986186 > > 9763672 1934 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181279214 79875:342509944 [2,1,0]p2 [2,1]p2 > > 2024-06-11T11:03:38.997373+0000 2024-06-11T11:03:38.997373+0000 > > 3970 periodic scrub scheduled @ 2024-06-17T17:04:16.317699+0000 > > 10.4c 5217864 5217864 0 0 4194304 6104537460 > > 9679170 1948 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181081348 79875:341937760 [1,0,2]p1 [1,2]p1 > > 2024-06-12T06:09:49.667680+0000 2024-06-01T13:06:12.938513+0000 > > 520 periodic scrub scheduled @ 2024-06-18T11:39:03.277402+0000 > > 10.4d 5217604 4905278 0 0 8388608 6250536027 > > 9765571 1917 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'179562016 79875:387768172 [0,1,2]p0 [1,2]p1 > > 2024-06-11T13:38:28.479053+0000 2024-06-11T13:38:28.479053+0000 > > 2004 periodic scrub scheduled @ 2024-06-17T15:12:47.615309+0000 > > 10.4e 5220202 5220202 0 0 12215022 5781813383 > > 9291138 1940 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180360470 79875:377712114 [1,0,2]p1 [1,2]p1 > > 2024-06-01T22:06:12.347760+0000 2024-05-25T19:34:49.805104+0000 > > 982 queued for scrub > > 10.4f 5215243 4561466 0 0 0 6099447266 > > 9605810 1930 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181794065 79875:396660094 [2,1,0]p2 [2,1]p2 > > 2024-06-02T13:35:21.451128+0000 2024-06-02T13:35:21.451128+0000 > > 7316 queued for scrub > > 10.50 5215219 5184003 0 0 0 6151789627 > > 9674779 1871 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181695151 79875:405060298 [0,2,1]p0 [1,2]p1 > > 2024-06-11T16:42:00.221013+0000 2024-06-11T16:42:00.221013+0000 > > 2385 periodic scrub scheduled @ 2024-06-17T10:54:13.580253+0000 > > 10.51 5218666 5218666 0 0 8388608 5774655330 > > 9331373 1920 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180711664 79875:342978462 [0,1,2]p0 [1,2]p1 > > 2024-06-11T14:31:51.402844+0000 2024-05-24T11:26:15.539677+0000 > > 730 periodic scrub scheduled @ 2024-06-17T20:48:06.862863+0000 > > 10.52 5218435 5187050 0 0 4194304 6375954134 > > 9843553 1935 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180834874 79875:341502004 [0,1,2]p0 [1,2]p1 > > 2024-06-11T08:41:43.586833+0000 2024-06-11T08:41:43.586833+0000 > > 8368 periodic scrub scheduled @ 2024-06-16T08:59:08.248156+0000 > > 10.53 5216822 5216822 0 0 10001995 5786896551 > > 9368323 1928 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181337168 79875:402047940 [0,2,1]p0 [1,2]p1 > > 2024-06-12T04:54:21.147742+0000 2024-05-23T23:02:36.154347+0000 > > 557 periodic scrub scheduled @ 2024-06-17T11:39:03.144049+0000 > > 10.54 5213949 4567626 0 0 8388608 6095478377 > > 9664814 1944 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'182179986 79875:421118572 [2,1,0]p2 [2,1]p2 > > 2024-06-12T09:19:30.470319+0000 2024-06-01T17:56:32.089348+0000 > > 551 periodic scrub scheduled @ 2024-06-18T09:51:50.591138+0000 > > 10.55 5217389 5217389 0 0 4194948 5905845521 > > 9403344 1950 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'183481821 79875:341223864 [1,2,0]p1 [1,2]p1 > > 2024-06-11T15:31:05.524698+0000 2024-05-29T11:50:10.322128+0000 > > 584 periodic scrub scheduled @ 2024-06-17T04:28:55.809357+0000 > > 10.56 5219142 5219142 0 0 0 6311549101 > > 9773102 1897 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'184062267 79875:408792597 [1,2,0]p1 [1,2]p1 > > 2024-06-10T22:17:28.523159+0000 2024-06-10T22:17:28.523159+0000 > > 8211 periodic scrub scheduled @ 2024-06-17T03:16:39.913869+0000 > > 10.57 5219035 4894148 0 0 180 5862209193 > > 9396761 1895 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'181886155 79875:340148291 [0,1,2]p0 [1,2]p1 > > 2024-06-10T22:33:19.364284+0000 2024-05-28T12:05:26.762682+0000 > > 951 periodic scrub scheduled @ 2024-06-16T00:55:55.641233+0000 > > 10.58 5218318 5185905 0 0 9878587 5911568798 > > 9459707 1875 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179109213 79875:390200207 [2,0,1]p2 [2,1]p2 > > 2024-06-11T13:08:37.380203+0000 2024-05-30T10:19:35.091370+0000 > > 578 periodic scrub scheduled @ 2024-06-17T15:53:56.734391+0000 > > 10.59 5218315 5185838 0 0 4194304 5727052522 > > 9214424 1875 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181700141 79875:400297310 [2,0,1]p2 [2,1]p2 > > 2024-06-11T11:15:00.221765+0000 2024-05-21T11:28:39.056382+0000 > > 682 periodic deep scrub scheduled @ > 2024-06-16T13:00:43.750185+0000 > > 10.5a 5218301 5218301 0 0 4194304 5833097049 > > 9365354 1942 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180699909 79875:393344045 [1,2,0]p1 [1,2]p1 > > 2024-06-12T01:50:51.954645+0000 2024-05-25T00:11:49.917556+0000 > > 497 periodic scrub scheduled @ 2024-06-17T12:05:07.871109+0000 > > 10.5b 5215134 5215134 0 0 4194802 6070043467 > > 9585134 1892 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181149500 79875:342539489 [2,0,1]p2 [2,1]p2 > > 2024-06-12T08:19:32.265112+0000 2024-06-02T11:33:23.094569+0000 > > 714 periodic scrub scheduled @ 2024-06-17T23:10:14.905316+0000 > > 10.5c 5221503 5221503 0 0 4194304 5989312430 > > 9524556 1939 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180748682 79875:340750417 [1,2,0]p1 [1,2]p1 > > 2024-06-11T09:37:59.483197+0000 2024-05-28T03:41:58.475567+0000 > > 1021 periodic scrub scheduled @ 2024-06-17T16:20:33.624394+0000 > > 10.5d 5222202 5222202 0 0 8388608 6031557641 > > 9520790 1928 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179508607 79875:399919359 [1,0,2]p1 [1,2]p1 > > 2024-06-11T17:10:33.165278+0000 2024-05-30T05:53:10.772736+0000 > > 575 periodic scrub scheduled @ 2024-06-17T11:34:40.016966+0000 > > 10.5e 5212770 4506643 0 0 0 5830165136 > > 9388228 1938 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180002484 79875:391844220 [2,0,1]p2 [2,1]p2 > > 2024-06-11T22:32:49.026280+0000 2024-05-24T15:31:50.362016+0000 > > 523 periodic scrub scheduled @ 2024-06-18T06:19:36.998416+0000 > > 10.5f 5221504 5221504 0 0 4194394 5807975316 > > 9387383 1879 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181161493 79875:408530090 [0,2,1]p0 [1,2]p1 > > 2024-06-11T15:10:33.261745+0000 2024-05-25T06:37:01.629711+0000 > > 845 periodic scrub scheduled @ 2024-06-18T01:25:00.005508+0000 > > 10.60 5217784 5186615 0 0 4194304 6208553838 > > 9749044 1962 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181569072 79875:392211608 [0,1,2]p0 [1,2]p1 > > 2024-06-11T21:44:35.372861+0000 2024-06-11T21:44:35.372861+0000 > > 1505 periodic scrub scheduled @ 2024-06-17T19:37:54.401372+0000 > > 10.61 5221292 4586636 0 0 8388608 5766320671 > > 9280184 1910 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180945714 79875:344272511 [0,1,2]p0 [1,2]p1 > > 2024-06-11T16:51:25.519891+0000 2024-05-24T17:37:21.814566+0000 > > 566 periodic scrub scheduled @ 2024-06-18T02:56:43.473028+0000 > > 10.62 5218498 4699002 0 0 4194466 6027128219 > > 9555111 1884 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180387699 79875:341929273 [1,2,0]p1 [1,2]p1 > > 2024-06-12T03:10:08.547987+0000 2024-06-01T02:47:43.109035+0000 > > 509 periodic scrub scheduled @ 2024-06-17T23:58:36.771468+0000 > > 10.63 5219101 5219101 0 0 0 6040382110 > > 9500040 1885 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181623402 79875:389756489 [0,1,2]p0 [1,2]p1 > > 2024-06-02T23:50:14.947543+0000 2024-06-02T23:50:14.947543+0000 > > 13925 queued for scrub > > 10.64 5217025 5217025 0 0 2277792 6222337345 > > 9701950 1866 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181853168 79875:406174284 [1,2,0]p1 [1,2]p1 > > 2024-06-02T04:57:41.872723+0000 2024-06-02T04:57:41.872723+0000 > > 6394 queued for scrub > > 10.65 5217908 4590550 0 0 4194326 6223151288 > > 9709345 1916 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181196382 79875:340107600 [2,0,1]p2 [2,1]p2 > > 2024-06-11T15:53:28.725001+0000 2024-06-11T15:53:28.725001+0000 > > 1343 periodic scrub scheduled @ 2024-06-17T11:59:39.917825+0000 > > 10.66 5218911 4893654 0 0 4194304 5815703055 > > 9338987 1939 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'183062531 79875:395193073 [1,0,2]p1 [1,2]p1 > > 2024-06-12T01:10:56.919233+0000 2024-05-25T04:50:04.300293+0000 > > 590 periodic scrub scheduled @ 2024-06-18T11:39:55.201903+0000 > > 10.67 5217539 4587208 0 0 4194304 6095526883 > > 9640323 1955 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181875315 79875:342878178 [2,1,0]p2 [2,1]p2 > > 2024-06-12T07:58:41.522730+0000 2024-06-01T07:20:59.951966+0000 > > 553 periodic scrub scheduled @ 2024-06-18T05:40:51.996147+0000 > > 10.68 5215852 4505902 0 0 8388608 6093858586 > > 9596421 1926 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179490643 79875:379632203 [0,1,2]p0 [1,2]p1 > > 2024-06-11T16:02:14.995195+0000 2024-05-30T22:02:42.335910+0000 > > 526 periodic scrub scheduled @ 2024-06-17T02:53:33.980471+0000 > > 10.69 5216596 5216596 0 0 4194304 5833268044 > > 9400414 1899 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181876551 79875:387059449 [1,2,0]p1 [1,2]p1 > > 2024-06-11T17:00:57.111851+0000 2024-05-23T07:05:05.989427+0000 > > 572 periodic scrub scheduled @ 2024-06-17T18:52:42.850336+0000 > > 10.6a 5214106 4497370 0 0 0 5745504618 > > 9296938 1869 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180109669 79875:381871149 [2,1,0]p2 [2,1]p2 > > 2024-06-12T00:27:45.155477+0000 2024-05-25T09:22:19.570011+0000 > > 494 periodic scrub scheduled @ 2024-06-18T07:00:30.017009+0000 > > 10.6b 5219339 5219339 0 0 0 5801390847 > > 9370084 1957 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180041524 79875:342270608 [0,1,2]p0 [1,2]p1 > > 2024-06-12T03:01:39.358464+0000 2024-05-23T01:04:22.522035+0000 > > 488 periodic scrub scheduled @ 2024-06-17T19:03:41.161095+0000 > > 10.6c 5217917 5217917 0 0 554 6100979305 > > 9612250 1955 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180968918 79875:341488318 [1,2,0]p1 [1,2]p1 > > 2024-06-11T18:47:35.969345+0000 2024-05-31T11:54:59.559316+0000 > > 588 periodic scrub scheduled @ 2024-06-18T06:02:38.982765+0000 > > 10.6d 5221048 4550388 0 0 4194304 6024067862 > > 9621947 1927 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179373269 79875:388095690 [2,1,0]p2 [2,1]p2 > > 2024-06-12T03:26:37.931310+0000 2024-05-31T04:10:41.991457+0000 > > 502 periodic scrub scheduled @ 2024-06-18T08:52:34.478975+0000 > > 10.6e 5215550 4888179 0 0 8388608 6110450001 > > 9666892 1906 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'180885974 79875:380260263 [0,1,2]p0 [1,2]p1 > > 2024-06-11T20:18:23.657558+0000 2024-06-11T20:18:23.657558+0000 > > 1218 periodic scrub scheduled @ 2024-06-18T07:12:57.858451+0000 > > 10.6f 5215850 4566820 0 0 0 6107146788 > > 9634911 1923 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181752773 79875:396708021 [2,1,0]p2 [2,1]p2 > > 2024-06-10T17:32:51.735004+0000 2024-06-10T17:32:51.735004+0000 > > 10119 periodic scrub scheduled @ 2024-06-17T02:47:23.675814+0000 > > 10.70 5218211 4672319 0 0 4194304 5793441438 > > 9371563 1881 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180405012 79875:404578822 [1,2,0]p1 [1,2]p1 > > 2024-06-03T04:46:35.130293+0000 2024-05-22T07:50:20.898441+0000 > > 969 queued for deep scrub > > 10.71 5218299 4599312 0 0 4194304 5807950891 > > 9349720 1906 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180690682 79875:344108361 [0,2,1]p0 [1,2]p1 > > 2024-06-12T03:18:16.645927+0000 2024-05-23T09:32:56.055703+0000 > > 487 periodic scrub scheduled @ 2024-06-17T13:47:03.236787+0000 > > 10.72 5223470 4922347 0 0 8388608 6416468325 > > 9922777 1958 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'180797342 79875:342356916 [0,1,2]p0 [1,2]p1 > > 2024-06-12T05:42:47.545762+0000 2024-06-12T05:42:47.545762+0000 > > 1307 periodic scrub scheduled @ 2024-06-18T13:20:41.826465+0000 > > 10.73 5218301 4520651 0 0 0 6293832301 > > 9738943 1953 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'182246986 79875:403792984 [2,0,1]p2 [2,1]p2 > > 2024-06-11T20:59:02.079157+0000 2024-06-11T20:59:02.079157+0000 > > 1399 periodic scrub scheduled @ 2024-06-17T01:00:32.819334+0000 > > 10.74 5219061 4595112 0 0 0 5974775781 > > 9469653 1943 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181947609 79875:420571221 [2,1,0]p2 [2,1]p2 > > 2024-06-03T02:17:19.687482+0000 2024-05-28T08:43:38.825954+0000 > > 927 queued for scrub > > 10.75 5220994 4596634 0 0 25165824 5822315216 > > 9362548 1877 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180376305 79875:338847231 [2,0,1]p2 [2,1]p2 > > 2024-06-03T05:52:28.743255+0000 2024-05-24T05:11:00.211501+0000 > > 981 queued for scrub > > 10.76 5222111 4892295 0 0 0 6067998823 > > 9574254 1935 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'183871180 79875:409218100 [1,0,2]p1 [1,2]p1 > > 2024-06-03T04:14:04.513004+0000 2024-06-03T04:14:04.513004+0000 > > 6987 queued for scrub > > 10.77 5219020 4574683 0 0 0 5889074277 > > 9460824 1896 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181614825 79875:342558162 [2,0,1]p2 [2,1]p2 > > 2024-06-12T08:07:38.169523+0000 2024-05-27T15:15:35.587972+0000 > > 536 periodic scrub scheduled @ 2024-06-17T18:29:44.366900+0000 > > 10.78 5220442 4693235 0 0 8388608 5879566566 > > 9395501 1923 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179031456 79875:391462415 [1,0,2]p1 [1,2]p1 > > 2024-06-02T09:32:42.834344+0000 2024-05-27T10:41:23.822124+0000 > > 994 queued for scrub > > 10.79 5219734 4702117 0 0 4194304 6129933545 > > 9646669 1926 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'182574416 79875:402249812 [1,2,0]p1 [1,2]p1 > > 2024-06-12T05:52:02.959814+0000 2024-06-01T00:53:49.548055+0000 > > 555 periodic scrub scheduled @ 2024-06-17T17:48:00.189475+0000 > > 10.7a 5215303 4526846 0 0 0 6316171107 > > 9817029 1932 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181765231 79875:395877057 [2,0,1]p2 [2,1]p2 > > 2024-06-12T02:23:42.692096+0000 2024-06-12T02:23:42.692096+0000 > > 1457 periodic scrub scheduled @ 2024-06-17T22:13:46.037000+0000 > > 10.7b 5217213 4606816 0 0 4194304 6226859283 > > 9718621 1952 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181413541 79875:343968658 [1,2,0]p1 [1,2]p1 > > 2024-06-11T23:52:28.414752+0000 2024-06-11T23:52:28.414752+0000 > > 1308 periodic scrub scheduled @ 2024-06-17T18:52:39.473049+0000 > > 10.7c 5215075 5215075 0 0 4194304 6013877208 > > 9602345 1867 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180991860 79875:341775615 [0,2,1]p0 [1,2]p1 > > 2024-06-11T23:30:40.614412+0000 2024-05-31T14:10:30.542603+0000 > > 503 periodic scrub scheduled @ 2024-06-17T02:30:15.591036+0000 > > 10.7d 5220700 4569720 0 0 5837609 6258736025 > > 9764836 1897 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179741885 79875:401448508 [2,1,0]p2 [2,1]p2 > > 2024-06-12T04:45:05.030905+0000 2024-06-12T04:45:05.030905+0000 > > 1565 periodic scrub scheduled @ 2024-06-17T04:54:42.039122+0000 > > 10.7e 5217113 5217113 0 0 16777216 6072081676 > > 9598390 1902 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181040361 79875:391032348 [1,2,0]p1 [1,2]p1 > > 2024-06-11T17:37:26.589968+0000 2024-06-11T17:37:26.589968+0000 > > 1613 periodic scrub scheduled @ 2024-06-17T05:06:05.391340+0000 > > 10.7f 5218719 5218719 0 0 9909061 6130469539 > > 9657824 1943 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182283514 79875:408550888 [1,2,0]p1 [1,2]p1 > > 2024-06-11T13:05:03.552526+0000 2024-06-11T13:05:03.552526+0000 > > 1796 periodic scrub scheduled @ 2024-06-17T09:57:04.545523+0000 > > > > * NOTE: Omap statistics are gathered during deep scrub and may be > > inaccurate soon afterwards depending on utilization. See > > http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for > > further details. > > > > > > > > sudo ceph osd df > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > > AVAIL %USE VAR PGS STATUS > > 0 ssd 3.63869 1.00000 3.6 TiB 82 GiB 50 MiB 40 GiB 42 > GiB > > 3.6 TiB 2.21 0.03 0 up > > 1 ssd 3.63869 1.00000 3.6 TiB 2.0 TiB 16 GiB 1021 GiB 1.0 > TiB > > 1.6 TiB 55.41 0.75 161 up > > 2 ssd 3.63869 1.00000 3.6 TiB 3.0 TiB 16 GiB 1.1 TiB 1.9 > TiB > > 676 GiB 81.86 1.11 161 up > > > > > > sudo ceph df detail > > --- RAW STORAGE --- > > CLASS SIZE AVAIL USED RAW USED %RAW USED > > hdd 1.2 PiB 308 TiB 871 TiB 871 TiB 73.86 > > ssd 11 TiB 5.8 TiB 5.1 TiB 5.1 TiB 46.51 > > TOTAL 1.2 PiB 314 TiB 876 TiB 876 TiB 73.61 > > > > --- POOLS --- > > POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) > > (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR > > UNDER COMPR > > .mgr 1 1 827 MiB 827 MiB 0 B 208 1.6 GiB 1.6 GiB > > 0 B 0.11 734 GiB N/A 50 GiB N/A 0 B > > 0 B > > metadata 10 128 1.0 TiB 647 MiB 1.0 TiB 667.90M 2.1 TiB 1.3 GiB > > 2.1 TiB 59.45 719 GiB N/A 3.5 TiB N/A 0 B > > 0 B > > data 11 2048 288 TiB 288 TiB 0 B 566.97M 863 TiB 863 TiB > > 0 B 81.10 67 TiB N/A N/A N/A 3.4 TiB > > 6.8 TiB > > ssd-data 18 32 14 GiB 14 GiB 0 B 265.17k 29 GiB 29 GiB > > 0 B 1.90 734 GiB N/A N/A N/A 0 B > > 0 B > > > > > > sudo ceph balancer status > > { > > "active": true, > > "last_optimize_duration": "0:00:00.000170", > > "last_optimize_started": "Wed Jun 12 13:14:49 2024", > > "mode": "upmap", > > "no_optimization_needed": true, > > "optimize_result": "Some objects (0.172816) are degraded; try again > > later", > > "plans": [] > > } > > > > > > > > [image: ariadne.ai Logo] Lars Köppel > > Developer > > Email: lars.koeppel@ariadne.ai > > Phone: +49 6221 5993580 <+4962215993580> > > ariadne.ai (Germany) GmbH > > Häusserstraße 3, 69115 Heidelberg > > Amtsgericht Mannheim, HRB 744040 > > Geschäftsführer: Dr. Fabian Svara > > https://ariadne.ai > > > > > > On Wed, Jun 12, 2024 at 2:53 PM Anthony D'Atri < anthony.datri@gmail.com> > > wrote: > > > >> If you have: > >> > >> * pg_num too low (defaults are too low) > >> * pg_num not a power of 2 > >> * pg_num != number of OSDs in the pool > >> * balancer not enabled > >> > >> any of those might result in imbalance. > >> > >> > On Jun 12, 2024, at 07:33, Eugen Block <eblock@nde.ag> wrote: > >> > > >> > I don't have any good explanation at this point. Can you share some > more > >> information like: > >> > > >> > ceph pg ls-by-pool <cephfs_metadata> > >> > ceph osd df (for the relevant OSDs) > >> > ceph df > >> > > >> > Thanks, > >> > Eugen > >> > > >> > Zitat von Lars Köppel <lars.koeppel@ariadne.ai>: > >> > > >> >> Since my last update the size of the largest OSD increased by 0.4 TiB > >> while > >> >> the smallest one only increased by 0.1 TiB. How is this possible? > >> >> > >> >> Because the metadata pool reported to have only 900MB space left, I > >> stopped > >> >> the hot-standby MDS. This gave me 8GB back but these filled up in the > >> last > >> >> 2h. > >> >> I think I have to zap the next OSD because the filesystem is getting > >> read > >> >> only... > >> >> > >> >> How is it possible that an OSD has over 1 TiB less data on it after a > >> >> rebuild? And how is it possible to have so different sizes of OSDs? > >> >> > >> >> > >> >> [image: ariadne.ai Logo] Lars Köppel > >> >> Developer > >> >> Email: lars.koeppel@ariadne.ai > >> >> Phone: +49 6221 5993580 <+4962215993580> > >> >> ariadne.ai (Germany) GmbH > >> >> Häusserstraße 3, 69115 Heidelberg > >> >> Amtsgericht Mannheim, HRB 744040 > >> >> Geschäftsführer: Dr. Fabian Svara > >> >> https://ariadne.ai > >> >> > >> >> > >> >> On Tue, Jun 11, 2024 at 3:47 PM Lars Köppel < lars.koeppel@ariadne.ai > > > >> wrote: > >> >> > >> >>> Only in warning mode. And there were no PG splits or merges in the > >> last 2 > >> >>> month. > >> >>> > >> >>> > >> >>> [image: ariadne.ai Logo] Lars Köppel > >> >>> Developer > >> >>> Email: lars.koeppel@ariadne.ai > >> >>> Phone: +49 6221 5993580 <+4962215993580> > >> >>> ariadne.ai (Germany) GmbH > >> >>> Häusserstraße 3, 69115 Heidelberg > >> >>> Amtsgericht Mannheim, HRB 744040 > >> >>> Geschäftsführer: Dr. Fabian Svara > >> >>> https://ariadne.ai > >> >>> > >> >>> > >> >>> On Tue, Jun 11, 2024 at 3:32 PM Eugen Block <eblock@nde.ag> wrote: > >> >>> > >> >>>> I don't think scrubs can cause this. Do you have autoscaler > enabled? > >> >>>> > >> >>>> Zitat von Lars Köppel <lars.koeppel@ariadne.ai>: > >> >>>> > >> >>>> > Hi, > >> >>>> > > >> >>>> > thank you for your response. > >> >>>> > > >> >>>> > I don't think this thread covers my problem, because the OSDs for > >> the > >> >>>> > metadata pool fill up at different rates. So I would think this > is > >> no > >> >>>> > direct problem with the journal. > >> >>>> > Because we had earlier problems with the journal I changed some > >> >>>> > settings(see below). I already restarted all MDS multiple times > but > >> no > >> >>>> > change here. > >> >>>> > > >> >>>> > The health warnings regarding cache pressure resolve normally > after > >> a > >> >>>> > short period of time, when the heavy load on the client ends. > >> Sometimes > >> >>>> it > >> >>>> > stays a bit longer because an rsync is running and copying data > on > >> the > >> >>>> > cluster(rsync is not good at releasing the caps). > >> >>>> > > >> >>>> > Could it be a problem if scrubs run most of the time in the > >> background? > >> >>>> Can > >> >>>> > this block any other tasks or generate new data itself? > >> >>>> > > >> >>>> > Best regards, > >> >>>> > Lars > >> >>>> > > >> >>>> > > >> >>>> > global basic mds_cache_memory_limit > >> >>>> > 17179869184 > >> >>>> > global advanced mds_max_caps_per_client > >> >>>> > 16384 > >> >>>> > global advanced > >> >>>> mds_recall_global_max_decay_threshold > >> >>>> > 262144 > >> >>>> > global advanced mds_recall_max_decay_rate > >> >>>> > 1.000000 > >> >>>> > global advanced > mds_recall_max_decay_threshold > >> >>>> > 262144 > >> >>>> > mds advanced mds_cache_trim_threshold > >> >>>> > 131072 > >> >>>> > mds advanced mds_heartbeat_grace > >> >>>> > 120.000000 > >> >>>> > mds advanced mds_heartbeat_reset_grace > >> >>>> > 7400 > >> >>>> > mds advanced mds_tick_interval > >> >>>> > 3.000000 > >> >>>> > > >> >>>> > > >> >>>> > [image: ariadne.ai Logo] Lars Köppel > >> >>>> > Developer > >> >>>> > Email: lars.koeppel@ariadne.ai > >> >>>> > Phone: +49 6221 5993580 <+4962215993580> > >> >>>> > ariadne.ai (Germany) GmbH > >> >>>> > Häusserstraße 3, 69115 Heidelberg > >> >>>> > Amtsgericht Mannheim, HRB 744040 > >> >>>> > Geschäftsführer: Dr. Fabian Svara > >> >>>> > https://ariadne.ai > >> >>>> > > >> >>>> > > >> >>>> > On Tue, Jun 11, 2024 at 2:05 PM Eugen Block < eblock@nde.ag> > wrote: > >> >>>> > > >> >>>> >> Hi, > >> >>>> >> > >> >>>> >> can you check if this thread [1] applies to your situation? You > >> don't > >> >>>> >> have multi-active MDS enabled, but maybe it's still some journal > >> >>>> >> trimming, or maybe misbehaving clients? In your first post there > >> were > >> >>>> >> health warnings regarding cache pressure and cache size. Are > those > >> >>>> >> resolved? > >> >>>> >> > >> >>>> >> [1] > >> >>>> >> > >> >>>> >> > >> >>>> > >> >
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7U27L27FHHPD...
> >> >>>> >> > >> >>>> >> Zitat von Lars Köppel <lars.koeppel@ariadne.ai>: > >> >>>> >> > >> >>>> >> > Hello everyone, > >> >>>> >> > > >> >>>> >> > short update to this problem. > >> >>>> >> > The zapped OSD is rebuilt and it has now 1.9 TiB (the expected > >> size > >> >>>> >> ~50%). > >> >>>> >> > The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They > >> jumped up > >> >>>> and > >> >>>> >> > down a lot but the higher one has now also reached 'nearfull' > >> >>>> status. How > >> >>>> >> > is this possible? What is going on? > >> >>>> >> > > >> >>>> >> > Does anyone have a solution how to fix this without zapping > the > >> OSD? > >> >>>> >> > > >> >>>> >> > Best regards, > >> >>>> >> > Lars > >> >>>> >> > > >> >>>> >> > > >> >>>> >> > [image: ariadne.ai Logo] Lars Köppel > >> >>>> >> > Developer > >> >>>> >> > Email: lars.koeppel@ariadne.ai > >> >>>> >> > Phone: +49 6221 5993580 <+4962215993580> > >> >>>> >> > ariadne.ai (Germany) GmbH > >> >>>> >> > Häusserstraße 3, 69115 Heidelberg > >> >>>> >> > Amtsgericht Mannheim, HRB 744040 > >> >>>> >> > Geschäftsführer: Dr. Fabian Svara > >> >>>> >> > https://ariadne.ai > >> >>>> >> > _______________________________________________ > >> >>>> >> > ceph-users mailing list -- ceph-users@ceph.io > >> >>>> >> > To unsubscribe send an email to ceph-users-leave@ceph.io > >> >>>> >> > >> >>>> >> > >> >>>> >> _______________________________________________ > >> >>>> >> ceph-users mailing list -- ceph-users@ceph.io > >> >>>> >> To unsubscribe send an email to ceph-users-leave@ceph.io > >> >>>> >> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> > > >> > > >> > _______________________________________________ > >> > ceph-users mailing list -- ceph-users@ceph.io > >> > To unsubscribe send an email to ceph-users-leave@ceph.io > >> > >> > > > >