I've been going round and round in circles trying to work this one out but I'm getting nowhere. We're running a 4 node quincy cluster (17.2.6) which recently reported the following:
ceph.log-20230729.gz:2023-07-28T08:31:42.390003+0000 osd.26 (osd.26) 13834 : cluster [WRN] Large omap object found. Object: 5:6c65dd84:users.uid::callrecordings$callrecordings_rw.buckets:head PG: 5.21bba636 (5.16) Key count: 378454 Size (bytes): 75565579
This happened a week or so ago (only the key count was only just over the 200000 threshold on that occasion) and after much searching around, I found an article that suggested a deep scrub on the pg would likely resolve the issue, so I forced a deep scrub and shortly after, the warning cleared. Came into the office today to discover the above. It's on the same PG as before which is in the default.rgw.meta pool. This time, after forcing a deep-scrub on that PG, nothing changed. I did it a second time just to be sure but got the same result.
I keep finding a suse article that simply suggests increasing the threshold to the previous default of 2,000,000, but other articles I read say it was lowered for a reason and that by the time it hits that figure, it's too late so I don't want to just mask it. Problem is that I don't really understand it. I found a thread here from a bit over two years ago but their issue was in the default.rgw.buckets.index pool. A step in the solution was to list out the problematic object id and check the objects per shard however, if I issue the command "rados -p default.rgw.meta ls" it returns nothing. I get a big list from "rados -p default.rgw.buckets.index ls" just nothing from the first pool. I think it may be because the meta pool isn't indexed based on something I read, but I really don't know what I'm talking about tbh.
I don't know if this is helpful, but if I list out all the PGs for that pool, there are 32 PGs and 5.16 shows 80186950 bytes and 401505 keys. PG 5.c has 75298 and 384 keys. The remaining 30 PGs show zero bytes and zero keys. I'm really not sure how to troubleshoot and resolve from here. For the record, dynamic resharding is enabled in that no options have been set in the config and that is the default setting.
Based on the suse article I mentioned which also references the default.rgw.meta pool, I'm gathering our issue is because we have so many buckets that are all owned by the one user and the solution is either:
* delete unused buckets
* create multiple users and spread buckets evenly across all users (not something we can do)
* increase the threshold to stop the warning
Problem is that I'm having trouble verifying this is the issue. I've tried dumping out bucket stats to a file (radosgw-admin bucket stats > bucket_stats.txt) but after three hours this is still running with no output.
Thanks for your time,
Mark
Hello,
I have 3 Luminous (12.2.12) clusters setup with rgw multi-site. One cluster
recently suffered data loss from PGs lost to bad SSDs. We're looking at how
to recover from this scenario. The other two clusters are healthy.
Will RGW sync the missing data? Do buckets and objects need to be restored
using rclone?
From `radosgw-admin sync error list`, these are the error types:
$ cat sync-error-list | jq -r '.[].entries[].info.message' | sort -h |
uniq -c | sort -h
2 failed to sync bucket instance: (13) Permission denied
3 failed to sync object(13) Permission denied
100 failed to sync bucket instance: (125) Operation canceled
239 failed to sync bucket instance: (5) Input/output error
404 failed to read remote metadata entry: (11) Resource temporarily
unavailable
2148 failed to sync object(5) Input/output error
--
Gregory O’Neill
Hello Everyone ,
I am getting [WRN] LARGE_OMAP_OBJECTS: 18 large omap objects warning
in one of my clusters . I see one of the buckets has a huge number of
shards 1999 and "num_objects": 221185360 when I check bucket stats using
radosgw-admin bucket stats . However I see only 8 files when I actually do
list bucket using python boto3. I am not sure what those objects are
missing somewhere .
# radosgw-admin bucket stats --bucket=epbucket
{
"bucket": "epbucket",
"num_shards": 1999,
"tenant": "",
"zonegroup": "ac361c18-7420-4739-8e92-73109fc8ec80",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "f3e47d62-b9ff-4f38-b863-d7bfc155d698.35744173.268063",
"marker": "f3e47d62-b9ff-4f38-b863-d7bfc155d698.35744173.1225",
"index_type": "Normal",
"owner": "user",
"ver":
"0#3,1#1,2#1,3#9,4#1,5#3,6#1,7#3,8#1,9#8,10#7,11#1,12#3,13#1,14#1,15#1,16#1,17#1,18#5,19#1,20#1,21#3,22#3,23#1,24#3,25#1,26#1,27#1,28#3,29#3,30#2,31#3,32#2,33#1,34#3,35#1,36#3,37#1,38#3,39#1,40#3,41#1,42#1,43#1,44#1,45#1,46#3,47#1,48#3,49#1,50#3,51#1,52#1,53#1,54#1,55#1,56#3,57#1,58#1,59#5,60#1,61#1,62#1,63#3,64#1,65#1,66#3,67#1,68#5,69#1,70#1,71#3,72#1,73#3,74#2,75#3,76#1,77#3,78#1,79#1,80#3,81#3,82#3,83#3,84#1,85#1,86#3,87#3,88#8,89#1,90#1,91#1,92#5,93#7,94#1,95#1,96#3,97#1,98#5,99#1,100#1,101#3,102#5,103#3,104#1,105#1,106#1,107#1,108#1,109#239050,110#1,111#1,112#1,113#5,114#1,115#1,116#1,117#3,118#3,119#9,120#1,121#1,122#3,123#5,124#3,125#3,126#1,127#1,128#1,129#3,130#3,131#3,132#3,133#5,134#1,135#1,136#1,137#1,138#5,139#3,140#1,141#1,142#3,143#3,144#1,145#1,146#3,147#3,148#3,149#1,150#3,151#1,152#1,153#3,154#7,155#1,156#5,157#3,158#3,159#3,160#5,161#3,162#3,163#3,164#3,165#1,166#1,167#5,168#3,169#1,170#1,171#1,172#4,173#1,174#12,175#1,176#3,177#3,178#3,179#1,180#3,181#33,182#1,183#1,184#1,185#1,186#3,187#3,188#11,189#3,190#5,191#3,192#1,193#5,194#3,195#2,196#3,197#1,198#3,199#1,200#7,201#3,202#7,203#1,204#3,205#3,206#1,207#1,208#3,209#1,210#7,211#3,212#2,213#9,214#1,215#3,216#1,217#1,218#7,219#1,220#1,221#5,222#1,223#1,224#3,225#1,226#1,227#3,228#3,229#1,230#5,231#5,232#1,233#3,234#1,235#1,236#5,237#1,238#1,239#3,240#7,241#3,242#1,243#3,244#1,245#1,246#3,247#1,248#3,249#3,250#1,251#1,252#3,253#1,254#3,255#1,256#3,257#1,258#1,259#7,260#9,261#3,262#1,263#1,264#3,265#1,266#1,267#5,268#1,269#1,270#1,271#1,272#1,273#3,274#1,275#1,276#1,277#3,278#1,279#7,280#1,281#238803,282#3,283#1,284#1,285#1,286#1,287#3,288#1,289#1,290#1,291#16,292#3,293#3,294#1,295#1,296#5,297#3,298#3,299#7,300#5,301#5,302#1,303#1,304#1,305#3,306#3,307#5,308#1,309#1,310#3,311#3,312#1,313#1,314#5,315#3,316#7,317#1,318#1,319#3,320#3,321#3,322#1,323#9,324#3,325#1,326#5,327#1,328#3,329#9,330#3,331#1,332#3,333#3,334#3,335#3,336#3,337#9,338#1,339#1,340#1,341#3,342#3,343#3,344#2,345#3,346#3,347#1,348#3,349#3,350#5,351#1,352#5,353#8,354#5,355#1,356#3,357#1,358#3,359#1,360#1,361#1,362#1,363#5,364#1,365#1,366#1,367#1,368#1,369#1,370#3,371#3,372#3,373#3,374#1,375#3,376#5,377#8,378#1,379#1,380#1,381#1,382#3,383#3,384#1,385#3,386#1,387#1,388#1,389#2,390#1,391#1,392#1,393#1,394#5,395#5,396#1,397#5,398#3,399#1,400#1,401#7,402#1,403#5,404#1,405#1,406#3,407#3,408#3,409#1,410#1,411#1,412#1,413#1,414#1,415#1,416#1,417#1,418#3,419#3,420#1,421#5,422#5,423#5,424#1,425#1,426#1,427#1,428#9,429#1,430#3,431#1,432#3,433#3,434#1,435#1,436#3,437#5,438#3,439#3,440#1,441#1,442#1,443#1,444#1,445#5,446#3,447#9,448#1,449#1,450#1,451#1,452#1,453#3,454#1,455#1,456#3,457#5,458#5,459#3,460#9,461#1,462#1,463#1,464#1,465#1,466#1,467#3,468#1,469#3,470#8,471#1,472#1,473#1,474#7,475#3,476#1,477#11,478#10,479#1,480#1,481#1,482#3,483#1,484#7,485#1,486#1,487#3,488#8,489#1,490#1,491#8,492#3,493#1,494#3,495#5,496#1,497#1,498#1,499#3,500#1,501#5,502#1,503#1,504#3,505#1,506#1,507#1,508#3,509#3,510#3,511#1,512#7,513#3,514#1,515#3,516#5,517#1,518#3,519#1,520#3,521#3,522#3,523#5,524#3,525#3,526#1,527#1,528#1,529#1,530#1,531#3,532#1,533#3,534#5,535#5,536#1,537#3,538#3,539#5,540#1,541#1,542#1,543#1,544#7,545#3,546#1,547#1,548#1,549#3,550#1,551#1,552#3,553#1,554#3,555#3,556#1,557#3,558#5,559#3,560#3,561#8,562#1,563#3,564#7,565#3,566#5,567#1,568#3,569#3,570#3,571#3,572#5,573#1,574#1,575#5,576#5,577#3,578#7,579#1,580#1,581#5,582#1,583#3,584#1,585#1,586#9,587#9,588#3,589#1,590#1,591#3,592#3,593#1,594#10,595#3,596#1,597#3,598#1,599#1,600#1,601#3,602#3,603#1,604#3,605#5,606#1,607#1,608#1,609#10,610#3,611#264711,612#3,613#5,614#3,615#3,616#1,617#1,618#7,619#3,620#3,621#1,622#3,623#1,624#3,625#1,626#3,627#1,628#1,629#3,630#1,631#2,632#233833,633#1,634#1,635#5,636#1,637#1,638#1,639#1,640#3,641#1,642#1,643#3,644#1,645#1,646#10,647#1,648#1,649#1,650#3,651#1,652#1,653#5,654#1,655#7,656#7,657#1,658#1,659#5,660#3,661#3,662#1,663#3,664#1,665#1,666#3,667#1,668#7,669#1,670#3,671#3,672#1,673#1,674#1,675#1,676#1,677#3,678#1,679#11,680#1,681#7,682#5,683#1,684#1,685#1,686#1,687#3,688#1,689#1,690#3,691#3,692#1,693#1,694#1,695#1,696#1,697#1,698#1,699#5,700#1,701#1,702#1,703#1,704#3,705#1,706#8,707#3,708#3,709#8,710#1,711#2,712#3,713#1,714#1,715#10,716#3,717#1,718#1,719#3,720#1,721#7,722#3,723#1,724#5,725#1,726#5,727#1,728#3,729#1,730#3,731#1,732#7,733#3,734#3,735#1,736#5,737#1,738#5,739#1,740#1,741#1,742#1,743#1,744#1,745#3,746#1,747#1,748#1,749#5,750#12,751#1,752#9,753#7,754#5,755#1,756#1,757#1,758#1,759#5,760#1,761#3,762#3,763#3,764#1,765#1,766#11,767#5,768#3,769#3,770#5,771#3,772#1,773#1,774#1,775#1,776#7,777#5,778#3,779#3,780#2,781#5,782#1,783#1,784#5,785#1,786#3,787#1,788#3,789#5,790#10,791#3,792#7,793#1,794#1,795#3,796#1,797#3,798#1,799#1,800#1,801#1,802#3,803#1,804#3,805#1,806#1,807#1,808#1,809#3,810#1,811#3,812#2,813#1,814#1,815#1,816#3,817#1,818#3,819#1,820#3,821#1,822#1,823#1,824#5,825#5,826#3,827#1,828#3,829#3,830#1,831#1,832#3,833#5,834#3,835#3,836#3,837#7,838#7,839#3,840#1,841#3,842#1,843#1,844#1,845#12,846#1,847#3,848#1,849#1,850#1,851#1,852#1,853#5,854#3,855#1,856#7,857#8,858#3,859#1,860#3,861#1,862#1,863#1,864#1,865#1,866#1,867#5,868#3,869#1,870#3,871#3,872#7,873#9,874#7,875#1,876#1,877#3,878#1,879#3,880#1,881#1,882#1,883#1,884#3,885#7,886#1,887#1,888#1,889#3,890#1,891#1,892#1,893#1,894#3,895#1,896#1,897#3,898#1,899#3,900#3,901#1,902#1,903#1,904#1,905#1,906#1,907#3,908#3,909#3,910#3,911#1,912#1,913#5,914#3,915#3,916#1,917#1,918#1,919#1,920#3,921#1,922#7,923#2,924#3,925#1,926#3,927#1,928#7,929#1,930#5,931#1,932#3,933#3,934#1,935#1,936#3,937#1,938#3,939#1,940#5,941#1,942#5,943#1,944#1,945#12,946#1,947#5,948#1,949#1,950#3,951#7,952#1,953#1,954#1,955#3,956#1,957#5,958#3,959#3,960#3,961#3,962#3,963#3,964#3,965#3,966#1,967#3,968#3,969#3,970#1,971#18,972#3,973#3,974#1,975#1,976#1,977#3,978#1,979#1,980#1,981#1,982#3,983#8,984#5,985#1,986#8,987#3,988#1,989#1,990#1,991#1,992#1,993#5,994#1,995#373523,996#3,997#3,998#3,999#1,1000#1,1001#1,1002#1,1003#1,1004#3,1005#1,1006#1,1007#1,1008#1,1009#7,1010#1,1011#3,1012#1,1013#1,1014#1,1015#1,1016#3,1017#3,1018#1,1019#5,1020#1,1021#1,1022#5,1023#3,1024#1,1025#1,1026#7,1027#3,1028#3,1029#3,1030#1,1031#3,1032#1,1033#1,1034#3,1035#9,1036#1,1037#1,1038#3,1039#5,1040#5,1041#5,1042#3,1043#1,1044#3,1045#7,1046#1,1047#3,1048#3,1049#1,1050#1,1051#1,1052#3,1053#1,1054#3,1055#3,1056#5,1057#8,1058#3,1059#1,1060#3,1061#3,1062#7,1063#1,1064#1,1065#7,1066#1,1067#1,1068#1,1069#3,1070#3,1071#3,1072#3,1073#7,1074#1,1075#3,1076#1,1077#3,1078#1,1079#1,1080#5,1081#1,1082#1,1083#1,1084#3,1085#1,1086#3,1087#1,1088#1,1089#1,1090#1,1091#1,1092#5,1093#1,1094#5,1095#3,1096#1,1097#1,1098#3,1099#1,1100#1,1101#5,1102#1,1103#5,1104#3,1105#1,1106#1,1107#3,1108#3,1109#1,1110#1,1111#1,1112#1,1113#3,1114#1,1115#3,1116#3,1117#1,1118#1,1119#1,1120#1,1121#1,1122#3,1123#1,1124#5,1125#1,1126#1,1127#5,1128#9,1129#5,1130#3,1131#1,1132#7,1133#1,1134#3,1135#2,1136#1,1137#1,1138#1,1139#1,1140#1,1141#1,1142#5,1143#3,1144#1,1145#5,1146#10,1147#235181,1148#1,1149#3,1150#1,1151#3,1152#1,1153#3,1154#7,1155#1,1156#1,1157#3,1158#1,1159#1,1160#3,1161#5,1162#1,1163#1,1164#1,1165#1,1166#5,1167#1,1168#3,1169#1,1170#1,1171#3,1172#3,1173#2,1174#1,1175#1,1176#1,1177#1,1178#1,1179#1,1180#5,1181#5,1182#1,1183#1,1184#1,1185#3,1186#1,1187#1,1188#1,1189#1,1190#5,1191#9,1192#3,1193#8,1194#1,1195#1,1196#1,1197#1,1198#1,1199#1,1200#3,1201#1,1202#1,1203#1,1204#3,1205#1,1206#1,1207#1,1208#1,1209#1,1210#1,1211#1,1212#1,1213#2,1214#1,1215#1,1216#3,1217#3,1218#3,1219#3,1220#1,1221#3,1222#1,1223#1,1224#3,1225#17,1226#3,1227#3,1228#239050,1229#1,1230#3,1231#3,1232#1,1233#5,1234#3,1235#1,1236#1,1237#1,1238#1,1239#1,1240#1,1241#5,1242#1,1243#1,1244#3,1245#5,1246#1,1247#3,1248#1,1249#3,1250#1,1251#3,1252#3,1253#1,1254#1,1255#1,1256#1,1257#1,1258#1,1259#3,1260#5,1261#3,1262#3,1263#1,1264#1,1265#1,1266#1,1267#1,1268#5,1269#3,1270#3,1271#5,1272#1,1273#1,1274#1,1275#5,1276#3,1277#1,1278#8,1279#5,1280#5,1281#1,1282#5,1283#1,1284#1,1285#5,1286#1,1287#3,1288#1,1289#1,1290#3,1291#1,1292#1,1293#3,1294#1,1295#1,1296#1,1297#9,1298#3,1299#1,1300#1,1301#1,1302#3,1303#1,1304#3,1305#3,1306#1,1307#3,1308#1,1309#1,1310#3,1311#3,1312#1,1313#3,1314#1,1315#1,1316#1,1317#5,1318#3,1319#1,1320#1,1321#1,1322#1,1323#3,1324#1,1325#3,1326#8,1327#1,1328#7,1329#6,1330#1,1331#1,1332#1,1333#3,1334#1,1335#1,1336#1,1337#1,1338#1,1339#1,1340#3,1341#3,1342#1,1343#3,1344#5,1345#1,1346#1,1347#1,1348#3,1349#3,1350#1,1351#1,1352#1,1353#1,1354#8,1355#1,1356#3,1357#3,1358#3,1359#3,1360#1,1361#1,1362#1,1363#5,1364#3,1365#1,1366#1,1367#3,1368#5,1369#5,1370#1,1371#1,1372#1,1373#3,1374#1,1375#1,1376#1,1377#7,1378#1,1379#3,1380#1,1381#1,1382#1,1383#1,1384#1,1385#6,1386#1,1387#1,1388#3,1389#1,1390#1,1391#1,1392#1,1393#3,1394#1,1395#3,1396#1,1397#1,1398#1,1399#1,1400#5,1401#3,1402#5,1403#3,1404#5,1405#5,1406#3,1407#1,1408#3,1409#1,1410#3,1411#1,1412#1,1413#3,1414#1,1415#1,1416#5,1417#3,1418#12,1419#3,1420#1,1421#1,1422#3,1423#1,1424#5,1425#7,1426#3,1427#1,1428#1,1429#1,1430#3,1431#1,1432#3,1433#1,1434#1,1435#1,1436#1,1437#1,1438#3,1439#1,1440#1,1441#3,1442#3,1443#1,1444#1,1445#1,1446#1,1447#1,1448#3,1449#3,1450#1,1451#1,1452#3,1453#1,1454#1,1455#1,1456#3,1457#1,1458#5,1459#3,1460#1,1461#3,1462#1,1463#1,1464#1,1465#1,1466#1,1467#1,1468#6,1469#1,1470#1,1471#1,1472#1,1473#1,1474#1,1475#1,1476#5,1477#1,1478#3,1479#8,1480#1,1481#3,1482#1,1483#1,1484#7,1485#1,1486#1,1487#3,1488#3,1489#1,1490#1,1491#1,1492#1,1493#1,1494#1,1495#5,1496#5,1497#1,1498#8,1499#1,1500#3,1501#3,1502#1,1503#1,1504#5,1505#1,1506#1,1507#3,1508#3,1509#3,1510#1,1511#3,1512#3,1513#1,1514#1,1515#1,1516#1,1517#3,1518#1,1519#3,1520#1,1521#1,1522#3,1523#1,1524#7,1525#1,1526#1,1527#1,1528#1,1529#1,1530#3,1531#1,1532#1,1533#1,1534#3,1535#5,1536#3,1537#3,1538#7,1539#3,1540#1,1541#1,1542#1,1543#3,1544#1,1545#1,1546#1,1547#1,1548#1,1549#7,1550#1,1551#1,1552#3,1553#1,1554#5,1555#3,1556#1,1557#1,1558#1,1559#5,1560#1,1561#1,1562#1,1563#1,1564#5,1565#3,1566#3,1567#3,1568#1,1569#3,1570#1,1571#3,1572#1,1573#13,1574#1,1575#1,1576#1,1577#1,1578#1,1579#1,1580#1,1581#1,1582#3,1583#1,1584#1,1585#1,1586#3,1587#1,1588#3,1589#5,1590#10,1591#3,1592#3,1593#1,1594#7,1595#1,1596#3,1597#5,1598#1,1599#1,1600#5,1601#3,1602#1,1603#1,1604#1,1605#3,1606#1,1607#1,1608#5,1609#15,1610#3,1611#3,1612#3,1613#1,1614#3,1615#1,1616#1,1617#1,1618#3,1619#3,1620#1,1621#3,1622#7,1623#1,1624#5,1625#1,1626#3,1627#3,1628#1,1629#1,1630#1,1631#3,1632#1,1633#1,1634#1,1635#1,1636#1,1637#3,1638#1,1639#3,1640#1,1641#1,1642#1,1643#1,1644#3,1645#1,1646#1,1647#3,1648#5,1649#1,1650#1,1651#1,1652#7,1653#3,1654#1,1655#1,1656#13,1657#1,1658#3,1659#3,1660#9,1661#1,1662#7,1663#3,1664#3,1665#1,1666#1,1667#1,1668#1,1669#1,1670#1,1671#16,1672#3,1673#1,1674#3,1675#1,1676#1,1677#1,1678#3,1679#1,1680#1,1681#1,1682#1,1683#1,1684#3,1685#1,1686#1,1687#1,1688#5,1689#5,1690#1,1691#3,1692#1,1693#1,1694#3,1695#3,1696#1,1697#5,1698#1,1699#3,1700#1,1701#1,1702#1,1703#3,1704#1,1705#5,1706#1,1707#9,1708#1,1709#1,1710#3,1711#3,1712#1,1713#5,1714#1,1715#1,1716#1,1717#1,1718#1,1719#1,1720#3,1721#3,1722#1,1723#1,1724#3,1725#1,1726#1,1727#1,1728#1,1729#3,1730#1,1731#3,1732#3,1733#10,1734#3,1735#3,1736#3,1737#3,1738#1,1739#1,1740#5,1741#1,1742#1,1743#1,1744#3,1745#1,1746#5,1747#3,1748#5,1749#1,1750#3,1751#9,1752#3,1753#1,1754#1,1755#3,1756#1,1757#3,1758#1,1759#3,1760#3,1761#3,1762#3,1763#1,1764#1,1765#5,1766#1,1767#1,1768#5,1769#5,1770#1,1771#1,1772#7,1773#7,1774#5,1775#3,1776#3,1777#1,1778#3,1779#1,1780#1,1781#3,1782#1,1783#1,1784#3,1785#1,1786#1,1787#1,1788#1,1789#1,1790#3,1791#3,1792#3,1793#1,1794#1,1795#1,1796#3,1797#1,1798#8,1799#3,1800#5,1801#9,1802#4,1803#3,1804#3,1805#1,1806#1,1807#3,1808#3,1809#3,1810#1,1811#5,1812#3,1813#5,1814#10,1815#1,1816#3,1817#1,1818#1,1819#3,1820#1,1821#1,1822#1,1823#3,1824#1,1825#1,1826#5,1827#3,1828#1,1829#3,1830#8,1831#3,1832#1,1833#3,1834#5,1835#1,1836#3,1837#3,1838#1,1839#3,1840#1,1841#3,1842#1,1843#1,1844#1,1845#8,1846#1,1847#1,1848#7,1849#1,1850#1,1851#3,1852#1,1853#1,1854#1,1855#1,1856#3,1857#1,1858#3,1859#1,1860#1,1861#589361,1862#1,1863#1,1864#2,1865#3,1866#4,1867#3,1868#1,1869#3,1870#3,1871#7,1872#1,1873#1,1874#1,1875#1,1876#1,1877#9,1878#7,1879#1,1880#1,1881#1,1882#3,1883#3,1884#1,1885#10,1886#8,1887#1,1888#1,1889#1,1890#1,1891#1,1892#1,1893#12,1894#1,1895#1,1896#1,1897#3,1898#1,1899#1,1900#5,1901#12,1902#1,1903#1,1904#1,1905#1,1906#1,1907#1,1908#1,1909#1,1910#3,1911#1,1912#1,1913#1,1914#7,1915#1,1916#5,1917#1,1918#1,1919#233919,1920#1,1921#3,1922#1,1923#1,1924#1,1925#3,1926#1,1927#3,1928#1,1929#1,1930#1,1931#1,1932#1,1933#1,1934#3,1935#1,1936#1,1937#3,1938#3,1939#1,1940#1,1941#3,1942#1,1943#1,1944#3,1945#1,1946#1,1947#7,1948#1,1949#3,1950#3,1951#1,1952#1,1953#1,1954#1,1955#9,1956#1,1957#1,1958#1,1959#1,1960#1,1961#1,1962#3,1963#1,1964#1,1965#2,1966#1,1967#3,1968#4,1969#1,1970#3,1971#1,1972#5,1973#1,1974#3,1975#1,1976#1,1977#5,1978#1,1979#8,1980#1,1981#1,1982#5,1983#1,1984#1,1985#1,1986#1,1987#1,1988#1,1989#1,1990#1,1991#1,1992#1,1993#1,1994#1,1995#3,1996#1,1997#1,1998#3",
"master_ver":
"0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0,197#0,198#0,199#0,200#0,201#0,202#0,203#0,204#0,205#0,206#0,207#0,208#0,209#0,210#0,211#0,212#0,213#0,214#0,215#0,216#0,217#0,218#0,219#0,220#0,221#0,222#0,223#0,224#0,225#0,226#0,227#0,228#0,229#0,230#0,231#0,232#0,233#0,234#0,235#0,236#0,237#0,238#0,239#0,240#0,241#0,242#0,243#0,244#0,245#0,246#0,247#0,248#0,249#0,250#0,251#0,252#0,253#0,254#0,255#0,256#0,257#0,258#0,259#0,260#0,261#0,262#0,263#0,264#0,265#0,266#0,267#0,268#0,269#0,270#0,271#0,272#0,273#0,274#0,275#0,276#0,277#0,278#0,279#0,280#0,281#0,282#0,283#0,284#0,285#0,286#0,287#0,288#0,289#0,290#0,291#0,292#0,293#0,294#0,295#0,296#0,297#0,298#0,299#0,300#0,301#0,302#0,303#0,304#0,305#0,306#0,307#0,308#0,309#0,310#0,311#0,312#0,313#0,314#0,315#0,316#0,317#0,318#0,319#0,320#0,321#0,322#0,323#0,324#0,325#0,326#0,327#0,328#0,329#0,330#0,331#0,332#0,333#0,334#0,335#0,336#0,337#0,338#0,339#0,340#0,341#0,342#0,343#0,344#0,345#0,346#0,347#0,348#0,349#0,350#0,351#0,352#0,353#0,354#0,355#0,356#0,357#0,358#0,359#0,360#0,361#0,362#0,363#0,364#0,365#0,366#0,367#0,368#0,369#0,370#0,371#0,372#0,373#0,374#0,375#0,376#0,377#0,378#0,379#0,380#0,381#0,382#0,383#0,384#0,385#0,386#0,387#0,388#0,389#0,390#0,391#0,392#0,393#0,394#0,395#0,396#0,397#0,398#0,399#0,400#0,401#0,402#0,403#0,404#0,405#0,406#0,407#0,408#0,409#0,410#0,411#0,412#0,413#0,414#0,415#0,416#0,417#0,418#0,419#0,420#0,421#0,422#0,423#0,424#0,425#0,426#0,427#0,428#0,429#0,430#0,431#0,432#0,433#0,434#0,435#0,436#0,437#0,438#0,439#0,440#0,441#0,442#0,443#0,444#0,445#0,446#0,447#0,448#0,449#0,450#0,451#0,452#0,453#0,454#0,455#0,456#0,457#0,458#0,459#0,460#0,461#0,462#0,463#0,464#0,465#0,466#0,467#0,468#0,469#0,470#0,471#0,472#0,473#0,474#0,475#0,476#0,477#0,478#0,479#0,480#0,481#0,482#0,483#0,484#0,485#0,486#0,487#0,488#0,489#0,490#0,491#0,492#0,493#0,494#0,495#0,496#0,497#0,498#0,499#0,500#0,501#0,502#0,503#0,504#0,505#0,506#0,507#0,508#0,509#0,510#0,511#0,512#0,513#0,514#0,515#0,516#0,517#0,518#0,519#0,520#0,521#0,522#0,523#0,524#0,525#0,526#0,527#0,528#0,529#0,530#0,531#0,532#0,533#0,534#0,535#0,536#0,537#0,538#0,539#0,540#0,541#0,542#0,543#0,544#0,545#0,546#0,547#0,548#0,549#0,550#0,551#0,552#0,553#0,554#0,555#0,556#0,557#0,558#0,559#0,560#0,561#0,562#0,563#0,564#0,565#0,566#0,567#0,568#0,569#0,570#0,571#0,572#0,573#0,574#0,575#0,576#0,577#0,578#0,579#0,580#0,581#0,582#0,583#0,584#0,585#0,586#0,587#0,588#0,589#0,590#0,591#0,592#0,593#0,594#0,595#0,596#0,597#0,598#0,599#0,600#0,601#0,602#0,603#0,604#0,605#0,606#0,607#0,608#0,609#0,610#0,611#0,612#0,613#0,614#0,615#0,616#0,617#0,618#0,619#0,620#0,621#0,622#0,623#0,624#0,625#0,626#0,627#0,628#0,629#0,630#0,631#0,632#0,633#0,634#0,635#0,636#0,637#0,638#0,639#0,640#0,641#0,642#0,643#0,644#0,645#0,646#0,647#0,648#0,649#0,650#0,651#0,652#0,653#0,654#0,655#0,656#0,657#0,658#0,659#0,660#0,661#0,662#0,663#0,664#0,665#0,666#0,667#0,668#0,669#0,670#0,671#0,672#0,673#0,674#0,675#0,676#0,677#0,678#0,679#0,680#0,681#0,682#0,683#0,684#0,685#0,686#0,687#0,688#0,689#0,690#0,691#0,692#0,693#0,694#0,695#0,696#0,697#0,698#0,699#0,700#0,701#0,702#0,703#0,704#0,705#0,706#0,707#0,708#0,709#0,710#0,711#0,712#0,713#0,714#0,715#0,716#0,717#0,718#0,719#0,720#0,721#0,722#0,723#0,724#0,725#0,726#0,727#0,728#0,729#0,730#0,731#0,732#0,733#0,734#0,735#0,736#0,737#0,738#0,739#0,740#0,741#0,742#0,743#0,744#0,745#0,746#0,747#0,748#0,749#0,750#0,751#0,752#0,753#0,754#0,755#0,756#0,757#0,758#0,759#0,760#0,761#0,762#0,763#0,764#0,765#0,766#0,767#0,768#0,769#0,770#0,771#0,772#0,773#0,774#0,775#0,776#0,777#0,778#0,779#0,780#0,781#0,782#0,783#0,784#0,785#0,786#0,787#0,788#0,789#0,790#0,791#0,792#0,793#0,794#0,795#0,796#0,797#0,798#0,799#0,800#0,801#0,802#0,803#0,804#0,805#0,806#0,807#0,808#0,809#0,810#0,811#0,812#0,813#0,814#0,815#0,816#0,817#0,818#0,819#0,820#0,821#0,822#0,823#0,824#0,825#0,826#0,827#0,828#0,829#0,830#0,831#0,832#0,833#0,834#0,835#0,836#0,837#0,838#0,839#0,840#0,841#0,842#0,843#0,844#0,845#0,846#0,847#0,848#0,849#0,850#0,851#0,852#0,853#0,854#0,855#0,856#0,857#0,858#0,859#0,860#0,861#0,862#0,863#0,864#0,865#0,866#0,867#0,868#0,869#0,870#0,871#0,872#0,873#0,874#0,875#0,876#0,877#0,878#0,879#0,880#0,881#0,882#0,883#0,884#0,885#0,886#0,887#0,888#0,889#0,890#0,891#0,892#0,893#0,894#0,895#0,896#0,897#0,898#0,899#0,900#0,901#0,902#0,903#0,904#0,905#0,906#0,907#0,908#0,909#0,910#0,911#0,912#0,913#0,914#0,915#0,916#0,917#0,918#0,919#0,920#0,921#0,922#0,923#0,924#0,925#0,926#0,927#0,928#0,929#0,930#0,931#0,932#0,933#0,934#0,935#0,936#0,937#0,938#0,939#0,940#0,941#0,942#0,943#0,944#0,945#0,946#0,947#0,948#0,949#0,950#0,951#0,952#0,953#0,954#0,955#0,956#0,957#0,958#0,959#0,960#0,961#0,962#0,963#0,964#0,965#0,966#0,967#0,968#0,969#0,970#0,971#0,972#0,973#0,974#0,975#0,976#0,977#0,978#0,979#0,980#0,981#0,982#0,983#0,984#0,985#0,986#0,987#0,988#0,989#0,990#0,991#0,992#0,993#0,994#0,995#0,996#0,997#0,998#0,999#0,1000#0,1001#0,1002#0,1003#0,1004#0,1005#0,1006#0,1007#0,1008#0,1009#0,1010#0,1011#0,1012#0,1013#0,1014#0,1015#0,1016#0,1017#0,1018#0,1019#0,1020#0,1021#0,1022#0,1023#0,1024#0,1025#0,1026#0,1027#0,1028#0,1029#0,1030#0,1031#0,1032#0,1033#0,1034#0,1035#0,1036#0,1037#0,1038#0,1039#0,1040#0,1041#0,1042#0,1043#0,1044#0,1045#0,1046#0,1047#0,1048#0,1049#0,1050#0,1051#0,1052#0,1053#0,1054#0,1055#0,1056#0,1057#0,1058#0,1059#0,1060#0,1061#0,1062#0,1063#0,1064#0,1065#0,1066#0,1067#0,1068#0,1069#0,1070#0,1071#0,1072#0,1073#0,1074#0,1075#0,1076#0,1077#0,1078#0,1079#0,1080#0,1081#0,1082#0,1083#0,1084#0,1085#0,1086#0,1087#0,1088#0,1089#0,1090#0,1091#0,1092#0,1093#0,1094#0,1095#0,1096#0,1097#0,1098#0,1099#0,1100#0,1101#0,1102#0,1103#0,1104#0,1105#0,1106#0,1107#0,1108#0,1109#0,1110#0,1111#0,1112#0,1113#0,1114#0,1115#0,1116#0,1117#0,1118#0,1119#0,1120#0,1121#0,1122#0,1123#0,1124#0,1125#0,1126#0,1127#0,1128#0,1129#0,1130#0,1131#0,1132#0,1133#0,1134#0,1135#0,1136#0,1137#0,1138#0,1139#0,1140#0,1141#0,1142#0,1143#0,1144#0,1145#0,1146#0,1147#0,1148#0,1149#0,1150#0,1151#0,1152#0,1153#0,1154#0,1155#0,1156#0,1157#0,1158#0,1159#0,1160#0,1161#0,1162#0,1163#0,1164#0,1165#0,1166#0,1167#0,1168#0,1169#0,1170#0,1171#0,1172#0,1173#0,1174#0,1175#0,1176#0,1177#0,1178#0,1179#0,1180#0,1181#0,1182#0,1183#0,1184#0,1185#0,1186#0,1187#0,1188#0,1189#0,1190#0,1191#0,1192#0,1193#0,1194#0,1195#0,1196#0,1197#0,1198#0,1199#0,1200#0,1201#0,1202#0,1203#0,1204#0,1205#0,1206#0,1207#0,1208#0,1209#0,1210#0,1211#0,1212#0,1213#0,1214#0,1215#0,1216#0,1217#0,1218#0,1219#0,1220#0,1221#0,1222#0,1223#0,1224#0,1225#0,1226#0,1227#0,1228#0,1229#0,1230#0,1231#0,1232#0,1233#0,1234#0,1235#0,1236#0,1237#0,1238#0,1239#0,1240#0,1241#0,1242#0,1243#0,1244#0,1245#0,1246#0,1247#0,1248#0,1249#0,1250#0,1251#0,1252#0,1253#0,1254#0,1255#0,1256#0,1257#0,1258#0,1259#0,1260#0,1261#0,1262#0,1263#0,1264#0,1265#0,1266#0,1267#0,1268#0,1269#0,1270#0,1271#0,1272#0,1273#0,1274#0,1275#0,1276#0,1277#0,1278#0,1279#0,1280#0,1281#0,1282#0,1283#0,1284#0,1285#0,1286#0,1287#0,1288#0,1289#0,1290#0,1291#0,1292#0,1293#0,1294#0,1295#0,1296#0,1297#0,1298#0,1299#0,1300#0,1301#0,1302#0,1303#0,1304#0,1305#0,1306#0,1307#0,1308#0,1309#0,1310#0,1311#0,1312#0,1313#0,1314#0,1315#0,1316#0,1317#0,1318#0,1319#0,1320#0,1321#0,1322#0,1323#0,1324#0,1325#0,1326#0,1327#0,1328#0,1329#0,1330#0,1331#0,1332#0,1333#0,1334#0,1335#0,1336#0,1337#0,1338#0,1339#0,1340#0,1341#0,1342#0,1343#0,1344#0,1345#0,1346#0,1347#0,1348#0,1349#0,1350#0,1351#0,1352#0,1353#0,1354#0,1355#0,1356#0,1357#0,1358#0,1359#0,1360#0,1361#0,1362#0,1363#0,1364#0,1365#0,1366#0,1367#0,1368#0,1369#0,1370#0,1371#0,1372#0,1373#0,1374#0,1375#0,1376#0,1377#0,1378#0,1379#0,1380#0,1381#0,1382#0,1383#0,1384#0,1385#0,1386#0,1387#0,1388#0,1389#0,1390#0,1391#0,1392#0,1393#0,1394#0,1395#0,1396#0,1397#0,1398#0,1399#0,1400#0,1401#0,1402#0,1403#0,1404#0,1405#0,1406#0,1407#0,1408#0,1409#0,1410#0,1411#0,1412#0,1413#0,1414#0,1415#0,1416#0,1417#0,1418#0,1419#0,1420#0,1421#0,1422#0,1423#0,1424#0,1425#0,1426#0,1427#0,1428#0,1429#0,1430#0,1431#0,1432#0,1433#0,1434#0,1435#0,1436#0,1437#0,1438#0,1439#0,1440#0,1441#0,1442#0,1443#0,1444#0,1445#0,1446#0,1447#0,1448#0,1449#0,1450#0,1451#0,1452#0,1453#0,1454#0,1455#0,1456#0,1457#0,1458#0,1459#0,1460#0,1461#0,1462#0,1463#0,1464#0,1465#0,1466#0,1467#0,1468#0,1469#0,1470#0,1471#0,1472#0,1473#0,1474#0,1475#0,1476#0,1477#0,1478#0,1479#0,1480#0,1481#0,1482#0,1483#0,1484#0,1485#0,1486#0,1487#0,1488#0,1489#0,1490#0,1491#0,1492#0,1493#0,1494#0,1495#0,1496#0,1497#0,1498#0,1499#0,1500#0,1501#0,1502#0,1503#0,1504#0,1505#0,1506#0,1507#0,1508#0,1509#0,1510#0,1511#0,1512#0,1513#0,1514#0,1515#0,1516#0,1517#0,1518#0,1519#0,1520#0,1521#0,1522#0,1523#0,1524#0,1525#0,1526#0,1527#0,1528#0,1529#0,1530#0,1531#0,1532#0,1533#0,1534#0,1535#0,1536#0,1537#0,1538#0,1539#0,1540#0,1541#0,1542#0,1543#0,1544#0,1545#0,1546#0,1547#0,1548#0,1549#0,1550#0,1551#0,1552#0,1553#0,1554#0,1555#0,1556#0,1557#0,1558#0,1559#0,1560#0,1561#0,1562#0,1563#0,1564#0,1565#0,1566#0,1567#0,1568#0,1569#0,1570#0,1571#0,1572#0,1573#0,1574#0,1575#0,1576#0,1577#0,1578#0,1579#0,1580#0,1581#0,1582#0,1583#0,1584#0,1585#0,1586#0,1587#0,1588#0,1589#0,1590#0,1591#0,1592#0,1593#0,1594#0,1595#0,1596#0,1597#0,1598#0,1599#0,1600#0,1601#0,1602#0,1603#0,1604#0,1605#0,1606#0,1607#0,1608#0,1609#0,1610#0,1611#0,1612#0,1613#0,1614#0,1615#0,1616#0,1617#0,1618#0,1619#0,1620#0,1621#0,1622#0,1623#0,1624#0,1625#0,1626#0,1627#0,1628#0,1629#0,1630#0,1631#0,1632#0,1633#0,1634#0,1635#0,1636#0,1637#0,1638#0,1639#0,1640#0,1641#0,1642#0,1643#0,1644#0,1645#0,1646#0,1647#0,1648#0,1649#0,1650#0,1651#0,1652#0,1653#0,1654#0,1655#0,1656#0,1657#0,1658#0,1659#0,1660#0,1661#0,1662#0,1663#0,1664#0,1665#0,1666#0,1667#0,1668#0,1669#0,1670#0,1671#0,1672#0,1673#0,1674#0,1675#0,1676#0,1677#0,1678#0,1679#0,1680#0,1681#0,1682#0,1683#0,1684#0,1685#0,1686#0,1687#0,1688#0,1689#0,1690#0,1691#0,1692#0,1693#0,1694#0,1695#0,1696#0,1697#0,1698#0,1699#0,1700#0,1701#0,1702#0,1703#0,1704#0,1705#0,1706#0,1707#0,1708#0,1709#0,1710#0,1711#0,1712#0,1713#0,1714#0,1715#0,1716#0,1717#0,1718#0,1719#0,1720#0,1721#0,1722#0,1723#0,1724#0,1725#0,1726#0,1727#0,1728#0,1729#0,1730#0,1731#0,1732#0,1733#0,1734#0,1735#0,1736#0,1737#0,1738#0,1739#0,1740#0,1741#0,1742#0,1743#0,1744#0,1745#0,1746#0,1747#0,1748#0,1749#0,1750#0,1751#0,1752#0,1753#0,1754#0,1755#0,1756#0,1757#0,1758#0,1759#0,1760#0,1761#0,1762#0,1763#0,1764#0,1765#0,1766#0,1767#0,1768#0,1769#0,1770#0,1771#0,1772#0,1773#0,1774#0,1775#0,1776#0,1777#0,1778#0,1779#0,1780#0,1781#0,1782#0,1783#0,1784#0,1785#0,1786#0,1787#0,1788#0,1789#0,1790#0,1791#0,1792#0,1793#0,1794#0,1795#0,1796#0,1797#0,1798#0,1799#0,1800#0,1801#0,1802#0,1803#0,1804#0,1805#0,1806#0,1807#0,1808#0,1809#0,1810#0,1811#0,1812#0,1813#0,1814#0,1815#0,1816#0,1817#0,1818#0,1819#0,1820#0,1821#0,1822#0,1823#0,1824#0,1825#0,1826#0,1827#0,1828#0,1829#0,1830#0,1831#0,1832#0,1833#0,1834#0,1835#0,1836#0,1837#0,1838#0,1839#0,1840#0,1841#0,1842#0,1843#0,1844#0,1845#0,1846#0,1847#0,1848#0,1849#0,1850#0,1851#0,1852#0,1853#0,1854#0,1855#0,1856#0,1857#0,1858#0,1859#0,1860#0,1861#0,1862#0,1863#0,1864#0,1865#0,1866#0,1867#0,1868#0,1869#0,1870#0,1871#0,1872#0,1873#0,1874#0,1875#0,1876#0,1877#0,1878#0,1879#0,1880#0,1881#0,1882#0,1883#0,1884#0,1885#0,1886#0,1887#0,1888#0,1889#0,1890#0,1891#0,1892#0,1893#0,1894#0,1895#0,1896#0,1897#0,1898#0,1899#0,1900#0,1901#0,1902#0,1903#0,1904#0,1905#0,1906#0,1907#0,1908#0,1909#0,1910#0,1911#0,1912#0,1913#0,1914#0,1915#0,1916#0,1917#0,1918#0,1919#0,1920#0,1921#0,1922#0,1923#0,1924#0,1925#0,1926#0,1927#0,1928#0,1929#0,1930#0,1931#0,1932#0,1933#0,1934#0,1935#0,1936#0,1937#0,1938#0,1939#0,1940#0,1941#0,1942#0,1943#0,1944#0,1945#0,1946#0,1947#0,1948#0,1949#0,1950#0,1951#0,1952#0,1953#0,1954#0,1955#0,1956#0,1957#0,1958#0,1959#0,1960#0,1961#0,1962#0,1963#0,1964#0,1965#0,1966#0,1967#0,1968#0,1969#0,1970#0,1971#0,1972#0,1973#0,1974#0,1975#0,1976#0,1977#0,1978#0,1979#0,1980#0,1981#0,1982#0,1983#0,1984#0,1985#0,1986#0,1987#0,1988#0,1989#0,1990#0,1991#0,1992#0,1993#0,1994#0,1995#0,1996#0,1997#0,1998#0",
"mtime": "2023-07-11T17:16:21.434940Z",
"creation_time": "2022-06-14T23:31:53.669280Z",
"max_marker":
"0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#,197#,198#,199#,200#,201#,202#,203#,204#,205#,206#,207#,208#,209#,210#,211#,212#,213#,214#,215#,216#,217#,218#,219#,220#,221#,222#,223#,224#,225#,226#,227#,228#,229#,230#,231#,232#,233#,234#,235#,236#,237#,238#,239#,240#,241#,242#,243#,244#,245#,246#,247#,248#,249#,250#,251#,252#,253#,254#,255#,256#,257#,258#,259#,260#,261#,262#,263#,264#,265#,266#,267#,268#,269#,270#,271#,272#,273#,274#,275#,276#,277#,278#,279#,280#,281#,282#,283#,284#,285#,286#,287#,288#,289#,290#,291#,292#,293#,294#,295#,296#,297#,298#,299#,300#,301#,302#,303#,304#,305#,306#,307#,308#,309#,310#,311#,312#,313#,314#,315#,316#,317#,318#,319#,320#,321#,322#,323#,324#,325#,326#,327#,328#,329#,330#,331#,332#,333#,334#,335#,336#,337#,338#,339#,340#,341#,342#,343#,344#,345#,346#,347#,348#,349#,350#,351#,352#,353#,354#,355#,356#,357#,358#,359#,360#,361#,362#,363#,364#,365#,366#,367#,368#,369#,370#,371#,372#,373#,374#,375#,376#,377#,378#,379#,380#,381#,382#,383#,384#,385#,386#,387#,388#,389#,390#,391#,392#,393#,394#,395#,396#,397#,398#,399#,400#,401#,402#,403#,404#,405#,406#,407#,408#,409#,410#,411#,412#,413#,414#,415#,416#,417#,418#,419#,420#,421#,422#,423#,424#,425#,426#,427#,428#,429#,430#,431#,432#,433#,434#,435#,436#,437#,438#,439#,440#,441#,442#,443#,444#,445#,446#,447#,448#,449#,450#,451#,452#,453#,454#,455#,456#,457#,458#,459#,460#,461#,462#,463#,464#,465#,466#,467#,468#,469#,470#,471#,472#,473#,474#,475#,476#,477#,478#,479#,480#,481#,482#,483#,484#,485#,486#,487#,488#,489#,490#,491#,492#,493#,494#,495#,496#,497#,498#,499#,500#,501#,502#,503#,504#,505#,506#,507#,508#,509#,510#,511#,512#,513#,514#,515#,516#,517#,518#,519#,520#,521#,522#,523#,524#,525#,526#,527#,528#,529#,530#,531#,532#,533#,534#,535#,536#,537#,538#,539#,540#,541#,542#,543#,544#,545#,546#,547#,548#,549#,550#,551#,552#,553#,554#,555#,556#,557#,558#,559#,560#,561#,562#,563#,564#,565#,566#,567#,568#,569#,570#,571#,572#,573#,574#,575#,576#,577#,578#,579#,580#,581#,582#,583#,584#,585#,586#,587#,588#,589#,590#,591#,592#,593#,594#,595#,596#,597#,598#,599#,600#,601#,602#,603#,604#,605#,606#,607#,608#,609#,610#,611#,612#,613#,614#,615#,616#,617#,618#,619#,620#,621#,622#,623#,624#,625#,626#,627#,628#,629#,630#,631#,632#,633#,634#,635#,636#,637#,638#,639#,640#,641#,642#,643#,644#,645#,646#,647#,648#,649#,650#,651#,652#,653#,654#,655#,656#,657#,658#,659#,660#,661#,662#,663#,664#,665#,666#,667#,668#,669#,670#,671#,672#,673#,674#,675#,676#,677#,678#,679#,680#,681#,682#,683#,684#,685#,686#,687#,688#,689#,690#,691#,692#,693#,694#,695#,696#,697#,698#,699#,700#,701#,702#,703#,704#,705#,706#,707#,708#,709#,710#,711#,712#,713#,714#,715#,716#,717#,718#,719#,720#,721#,722#,723#,724#,725#,726#,727#,728#,729#,730#,731#,732#,733#,734#,735#,736#,737#,738#,739#,740#,741#,742#,743#,744#,745#,746#,747#,748#,749#,750#,751#,752#,753#,754#,755#,756#,757#,758#,759#,760#,761#,762#,763#,764#,765#,766#,767#,768#,769#,770#,771#,772#,773#,774#,775#,776#,777#,778#,779#,780#,781#,782#,783#,784#,785#,786#,787#,788#,789#,790#,791#,792#,793#,794#,795#,796#,797#,798#,799#,800#,801#,802#,803#,804#,805#,806#,807#,808#,809#,810#,811#,812#,813#,814#,815#,816#,817#,818#,819#,820#,821#,822#,823#,824#,825#,826#,827#,828#,829#,830#,831#,832#,833#,834#,835#,836#,837#,838#,839#,840#,841#,842#,843#,844#,845#,846#,847#,848#,849#,850#,851#,852#,853#,854#,855#,856#,857#,858#,859#,860#,861#,862#,863#,864#,865#,866#,867#,868#,869#,870#,871#,872#,873#,874#,875#,876#,877#,878#,879#,880#,881#,882#,883#,884#,885#,886#,887#,888#,889#,890#,891#,892#,893#,894#,895#,896#,897#,898#,899#,900#,901#,902#,903#,904#,905#,906#,907#,908#,909#,910#,911#,912#,913#,914#,915#,916#,917#,918#,919#,920#,921#,922#,923#,924#,925#,926#,927#,928#,929#,930#,931#,932#,933#,934#,935#,936#,937#,938#,939#,940#,941#,942#,943#,944#,945#,946#,947#,948#,949#,950#,951#,952#,953#,954#,955#,956#,957#,958#,959#,960#,961#,962#,963#,964#,965#,966#,967#,968#,969#,970#,971#,972#,973#,974#,975#,976#,977#,978#,979#,980#,981#,982#,983#,984#,985#,986#,987#,988#,989#,990#,991#,992#,993#,994#,995#,996#,997#,998#,999#,1000#,1001#,1002#,1003#,1004#,1005#,1006#,1007#,1008#,1009#,1010#,1011#,1012#,1013#,1014#,1015#,1016#,1017#,1018#,1019#,1020#,1021#,1022#,1023#,1024#,1025#,1026#,1027#,1028#,1029#,1030#,1031#,1032#,1033#,1034#,1035#,1036#,1037#,1038#,1039#,1040#,1041#,1042#,1043#,1044#,1045#,1046#,1047#,1048#,1049#,1050#,1051#,1052#,1053#,1054#,1055#,1056#,1057#,1058#,1059#,1060#,1061#,1062#,1063#,1064#,1065#,1066#,1067#,1068#,1069#,1070#,1071#,1072#,1073#,1074#,1075#,1076#,1077#,1078#,1079#,1080#,1081#,1082#,1083#,1084#,1085#,1086#,1087#,1088#,1089#,1090#,1091#,1092#,1093#,1094#,1095#,1096#,1097#,1098#,1099#,1100#,1101#,1102#,1103#,1104#,1105#,1106#,1107#,1108#,1109#,1110#,1111#,1112#,1113#,1114#,1115#,1116#,1117#,1118#,1119#,1120#,1121#,1122#,1123#,1124#,1125#,1126#,1127#,1128#,1129#,1130#,1131#,1132#,1133#,1134#,1135#,1136#,1137#,1138#,1139#,1140#,1141#,1142#,1143#,1144#,1145#,1146#,1147#,1148#,1149#,1150#,1151#,1152#,1153#,1154#,1155#,1156#,1157#,1158#,1159#,1160#,1161#,1162#,1163#,1164#,1165#,1166#,1167#,1168#,1169#,1170#,1171#,1172#,1173#,1174#,1175#,1176#,1177#,1178#,1179#,1180#,1181#,1182#,1183#,1184#,1185#,1186#,1187#,1188#,1189#,1190#,1191#,1192#,1193#,1194#,1195#,1196#,1197#,1198#,1199#,1200#,1201#,1202#,1203#,1204#,1205#,1206#,1207#,1208#,1209#,1210#,1211#,1212#,1213#,1214#,1215#,1216#,1217#,1218#,1219#,1220#,1221#,1222#,1223#,1224#,1225#,1226#,1227#,1228#,1229#,1230#,1231#,1232#,1233#,1234#,1235#,1236#,1237#,1238#,1239#,1240#,1241#,1242#,1243#,1244#,1245#,1246#,1247#,1248#,1249#,1250#,1251#,1252#,1253#,1254#,1255#,1256#,1257#,1258#,1259#,1260#,1261#,1262#,1263#,1264#,1265#,1266#,1267#,1268#,1269#,1270#,1271#,1272#,1273#,1274#,1275#,1276#,1277#,1278#,1279#,1280#,1281#,1282#,1283#,1284#,1285#,1286#,1287#,1288#,1289#,1290#,1291#,1292#,1293#,1294#,1295#,1296#,1297#,1298#,1299#,1300#,1301#,1302#,1303#,1304#,1305#,1306#,1307#,1308#,1309#,1310#,1311#,1312#,1313#,1314#,1315#,1316#,1317#,1318#,1319#,1320#,1321#,1322#,1323#,1324#,1325#,1326#,1327#,1328#,1329#,1330#,1331#,1332#,1333#,1334#,1335#,1336#,1337#,1338#,1339#,1340#,1341#,1342#,1343#,1344#,1345#,1346#,1347#,1348#,1349#,1350#,1351#,1352#,1353#,1354#,1355#,1356#,1357#,1358#,1359#,1360#,1361#,1362#,1363#,1364#,1365#,1366#,1367#,1368#,1369#,1370#,1371#,1372#,1373#,1374#,1375#,1376#,1377#,1378#,1379#,1380#,1381#,1382#,1383#,1384#,1385#,1386#,1387#,1388#,1389#,1390#,1391#,1392#,1393#,1394#,1395#,1396#,1397#,1398#,1399#,1400#,1401#,1402#,1403#,1404#,1405#,1406#,1407#,1408#,1409#,1410#,1411#,1412#,1413#,1414#,1415#,1416#,1417#,1418#,1419#,1420#,1421#,1422#,1423#,1424#,1425#,1426#,1427#,1428#,1429#,1430#,1431#,1432#,1433#,1434#,1435#,1436#,1437#,1438#,1439#,1440#,1441#,1442#,1443#,1444#,1445#,1446#,1447#,1448#,1449#,1450#,1451#,1452#,1453#,1454#,1455#,1456#,1457#,1458#,1459#,1460#,1461#,1462#,1463#,1464#,1465#,1466#,1467#,1468#,1469#,1470#,1471#,1472#,1473#,1474#,1475#,1476#,1477#,1478#,1479#,1480#,1481#,1482#,1483#,1484#,1485#,1486#,1487#,1488#,1489#,1490#,1491#,1492#,1493#,1494#,1495#,1496#,1497#,1498#,1499#,1500#,1501#,1502#,1503#,1504#,1505#,1506#,1507#,1508#,1509#,1510#,1511#,1512#,1513#,1514#,1515#,1516#,1517#,1518#,1519#,1520#,1521#,1522#,1523#,1524#,1525#,1526#,1527#,1528#,1529#,1530#,1531#,1532#,1533#,1534#,1535#,1536#,1537#,1538#,1539#,1540#,1541#,1542#,1543#,1544#,1545#,1546#,1547#,1548#,1549#,1550#,1551#,1552#,1553#,1554#,1555#,1556#,1557#,1558#,1559#,1560#,1561#,1562#,1563#,1564#,1565#,1566#,1567#,1568#,1569#,1570#,1571#,1572#,1573#,1574#,1575#,1576#,1577#,1578#,1579#,1580#,1581#,1582#,1583#,1584#,1585#,1586#,1587#,1588#,1589#,1590#,1591#,1592#,1593#,1594#,1595#,1596#,1597#,1598#,1599#,1600#,1601#,1602#,1603#,1604#,1605#,1606#,1607#,1608#,1609#,1610#,1611#,1612#,1613#,1614#,1615#,1616#,1617#,1618#,1619#,1620#,1621#,1622#,1623#,1624#,1625#,1626#,1627#,1628#,1629#,1630#,1631#,1632#,1633#,1634#,1635#,1636#,1637#,1638#,1639#,1640#,1641#,1642#,1643#,1644#,1645#,1646#,1647#,1648#,1649#,1650#,1651#,1652#,1653#,1654#,1655#,1656#,1657#,1658#,1659#,1660#,1661#,1662#,1663#,1664#,1665#,1666#,1667#,1668#,1669#,1670#,1671#,1672#,1673#,1674#,1675#,1676#,1677#,1678#,1679#,1680#,1681#,1682#,1683#,1684#,1685#,1686#,1687#,1688#,1689#,1690#,1691#,1692#,1693#,1694#,1695#,1696#,1697#,1698#,1699#,1700#,1701#,1702#,1703#,1704#,1705#,1706#,1707#,1708#,1709#,1710#,1711#,1712#,1713#,1714#,1715#,1716#,1717#,1718#,1719#,1720#,1721#,1722#,1723#,1724#,1725#,1726#,1727#,1728#,1729#,1730#,1731#,1732#,1733#,1734#,1735#,1736#,1737#,1738#,1739#,1740#,1741#,1742#,1743#,1744#,1745#,1746#,1747#,1748#,1749#,1750#,1751#,1752#,1753#,1754#,1755#,1756#,1757#,1758#,1759#,1760#,1761#,1762#,1763#,1764#,1765#,1766#,1767#,1768#,1769#,1770#,1771#,1772#,1773#,1774#,1775#,1776#,1777#,1778#,1779#,1780#,1781#,1782#,1783#,1784#,1785#,1786#,1787#,1788#,1789#,1790#,1791#,1792#,1793#,1794#,1795#,1796#,1797#,1798#,1799#,1800#,1801#,1802#,1803#,1804#,1805#,1806#,1807#,1808#,1809#,1810#,1811#,1812#,1813#,1814#,1815#,1816#,1817#,1818#,1819#,1820#,1821#,1822#,1823#,1824#,1825#,1826#,1827#,1828#,1829#,1830#,1831#,1832#,1833#,1834#,1835#,1836#,1837#,1838#,1839#,1840#,1841#,1842#,1843#,1844#,1845#,1846#,1847#,1848#,1849#,1850#,1851#,1852#,1853#,1854#,1855#,1856#,1857#,1858#,1859#,1860#,1861#,1862#,1863#,1864#,1865#,1866#,1867#,1868#,1869#,1870#,1871#,1872#,1873#,1874#,1875#,1876#,1877#,1878#,1879#,1880#,1881#,1882#,1883#,1884#,1885#,1886#,1887#,1888#,1889#,1890#,1891#,1892#,1893#,1894#,1895#,1896#,1897#,1898#,1899#,1900#,1901#,1902#,1903#,1904#,1905#,1906#,1907#,1908#,1909#,1910#,1911#,1912#,1913#,1914#,1915#,1916#,1917#,1918#,1919#,1920#,1921#,1922#,1923#,1924#,1925#,1926#,1927#,1928#,1929#,1930#,1931#,1932#,1933#,1934#,1935#,1936#,1937#,1938#,1939#,1940#,1941#,1942#,1943#,1944#,1945#,1946#,1947#,1948#,1949#,1950#,1951#,1952#,1953#,1954#,1955#,1956#,1957#,1958#,1959#,1960#,1961#,1962#,1963#,1964#,1965#,1966#,1967#,1968#,1969#,1970#,1971#,1972#,1973#,1974#,1975#,1976#,1977#,1978#,1979#,1980#,1981#,1982#,1983#,1984#,1985#,1986#,1987#,1988#,1989#,1990#,1991#,1992#,1993#,1994#,1995#,1996#,1997#,1998#",
"usage": {
"rgw.main": {
"size": 209866840,
"size_actual": 209903616,
"size_utilized": 209866840,
"size_kb": 204949,
"size_kb_actual": 204984,
"size_kb_utilized": 204949,
"num_objects": 169433938
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 837,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 1,
"num_objects": 31
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
I am not sure what those objects are and why they have so many
shards, since actual readable objects are less than 10 . Any help with that
is really appreciated.
Hello,
This is a strange one for me. My ceph dashboard just stopped loading,
nothing but a white page. I don't see anything in the logs and on browser
side the only error I see is Failed to load resource:
net::ERR_CONTENT_LENGTH_MISMATCH in chrome and Uncaught SyntaxError: expected
expression, got end of script for Firefox on the
file main.ddd4de0999172734.js.
Nothing in the logs files, put mgr up to 20 on the log level. Any
suggestions?
Thanks,
Curt
Hi,
Quick question about cephadm and its logs. On my cluster I have every logs that goes to journald. But on each machine, I still have /var/log/ceph/cephadm.log that is alive.
Is there a way to make cephadm log to journald instead of a file? If yes did I miss it on the documentation? Of if not is there any reason to log into a file while everything else logs to journald?
Thanks
Luis Domingues
Proton AG
Hi,
I'm using Pacific v16.2.10 container image, deployed by cephadm.
I used to manually build config file for rgw, deploy rgw, put config file in place
and restart rgw. It works fine.
Now, I'd like to put rgw config into config db. I tried with client.rgw, but the config
is not taken by rgw. Also "config show" doesn't work. It always says "no config state".
```
# ceph orch ps | grep rgw
rgw.qa.ceph-1.hzfrwq ceph-1 10.250.80.100:80 running (10m) 10m ago 53m 51.4M - 16.2.10 32214388de9d 13169a213bc5
# ceph config get client.rgw | grep frontends
client.rgw basic rgw_frontends beast port=8086 *
# ceph config show rgw.qa.ceph-1.hzfrwq
Error ENOENT: no config state for daemon rgw.qa.ceph-1.hzfrwq
# ceph config show client.rgw.qa.ceph-1.hzfrwq
Error ENOENT: no config state for daemon client.rgw.qa.ceph-1.hzfrwq
# radosgw-admin --show-config -n client.rgw.qa.ceph-1.hzfrwq | grep frontends
rgw_frontends = beast port=7480
```
Any clues what I am missing here?
Thanks!
Tony
Currently I am checking usage on ssd drives with
ceph osd df| egrep 'CLASS|ssd'
I have a use % between 48% and 57%, and assume that with a node failure 1/3 (only using 3x repl.) of this 57% needs to be able to migrate and added to a different node.
Is there a better way of checking this (on old Nautilus)?