- ceph-users - lists.ceph.io

veeam backup on rgw - error - op->ERRORHANDLER: err_no=-2 new_err_no=-2
by xadhoom76＠gmail.com 01 Aug '23

01 Aug '23

Hi , we have random error with rgw during a backup from veeam . Daemons go in error state. Where we can find the appropriate logs about it ? I just find out something related to this -1788> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s getting op 0 -1787> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj verifying requester -1786> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj normalizing buckets and tenants -1785> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj init permissions -1784> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj recalculating target -1783> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj reading permissions -1782> 2023-07-31T19:51:21.173+0000 7f04567d3700 1 req 10656715914266436796 0.004000044s op->ERRORHANDLER: err_no=-2 new_err_no=-2 -1781> 2023-07-31T19:51:21.173+0000 7f04567d3700 2 req 10656715914266436796 0.004000044s s3:get_obj op status=0 -1780> 2023-07-31T19:51:21.173+0000 7f04567d3700 2 req 10656715914266436796 0.004000044s s3:get_obj http status=404 -1779> 2023-07-31T19:51:21.173+0000 7f04567d3700 1 ====== req done req=0x7f03aa67a710 op status=0 http_status=404 latency=0.004000044s ====== -1778> 2023-07-31T19:51:21.173+0000 7f04567d3700 1 beast: 0x7f03aa67a710: 172.16.199.11 - veeam90 [31/Jul/2023:19:51:21.169 +0000] "GET /veeam90/Veeam/Backup/veeam90/Clients/%7Bd14cd688-57b4-4809-a1d9-14cafd191b11%7D/15fb2d73-11f5-4ef2-8d17-c9848eb40c93/Metadata/Lock/create.checkpoint/lock HTTP/1.1" 404 207 - "APN/1.0 Veeam/1.0 Backup/12.0" - latency=0.004000044s Best regards

1 0

RGW multi-site recovery
by Gregory O'Neill 31 Jul '23

31 Jul '23

Hello, I have 3 Luminous (12.2.12) clusters setup with rgw multi-site. One cluster recently suffered data loss from PGs lost to bad SSDs. We're looking at how to recover from this scenario. The other two clusters are healthy. Will RGW sync the missing data? Do buckets and objects need to be restored using rclone? From `radosgw-admin sync error list`, these are the error types: $ cat sync-error-list | jq -r '.[].entries[].info.message' | sort -h | uniq -c | sort -h 2 failed to sync bucket instance: (13) Permission denied 3 failed to sync object(13) Permission denied 100 failed to sync bucket instance: (125) Operation canceled 239 failed to sync bucket instance: (5) Input/output error 404 failed to read remote metadata entry: (11) Resource temporarily unavailable 2148 failed to sync object(5) Input/output error -- Gregory O’Neill

1 0

LARGE_OMAP_OBJECTS warning and bucket has lot of unknown objects and 1999 shards.
by Uday Bhaskar Jalagam 31 Jul '23

31 Jul '23

Hello Everyone , I am getting [WRN] LARGE_OMAP_OBJECTS: 18 large omap objects warning in one of my clusters . I see one of the buckets has a huge number of shards 1999 and "num_objects": 221185360 when I check bucket stats using radosgw-admin bucket stats . However I see only 8 files when I actually do list bucket using python boto3. I am not sure what those objects are missing somewhere . # radosgw-admin bucket stats --bucket=epbucket { "bucket": "epbucket", "num_shards": 1999, "tenant": "", "zonegroup": "ac361c18-7420-4739-8e92-73109fc8ec80", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "f3e47d62-b9ff-4f38-b863-d7bfc155d698.35744173.268063", "marker": "f3e47d62-b9ff-4f38-b863-d7bfc155d698.35744173.1225", "index_type": "Normal", "owner": "user", "ver": "0#3,1#1,2#1,3#9,4#1,5#3,6#1,7#3,8#1,9#8,10#7,11#1,12#3,13#1,14#1,15#1,16#1,17#1,18#5,19#1,20#1,21#3,22#3,23#1,24#3,25#1,26#1,27#1,28#3,29#3,30#2,31#3,32#2,33#1,34#3,35#1,36#3,37#1,38#3,39#1,40#3,41#1,42#1,43#1,44#1,45#1,46#3,47#1,48#3,49#1,50#3,51#1,52#1,53#1,54#1,55#1,56#3,57#1,58#1,59#5,60#1,61#1,62#1,63#3,64#1,65#1,66#3,67#1,68#5,69#1,70#1,71#3,72#1,73#3,74#2,75#3,76#1,77#3,78#1,79#1,80#3,81#3,82#3,83#3,84#1,85#1,86#3,87#3,88#8,89#1,90#1,91#1,92#5,93#7,94#1,95#1,96#3,97#1,98#5,99#1,100#1,101#3,102#5,103#3,104#1,105#1,106#1,107#1,108#1,109#239050,110#1,111#1,112#1,113#5,114#1,115#1,116#1,117#3,118#3,119#9,120#1,121#1,122#3,123#5,124#3,125#3,126#1,127#1,128#1,129#3,130#3,131#3,132#3,133#5,134#1,135#1,136#1,137#1,138#5,139#3,140#1,141#1,142#3,143#3,144#1,145#1,146#3,147#3,148#3,149#1,150#3,151#1,152#1,153#3,154#7,155#1,156#5,157#3,158#3,159#3,160#5,161#3,162#3,163#3,164#3,165#1,166#1,167#5,168#3,169#1,170#1,171#1,172#4,173#1,174#12,175#1,176#3,177#3,178#3,179#1,180#3,181#33,182#1,183#1,184#1,185#1,186#3,187#3,188#11,189#3,190#5,191#3,192#1,193#5,194#3,195#2,196#3,197#1,198#3,199#1,200#7,201#3,202#7,203#1,204#3,205#3,206#1,207#1,208#3,209#1,210#7,211#3,212#2,213#9,214#1,215#3,216#1,217#1,218#7,219#1,220#1,221#5,222#1,223#1,224#3,225#1,226#1,227#3,228#3,229#1,230#5,231#5,232#1,233#3,234#1,235#1,236#5,237#1,238#1,239#3,240#7,241#3,242#1,243#3,244#1,245#1,246#3,247#1,248#3,249#3,250#1,251#1,252#3,253#1,254#3,255#1,256#3,257#1,258#1,259#7,260#9,261#3,262#1,263#1,264#3,265#1,266#1,267#5,268#1,269#1,270#1,271#1,272#1,273#3,274#1,275#1,276#1,277#3,278#1,279#7,280#1,281#238803,282#3,283#1,284#1,285#1,286#1,287#3,288#1,289#1,290#1,291#16,292#3,293#3,294#1,295#1,296#5,297#3,298#3,299#7,300#5,301#5,302#1,303#1,304#1,305#3,306#3,307#5,308#1,309#1,310#3,311#3,312#1,313#1,314#5,315#3,316#7,317#1,318#1,319#3,320#3,321#3,322#1,323#9,324#3,325#1,326#5,327#1,328#3,329#9,330#3,331#1,332#3,333#3,334#3,335#3,336#3,337#9,338#1,339#1,340#1,341#3,342#3,343#3,344#2,345#3,346#3,347#1,348#3,349#3,350#5,351#1,352#5,353#8,354#5,355#1,356#3,357#1,358#3,359#1,360#1,361#1,362#1,363#5,364#1,365#1,366#1,367#1,368#1,369#1,370#3,371#3,372#3,373#3,374#1,375#3,376#5,377#8,378#1,379#1,380#1,381#1,382#3,383#3,384#1,385#3,386#1,387#1,388#1,389#2,390#1,391#1,392#1,393#1,394#5,395#5,396#1,397#5,398#3,399#1,400#1,401#7,402#1,403#5,404#1,405#1,406#3,407#3,408#3,409#1,410#1,411#1,412#1,413#1,414#1,415#1,416#1,417#1,418#3,419#3,420#1,421#5,422#5,423#5,424#1,425#1,426#1,427#1,428#9,429#1,430#3,431#1,432#3,433#3,434#1,435#1,436#3,437#5,438#3,439#3,440#1,441#1,442#1,443#1,444#1,445#5,446#3,447#9,448#1,449#1,450#1,451#1,452#1,453#3,454#1,455#1,456#3,457#5,458#5,459#3,460#9,461#1,462#1,463#1,464#1,465#1,466#1,467#3,468#1,469#3,470#8,471#1,472#1,473#1,474#7,475#3,476#1,477#11,478#10,479#1,480#1,481#1,482#3,483#1,484#7,485#1,486#1,487#3,488#8,489#1,490#1,491#8,492#3,493#1,494#3,495#5,496#1,497#1,498#1,499#3,500#1,501#5,502#1,503#1,504#3,505#1,506#1,507#1,508#3,509#3,510#3,511#1,512#7,513#3,514#1,515#3,516#5,517#1,518#3,519#1,520#3,521#3,522#3,523#5,524#3,525#3,526#1,527#1,528#1,529#1,530#1,531#3,532#1,533#3,534#5,535#5,536#1,537#3,538#3,539#5,540#1,541#1,542#1,543#1,544#7,545#3,546#1,547#1,548#1,549#3,550#1,551#1,552#3,553#1,554#3,555#3,556#1,557#3,558#5,559#3,560#3,561#8,562#1,563#3,564#7,565#3,566#5,567#1,568#3,569#3,570#3,571#3,572#5,573#1,574#1,575#5,576#5,577#3,578#7,579#1,580#1,581#5,582#1,583#3,584#1,585#1,586#9,587#9,588#3,589#1,590#1,591#3,592#3,593#1,594#10,595#3,596#1,597#3,598#1,599#1,600#1,601#3,602#3,603#1,604#3,605#5,606#1,607#1,608#1,609#10,610#3,611#264711,612#3,613#5,614#3,615#3,616#1,617#1,618#7,619#3,620#3,621#1,622#3,623#1,624#3,625#1,626#3,627#1,628#1,629#3,630#1,631#2,632#233833,633#1,634#1,635#5,636#1,637#1,638#1,639#1,640#3,641#1,642#1,643#3,644#1,645#1,646#10,647#1,648#1,649#1,650#3,651#1,652#1,653#5,654#1,655#7,656#7,657#1,658#1,659#5,660#3,661#3,662#1,663#3,664#1,665#1,666#3,667#1,668#7,669#1,670#3,671#3,672#1,673#1,674#1,675#1,676#1,677#3,678#1,679#11,680#1,681#7,682#5,683#1,684#1,685#1,686#1,687#3,688#1,689#1,690#3,691#3,692#1,693#1,694#1,695#1,696#1,697#1,698#1,699#5,700#1,701#1,702#1,703#1,704#3,705#1,706#8,707#3,708#3,709#8,710#1,711#2,712#3,713#1,714#1,715#10,716#3,717#1,718#1,719#3,720#1,721#7,722#3,723#1,724#5,725#1,726#5,727#1,728#3,729#1,730#3,731#1,732#7,733#3,734#3,735#1,736#5,737#1,738#5,739#1,740#1,741#1,742#1,743#1,744#1,745#3,746#1,747#1,748#1,749#5,750#12,751#1,752#9,753#7,754#5,755#1,756#1,757#1,758#1,759#5,760#1,761#3,762#3,763#3,764#1,765#1,766#11,767#5,768#3,769#3,770#5,771#3,772#1,773#1,774#1,775#1,776#7,777#5,778#3,779#3,780#2,781#5,782#1,783#1,784#5,785#1,786#3,787#1,788#3,789#5,790#10,791#3,792#7,793#1,794#1,795#3,796#1,797#3,798#1,799#1,800#1,801#1,802#3,803#1,804#3,805#1,806#1,807#1,808#1,809#3,810#1,811#3,812#2,813#1,814#1,815#1,816#3,817#1,818#3,819#1,820#3,821#1,822#1,823#1,824#5,825#5,826#3,827#1,828#3,829#3,830#1,831#1,832#3,833#5,834#3,835#3,836#3,837#7,838#7,839#3,840#1,841#3,842#1,843#1,844#1,845#12,846#1,847#3,848#1,849#1,850#1,851#1,852#1,853#5,854#3,855#1,856#7,857#8,858#3,859#1,860#3,861#1,862#1,863#1,864#1,865#1,866#1,867#5,868#3,869#1,870#3,871#3,872#7,873#9,874#7,875#1,876#1,877#3,878#1,879#3,880#1,881#1,882#1,883#1,884#3,885#7,886#1,887#1,888#1,889#3,890#1,891#1,892#1,893#1,894#3,895#1,896#1,897#3,898#1,899#3,900#3,901#1,902#1,903#1,904#1,905#1,906#1,907#3,908#3,909#3,910#3,911#1,912#1,913#5,914#3,915#3,916#1,917#1,918#1,919#1,920#3,921#1,922#7,923#2,924#3,925#1,926#3,927#1,928#7,929#1,930#5,931#1,932#3,933#3,934#1,935#1,936#3,937#1,938#3,939#1,940#5,941#1,942#5,943#1,944#1,945#12,946#1,947#5,948#1,949#1,950#3,951#7,952#1,953#1,954#1,955#3,956#1,957#5,958#3,959#3,960#3,961#3,962#3,963#3,964#3,965#3,966#1,967#3,968#3,969#3,970#1,971#18,972#3,973#3,974#1,975#1,976#1,977#3,978#1,979#1,980#1,981#1,982#3,983#8,984#5,985#1,986#8,987#3,988#1,989#1,990#1,991#1,992#1,993#5,994#1,995#373523,996#3,997#3,998#3,999#1,1000#1,1001#1,1002#1,1003#1,1004#3,1005#1,1006#1,1007#1,1008#1,1009#7,1010#1,1011#3,1012#1,1013#1,1014#1,1015#1,1016#3,1017#3,1018#1,1019#5,1020#1,1021#1,1022#5,1023#3,1024#1,1025#1,1026#7,1027#3,1028#3,1029#3,1030#1,1031#3,1032#1,1033#1,1034#3,1035#9,1036#1,1037#1,1038#3,1039#5,1040#5,1041#5,1042#3,1043#1,1044#3,1045#7,1046#1,1047#3,1048#3,1049#1,1050#1,1051#1,1052#3,1053#1,1054#3,1055#3,1056#5,1057#8,1058#3,1059#1,1060#3,1061#3,1062#7,1063#1,1064#1,1065#7,1066#1,1067#1,1068#1,1069#3,1070#3,1071#3,1072#3,1073#7,1074#1,1075#3,1076#1,1077#3,1078#1,1079#1,1080#5,1081#1,1082#1,1083#1,1084#3,1085#1,1086#3,1087#1,1088#1,1089#1,1090#1,1091#1,1092#5,1093#1,1094#5,1095#3,1096#1,1097#1,1098#3,1099#1,1100#1,1101#5,1102#1,1103#5,1104#3,1105#1,1106#1,1107#3,1108#3,1109#1,1110#1,1111#1,1112#1,1113#3,1114#1,1115#3,1116#3,1117#1,1118#1,1119#1,1120#1,1121#1,1122#3,1123#1,1124#5,1125#1,1126#1,1127#5,1128#9,1129#5,1130#3,1131#1,1132#7,1133#1,1134#3,1135#2,1136#1,1137#1,1138#1,1139#1,1140#1,1141#1,1142#5,1143#3,1144#1,1145#5,1146#10,1147#235181,1148#1,1149#3,1150#1,1151#3,1152#1,1153#3,1154#7,1155#1,1156#1,1157#3,1158#1,1159#1,1160#3,1161#5,1162#1,1163#1,1164#1,1165#1,1166#5,1167#1,1168#3,1169#1,1170#1,1171#3,1172#3,1173#2,1174#1,1175#1,1176#1,1177#1,1178#1,1179#1,1180#5,1181#5,1182#1,1183#1,1184#1,1185#3,1186#1,1187#1,1188#1,1189#1,1190#5,1191#9,1192#3,1193#8,1194#1,1195#1,1196#1,1197#1,1198#1,1199#1,1200#3,1201#1,1202#1,1203#1,1204#3,1205#1,1206#1,1207#1,1208#1,1209#1,1210#1,1211#1,1212#1,1213#2,1214#1,1215#1,1216#3,1217#3,1218#3,1219#3,1220#1,1221#3,1222#1,1223#1,1224#3,1225#17,1226#3,1227#3,1228#239050,1229#1,1230#3,1231#3,1232#1,1233#5,1234#3,1235#1,1236#1,1237#1,1238#1,1239#1,1240#1,1241#5,1242#1,1243#1,1244#3,1245#5,1246#1,1247#3,1248#1,1249#3,1250#1,1251#3,1252#3,1253#1,1254#1,1255#1,1256#1,1257#1,1258#1,1259#3,1260#5,1261#3,1262#3,1263#1,1264#1,1265#1,1266#1,1267#1,1268#5,1269#3,1270#3,1271#5,1272#1,1273#1,1274#1,1275#5,1276#3,1277#1,1278#8,1279#5,1280#5,1281#1,1282#5,1283#1,1284#1,1285#5,1286#1,1287#3,1288#1,1289#1,1290#3,1291#1,1292#1,1293#3,1294#1,1295#1,1296#1,1297#9,1298#3,1299#1,1300#1,1301#1,1302#3,1303#1,1304#3,1305#3,1306#1,1307#3,1308#1,1309#1,1310#3,1311#3,1312#1,1313#3,1314#1,1315#1,1316#1,1317#5,1318#3,1319#1,1320#1,1321#1,1322#1,1323#3,1324#1,1325#3,1326#8,1327#1,1328#7,1329#6,1330#1,1331#1,1332#1,1333#3,1334#1,1335#1,1336#1,1337#1,1338#1,1339#1,1340#3,1341#3,1342#1,1343#3,1344#5,1345#1,1346#1,1347#1,1348#3,1349#3,1350#1,1351#1,1352#1,1353#1,1354#8,1355#1,1356#3,1357#3,1358#3,1359#3,1360#1,1361#1,1362#1,1363#5,1364#3,1365#1,1366#1,1367#3,1368#5,1369#5,1370#1,1371#1,1372#1,1373#3,1374#1,1375#1,1376#1,1377#7,1378#1,1379#3,1380#1,1381#1,1382#1,1383#1,1384#1,1385#6,1386#1,1387#1,1388#3,1389#1,1390#1,1391#1,1392#1,1393#3,1394#1,1395#3,1396#1,1397#1,1398#1,1399#1,1400#5,1401#3,1402#5,1403#3,1404#5,1405#5,1406#3,1407#1,1408#3,1409#1,1410#3,1411#1,1412#1,1413#3,1414#1,1415#1,1416#5,1417#3,1418#12,1419#3,1420#1,1421#1,1422#3,1423#1,1424#5,1425#7,1426#3,1427#1,1428#1,1429#1,1430#3,1431#1,1432#3,1433#1,1434#1,1435#1,1436#1,1437#1,1438#3,1439#1,1440#1,1441#3,1442#3,1443#1,1444#1,1445#1,1446#1,1447#1,1448#3,1449#3,1450#1,1451#1,1452#3,1453#1,1454#1,1455#1,1456#3,1457#1,1458#5,1459#3,1460#1,1461#3,1462#1,1463#1,1464#1,1465#1,1466#1,1467#1,1468#6,1469#1,1470#1,1471#1,1472#1,1473#1,1474#1,1475#1,1476#5,1477#1,1478#3,1479#8,1480#1,1481#3,1482#1,1483#1,1484#7,1485#1,1486#1,1487#3,1488#3,1489#1,1490#1,1491#1,1492#1,1493#1,1494#1,1495#5,1496#5,1497#1,1498#8,1499#1,1500#3,1501#3,1502#1,1503#1,1504#5,1505#1,1506#1,1507#3,1508#3,1509#3,1510#1,1511#3,1512#3,1513#1,1514#1,1515#1,1516#1,1517#3,1518#1,1519#3,1520#1,1521#1,1522#3,1523#1,1524#7,1525#1,1526#1,1527#1,1528#1,1529#1,1530#3,1531#1,1532#1,1533#1,1534#3,1535#5,1536#3,1537#3,1538#7,1539#3,1540#1,1541#1,1542#1,1543#3,1544#1,1545#1,1546#1,1547#1,1548#1,1549#7,1550#1,1551#1,1552#3,1553#1,1554#5,1555#3,1556#1,1557#1,1558#1,1559#5,1560#1,1561#1,1562#1,1563#1,1564#5,1565#3,1566#3,1567#3,1568#1,1569#3,1570#1,1571#3,1572#1,1573#13,1574#1,1575#1,1576#1,1577#1,1578#1,1579#1,1580#1,1581#1,1582#3,1583#1,1584#1,1585#1,1586#3,1587#1,1588#3,1589#5,1590#10,1591#3,1592#3,1593#1,1594#7,1595#1,1596#3,1597#5,1598#1,1599#1,1600#5,1601#3,1602#1,1603#1,1604#1,1605#3,1606#1,1607#1,1608#5,1609#15,1610#3,1611#3,1612#3,1613#1,1614#3,1615#1,1616#1,1617#1,1618#3,1619#3,1620#1,1621#3,1622#7,1623#1,1624#5,1625#1,1626#3,1627#3,1628#1,1629#1,1630#1,1631#3,1632#1,1633#1,1634#1,1635#1,1636#1,1637#3,1638#1,1639#3,1640#1,1641#1,1642#1,1643#1,1644#3,1645#1,1646#1,1647#3,1648#5,1649#1,1650#1,1651#1,1652#7,1653#3,1654#1,1655#1,1656#13,1657#1,1658#3,1659#3,1660#9,1661#1,1662#7,1663#3,1664#3,1665#1,1666#1,1667#1,1668#1,1669#1,1670#1,1671#16,1672#3,1673#1,1674#3,1675#1,1676#1,1677#1,1678#3,1679#1,1680#1,1681#1,1682#1,1683#1,1684#3,1685#1,1686#1,1687#1,1688#5,1689#5,1690#1,1691#3,1692#1,1693#1,1694#3,1695#3,1696#1,1697#5,1698#1,1699#3,1700#1,1701#1,1702#1,1703#3,1704#1,1705#5,1706#1,1707#9,1708#1,1709#1,1710#3,1711#3,1712#1,1713#5,1714#1,1715#1,1716#1,1717#1,1718#1,1719#1,1720#3,1721#3,1722#1,1723#1,1724#3,1725#1,1726#1,1727#1,1728#1,1729#3,1730#1,1731#3,1732#3,1733#10,1734#3,1735#3,1736#3,1737#3,1738#1,1739#1,1740#5,1741#1,1742#1,1743#1,1744#3,1745#1,1746#5,1747#3,1748#5,1749#1,1750#3,1751#9,1752#3,1753#1,1754#1,1755#3,1756#1,1757#3,1758#1,1759#3,1760#3,1761#3,1762#3,1763#1,1764#1,1765#5,1766#1,1767#1,1768#5,1769#5,1770#1,1771#1,1772#7,1773#7,1774#5,1775#3,1776#3,1777#1,1778#3,1779#1,1780#1,1781#3,1782#1,1783#1,1784#3,1785#1,1786#1,1787#1,1788#1,1789#1,1790#3,1791#3,1792#3,1793#1,1794#1,1795#1,1796#3,1797#1,1798#8,1799#3,1800#5,1801#9,1802#4,1803#3,1804#3,1805#1,1806#1,1807#3,1808#3,1809#3,1810#1,1811#5,1812#3,1813#5,1814#10,1815#1,1816#3,1817#1,1818#1,1819#3,1820#1,1821#1,1822#1,1823#3,1824#1,1825#1,1826#5,1827#3,1828#1,1829#3,1830#8,1831#3,1832#1,1833#3,1834#5,1835#1,1836#3,1837#3,1838#1,1839#3,1840#1,1841#3,1842#1,1843#1,1844#1,1845#8,1846#1,1847#1,1848#7,1849#1,1850#1,1851#3,1852#1,1853#1,1854#1,1855#1,1856#3,1857#1,1858#3,1859#1,1860#1,1861#589361,1862#1,1863#1,1864#2,1865#3,1866#4,1867#3,1868#1,1869#3,1870#3,1871#7,1872#1,1873#1,1874#1,1875#1,1876#1,1877#9,1878#7,1879#1,1880#1,1881#1,1882#3,1883#3,1884#1,1885#10,1886#8,1887#1,1888#1,1889#1,1890#1,1891#1,1892#1,1893#12,1894#1,1895#1,1896#1,1897#3,1898#1,1899#1,1900#5,1901#12,1902#1,1903#1,1904#1,1905#1,1906#1,1907#1,1908#1,1909#1,1910#3,1911#1,1912#1,1913#1,1914#7,1915#1,1916#5,1917#1,1918#1,1919#233919,1920#1,1921#3,1922#1,1923#1,1924#1,1925#3,1926#1,1927#3,1928#1,1929#1,1930#1,1931#1,1932#1,1933#1,1934#3,1935#1,1936#1,1937#3,1938#3,1939#1,1940#1,1941#3,1942#1,1943#1,1944#3,1945#1,1946#1,1947#7,1948#1,1949#3,1950#3,1951#1,1952#1,1953#1,1954#1,1955#9,1956#1,1957#1,1958#1,1959#1,1960#1,1961#1,1962#3,1963#1,1964#1,1965#2,1966#1,1967#3,1968#4,1969#1,1970#3,1971#1,1972#5,1973#1,1974#3,1975#1,1976#1,1977#5,1978#1,1979#8,1980#1,1981#1,1982#5,1983#1,1984#1,1985#1,1986#1,1987#1,1988#1,1989#1,1990#1,1991#1,1992#1,1993#1,1994#1,1995#3,1996#1,1997#1,1998#3", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0,197#0,198#0,199#0,200#0,201#0,202#0,203#0,204#0,205#0,206#0,207#0,208#0,209#0,210#0,211#0,212#0,213#0,214#0,215#0,216#0,217#0,218#0,219#0,220#0,221#0,222#0,223#0,224#0,225#0,226#0,227#0,228#0,229#0,230#0,231#0,232#0,233#0,234#0,235#0,236#0,237#0,238#0,239#0,240#0,241#0,242#0,243#0,244#0,245#0,246#0,247#0,248#0,249#0,250#0,251#0,252#0,253#0,254#0,255#0,256#0,257#0,258#0,259#0,260#0,261#0,262#0,263#0,264#0,265#0,266#0,267#0,268#0,269#0,270#0,271#0,272#0,273#0,274#0,275#0,276#0,277#0,278#0,279#0,280#0,281#0,282#0,283#0,284#0,285#0,286#0,287#0,288#0,289#0,290#0,291#0,292#0,293#0,294#0,295#0,296#0,297#0,298#0,299#0,300#0,301#0,302#0,303#0,304#0,305#0,306#0,307#0,308#0,309#0,310#0,311#0,312#0,313#0,314#0,315#0,316#0,317#0,318#0,319#0,320#0,321#0,322#0,323#0,324#0,325#0,326#0,327#0,328#0,329#0,330#0,331#0,332#0,333#0,334#0,335#0,336#0,337#0,338#0,339#0,340#0,341#0,342#0,343#0,344#0,345#0,346#0,347#0,348#0,349#0,350#0,351#0,352#0,353#0,354#0,355#0,356#0,357#0,358#0,359#0,360#0,361#0,362#0,363#0,364#0,365#0,366#0,367#0,368#0,369#0,370#0,371#0,372#0,373#0,374#0,375#0,376#0,377#0,378#0,379#0,380#0,381#0,382#0,383#0,384#0,385#0,386#0,387#0,388#0,389#0,390#0,391#0,392#0,393#0,394#0,395#0,396#0,397#0,398#0,399#0,400#0,401#0,402#0,403#0,404#0,405#0,406#0,407#0,408#0,409#0,410#0,411#0,412#0,413#0,414#0,415#0,416#0,417#0,418#0,419#0,420#0,421#0,422#0,423#0,424#0,425#0,426#0,427#0,428#0,429#0,430#0,431#0,432#0,433#0,434#0,435#0,436#0,437#0,438#0,439#0,440#0,441#0,442#0,443#0,444#0,445#0,446#0,447#0,448#0,449#0,450#0,451#0,452#0,453#0,454#0,455#0,456#0,457#0,458#0,459#0,460#0,461#0,462#0,463#0,464#0,465#0,466#0,467#0,468#0,469#0,470#0,471#0,472#0,473#0,474#0,475#0,476#0,477#0,478#0,479#0,480#0,481#0,482#0,483#0,484#0,485#0,486#0,487#0,488#0,489#0,490#0,491#0,492#0,493#0,494#0,495#0,496#0,497#0,498#0,499#0,500#0,501#0,502#0,503#0,504#0,505#0,506#0,507#0,508#0,509#0,510#0,511#0,512#0,513#0,514#0,515#0,516#0,517#0,518#0,519#0,520#0,521#0,522#0,523#0,524#0,525#0,526#0,527#0,528#0,529#0,530#0,531#0,532#0,533#0,534#0,535#0,536#0,537#0,538#0,539#0,540#0,541#0,542#0,543#0,544#0,545#0,546#0,547#0,548#0,549#0,550#0,551#0,552#0,553#0,554#0,555#0,556#0,557#0,558#0,559#0,560#0,561#0,562#0,563#0,564#0,565#0,566#0,567#0,568#0,569#0,570#0,571#0,572#0,573#0,574#0,575#0,576#0,577#0,578#0,579#0,580#0,581#0,582#0,583#0,584#0,585#0,586#0,587#0,588#0,589#0,590#0,591#0,592#0,593#0,594#0,595#0,596#0,597#0,598#0,599#0,600#0,601#0,602#0,603#0,604#0,605#0,606#0,607#0,608#0,609#0,610#0,611#0,612#0,613#0,614#0,615#0,616#0,617#0,618#0,619#0,620#0,621#0,622#0,623#0,624#0,625#0,626#0,627#0,628#0,629#0,630#0,631#0,632#0,633#0,634#0,635#0,636#0,637#0,638#0,639#0,640#0,641#0,642#0,643#0,644#0,645#0,646#0,647#0,648#0,649#0,650#0,651#0,652#0,653#0,654#0,655#0,656#0,657#0,658#0,659#0,660#0,661#0,662#0,663#0,664#0,665#0,666#0,667#0,668#0,669#0,670#0,671#0,672#0,673#0,674#0,675#0,676#0,677#0,678#0,679#0,680#0,681#0,682#0,683#0,684#0,685#0,686#0,687#0,688#0,689#0,690#0,691#0,692#0,693#0,694#0,695#0,696#0,697#0,698#0,699#0,700#0,701#0,702#0,703#0,704#0,705#0,706#0,707#0,708#0,709#0,710#0,711#0,712#0,713#0,714#0,715#0,716#0,717#0,718#0,719#0,720#0,721#0,722#0,723#0,724#0,725#0,726#0,727#0,728#0,729#0,730#0,731#0,732#0,733#0,734#0,735#0,736#0,737#0,738#0,739#0,740#0,741#0,742#0,743#0,744#0,745#0,746#0,747#0,748#0,749#0,750#0,751#0,752#0,753#0,754#0,755#0,756#0,757#0,758#0,759#0,760#0,761#0,762#0,763#0,764#0,765#0,766#0,767#0,768#0,769#0,770#0,771#0,772#0,773#0,774#0,775#0,776#0,777#0,778#0,779#0,780#0,781#0,782#0,783#0,784#0,785#0,786#0,787#0,788#0,789#0,790#0,791#0,792#0,793#0,794#0,795#0,796#0,797#0,798#0,799#0,800#0,801#0,802#0,803#0,804#0,805#0,806#0,807#0,808#0,809#0,810#0,811#0,812#0,813#0,814#0,815#0,816#0,817#0,818#0,819#0,820#0,821#0,822#0,823#0,824#0,825#0,826#0,827#0,828#0,829#0,830#0,831#0,832#0,833#0,834#0,835#0,836#0,837#0,838#0,839#0,840#0,841#0,842#0,843#0,844#0,845#0,846#0,847#0,848#0,849#0,850#0,851#0,852#0,853#0,854#0,855#0,856#0,857#0,858#0,859#0,860#0,861#0,862#0,863#0,864#0,865#0,866#0,867#0,868#0,869#0,870#0,871#0,872#0,873#0,874#0,875#0,876#0,877#0,878#0,879#0,880#0,881#0,882#0,883#0,884#0,885#0,886#0,887#0,888#0,889#0,890#0,891#0,892#0,893#0,894#0,895#0,896#0,897#0,898#0,899#0,900#0,901#0,902#0,903#0,904#0,905#0,906#0,907#0,908#0,909#0,910#0,911#0,912#0,913#0,914#0,915#0,916#0,917#0,918#0,919#0,920#0,921#0,922#0,923#0,924#0,925#0,926#0,927#0,928#0,929#0,930#0,931#0,932#0,933#0,934#0,935#0,936#0,937#0,938#0,939#0,940#0,941#0,942#0,943#0,944#0,945#0,946#0,947#0,948#0,949#0,950#0,951#0,952#0,953#0,954#0,955#0,956#0,957#0,958#0,959#0,960#0,961#0,962#0,963#0,964#0,965#0,966#0,967#0,968#0,969#0,970#0,971#0,972#0,973#0,974#0,975#0,976#0,977#0,978#0,979#0,980#0,981#0,982#0,983#0,984#0,985#0,986#0,987#0,988#0,989#0,990#0,991#0,992#0,993#0,994#0,995#0,996#0,997#0,998#0,999#0,1000#0,1001#0,1002#0,1003#0,1004#0,1005#0,1006#0,1007#0,1008#0,1009#0,1010#0,1011#0,1012#0,1013#0,1014#0,1015#0,1016#0,1017#0,1018#0,1019#0,1020#0,1021#0,1022#0,1023#0,1024#0,1025#0,1026#0,1027#0,1028#0,1029#0,1030#0,1031#0,1032#0,1033#0,1034#0,1035#0,1036#0,1037#0,1038#0,1039#0,1040#0,1041#0,1042#0,1043#0,1044#0,1045#0,1046#0,1047#0,1048#0,1049#0,1050#0,1051#0,1052#0,1053#0,1054#0,1055#0,1056#0,1057#0,1058#0,1059#0,1060#0,1061#0,1062#0,1063#0,1064#0,1065#0,1066#0,1067#0,1068#0,1069#0,1070#0,1071#0,1072#0,1073#0,1074#0,1075#0,1076#0,1077#0,1078#0,1079#0,1080#0,1081#0,1082#0,1083#0,1084#0,1085#0,1086#0,1087#0,1088#0,1089#0,1090#0,1091#0,1092#0,1093#0,1094#0,1095#0,1096#0,1097#0,1098#0,1099#0,1100#0,1101#0,1102#0,1103#0,1104#0,1105#0,1106#0,1107#0,1108#0,1109#0,1110#0,1111#0,1112#0,1113#0,1114#0,1115#0,1116#0,1117#0,1118#0,1119#0,1120#0,1121#0,1122#0,1123#0,1124#0,1125#0,1126#0,1127#0,1128#0,1129#0,1130#0,1131#0,1132#0,1133#0,1134#0,1135#0,1136#0,1137#0,1138#0,1139#0,1140#0,1141#0,1142#0,1143#0,1144#0,1145#0,1146#0,1147#0,1148#0,1149#0,1150#0,1151#0,1152#0,1153#0,1154#0,1155#0,1156#0,1157#0,1158#0,1159#0,1160#0,1161#0,1162#0,1163#0,1164#0,1165#0,1166#0,1167#0,1168#0,1169#0,1170#0,1171#0,1172#0,1173#0,1174#0,1175#0,1176#0,1177#0,1178#0,1179#0,1180#0,1181#0,1182#0,1183#0,1184#0,1185#0,1186#0,1187#0,1188#0,1189#0,1190#0,1191#0,1192#0,1193#0,1194#0,1195#0,1196#0,1197#0,1198#0,1199#0,1200#0,1201#0,1202#0,1203#0,1204#0,1205#0,1206#0,1207#0,1208#0,1209#0,1210#0,1211#0,1212#0,1213#0,1214#0,1215#0,1216#0,1217#0,1218#0,1219#0,1220#0,1221#0,1222#0,1223#0,1224#0,1225#0,1226#0,1227#0,1228#0,1229#0,1230#0,1231#0,1232#0,1233#0,1234#0,1235#0,1236#0,1237#0,1238#0,1239#0,1240#0,1241#0,1242#0,1243#0,1244#0,1245#0,1246#0,1247#0,1248#0,1249#0,1250#0,1251#0,1252#0,1253#0,1254#0,1255#0,1256#0,1257#0,1258#0,1259#0,1260#0,1261#0,1262#0,1263#0,1264#0,1265#0,1266#0,1267#0,1268#0,1269#0,1270#0,1271#0,1272#0,1273#0,1274#0,1275#0,1276#0,1277#0,1278#0,1279#0,1280#0,1281#0,1282#0,1283#0,1284#0,1285#0,1286#0,1287#0,1288#0,1289#0,1290#0,1291#0,1292#0,1293#0,1294#0,1295#0,1296#0,1297#0,1298#0,1299#0,1300#0,1301#0,1302#0,1303#0,1304#0,1305#0,1306#0,1307#0,1308#0,1309#0,1310#0,1311#0,1312#0,1313#0,1314#0,1315#0,1316#0,1317#0,1318#0,1319#0,1320#0,1321#0,1322#0,1323#0,1324#0,1325#0,1326#0,1327#0,1328#0,1329#0,1330#0,1331#0,1332#0,1333#0,1334#0,1335#0,1336#0,1337#0,1338#0,1339#0,1340#0,1341#0,1342#0,1343#0,1344#0,1345#0,1346#0,1347#0,1348#0,1349#0,1350#0,1351#0,1352#0,1353#0,1354#0,1355#0,1356#0,1357#0,1358#0,1359#0,1360#0,1361#0,1362#0,1363#0,1364#0,1365#0,1366#0,1367#0,1368#0,1369#0,1370#0,1371#0,1372#0,1373#0,1374#0,1375#0,1376#0,1377#0,1378#0,1379#0,1380#0,1381#0,1382#0,1383#0,1384#0,1385#0,1386#0,1387#0,1388#0,1389#0,1390#0,1391#0,1392#0,1393#0,1394#0,1395#0,1396#0,1397#0,1398#0,1399#0,1400#0,1401#0,1402#0,1403#0,1404#0,1405#0,1406#0,1407#0,1408#0,1409#0,1410#0,1411#0,1412#0,1413#0,1414#0,1415#0,1416#0,1417#0,1418#0,1419#0,1420#0,1421#0,1422#0,1423#0,1424#0,1425#0,1426#0,1427#0,1428#0,1429#0,1430#0,1431#0,1432#0,1433#0,1434#0,1435#0,1436#0,1437#0,1438#0,1439#0,1440#0,1441#0,1442#0,1443#0,1444#0,1445#0,1446#0,1447#0,1448#0,1449#0,1450#0,1451#0,1452#0,1453#0,1454#0,1455#0,1456#0,1457#0,1458#0,1459#0,1460#0,1461#0,1462#0,1463#0,1464#0,1465#0,1466#0,1467#0,1468#0,1469#0,1470#0,1471#0,1472#0,1473#0,1474#0,1475#0,1476#0,1477#0,1478#0,1479#0,1480#0,1481#0,1482#0,1483#0,1484#0,1485#0,1486#0,1487#0,1488#0,1489#0,1490#0,1491#0,1492#0,1493#0,1494#0,1495#0,1496#0,1497#0,1498#0,1499#0,1500#0,1501#0,1502#0,1503#0,1504#0,1505#0,1506#0,1507#0,1508#0,1509#0,1510#0,1511#0,1512#0,1513#0,1514#0,1515#0,1516#0,1517#0,1518#0,1519#0,1520#0,1521#0,1522#0,1523#0,1524#0,1525#0,1526#0,1527#0,1528#0,1529#0,1530#0,1531#0,1532#0,1533#0,1534#0,1535#0,1536#0,1537#0,1538#0,1539#0,1540#0,1541#0,1542#0,1543#0,1544#0,1545#0,1546#0,1547#0,1548#0,1549#0,1550#0,1551#0,1552#0,1553#0,1554#0,1555#0,1556#0,1557#0,1558#0,1559#0,1560#0,1561#0,1562#0,1563#0,1564#0,1565#0,1566#0,1567#0,1568#0,1569#0,1570#0,1571#0,1572#0,1573#0,1574#0,1575#0,1576#0,1577#0,1578#0,1579#0,1580#0,1581#0,1582#0,1583#0,1584#0,1585#0,1586#0,1587#0,1588#0,1589#0,1590#0,1591#0,1592#0,1593#0,1594#0,1595#0,1596#0,1597#0,1598#0,1599#0,1600#0,1601#0,1602#0,1603#0,1604#0,1605#0,1606#0,1607#0,1608#0,1609#0,1610#0,1611#0,1612#0,1613#0,1614#0,1615#0,1616#0,1617#0,1618#0,1619#0,1620#0,1621#0,1622#0,1623#0,1624#0,1625#0,1626#0,1627#0,1628#0,1629#0,1630#0,1631#0,1632#0,1633#0,1634#0,1635#0,1636#0,1637#0,1638#0,1639#0,1640#0,1641#0,1642#0,1643#0,1644#0,1645#0,1646#0,1647#0,1648#0,1649#0,1650#0,1651#0,1652#0,1653#0,1654#0,1655#0,1656#0,1657#0,1658#0,1659#0,1660#0,1661#0,1662#0,1663#0,1664#0,1665#0,1666#0,1667#0,1668#0,1669#0,1670#0,1671#0,1672#0,1673#0,1674#0,1675#0,1676#0,1677#0,1678#0,1679#0,1680#0,1681#0,1682#0,1683#0,1684#0,1685#0,1686#0,1687#0,1688#0,1689#0,1690#0,1691#0,1692#0,1693#0,1694#0,1695#0,1696#0,1697#0,1698#0,1699#0,1700#0,1701#0,1702#0,1703#0,1704#0,1705#0,1706#0,1707#0,1708#0,1709#0,1710#0,1711#0,1712#0,1713#0,1714#0,1715#0,1716#0,1717#0,1718#0,1719#0,1720#0,1721#0,1722#0,1723#0,1724#0,1725#0,1726#0,1727#0,1728#0,1729#0,1730#0,1731#0,1732#0,1733#0,1734#0,1735#0,1736#0,1737#0,1738#0,1739#0,1740#0,1741#0,1742#0,1743#0,1744#0,1745#0,1746#0,1747#0,1748#0,1749#0,1750#0,1751#0,1752#0,1753#0,1754#0,1755#0,1756#0,1757#0,1758#0,1759#0,1760#0,1761#0,1762#0,1763#0,1764#0,1765#0,1766#0,1767#0,1768#0,1769#0,1770#0,1771#0,1772#0,1773#0,1774#0,1775#0,1776#0,1777#0,1778#0,1779#0,1780#0,1781#0,1782#0,1783#0,1784#0,1785#0,1786#0,1787#0,1788#0,1789#0,1790#0,1791#0,1792#0,1793#0,1794#0,1795#0,1796#0,1797#0,1798#0,1799#0,1800#0,1801#0,1802#0,1803#0,1804#0,1805#0,1806#0,1807#0,1808#0,1809#0,1810#0,1811#0,1812#0,1813#0,1814#0,1815#0,1816#0,1817#0,1818#0,1819#0,1820#0,1821#0,1822#0,1823#0,1824#0,1825#0,1826#0,1827#0,1828#0,1829#0,1830#0,1831#0,1832#0,1833#0,1834#0,1835#0,1836#0,1837#0,1838#0,1839#0,1840#0,1841#0,1842#0,1843#0,1844#0,1845#0,1846#0,1847#0,1848#0,1849#0,1850#0,1851#0,1852#0,1853#0,1854#0,1855#0,1856#0,1857#0,1858#0,1859#0,1860#0,1861#0,1862#0,1863#0,1864#0,1865#0,1866#0,1867#0,1868#0,1869#0,1870#0,1871#0,1872#0,1873#0,1874#0,1875#0,1876#0,1877#0,1878#0,1879#0,1880#0,1881#0,1882#0,1883#0,1884#0,1885#0,1886#0,1887#0,1888#0,1889#0,1890#0,1891#0,1892#0,1893#0,1894#0,1895#0,1896#0,1897#0,1898#0,1899#0,1900#0,1901#0,1902#0,1903#0,1904#0,1905#0,1906#0,1907#0,1908#0,1909#0,1910#0,1911#0,1912#0,1913#0,1914#0,1915#0,1916#0,1917#0,1918#0,1919#0,1920#0,1921#0,1922#0,1923#0,1924#0,1925#0,1926#0,1927#0,1928#0,1929#0,1930#0,1931#0,1932#0,1933#0,1934#0,1935#0,1936#0,1937#0,1938#0,1939#0,1940#0,1941#0,1942#0,1943#0,1944#0,1945#0,1946#0,1947#0,1948#0,1949#0,1950#0,1951#0,1952#0,1953#0,1954#0,1955#0,1956#0,1957#0,1958#0,1959#0,1960#0,1961#0,1962#0,1963#0,1964#0,1965#0,1966#0,1967#0,1968#0,1969#0,1970#0,1971#0,1972#0,1973#0,1974#0,1975#0,1976#0,1977#0,1978#0,1979#0,1980#0,1981#0,1982#0,1983#0,1984#0,1985#0,1986#0,1987#0,1988#0,1989#0,1990#0,1991#0,1992#0,1993#0,1994#0,1995#0,1996#0,1997#0,1998#0", "mtime": "2023-07-11T17:16:21.434940Z", "creation_time": "2022-06-14T23:31:53.669280Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#,197#,198#,199#,200#,201#,202#,203#,204#,205#,206#,207#,208#,209#,210#,211#,212#,213#,214#,215#,216#,217#,218#,219#,220#,221#,222#,223#,224#,225#,226#,227#,228#,229#,230#,231#,232#,233#,234#,235#,236#,237#,238#,239#,240#,241#,242#,243#,244#,245#,246#,247#,248#,249#,250#,251#,252#,253#,254#,255#,256#,257#,258#,259#,260#,261#,262#,263#,264#,265#,266#,267#,268#,269#,270#,271#,272#,273#,274#,275#,276#,277#,278#,279#,280#,281#,282#,283#,284#,285#,286#,287#,288#,289#,290#,291#,292#,293#,294#,295#,296#,297#,298#,299#,300#,301#,302#,303#,304#,305#,306#,307#,308#,309#,310#,311#,312#,313#,314#,315#,316#,317#,318#,319#,320#,321#,322#,323#,324#,325#,326#,327#,328#,329#,330#,331#,332#,333#,334#,335#,336#,337#,338#,339#,340#,341#,342#,343#,344#,345#,346#,347#,348#,349#,350#,351#,352#,353#,354#,355#,356#,357#,358#,359#,360#,361#,362#,363#,364#,365#,366#,367#,368#,369#,370#,371#,372#,373#,374#,375#,376#,377#,378#,379#,380#,381#,382#,383#,384#,385#,386#,387#,388#,389#,390#,391#,392#,393#,394#,395#,396#,397#,398#,399#,400#,401#,402#,403#,404#,405#,406#,407#,408#,409#,410#,411#,412#,413#,414#,415#,416#,417#,418#,419#,420#,421#,422#,423#,424#,425#,426#,427#,428#,429#,430#,431#,432#,433#,434#,435#,436#,437#,438#,439#,440#,441#,442#,443#,444#,445#,446#,447#,448#,449#,450#,451#,452#,453#,454#,455#,456#,457#,458#,459#,460#,461#,462#,463#,464#,465#,466#,467#,468#,469#,470#,471#,472#,473#,474#,475#,476#,477#,478#,479#,480#,481#,482#,483#,484#,485#,486#,487#,488#,489#,490#,491#,492#,493#,494#,495#,496#,497#,498#,499#,500#,501#,502#,503#,504#,505#,506#,507#,508#,509#,510#,511#,512#,513#,514#,515#,516#,517#,518#,519#,520#,521#,522#,523#,524#,525#,526#,527#,528#,529#,530#,531#,532#,533#,534#,535#,536#,537#,538#,539#,540#,541#,542#,543#,544#,545#,546#,547#,548#,549#,550#,551#,552#,553#,554#,555#,556#,557#,558#,559#,560#,561#,562#,563#,564#,565#,566#,567#,568#,569#,570#,571#,572#,573#,574#,575#,576#,577#,578#,579#,580#,581#,582#,583#,584#,585#,586#,587#,588#,589#,590#,591#,592#,593#,594#,595#,596#,597#,598#,599#,600#,601#,602#,603#,604#,605#,606#,607#,608#,609#,610#,611#,612#,613#,614#,615#,616#,617#,618#,619#,620#,621#,622#,623#,624#,625#,626#,627#,628#,629#,630#,631#,632#,633#,634#,635#,636#,637#,638#,639#,640#,641#,642#,643#,644#,645#,646#,647#,648#,649#,650#,651#,652#,653#,654#,655#,656#,657#,658#,659#,660#,661#,662#,663#,664#,665#,666#,667#,668#,669#,670#,671#,672#,673#,674#,675#,676#,677#,678#,679#,680#,681#,682#,683#,684#,685#,686#,687#,688#,689#,690#,691#,692#,693#,694#,695#,696#,697#,698#,699#,700#,701#,702#,703#,704#,705#,706#,707#,708#,709#,710#,711#,712#,713#,714#,715#,716#,717#,718#,719#,720#,721#,722#,723#,724#,725#,726#,727#,728#,729#,730#,731#,732#,733#,734#,735#,736#,737#,738#,739#,740#,741#,742#,743#,744#,745#,746#,747#,748#,749#,750#,751#,752#,753#,754#,755#,756#,757#,758#,759#,760#,761#,762#,763#,764#,765#,766#,767#,768#,769#,770#,771#,772#,773#,774#,775#,776#,777#,778#,779#,780#,781#,782#,783#,784#,785#,786#,787#,788#,789#,790#,791#,792#,793#,794#,795#,796#,797#,798#,799#,800#,801#,802#,803#,804#,805#,806#,807#,808#,809#,810#,811#,812#,813#,814#,815#,816#,817#,818#,819#,820#,821#,822#,823#,824#,825#,826#,827#,828#,829#,830#,831#,832#,833#,834#,835#,836#,837#,838#,839#,840#,841#,842#,843#,844#,845#,846#,847#,848#,849#,850#,851#,852#,853#,854#,855#,856#,857#,858#,859#,860#,861#,862#,863#,864#,865#,866#,867#,868#,869#,870#,871#,872#,873#,874#,875#,876#,877#,878#,879#,880#,881#,882#,883#,884#,885#,886#,887#,888#,889#,890#,891#,892#,893#,894#,895#,896#,897#,898#,899#,900#,901#,902#,903#,904#,905#,906#,907#,908#,909#,910#,911#,912#,913#,914#,915#,916#,917#,918#,919#,920#,921#,922#,923#,924#,925#,926#,927#,928#,929#,930#,931#,932#,933#,934#,935#,936#,937#,938#,939#,940#,941#,942#,943#,944#,945#,946#,947#,948#,949#,950#,951#,952#,953#,954#,955#,956#,957#,958#,959#,960#,961#,962#,963#,964#,965#,966#,967#,968#,969#,970#,971#,972#,973#,974#,975#,976#,977#,978#,979#,980#,981#,982#,983#,984#,985#,986#,987#,988#,989#,990#,991#,992#,993#,994#,995#,996#,997#,998#,999#,1000#,1001#,1002#,1003#,1004#,1005#,1006#,1007#,1008#,1009#,1010#,1011#,1012#,1013#,1014#,1015#,1016#,1017#,1018#,1019#,1020#,1021#,1022#,1023#,1024#,1025#,1026#,1027#,1028#,1029#,1030#,1031#,1032#,1033#,1034#,1035#,1036#,1037#,1038#,1039#,1040#,1041#,1042#,1043#,1044#,1045#,1046#,1047#,1048#,1049#,1050#,1051#,1052#,1053#,1054#,1055#,1056#,1057#,1058#,1059#,1060#,1061#,1062#,1063#,1064#,1065#,1066#,1067#,1068#,1069#,1070#,1071#,1072#,1073#,1074#,1075#,1076#,1077#,1078#,1079#,1080#,1081#,1082#,1083#,1084#,1085#,1086#,1087#,1088#,1089#,1090#,1091#,1092#,1093#,1094#,1095#,1096#,1097#,1098#,1099#,1100#,1101#,1102#,1103#,1104#,1105#,1106#,1107#,1108#,1109#,1110#,1111#,1112#,1113#,1114#,1115#,1116#,1117#,1118#,1119#,1120#,1121#,1122#,1123#,1124#,1125#,1126#,1127#,1128#,1129#,1130#,1131#,1132#,1133#,1134#,1135#,1136#,1137#,1138#,1139#,1140#,1141#,1142#,1143#,1144#,1145#,1146#,1147#,1148#,1149#,1150#,1151#,1152#,1153#,1154#,1155#,1156#,1157#,1158#,1159#,1160#,1161#,1162#,1163#,1164#,1165#,1166#,1167#,1168#,1169#,1170#,1171#,1172#,1173#,1174#,1175#,1176#,1177#,1178#,1179#,1180#,1181#,1182#,1183#,1184#,1185#,1186#,1187#,1188#,1189#,1190#,1191#,1192#,1193#,1194#,1195#,1196#,1197#,1198#,1199#,1200#,1201#,1202#,1203#,1204#,1205#,1206#,1207#,1208#,1209#,1210#,1211#,1212#,1213#,1214#,1215#,1216#,1217#,1218#,1219#,1220#,1221#,1222#,1223#,1224#,1225#,1226#,1227#,1228#,1229#,1230#,1231#,1232#,1233#,1234#,1235#,1236#,1237#,1238#,1239#,1240#,1241#,1242#,1243#,1244#,1245#,1246#,1247#,1248#,1249#,1250#,1251#,1252#,1253#,1254#,1255#,1256#,1257#,1258#,1259#,1260#,1261#,1262#,1263#,1264#,1265#,1266#,1267#,1268#,1269#,1270#,1271#,1272#,1273#,1274#,1275#,1276#,1277#,1278#,1279#,1280#,1281#,1282#,1283#,1284#,1285#,1286#,1287#,1288#,1289#,1290#,1291#,1292#,1293#,1294#,1295#,1296#,1297#,1298#,1299#,1300#,1301#,1302#,1303#,1304#,1305#,1306#,1307#,1308#,1309#,1310#,1311#,1312#,1313#,1314#,1315#,1316#,1317#,1318#,1319#,1320#,1321#,1322#,1323#,1324#,1325#,1326#,1327#,1328#,1329#,1330#,1331#,1332#,1333#,1334#,1335#,1336#,1337#,1338#,1339#,1340#,1341#,1342#,1343#,1344#,1345#,1346#,1347#,1348#,1349#,1350#,1351#,1352#,1353#,1354#,1355#,1356#,1357#,1358#,1359#,1360#,1361#,1362#,1363#,1364#,1365#,1366#,1367#,1368#,1369#,1370#,1371#,1372#,1373#,1374#,1375#,1376#,1377#,1378#,1379#,1380#,1381#,1382#,1383#,1384#,1385#,1386#,1387#,1388#,1389#,1390#,1391#,1392#,1393#,1394#,1395#,1396#,1397#,1398#,1399#,1400#,1401#,1402#,1403#,1404#,1405#,1406#,1407#,1408#,1409#,1410#,1411#,1412#,1413#,1414#,1415#,1416#,1417#,1418#,1419#,1420#,1421#,1422#,1423#,1424#,1425#,1426#,1427#,1428#,1429#,1430#,1431#,1432#,1433#,1434#,1435#,1436#,1437#,1438#,1439#,1440#,1441#,1442#,1443#,1444#,1445#,1446#,1447#,1448#,1449#,1450#,1451#,1452#,1453#,1454#,1455#,1456#,1457#,1458#,1459#,1460#,1461#,1462#,1463#,1464#,1465#,1466#,1467#,1468#,1469#,1470#,1471#,1472#,1473#,1474#,1475#,1476#,1477#,1478#,1479#,1480#,1481#,1482#,1483#,1484#,1485#,1486#,1487#,1488#,1489#,1490#,1491#,1492#,1493#,1494#,1495#,1496#,1497#,1498#,1499#,1500#,1501#,1502#,1503#,1504#,1505#,1506#,1507#,1508#,1509#,1510#,1511#,1512#,1513#,1514#,1515#,1516#,1517#,1518#,1519#,1520#,1521#,1522#,1523#,1524#,1525#,1526#,1527#,1528#,1529#,1530#,1531#,1532#,1533#,1534#,1535#,1536#,1537#,1538#,1539#,1540#,1541#,1542#,1543#,1544#,1545#,1546#,1547#,1548#,1549#,1550#,1551#,1552#,1553#,1554#,1555#,1556#,1557#,1558#,1559#,1560#,1561#,1562#,1563#,1564#,1565#,1566#,1567#,1568#,1569#,1570#,1571#,1572#,1573#,1574#,1575#,1576#,1577#,1578#,1579#,1580#,1581#,1582#,1583#,1584#,1585#,1586#,1587#,1588#,1589#,1590#,1591#,1592#,1593#,1594#,1595#,1596#,1597#,1598#,1599#,1600#,1601#,1602#,1603#,1604#,1605#,1606#,1607#,1608#,1609#,1610#,1611#,1612#,1613#,1614#,1615#,1616#,1617#,1618#,1619#,1620#,1621#,1622#,1623#,1624#,1625#,1626#,1627#,1628#,1629#,1630#,1631#,1632#,1633#,1634#,1635#,1636#,1637#,1638#,1639#,1640#,1641#,1642#,1643#,1644#,1645#,1646#,1647#,1648#,1649#,1650#,1651#,1652#,1653#,1654#,1655#,1656#,1657#,1658#,1659#,1660#,1661#,1662#,1663#,1664#,1665#,1666#,1667#,1668#,1669#,1670#,1671#,1672#,1673#,1674#,1675#,1676#,1677#,1678#,1679#,1680#,1681#,1682#,1683#,1684#,1685#,1686#,1687#,1688#,1689#,1690#,1691#,1692#,1693#,1694#,1695#,1696#,1697#,1698#,1699#,1700#,1701#,1702#,1703#,1704#,1705#,1706#,1707#,1708#,1709#,1710#,1711#,1712#,1713#,1714#,1715#,1716#,1717#,1718#,1719#,1720#,1721#,1722#,1723#,1724#,1725#,1726#,1727#,1728#,1729#,1730#,1731#,1732#,1733#,1734#,1735#,1736#,1737#,1738#,1739#,1740#,1741#,1742#,1743#,1744#,1745#,1746#,1747#,1748#,1749#,1750#,1751#,1752#,1753#,1754#,1755#,1756#,1757#,1758#,1759#,1760#,1761#,1762#,1763#,1764#,1765#,1766#,1767#,1768#,1769#,1770#,1771#,1772#,1773#,1774#,1775#,1776#,1777#,1778#,1779#,1780#,1781#,1782#,1783#,1784#,1785#,1786#,1787#,1788#,1789#,1790#,1791#,1792#,1793#,1794#,1795#,1796#,1797#,1798#,1799#,1800#,1801#,1802#,1803#,1804#,1805#,1806#,1807#,1808#,1809#,1810#,1811#,1812#,1813#,1814#,1815#,1816#,1817#,1818#,1819#,1820#,1821#,1822#,1823#,1824#,1825#,1826#,1827#,1828#,1829#,1830#,1831#,1832#,1833#,1834#,1835#,1836#,1837#,1838#,1839#,1840#,1841#,1842#,1843#,1844#,1845#,1846#,1847#,1848#,1849#,1850#,1851#,1852#,1853#,1854#,1855#,1856#,1857#,1858#,1859#,1860#,1861#,1862#,1863#,1864#,1865#,1866#,1867#,1868#,1869#,1870#,1871#,1872#,1873#,1874#,1875#,1876#,1877#,1878#,1879#,1880#,1881#,1882#,1883#,1884#,1885#,1886#,1887#,1888#,1889#,1890#,1891#,1892#,1893#,1894#,1895#,1896#,1897#,1898#,1899#,1900#,1901#,1902#,1903#,1904#,1905#,1906#,1907#,1908#,1909#,1910#,1911#,1912#,1913#,1914#,1915#,1916#,1917#,1918#,1919#,1920#,1921#,1922#,1923#,1924#,1925#,1926#,1927#,1928#,1929#,1930#,1931#,1932#,1933#,1934#,1935#,1936#,1937#,1938#,1939#,1940#,1941#,1942#,1943#,1944#,1945#,1946#,1947#,1948#,1949#,1950#,1951#,1952#,1953#,1954#,1955#,1956#,1957#,1958#,1959#,1960#,1961#,1962#,1963#,1964#,1965#,1966#,1967#,1968#,1969#,1970#,1971#,1972#,1973#,1974#,1975#,1976#,1977#,1978#,1979#,1980#,1981#,1982#,1983#,1984#,1985#,1986#,1987#,1988#,1989#,1990#,1991#,1992#,1993#,1994#,1995#,1996#,1997#,1998#", "usage": { "rgw.main": { "size": 209866840, "size_actual": 209903616, "size_utilized": 209866840, "size_kb": 204949, "size_kb_actual": 204984, "size_kb_utilized": 204949, "num_objects": 169433938 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 837, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 1, "num_objects": 31 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } I am not sure what those objects are and why they have so many shards, since actual readable objects are less than 10 . Any help with that is really appreciated.

2 2

Blank dashboard
by Curt 31 Jul '23

31 Jul '23

Hello, This is a strange one for me. My ceph dashboard just stopped loading, nothing but a white page. I don't see anything in the logs and on browser side the only error I see is Failed to load resource: net::ERR_CONTENT_LENGTH_MISMATCH in chrome and Uncaught SyntaxError: expected expression, got end of script for Firefox on the file main.ddd4de0999172734.js. Nothing in the logs files, put mgr up to 20 on the log level. Any suggestions? Thanks, Curt

2 2

[rbd-mirror] can't enable journal-based image mirroring
by Tony Liu 31 Jul '23

31 Jul '23

Hi, The Ceph cluster is with Pacific v16.2.10. "rbd mirror image enable <image> journal" seems not working. Any clues what I'm missing? There is no error messages from the CLI. Any way to troubleshooting? ``` # rbd mirror pool info volume-ssd Mode: image Site Name: 35d050c0-77c0-11eb-9242-2cea7ff9d07c Peer Sites: UUID: 86eacc0f-6657-4742-8daf-2942ea23affd Name: qa Mirror UUID: Direction: rx-tx Client: client.infra # rbd feature enable volume-ssd/volume-aceee005-265e-44ea-a591-b6dda639a76b journaling # rbd mirror image enable volume-ssd/volume-aceee005-265e-44ea-a591-b6dda639a76b journal # rbd info volume-ssd/volume-aceee005-265e-44ea-a591-b6dda639a76b rbd image 'volume-aceee005-265e-44ea-a591-b6dda639a76b': size 40 GiB in 10240 objects order 22 (4 MiB objects) snapshot_count: 0 id: e6fce674350e55 block_name_prefix: rbd_data.e6fce674350e55 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Sun Jul 30 22:21:09 2023 access_timestamp: Mon Jul 31 00:36:20 2023 modify_timestamp: Mon Jul 31 08:01:26 2023 parent: image/801d1850-de2f-443b-a30c-71966e90c118@snap overlap: 10 GiB journal: e6fce674350e55 mirroring state: disabled # rbd mirror image status volume-ssd/volume-aceee005-265e-44ea-a591-b6dda639a76b rbd: mirroring not enabled on the image ``` Thanks! Tony

1 1

Some Ceph OSD metrics are zero
by GOSSET, Alexandre 31 Jul '23

31 Jul '23

Hello, We have a cluster in version 17.2.5 and on 2 nodes (out of 5), some metrics are at 0 on all OSD. The problem is really global to the node. I performed an upgrade to 17.2.6 on the cluster like this : * upgrade all monitors * upgrade all mgr * then reboot each node one by one (there was a pending upgrade of linux kernel image) to upgrade all OSDs This fixed for one node of the cluster but not the other so I stil don't understand. Here is an example of such OSD : { "AsyncMessenger::Worker-0": { "msgr_recv_messages": 6843132, "msgr_send_messages": 6880681, "msgr_recv_bytes": 63926805259, "msgr_send_bytes": 3298578019, "msgr_created_connections": 354, "msgr_active_connections": 20, "msgr_running_total_time": 361.495252541, "msgr_running_send_time": 88.204996929, "msgr_running_recv_time": 157.610653554, "msgr_running_fast_dispatch_time": 94.944377551, "msgr_send_messages_queue_lat": { "avgcount": 6880679, "sum": 398.607948692, "avgtime": 0.000057931 }, "msgr_handle_ack_lat": { "avgcount": 8509969, "sum": 1.558485881, "avgtime": 0.000000183 } }, "AsyncMessenger::Worker-1": { "msgr_recv_messages": 2332821, "msgr_send_messages": 2357957, "msgr_recv_bytes": 17554824362, "msgr_send_bytes": 3407896999, "msgr_created_connections": 177, "msgr_active_connections": 18, "msgr_running_total_time": 112.862256919, "msgr_running_send_time": 34.874196263, "msgr_running_recv_time": 46.995625507, "msgr_running_fast_dispatch_time": 25.082966322, "msgr_send_messages_queue_lat": { "avgcount": 2357955, "sum": 85.725096545, "avgtime": 0.000036355 }, "msgr_handle_ack_lat": { "avgcount": 1140003, "sum": 0.241074745, "avgtime": 0.000000211 } }, "AsyncMessenger::Worker-2": { "msgr_recv_messages": 4326174, "msgr_send_messages": 4330697, "msgr_recv_bytes": 31752213275, "msgr_send_bytes": 3567666813, "msgr_created_connections": 100, "msgr_active_connections": 20, "msgr_running_total_time": 218.260025879, "msgr_running_send_time": 58.635931148, "msgr_running_recv_time": 92.152198211, "msgr_running_fast_dispatch_time": 54.438590557, "msgr_send_messages_queue_lat": { "avgcount": 4330696, "sum": 250.493265066, "avgtime": 0.000057841 }, "msgr_handle_ack_lat": { "avgcount": 4229647, "sum": 0.806930547, "avgtime": 0.000000190 } }, "bluefs": { "db_total_bytes": 1920362086400, "db_used_bytes": 1089011712, "wal_total_bytes": 0, "wal_used_bytes": 0, "slow_total_bytes": 0, "slow_used_bytes": 0, "num_files": 125, "log_bytes": 2646016, "log_compactions": 812, "logged_bytes": 13500940288, "files_written_wal": 1575, "files_written_sst": 182, "bytes_written_wal": 25542598656, "bytes_written_sst": 2466263040, "bytes_written_slow": 0, "max_bytes_wal": 0, "max_bytes_db": 2035351552, "max_bytes_slow": 0, "alloc_unit_main": 0, "alloc_unit_db": 65536, "alloc_unit_wal": 0, "read_random_count": 201339, "read_random_bytes": 3080889493, "read_random_disk_count": 101652, "read_random_disk_bytes": 2645938201, "read_random_disk_bytes_wal": 0, "read_random_disk_bytes_db": 2645938201, "read_random_disk_bytes_slow": 0, "read_random_buffer_count": 110722, "read_random_buffer_bytes": 434951292, "read_count": 80250, "read_bytes": 1159472505, "read_disk_count": 68083, "read_disk_bytes": 1053351936, "read_disk_bytes_wal": 0, "read_disk_bytes_db": 1053356032, "read_disk_bytes_slow": 0, "read_prefetch_count": 78665, "read_prefetch_bytes": 1108068939, "compact_lat": { "avgcount": 812, "sum": 1.246095373, "avgtime": 0.001534600 }, "compact_lock_lat": { "avgcount": 812, "sum": 0.466894228, "avgtime": 0.000574992 }, "alloc_slow_fallback": 0, "alloc_slow_size_fallback": 0, "read_zeros_candidate": 0, "read_zeros_errors": 0 }, "bluestore": { "allocated": 180780802048, "stored": 188948817556, "fragmentation_micros": 2, "alloc_unit": 4096, "state_prepare_lat": { "avgcount": 3979669, "sum": 426.352439741, "avgtime": 0.000107132 }, "state_aio_wait_lat": { "avgcount": 3979669, "sum": 662.940570063, "avgtime": 0.000166581 }, "state_io_done_lat": { "avgcount": 3979669, "sum": 7.323425924, "avgtime": 0.000001840 }, "state_kv_queued_lat": { "avgcount": 3979669, "sum": 1900.856846444, "avgtime": 0.000477641 }, "state_kv_commiting_lat": { "avgcount": 3979669, "sum": 3192.442512062, "avgtime": 0.000802187 }, "state_kv_done_lat": { "avgcount": 3979669, "sum": 1.143240407, "avgtime": 0.000000287 }, "state_finishing_lat": { "avgcount": 3979665, "sum": 0.363208449, "avgtime": 0.000000091 }, "state_done_lat": { "avgcount": 3979658, "sum": 123255.522846135, "avgtime": 0.030971385 }, "state_deferred_queued_lat": { "avgcount": 356134, "sum": 455582.851504787, "avgtime": 1.279245597 }, "state_deferred_aio_wait_lat": { "avgcount": 356134, "sum": 64.258012775, "avgtime": 0.000180432 }, "state_deferred_cleanup_lat": { "avgcount": 356134, "sum": 58342.990966896, "avgtime": 0.163823142 }, "txc_commit_lat": { "avgcount": 3979669, "sum": 6190.519490871, "avgtime": 0.001555536 }, "txc_throttle_lat": { "avgcount": 3979669, "sum": 61.112847717, "avgtime": 0.000015356 }, "txc_submit_lat": { "avgcount": 3979669, "sum": 495.416650342, "avgtime": 0.000124486 }, "txc_count": 3979669, "read_onode_meta_lat": { "avgcount": 1197781, "sum": 19.722806609, "avgtime": 0.000016466 }, "read_wait_aio_lat": { "avgcount": 929167, "sum": 537.061469380, "avgtime": 0.000578003 }, "csum_lat": { "avgcount": 2133026, "sum": 13.996152575, "avgtime": 0.000006561 }, "read_eio": 0, "reads_with_retries": 0, "read_lat": { "avgcount": 268614, "sum": 520.246577964, "avgtime": 0.001936781 }, "kv_flush_lat": { "avgcount": 3291697, "sum": 403.833685777, "avgtime": 0.000122682 }, "kv_commit_lat": { "avgcount": 3291697, "sum": 2311.502280053, "avgtime": 0.000702222 }, "kv_sync_lat": { "avgcount": 3291697, "sum": 2715.335965830, "avgtime": 0.000824904 }, "kv_final_lat": { "avgcount": 3291667, "sum": 65.461458609, "avgtime": 0.000019887 }, "write_big": 3712142, "write_big_bytes": 94437191680, "write_big_blobs": 4685989, "write_big_deferred": 0, "write_small": 584115, "write_small_bytes": 960510935, "write_small_unused": 13820, "write_small_pre_read": 567335, "write_pad_bytes": 217785305, "write_penalty_read_ops": 688111, "write_new": 4702769, "issued_deferred_writes": 567335, "issued_deferred_write_bytes": 2323804160, "submitted_deferred_writes": 251455, "submitted_deferred_write_bytes": 1235841024, "write_big_skipped_blobs": 0, "write_big_skipped_bytes": 0, "write_small_skipped": 0, "write_small_skipped_bytes": 0, "compressed": 0, "compressed_allocated": 0, "compressed_original": 0, "compress_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "decompress_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "compress_success_count": 0, "compress_rejected_count": 0, "onodes": 42417, "onodes_pinned": 0, "onode_hits": 8904311, "onode_misses": 87317, "onode_shard_hits": 6533806, "onode_shard_misses": 270931, "onode_extents": 1288567, "onode_blobs": 1116911, "buffers": 956, "buffer_bytes": 4562944, "buffer_hit_bytes": 652455498, "buffer_miss_bytes": 115020768720, "onode_reshard": 54721, "blob_split": 1236, "extent_compress": 4199105, "gc_merged": 0, "omap_seek_to_first_lat": { "avgcount": 31054, "sum": 1.622677452, "avgtime": 0.000052253 }, "omap_upper_bound_lat": { "avgcount": 4, "sum": 0.016788155, "avgtime": 0.004197038 }, "omap_lower_bound_lat": { "avgcount": 82, "sum": 0.001473844, "avgtime": 0.000017973 }, "omap_next_lat": { "avgcount": 791854, "sum": 2.400923607, "avgtime": 0.000003032 }, "omap_get_keys_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "omap_get_values_lat": { "avgcount": 26905, "sum": 4.060382074, "avgtime": 0.000150915 }, "omap_clear_lat": { "avgcount": 3, "sum": 0.001249818, "avgtime": 0.000416606 }, "clist_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "remove_lat": { "avgcount": 35724, "sum": 28.130138084, "avgtime": 0.000787429 }, "truncate_lat": { "avgcount": 6300, "sum": 0.361881579, "avgtime": 0.000057441 } }, "bluestore-pricache": { "target_bytes": 4294967296, "mapped_bytes": 2529673216, "unmapped_bytes": 230645760, "heap_bytes": 2760318976, "cache_bytes": 2845415832 }, "bluestore-pricache:data": { "pri0_bytes": 0, "pri1_bytes": 0, "pri2_bytes": 0, "pri3_bytes": 0, "pri4_bytes": 24576, "pri5_bytes": 57344, "pri6_bytes": 81920, "pri7_bytes": 0, "pri8_bytes": 0, "pri9_bytes": 0, "pri10_bytes": 0, "pri11_bytes": 77126679, "reserved_bytes": 73704425, "committed_bytes": 150994944 }, "bluestore-pricache:kv": { "pri0_bytes": 67108864, "pri1_bytes": 0, "pri2_bytes": 0, "pri3_bytes": 4375736, "pri4_bytes": 0, "pri5_bytes": 112032, "pri6_bytes": 0, "pri7_bytes": 0, "pri8_bytes": 0, "pri9_bytes": 0, "pri10_bytes": 0, "pri11_bytes": 578450095, "reserved_bytes": 71373561, "committed_bytes": 721420288 }, "bluestore-pricache:kv_onode": { "pri0_bytes": 67108864, "pri1_bytes": 0, "pri2_bytes": 0, "pri3_bytes": 0, "pri4_bytes": 0, "pri5_bytes": 0, "pri6_bytes": 0, "pri7_bytes": 0, "pri8_bytes": 0, "pri9_bytes": 0, "pri10_bytes": 5099312, "pri11_bytes": 51417786, "reserved_bytes": 77700630, "committed_bytes": 201326592 }, "bluestore-pricache:meta": { "pri0_bytes": 0, "pri1_bytes": 90067, "pri2_bytes": 90067, "pri3_bytes": 1200904, "pri4_bytes": 1486119, "pri5_bytes": 3047296, "pri6_bytes": 3632737, "pri7_bytes": 0, "pri8_bytes": 0, "pri9_bytes": 0, "pri10_bytes": 0, "pri11_bytes": 578450095, "reserved_bytes": 83091355, "committed_bytes": 671088640 }, "cct": { "total_workers": 16, "unhealthy_workers": 0 }, "finisher-commit_finisher": { "queue_len": 0, "complete_latency": { "avgcount": 286, "sum": 0.262805597, "avgtime": 0.000918900 } }, "finisher-objecter-finisher-0": { "queue_len": 0, "complete_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluestore_alloc_bytes": 125943744, "bluestore_alloc_items": 8193228, "bluestore_cache_data_bytes": 1884109, "bluestore_cache_data_items": 829, "bluestore_cache_onode_bytes": 28842880, "bluestore_cache_onode_items": 42416, "bluestore_cache_meta_bytes": 50594253, "bluestore_cache_meta_items": 5315174, "bluestore_cache_other_bytes": 229414948, "bluestore_cache_other_items": 23631685, "bluestore_Buffer_bytes": 108416, "bluestore_Buffer_items": 968, "bluestore_Extent_bytes": 61847760, "bluestore_Extent_items": 1288495, "bluestore_Blob_bytes": 140450128, "bluestore_Blob_items": 1254019, "bluestore_SharedBlob_bytes": 125093808, "bluestore_SharedBlob_items": 1116909, "bluestore_inline_bl_bytes": 369130, "bluestore_inline_bl_items": 725, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_txc_bytes": 5888, "bluestore_txc_items": 8, "bluestore_writing_deferred_bytes": 3317593, "bluestore_writing_deferred_items": 794, "bluestore_writing_bytes": 380783, "bluestore_writing_items": 47, "bluefs_bytes": 114472, "bluefs_items": 6282, "bluefs_file_reader_bytes": 705280, "bluefs_file_reader_items": 72, "bluefs_file_writer_bytes": 768, "bluefs_file_writer_items": 4, "buffer_anon_bytes": 13417560, "buffer_anon_items": 53741, "buffer_meta_bytes": 3034064, "buffer_meta_items": 34478, "osd_bytes": 1998496, "osd_items": 173, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 205086276, "osd_pglog_items": 814012, "osdmap_bytes": 515864, "osdmap_items": 11775, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "mds_co_bytes": 0, "mds_co_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "op_active": 0, "op_laggy": 0, "op_send": 0, "op_send_bytes": 0, "op_resend": 0, "op_reply": 0, "oplen_avg": { "avgcount": 0, "sum": 0 }, "op": 0, "op_r": 0, "op_w": 0, "op_rmw": 0, "op_pg": 0, "osdop_stat": 0, "osdop_create": 0, "osdop_read": 0, "osdop_write": 0, "osdop_writefull": 0, "osdop_writesame": 0, "osdop_append": 0, "osdop_zero": 0, "osdop_truncate": 0, "osdop_delete": 0, "osdop_mapext": 0, "osdop_sparse_read": 0, "osdop_clonerange": 0, "osdop_getxattr": 0, "osdop_setxattr": 0, "osdop_cmpxattr": 0, "osdop_rmxattr": 0, "osdop_resetxattrs": 0, "osdop_call": 0, "osdop_watch": 0, "osdop_notify": 0, "osdop_src_cmpxattr": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_other": 0, "linger_active": 0, "linger_send": 0, "linger_resend": 0, "linger_ping": 0, "poolop_active": 0, "poolop_send": 0, "poolop_resend": 0, "poolstat_active": 0, "poolstat_send": 0, "poolstat_resend": 0, "statfs_active": 0, "statfs_send": 0, "statfs_resend": 0, "command_active": 0, "command_send": 0, "command_resend": 0, "map_epoch": 12426, "map_full": 1, "map_inc": 287, "osd_sessions": 0, "osd_session_open": 0, "osd_session_close": 0, "osd_laggy": 0, "omap_wr": 0, "omap_rd": 0, "omap_del": 0 }, "osd": { "op_wip": 0, "op": 0, "op_in_bytes": 0, "op_out_bytes": 0, "op_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_process_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_prepare_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_r": 0, "op_r_out_bytes": 0, "op_r_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_r_process_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_r_prepare_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_w": 0, "op_w_in_bytes": 0, "op_w_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_w_process_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_w_prepare_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_rw": 0, "op_rw_in_bytes": 0, "op_rw_out_bytes": 0, "op_rw_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_rw_process_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_rw_prepare_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "op_before_queue_op_lat": { "avgcount": 3965535, "sum": 333.780003675, "avgtime": 0.000084170 }, "op_before_dequeue_op_lat": { "avgcount": 3965535, "sum": 434.995411447, "avgtime": 0.000109694 }, "subop": 3944307, "subop_in_bytes": 98738982548, "subop_latency": { "avgcount": 3944307, "sum": 6756.175478507, "avgtime": 0.001712892 }, "subop_w": 3944307, "subop_w_in_bytes": 98738982548, "subop_w_latency": { "avgcount": 3944307, "sum": 6756.175478507, "avgtime": 0.001712892 }, "subop_pull": 0, "subop_pull_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "subop_push": 0, "subop_push_in_bytes": 0, "subop_push_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "pull": 0, "push": 0, "push_out_bytes": 0, "recovery_ops": 0, "recovery_bytes": 0, "loadavg": 40, "cached_crc": 0, "cached_crc_adjusted": 0, "missed_crc": 0, "numpg": 173, "numpg_primary": 0, "numpg_replica": 173, "numpg_stray": 0, "numpg_removing": 0, "heartbeat_to_peers": 10, "map_messages": 2085, "map_message_epochs": 2223, "map_message_epoch_dups": 1935, "messages_delayed_for_map": 0, "osd_map_cache_hit": 50571, "osd_map_cache_miss": 1, "osd_map_cache_miss_low": 0, "osd_map_cache_miss_low_avg": { "avgcount": 0, "sum": 0 }, "osd_map_bl_cache_hit": 39, "osd_map_bl_cache_miss": 317, "stat_bytes": 1920362086400, "stat_bytes_used": 181869830144, "stat_bytes_avail": 1738492256256, "copyfrom": 0, "tier_promote": 0, "tier_flush": 0, "tier_flush_fail": 0, "tier_try_flush": 0, "tier_try_flush_fail": 0, "tier_evict": 0, "tier_whiteout": 0, "tier_dirty": 0, "tier_clean": 0, "tier_delay": 0, "tier_proxy_read": 0, "tier_proxy_write": 0, "agent_wake": 0, "agent_skip": 0, "agent_flush": 0, "agent_evict": 0, "object_ctx_cache_hit": 0, "object_ctx_cache_total": 0, "op_cache_hit": 0, "osd_tier_flush_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "osd_tier_promote_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "osd_tier_r_lat": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "osd_pg_info": 3979232, "osd_pg_fastinfo": 3825183, "osd_pg_biginfo": 33887 }, "recoverystate_perf": { "initial_latency": { "avgcount": 173, "sum": 6.060385414, "avgtime": 0.035031129 }, "started_latency": { "avgcount": 173, "sum": 35.126913452, "avgtime": 0.203045742 }, "reset_latency": { "avgcount": 346, "sum": 696.523139606, "avgtime": 2.013072657 }, "start_latency": { "avgcount": 346, "sum": 0.000675707, "avgtime": 0.000001952 }, "primary_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "peering_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "backfilling_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitremotebackfillreserved_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitlocalbackfillreserved_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "notbackfilling_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "repnotrecovering_latency": { "avgcount": 88, "sum": 14.954158341, "avgtime": 0.169933617 }, "repwaitrecoveryreserved_latency": { "avgcount": 88, "sum": 43.849329328, "avgtime": 0.498287833 }, "repwaitbackfillreserved_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "reprecovering_latency": { "avgcount": 88, "sum": 16.136443540, "avgtime": 0.183368676 }, "activating_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitlocalrecoveryreserved_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitremoterecoveryreserved_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "recovering_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "recovered_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "clean_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "active_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "replicaactive_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "stray_latency": { "avgcount": 346, "sum": 212.848549950, "avgtime": 0.615169219 }, "getinfo_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "getlog_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitactingchange_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "incomplete_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "down_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "getmissing_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "waitupthru_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "notrecovering_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "rocksdb": { "get_latency": { "avgcount": 790064, "sum": 51.480388766, "avgtime": 0.000065159 }, "submit_latency": { "avgcount": 3979669, "sum": 377.602827079, "avgtime": 0.000094882 }, "submit_sync_latency": { "avgcount": 3291698, "sum": 1909.117762596, "avgtime": 0.000579979 }, "compact": 0, "compact_range": 0, "compact_queue_merge": 0, "compact_queue_len": 0, "rocksdb_write_wal_time": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "rocksdb_write_memtable_time": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "rocksdb_write_delay_time": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "rocksdb_write_pre_and_post_time": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } } } We monitor osd.op_w_latency to check for problems on our clusters. Does someone know how to fix this ? Best regards, [cid:e90311cc-51d8-4266-b5ee-347da8fe1b72] Alexandre GOSSET Ingénieur Système et Réseau Fixe : +33 (0)3 59 82 20 20 Cyllene Lille - 393 rue du général de Gaulle - 59700 Marcq-en-baroeul www.groupe-cyllene.com<http://www.groupe-cyllene.com/>

1 0

cephadm logs
by Luis Domingues 30 Jul '23

30 Jul '23

Hi, Quick question about cephadm and its logs. On my cluster I have every logs that goes to journald. But on each machine, I still have /var/log/ceph/cephadm.log that is alive. Is there a way to make cephadm log to journald instead of a file? If yes did I miss it on the documentation? Of if not is there any reason to log into a file while everything else logs to journald? Thanks Luis Domingues Proton AG

3 3

OSD stuck on booting state after upgrade (v15.2.17 -> v17.2.6)
by s.smagul94＠gmail.com 30 Jul '23

30 Jul '23

Updating the Ceph cluster from the Octopus version (v15.2.17) to Quincy (v17.2.6). We used ceph-deploy to update all ceph packages on all hosts, and then we restarted the services one by one (mon -> mgr -> osd -> rgw). During the restart on the first node, all osd encountered an issue where they didn't change to the "up" state and got stuck in "booting" state. # ceph daemon osd.3 status { "cluster_fsid": "f95b201c-4cd6-4c36-a54e-7f2b68608b8f", "osd_fsid": "b0141718-a2ac-4a26-808b-17b6741b789e", "whoami": 3, "state": "booting", "oldest_map": 4437792, "newest_map": 4441114, "num_pgs": 29 } While changing "ceph osd require-osd-release quincy," the monitor service crashed with errors. # ceph report | jq '.osdmap.require_osd_release' "nautilus" -2> 2023-07-25T12:10:20.977+0600 7f245a84f700 5 mon.ceph-ph-mon1-dc3(a)0(leader).paxos(paxos updating c 81819224..81819937) is_readable = 1 - now=2023-07-25T12:10:20.981801+0600 lease_expire=2023-07-25T12:10:25.959818+0600 has v0 lc 81819937 -1> 2023-07-25T12:10:20.997+0600 7f245a84f700 -1 /build/ceph-17.2.6/src/mon/OSDMonitor.cc: In function 'bool OSDMonitor::prepare_command_impl(MonOpRequestRef, const cmdmap_t&)' thread 7f245a84f700 time 2023-07-25T12:10:20.981991+0600 /build/ceph-17.2.6/src/mon/OSDMonitor.cc: 11631: FAILED ceph_assert(osdmap.require_osd_release >= ceph_release_t::octopus) ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f24629d3878] 2: /usr/lib/ceph/libceph-common.so.2(+0x27da8a) [0x7f24629d3a8a] 3: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_stri ng<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basi c_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std: :__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std: :vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<lo ng, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&)+0xcb13) [0x5569f209a823] 4: (OSDMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x45f) [0x5569f20ab89f] 5: (OSDMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x162) [0x5569f20baa42] 6: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x716) [0x5569f201fd86] 7: (PaxosService::C_RetryMessage::_finish(int)+0x6c) [0x5569f1f4f93c] 8: (C_MonOp::finish(int)+0x4b) [0x5569f1ebbb3b] 9: (Context::complete(int)+0xd) [0x5569f1ebaa0d] 10: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xb0) [0x5569f1ef11e0] 11: (Paxos::finish_round()+0xb1) [0x5569f2015a61] 12: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0x11e3) [0x5569f20172a3] 13: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x49f) [0x5569f2019f7f] 14: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x14f4) [0x5569f1eb7f34] 15: (Monitor::_ms_dispatch(Message*)+0xa68) [0x5569f1eb8bd8] 16: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x5569f1ef2c4d] 17: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f2462c71da0] 18: (DispatchQueue::entry()+0x58f) [0x7f2462c6f63f] 19: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2462d40b61] 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f24624f4609] 21: clone() 0> 2023-07-25T12:10:21.009+0600 7f245a84f700 -1 *** Caught signal (Aborted) ** in thread 7f245a84f700 thread_name:ms_dispatch ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f24625003c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7f24629d38e0] 5: /usr/lib/ceph/libceph-common.so.2(+0x27da8a) [0x7f24629d3a8a] 6: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_stri ng<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basi c_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std: :__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std: :vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<lo ng, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&)+0xcb13) [0x5569f209a823] 7: (OSDMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x45f) [0x5569f20ab89f] 8: (OSDMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x162) [0x5569f20baa42] 9: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x716) [0x5569f201fd86] 10: (PaxosService::C_RetryMessage::_finish(int)+0x6c) [0x5569f1f4f93c] 11: (C_MonOp::finish(int)+0x4b) [0x5569f1ebbb3b] 12: (Context::complete(int)+0xd) [0x5569f1ebaa0d] 13: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xb0) [0x5569f1ef11e0] 14: (Paxos::finish_round()+0xb1) [0x5569f2015a61] 15: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0x11e3) [0x5569f20172a3] 16: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x49f) [0x5569f2019f7f] 17: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x14f4) [0x5569f1eb7f34] 18: (Monitor::_ms_dispatch(Message*)+0xa68) [0x5569f1eb8bd8] 19: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x5569f1ef2c4d] 20: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f2462c71da0] 21: (DispatchQueue::entry()+0x58f) [0x7f2462c6f63f] 22: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2462d40b61] 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f24624f4609] 24: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Now the cluster looks as follows: # ceph -s cluster: id: f95b201c-4cd6-4c36-a54e-7f2b68608b8f health: HEALTH_WARN noout flag(s) set 12 osds down 1 host (12 osds) down all OSDs are running octopus or later but require_osd_release < octopus Degraded data redundancy: 5731463/11463184 objects degraded (49.999%), 240 pgs degraded, 315 pgs undersized services: mon: 3 daemons, quorum ceph-ph-mon1-dc3,hw-ceph-ph3-dc3,hw-ceph-ph4-dc3 (age 68m) mgr: ceph-ph-mon1-dc3.alahd.kz.test.bash.kz(active, since 45m) osd: 25 osds: 13 up (since 26h), 25 in (since 21h); 6 remapped pgs flags noout rgw: 1 daemon active (1 hosts, 1 zones) data: pools: 8 pools, 321 pgs objects: 5.73M objects, 4.4 TiB usage: 8.8 TiB used, 32 TiB / 40 TiB avail pgs: 5731463/11463184 objects degraded (49.999%) 129/11463184 objects misplaced (0.001%) 240 active+undersized+degraded 75 active+undersized 6 active+clean+remapped # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 42.23981 - 22 TiB 4.7 TiB 4.6 TiB 1.2 GiB 22 GiB 17 TiB 0 0 - root default -2 20.19991 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host hw-ceph-ph3 0 hdd 1.81999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down osd.0 1 hdd 1.81999 1.00000 1.8 TiB 401 GiB 396 GiB 76 MiB 1.2 GiB 1.4 TiB 21.47 0.98 0 down osd.1 2 hdd 1.81999 1.00000 1.8 TiB 365 GiB 362 GiB 45 KiB 1.1 GiB 1.5 TiB 19.59 0.90 0 down osd.2 3 hdd 1.81999 1.00000 1.8 TiB 584 GiB 580 GiB 825 KiB 1.7 GiB 1.3 TiB 31.29 1.43 0 down osd.3 4 hdd 1.81999 1.00000 1.8 TiB 365 GiB 362 GiB 621 KiB 1.1 GiB 1.5 TiB 19.57 0.90 0 down osd.4 5 hdd 1.81999 1.00000 1.8 TiB 583 GiB 579 GiB 31 KiB 1.7 GiB 1.3 TiB 31.25 1.43 0 down osd.5 6 hdd 1.81999 1.00000 1.8 TiB 365 GiB 362 GiB 43 KiB 1.1 GiB 1.5 TiB 19.55 0.90 0 down osd.6 7 hdd 1.81999 1.00000 1.8 TiB 365 GiB 362 GiB 12 KiB 1.1 GiB 1.5 TiB 19.58 0.90 0 down osd.7 8 hdd 1.81999 1.00000 1.8 TiB 365 GiB 362 GiB 27 KiB 1.1 GiB 1.5 TiB 19.58 0.90 0 down osd.8 9 hdd 1.81999 1.00000 1.8 TiB 547 GiB 543 GiB 47 KiB 1.6 GiB 1.3 TiB 29.32 1.34 0 down osd.9 10 hdd 1.81999 1.00000 1.8 TiB 330 GiB 327 GiB 68 KiB 987 MiB 1.5 TiB 17.67 0.81 0 down osd.10 12 ssd 0.17999 1.00000 186 GiB 1.2 GiB 67 MiB 1.1 GiB 86 MiB 185 GiB 0.65 0.03 0 down osd.12 -3 22.03990 - 22 TiB 4.7 TiB 4.6 TiB 1.2 GiB 22 GiB 17 TiB 21.11 0.97 - host hw-ceph-ph4 11 hdd 1.81999 1.00000 1.8 TiB 550 GiB 546 GiB 1012 KiB 2.3 GiB 1.3 TiB 29.48 1.35 34 up osd.11 13 hdd 1.81999 1.00000 1.8 TiB 332 GiB 329 GiB 2.8 MiB 1.5 GiB 1.5 TiB 17.78 0.82 22 up osd.13 14 hdd 1.81999 1.00000 1.8 TiB 550 GiB 547 GiB 4.4 MiB 2.1 GiB 1.3 TiB 29.50 1.35 25 up osd.14 15 hdd 1.81999 1.00000 1.8 TiB 259 GiB 256 GiB 1.6 MiB 1.2 GiB 1.6 TiB 13.86 0.64 20 up osd.15 16 hdd 1.81999 1.00000 1.8 TiB 477 GiB 474 GiB 2.7 MiB 1.9 GiB 1.4 TiB 25.58 1.17 23 up osd.16 17 hdd 1.81999 1.00000 1.8 TiB 403 GiB 400 GiB 1.7 MiB 1.7 GiB 1.4 TiB 21.63 0.99 32 up osd.17 18 hdd 1.81999 1.00000 1.8 TiB 294 GiB 291 GiB 79 MiB 1.4 GiB 1.5 TiB 15.79 0.72 25 up osd.18 19 hdd 1.81999 1.00000 1.8 TiB 294 GiB 291 GiB 1.6 MiB 1.4 GiB 1.5 TiB 15.76 0.72 27 up osd.19 20 hdd 1.81999 1.00000 1.8 TiB 477 GiB 474 GiB 2.3 MiB 1.9 GiB 1.4 TiB 25.60 1.17 23 up osd.20 21 hdd 1.81999 1.00000 1.8 TiB 473 GiB 470 GiB 23 MiB 2.0 GiB 1.4 TiB 25.36 1.16 24 up osd.21 22 hdd 1.81999 1.00000 1.8 TiB 404 GiB 401 GiB 3.6 MiB 2.3 GiB 1.4 TiB 21.69 0.99 22 up osd.22 23 hdd 1.81999 1.00000 1.8 TiB 258 GiB 255 GiB 2.7 MiB 1.3 GiB 1.6 TiB 13.84 0.63 18 up osd.23 24 ssd 0.20000 1.00000 238 GiB 2.2 GiB 67 MiB 1.1 GiB 1.0 GiB 236 GiB 0.91 0.04 32 up osd.24 TOTAL 40 TiB 8.8 TiB 8.8 TiB 2.3 GiB 34 GiB 32 TiB 21.81 MIN/MAX VAR: 0/1.43 STDDEV: 8.97 We checked the network connectivity between the hosts, and everything is fine there. We have not restarted the OSD services on the second node because we suspect that the entire cluster might crash. Please give advise how to solve the problem

5 6

configure rgw
by Tony Liu 30 Jul '23

30 Jul '23

Hi, I'm using Pacific v16.2.10 container image, deployed by cephadm. I used to manually build config file for rgw, deploy rgw, put config file in place and restart rgw. It works fine. Now, I'd like to put rgw config into config db. I tried with client.rgw, but the config is not taken by rgw. Also "config show" doesn't work. It always says "no config state". ``` # ceph orch ps | grep rgw rgw.qa.ceph-1.hzfrwq ceph-1 10.250.80.100:80 running (10m) 10m ago 53m 51.4M - 16.2.10 32214388de9d 13169a213bc5 # ceph config get client.rgw | grep frontends client.rgw basic rgw_frontends beast port=8086 * # ceph config show rgw.qa.ceph-1.hzfrwq Error ENOENT: no config state for daemon rgw.qa.ceph-1.hzfrwq # ceph config show client.rgw.qa.ceph-1.hzfrwq Error ENOENT: no config state for daemon client.rgw.qa.ceph-1.hzfrwq # radosgw-admin --show-config -n client.rgw.qa.ceph-1.hzfrwq | grep frontends rgw_frontends = beast port=7480 ``` Any clues what I am missing here? Thanks! Tony

1 2

precise/best way to check ssd usage
by Marc 29 Jul '23

29 Jul '23

Currently I am checking usage on ssd drives with ceph osd df| egrep 'CLASS|ssd' I have a use % between 48% and 57%, and assume that with a node failure 1/3 (only using 3x repl.) of this 57% needs to be able to migrate and added to a different node. Is there a better way of checking this (on old Nautilus)?

2 2