18 Apr
2020
18 Apr
'20
7:15 a.m.
Hi,
I am trying to understand how the PG peering count could increase between
two OSDMAP epochs when none of the OSDs went down or up, nor were there
changes in their weights.
This behaviour was seen on a Hammer cluster, but I guess it should be no
different from the recent releases.
I'm pasting the relevant piece of monitor logs below and attaching the
relevant odmap dumps.
As it can be seen, the changes to the osdmap between dump_e1053237
and dump_e10532398 seem to be restricted to a few pg_temp additions and
changes to up_thru, but the OSD weights and OSD up/down state have remained
the same.
Kindly let me know what I'm missing here in my understanding.
Thanks,
K.Prasad
Monitor logs
--------------------
2020-04-15 08:06:10.956681 mon.1 10.33.89.159:6789/0 52232955 : cluster
[INF] pgmap v107549051: 18432 pgs: 18332 active+clean, 1 peering, 99
active+remapped+backfill_toofull; 330 TB data, 518 TB used, 596 TB / 1114
TB avail; 34776/1684702815 objects degraded (0.002%); 1766213/1684702815
objects misplaced (0.105%)
2020-04-15 08:06:11.862613 mon.1 10.33.89.159:6789/0 52232956 : cluster
[INF] osdmap e1053236: 388 osds: 315 up, 308 in
2020-04-15 08:06:11.868380 mon.1 10.33.89.159:6789/0 52232957 : cluster
[INF] pgmap v107549052: 18432 pgs: 18332 active+clean, 1 peering, 99
active+remapped+backfill_toofull; 330 TB data, 518 TB used, 596 TB / 1114
TB avail; 34776/1684702815 objects degraded (0.002%); 1766213/1684702815
objects misplaced (0.105%)
2020-04-15 08:06:10.460979 osd.57 10.33.249.161:6811/4066471 200448 :
cluster [INF] 5.356s0 restarting backfill on osd.382(0) from
(936954'62108,945456'62308] 0//0//-1 to 1053229'64769
2020-04-15 08:06:12.696839 mon.1 10.33.89.159:6789/0 52232959 : cluster
[INF] osdmap e1053237: 388 osds: 315 up, 308 in
2020-04-15 08:06:12.703424 mon.1 10.33.89.159:6789/0 52232960 : cluster
[INF] pgmap v107549053: 18432 pgs: 18332 active+clean, 1 peering, 99
active+remapped+backfill_toofull; 330 TB data, 518 TB used, 596 TB / 1114
TB avail; 34776/1684702815 objects degraded (0.002%); 1766213/1684702815
objects misplaced (0.105%)
2020-04-15 08:06:13.745554 mon.1 10.33.89.159:6789/0 52232961 : cluster
[INF] osdmap e1053238: 388 osds: 315 up, 308 in
2020-04-15 08:06:13.899273 mon.1 10.33.89.159:6789/0 52232962 : cluster
[INF] pgmap v107549054: 18432 pgs: 1 inactive, 1
active+remapped+wait_backfill+backfill_toofull, 2 activating, 17078
active+clean, 1251 peering, 2 activating+remapped, 5 remapped+peering, 1
activating+degraded, 91 active+remapped+backfill_toofull; 329 TB data, 518
TB used, 600 TB / 1118 TB avail; 32298/1678972195 objects degraded
(0.002%); 1666716/1678972195 objects misplaced (0.099%)
2020-04-15 08:06:14.766039 mon.1 10.33.89.159:6789/0 52232963 : cluster
[INF] osdmap e1053239: 388 osds: 315 up, 308 in
2020-04-15 08:06:14.833707 mon.1 10.33.89.159:6789/0 52232964 : cluster
[INF] pgmap v107549055: 18432 pgs: 2 inactive, 1
active+remapped+wait_backfill+backfill_toofull, 3 activating, 16496
active+clean, 1794 peering, 4 activating+remapped, 42 remapped+peering, 3
activating+degraded, 87 active+remapped+backfill_toofull; 329 TB data, 518
TB used, 600 TB / 1118 TB avail; 30842/1678959760 objects degraded
(0.002%); 1576530/1678959760 objects misplaced (0.094%)
2020-04-15 08:06:15.730301 mon.1 10.33.89.159:6789/0 52232965 : cluster
[INF] osdmap e1053240: 388 osds: 315 up, 308 in
2020-04-15 08:06:15.774021 mon.1 10.33.89.159:6789/0 52232966 : cluster
[INF] pgmap v107549056: 18432 pgs: 2 inactive, 1
active+remapped+wait_backfill+backfill_toofull, 3 activating, 16139
active+clean, 2103 peering, 6 activating+remapped, 92 remapped+peering, 3
activating+degraded, 83 active+remapped+backfill_toofull; 329 TB data, 518
TB used, 600 TB / 1118 TB avail; 29414/1678947337 objects degraded
(0.002%); 1490753/1678947337 objects misplaced (0.089%)
2020-04-15 08:06:09.424940 osd.146 10.33.129.28:6800/2689 125912 : cluster
[INF] 5.252fs0 restarting backfill on osd.382(3) from
(935430'62507,945456'62707] 0//0//-1 to 1053229'65086
2020-04-15 08:06:16.713440 mon.1 10.33.89.159:6789/0 52232967 : cluster
[INF] osdmap e1053241: 388 osds: 315 up, 308 in
--
*-----------------------------------------------------------------------------------------*
*This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error, please notify the
system manager. This message contains confidential information and is
intended only for the individual named. If you are not the named addressee,
you should not disseminate, distribute or copy this email. Please notify
the sender immediately by email if you have received this email by mistake
and delete this email from your system. If you are not the intended
recipient, you are notified that disclosing, copying, distributing or
taking any action in reliance on the contents of this information is
strictly prohibited.*****
****
*Any views or opinions presented in this
email are solely those of the author and do not necessarily represent those
of the organization. Any information on shares, debentures or similar
instruments, recommended product pricing, valuations and the like are for
information purposes only. It is not meant to be an instruction or
recommendation, as the case may be, to buy or to sell securities, products,
services nor an offer to buy or sell securities, products or services
unless specifically stated to be so on behalf of the Flipkart group.
Employees of the Flipkart group of companies are expressly required not to
make defamatory statements and not to infringe or authorise any
infringement of copyright or any other legal right by email communications.
Any such communication is contrary to organizational policy and outside the
scope of the employment of the individual concerned. The organization will
not accept any liability in respect of such communication, and the employee
responsible will be personally liable for any damages or other liability
arising.*****
****
*Our organization accepts no liability for the
content of this email, or for the consequences of any actions taken on the
basis of the information *provided,* unless that information is
subsequently confirmed in writing. If you are not the intended recipient,
you are notified that disclosing, copying, distributing or taking any
action in reliance on the contents of this information is strictly
prohibited.*
_-----------------------------------------------------------------------------------------_