why osd's heartbeat partner comes from another root tree? - ceph-users

27 Nov 2019

Hello:
According to my understanding, osd's heartbeat partners only come from
those osds who assume the same pg
See below(# ceph osd tree), osd.10 and osd.0-6 cannot assume the same pg,
because osd.10 and osd.0-6 are from different root tree, and pg in my
cluster doesn't map across root trees(# ceph osd crush rule dump). so,
osd.0-6 cannot become the heartbeat partner of osd.10

But, below is the log on osd.10, It can be seen that the osd.10's heartbeat
partner include osd.0/1/2/5, why?

thanks for any help

...
  # osd.10 log
 2019-11-20 09:21:50.431799 7fbb369fb700 -1 osd.10 7344 heartbeat_check: no
 reply from 10.13.6.162:6806 osd.2 since back 2019-11-20 09:21:19.979712
 front 2019-11-20 09:21:19.979712 (cutoff 2019-11-20 09:21:30.431768)
 2019-11-20 13:15:59.175060 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no
 reply from 10.13.6.162:6806 <http://10.13.6.152:6806> osd.2 since back
 2019-11-20 13:15:38.710424 front 2019-11-20 13:15:38.710424 (cutoff
 2019-11-20 13:15:39.175058)
 2019-11-20 13:15:59.175110 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no
 reply from 10.13.6.160:6803 osd.0 since back 2019-11-20 13:15:38.710424
 front 2019-11-20 13:15:38.710424 (cutoff 2019-11-20 13:15:39.175058)
 2019-11-20 13:15:59.175118 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no
 reply from 10.13.6.161:6803 osd.1 since back 2019-11-20 13:15:38.710424
 front 2019-11-20 13:15:38.710424 (cutoff 2019-11-20 13:15:39.175058)
 2019-11-21 02:52:24.656783 7fbb369fb700 -1 osd.10 7374 heartbeat_check: no
 reply from 10.13.6.158:6810 osd.5 since back 2019-11-21 02:52:04.557548
 front 2019-11-21 02:52:04.557548 (cutoff 2019-11-21 02:52:04.656781)

# ceph osd tree

...
  -17       3.29095 root ssd-storage

 -25       1.09698     rack rack-ssd-A

 -18       1.09698         host ssd-osd01
  10   hdd 1.09698             osd.10                       up  1.00000
 1.00000
 -26       1.09698     rack rack-ssd-B

 -19       1.09698         host ssd-osd02
  11   hdd 1.09698             osd.11                        up  1.00000
 1.00000
 -27       1.09698     rack rack-ssd-C

 -20       1.09698         host ssd-osd03
  12   hdd 1.09698             osd.12                        up  1.00000
 1.00000
  -1       3.22256 root default

  -3       0.29300     host test-osd01

   0   hdd 0.29300         osd.0                             up  1.00000
 1.00000
  -5       0.29300     host test-osd02

   1   hdd 0.29300         osd.1                             up  0.89999
 1.00000
  -7       0.29300     host test-osd03

   2   hdd 0.29300         osd.2                             up  0.79999
 1.00000
  -9       0.29300     host test-osd04

   3   hdd 0.29300         osd.3                             up  1.00000
 1.00000
 -11       0.29300     host test-osd05

   4   hdd 0.29300         osd.4                             up  1.00000
 1.00000
 -13       0.29300     host test-osd06

   5   hdd 0.29300         osd.5                             up  1.00000
 1.00000
 -15       0.29300     host test-osd07

   6   hdd 0.29300         osd.6                             up  1.00000
 1.00000 

# ceph osd crush rule dump

...

 [
     {
         "rule_id": 0,
         "rule_name": "replicated_rule",
         "ruleset": 0,
         "type": 1,
         "min_size": 1,
         "max_size": 10,
         "steps": [
             {
                 "op": "take",
                 "item": -1,
                 "item_name": "default"
             },
             {
                 "op": "chooseleaf_firstn",
                 "num": 0,
                 "type": "host"
             },
             {
                 "op": "emit"
             }
         ]
     },
     {
         "rule_id": 1,
         "rule_name": "replicated_rule_ssd",
         "ruleset": 1,
         "type": 1,
         "min_size": 1,
         "max_size": 10,
         "steps": [
             {
                 "op": "take",
                 "item": -17,
                 "item_name": "ssd-storage"
             },
             {
                 "op": "chooseleaf_firstn",
                 "num": 0,
                 "type": "rack"
             },
             {
                 "op": "emit"
             }
         ]
     }
 ]

# some parameters

...
  "mon_osd_down_out_interval":
"600",
 "mon_osd_down_out_subtree_limit": "rack",
 "mds_debug_subtrees": "false",
 "mon_osd_down_out_subtree_limit": "rack",
 "mon_osd_reporter_subtree_level": "host", 