since quincy i'm randomly getting authentication issues from clients to osds. symptom is qemu hangs, but when it happens, i can reproduce it using:

ceph tell osd.\* version

some - but only some - osds will never respond, but only to clients on _some_ hosts. the client gets stuck in a loop with this error

2023-03-14T10:09:38.492+0100 7f38f5d95700 1 --2- 10.180.10.36:0/329477069 >> [v2:10.180.10.24:6810/697584,v1:10.180.10.24:6811/697584] conn(0x7f38f0107990 0x7f38f0107d60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)

restarting the affected OSD helps for a few hours. in the osd log i see only

2023-03-14T09:27:27.801+0000 7fb79020a700 10 osd.4 114909 ms_handle_authentication session 0x55880cd58b40 client.admin h as caps osdcap[grant(*)] 'allow *' 2023-03-14T09:27:27.805+0000 7fb781a3c700 2 osd.4 114909 ms_handle_reset con 0x55880a7fec00 session 0x55880cd58b40

searching for this issue gives me people whos mon is dead, but i dont think "tell" is supposed to go through mon, beyond the initial listing, which succeeds. but here's the full auth log from mon anyway if it helps: 2023-03-14T09:34:48.847+0000 7fcc8a5c7700 10 In get_auth_session_handler for protocol 0 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 start_session entity_name=client.admin global_id=6751719 is_new_global_id=1 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx server client.admin: start_session server_challenge 20aa2b96857f41cf 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 start_session entity_name=client.admin global_id=6751722 is_new_global_id=1 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx server client.admin: start_session server_challenge 6066dd1200ddc855 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx server client.admin: handle_request get_auth_session_key for client.admin 2023-03-14T09:34:48.847+0000 7fcc84dbc700 20 cephx server client.admin: checking key: req.key=92ed7ea281e9ac0c expected_key=92ed7ea281e9ac0c 2023-03-14T09:34:48.847+0000 7fcc84dbc700 20 cephx server client.admin: checking old_ticket: secret_id=0 len=0, old_ticket_may_be_omitted=0 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx server client.admin: new global_id 6751719 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx: build_service_ticket_reply encoding 1 tickets with secret REDACTED== 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx: build_service_ticket service auth secret_id 160 ticket_info.ticket.name=client.admin ticket.global_id 6751719 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx keyserverdata: get_caps: name=client.admin 2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx keyserverdata: get_secret: num of caps=4 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx server client.admin: handle_request get_auth_session_key for client.admin 2023-03-14T09:34:48.847+0000 7fcc865bf700 20 cephx server client.admin: checking key: req.key=3c1f6182caf84073 expected_key=3c1f6182caf84073 2023-03-14T09:34:48.847+0000 7fcc865bf700 20 cephx server client.admin: checking old_ticket: secret_id=0 len=0, old_ticket_may_be_omitted=0 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx server client.admin: new global_id 6751722 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx: build_service_ticket_reply encoding 1 tickets with secret REDACTED== 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx: build_service_ticket service auth secret_id 160 ticket_info.ticket.name=client.admin ticket.global_id 6751722 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx keyserverdata: get_caps: name=client.admin 2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx keyserverdata: get_secret: num of caps=4 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 start_session entity_name=client.admin global_id=6751725 is_new_global_id=1 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: start_session server_challenge 22fa068f8da1fb28 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: handle_request get_auth_session_key for client.admin 2023-03-14T09:34:48.851+0000 7fcc84dbc700 20 cephx server client.admin: checking key: req.key=fc7fdedb8e669347 expected_key=fc7fdedb8e669347 2023-03-14T09:34:48.851+0000 7fcc84dbc700 20 cephx server client.admin: checking old_ticket: secret_id=0 len=0, old_ticket_may_be_omitted=0 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: new global_id 6751725 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket_reply encoding 1 tickets with secret REDACTED== 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket service auth secret_id 160 ticket_info.ticket.name=client.admin ticket.global_id 6751725 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_caps: name=client.admin 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_secret: num of caps=4 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: adding key for service mon 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: adding key for service osd 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_caps: name=client.admin 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_secret: num of caps=4 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server client.admin: adding key for service mgr 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_caps: name=client.admin 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata: get_secret: num of caps=4 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket_reply encoding 3 tickets with secret REDACTED== 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket service mon secret_id 11455 ticket_info.ticket.name=client.admin ticket.global_id 6751725 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket service osd secret_id 11455 ticket_info.ticket.name=client.admin ticket.global_id 6751725 2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx: build_service_ticket service mgr secret_id 11455 ticket_info.ticket.name=client.admin ticket.global_id 6751725 2023-03-14T09:34:48.851+0000 7fcc865bf700 0 mon.mon-yca4ceph@0(leader) e8 handle_command mon_command({"prefix": "osd ls"} v 0) v1 2023-03-14T09:34:48.851+0000 7fcc865bf700 0 log_channel(audit) log [DBG] : from='client.? 10.180.10.36:0/329477069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch -- +4916093821054