- ceph-users - lists.ceph.io

Strange (incorrect?) upmap entries in OSD map
by Andras Pataki 16 Jun '21

16 Jun '21

I've been working on some improvements to our large cluster's space balancing, when I noticed that sometimes the OSD maps have strange upmap entries. Here is an example on a clean cluster (PGs are active+clean): { "pgid": "1.1cb7", ... "up": [ 891, 170, 1338 ], "acting": [ 891, 170, 1338 ], ... }, with an upmap entry: pg_upmap_items 1.1cb7 [170,891] this would make the "up" list [ 170, 170, 1338 ], which isn't allowed. So the cluster just seems to ignore this upmap. When I remove the upmap, nothing changes in the PG state, and I can even re-insert it (without any effect). Any ideas why this upmap doesn't simply get rejected/removed? However, if I were to insert an upmap [170, 892], it gets rejected correctly (since 891 and 892 are on the same host - violating crush rules). Any insights would be helpful, Andras

1 0

ceph osd df return null
by julien lenseigne 16 Jun '21

16 Jun '21

Hi, when i do ceph osd df, some osd returns null size. For example : 0 hdd 7.27699 1.00000 0B 0B 0B 0B 0B 0B 0 0 29 1 hdd 7.27698 1.00000 0B 0B 0B 0B 0B 0B 0 0 23 11 ssd 0.50000 1.00000 0B 0B 0B 0B 0B 0B 0 0 49 2 hdd 7.27699 1.00000 0B 0B 0B 0B 0B 0B 0 0 30 4 hdd 7.27699 1.00000 0B 0B 0B 0B 0B 0B 0 0 37 12 ssd 0.50000 1.00000 0B 0B 0B 0B 0B 0B 0 0 39 7 hdd 24.55698 1.00000 24.6TiB 663GiB 663GiB 0B 0B 23.9TiB 2.64 0.79 302 42 hdd 10.69179 1.00000 10.7TiB 346GiB 344GiB 30.0MiB 1.13GiB 10.4TiB 3.16 0.95 98 43 hdd 10.69179 1.00000 10.7TiB 357GiB 355GiB 49.1MiB 1.17GiB 10.3TiB 3.26 0.98 131 44 hdd 10.69179 1.00000 10.7TiB 298GiB 297GiB 30.4MiB 1.05GiB 10.4TiB 2.72 0.81 92 45 hdd 10.69179 1.00000 10.7TiB 342GiB 341GiB 26.6MiB 1.11GiB 10.4TiB 3.12 0.94 105 40 ssd 0.87270 1.00000 894GiB 180GiB 179GiB 33.5MiB 990MiB 714GiB 20.10 6.02 53 41 ssd 0.87270 1.00000 894GiB 254GiB 253GiB 39.0MiB 1.00GiB 640GiB 28.38 8.50 61 46 ssd 0.87270 1.00000 894GiB 255GiB 254GiB 26.8MiB 1.03GiB 639GiB 28.55 8.55 76 3 hdd 24.55699 1.00000 24.6TiB 537GiB 536GiB 67.9MiB 1.36GiB 24.0TiB 2.14 0.64 252 do you know where it might come from ? Thanks. -- Julien Lenseigne Responsable informatique LMD Tel: 0169335172 Ecole Polytechnique, route de saclay, 91128 Palaiseau Bat 83 - Bureau 83.30.13

2 1

Drop old SDD / HDD Host crushmap rules
by Denny Fuchs 16 Jun '21

16 Jun '21

Hello, i have from the beginning on one DC very old crush map rules, to split HDD and SSD disks. It is obsolete since Luminous and I want to drop them: # ceph osd crush rule ls replicated_rule fc-r02-ssdpool fc-r02-satapool fc-r02-ssd ================= [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "fc-r02-ssdpool", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -15, "item_name": "r02-ssds" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "fc-r02-satapool", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -16, "item_name": "r02-sata" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 3, "rule_name": "fc-r02-ssd", "ruleset": 3, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -4, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ========================== # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -14 0 root sata -18 0 datacenter fc-sata -16 0 rack r02-sata -13 18.55234 root ssds -17 18.55234 datacenter fc-ssds -15 18.55234 rack r02-ssds -6 3.09206 host fc-r02-ceph-osd-01 41 nvme 0.36388 osd.41 up 1.00000 1.00000 0 ssd 0.45470 osd.0 up 1.00000 1.00000 1 ssd 0.45470 osd.1 up 1.00000 1.00000 2 ssd 0.45470 osd.2 up 1.00000 1.00000 3 ssd 0.45470 osd.3 up 1.00000 1.00000 4 ssd 0.45470 osd.4 up 1.00000 1.00000 5 ssd 0.45470 osd.5 up 1.00000 1.00000 -2 3.09206 host fc-r02-ceph-osd-02 36 nvme 0.36388 osd.36 up 1.00000 1.00000 6 ssd 0.45470 osd.6 up 1.00000 1.00000 7 ssd 0.45470 osd.7 up 1.00000 1.00000 8 ssd 0.45470 osd.8 up 1.00000 1.00000 9 ssd 0.45470 osd.9 up 1.00000 1.00000 10 ssd 0.45470 osd.10 up 1.00000 1.00000 29 ssd 0.45470 osd.29 up 1.00000 1.00000 -5 3.45593 host fc-r02-ceph-osd-03 38 nvme 0.36388 osd.38 up 1.00000 1.00000 40 nvme 0.36388 osd.40 up 1.00000 1.00000 11 ssd 0.45470 osd.11 up 1.00000 1.00000 12 ssd 0.45470 osd.12 up 1.00000 1.00000 13 ssd 0.45470 osd.13 up 1.00000 1.00000 14 ssd 0.45470 osd.14 up 1.00000 1.00000 15 ssd 0.45470 osd.15 up 1.00000 1.00000 16 ssd 0.45470 osd.16 up 1.00000 1.00000 -9 3.09206 host fc-r02-ceph-osd-04 37 nvme 0.36388 osd.37 up 1.00000 1.00000 30 ssd 0.45470 osd.30 up 1.00000 1.00000 31 ssd 0.45470 osd.31 up 1.00000 1.00000 32 ssd 0.45470 osd.32 up 1.00000 1.00000 33 ssd 0.45470 osd.33 up 1.00000 1.00000 34 ssd 0.45470 osd.34 up 1.00000 1.00000 35 ssd 0.45470 osd.35 up 1.00000 1.00000 -11 2.72818 host fc-r02-ceph-osd-05 17 ssd 0.45470 osd.17 up 1.00000 1.00000 18 ssd 0.45470 osd.18 up 1.00000 1.00000 19 ssd 0.45470 osd.19 up 1.00000 1.00000 20 ssd 0.45470 osd.20 up 1.00000 1.00000 21 ssd 0.45470 osd.21 up 1.00000 1.00000 22 ssd 0.45470 osd.22 up 1.00000 1.00000 -25 3.09206 host fc-r02-ceph-osd-06 39 nvme 0.36388 osd.39 up 1.00000 1.00000 23 ssd 0.45470 osd.23 up 1.00000 1.00000 24 ssd 0.45470 osd.24 up 1.00000 1.00000 25 ssd 0.45470 osd.25 up 1.00000 1.00000 26 ssd 0.45470 osd.26 up 1.00000 1.00000 27 ssd 0.45470 osd.27 up 1.00000 1.00000 28 ssd 0.45470 osd.28 up 1.00000 1.00000 -1 0 root default =================================================== the question is now: what is the best way, to drop them and use the defaults from Ceph. Any suggestions ? cu denny

1 0

RADOSGW Keystone integration - S3 bucket policies targeting not just other tenants / projects ?
by Christian Rohmann 16 Jun '21

16 Jun '21

Hallo Ceph-Users, I've been wondering about the state of OpenStack Keystone Auth in RADOSGW. 1) Even though the general documentation on RADOSGW S3 bucket policies is a little "misleading" https://docs.ceph.com/en/latest/radosgw/bucketpolicy/#creation-and-removal in showing users being referred as Principal, the documentation about Keystone integration at https://docs.ceph.com/en/latest/radosgw/keystone/#integrating-with-openstac… clearly states, that "A Ceph Object Gateway user is mapped into a Keystone <tenant>"||. In the keystone authentication code it strictly only takes the project from the authenticating user: * https://github.com/ceph/ceph/blob/6ce6874bae8fbac8921f0bdfc3931371fc61d4ff/… * https://github.com/ceph/ceph/blob/6ce6874bae8fbac8921f0bdfc3931371fc61d4ff/… This is rather unfortunate as this renders the usually powerful S3 bucket policies to be rather basic with granting access to all users (with a certain role) of a project or more importantly all users of another project / tenant, as in using arn:aws:iam::$OS_REMOTE_PROJECT_ID:root as principal. Or am I just misreading anything here or is this really all that can be done if using native keystone auth? 2) There is a PR open implementing generic external authentication https://github.com/ceph/ceph/pull/34093 Apparently this seems to also address the lack of support for subusers for Keystone - if I understand this correctly I could then grant access to users arn:aws:iam::$OS_REMOTE_PROJECT_ID:$user Are there any plans on the roadmap to extend the functionality in regards to keystone as authentication backend? I know a similar question as been asked before (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/GY7VUKCQ5QU…) but unfortunately with no discussion / responses then. Regards Christian

1 0

libceph: monX session lost, hunting for new mon
by Magnus HAGDORN 16 Jun '21

16 Jun '21

Hi all, I know this came up before but I couldn't find a resolution. We get the error libceph: monX session lost, hunting for new mon a lot on our samba servers that reexport cephfs. A lot means more than once a minute. On other machines that are less busy we get it about every 10-30 minutes. We only use a single network for both client and backend traffic on bonded 10GE links. So, my questions are: is this expected and normal behaviour? how to track this problem down? Regards magnus The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

1 0

Strategy for add new osds
by Jorge JP 16 Jun '21

16 Jun '21

Hello, I have a ceph cluster with 5 nodes (1 hdd each node). I want to add 5 more drives (hdd) to expand my cluster. What is the best strategy for this? I will add each drive in each node but is a good strategy add one drive and wait to rebalance the data to new osd for add new osd? or maybe.. I should be add the 5 drives without wait rebalancing and ceph rebalancing the data to all new osd? Thank you.

5 4

Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG
by Joe Comeau 16 Jun '21

16 Jun '21

We also run with Dell VLT switches (40 GB) everything is active/active, so multiple paths as Andrew describes in his config Our config allows us: bring down one of the switches for upgrades bring down an iscsi gatway for patching all the while at least one path is up and servicing Thanks Joe >>> Andrew Walker-Brown <andrew_jbrown(a)hotmail.com> 6/15/2021 10:26 AM >>> With an unstable link/port you could see the issues you describe. Ping doesn’t have the packet rate for you to necessarily have a packet in transit at exactly the same time as the port fails temporarily. Iperf on the other hand could certainly show the issue, higher packet rate and more likely to have packets in flight at the time of a link fail...combined with packet loss/retries gives poor throughput. Depending on what you want to happen, there are a number of tuning options both on the switches and Linux. If you want the LAG to be down if any link fails, the you should be able to config this on the switches and/or Linux (minimum number of links = 2 if you have 2 links in the lag). You can also tune the link monitoring, how frequently the links are checked (e.g. miimon) etc. Bringing this value down from the default of 100ms may allow you to detect a link failure more quickly. But you then run into the chance if detecting a transient failure that wouldn’t have caused any issues....and the LAG becoming more unstable. Flapping/unstable links are the worst kind of situation. Ideally you’d pick that up quickly from monitoring/alerts and either fix immediately or take the link down until you can fix it. I run 2x10G from my hosts into separate switches (Dell S series – VLT between switches). Pulling a single interface has no impact on Ceph, any packet loss is tiny and we’re not exceeding 10G bandwidth per host. If you’re running 1G links and the LAG is already busy, a link failure could be causing slow writes to the host, just down to congestion...which then starts to impact the wider cluster based on how Ceph works. Just caveating the above with - I’m relatively new to Ceph myself.... Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: huxiaoyu(a)horebdata.cn<mailto:huxiaoyu@horebdata.cn> Sent: 15 June 2021 17:52 To: Serkan Çoban<mailto:cobanserkan@gmail.com> Cc: ceph-users<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Issues with Ceph network redundancy using L2 MC-LAG When i pull out the cable, then the bond is working properly. Does it mean that the port is somehow flapping? Ping can still work, but the iperf test yields very low results. huxiaoyu(a)horebdata.cn From: Serkan Çoban Date: 2021-06-15 18:47 To: huxiaoyu(a)horebdata.cn CC: ceph-users Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2 MC-LAG Do you observe the same behaviour when you pull a cable? Maybe a flapping port might cause this kind of behaviour, other than that you should't see any network disconnects. Are you sure about LACP configuration, what is the output of 'cat /proc/net/bonding/bond0' On Tue, Jun 15, 2021 at 7:19 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > > Dear Cephers, > > I encountered the following networking issue several times, and i wonder whether there is a solution for networking HA solution. > > We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to provide switch redundancy. On each host, we use 802.3ad, LACP > mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate. > > Did i miss something important here? and how to really achieve networking HA in Ceph cluster? > > best regards, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 4

Re: Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG
by Serkan Çoban 16 Jun '21

16 Jun '21

You cannot do much if the link is flapping or the cable is bad. Maybe you can write some rules to shut the port down on the switch if the error packet ratio goes up. I also remember there are some config on the switch side for link flapping. On Wed, Jun 16, 2021 at 10:57 AM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > > Is it true that MC-LAG and 803.2ad, by its default, are working on active-active. > > What else should i take care to ensure fault tolerance when one path is bad? > > best regards, > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: Joe Comeau > Date: 2021-06-15 23:44 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG > We also run with Dell VLT switches (40 GB) > everything is active/active, so multiple paths as Andrew describes in > his config > Our config allows us: > bring down one of the switches for upgrades > bring down an iscsi gatway for patching > all the while at least one path is up and servicing > Thanks Joe > > > >>> Andrew Walker-Brown <andrew_jbrown(a)hotmail.com> 6/15/2021 10:26 AM > >>> > With an unstable link/port you could see the issues you describe. Ping > doesn’t have the packet rate for you to necessarily have a packet in > transit at exactly the same time as the port fails temporarily. Iperf > on the other hand could certainly show the issue, higher packet rate and > more likely to have packets in flight at the time of a link > fail...combined with packet loss/retries gives poor throughput. > > Depending on what you want to happen, there are a number of tuning > options both on the switches and Linux. If you want the LAG to be down > if any link fails, the you should be able to config this on the switches > and/or Linux (minimum number of links = 2 if you have 2 links in the > lag). > > You can also tune the link monitoring, how frequently the links are > checked (e.g. miimon) etc. Bringing this value down from the default of > 100ms may allow you to detect a link failure more quickly. But you then > run into the chance if detecting a transient failure that wouldn’t have > caused any issues....and the LAG becoming more unstable. > > Flapping/unstable links are the worst kind of situation. Ideally you’d > pick that up quickly from monitoring/alerts and either fix immediately > or take the link down until you can fix it. > > I run 2x10G from my hosts into separate switches (Dell S series – VLT > between switches). Pulling a single interface has no impact on Ceph, > any packet loss is tiny and we’re not exceeding 10G bandwidth per host. > > If you’re running 1G links and the LAG is already busy, a link failure > could be causing slow writes to the host, just down to > congestion...which then starts to impact the wider cluster based on how > Ceph works. > > Just caveating the above with - I’m relatively new to Ceph myself.... > > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for > Windows 10 > > From: huxiaoyu(a)horebdata.cn<mailto:huxiaoyu@horebdata.cn> > Sent: 15 June 2021 17:52 > To: Serkan Çoban<mailto:cobanserkan@gmail.com> > Cc: ceph-users<mailto:ceph-users@ceph.io> > Subject: [ceph-users] Re: Issues with Ceph network redundancy using L2 > MC-LAG > > When i pull out the cable, then the bond is working properly. > > Does it mean that the port is somehow flapping? Ping can still work, > but the iperf test yields very low results. > > > > > > huxiaoyu(a)horebdata.cn > > From: Serkan Çoban > Date: 2021-06-15 18:47 > To: huxiaoyu(a)horebdata.cn > CC: ceph-users > Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2 > MC-LAG > Do you observe the same behaviour when you pull a cable? > Maybe a flapping port might cause this kind of behaviour, other than > that you should't see any network disconnects. > Are you sure about LACP configuration, what is the output of 'cat > /proc/net/bonding/bond0' > > On Tue, Jun 15, 2021 at 7:19 PM huxiaoyu(a)horebdata.cn > <huxiaoyu(a)horebdata.cn> wrote: > > > > Dear Cephers, > > > > I encountered the following networking issue several times, and i > wonder whether there is a solution for networking HA solution. > > > > We build ceph using L2 multi chassis link aggregation group (MC-LAG ) > to provide switch redundancy. On each host, we use 802.3ad, LACP > > mode for NIC redundancy. However, we observe several times, when a > single network port, either the cable, or the SFP+ optical module fails, > Ceph cluster is badly affected by networking, although in theory it > should be able to tolerate. > > > > Did i miss something important here? and how to really achieve > networking HA in Ceph cluster? > > > > best regards, > > > > Samuel > > > > > > > > > > huxiaoyu(a)horebdata.cn > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: cephfs mount problems with 5.11 kernel - not a ipv6 problem
by Ilya Dryomov 16 Jun '21

16 Jun '21

On Mon, May 3, 2021 at 12:27 PM Magnus Harlander <magnus(a)harlan.de> wrote: > > Am 03.05.21 um 12:25 schrieb Ilya Dryomov: > > ceph osd setmaxosd 10 > > Bingo! Mount works again. > > Veeeery strange things are going on here (-: > > Thanx a lot for now!! If I can help to track it down, please let me know. Good to know it helped! I'll think about this some more and probably plan to patch the kernel client to be less stringent and not choke on this sort of misconfiguration. Thanks, Ilya

3 6

Docs on Containerized Mon Maintenance
by Phil Merricks 16 Jun '21

16 Jun '21

Hey folks, I'm working through some basic ops drills, and noticed what I think is an inconsistency in the Cephadm Docs. Some Googling appears to show this is a known thing, but I didn't find a clear direction on cooking up a solution yet. On a cluster with 5 mons, 2 were abruptly removed when their host OS decided to do scheduled maintenance without asking first. Those hosts only had mons running on them (and mds/crash/node exporter), so I still have 3 mon quorum and the cluster is happy. It's not clear to me how I add these hosts back in as mons though. In the troubleshooting docs it describes bringing all mons down, then extracting a monmap. I tried this through various iterations of bringing all down, bringing one back up and entering the container; bringing all down and trying to use ceph-mon from cephadm shell and so on. I either got rocksdb lock issues presumably because a mon node was running, or an error that the path to the mon data didn't exist, presumably for the opposite reason. Is there guidance on the container-friendly way to perform the monmap maintenance? I did think that because I still have quorum, I could simply do ceph orch apply mon label:mon instead, but I am nervous this might upset my remaining mons. Looking at the ceph orch ls output I see: root@kida:/# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager 1/1 7m ago 2h count:1 crash 5/5 9m ago 2h * grafana 1/1 7m ago 2h count:1 mds.media 3/3 9m ago 2h thebends;okcomputer;amnesiac mgr 2/2 9m ago 2h count:2 mon 3/5 9m ago 2h label:mon node-exporter 5/5 9m ago 2h * osd.all-available-devices 5/10 9m ago 2h * prometheus 1/1 7m ago 2h count:1 root@kida:/# So is it expecting 2 more mons, or has it autoscaled down cleverly? Looking at ceph orch ps I see: root@kida:/# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE VERSION IMAGE ID CONTAINER ID alertmanager.kida kida *:9093,9094 running (2h) 8m ago 2h 0.20.0 0881eb8f169f 89c604455194 crash.amnesiac amnesiac running (11h) 8m ago 11h 16.2.4 8d91d370c2b8 bff086c930db crash.kida kida running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 b0ac059be109 crash.kingoflimbs kingoflimbs running (13h) 8m ago 13h 16.2.4 8d91d370c2b8 b0955309a8b9 crash.okcomputer okcomputer running (2h) 10m ago 2h 16.2.4 8d91d370c2b8 a75cf65ef235 crash.thebends thebends running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 befe9c1015f3 grafana.kida kida *:3000 running (2h) 8m ago 2h 6.7.4 ae5c36c3d3cd f85747138299 mds.media.amnesiac.uujwlk amnesiac running (11h) 8m ago 2h 16.2.4 8d91d370c2b8 512a2fcc0f97 mds.media.okcomputer.nednib okcomputer running (2h) 10m ago 2h 16.2.4 8d91d370c2b8 10c6244a9308 mds.media.thebends.pqsfeb thebends running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 c1b75831a973 mgr.kida.kchysa kida *:9283 running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 602acc0d8df3 mgr.okcomputer.rjtrqw okcomputer *:8443,9283 running (2h) 10m ago 2h 16.2.4 8d91d370c2b8 605a8a25a604 mon.amnesiac amnesiac stopped 8m ago 2h <unknown> <unknown> <unknown> mon.kida kida running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 a441563a978d mon.kingoflimbs kingoflimbs stopped 8m ago 2h <unknown> <unknown> <unknown> mon.okcomputer okcomputer running (2h) 10m ago 2h 16.2.4 8d91d370c2b8 c4297efafe27 mon.thebends thebends running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 e2394d5f152b node-exporter.amnesiac amnesiac *:9100 running (11h) 8m ago 2h 0.18.1 e5a616e4b9cf da3c69057c4f node-exporter.kida kida *:9100 running (2h) 8m ago 2h 0.18.1 e5a616e4b9cf 5c9219a29257 node-exporter.kingoflimbs kingoflimbs *:9100 running (13h) 8m ago 2h 0.18.1 e5a616e4b9cf c2236491fb6e node-exporter.okcomputer okcomputer *:9100 running (2h) 10m ago 2h 0.18.1 e5a616e4b9cf 2e53a82eed32 node-exporter.thebends thebends *:9100 running (2h) 8m ago 2h 0.18.1 e5a616e4b9cf def6bdd359d6 osd.0 kida running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 c1419a29ddd8 osd.1 kida running (85m) 8m ago 2h 16.2.4 8d91d370c2b8 dcb172c628ec osd.2 thebends running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 4826e3da8d14 osd.3 okcomputer running (2h) 10m ago 2h 16.2.4 8d91d370c2b8 5424d437c270 osd.4 thebends running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 47e682c3727d prometheus.kida kida *:9095 running (2h) 8m ago 2h 2.18.1 de242295e225 4c8e7fdd89a8 root@kida:/# So those mon containers are still there, stopped. ceph orch daemon restart mon.amnesiac gives notice that a restart is scheduled on that mon. The container status updates in ceph orch ps to running, but version, image ID and container ID are <unknown> and I don't see that mon unit in any status output or log. cephadm unit --name mon.amnesiac restart --fsid yadda-yadda-yadda errors with daemon not found, it seems like the cephadm cli command is scoped to the daemons running on the same host it's being executed on, rather than cluster-wide like ceph orch. Any clues offered to further investigation are welcomed. Best regards Phil

1 0