Re: cephadm quincy bootstrap error

Hi Adam, I noticed a bug using cephadm quincy: when using cephadm from https://github.com/ceph/ceph/raw/quincy/src/cephadm/cephadm as written on https://docs.ceph.com/en/quincy/cephadm/install/ I get following error: root@ceph-01:~# cephadm bootstrap --ssh-user cephadm --ssh-public-key /home/cephadm/.ssh/cephadm.pub --ssh-private-key /home/cephadm/.ssh/cephadm --mon-ip 10.82.71.11 Verifying ssh connectivity ... Adding key to cephadm@localhost authorized_keys... key already in cephadm@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chrony.service is enabled and running Repeating the final host check... podman (/usr/bin/podman) version 4.3.1 is present systemctl is present lvcreate is present Unit chrony.service is enabled and running Host looks OK Cluster fsid: e460fb6a-c315-11ed-8abc-02000a52470b Verifying IP 10.82.71.11 port 3300 ... Verifying IP 10.82.71.11 port 6789 ... Mon IP `10.82.71.11` is in CIDR network `10.82.71.0/24` <http://10.82.71.0/24> Mon IP `10.82.71.11` is in CIDR network `10.82.71.0/24` <http://10.82.71.0/24> Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Pulling container image quay.io/ceph/ceph:v17... Ceph version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting mon public_network to 10.82.71.0/24 Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 9283 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Using provided ssh keys... Adding key to cephadm@localhost authorized_keys... key already in cephadm@localhost authorized_keys... Adding host ceph-01... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Deploying ceph-exporter service with default placement... Non-zero exit code 22 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=ceph-01 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e460fb6a-c315-11ed-8abc-02000a52470b:/var/log/ceph:z -v /tmp/ceph-tmpdj4jpaw8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpqrstws24:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch apply ceph-exporter /usr/bin/ceph: stderr Error EINVAL: Usage: /usr/bin/ceph: stderr ceph orch apply -i <yaml spec> [--dry-run] /usr/bin/ceph: stderr ceph orch apply <service_type> [--placement=<placement_string>] [--unmanaged] /usr/bin/ceph: stderr Traceback (most recent call last): File "/usr/local/bin/cephadm", line 9653, in <module> main() File "/usr/local/bin/cephadm", line 9641, in main r = ctx.func(ctx) ^^^^^^^^^^^^^ File "/usr/local/bin/cephadm", line 2205, in _default_image return func(ctx) ^^^^^^^^^ File "/usr/local/bin/cephadm", line 5774, in command_bootstrap prepare_ssh(ctx, cli, wait_for_mgr_restart) File "/usr/local/bin/cephadm", line 5275, in prepare_ssh cli(['orch', 'apply', t]) File "/usr/local/bin/cephadm", line 5714, in cli ).run(timeout=timeout, verbosity=verbosity) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/bin/cephadm", line 4144, in run out, _, _ = call_throws(self.ctx, self.run_cmd(), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/bin/cephadm", line 1853, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=ceph-01 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e460fb6a-c315-11ed-8abc-02000a52470b:/var/log/ceph:z -v /tmp/ceph-tmpdj4jpaw8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpqrstws24:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch apply ceph-exporter If I skip the monitoring stack with "--skip-monitoring-stack" boostraping works as expected. Using cephadm from https://github.com/ceph/ceph/raw/v17.2.5/src/cephadm/cephadm ( "v17.2.5" instead of "quincy" ) also works as expected. If you have any questions please get in touch. thanks! BR Tobi -- Mit freundlichen Grüßen Tobias Fischer Head of Ceph Clyso GmbH p: +49 89 21552391 12 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: tobias.fischer(a)clyso.com We are hiring: https://www.clyso.com/jobs/ --- Geschäftsführer: Dipl. Inf. (FH) Joachim Kraftmayer Unternehmenssitz: Utting am Ammersee Handelsregister beim Amtsgericht: Augsburg Handelsregister-Nummer: HRB 25866 USt. ID-Nr.: DE275430677