The other hosts are still online and the cluster only lost 1/3 of its services.
Am 16.09.2023 um 12:53 schrieb Eugen Block <eblock@nde.ag>:
I don’t have time to look into all the details, but I’m wondering how you seem to be able to start mgr services with the orchestrator if all mgr daemons are down. The orchestrator is a mgr module, so that’s a bit weird, isn’t it?
Zitat von Boris Behrens <bb@kervyn.de>:
Hi Eugen, the test-test cluster where we started with simple ceph and the adoption when straight forward are working fine.
But this test cluster was all over the place. We had an old running update via orchestrator which was still in the pipeline, the adoption process was stopped a year ago and now got picked up again, and so on and so forth.
But now we have it clean, at least we think it's clean.
After a reboot, the services are not available. I have to start the via ceph orch. root@0cc47a6df14e:~# systemctl list-units | grep ceph ceph-crash.service loaded active running Ceph crash dump collector ceph-fuse.target loaded active active ceph target allowing to start/stop all ceph-fuse@.service instances at once ceph-mds.target loaded active active ceph target allowing to start/stop all ceph-mds@.service instances at once ceph-mgr.target loaded active active ceph target allowing to start/stop all ceph-mgr@.service instances at once ceph-mon.target loaded active active ceph target allowing to start/stop all ceph-mon@.service instances at once ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once ceph-radosgw.target loaded active active ceph target allowing to start/stop all ceph-radosgw@.service instances at once ceph.target loaded active active All Ceph clusters and services root@0cc47a6df14e:~# ceph orch start mgr Scheduled to start mgr.0cc47a6df14e.nvjlcx on host '0cc47a6df14e' Scheduled to start mgr.0cc47a6df330.aznjao on host '0cc47a6df330' Scheduled to start mgr.0cc47aad8ce8.ifiydp on host '0cc47aad8ce8' root@0cc47a6df14e:~# ceph orch start mon Scheduled to start mon.0cc47a6df14e on host '0cc47a6df14e' Scheduled to start mon.0cc47a6df330 on host '0cc47a6df330' Scheduled to start mon.0cc47aad8ce8 on host '0cc47aad8ce8' root@0cc47a6df14e:~# ceph orch start osd.all-flash-over-1tb Scheduled to start osd.2 on host '0cc47a6df14e' Scheduled to start osd.5 on host '0cc47a6df14e' Scheduled to start osd.3 on host '0cc47a6df330' Scheduled to start osd.0 on host '0cc47a6df330' Scheduled to start osd.4 on host '0cc47aad8ce8' Scheduled to start osd.1 on host '0cc47aad8ce8' root@0cc47a6df14e:~# systemctl list-units | grep ceph ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@mgr.0cc47a6df14e.nvjlcx.service loaded active running Ceph mgr.0cc47a6df14e.nvjlcx for 03977a23-f00f-4bb0-b9a7-de57f40ba853 ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@mon.0cc47a6df14e.service loaded active running Ceph mon.0cc47a6df14e for 03977a23-f00f-4bb0-b9a7-de57f40ba853 ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.2.service loaded active running Ceph osd.2 for 03977a23-f00f-4bb0-b9a7-de57f40ba853 ceph-crash.service loaded active running Ceph crash dump collector system-ceph\x2d03977a23\x2df00f\x2d4bb0\x2db9a7\x2dde57f40ba853.slice loaded active active system-ceph\x2d03977a23\x2df00f\x2d4bb0\x2db9a7\x2dde57f40ba853.slice ceph-fuse.target loaded active active ceph target allowing to start/stop all ceph-fuse@.service instances at once ceph-mds.target loaded active active ceph target allowing to start/stop all ceph-mds@.service instances at once ceph-mgr.target loaded active active ceph target allowing to start/stop all ceph-mgr@.service instances at once ceph-mon.target loaded active active ceph target allowing to start/stop all ceph-mon@.service instances at once ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once ceph-radosgw.target loaded active active ceph target allowing to start/stop all ceph-radosgw@.service instances at once ceph.target loaded active active All Ceph clusters and services root@0cc47a6df14e:~# systemctl status ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@mgr.0cc47a6df14e.nvjlcx.service ● ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@mgr.0cc47a6df14e.nvjlcx.service - Ceph mgr.0cc47a6df14e.nvjlcx for 03977a23-f00f-4bb0-b9a7-de57f40ba853 Loaded: loaded (/etc/systemd/system/ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2023-09-16 09:18:53 UTC; 51s ago Process: 4828 ExecStartPre=/bin/rm -f /run/ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@mgr.0cc47a6df14e.nvjlcx.service-pid /run/ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@mgr.0cc47a6df1> Process: 4829 ExecStart=/bin/bash /var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/mgr.0cc47a6df14e.nvjlcx/unit.run (code=exited, status=0/SUCCESS) Main PID: 5132 (conmon) Tasks: 36 (limit: 309227) Memory: 512.0M CGroup: /system.slice/system-ceph\x2d03977a23\x2df00f\x2d4bb0\x2db9a7\x2dde57f40ba853.slice/ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@mgr.0cc47a6df14e.nvjlcx.service ├─container │ ├─5136 /dev/init -- /usr/bin/ceph-mgr -n mgr.0cc47a6df14e.nvjlcx -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log> │ └─5139 /usr/bin/ceph-mgr -n mgr.0cc47a6df14e.nvjlcx -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=fa> └─supervisor └─5132 /usr/libexec/podman/conmon --api-version 1 -c 0165b4f78867ad284cc65fbece46013e6547a2f3ecf99cc7ffb8b720f705ee66 -u 0165b4f78867ad284cc65fbece46013e6547a2f3ecf99cc7ff>
Sep 16 09:19:04 0cc47a6df14e.f00f.gridscale.dev ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df14e-nvjlcx[5132]: 2023-09-16T09:19:04.333+0000 7f4fcc0a91c0 -1 mgr[py] Module alert> Sep 16 09:19:04 0cc47a6df14e.f00f.gridscale.dev ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df14e-nvjlcx[5132]: 2023-09-16T09:19:04.501+0000 7f4fcc0a91c0 -1 mgr[py] Module iosta> Sep 16 09:19:05 0cc47a6df14e.f00f.gridscale.dev ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df14e-nvjlcx[5132]: 2023-09-16T09:19:05.249+0000 7f4fcc0a91c0 -1 mgr[py] Module orche> Sep 16 09:19:05 0cc47a6df14e.f00f.gridscale.dev ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df14e-nvjlcx[5132]: 2023-09-16T09:19:05.481+0000 7f4fcc0a91c0 -1 mgr[py] Module rbd_s> Sep 16 09:19:06 0cc47a6df14e.f00f.gridscale.dev ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df14e-nvjlcx[5132]: [16/Sep/2023:09:19:06] ENGINE Bus STARTING Sep 16 09:19:06 0cc47a6df14e.f00f.gridscale.dev ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df14e-nvjlcx[5132]: CherryPy Checker: Sep 16 09:19:06 0cc47a6df14e.f00f.gridscale.dev ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df14e-nvjlcx[5132]: The Application mounted at '' has an empty config. Sep 16 09:19:06 0cc47a6df14e.f00f.gridscale.dev ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df14e-nvjlcx[5132]: Sep 16 09:19:06 0cc47a6df14e.f00f.gridscale.dev ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df14e-nvjlcx[5132]: [16/Sep/2023:09:19:06] ENGINE Serving on http://:::9283 Sep 16 09:19:06 0cc47a6df14e.f00f.gridscale.dev ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-mgr-0cc47a6df14e-nvjlcx[5132]: [16/Sep/2023:09:19:06] ENGINE Bus STARTED
This seems to be the cephadm log:
cephadm ['adopt', '--style', 'legacy', '--name', 'osd.3'] 2023-09-15 11:32:44,290 7fef7b041740 INFO Pulling container image quay.io/ceph/ceph:v17... 2023-09-15 11:32:47,128 7fef7b041740 INFO Found online OSD at //var/lib/ceph/osd/ceph-3/fsid 2023-09-15 11:32:47,129 7fef7b041740 INFO objectstore_type is bluestore 2023-09-15 11:32:47,150 7fef7b041740 INFO Stopping old systemd unit ceph-osd@3... 2023-09-15 11:32:48,560 7fef7b041740 INFO Disabling old systemd unit ceph-osd@3... 2023-09-15 11:32:49,157 7fef7b041740 INFO Moving data... 2023-09-15 11:32:49,158 7fef7b041740 DEBUG move file '//var/lib/ceph/osd/ceph-3/require_osd_release' -> '/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/require_osd_release' 2023-09-15 11:32:49,158 7fef7b041740 DEBUG chown 167:167 `/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/require_osd_release` 2023-09-15 11:32:49,158 7fef7b041740 DEBUG symlink '/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/block' -> '/dev/ceph-66d3bb27-cd5c-4897-aa76-684bc46d1c8b/osd-block-4bfc2101-e9b2-468d-8f54-a05f080ebdfe' 2023-09-15 11:32:49,158 7fef7b041740 DEBUG move file '//var/lib/ceph/osd/ceph-3/ready' -> '/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/ready' 2023-09-15 11:32:49,159 7fef7b041740 DEBUG chown 167:167 `/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/ready` 2023-09-15 11:32:49,159 7fef7b041740 DEBUG move file '//var/lib/ceph/osd/ceph-3/type' -> '/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/type' 2023-09-15 11:32:49,159 7fef7b041740 DEBUG chown 167:167 `/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/type` 2023-09-15 11:32:49,159 7fef7b041740 DEBUG move file '//var/lib/ceph/osd/ceph-3/fsid' -> '/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/fsid' 2023-09-15 11:32:49,159 7fef7b041740 DEBUG chown 167:167 `/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/fsid` 2023-09-15 11:32:49,160 7fef7b041740 DEBUG move file '//var/lib/ceph/osd/ceph-3/ceph_fsid' -> '/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/ceph_fsid' 2023-09-15 11:32:49,160 7fef7b041740 DEBUG chown 167:167 `/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/ceph_fsid` 2023-09-15 11:32:49,160 7fef7b041740 DEBUG move file '//var/lib/ceph/osd/ceph-3/keyring' -> '/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/keyring' 2023-09-15 11:32:49,160 7fef7b041740 DEBUG chown 167:167 `/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/keyring` 2023-09-15 11:32:49,160 7fef7b041740 DEBUG move file '//var/lib/ceph/osd/ceph-3/whoami' -> '/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/whoami' 2023-09-15 11:32:49,161 7fef7b041740 DEBUG chown 167:167 `/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/whoami` 2023-09-15 11:32:49,161 7fef7b041740 DEBUG Remove dir `//var/lib/ceph/osd/ceph-3` 2023-09-15 11:32:49,166 7fef7b041740 INFO Chowning content... 2023-09-15 11:32:49,171 7fef7b041740 DEBUG chown: stdout changed ownership of '/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/block' from root:root to 167:167 2023-09-15 11:32:49,172 7fef7b041740 INFO Chowning /var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/block... 2023-09-15 11:32:49,172 7fef7b041740 INFO Disabling host unit ceph-volume@ lvm unit... 2023-09-15 11:32:49,649 7fef7b041740 DEBUG systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-3-4bfc2101-e9b2-468d-8f54-a05f080ebdfe.service. 2023-09-15 11:32:49,650 7fef7b041740 DEBUG copy file `//etc/ceph/ceph.conf` -> `/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/config` 2023-09-15 11:32:49,650 7fef7b041740 DEBUG chown 167:167 `/var/lib/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/osd.3/config` 2023-09-15 11:32:49,650 7fef7b041740 INFO Moving logs... 2023-09-15 11:32:49,651 7fef7b041740 DEBUG move file '//var/log/ceph/ceph-osd.3.log' -> '/var/log/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/ceph-osd.3.log' 2023-09-15 11:32:49,651 7fef7b041740 DEBUG chown 167:167 `/var/log/ceph/03977a23-f00f-4bb0-b9a7-de57f40ba853/ceph-osd.3.log` 2023-09-15 11:32:49,651 7fef7b041740 INFO Creating new units... 2023-09-15 11:32:50,803 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.d/10-console-messages.conf ... 2023-09-15 11:32:50,803 7fef7b041740 DEBUG sysctl: stdout kernel.printk = 4 4 1 7 2023-09-15 11:32:50,803 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.d/10-ipv6-privacy.conf ... 2023-09-15 11:32:50,803 7fef7b041740 DEBUG sysctl: stdout net.ipv6.conf.all.use_tempaddr = 2 2023-09-15 11:32:50,803 7fef7b041740 DEBUG sysctl: stdout net.ipv6.conf.default.use_tempaddr = 2 2023-09-15 11:32:50,803 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.d/10-kernel-hardening.conf ... 2023-09-15 11:32:50,803 7fef7b041740 DEBUG sysctl: stdout kernel.kptr_restrict = 1 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.d/10-link-restrictions.conf ... 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout fs.protected_hardlinks = 1 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout fs.protected_symlinks = 1 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.d/10-magic-sysrq.conf ... 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout kernel.sysrq = 176 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.d/10-network-security.conf ... 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout net.ipv4.conf.default.rp_filter = 2 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout net.ipv4.conf.all.rp_filter = 2 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.d/10-ptrace.conf ... 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout kernel.yama.ptrace_scope = 1 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.d/10-zeropage.conf ... 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout vm.mmap_min_addr = 65536 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.d/30-ceph-osd.conf ... 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout fs.aio-max-nr = 1048576 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout kernel.pid_max = 4194304 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-coredump.conf ... 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-default.conf ... 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout net.ipv4.conf.default.promote_secondaries = 1 2023-09-15 11:32:50,804 7fef7b041740 DEBUG sysctl: stdout net.ipv4.ping_group_range = 0 2147483647 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout net.core.default_qdisc = fq_codel 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout fs.protected_regular = 1 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout fs.protected_fifos = 1 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-pid-max.conf ... 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout kernel.pid_max = 4194304 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.d/90-ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd.conf ... 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout fs.aio-max-nr = 1048576 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout kernel.pid_max = 4194304 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.d/99-sysctl.conf ... 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/protect-links.conf ... 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout fs.protected_fifos = 1 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout fs.protected_hardlinks = 1 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout fs.protected_regular = 2 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout fs.protected_symlinks = 1 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stdout * Applying /etc/sysctl.conf ... 2023-09-15 11:32:50,805 7fef7b041740 DEBUG sysctl: stderr sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument 2023-09-15 11:32:51,469 7fef7b041740 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.3 2023-09-15 11:32:51,469 7fef7b041740 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.3.service: Unit ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.3.service not loaded. 2023-09-15 11:32:51,954 7fef7b041740 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853.target.wants/ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.3.service → /etc/systemd/system/ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@.service. 2023-09-15 11:32:54,331 7fef7b041740 DEBUG firewalld does not appear to be present
Am Sa., 16. Sept. 2023 um 10:25 Uhr schrieb Eugen Block <eblock@nde.ag>:
That sounds a bit strange to me, because all clusters we adopted so far successfully converted the previous systemd-units into systemd units targeting the pods. This process also should have been logged (stdout, probably in the cephadm.log as well), resulting in "enabled" systemd units. Can you paste the output of 'systemctl status ceph-<FSID>@mon.<MON>'? If you have it, please also share the logs from the adoption process. What I did notice in a test cluster a while ago was that I had to reboot a node where I had to "play around" a bit with removed and redeployed osd containers. At some point they didn't react to systemctl commands anymore, but a reboot fixed that. But I haven't seen that in a production cluster yet, so some more details would be useful.
Zitat von Boris Behrens <bb@kervyn.de>:
Hi, is there a way to have the pods start again after reboot? Currently I need to start them by hand via ceph orch start mon/mgr/osd/...
I imagine this will lead to a lot of headache when the ceph cluster gets a powercycle and the mon pods will not start automatically.
I've spun up a test cluster and there the pods start very fast. On the legacy test cluster, which got adopted to cephadm, it does not.
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.