Cephadm upgrade to Pacific problem
Hello, Cluster is 3 nodes Debian 10. Started cephadm upgrade on healthy 15.2.10 cluster. Managers were upgraded fine then first monitor went down for upgrade and never came back. Researching at the unit files container fails to run because of an error: root@host1:/var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1# cat unit.run set -e /usr/bin/install -d -m0770 -o 167 -g 167 /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6 # mon.host1 ! /usr/bin/docker rm -f ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 2> /dev/null /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 -e CONTAINER_IMAGE=ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -e NODE_NAME=host1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/run/ceph:z -v /var/log/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/log/ceph:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1:/var/lib/ceph/mon/ceph-host1:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -n mon.host1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true root@host1:/var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1# /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 -e CONTAINER_IMAGE=ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -e NODE_NAME=host1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/run/ceph:z -v /var/log/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/log/ceph:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1:/var/lib/ceph/mon/ceph-host1:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -n mon.host1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true /usr/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"/dev/init\": stat /dev/init: no such file or directory": unknown. Any suggestions how to resolve that ? Thank you.
Radoslav, I ran into the same. For Debian 10 - recent updates - you have to add 'cgroup_enable=memory swapaccount=1' to the kernel command line (/etc/default/grub). The reference I found said that Debian decided to disable this by default and make us turn it on if we want to run containers. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu On Wed, Apr 14, 2021 at 12:51 PM Radoslav Milanov < radoslav.milanov@gmail.com> wrote:
Hello,
Cluster is 3 nodes Debian 10. Started cephadm upgrade on healthy 15.2.10 cluster. Managers were upgraded fine then first monitor went down for upgrade and never came back. Researching at the unit files container fails to run because of an error:
root@host1:/var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1# cat unit.run
set -e /usr/bin/install -d -m0770 -o 167 -g 167 /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6 # mon.host1 ! /usr/bin/docker rm -f ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 2> /dev/null /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 -e CONTAINER_IMAGE=ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a
-e NODE_NAME=host1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/run/ceph:z -v /var/log/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/log/ceph:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/crash:/var/lib/ceph/crash:z
-v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1:/var/lib/ceph/mon/ceph-host1:z
-v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1/config:/etc/ceph/ceph.conf:z
-v /dev:/dev -v /run/udev:/run/udev ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a
-n mon.host1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true
root@host1:/var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1# /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 -e CONTAINER_IMAGE=ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a
-e NODE_NAME=host1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/run/ceph:z -v /var/log/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/log/ceph:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/crash:/var/lib/ceph/crash:z
-v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1:/var/lib/ceph/mon/ceph-host1:z
-v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1/config:/etc/ceph/ceph.conf:z
-v /dev:/dev -v /run/udev:/run/udev ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a
-n mon.host1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true
/usr/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"/dev/init\": stat /dev/init: no such file or directory": unknown.
Any suggestions how to resolve that ?
Thank you. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the pointer Dave, in my case though problem proved to be old docker version (18) provided by OS repos. Installing latest docker-ce from docker.com resolves the problem. It would be nice though if host was checked for compatibility before starting an upgrade. On 14.4.2021 г. 13:15 ч., Dave Hall wrote:
Radoslav,
I ran into the same. For Debian 10 - recent updates - you have to add 'cgroup_enable=memory swapaccount=1' to the kernel command line (/etc/default/grub). The reference I found said that Debian decided to disable this by default and make us turn it on if we want to run containers.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu <mailto:kdhall@binghamton.edu>
On Wed, Apr 14, 2021 at 12:51 PM Radoslav Milanov <radoslav.milanov@gmail.com <mailto:radoslav.milanov@gmail.com>> wrote:
Hello,
Cluster is 3 nodes Debian 10. Started cephadm upgrade on healthy 15.2.10 cluster. Managers were upgraded fine then first monitor went down for upgrade and never came back. Researching at the unit files container fails to run because of an error:
root@host1:/var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1#
cat unit.run
set -e /usr/bin/install -d -m0770 -o 167 -g 167 /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6 # mon.host1 ! /usr/bin/docker rm -f ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 2> /dev/null /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 -e CONTAINER_IMAGE=ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a
-e NODE_NAME=host1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/run/ceph:z -v /var/log/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/log/ceph:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/crash:/var/lib/ceph/crash:z
-v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1:/var/lib/ceph/mon/ceph-host1:z
-v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1/config:/etc/ceph/ceph.conf:z
-v /dev:/dev -v /run/udev:/run/udev ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a
-n mon.host1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true
root@host1:/var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1#
/usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 -e CONTAINER_IMAGE=ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a
-e NODE_NAME=host1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/run/ceph:z -v /var/log/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/log/ceph:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/crash:/var/lib/ceph/crash:z
-v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1:/var/lib/ceph/mon/ceph-host1:z
-v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1/config:/etc/ceph/ceph.conf:z
-v /dev:/dev -v /run/udev:/run/udev ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a
-n mon.host1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true
/usr/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"/dev/init\": stat /dev/init: no such file or directory": unknown.
Any suggestions how to resolve that ?
Thank you. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Hi Dave, El 14/4/21 a las 19:15, Dave Hall escribió:
Radoslav,
I ran into the same. For Debian 10 - recent updates - you have to add 'cgroup_enable=memory swapaccount=1' to the kernel command line (/etc/default/grub). The reference I found said that Debian decided to disable this by default and make us turn it on if we want to run containers. I find this quite strange. We have several updated Debian 10 servers running Docker containers (not related to Ceph) without needing this tuning. Docker is from Debian repos but we had a couple with docker.io version without issues.
Cheers Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/
Eneko, For clarification, this is the link I used to fix my particular docker issue: https://github.com/Debian/docker.io/blob/master/debian/README.Debian The specific issue for me was as follows: My cluster is running: - Debian 10 with DefaultRelease=buster-backports - Ceph packages from the Debian repo - Non-container installation (except for Grafana/Prometheus/Node Exporter) - Docker.io repo added by ceph-ansible The Grafana and Prometheus containers stopped working after running apt-get dist-upgrade (via ceph-ansible) to pick up the latest Debian Ceph packages . The systemd log messages were similar to those reported by Radoslav. A Google search led me to the link above. The suggested addition to the kernel command line fixed the issue. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu On Thu, Apr 15, 2021 at 4:07 AM Eneko Lacunza <elacunza@binovo.es> wrote:
Hi Dave,
El 14/4/21 a las 19:15, Dave Hall escribió:
Radoslav,
I ran into the same. For Debian 10 - recent updates - you have to add 'cgroup_enable=memory swapaccount=1' to the kernel command line (/etc/default/grub). The reference I found said that Debian decided to disable this by default and make us turn it on if we want to run containers. I find this quite strange. We have several updated Debian 10 servers running Docker containers (not related to Ceph) without needing this tuning. Docker is from Debian repos but we had a couple with docker.io version without issues.
Cheers
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dave, I see now what was the problem. Thanks a lot for the link. Cheers El 15/4/21 a las 14:55, Dave Hall escribió:
Eneko,
For clarification, this is the link I used to fix my particular docker issue:
https://github.com/Debian/docker.io/blob/master/debian/README.Debian <https://github.com/Debian/docker.io/blob/master/debian/README.Debian>
The specific issue for me was as follows:
My cluster is running:
* Debian 10 with DefaultRelease=buster-backports * Ceph packages from the Debian repo * Non-container installation (except for Grafana/Prometheus/Node Exporter) * Docker.io repo added by ceph-ansible
The Grafana and Prometheus containers stopped working after running apt-get dist-upgrade (via ceph-ansible) to pick up the latest Debian Ceph packages . The systemd log messages were similar to those reported by Radoslav. A Google search led me to the link above. The suggested addition to the kernel command line fixed the issue.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu <mailto:kdhall@binghamton.edu>
On Thu, Apr 15, 2021 at 4:07 AM Eneko Lacunza <elacunza@binovo.es <mailto:elacunza@binovo.es>> wrote:
Hi Dave,
El 14/4/21 a las 19:15, Dave Hall escribió: > Radoslav, > > I ran into the same. For Debian 10 - recent updates - you have to add > 'cgroup_enable=memory swapaccount=1' to the kernel command line > (/etc/default/grub). The reference I found said that Debian decided to > disable this by default and make us turn it on if we want to run containers. I find this quite strange. We have several updated Debian 10 servers running Docker containers (not related to Ceph) without needing this tuning. Docker is from Debian repos but we had a couple with docker.io <http://docker.io> version without issues.
Cheers
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es <https://www.binovo.es> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO <https://www.youtube.com/user/CANALBINOVO> https://www.linkedin.com/company/37269706/ <https://www.linkedin.com/company/37269706/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
EnekoLacunza Director Técnico | Zuzendari teknikoa Binovo IT Human Project 943 569 206 <tel:943 569 206> elacunza@binovo.es <mailto:elacunza@binovo.es> binovo.es <//binovo.es> Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun youtube <https://www.youtube.com/user/CANALBINOVO/> linkedin <https://www.linkedin.com/company/37269706/>
participants (3)
-
Dave Hall
-
Eneko Lacunza
-
Radoslav Milanov