after upgrade to 16.2.3 16.2.4 and after adding few hdd's OSD's started to fail 1 by 1.
Hello, I will try to keep it sad and short :) :( PS sorry if this dublicate I tried post it from web also. Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds. After data migration for few hours, 1 SSD failed, then another and another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same host has SSD and HDD, but only SSD's are failing so I think this has to be balancing refiling or something bug and probably not upgrade bug. Cluster has been in pause for 4 hours and no more OSD's are failing. full trace https://pastebin.com/UxbfFYpb Now I m googling and learning but, Is there a way how to easily test lets say 15.2.XX version on osd without losing anything? Any help would be appreciated. Error start like this May 14 16:58:52 dragon-ball-radar systemd[1]: Starting Ceph osd.2 for 4e01640b-951b-4f75-8dca-0bad4faf1b11... May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.057836433 +0000 UTC m=+0.454352919 container create 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, GIT_BRANCH=HEAD, maintainer=D May 14 16:58:53 dragon-ball-radar systemd[1]: Started libcrun container. May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.3394116 +0000 UTC m=+0.735928098 container init 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, maintainer=Dimitri Savineau <dsav May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.446921192 +0000 UTC m=+0.843437626 container start 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, GIT_BRANCH=HEAD, org.label-sch May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.447050119 +0000 UTC m=+0.843566553 container attach 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, org.label-schema.name=CentOS May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 --path /var/lib/ceph/osd/ceph-2 --no-mon-config May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/ln -snf /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 /var/lib/ceph/osd/ceph-2/block May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 May 14 16:58:53 dragon-ball-radar bash[113558]: --> ceph-volume lvm activate successful for osd ID: 2 May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.8147653 +0000 UTC m=+1.211281741 container died 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate) May 14 16:58:55 dragon-ball-radar podman[113650]: 2021-05-14 16:58:55.044964534 +0000 UTC m=+2.441480996 container remove 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, CEPH_POINT_RELEASE=-16.2.4, R May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.594265612 +0000 UTC m=+0.369978347 container create 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, RELEASE=HEAD, org.label-schema.build-d May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.864589286 +0000 UTC m=+0.640302021 container init 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, org.label-schema.schema-version=1.0, GIT May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 set uid:gid to 167:167 (ceph:ceph) May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-osd, pid 2 May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 pidfile_write: ignore empty --pid-file May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 466 GiB May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) close May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.972267166 +0000 UTC m=+0.747979911 container start 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, ceph=True, GIT_REPO=https://github.com/ May 14 16:58:55 dragon-ball-radar bash[113558]: 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da May 14 16:58:55 dragon-ball-radar systemd[1]: Started Ceph osd.2 for 4e01640b-951b-4f75-8dca-0bad4faf1b11. May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.184+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) close May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 1 objectstore numa_node 0 May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 0 starting osd.2 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 address in networks '10.0.199.0/24' interfaces '' May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 address in networks '172.16.199.0/24' interfaces '' May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.452+0000 7fcf16aa2080 0 load: jerasure load: lrc load: isa May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) close
On Fri, May 14, 2021 at 10:47 AM Andrius Jurkus <andrius.jurkus@unishop.lt> wrote:
Hello, I will try to keep it sad and short :) :( PS sorry if this dublicate I tried post it from web also.
Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds. After data migration for few hours, 1 SSD failed, then another and another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same host has SSD and HDD, but only SSD's are failing so I think this has to be balancing refiling or something bug and probably not upgrade bug.
Cluster has been in pause for 4 hours and no more OSD's are failing.
full trace https://pastebin.com/UxbfFYpb
This looks very similar to https://tracker.ceph.com/issues/50656. Adding Igor for more ideas. Neha
Now I m googling and learning but, Is there a way how to easily test lets say 15.2.XX version on osd without losing anything?
Any help would be appreciated.
Error start like this
May 14 16:58:52 dragon-ball-radar systemd[1]: Starting Ceph osd.2 for 4e01640b-951b-4f75-8dca-0bad4faf1b11... May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.057836433 +0000 UTC m=+0.454352919 container create 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, GIT_BRANCH=HEAD, maintainer=D May 14 16:58:53 dragon-ball-radar systemd[1]: Started libcrun container. May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.3394116 +0000 UTC m=+0.735928098 container init 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, maintainer=Dimitri Savineau <dsav May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.446921192 +0000 UTC m=+0.843437626 container start 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, GIT_BRANCH=HEAD, org.label-sch May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.447050119 +0000 UTC m=+0.843566553 container attach 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, org.label-schema.name=CentOS May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 --path /var/lib/ceph/osd/ceph-2 --no-mon-config May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/ln -snf /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 /var/lib/ceph/osd/ceph-2/block May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 May 14 16:58:53 dragon-ball-radar bash[113558]: --> ceph-volume lvm activate successful for osd ID: 2 May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.8147653 +0000 UTC m=+1.211281741 container died 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate) May 14 16:58:55 dragon-ball-radar podman[113650]: 2021-05-14 16:58:55.044964534 +0000 UTC m=+2.441480996 container remove 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, CEPH_POINT_RELEASE=-16.2.4, R May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.594265612 +0000 UTC m=+0.369978347 container create 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, RELEASE=HEAD, org.label-schema.build-d May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.864589286 +0000 UTC m=+0.640302021 container init 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, org.label-schema.schema-version=1.0, GIT May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 set uid:gid to 167:167 (ceph:ceph) May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-osd, pid 2 May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 pidfile_write: ignore empty --pid-file May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 466 GiB May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) close May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.972267166 +0000 UTC m=+0.747979911 container start 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, ceph=True, GIT_REPO=https://github.com/ May 14 16:58:55 dragon-ball-radar bash[113558]: 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da May 14 16:58:55 dragon-ball-radar systemd[1]: Started Ceph osd.2 for 4e01640b-951b-4f75-8dca-0bad4faf1b11. May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.184+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) close May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 1 objectstore numa_node 0 May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 0 starting osd.2 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 address in networks '10.0.199.0/24' interfaces '' May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 address in networks '172.16.199.0/24' interfaces '' May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.452+0000 7fcf16aa2080 0 load: jerasure load: lrc load: isa May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) close _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This looks similar to #50656 indeed. Hopefully will fix that next week. Thanks, Igor On 5/14/2021 9:09 PM, Neha Ojha wrote:
On Fri, May 14, 2021 at 10:47 AM Andrius Jurkus <andrius.jurkus@unishop.lt> wrote:
Hello, I will try to keep it sad and short :) :( PS sorry if this dublicate I tried post it from web also.
Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds. After data migration for few hours, 1 SSD failed, then another and another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same host has SSD and HDD, but only SSD's are failing so I think this has to be balancing refiling or something bug and probably not upgrade bug.
Cluster has been in pause for 4 hours and no more OSD's are failing.
full trace https://pastebin.com/UxbfFYpb This looks very similar to https://tracker.ceph.com/issues/50656. Adding Igor for more ideas.
Neha
Now I m googling and learning but, Is there a way how to easily test lets say 15.2.XX version on osd without losing anything?
Any help would be appreciated.
Error start like this
May 14 16:58:52 dragon-ball-radar systemd[1]: Starting Ceph osd.2 for 4e01640b-951b-4f75-8dca-0bad4faf1b11... May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.057836433 +0000 UTC m=+0.454352919 container create 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, GIT_BRANCH=HEAD, maintainer=D May 14 16:58:53 dragon-ball-radar systemd[1]: Started libcrun container. May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.3394116 +0000 UTC m=+0.735928098 container init 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, maintainer=Dimitri Savineau <dsav May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.446921192 +0000 UTC m=+0.843437626 container start 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, GIT_BRANCH=HEAD, org.label-sch May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.447050119 +0000 UTC m=+0.843566553 container attach 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, org.label-schema.name=CentOS May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 --path /var/lib/ceph/osd/ceph-2 --no-mon-config May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/ln -snf /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 /var/lib/ceph/osd/ceph-2/block May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 May 14 16:58:53 dragon-ball-radar bash[113558]: --> ceph-volume lvm activate successful for osd ID: 2 May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.8147653 +0000 UTC m=+1.211281741 container died 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate) May 14 16:58:55 dragon-ball-radar podman[113650]: 2021-05-14 16:58:55.044964534 +0000 UTC m=+2.441480996 container remove 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, CEPH_POINT_RELEASE=-16.2.4, R May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.594265612 +0000 UTC m=+0.369978347 container create 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, RELEASE=HEAD, org.label-schema.build-d May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.864589286 +0000 UTC m=+0.640302021 container init 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, org.label-schema.schema-version=1.0, GIT May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 set uid:gid to 167:167 (ceph:ceph) May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-osd, pid 2 May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 pidfile_write: ignore empty --pid-file May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 466 GiB May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) close May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.972267166 +0000 UTC m=+0.747979911 container start 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, ceph=True, GIT_REPO=https://github.com/ May 14 16:58:55 dragon-ball-radar bash[113558]: 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da May 14 16:58:55 dragon-ball-radar systemd[1]: Started Ceph osd.2 for 4e01640b-951b-4f75-8dca-0bad4faf1b11. May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.184+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) close May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 1 objectstore numa_node 0 May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 0 starting osd.2 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 address in networks '10.0.199.0/24' interfaces '' May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 address in networks '172.16.199.0/24' interfaces '' May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.452+0000 7fcf16aa2080 0 load: jerasure load: lrc load: isa May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) close _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Today I had a very similar case: 2 nvme OSDs got down and out. I had freshly installed 16.2.1 version. Before failure disks were under some load ~1.5k read IOPS + ~600 write IOPS. When they failed, nothing helped. After every trial of resterting them I was finding in logs messages containing: bluefs _allocate unable to allocate The bluestore_allocator was default i.e. hybrid. I had changed it to bitmap just like in the issue mentioned by Neha Ojha (thanks), and OSDs got in and up. Now the disks are OK, but they are under very small load. Thus, I am not certain, whether the bitmap allocator is stable. Kind regards, -- Bartosz Lis On 5/15/2021 01:10:57 CEST Igor Fedotov wrote:
This looks similar to #50656 indeed.
Hopefully will fix that next week.
Thanks,
Igor
On 5/14/2021 9:09 PM, Neha Ojha wrote:
On Fri, May 14, 2021 at 10:47 AM Andrius Jurkus
<andrius.jurkus@unishop.lt> wrote:
Hello, I will try to keep it sad and short :) :( PS sorry if this dublicate I tried post it from web also.
Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds. After data migration for few hours, 1 SSD failed, then another and another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same host has SSD and HDD, but only SSD's are failing so I think this has to be balancing refiling or something bug and probably not upgrade bug.
Cluster has been in pause for 4 hours and no more OSD's are failing.
full trace https://pastebin.com/UxbfFYpb
This looks very similar to https://tracker.ceph.com/issues/50656. Adding Igor for more ideas. [---]
Thanks for fast responses. For now I m running with bluestore_allocator = bitmap and it doesnt crash anymore. (ps other messages didint appear for some reason) On 2021-05-15 02:10, Igor Fedotov wrote:
This looks similar to #50656 indeed.
Hopefully will fix that next week.
Thanks,
Igor
On 5/14/2021 9:09 PM, Neha Ojha wrote:
On Fri, May 14, 2021 at 10:47 AM Andrius Jurkus <andrius.jurkus@unishop.lt> wrote:
Hello, I will try to keep it sad and short :) :( PS sorry if this dublicate I tried post it from web also.
Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds. After data migration for few hours, 1 SSD failed, then another and another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same host has SSD and HDD, but only SSD's are failing so I think this has to be balancing refiling or something bug and probably not upgrade bug.
Cluster has been in pause for 4 hours and no more OSD's are failing.
full trace https://pastebin.com/UxbfFYpb This looks very similar to https://tracker.ceph.com/issues/50656. Adding Igor for more ideas.
Neha
Now I m googling and learning but, Is there a way how to easily test lets say 15.2.XX version on osd without losing anything?
Any help would be appreciated.
Error start like this
May 14 16:58:52 dragon-ball-radar systemd[1]: Starting Ceph osd.2 for 4e01640b-951b-4f75-8dca-0bad4faf1b11... May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.057836433 +0000 UTC m=+0.454352919 container create 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, GIT_BRANCH=HEAD, maintainer=D May 14 16:58:53 dragon-ball-radar systemd[1]: Started libcrun container. May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.3394116 +0000 UTC m=+0.735928098 container init 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, maintainer=Dimitri Savineau <dsav May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.446921192 +0000 UTC m=+0.843437626 container start 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, GIT_BRANCH=HEAD, org.label-sch May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.447050119 +0000 UTC m=+0.843566553 container attach 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, org.label-schema.name=CentOS May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 --path /var/lib/ceph/osd/ceph-2 --no-mon-config May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/ln -snf /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 /var/lib/ceph/osd/ceph-2/block May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 May 14 16:58:53 dragon-ball-radar bash[113558]: --> ceph-volume lvm activate successful for osd ID: 2 May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.8147653 +0000 UTC m=+1.211281741 container died 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate) May 14 16:58:55 dragon-ball-radar podman[113650]: 2021-05-14 16:58:55.044964534 +0000 UTC m=+2.441480996 container remove 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, CEPH_POINT_RELEASE=-16.2.4, R May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.594265612 +0000 UTC m=+0.369978347 container create 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, RELEASE=HEAD, org.label-schema.build-d May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.864589286 +0000 UTC m=+0.640302021 container init 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, org.label-schema.schema-version=1.0, GIT May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 set uid:gid to 167:167 (ceph:ceph) May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-osd, pid 2 May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 pidfile_write: ignore empty --pid-file May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 466 GiB May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) close May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.972267166 +0000 UTC m=+0.747979911 container start 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, ceph=True, GIT_REPO=https://github.com/ May 14 16:58:55 dragon-ball-radar bash[113558]: 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da May 14 16:58:55 dragon-ball-radar systemd[1]: Started Ceph osd.2 for 4e01640b-951b-4f75-8dca-0bad4faf1b11. May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.184+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) close May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 1 objectstore numa_node 0 May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 0 starting osd.2 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 address in networks '10.0.199.0/24' interfaces '' May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 address in networks '172.16.199.0/24' interfaces '' May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.452+0000 7fcf16aa2080 0 load: jerasure load: lrc load: isa May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) close _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Andrius Jurkus
-
Bartosz Lis
-
Igor Fedotov
-
Neha Ojha