14 May
2021
14 May
'21
10:16 p.m.
Hello, I will try to keep it sad and short :) :( PS sorry if this
dublicate I tried post it from web also.
Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds.
After data migration for few hours, 1 SSD failed, then another and
another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same
host has SSD and HDD, but only SSD's are failing so I think this has to
be balancing refiling or something bug and probably not upgrade bug.
Cluster has been in pause for 4 hours and no more OSD's are failing.
full trace
https://pastebin.com/UxbfFYpb
Now I m googling and learning but, Is there a way how to easily test
lets say 15.2.XX version on osd without losing anything?
Any help would be appreciated.
Error start like this
May 14 16:58:52 dragon-ball-radar systemd[1]: Starting Ceph osd.2 for
4e01640b-951b-4f75-8dca-0bad4faf1b11...
May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
16:58:53.057836433 +0000 UTC m=+0.454352919 container create
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
GIT_BRANCH=HEAD, maintainer=D
May 14 16:58:53 dragon-ball-radar systemd[1]: Started libcrun container.
May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
16:58:53.3394116 +0000 UTC m=+0.735928098 container init
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
maintainer=Dimitri Savineau <dsav
May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
16:58:53.446921192 +0000 UTC m=+0.843437626 container start
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
GIT_BRANCH=HEAD, org.label-sch
May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
16:58:53.447050119 +0000 UTC m=+0.843566553 container attach
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
org.label-schema.name=CentOS
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25
--path /var/lib/ceph/osd/ceph-2 --no-mon-config
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/ln -snf
/dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25
/var/lib/ceph/osd/ceph-2/block
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/chown -R ceph:ceph /dev/dm-1
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
May 14 16:58:53 dragon-ball-radar bash[113558]: --> ceph-volume lvm
activate successful for osd ID: 2
May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
16:58:53.8147653 +0000 UTC m=+1.211281741 container died
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate)
May 14 16:58:55 dragon-ball-radar podman[113650]: 2021-05-14
16:58:55.044964534 +0000 UTC m=+2.441480996 container remove
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
CEPH_POINT_RELEASE=-16.2.4, R
May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
16:58:55.594265612 +0000 UTC m=+0.369978347 container create
31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, RELEASE=HEAD,
org.label-schema.build-d
May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
16:58:55.864589286 +0000 UTC m=+0.640302021 container init
31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2,
org.label-schema.schema-version=1.0, GIT
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 set uid:gid to 167:167
(ceph:ceph)
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 ceph version 16.2.4
(3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process
ceph-osd, pid 2
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 pidfile_write: ignore empty
--pid-file
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.896+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
/var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
/var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1
bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size
3221225472 meta 0.45 kv 0.45 data 0.06
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
/var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
/var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluefs add_block_device bdev
1 path /var/lib/ceph/osd/ceph-2/block size 466 GiB
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
/var/lib/ceph/osd/ceph-2/block) close
May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
16:58:55.972267166 +0000 UTC m=+0.747979911 container start
31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, ceph=True,
GIT_REPO=https://github.com/
May 14 16:58:55 dragon-ball-radar bash[113558]:
31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
May 14 16:58:55 dragon-ball-radar systemd[1]: Started Ceph osd.2 for
4e01640b-951b-4f75-8dca-0bad4faf1b11.
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.184+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
/var/lib/ceph/osd/ceph-2/block) close
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.444+0000 7fcf16aa2080 1 objectstore numa_node 0
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.444+0000 7fcf16aa2080 0 starting osd.2 osd_data
/var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4
address in networks '10.0.199.0/24' interfaces ''
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4
address in networks '172.16.199.0/24' interfaces ''
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.452+0000 7fcf16aa2080 0 load: jerasure load: lrc
load: isa
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
/var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
/var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.456+0000 7fcf16aa2080 1
bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size
3221225472 meta 0.45 kv 0.45 data 0.06
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
/var/lib/ceph/osd/ceph-2/block) close
14 May
14 May
10:39 p.m.
On Fri, May 14, 2021 at 10:47 AM Andrius Jurkus
<andrius.jurkus(a)unishop.lt> wrote:
Hello, I will try to keep it sad and short :) :( PS sorry if this
dublicate I tried post it from web also.
Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds.
After data migration for few hours, 1 SSD failed, then another and
another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same
host has SSD and HDD, but only SSD's are failing so I think this has to
be balancing refiling or something bug and probably not upgrade bug.
Cluster has been in pause for 4 hours and no more OSD's are failing.
full trace
https://pastebin.com/UxbfFYpb
This looks very similar to https://tracker.ceph.com/issues/50656.
Adding Igor for more ideas.
Neha
Now I m googling and learning but, Is there a way how to easily test
lets say 15.2.XX version on osd without losing anything?
Any help would be appreciated.
Error start like this
May 14 16:58:52 dragon-ball-radar systemd[1]: Starting Ceph osd.2 for
4e01640b-951b-4f75-8dca-0bad4faf1b11...
May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
16:58:53.057836433 +0000 UTC m=+0.454352919 container create
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
GIT_BRANCH=HEAD, maintainer=D
May 14 16:58:53 dragon-ball-radar systemd[1]: Started libcrun container.
May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
16:58:53.3394116 +0000 UTC m=+0.735928098 container init
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
maintainer=Dimitri Savineau <dsav
May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
16:58:53.446921192 +0000 UTC m=+0.843437626 container start
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
GIT_BRANCH=HEAD, org.label-sch
May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
16:58:53.447050119 +0000 UTC m=+0.843566553 container attach
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
org.label-schema.name=CentOS
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25
--path /var/lib/ceph/osd/ceph-2 --no-mon-config
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/ln -snf
/dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25
/var/lib/ceph/osd/ceph-2/block
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/chown -R ceph:ceph /dev/dm-1
May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
/usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
May 14 16:58:53 dragon-ball-radar bash[113558]: --> ceph-volume lvm
activate successful for osd ID: 2
May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
16:58:53.8147653 +0000 UTC m=+1.211281741 container died
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate)
May 14 16:58:55 dragon-ball-radar podman[113650]: 2021-05-14
16:58:55.044964534 +0000 UTC m=+2.441480996 container remove
3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
CEPH_POINT_RELEASE=-16.2.4, R
May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
16:58:55.594265612 +0000 UTC m=+0.369978347 container create
31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, RELEASE=HEAD,
org.label-schema.build-d
May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
16:58:55.864589286 +0000 UTC m=+0.640302021 container init
31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2,
org.label-schema.schema-version=1.0, GIT
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 set uid:gid to 167:167
(ceph:ceph)
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 ceph version 16.2.4
(3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process
ceph-osd, pid 2
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 pidfile_write: ignore empty
--pid-file
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.896+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
/var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
/var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1
bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size
3221225472 meta 0.45 kv 0.45 data 0.06
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
/var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
/var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluefs add_block_device bdev
1 path /var/lib/ceph/osd/ceph-2/block size 466 GiB
May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
/var/lib/ceph/osd/ceph-2/block) close
May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
16:58:55.972267166 +0000 UTC m=+0.747979911 container start
31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, ceph=True,
GIT_REPO=https://github.com/
May 14 16:58:55 dragon-ball-radar bash[113558]:
31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
May 14 16:58:55 dragon-ball-radar systemd[1]: Started Ceph osd.2 for
4e01640b-951b-4f75-8dca-0bad4faf1b11.
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.184+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
/var/lib/ceph/osd/ceph-2/block) close
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.444+0000 7fcf16aa2080 1 objectstore numa_node 0
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.444+0000 7fcf16aa2080 0 starting osd.2 osd_data
/var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4
address in networks '10.0.199.0/24' interfaces ''
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4
address in networks '172.16.199.0/24' interfaces ''
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.452+0000 7fcf16aa2080 0 load: jerasure load: lrc
load: isa
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
/var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
/var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.456+0000 7fcf16aa2080 1
bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size
3221225472 meta 0.45 kv 0.45 data 0.06
May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
/var/lib/ceph/osd/ceph-2/block) close
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
15 May
15 May
3:40 a.m.
This looks similar to #50656 indeed.
Hopefully will fix that next week.
Thanks,
Igor
On 5/14/2021 9:09 PM, Neha Ojha wrote:
> On Fri, May 14, 2021 at 10:47 AM Andrius Jurkus
> <andrius.jurkus(a)unishop.lt> wrote:
>> Hello, I will try to keep it sad and short :) :( PS sorry if this
>> dublicate I tried post it from web also.
>>
>> Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds.
>> After data migration for few hours, 1 SSD failed, then another and
>> another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same
>> host has SSD and HDD, but only SSD's are failing so I think this has to
>> be balancing refiling or something bug and probably not upgrade bug.
>>
>> Cluster has been in pause for 4 hours and no more OSD's are failing.
>>
>> full trace
>> https://pastebin.com/UxbfFYpb
> This looks very similar to https://tracker.ceph.com/issues/50656.
> Adding Igor for more ideas.
>
> Neha
>
>> Now I m googling and learning but, Is there a way how to easily test
>> lets say 15.2.XX version on osd without losing anything?
>>
>> Any help would be appreciated.
>>
>> Error start like this
>>
>> May 14 16:58:52 dragon-ball-radar systemd[1]: Starting Ceph osd.2 for
>> 4e01640b-951b-4f75-8dca-0bad4faf1b11...
>> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
>> 16:58:53.057836433 +0000 UTC m=+0.454352919 container create
>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
>> GIT_BRANCH=HEAD, maintainer=D
>> May 14 16:58:53 dragon-ball-radar systemd[1]: Started libcrun container.
>> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
>> 16:58:53.3394116 +0000 UTC m=+0.735928098 container init
>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
>> maintainer=Dimitri Savineau <dsav
>> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
>> 16:58:53.446921192 +0000 UTC m=+0.843437626 container start
>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
>> GIT_BRANCH=HEAD, org.label-sch
>> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
>> 16:58:53.447050119 +0000 UTC m=+0.843566553 container attach
>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
>> org.label-schema.name=CentOS
>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>> /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
>>
/dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25
>> --path /var/lib/ceph/osd/ceph-2 --no-mon-config
>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>> /usr/bin/ln -snf
>>
/dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25
>> /var/lib/ceph/osd/ceph-2/block
>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>> /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block
>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>> /usr/bin/chown -R ceph:ceph /dev/dm-1
>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> May 14 16:58:53 dragon-ball-radar bash[113558]: --> ceph-volume lvm
>> activate successful for osd ID: 2
>> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
>> 16:58:53.8147653 +0000 UTC m=+1.211281741 container died
>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate)
>> May 14 16:58:55 dragon-ball-radar podman[113650]: 2021-05-14
>> 16:58:55.044964534 +0000 UTC m=+2.441480996 container remove
>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
>> CEPH_POINT_RELEASE=-16.2.4, R
>> May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
>> 16:58:55.594265612 +0000 UTC m=+0.369978347 container create
>> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, RELEASE=HEAD,
>> org.label-schema.build-d
>> May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
>> 16:58:55.864589286 +0000 UTC m=+0.640302021 container init
>> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2,
>> org.label-schema.schema-version=1.0, GIT
>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 set uid:gid to 167:167
>> (ceph:ceph)
>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 ceph version 16.2.4
>> (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process
>> ceph-osd, pid 2
>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 pidfile_write: ignore empty
>> --pid-file
>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
>> /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
>> /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
>> 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1
>> bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size
>> 3221225472 meta 0.45 kv 0.45 data 0.06
>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
>> /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
>> /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
>> 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluefs add_block_device bdev
>> 1 path /var/lib/ceph/osd/ceph-2/block size 466 GiB
>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
>> /var/lib/ceph/osd/ceph-2/block) close
>> May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
>> 16:58:55.972267166 +0000 UTC m=+0.747979911 container start
>> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, ceph=True,
>> GIT_REPO=https://github.com/
>> May 14 16:58:55 dragon-ball-radar bash[113558]:
>> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
>> May 14 16:58:55 dragon-ball-radar systemd[1]: Started Ceph osd.2 for
>> 4e01640b-951b-4f75-8dca-0bad4faf1b11.
>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:56.184+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
>> /var/lib/ceph/osd/ceph-2/block) close
>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 1 objectstore numa_node 0
>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 0 starting osd.2 osd_data
>> /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal
>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4
>> address in networks '10.0.199.0/24' interfaces ''
>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4
>> address in networks '172.16.199.0/24' interfaces ''
>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:56.452+0000 7fcf16aa2080 0 load: jerasure load: lrc
>> load: isa
>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
>> /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
>> /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
>> 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1
>> bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size
>> 3221225472 meta 0.45 kv 0.45 data 0.06
>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
>> /var/lib/ceph/osd/ceph-2/block) close
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
16 May
16 May
4:21 a.m.
Hi,
Today I had a very similar case: 2 nvme OSDs got down and out. I had freshly
installed 16.2.1 version. Before failure disks were under some load ~1.5k read
IOPS + ~600 write IOPS. When they failed, nothing helped. After every trial of
resterting them I was finding in logs messages containing:
bluefs _allocate unable to allocate
The bluestore_allocator was default i.e. hybrid. I had changed it to bitmap
just like in the issue mentioned by Neha Ojha (thanks), and OSDs got in and
up.
Now the disks are OK, but they are under very small load. Thus, I am not
certain, whether the bitmap allocator is stable.
Kind regards,
--
Bartosz Lis
On 5/15/2021 01:10:57 CEST Igor Fedotov wrote:
This looks similar to #50656 indeed.
Hopefully will fix that next week.
Thanks,
Igor
On 5/14/2021 9:09 PM, Neha Ojha wrote:
> On Fri, May 14, 2021 at 10:47 AM Andrius Jurkus
>
> <andrius.jurkus(a)unishop.lt> wrote:
>> Hello, I will try to keep it sad and short :) :( PS sorry if this
>> dublicate I tried post it from web also.
>>
>> Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds.
>> After data migration for few hours, 1 SSD failed, then another and
>> another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same
>> host has SSD and HDD, but only SSD's are failing so I think this has to
>> be balancing refiling or something bug and probably not upgrade bug.
>>
>> Cluster has been in pause for 4 hours and no more OSD's are failing.
>>
>> full trace
>> https://pastebin.com/UxbfFYpb
>
> This looks very similar to https://tracker.ceph.com/issues/50656.
> Adding Igor for more ideas.
[---]
17 May
17 May
8:37 a.m.
Thanks for fast responses.
For now I m running with
bluestore_allocator = bitmap
and it doesnt crash anymore. (ps other messages didint appear for some
reason)
On 2021-05-15 02:10, Igor Fedotov wrote:
> This looks similar to #50656 indeed.
>
> Hopefully will fix that next week.
>
>
> Thanks,
>
> Igor
>
> On 5/14/2021 9:09 PM, Neha Ojha wrote:
>> On Fri, May 14, 2021 at 10:47 AM Andrius Jurkus
>> <andrius.jurkus(a)unishop.lt> wrote:
>>> Hello, I will try to keep it sad and short :) :( PS sorry if this
>>> dublicate I tried post it from web also.
>>>
>>> Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds.
>>> After data migration for few hours, 1 SSD failed, then another and
>>> another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same
>>> host has SSD and HDD, but only SSD's are failing so I think this has
>>> to
>>> be balancing refiling or something bug and probably not upgrade bug.
>>>
>>> Cluster has been in pause for 4 hours and no more OSD's are failing.
>>>
>>> full trace
>>> https://pastebin.com/UxbfFYpb
>> This looks very similar to https://tracker.ceph.com/issues/50656.
>> Adding Igor for more ideas.
>>
>> Neha
>>
>>> Now I m googling and learning but, Is there a way how to easily test
>>> lets say 15.2.XX version on osd without losing anything?
>>>
>>> Any help would be appreciated.
>>>
>>> Error start like this
>>>
>>> May 14 16:58:52 dragon-ball-radar systemd[1]: Starting Ceph osd.2 for
>>> 4e01640b-951b-4f75-8dca-0bad4faf1b11...
>>> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
>>> 16:58:53.057836433 +0000 UTC m=+0.454352919 container create
>>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
>>> GIT_BRANCH=HEAD, maintainer=D
>>> May 14 16:58:53 dragon-ball-radar systemd[1]: Started libcrun
>>> container.
>>> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
>>> 16:58:53.3394116 +0000 UTC m=+0.735928098 container init
>>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
>>> maintainer=Dimitri Savineau <dsav
>>> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
>>> 16:58:53.446921192 +0000 UTC m=+0.843437626 container start
>>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
>>> GIT_BRANCH=HEAD, org.label-sch
>>> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
>>> 16:58:53.447050119 +0000 UTC m=+0.843566553 container attach
>>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
>>> org.label-schema.name=CentOS
>>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>>> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>>> /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
>>>
/dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25
>>> --path /var/lib/ceph/osd/ceph-2 --no-mon-config
>>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>>> /usr/bin/ln -snf
>>>
/dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25
>>> /var/lib/ceph/osd/ceph-2/block
>>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>>> /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block
>>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>>> /usr/bin/chown -R ceph:ceph /dev/dm-1
>>> May 14 16:58:53 dragon-ball-radar bash[113558]: Running command:
>>> /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>>> May 14 16:58:53 dragon-ball-radar bash[113558]: --> ceph-volume lvm
>>> activate successful for osd ID: 2
>>> May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14
>>> 16:58:53.8147653 +0000 UTC m=+1.211281741 container died
>>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate)
>>> May 14 16:58:55 dragon-ball-radar podman[113650]: 2021-05-14
>>> 16:58:55.044964534 +0000 UTC m=+2.441480996 container remove
>>> 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd
>>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate,
>>> CEPH_POINT_RELEASE=-16.2.4, R
>>> May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
>>> 16:58:55.594265612 +0000 UTC m=+0.369978347 container create
>>> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
>>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, RELEASE=HEAD,
>>> org.label-schema.build-d
>>> May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
>>> 16:58:55.864589286 +0000 UTC m=+0.640302021 container init
>>> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
>>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2,
>>> org.label-schema.schema-version=1.0, GIT
>>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 set uid:gid to 167:167
>>> (ceph:ceph)
>>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 ceph version 16.2.4
>>> (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process
>>> ceph-osd, pid 2
>>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 pidfile_write: ignore
>>> empty
>>> --pid-file
>>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:55.896+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
>>> /var/lib/ceph/osd/ceph-2/block) open path
>>> /var/lib/ceph/osd/ceph-2/block
>>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
>>> /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
>>> 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
>>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1
>>> bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size
>>> 3221225472 meta 0.45 kv 0.45 data 0.06
>>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
>>> /var/lib/ceph/osd/ceph-2/block) open path
>>> /var/lib/ceph/osd/ceph-2/block
>>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
>>> /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
>>> 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
>>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluefs add_block_device
>>> bdev
>>> 1 path /var/lib/ceph/osd/ceph-2/block size 466 GiB
>>> May 14 16:58:55 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00
>>> /var/lib/ceph/osd/ceph-2/block) close
>>> May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14
>>> 16:58:55.972267166 +0000 UTC m=+0.747979911 container start
>>> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
>>>
(image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,
>>> name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, ceph=True,
>>> GIT_REPO=https://github.com/
>>> May 14 16:58:55 dragon-ball-radar bash[113558]:
>>> 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da
>>> May 14 16:58:55 dragon-ball-radar systemd[1]: Started Ceph osd.2 for
>>> 4e01640b-951b-4f75-8dca-0bad4faf1b11.
>>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:56.184+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800
>>> /var/lib/ceph/osd/ceph-2/block) close
>>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 1 objectstore numa_node 0
>>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 0 starting osd.2 osd_data
>>> /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal
>>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4
>>> address in networks '10.0.199.0/24' interfaces ''
>>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4
>>> address in networks '172.16.199.0/24' interfaces ''
>>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:56.452+0000 7fcf16aa2080 0 load: jerasure load: lrc
>>> load: isa
>>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
>>> /var/lib/ceph/osd/ceph-2/block) open path
>>> /var/lib/ceph/osd/ceph-2/block
>>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
>>> /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000,
>>> 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported
>>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1
>>> bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size
>>> 3221225472 meta 0.45 kv 0.45 data 0.06
>>> May 14 16:58:56 dragon-ball-radar conmon[113957]: debug
>>> 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400
>>> /var/lib/ceph/osd/ceph-2/block) close
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>
1093
days inactive
1096
days old
4 comments
4 participants
participants (4)
-
Andrius Jurkus -
Bartosz Lis -
Igor Fedotov -
Neha Ojha