Hi,
I've tried to create ceph luminous cluster for testing porpuses with ceph-ansible on
my 3 machines hyperv vms, but I've got the below error with the following with the
following osd configuration:
---
dummy:
osd_scenario: lvm
lvm_volumes:
- data: osd1lv
data_vg: osd1
db: journal_lv
db_vg: journal
wal: journal_lv
wal_vg: journal
- data: osd2lv
data_vg: osd2
db: journal_lv
db_vg: journal
wal: journal_lv
wal_vg: journal
I have 3 disks, 2 for osd 1 for journal, I'd like to use the journal (sdb) for both
(sdc sdd) device.
I've also tried just simply use:
lvm_volumes:
- data: /dev/sdc
db: /dev/sdb
wal: /dev/sdb
- data: /dev/sdd
db: /dev/sdb
wal: /dev/sdb
But this one complains about partuuid not found for sdb.
I've also tried to create with parted a gpt partition to have the partuuid, but that
time it complains about gpt flag.
My setup is a centos 7, ansible 2.6, checked out the stable-3.2 ceph-ansible.
The all.yml is like this:
---
dummy:
ceph_origin: repository
ceph_repository: community
ceph_stable_release: luminous
monitor_interface: eth0
public_network: 172.19.15.0/24
cluster_network: 192.168.1.0/24
Is there anybody can show me a working configuration for ceph-volume with wal and db
configuration?
Thank you in advance.
The ansible dump is here:
failed: [ceph3] (item={'data': 'osd2lv', 'data_vg':
'osd2', 'db': 'journal_lv', 'db_vg': 'journal',
'wal': 'journal_lv', 'wal_vg': 'journal'}) =>
{"changed": true, "cmd": ["ceph-volume",
"--cluster", "ceph", "lvm", "create",
"--bluestore", "--data", "osd2/osd2lv",
"--block.db", "journal/journal_lv", "--block.wal",
"journal/journal_lv"], "delta": "0:00:06.029411",
"end": "2020-04-28 16:48:13.252098", "item":
{"data": "osd2lv", "data_vg": "osd2",
"db": "journal_lv", "db_vg": "journal",
"wal": "journal_lv", "wal_vg": "journal"},
"msg": "non-zero return code", "rc": 1, "start":
"2020-04-28 16:48:07.222687", "stderr": "Running command:
/bin/ceph-authtool --gen-print-key\nRunning command: /bin/ceph --cluster ceph --name
client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3\nRunning command: /bin/ceph-authtool
--gen-print-key\nRunning command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2\nRunning
command: restorecon /var/lib/ceph/osd/ceph-2\nRunning command: chown -h ceph:ceph
/dev/osd2/osd2lv\nRunning command: chown -R ceph:ceph /dev/dm-4\nRunning command: ln -s
/dev/osd2/osd2lv /var/lib/ceph/osd/ceph-2/block\nRunning command: ceph --cluster ceph
--name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap
-o /var/lib/ceph/osd/ceph-2/activate.monmap\n stderr: got monmap epoch 1\nRunning command:
ceph-authtool /var/lib/ceph/osd/ceph-2/keyring --create-keyring --name osd.2 --add-key
AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA==\n stdout: creating
/var/lib/ceph/osd/ceph-2/keyring\nadded entity osd.2 auth auth(auid = 18446744073709551615
key=AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA== with 0 caps)\nRunning command: chown -R
ceph:ceph /var/lib/ceph/osd/ceph-2/keyring\nRunning command: chown -R ceph:ceph
/var/lib/ceph/osd/ceph-2/\nRunning command: chown -h ceph:ceph
/dev/journal/journal_lv\nRunning command: chown -R ceph:ceph /dev/dm-5\nRunning command:
chown -h ceph:ceph /dev/journal/journal_lv\nRunning command: chown -R ceph:ceph
/dev/dm-5\nRunning command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore
--mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile -
--bluestore-block-wal-path /dev/journal/journal_lv --bluestore-block-db-path
/dev/journal/journal_lv --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid
842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3 --setuser ceph --setgroup ceph\n stderr: 2020-04-28
16:48:11.895597 7fb89cb53ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable
uuid\n stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with
L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time
2020-04-28 16:48:12.908859\n stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757: FAILED assert(available >= allocated)\n stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110)
[0x55f13e067ae0]\n stderr: 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned
long)+0x674) [0x55f13e01ae54]\n stderr: 3: (BlueFS::mount()+0x271) [0x55f13dff3411]\n
stderr: 4: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 5:
(BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 6:
(BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 7: (OSD::mkfs(CephContext*,
ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 8:
(main()+0x11a5) [0x55f13d9d4975]\n stderr: 9: (__libc_start_main()+0xf5)
[0x7fb899005505]\n stderr: 10: (()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of
the executable, or `objdump -rdS <executable>` is needed to interpret this.\n
stderr: 2020-04-28 16:48:12.911181 7fb89cb53ec0 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with
L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time
2020-04-28 16:48:12.908859\n stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757: FAILED assert(available >= allocated)\n stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110)
[0x55f13e067ae0]\n stderr: 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned
long)+0x674) [0x55f13e01ae54]\n stderr: 3: (BlueFS::mount()+0x271) [0x55f13dff3411]\n
stderr: 4: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 5:
(BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 6:
(BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 7: (OSD::mkfs(CephContext*,
ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 8:
(main()+0x11a5) [0x55f13d9d4975]\n stderr: 9: (__libc_start_main()+0xf5)
[0x7fb899005505]\n stderr: 10: (()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of
the executable, or `objdump -rdS <executable>` is needed to interpret this.\n
stderr: -253> 2020-04-28 16:48:11.895597 7fb89cb53ec0 -1
bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid\n stderr: 0> 2020-04-28
16:48:12.911181 7fb89cb53ec0 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with
L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time
2020-04-28 16:48:12.908859\n stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757: FAILED assert(available >= allocated)\n stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110)
[0x55f13e067ae0]\n stderr: 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned
long)+0x674) [0x55f13e01ae54]\n stderr: 3: (BlueFS::mount()+0x271) [0x55f13dff3411]\n
stderr: 4: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 5:
(BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 6:
(BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 7: (OSD::mkfs(CephContext*,
ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 8:
(main()+0x11a5) [0x55f13d9d4975]\n stderr: 9: (__libc_start_main()+0xf5)
[0x7fb899005505]\n stderr: 10: (()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of
the executable, or `objdump -rdS <executable>` is needed to interpret this.\n
stderr: *** Caught signal (Aborted) **\n stderr: in thread 7fb89cb53ec0
thread_name:ceph-osd\n stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1: (()+0xa76e51)
[0x55f13e028e51]\n stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]\n stderr: 3: (gsignal()+0x37)
[0x7fb899019337]\n stderr: 4: (abort()+0x148) [0x7fb89901aa28]\n stderr: 5:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284)
[0x55f13e067c54]\n stderr: 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned
long)+0x674) [0x55f13e01ae54]\n stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]\n
stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 9:
(BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 10:
(BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 11: (OSD::mkfs(CephContext*,
ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 12:
(main()+0x11a5) [0x55f13d9d4975]\n stderr: 13: (__libc_start_main()+0xf5)
[0x7fb899005505]\n stderr: 14: (()+0x4c4383) [0x55f13da76383]\n stderr: 2020-04-28
16:48:12.914234 7fb89cb53ec0 -1 *** Caught signal (Aborted) **\n stderr: in thread
7fb89cb53ec0 thread_name:ceph-osd\n stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1: (()+0xa76e51)
[0x55f13e028e51]\n stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]\n stderr: 3: (gsignal()+0x37)
[0x7fb899019337]\n stderr: 4: (abort()+0x148) [0x7fb89901aa28]\n stderr: 5:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284)
[0x55f13e067c54]\n stderr: 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned
long)+0x674) [0x55f13e01ae54]\n stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]\n
stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 9:
(BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 10:
(BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 11: (OSD::mkfs(CephContext*,
ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 12:
(main()+0x11a5) [0x55f13d9d4975]\n stderr: 13: (__libc_start_main()+0xf5)
[0x7fb899005505]\n stderr: 14: (()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of
the executable, or `objdump -rdS <executable>` is needed to interpret this.\n
stderr: 0> 2020-04-28 16:48:12.914234 7fb89cb53ec0 -1 *** Caught signal (Aborted) **\n
stderr: in thread 7fb89cb53ec0 thread_name:ceph-osd\n stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1: (()+0xa76e51)
[0x55f13e028e51]\n stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]\n stderr: 3: (gsignal()+0x37)
[0x7fb899019337]\n stderr: 4: (abort()+0x148) [0x7fb89901aa28]\n stderr: 5:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284)
[0x55f13e067c54]\n stderr: 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned
long)+0x674) [0x55f13e01ae54]\n stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]\n
stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 9:
(BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 10:
(BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 11: (OSD::mkfs(CephContext*,
ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 12:
(main()+0x11a5) [0x55f13d9d4975]\n stderr: 13: (__libc_start_main()+0xf5)
[0x7fb899005505]\n stderr: 14: (()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of
the executable, or `objdump -rdS <executable>` is needed to interpret this.\n-->
Was unable to complete a new OSD, will rollback changes\n--> OSD will be fully purged
from the cluster, because the ID was generated\nRunning command: ceph osd purge osd.2
--yes-i-really-mean-it\n stderr: purged osd.2\nTraceback (most recent call last):\n File
\"/sbin/ceph-volume\", line 9, in <module>\n
load_entry_point('ceph-volume==1.0.0', 'console_scripts',
'ceph-volume')()\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 38, in __init__\n
self.main(self.argv)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 59, in
newfunc\n return f(*a, **kw)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 149, in main\n
terminal.dispatch(self.mapper, subcommand_args)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in
dispatch\n instance.main()\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py\", line 40,
in main\n terminal.dispatch(self.mapper, self.argv)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in
dispatch\n instance.main()\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line 69,
in main\n self.create(args)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in
is_root\n return func(*a, **kw)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line 26,
in create\n prepare_step.safe_prepare(args)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line
219, in safe_prepare\n self.prepare()\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in
is_root\n return func(*a, **kw)\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line
320, in prepare\n osd_fsid,\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line
119, in prepare_bluestore\n db=db\n File
\"/usr/lib/python2.7/site-packages/ceph_volume/util/prepare.py\", line 397, in
osd_mkfs_bluestore\n raise RuntimeError('Command failed with exit code %s: %s'
% (returncode, ' '.join(command)))\nRuntimeError: Command failed with exit code
-6: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap
/var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --bluestore-block-wal-path
/dev/journal/journal_lv --bluestore-block-db-path /dev/journal/journal_lv --osd-data
/var/lib/ceph/osd/ceph-2/ --osd-uuid 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3 --setuser ceph
--setgroup ceph", "stderr_lines": ["Running command:
/bin/ceph-authtool --gen-print-key", "Running command: /bin/ceph --cluster ceph
--name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd
new 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3", "Running command: /bin/ceph-authtool
--gen-print-key", "Running command: mount -t tmpfs tmpfs
/var/lib/ceph/osd/ceph-2", "Running command: restorecon
/var/lib/ceph/osd/ceph-2", "Running command: chown -h ceph:ceph
/dev/osd2/osd2lv", "Running command: chown -R ceph:ceph /dev/dm-4",
"Running command: ln -s /dev/osd2/osd2lv /var/lib/ceph/osd/ceph-2/block",
"Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/osd/ceph-2/activate.monmap", " stderr: got monmap epoch 1",
"Running command: ceph-authtool /var/lib/ceph/osd/ceph-2/keyring --create-keyring
--name osd.2 --add-key AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA==", " stdout:
creating /var/lib/ceph/osd/ceph-2/keyring", "added entity osd.2 auth auth(auid =
18446744073709551615 key=AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA== with 0 caps)",
"Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring",
"Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/", "Running
command: chown -h ceph:ceph /dev/journal/journal_lv", "Running command: chown -R
ceph:ceph /dev/dm-5", "Running command: chown -h ceph:ceph
/dev/journal/journal_lv", "Running command: chown -R ceph:ceph /dev/dm-5",
"Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i
2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --bluestore-block-wal-path
/dev/journal/journal_lv --bluestore-block-db-path /dev/journal/journal_lv --osd-data
/var/lib/ceph/osd/ceph-2/ --osd-uuid 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3 --setuser ceph
--setgroup ceph", " stderr: 2020-04-28 16:48:11.895597 7fb89cb53ec0 -1
bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid", " stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with
L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time
2020-04-28 16:48:12.908859", " stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757: FAILED assert(available >= allocated)", " stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110)
[0x55f13e067ae0]", " stderr: 2: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 3: (BlueFS::mount()+0x271)
[0x55f13dff3411]", " stderr: 4: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]", " stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]", " stderr: 6: (BlueStore::mkfs()+0xe5a)
[0x55f13df0d3ba]", " stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*,
std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 8:
(main()+0x11a5) [0x55f13d9d4975]", " stderr: 9: (__libc_start_main()+0xf5)
[0x7fb899005505]", " stderr: 10: (()+0x4c4383) [0x55f13da76383]", "
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.", " stderr: 2020-04-28 16:48:12.911181 7fb89cb53ec0 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with
L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time
2020-04-28 16:48:12.908859", " stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757: FAILED assert(available >= allocated)", " stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110)
[0x55f13e067ae0]", " stderr: 2: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 3: (BlueFS::mount()+0x271)
[0x55f13dff3411]", " stderr: 4: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]", " stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]", " stderr: 6: (BlueStore::mkfs()+0xe5a)
[0x55f13df0d3ba]", " stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*,
std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 8:
(main()+0x11a5) [0x55f13d9d4975]", " stderr: 9: (__libc_start_main()+0xf5)
[0x7fb899005505]", " stderr: 10: (()+0x4c4383) [0x55f13da76383]", "
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.", " stderr: -253> 2020-04-28 16:48:11.895597 7fb89cb53ec0 -1
bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid", " stderr:
0> 2020-04-28 16:48:12.911181 7fb89cb53ec0 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with
L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time
2020-04-28 16:48:12.908859", " stderr:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h:
757: FAILED assert(available >= allocated)", " stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110)
[0x55f13e067ae0]", " stderr: 2: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 3: (BlueFS::mount()+0x271)
[0x55f13dff3411]", " stderr: 4: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]", " stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]", " stderr: 6: (BlueStore::mkfs()+0xe5a)
[0x55f13df0d3ba]", " stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*,
std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 8:
(main()+0x11a5) [0x55f13d9d4975]", " stderr: 9: (__libc_start_main()+0xf5)
[0x7fb899005505]", " stderr: 10: (()+0x4c4383) [0x55f13da76383]", "
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.", " stderr: *** Caught signal (Aborted) **", " stderr:
in thread 7fb89cb53ec0 thread_name:ceph-osd", " stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1:
(()+0xa76e51) [0x55f13e028e51]", " stderr: 2: (()+0xf5f0)
[0x7fb899ff95f0]", " stderr: 3: (gsignal()+0x37) [0x7fb899019337]", "
stderr: 4: (abort()+0x148) [0x7fb89901aa28]", " stderr: 5:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284)
[0x55f13e067c54]", " stderr: 6: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 7: (BlueFS::mount()+0x271)
[0x55f13dff3411]", " stderr: 8: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]", " stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]", " stderr: 10: (BlueStore::mkfs()+0xe5a)
[0x55f13df0d3ba]", " stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*,
std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 12:
(main()+0x11a5) [0x55f13d9d4975]", " stderr: 13: (__libc_start_main()+0xf5)
[0x7fb899005505]", " stderr: 14: (()+0x4c4383) [0x55f13da76383]", "
stderr: 2020-04-28 16:48:12.914234 7fb89cb53ec0 -1 *** Caught signal (Aborted) **",
" stderr: in thread 7fb89cb53ec0 thread_name:ceph-osd", " stderr: ceph
version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", "
stderr: 1: (()+0xa76e51) [0x55f13e028e51]", " stderr: 2: (()+0xf5f0)
[0x7fb899ff95f0]", " stderr: 3: (gsignal()+0x37) [0x7fb899019337]", "
stderr: 4: (abort()+0x148) [0x7fb89901aa28]", " stderr: 5:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284)
[0x55f13e067c54]", " stderr: 6: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 7: (BlueFS::mount()+0x271)
[0x55f13dff3411]", " stderr: 8: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]", " stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]", " stderr: 10: (BlueStore::mkfs()+0xe5a)
[0x55f13df0d3ba]", " stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*,
std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 12:
(main()+0x11a5) [0x55f13d9d4975]", " stderr: 13: (__libc_start_main()+0xf5)
[0x7fb899005505]", " stderr: 14: (()+0x4c4383) [0x55f13da76383]", "
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.", " stderr: 0> 2020-04-28 16:48:12.914234 7fb89cb53ec0 -1 ***
Caught signal (Aborted) **", " stderr: in thread 7fb89cb53ec0
thread_name:ceph-osd", " stderr: ceph version 12.2.13
(584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1:
(()+0xa76e51) [0x55f13e028e51]", " stderr: 2: (()+0xf5f0)
[0x7fb899ff95f0]", " stderr: 3: (gsignal()+0x37) [0x7fb899019337]", "
stderr: 4: (abort()+0x148) [0x7fb89901aa28]", " stderr: 5:
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284)
[0x55f13e067c54]", " stderr: 6: (BitmapAllocator::init_rm_free(unsigned long,
unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 7: (BlueFS::mount()+0x271)
[0x55f13dff3411]", " stderr: 8: (BlueStore::_open_db(bool)+0x1857)
[0x55f13df01827]", " stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7)
[0x55f13df37707]", " stderr: 10: (BlueStore::mkfs()+0xe5a)
[0x55f13df0d3ba]", " stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*,
std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 12:
(main()+0x11a5) [0x55f13d9d4975]", " stderr: 13: (__libc_start_main()+0xf5)
[0x7fb899005505]", " stderr: 14: (()+0x4c4383) [0x55f13da76383]", "
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.", "--> Was unable to complete a new OSD, will rollback
changes", "--> OSD will be fully purged from the cluster, because the ID was
generated", "Running command: ceph osd purge osd.2 --yes-i-really-mean-it",
" stderr: purged osd.2", "Traceback (most recent call last):", "
File \"/sbin/ceph-volume\", line 9, in <module>", "
load_entry_point('ceph-volume==1.0.0', 'console_scripts',
'ceph-volume')()", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 38, in
__init__", " self.main(self.argv)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 59, in
newfunc", " return f(*a, **kw)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 149, in
main", " terminal.dispatch(self.mapper, subcommand_args)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in
dispatch", " instance.main()", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py\", line 40,
in main", " terminal.dispatch(self.mapper, self.argv)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in
dispatch", " instance.main()", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line 69,
in main", " self.create(args)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in
is_root", " return func(*a, **kw)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line 26,
in create", " prepare_step.safe_prepare(args)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line
219, in safe_prepare", " self.prepare()", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in
is_root", " return func(*a, **kw)", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line
320, in prepare", " osd_fsid,", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line
119, in prepare_bluestore", " db=db", " File
\"/usr/lib/python2.7/site-packages/ceph_volume/util/prepare.py\", line 397, in
osd_mkfs_bluestore", " raise RuntimeError('Command failed with exit code
%s: %s' % (returncode, ' '.join(command)))", "RuntimeError: Command
failed with exit code -6: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs
-i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile -
--bluestore-block-wal-path /dev/journal/journal_lv --bluestore-block-db-path
/dev/journal/journal_lv --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid
842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3 --setuser ceph --setgroup ceph"],
"stdout": "", "stdout_lines": []}
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.