Hello Jan,
it would be nice to see how the state evolves. Is there a chance to get
a log with `debug_prioritycache=20` and `debug_bluestore=20`?
How I can create osd?
As a workaround you might want to poke with `osd_memory_cache_min`
which, by default, is just 128 MB.
CCing BlueStore folks.
Regards,
Radek
On Fri, Aug 4, 2023 at 12:58 AM Jan Marek <jmarek(a)jcu.cz> wrote:
>
> Hello,
>
> I would like to add OSD to cluster, but I've got this error:
>
> # ceph-volume --cluster ceph lvm create --bluestore --data /dev/sda --block.wal
/dev/ceph/wal.a --block.db /dev/ceph/db.a
> Running command: /usr/bin/ceph-authtool --gen-print-key
> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
459cc86e-999a-4398-bd20-d03031f98a10
> Running command: vgcreate --force --yes ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb
/dev/sda
> stdout: Physical volume "/dev/sda" successfully created.
> stdout: Volume group "ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb"
successfully created
> Running command: lvcreate --yes -l 1907721 -n
osd-block-459cc86e-999a-4398-bd20-d03031f98a10 ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb
> stdout: Logical volume "osd-block-459cc86e-999a-4398-bd20-d03031f98a10"
created.
> Running command: /usr/bin/ceph-authtool --gen-print-key
> Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
> --> Executable selinuxenabled not in PATH:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> Running command: /usr/bin/chown -h ceph:ceph
/dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10
> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-24
> Running command: /usr/bin/ln -s
/dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10
/var/lib/ceph/osd/ceph-0/block
> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/osd/ceph-0/activate.monmap
> stderr: 2023-02-09T13:10:37.252+0100 7fdc2b98a700 -1 auth: unable to find a keyring
on
/etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:
(2) No such file or directory
> 2023-02-09T13:10:37.252+0100 7fdc2b98a700 -1 AuthRegistry(0x7fdc240607d0) no keyring
found at
/etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,
disabling cephx
> stderr: got monmap epoch 254
> --> Creating keyring file for osd.0
> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
> Running command: /usr/bin/chown -h ceph:ceph /dev/ceph/wal.a
> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-22
> Running command: /usr/bin/chown -h ceph:ceph /dev/ceph/db.a
> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-23
> Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs
-i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile -
--bluestore-block-wal-path /dev/ceph/wal.a --bluestore-block-db-path /dev/ceph/db.a
--osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 459cc86e-999a-4398-bd20-d03031f98a10
--setuser ceph --setgroup ceph
> stderr: 2023-02-09T13:10:37.728+0100 7f54531aa240 -1
bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
> stderr: ./src/common/PriorityCache.cc: In function 'void
PriorityCache::Manager::balance()' thread 7f5446944700 time
2023-02-09T13:10:39.843937+0100
> stderr: ./src/common/PriorityCache.cc: 300: FAILED ceph_assert(mem_avail >= 0)
> stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy
(stable)
> stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x124) [0x563da783bf70]
> stderr: 2: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e]
> stderr: 3: (PriorityCache::Manager::balance()+0x478) [0x563da8459938]
> stderr: 4: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a]
> stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7]
> stderr: 6: clone()
> stderr: *** Caught signal (Aborted) **
> stderr: in thread 7f5446944700 thread_name:bstore_mempool
> stderr: 2023-02-09T13:10:39.844+0100 7f5446944700 -1 ./src/common/PriorityCache.cc:
In function 'void PriorityCache::Manager::balance()' thread 7f5446944700 time
2023-02-09T13:10:39.843937+0100
> stderr: ./src/common/PriorityCache.cc: 300: FAILED ceph_assert(mem_avail >= 0)
> stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy
(stable)
> stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x124) [0x563da783bf70]
> stderr: 2: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e]
> stderr: 3: (PriorityCache::Manager::balance()+0x478) [0x563da8459938]
> stderr: 4: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a]
> stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7]
> stderr: 6: clone()
> stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy
(stable)
> stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f5453810140]
> stderr: 2: gsignal()
> stderr: 3: abort()
> stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x17e) [0x563da783bfca]
> stderr: 5: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e]
> stderr: 6: (PriorityCache::Manager::balance()+0x478) [0x563da8459938]
> stderr: 7: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a]
> stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7]
> stderr: 9: clone()
> stderr: 2023-02-09T13:10:39.852+0100 7f5446944700 -1 *** Caught signal (Aborted) **
> stderr: in thread 7f5446944700 thread_name:bstore_mempool
> stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy
(stable)
> stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f5453810140]
> stderr: 2: gsignal()
> stderr: 3: abort()
> stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x17e) [0x563da783bfca]
> stderr: 5: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e]
> stderr: 6: (PriorityCache::Manager::balance()+0x478) [0x563da8459938]
> stderr: 7: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a]
> stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7]
> stderr: 9: clone()
> stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
> stderr: -3212> 2023-02-09T13:10:37.728+0100 7f54531aa240 -1
bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
> stderr: -1> 2023-02-09T13:10:39.844+0100 7f5446944700 -1
./src/common/PriorityCache.cc: In function 'void
PriorityCache::Manager::balance()' thread 7f5446944700 time
2023-02-09T13:10:39.843937+0100
> stderr: ./src/common/PriorityCache.cc: 300: FAILED ceph_assert(mem_avail >= 0)
> stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy
(stable)
> stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x124) [0x563da783bf70]
> stderr: 2: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e]
> stderr: 3: (PriorityCache::Manager::balance()+0x478) [0x563da8459938]
> stderr: 4: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a]
> stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7]
> stderr: 6: clone()
> stderr: 0> 2023-02-09T13:10:39.852+0100 7f5446944700 -1 *** Caught signal
(Aborted) **
> stderr: in thread 7f5446944700 thread_name:bstore_mempool
> stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy
(stable)
> stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f5453810140]
> stderr: 2: gsignal()
> stderr: 3: abort()
> stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x17e) [0x563da783bfca]
> stderr: 5: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e]
> stderr: 6: (PriorityCache::Manager::balance()+0x478) [0x563da8459938]
> stderr: 7: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a]
> stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7]
> stderr: 9: clone()
> stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
> --> Was unable to complete a new OSD, will rollback changes
> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
> stderr: 2023-02-09T13:10:40.072+0100 7f6963aec700 -1 auth: unable to find a keyring
on
/etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:
(2) No such file or directory
> 2023-02-09T13:10:40.072+0100 7f6963aec700 -1 AuthRegistry(0x7f695c0607d0) no keyring
found at
/etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,
disabling cephx
> stderr: purged osd.0
> --> Zapping: /dev/ceph/db.a
> --> Unmounting /var/lib/ceph/osd/ceph-0
> Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-0
> stderr: umount: /var/lib/ceph/osd/ceph-0 odpojen
> Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph/db.a bs=1M count=10
conv=fsync
> stderr: 10+0 záznamů přečteno
> 10+0 záznamů zapsáno
> 10485760 bajtů (10 MB, 10 MiB) zkopírováno, 0,0144712 s, 725 MB/s
> --> More than 1 LV left in VG, will proceed to destroy LV only
> --> Removing LV because --destroy was given: /dev/ceph/db.a
> Running command: lvremove -v -f /dev/ceph/db.a
> stdout: Logical volume "db.a" successfully removed
> stderr: Removing ceph-db.a (254:23)
> stderr: Archiving volume group "ceph" metadata (seqno 85).
> stderr: Releasing logical volume "db.a"
> stderr: Creating volume group backup "/etc/lvm/backup/ceph" (seqno 86).
> --> Zapping:
/dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10
> Running command: /usr/bin/dd if=/dev/zero
of=/dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10
bs=1M count=10 conv=fsync
> stderr: 10+0 záznamů přečteno
> 10+0 záznamů zapsáno
> stderr: 10485760 bajtů (10 MB, 10 MiB) zkopírováno, 0,096139 s, 109 MB/s
> --> Only 1 LV left in VG, will proceed to destroy volume group
ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb
> Running command: vgremove -v -f ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb
> stderr: Removing
ceph--8f15ecd8--3b3f--404f--a851--8bbba15e30bb-osd--block--459cc86e--999a--4398--bd20--d03031f98a10
(254:24)
> stderr: Archiving volume group "ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb"
metadata (seqno 5).
> stderr: Releasing logical volume
"osd-block-459cc86e-999a-4398-bd20-d03031f98a10"
> stderr: Creating volume group backup
"/etc/lvm/backup/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb" (seqno 6).
> stdout: Logical volume "osd-block-459cc86e-999a-4398-bd20-d03031f98a10"
successfully removed
> stderr: Removing physical volume "/dev/sda" from volume group
"ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb"
> stdout: Volume group "ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb"
successfully removed
> Running command: pvremove -v -f -f /dev/sda
> stdout: Labels on physical volume "/dev/sda" successfully wiped.
> --> Zapping: /dev/ceph/wal.a
> Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph/wal.a bs=1M count=10
conv=fsync
> stderr: 10+0 záznamů přečteno
> 10+0 záznamů zapsáno
> 10485760 bajtů (10 MB, 10 MiB) zkopírováno, 0,014648 s, 716 MB/s
> --> More than 1 LV left in VG, will proceed to destroy LV only
> --> Removing LV because --destroy was given: /dev/ceph/wal.a
> Running command: lvremove -v -f /dev/ceph/wal.a
> stdout: Logical volume "wal.a" successfully removed
> stderr: Removing ceph-wal.a (254:22)
> stderr: Archiving volume group "ceph" metadata (seqno 86).
> stderr: Releasing logical volume "wal.a"
> stderr: Creating volume group backup "/etc/lvm/backup/ceph" (seqno 87).
> --> Zapping successful for OSD: 0
> --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster
ceph --osd-objectstore bluestore --mkfs -i 0 --monmap
/var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --bluestore-block-wal-path
/dev/ceph/wal.a --bluestore-block-db-path /dev/ceph/db.a --osd-data
/var/lib/ceph/osd/ceph-0/ --osd-uuid 459cc86e-999a-4398-bd20-d03031f98a10 --setuser ceph
--setgroup ceph
>
> My server have 64GB RAM.
>
> Version of ceph package is 17.2.5
>
> ceph config dump | grep osd
>
> global dev osd_max_pg_log_entries 10
> global advanced osd_memory_target_autotune true
> global dev osd_min_pg_log_entries 10
> global dev osd_pg_log_dups_tracked 10
> global dev osd_pg_log_trim_min 10
> global advanced osd_pool_default_min_size 2
> global advanced osd_pool_default_size 3
> osd dev bluefs_allocator bitmap
*
> osd advanced bluestore_allocator bitmap
*
> osd advanced bluestore_default_buffered_write true
> osd dev bluestore_fsck_quick_fix_on_mount true
> osd dev osd_memory_cache_min
134217728
> osd host:c-osd2 basic osd_memory_target
3927826739
> osd host:c-osd3 basic osd_memory_target
3927839880
> osd host:c-osd4 basic osd_memory_target
3927826739
> osd host:c-osd5 basic osd_memory_target
3927826739
> osd host:c-osd6 basic osd_memory_target
3927826739
> osd basic osd_memory_target
1073741824
>
How I can create osd?
>
> I've tried `ceph orch`, but its got the same error...
>
> Sincerely
> Jan Marek
> --
> Ing. Jan Marek
> University of South Bohemia
> Academic Computer Centre
> Phone: +420389032080
>
http://www.gnu.org/philosophy/no-word-attachments.cs.html
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io