Hello Jan, it would be nice to see how the state evolves. Is there a chance to get a log with `debug_prioritycache=20` and `debug_bluestore=20`?
How I can create osd?
As a workaround you might want to poke with `osd_memory_cache_min` which, by default, is just 128 MB. CCing BlueStore folks. Regards, Radek On Fri, Aug 4, 2023 at 12:58 AM Jan Marek <jmarek@jcu.cz> wrote:
Hello,
I would like to add OSD to cluster, but I've got this error:
# ceph-volume --cluster ceph lvm create --bluestore --data /dev/sda --block.wal /dev/ceph/wal.a --block.db /dev/ceph/db.a Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 459cc86e-999a-4398-bd20-d03031f98a10 Running command: vgcreate --force --yes ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb /dev/sda stdout: Physical volume "/dev/sda" successfully created. stdout: Volume group "ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb" successfully created Running command: lvcreate --yes -l 1907721 -n osd-block-459cc86e-999a-4398-bd20-d03031f98a10 ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb stdout: Logical volume "osd-block-459cc86e-999a-4398-bd20-d03031f98a10" created. Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-24 Running command: /usr/bin/ln -s /dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap stderr: 2023-02-09T13:10:37.252+0100 7fdc2b98a700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2023-02-09T13:10:37.252+0100 7fdc2b98a700 -1 AuthRegistry(0x7fdc240607d0) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx stderr: got monmap epoch 254 --> Creating keyring file for osd.0 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ Running command: /usr/bin/chown -h ceph:ceph /dev/ceph/wal.a Running command: /usr/bin/chown -R ceph:ceph /dev/dm-22 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph/db.a Running command: /usr/bin/chown -R ceph:ceph /dev/dm-23 Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --bluestore-block-wal-path /dev/ceph/wal.a --bluestore-block-db-path /dev/ceph/db.a --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 459cc86e-999a-4398-bd20-d03031f98a10 --setuser ceph --setgroup ceph stderr: 2023-02-09T13:10:37.728+0100 7f54531aa240 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid stderr: ./src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f5446944700 time 2023-02-09T13:10:39.843937+0100 stderr: ./src/common/PriorityCache.cc: 300: FAILED ceph_assert(mem_avail >= 0) stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x563da783bf70] stderr: 2: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] stderr: 3: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] stderr: 4: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] stderr: 6: clone() stderr: *** Caught signal (Aborted) ** stderr: in thread 7f5446944700 thread_name:bstore_mempool stderr: 2023-02-09T13:10:39.844+0100 7f5446944700 -1 ./src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f5446944700 time 2023-02-09T13:10:39.843937+0100 stderr: ./src/common/PriorityCache.cc: 300: FAILED ceph_assert(mem_avail >= 0) stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x563da783bf70] stderr: 2: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] stderr: 3: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] stderr: 4: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] stderr: 6: clone() stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f5453810140] stderr: 2: gsignal() stderr: 3: abort() stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17e) [0x563da783bfca] stderr: 5: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] stderr: 6: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] stderr: 7: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] stderr: 9: clone() stderr: 2023-02-09T13:10:39.852+0100 7f5446944700 -1 *** Caught signal (Aborted) ** stderr: in thread 7f5446944700 thread_name:bstore_mempool stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f5453810140] stderr: 2: gsignal() stderr: 3: abort() stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17e) [0x563da783bfca] stderr: 5: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] stderr: 6: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] stderr: 7: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] stderr: 9: clone() stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: -3212> 2023-02-09T13:10:37.728+0100 7f54531aa240 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid stderr: -1> 2023-02-09T13:10:39.844+0100 7f5446944700 -1 ./src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f5446944700 time 2023-02-09T13:10:39.843937+0100 stderr: ./src/common/PriorityCache.cc: 300: FAILED ceph_assert(mem_avail >= 0) stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x563da783bf70] stderr: 2: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] stderr: 3: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] stderr: 4: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] stderr: 6: clone() stderr: 0> 2023-02-09T13:10:39.852+0100 7f5446944700 -1 *** Caught signal (Aborted) ** stderr: in thread 7f5446944700 thread_name:bstore_mempool stderr: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f5453810140] stderr: 2: gsignal() stderr: 3: abort() stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17e) [0x563da783bfca] stderr: 5: /usr/bin/ceph-osd(+0xc2310e) [0x563da783c10e] stderr: 6: (PriorityCache::Manager::balance()+0x478) [0x563da8459938] stderr: 7: (BlueStore::MempoolThread::entry()+0x82a) [0x563da7e9e27a] stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f5453804ea7] stderr: 9: clone() stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: 2023-02-09T13:10:40.072+0100 7f6963aec700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2023-02-09T13:10:40.072+0100 7f6963aec700 -1 AuthRegistry(0x7f695c0607d0) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx stderr: purged osd.0 --> Zapping: /dev/ceph/db.a --> Unmounting /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-0 stderr: umount: /var/lib/ceph/osd/ceph-0 odpojen Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph/db.a bs=1M count=10 conv=fsync stderr: 10+0 záznamů přečteno 10+0 záznamů zapsáno 10485760 bajtů (10 MB, 10 MiB) zkopírováno, 0,0144712 s, 725 MB/s --> More than 1 LV left in VG, will proceed to destroy LV only --> Removing LV because --destroy was given: /dev/ceph/db.a Running command: lvremove -v -f /dev/ceph/db.a stdout: Logical volume "db.a" successfully removed stderr: Removing ceph-db.a (254:23) stderr: Archiving volume group "ceph" metadata (seqno 85). stderr: Releasing logical volume "db.a" stderr: Creating volume group backup "/etc/lvm/backup/ceph" (seqno 86). --> Zapping: /dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10 Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb/osd-block-459cc86e-999a-4398-bd20-d03031f98a10 bs=1M count=10 conv=fsync stderr: 10+0 záznamů přečteno 10+0 záznamů zapsáno stderr: 10485760 bajtů (10 MB, 10 MiB) zkopírováno, 0,096139 s, 109 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb Running command: vgremove -v -f ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb stderr: Removing ceph--8f15ecd8--3b3f--404f--a851--8bbba15e30bb-osd--block--459cc86e--999a--4398--bd20--d03031f98a10 (254:24) stderr: Archiving volume group "ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb" metadata (seqno 5). stderr: Releasing logical volume "osd-block-459cc86e-999a-4398-bd20-d03031f98a10" stderr: Creating volume group backup "/etc/lvm/backup/ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb" (seqno 6). stdout: Logical volume "osd-block-459cc86e-999a-4398-bd20-d03031f98a10" successfully removed stderr: Removing physical volume "/dev/sda" from volume group "ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb" stdout: Volume group "ceph-8f15ecd8-3b3f-404f-a851-8bbba15e30bb" successfully removed Running command: pvremove -v -f -f /dev/sda stdout: Labels on physical volume "/dev/sda" successfully wiped. --> Zapping: /dev/ceph/wal.a Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph/wal.a bs=1M count=10 conv=fsync stderr: 10+0 záznamů přečteno 10+0 záznamů zapsáno 10485760 bajtů (10 MB, 10 MiB) zkopírováno, 0,014648 s, 716 MB/s --> More than 1 LV left in VG, will proceed to destroy LV only --> Removing LV because --destroy was given: /dev/ceph/wal.a Running command: lvremove -v -f /dev/ceph/wal.a stdout: Logical volume "wal.a" successfully removed stderr: Removing ceph-wal.a (254:22) stderr: Archiving volume group "ceph" metadata (seqno 86). stderr: Releasing logical volume "wal.a" stderr: Creating volume group backup "/etc/lvm/backup/ceph" (seqno 87). --> Zapping successful for OSD: 0 --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --bluestore-block-wal-path /dev/ceph/wal.a --bluestore-block-db-path /dev/ceph/db.a --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 459cc86e-999a-4398-bd20-d03031f98a10 --setuser ceph --setgroup ceph
My server have 64GB RAM.
Version of ceph package is 17.2.5
ceph config dump | grep osd
global dev osd_max_pg_log_entries 10 global advanced osd_memory_target_autotune true global dev osd_min_pg_log_entries 10 global dev osd_pg_log_dups_tracked 10 global dev osd_pg_log_trim_min 10 global advanced osd_pool_default_min_size 2 global advanced osd_pool_default_size 3 osd dev bluefs_allocator bitmap * osd advanced bluestore_allocator bitmap * osd advanced bluestore_default_buffered_write true osd dev bluestore_fsck_quick_fix_on_mount true osd dev osd_memory_cache_min 134217728 osd host:c-osd2 basic osd_memory_target 3927826739 osd host:c-osd3 basic osd_memory_target 3927839880 osd host:c-osd4 basic osd_memory_target 3927826739 osd host:c-osd5 basic osd_memory_target 3927826739 osd host:c-osd6 basic osd_memory_target 3927826739 osd basic osd_memory_target 1073741824
How I can create osd?
I've tried `ceph orch`, but its got the same error...
Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io