- ceph-users - lists.ceph.io

ceph-volume quite buggy compared to ceph-disk
by Marc Roos 19 Sep '20

19 Sep '20

[@]# ceph-volume lvm activate 36 82b94115-4dfb-4ed0-8801-def59a432b0a Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-36 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-36/lockbox.keyring --create-keyring --name client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --add-key AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ== stdout: creating /var/lib/ceph/osd/ceph-36/lockbox.keyring added entity client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a auth(key=AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ==) Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/lockbox.keyring Running command: /usr/bin/ceph --cluster ceph --name client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --keyring /var/lib/ceph/osd/ceph-36/lockbox.keyring config-key get dm-crypt/osd/82b94115-4dfb-4ed0-8801-def59a432b0a/luks Running command: /usr/sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-9263e83b-7660-4f5b-843a-2111e882a17e/osd-block-82b94115-4dfb-4 ed0-8801-def59a432b0a I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb stderr: Device I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb already exists. Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb --path /var/lib/ceph/osd/ceph-36 --no-mon-config stderr: failed to read label for /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb: (2) No such file or directory --> RuntimeError: command returned non-zero exit status: 1 dmsetup ls lists this???? Where is an option to set the weight? As far as I can see you can only set this after peering started? How can I mount this tmpfs manually to inspect this? Maybe put in the manual[1]? [1] https://docs.ceph.com/en/latest/ceph-volume/lvm/activate/

1 0

Re: Ceph RDMA GID Selection Problem
by Lazuardi Nasution 19 Sep '20

19 Sep '20

Hi Samuel, I resent my reply since CC will not be accepted by this mailing list. I'm using Octopus. If it is the release problem, why it is successful with only ms_async_rdma_device_name and ms_async_rdma_gid_idx? Best regards. On Sat, Sep 19, 2020 at 3:33 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > Which Ceph version are you using? Just wondering Ceph RDMA support is > officially announced, or still on development? > > best regards, > > samuel > > ------------------------------ > huxiaoyu(a)horebdata.cn > > > *From:* Lazuardi Nasution <mrxlazuardin(a)gmail.com> > *Date:* 2020-09-18 19:21 > *To:* ceph-users <ceph-users(a)ceph.io> > *Subject:* [ceph-users] Ceph RDMA GID Selection Problem > Hi, > > I have something weird about GID selection for Ceph with RDMA. When I do > configuration with ms_async_rdma_device_name and ms_async_rdma_gid_idx, > Ceph with RDMA running successfully. But, when I do configuration with > ms_async_rdma_device_name, ms_async_rdma_local_gid and > ms_async_rdma_roce_ver, Ceph with RDMA is not working, OSDs are down in > seconds after they up. The GID index which is used on the first attempt is > associated with the GID and RoCE version which are used on the second > attempt. Is this the string matter (maybe because GID is using colon > characters) or something else? Using the GID index sometimes gives me > problems due to it not persisting and changes happen every time I do > network reconfiguration (for example: adding/removing VLAN), or even > rebooting. > > Best regards, > >

1 0

Re: Ceph RDMA GID Selection Problem
by Lazuardi Nasution 19 Sep '20

19 Sep '20

Hi Samuel, I'm using Octopus. If it is the release problem, why it is successful with only ms_async_rdma_device_name and ms_async_rdma_gid_idx? Best regards, On Sat, Sep 19, 2020 at 3:33 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > Which Ceph version are you using? Just wondering Ceph RDMA support is > officially announced, or still on development? > > best regards, > > samuel > > ------------------------------ > huxiaoyu(a)horebdata.cn > > > *From:* Lazuardi Nasution <mrxlazuardin(a)gmail.com> > *Date:* 2020-09-18 19:21 > *To:* ceph-users <ceph-users(a)ceph.io> > *Subject:* [ceph-users] Ceph RDMA GID Selection Problem > Hi, > > I have something weird about GID selection for Ceph with RDMA. When I do > configuration with ms_async_rdma_device_name and ms_async_rdma_gid_idx, > Ceph with RDMA running successfully. But, when I do configuration with > ms_async_rdma_device_name, ms_async_rdma_local_gid and > ms_async_rdma_roce_ver, Ceph with RDMA is not working, OSDs are down in > seconds after they up. The GID index which is used on the first attempt is > associated with the GID and RoCE version which are used on the second > attempt. Is this the string matter (maybe because GID is using colon > characters) or something else? Using the GID index sometimes gives me > problems due to it not persisting and changes happen every time I do > network reconfiguration (for example: adding/removing VLAN), or even > rebooting. > > Best regards, > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >

1 0

Ceph RDMA GID Selection Problem
by Lazuardi Nasution 19 Sep '20

19 Sep '20

Hi, I have something weird about GID selection for Ceph with RDMA. When I do configuration with ms_async_rdma_device_name and ms_async_rdma_gid_idx, Ceph with RDMA running successfully. But, when I do configuration with ms_async_rdma_device_name, ms_async_rdma_local_gid and ms_async_rdma_roce_ver, Ceph with RDMA is not working, OSDs are down in seconds after they up. The GID index which is used on the first attempt is associated with the GID and RoCE version which are used on the second attempt. Is this the string matter (maybe because GID is using colon characters) or something else? Using the GID index sometimes gives me problems due to it not persisting and changes happen every time I do network reconfiguration (for example: adding/removing VLAN), or even rebooting. Best regards,

2 1

Re: Benchmark WAL/DB on SSD and HDD for RGW RBD CephFS
by vitalif＠yourcmc.ru 19 Sep '20

19 Sep '20

> we did test dm-cache, bcache and dm-writecache, we found the later to be > much better. Did you set bcache block size to 4096 during your tests? Without this setting it's slow because 99.9% SSDs don't handle 512 byte overwrites well. Otherwise I don't think bcache should be worse than dm-writecache. Also dm-writecache only caches writes, and bcache also caches reads. And lvmcache is trash because it only writes to SSD when the block is already on the SSD. Please post some details about the comparison if you have them :)

3 2

RuntimeError: Unable check if OSD id exists
by Marc Roos 19 Sep '20

19 Sep '20

I have still ceph-disk created osd's in nautilus. Thought about using this ceph-volume, but looks like this manual for replacing ceph-disk[1] is not complete. Getting already this error RuntimeError: Unable check if OSD id exists: [1] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/#rados-repl…

1 0

Benchmark WAL/DB on SSD and HDD for RGW RBD CephFS
by Danni Setiawan 19 Sep '20

19 Sep '20

Hi all, I'm trying to find performance penalty with OSD HDD when using WAL/DB in faster device (SSD/NVMe) vs WAL/DB in same device (HDD) for different workload (RBD, RGW with index bucket in SSD pool, and CephFS with metadata in SSD pool). I want to know if giving up disk slot for WAL/DB device is worth vs adding more OSD. Unfortunately I cannot find the benchmark for these kind workload. Has anyone ever done this benchmark? Thank you.

8 10

Using cephadm shell/ceph-volume
by tri＠postix.net 18 Sep '20

18 Sep '20

Hi all, I'm having problem creating an osd using ceph-volume (by the way of cephadm). This is on an octopus installation with cephadm. So I use "cephadm shell" and then "ceph-volume" but got the following error: root@furry:/var/lib# ceph-volume lvm prepare --data /dev/sda --block.db /dev/vg/sda.db --dmcrypt Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 29d95564-733d-4f2a-a2c8-1bb9ceb5a14b stderr: [errno 13] RADOS permission denied (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id if I pass the cluster ID, and use the correct key ring (using the client.admin keyring), I got a bit further root@furry:/var/lib# ceph-volume --cluster c258000c-f3e4-11ea-9ebe-c3c75e8e9028 lvm prepare --data /dev/sda --block.db /dev/vg/sda.db --dmcrypt Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster c258000c-f3e4-11ea-9ebe-c3c75e8e9028 --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/c258000c-f3e4-11ea-9ebe-c3c75e8e9028.keyring -i - osd new aa13c362-c9cf-4d03-9a86-d6118fbc312c stderr: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)',) --> RuntimeError: Unable to create a new OSD id Any idea on how to get pass these errors? Thanks. --Tri Hoang

2 1

It is an advantage to use assignment help in Australia
by john seena 18 Sep '20

18 Sep '20

Do you want to connect with Australian assignment helpers to feel the low stress of submission? Are you seeking the best source of help for composing your academic documents? In this context, use Assignment Help Services and get completed papers without any delay. As I can understand that, whenever you have issues in writing your assignments, it gets hard for you to write flawless papers. When you are busy with lots of activities, you can’t focus on assignment writing and collecting requisite information for it. So, if you require a reliable way of finishing your homework, choosing the assistance of professionals is an advantage. You can manage your effort and save your time using the services of online academic writing. I can say so because I already used these services when I was studying Australia. You don’t have to think a lot because using online assignment help services is the best choice for completing your work without any issue. Don’t face any delay if you know about online academic writing service in Australia. https://www.greatassignmenthelp.com/au/

1 0

September Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 18 Sep '20

18 Sep '20

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday September 23rd. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20200923 <https://pad.ceph.com/p/Ceph_Science_User_Group_20200923> We try to keep it to an hour or less. Ceph calendar event details: September 23, 2020 14:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

1 0