- ceph-users - lists.ceph.io

Cannot create new OSD node - _read_fsid unparsable uuid
by Ha, Son Hai 22 Jan '21

22 Jan '21

Hello everyone, I'm trying to add an OSD node to my current cluster. I created an lvm volume for this node to use for OSD. My current Ceph version is 14.2.6 and it runs on an RHEL 7 OS. However, I got error when trying to activate the node. I'm confused with the output. I tried to see what really happened, but I don't know how to proceed to pinpoint the issues. I appreciate so much for any help to clear the confusion. * Why did the "mon getmap" command send the result to /dev/stderr? I saw the monmap file having value and it looked like it got the monmap, is there any way to check if the monmap having problem? * What caused "_read_fsid unparsable uuid"? Sincerely, Hai [xxx@xxx.com@xxx430 ~]$ sudo ceph-volume --cluster aap-storage lvm create --data /dev/kubernetes/ceph-osd Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring -i - osd new c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90 Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/aap-storage-5 Running command: /sbin/restorecon /var/lib/ceph/osd/aap-storage-5 Running command: /bin/chown -h ceph:ceph /dev/kubernetes/ceph-osd Running command: /bin/chown -R ceph:ceph /dev/dm-9 Running command: /bin/ln -s /dev/kubernetes/ceph-osd /var/lib/ceph/osd/aap-storage-5/block Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring mon getmap -o /var/lib/ceph/osd/aap-storage-5/activate.monmap stderr: got monmap epoch 8 Running command: /bin/ceph-authtool /var/lib/ceph/osd/aap-storage-5/keyring --create-keyring --name osd.5 --add-key AQB2RAhgnkcLIBAAdcdR5N4YKzSJmfoA6G6XvA== stdout: creating /var/lib/ceph/osd/aap-storage-5/keyring added entity osd.5 auth(key=AQB2RAhgnkcLIBAAdcdR5N4YKzSJmfoA6G6XvA==) Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5/keyring Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5/ Running command: /bin/ceph-osd --cluster aap-storage --osd-objectstore bluestore --mkfs -i 5 --monmap /var/lib/ceph/osd/aap-storage-5/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/aap-storage-5/ --osd-uuid c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90 --setuser ceph --setgroup ceph stderr: 2021-01-20 15:55:52.319 7f0970d3ba80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: kubernetes/ceph-osd Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5 Running command: /bin/ceph-bluestore-tool --cluster=aap-storage prime-osd-dir --dev /dev/kubernetes/ceph-osd --path /var/lib/ceph/osd/aap-storage-5 --no-mon-config Running command: /bin/ln -snf /dev/kubernetes/ceph-osd /var/lib/ceph/osd/aap-storage-5/block Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/aap-storage-5/block Running command: /bin/chown -R ceph:ceph /dev/dm-9 Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5 Running command: /bin/systemctl enable ceph-volume@lvm-5-c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-5-c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90.service to /usr/lib/systemd/system/ceph-volume@.service. Running command: /bin/systemctl enable --runtime ceph-osd@5 stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)5.service to /usr/lib/systemd/system/ceph-osd@.service. Running command: /bin/systemctl start ceph-osd@5 stderr: Job for ceph-osd(a)5.service failed because the control process exited with error code. See "systemctl status ceph-osd(a)5.service" and "journalctl -xe" for details. --> Was unable to complete a new OSD, will rollback changes Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring osd purge-new osd.5 --yes-i-really-mean-it stderr: purged osd.5 --> RuntimeError: command returned non-zero exit status: 1 So I break the procedure into steps, and proceed them one by one. I saw the problem happen with the bluestore. [xxx@xxx.com@xxx430 ~]$ sudo ceph-osd -c /etc/ceph/${CLUSTER_NAME}.conf -k /etc/ceph/${CLUSTER_NAME}.client.admin.keyring -i $ID --mkfs \ > --monmap /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/activate.monmap --osd-uuid $UUID --no-mon-config \ > --osd-data /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/ --setuser ceph --setgroup ceph 2021-01-21 16:59:14.907 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5//block) _read_bdev_label failed to open /var/lib/ceph/osd/aap-storage-5//block: (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5//block) _read_bdev_label failed to open /var/lib/ceph/osd/aap-storage-5//block: (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _setup_block_symlink_or_file failed to open block file: (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) mkfs failed, (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/aap-storage-5/: (13) Permission denied Because of the permission problem, so I avoid using the ceph user for the ceph service, however, the problem still persisted. [xxx@xxx.com@xxx430 ~]$ sudo ceph-osd -c /etc/ceph/${CLUSTER_NAME}.conf -k /etc/ceph/${CLUSTER_NAME}.client.admin.keyring -i $ID --mkfs --monmap /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/activate.monmap --osd-uuid $UUID --no-mon-config --osd-data /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/ 2021-01-21 17:00:08.583 7f3acc546a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid -- KPMG IT Service GmbH Sitz/Registergericht: Berlin/Amtsgericht Charlottenburg, HRB 87521 B Geschäftsführer: Hans-Christian Schwieger, Helmar Symmank Aufsichtsratsvorsitzender: WP StB Klaus Becker Allgemeine Informationen zur Datenverarbeitung im Rahmen unserer allgemeinen Geschäftstätigkeit sowie im Mandatsverhältnis gemäß EU Datenschutz-Grundverordnung sind hier <https://home.kpmg.com/content/dam/kpmg/de/pdf/Themen/2018/datenschutzinform…> abrufbar. Die Information in dieser E-Mail ist vertraulich und kann dem Berufsgeheimnis unterliegen. Sie ist ausschließlich für den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Sollten Sie nicht der für diese E-Mail bestimmte Adressat sein, ist Ihnen jede Veröffentlichung, Vervielfältigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Maßnahmen im Vertrauen auf erlangte Information untersagt. In dieser E-Mail enthaltene Meinungen oder Empfehlungen unterliegen den Bedingungen des jeweiligen Mandatsverhältnisses mit dem Adressaten. The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing KPMG client engagement letter.

1 0

mds openfiles table shards
by Dan van der Ster 22 Jan '21

22 Jan '21

Hi all, During rejoin an MDS can sometimes go OOM if the openfiles table is too large. The workaround has been described by ceph devs as "rados rm -p cephfs_metadata mds0_openfiles.0". On our cluster we have several such objects for rank 0: mds0_openfiles.0 exists with size: 199978 mds0_openfiles.1 exists with size: 153650 mds0_openfiles.2 exists with size: 40987 mds0_openfiles.3 exists with size: 7746 mds0_openfiles.4 exists with size: 413 If we suffer such an OOM, do we need to rm *all* of those objects or only the `.0` object? Best Regards, Dan

1 1

Large rbd
by Chris Dunlop 21 Jan '21

21 Jan '21

Hi, What limits are there on the "reasonable size" of an rbd? E.g. when I try to create a 1 PB rbd with default 4 MiB objects on my octopus cluster: $ rbd create --size 1P --data-pool rbd.ec rbd.meta/fs 2021-01-20T18:19:35.799+1100 7f47a99253c0 -1 librbd::image::CreateRequest: validate_layout: image size not compatible with object ...which somes from: == src/librbd/image/CreateRequest.cc bool validate_layout(CephContext *cct, uint64_t size, file_layout_t &layout) { if (!librbd::ObjectMap<>::is_compatible(layout, size)) { lderr(cct) << "image size not compatible with object map" << dendl; return false; } == src/librbd/ObjectMap.cc template <typename I> bool ObjectMap<I>::is_compatible(const file_layout_t& layout, uint64_t size) { uint64_t object_count = Striper::get_num_objects(layout, size); return (object_count <= cls::rbd::MAX_OBJECT_MAP_OBJECT_COUNT); } == src/cls/rbd/cls_rbd_types.h static const uint32_t MAX_OBJECT_MAP_OBJECT_COUNT = 256000000; For 4 MiB objects that object count equates to just over 976 TB. Is there any particular reason for that MAX_OBJECT_MAP_OBJECT_COUNT, or it just "this is crazy large, if you're trying to go over this you're doing something wrong, rethink your life..."? Yes, I realise I can increase the size of the objects to get a larger rbd, or drop the object-map support (and the fast-diff that goes along with it). I'm SO glad I found this limit now rather than starting on a smaller rbd and a finding the limit when I tried to grow the rbd underneath a rapidly filling filesystem. What else should I know? Background: I currently have nearly 0.5 PB on XFS (on lvm / raid6) and ZFS that I'm looking to move over to ceph. XFS is a requirement, for the reflinking (sadly not yet available in CephFS: https://tracker.ceph.com/issues/1680) The recommendation for XFS is to start larger, on a thin-provisioned store (hello rbd!), rather than start smaller and grow as needed - e.g. see the thread surrounding: https://www.spinics.net/lists/linux-xfs/msg20099.html Rather than a single large rbd, should I be looking at multiple smaller rbds linked together using lvm or somesuch? What are the tradeoffs? And whilst we're here... for an rbd with the data on an erasure-coded pool, how do you calculate the amount of rbd metadata required if/when the rbd data is fully allocated? Cheers, Chris

8 9

RBD-Mirror Snapshot Scalability
by Adam Boyhan 21 Jan '21

21 Jan '21

I have noticed that RBD-Mirror snapshot mode can only manage to take 1 snapshot per second. For example I have 21 images in a single pool. When the schedule is triggered it takes the mirror snapshot of each image 1 at a time. It doesn't feel or look like a performance issue as the OSD's are Micron 9300 PRO NVMe's and each server has 2x Intel Platinum 8268 CPU's. I was hoping that adding more RDB-Mirror instance would help, but that only seems to help with overall throughput. As it sits I have 3 RBD-Mirror instances running on each cluster. We run a 30 minute snapshot schedule to our remote site as it is, based on that I can only squeeze 1800 mirror snaps every 30 minutes. I was hoping there might be something I am missing with RBD-Mirror as far as scaling goes. Maybe multiple pools would be a solution and have other benefits?

2 3

RBD-Mirror Mirror Snapshot stuck
by Adam Boyhan 21 Jan '21

21 Jan '21

I have a rbd-mirror snapshot on 1 image that failed to replicate and now its not getting cleaned up. The cause of this was my fault based on my steps. Just trying to understand how to clean up/handle the situation. Here is how I got into this situation. - Created manual rbd snapshot on the image - On the remote cluster I cloned the snapshot - While cloned on the secondary cluster I made the mistake of deleting the snapshot on the primary - The subsequent mirror snapshot failed - I then removed the clone - The next mirror snapshot was successful but I was left with this mirror snapshot on the primary that I can't seem to get rid of root@Ccscephtest1:/var/log/ceph# rbd snap ls --all CephTestPool1/vm-100-disk-0 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 10082 .mirror.primary.90c53c21-6951-4218-9f07-9e983d490993.e0c63479-b09e-4c66-a65b-085b67a19907 2 TiB Thu Jan 21 07:10:09 2021 mirror (primary peer_uuids:[]) 10243 .mirror.primary.90c53c21-6951-4218-9f07-9e983d490993.483e55aa-2f64-4bb0-ac0f-7b5aac59830e 2 TiB Thu Jan 21 07:30:08 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770]) I have tried deleting the snap with "rbd snap rm" like normal user created snaps, but no luck. Anyway to force the deletion?

2 2

How to make HEALTH_ERR quickly and pain-free
by George Shuklin 21 Jan '21

21 Jan '21

I have hell of the question: how to make HEALTH_ERR status for a cluster without consequences? I'm working on CI tests and I need to check if our reaction to HEALTH_ERR is good. For this I need to take an empty cluster with an empty pool and do something. Preferably quick and reversible. For HEALTH_WARN the best thing I found is to change pool size to 1, it raises "1 pool(s) have no replicas configured" warning almost instantly and it can be reverted very quickly for empty pool. But HEALTH_ERR is a bit more tricky. Any ideas?

2 5

cephfs: massive drop in MDS requests per second with increasing number of caps
by Dietmar Rieder 20 Jan '21

20 Jan '21

Hi all, we noticed a massive drop in requests per second a cephfs client is able to perform when we do a recursive chown over a directory with millions of files. As soon as we see about 170k caps on the MDS, the client performance drops from about 660 reqs/sec to 70 reqs/sec. When we then clear dentries and inodes using "sync; echo 2 > /proc/sys/vm/drop_caches" on the client, the request go up to ~660 again just to drop again when reaching about 170k caps. See the attached screenshots. When we stop the chown process for a while and restart it ~25min later again it still performs very slowly and the MDS reqs/sec remain low (~60/sec.). Clearing the cache (dentries and inodes) on the client restores the performance again. When we run the same chown on another client in parallel, it starts again with reasonable good performance (while the first client is poorly performing) but eventually it gets slow again just like the first client. Can someone comment on this and explain it? How can this be solved, so that the performance remains stable? We are running ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) on all ceph cluster nodes and on all clients. The OS on all ceph cluster nodes and client nodes is CentOS 7.9. The filesystem is mounted via CentOS kernel client (latest official version). Thanks in advance. ~Best Dietmar -- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder(a)i-med.ac.at Web: http://www.icbi.at

3 8

RBD on windows
by Szabo, Istvan (Agoda) 20 Jan '21

20 Jan '21

Hi, I'm looking the suse documentation regarding their option to have rbd on win. I want to try on windows server 2019 vm, but I got this error: PS C:\Users\$admin$> rbd create image01 --size 4096 --pool windowstest -m 10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring C:/ProgramData/ceph/keyring 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: error parsing file C:/ProgramData/ceph/keyring: cannot parse buffer: Malformed input 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: failed to load C:/ProgramData/ceph/keyring: (5) Input/output error 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: error parsing file C:/ProgramData/ceph/keyring: cannot parse buffer: Malformed input 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: failed to load C:/ProgramData/ceph/keyring: (5) Input/output error 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: error parsing file C:/ProgramData/ceph/keyring: cannot parse buffer: Malformed input rbd: couldn't connect to the cluster! 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: failed to load C:/ProgramData/ceph/keyring: (5) Input/output error 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 monclient: keyring not found This is the keyring file: [client.windowstest] key = AQBJ7wdgdWLIMhAAle+/pg+26XvWsDv8PyPcvw== caps mon = "allow rw" caps osd = "allow rwx pool=windowstest" And this is the ceph.conf file on the windows client: [global] log to stderr = true run dir = C:/ProgramData/ceph crash dir = C:/ProgramData/ceph [client] keyring = C:/ProgramData/ceph/keyring log file = C:/ProgramData/ceph/$name.$pid.log admin socket = C:/ProgramData/ceph/$name.$pid.asok [global] mon host = [v2:10.118.199.231:3300,v1:10.118.199.231:6789] [v2:10.118.199.232:3300,v1:10.118.199.232:6789] [v2:10.118.199.233:3300,v1:10.118.199.233:6789] Commands I've tried: rbd create image01 --size 4096 --pool windowstest -m 10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring C:/ProgramData/ceph/keyring rbd create image01 --size 4096 --pool windowstest -m 10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring C:\ProgramData\ceph\keyring rbd create image01 --size 4096 --pool windowstest -m 10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring "C:/ProgramData/ceph/keyring" rbd create image01 --size 4096 --pool windowstest -m 10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring "C:\ProgramData\ceph\keyring" rbd create blank_image --size=1G The ceph version is luminous 12.2.8. Don't know why they don't find mon keyring. Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

[Query] Safe to discard bucket lock objects in reshard pool?
by Prasad Krishnan 20 Jan '21

20 Jan '21

Dear Ceph users, We have a slightly dated version of Luminous cluster in which dynamic bucket resharding was accidentally enabled due to a misconfig (we don't use this feature since the number of objects per bucket is capped). This resulted in creation of the RGW reshard pool with lots of bucket reshard lock objects (we have thousands of buckets) which is leading to clutter. Also, we've run into a malloc failure issue (similar to https://tracker.ceph.com/issues/21826 but not the same since we already use tcmalloc) on the OSDs in which these reshard lock objects are located and we'd like to reduce the objects that have to be copied out. My question to the community is: "Is it safe to discard the bucket reshard lock objects if we know that we'll never use the reshard feature on the cluster again?". The RGWs performed resharding several months ago due to a misconfiguration and we already have stale bucket instances which are due for cleanup on this cluster. Thanks, Prasad Krishnan -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

3 2

Re: Samsung PM883 3.84TB SSD performance
by mj 19 Jan '21

19 Jan '21

Hi both, Thanks for both quick replies, and both (of course) 100% spot-on! With 4k, IOPS is around 18530 :-) Thank you both, and apologies for the noise! Best, MJ On 19/01/2021 14:57, Marc Roos wrote: > You should test with 4k not 4M. > > >> -----Original Message----- >> From: mj <lists(a)merit.unu.edu> >> Sent: 19 January 2021 14:33 >> To: ceph-users <ceph-users(a)ceph.io> >> Subject: [ceph-users] Samsung PM883 3.84TB SSD performance >> >> Hi, >> >> We purchased Samsung PM883 3.84TB SSDs to be used as BlueStore SSDs in >> our cluster. >> >> I ran some benchmarks (write cache disabled, and with -sync=1 and >> -fsync=1) but with my little understandig, the results look terribly >> bad. (iops around 82!) >> >> Reading pages like yourcmc.ru, we should at least aim for > 1000s IOPS. >> (like 10000 up to 20000) >> >> Here is our complete output: >> >>> root@pve:~# hdparm -W 0 /dev/sdd >>> >>> /dev/sdd: >>> setting drive write-caching to 0 (off) >>> write-caching = 0 (off) >>> root@pve:~# fio -ioengine=libaio -fsync=1 -invalidate=1 -name=test - >> bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd >>> test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, >> iodepth=32 >>> fio-2.16 >>> Starting 1 process >>> Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta >> 00m:00s] >>> test: (groupid=0, jobs=1): err= 0: pid=3268778: Tue Jan 19 14:13:44 >> 2021 >>> write: io=19728MB, bw=336674KB/s, iops=82, runt= 60003msec >>> slat (usec): min=1853, max=4652, avg=2428.94, stdev=361.92 >>> clat (msec): min=9, max=652, avg=376.87, stdev=22.88 >>> lat (msec): min=11, max=655, avg=379.30, stdev=22.94 >>> clat percentiles (msec): >>> | 1.00th=[ 363], 5.00th=[ 367], 10.00th=[ 371], 20.00th=[ >> 371], >>> | 30.00th=[ 371], 40.00th=[ 375], 50.00th=[ 375], 60.00th=[ >> 379], >>> | 70.00th=[ 379], 80.00th=[ 383], 90.00th=[ 388], 95.00th=[ >> 392], >>> | 99.00th=[ 404], 99.50th=[ 420], 99.90th=[ 611], 99.95th=[ >> 635], >>> | 99.99th=[ 652] >>> lat (msec) : 10=0.02%, 50=0.06%, 100=0.10%, 250=0.24%, 500=99.25% >>> lat (msec) : 750=0.32% >>> cpu : usr=2.95%, sys=28.30%, ctx=47835, majf=0, minf=26 >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >>> =64=0.0% >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>> =64=0.0% >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >>> =64=0.0% >>> issued : total=r=0/w=4932/d=0, short=r=0/w=0/d=0, >> drop=r=0/w=0/d=0 >>> latency : target=0, window=0, percentile=100.00%, depth=32 >>> >>> Run status group 0 (all jobs): >>> WRITE: io=19728MB, aggrb=336674KB/s, minb=336674KB/s, >> maxb=336674KB/s, mint=60003msec, maxt=60003msec >>> >>> Disk stats (read/write): >>> sdd: ios=41/39330, merge=0/4995672, ticks=8/231816, in_queue=231440, >> util=79.43% >>> root@pve:~# fio -ioengine=libaio -sync=1 -invalidate=1 -name=test - >> bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd >>> test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, >> iodepth=32 >>> fio-2.16 >>> Starting 1 process >>> Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta >> 00m:00s] >>> test: (groupid=0, jobs=1): err= 0: pid=3270282: Tue Jan 19 14:14:58 >> 2021 >>> write: io=19468MB, bw=332248KB/s, iops=81, runt= 60001msec >>> slat (usec): min=11658, max=14287, avg=12318.59, stdev=371.39 >>> clat (usec): min=8, max=400078, avg=380931.63, stdev=19107.88 >>> lat (msec): min=11, max=412, avg=393.25, stdev=19.16 >>> clat percentiles (msec): >>> | 1.00th=[ 371], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ >> 375], >>> | 30.00th=[ 375], 40.00th=[ 379], 50.00th=[ 379], 60.00th=[ >> 383], >>> | 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 392], 95.00th=[ >> 396], >>> | 99.00th=[ 396], 99.50th=[ 400], 99.90th=[ 400], 99.95th=[ >> 400], >>> | 99.99th=[ 400] >>> lat (usec) : 10=0.02% >>> lat (msec) : 20=0.02%, 50=0.04%, 100=0.08%, 250=0.25%, 500=99.59% >>> cpu : usr=2.22%, sys=30.01%, ctx=53455, majf=0, minf=25 >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >>> =64=0.0% >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>> =64=0.0% >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >>> =64=0.0% >>> issued : total=r=0/w=4867/d=0, short=r=0/w=0/d=0, >> drop=r=0/w=0/d=0 >>> latency : target=0, window=0, percentile=100.00%, depth=32 >>> >>> Run status group 0 (all jobs): >>> WRITE: io=19468MB, aggrb=332248KB/s, minb=332248KB/s, >> maxb=332248KB/s, mint=60001msec, maxt=60001msec >>> >>> Disk stats (read/write): >>> sdd: ios=41/38830, merge=0/4931664, ticks=4/203944, in_queue=203700, >> util=79.55% >>> >>> >> >> The disk seems to be properly detected properly by the OS: >> >>> [4145961.899584] scsi 0:0:16:0: Direct-Access ATA SAMSUNG >> MZ7LH3T8 904Q PQ: 0 ANSI: 6 >>> [4145961.899596] scsi 0:0:16:0: SATA: handle(0x001a), >> sas_addr(0x500304801ef80883), phy(3), device_name(0x0000000000000000) >>> [4145961.899598] scsi 0:0:16:0: enclosure logical id >> (0x500304801ef808bf), slot(3) >>> [4145961.899599] scsi 0:0:16:0: enclosure level(0x0000), connector >> name( ) >>> [4145961.899649] scsi 0:0:16:0: atapi(n), ncq(y), asyn_notify(n), >> smart(y), fua(y), sw_preserve(y) >>> [4145961.900471] sd 0:0:16:0: Power-on or device reset occurred >>> [4145961.900544] sd 0:0:16:0: Attached scsi generic sg3 type 0 >>> [4145961.902367] sd 0:0:16:0: [sdd] 7501476528 512-byte logical >> blocks: (3.84 TB/3.49 TiB) >>> [4145961.902369] sd 0:0:16:0: [sdd] 4096-byte physical blocks >>> [4145961.904264] sd 0:0:16:0: [sdd] Write Protect is off >>> [4145961.904266] sd 0:0:16:0: [sdd] Mode Sense: 9b 00 10 08 >>> [4145961.904756] sd 0:0:16:0: [sdd] Write cache: enabled, read cache: >> enabled, supports DPO and FUA >>> [4145961.915996] sd 0:0:16:0: [sdd] Attached SCSI disk >> >> Anyone with a idea what we could be doing wrong? Or are these disks >> really unsuitable for OSD use? >> >> MJ >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0