April 2021 - ceph-users - lists.ceph.io

by Dan van der Ster

Hi all, bdev_enable_discard has been in ceph for several major releases now but it is still off by default. Did anyone try it recently -- is it safe to use? And do you have perf numbers before and after enabling? Cheers, Dan

3 years, 1 month

3
5
0 0

Revisit Large OMAP Objects

by DHilsbos＠performair.com

All; We run 2 Nautilus clusters, with RADOSGW replication (14.2.11 --> 14.2.16). Initially our bucket grew very quickly, as I was loading old data into it and we quickly ran into Large OMAP Object warnings. I have since done a couple manual reshards, which has fixed the warning on the primary cluster. I have never been able to get rid of the issue on the cluster with the replica. I prior conversation on this list led me to this command: radosgw-admin reshard stale-instances list --yes-i-really-mean-it The results of which look like this: [ "nextcloud-ra:f91aeff8-a365-47b4-a1c8-928cd66134e8.185262.1", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.6", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.2", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.5", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.4", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.3", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.1", "3520ae821f974340afd018110c1065b8/OS Development:f91aeff8-a365-47b4-a1c8-928cd66134e8.4298264.1", "10dfdfadb7374ea1ba37bee1435d87ad/volumebackups:f91aeff8-a365-47b4-a1c8-928cd66134e8.4298264.2", "WorkOrder:f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1" ] I find this particularly interesting, as nextcloud-ra, <swift>/OS Development, <swift>/volumbackups, and WorkOrder buckets no longer exist. When I run: for obj in $(rados -p 300.rgw.buckets.index ls | grep f91aeff8-a365-47b4-a1c8-928cd66134e8.3512190.1); do printf "%-60s %7d\n" $obj $(rados -p 300.rgw.buckets.index listomapkeys $obj | wc -l); done I get the expected 64 entries, with counts around 20000 +/- 1000. Are the above listed stale instances ok to delete? If so, how do I go about doing so? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

3 years, 1 month

4
5
0 0

ERROR: read_key_entry() idx= 1000_ ret=-2

by by morphin

Hello everyone! I'm running nautilus 14.2.16 and I'm using RGW with Beast frontend. I see this eror log in every SSD osd which is using for rgw index. Can you please tell me what is the problem? OSD LOG: cls_rgw.cc:1102: ERROR: read_key_entry() idx=�1000_matches/xdir/05/21/27260.jpg ret=-2 cls_rgw.cc:1102: ERROR: read_key_entry() idx=�1000_matches/xdir/05/21/27253.jpg ret=-2 RADOSGW LOG: 2021-04-15 01:53:54.385 7f2e0f8e7700 1 beast: 0x55a4439f8710: 10.151.101.15 - - [2021-04-15 01:53:54.0.385327s] "HEAD /xdir/04/13/704745.jpg HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1062.12.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.251-b08 java/1.8.0_251 groovy/2.4.3 vendor/Oracle_Corporation" - 2021-04-15 01:53:54.385 7f2d8b7df700 1 ====== starting new request req=0x55a4439f8710 ===== 2021-04-15 01:53:54.405 7f2e008c9700 1 ====== req done req=0x55a43dbc6710 op status=0 http_status=204 latency=0.300003s ====== 2021-04-15 01:53:54.405 7f2e008c9700 1 beast: 0x55a43dbc6710: 10.151.101.15 - - [2021-04-15 01:53:54.0.405327s] "DELETE /xdir/05/21/21586.gz HTTP/1.1" 204 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1062.12.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.251-b08 java/1.8.0_251 groovy/2.4.3 vendor/Oracle_Corporation" - 2021-04-15 01:53:54.405 7f2d92fee700 1 ====== starting new request req=0x55a43dbc6710 ===== 2021-04-15 01:53:54.405 7f2d92fee700 0 WARNING: couldn't find acl header for object, generating default 2021-04-15 01:53:54.405 7f2d92fee700 1 ====== req done req=0x55a43dbc6710 op status=0 http_status=200 latency=0s ====== 2021-04-15 01:53:54.405 7f2d92fee700 1 beast: 0x55a43dbc6710: 10.151.101.15 - - [2021-04-15 01:53:54.0.405327s] "HEAD /xdir/2013/11/20/2a67508e-d7dd-4e0f-b959-d7575d5f65b1 HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281 groovy/2.5.6 vendor/Oracle_Corporation" - CEPH OSD DF ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 19 ssd 0.87320 1.00000 894 GiB 436 GiB 101 GiB 332 GiB 2.5 GiB 458 GiB 48.75 1.84 115 up 208 ssd 0.87329 1.00000 894 GiB 161 GiB 87 GiB 73 GiB 978 MiB 733 GiB 18.00 0.68 113 up 199 ssd 0.87320 1.00000 894 GiB 272 GiB 106 GiB 163 GiB 2.4 GiB 623 GiB 30.37 1.14 123 up 202 ssd 0.87329 1.00000 894 GiB 239 GiB 73 GiB 165 GiB 1.4 GiB 655 GiB 26.77 1.01 106 up 39 ssd 0.87320 1.00000 894 GiB 450 GiB 87 GiB 361 GiB 2.3 GiB 444 GiB 50.36 1.90 113 up 207 ssd 0.87329 1.00000 894 GiB 204 GiB 100 GiB 98 GiB 6.0 GiB 691 GiB 22.76 0.86 118 up 59 ssd 0.87320 1.00000 894 GiB 372 GiB 107 GiB 263 GiB 3.0 GiB 522 GiB 41.64 1.57 122 up 203 ssd 0.87329 1.00000 894 GiB 206 GiB 79 GiB 124 GiB 2.4 GiB 689 GiB 23.00 0.87 117 up 79 ssd 0.87320 1.00000 894 GiB 447 GiB 103 GiB 342 GiB 1.8 GiB 447 GiB 49.97 1.88 120 up 206 ssd 0.87329 1.00000 894 GiB 200 GiB 81 GiB 119 GiB 1.0 GiB 694 GiB 22.38 0.84 94 up 99 ssd 0.87320 1.00000 894 GiB 333 GiB 87 GiB 244 GiB 2.0 GiB 562 GiB 37.19 1.40 106 up 205 ssd 0.87329 1.00000 894 GiB 316 GiB 83 GiB 232 GiB 1.1 GiB 579 GiB 35.29 1.33 117 up 114 ssd 0.87329 1.00000 894 GiB 256 GiB 100 GiB 154 GiB 1.7 GiB 638 GiB 28.61 1.08 113 up 200 ssd 0.87329 1.00000 894 GiB 266 GiB 100 GiB 165 GiB 1.1 GiB 628 GiB 29.76 1.12 128 up 139 ssd 0.87320 1.00000 894 GiB 234 GiB 79 GiB 153 GiB 1.7 GiB 660 GiB 26.14 0.98 104 up 204 ssd 0.87329 1.00000 894 GiB 173 GiB 113 GiB 59 GiB 1.2 GiB 721 GiB 19.37 0.73 124 up 119 ssd 0.87329 1.00000 894 GiB 248 GiB 108 GiB 139 GiB 1.9 GiB 646 GiB 27.76 1.05 130 up 159 ssd 0.87329 1.00000 894 GiB 196 GiB 94 GiB 99 GiB 2.6 GiB 699 GiB 21.87 0.82 109 up 179 ssd 0.87329 1.00000 894 GiB 427 GiB 81 GiB 341 GiB 4.7 GiB 467 GiB 47.73 1.80 114 up 201 ssd 0.87329 1.00000 894 GiB 346 GiB 102 GiB 242 GiB 1.8 GiB 548 GiB 38.71 1.46 128 up CEPH IOSTAT +---------------+---------------+---------------+---------------+---------------+---------------+ | Read | Write | Total | Read IOPS | Write IOPS | Total IOPS | +---------------+---------------+---------------+---------------+---------------+---------------+ | 329 MiB/s | 39 MiB/s | 368 MiB/s | 109027 | 1646 | 110673 | | 329 MiB/s | 39 MiB/s | 368 MiB/s | 109027 | 1646 | 110673 | | 331 MiB/s | 39 MiB/s | 371 MiB/s | 114915 | 1631 | 116547 | | 331 MiB/s | 39 MiB/s | 371 MiB/s | 114915 | 1631 | 116547 | | 308 MiB/s | 42 MiB/s | 350 MiB/s | 108469 | 1635 | 110104 | | 308 MiB/s | 42 MiB/s | 350 MiB/s | 108469 | 1635 | 110104 | | 291 MiB/s | 44 MiB/s | 335 MiB/s | 105828 | 1687 | 107516 |

3 years, 1 month

1
1
0 0

How to disable ceph-grafana during cephadm bootstrap

by mabi

Hello, When bootstrapping a new ceph Octopus cluster with "cephadm bootstrap", how can I tell the cephadm bootstrap NOT to install the ceph-grafana container? Thank you very much in advance for your answer. Best regards, Mabi

3 years, 1 month

2
2
0 0

Ceph Month June 2021 Event

by Mike Perez

Hi everyone, In June 2021, we're hosting a month of Ceph presentations, lightning talks, and unconference sessions such as BOFs. There is no registration or cost to attend this event. The CFP is now open until May 12th. https://ceph.io/events/ceph-month-june-2021/cfp Speakers will receive confirmation that their presentation is accepted and further instructions for scheduling by May 16th. The schedule will be available on May 19th. Join the Ceph community as we discuss how Ceph, the massively scalable, open-source, software-defined storage system, can radically improve the economics and management of data storage for your enterprise. -- Mike Perez

3 years, 1 month

1
0
0 0

Monitor dissapears/stopped after testing monitor-host loss and recovery

by Kai Börnert

Hi, I'm currently testing some disaster scenarios. When removing one osd/monitor host, I see that a new quorum is build without the missing host. The missing host is listed in the dashboard under Not In Quorum, so probably everything as expected. After restarting the host, I see that the osd's come back online and everything appears to be working, however the quorum is still only with two monitors. Looking at the services, I can see that it is somehow stopped. Is this expected and I must start it manually somehow, or should it work? The whole cluster is deployed using cephadm (the node was the initial bootstrap one, if that is important) Greetings, Kai

3 years, 1 month

1
0
0 0

Exporting CephFS using Samba preferred method

by Martin Palma

Hello, what is the currently preferred method, in terms of stability and performance, for exporting a CephFS directory with Samba? - locally mount the CephFS directory and export it via Samba? - using the "vfs_ceph" module of Samba? Best, Martin

3 years, 1 month

4
3
0 0

As the cluster is filling up, write performance decreases

by zp_8483

Backend: XFS for the filestore back-end. In our testing, we found the performance decreases when cluster usage exceed default nearfull ratio(85%), is it by design?

3 years, 1 month

3
2
0 0

ceph rgw why are reads faster for larger than 64kb object size

by Ronnie Puthukkeril

Environment: Ceph Nautilus 14.2.8 Object Storage Data nodes: 12 * HDD OSDs drives each with a 12TB capacity + 2 * SSD OSDs drives for rgw bucket index pool & rgw meta pool. Custom configs (since we dealing with a majority smaller sized objects) bluestore_min_alloc_size_ssd 4096 bluestore_min_alloc_size_hdd 4096 Observations from cosbench performance tests Stage Op-Type Op-Count Byte-Count Avg-ResTime s7-read1KB 48W read 2004202 2004202000 43.11 s13-read2KB 48W read 2013906 4027812000 42.9 s19-read4KB 48W read 2014701 8058804000 42.88 s25-read8KB 48W read 2002337 16018696000 43.15 s31-read16KB 48W read 1987785 31804560000 43.46 s37-read32KB 48W read 1976190 63238080000 43.7 s43-read64KB 48W read 1929183 123467712000 44.78 s49-read128KB 48W read 9965032 1275524096000 8.67 s55-read256KB 48W read 6505554 1665421824000 13.28 The response time improves drastically when the object size is greater than 64KB. What could be the reason? Thanks, Ronnie [https://d1dejaj6dcqv24.cloudfront.net/asset/image/email-banner-384-2x.png]<https://www.qualys.com/email-banner> This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. NOTE that all incoming emails sent to Qualys email accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional emails ("spam"). If you have any concerns about this process, please contact us.

3 years, 1 month

1
3
0 0

Announcing go-ceph v0.9.0

by John Mulligan

I'm happy to announce another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.9.0 Changes in the release are detailed in the link above. The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. Specific questions, comments, bugs etc are best directed at our github issues tracker or github discussions forum. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

3 years, 1 month

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users April 2021