- ceph-users - lists.ceph.io

IO500 Revised Call For Submissions Mid-2020 List
by committee＠io500.org 09 Jun '20

09 Jun '20

New Deadline: 13 July 2020 AoE NOTE: Given the short timeframe from the original announcement and complexities with some of the changes for this list, the deadline has been pushed out to give the community more time to participate. The BoF announcing the winners will be online 23 July 2020. Announcement: The IO500 [1] is now accepting and encouraging submissions for the upcoming 6th IO500 list. Once again, we are also accepting submissions to the 10 Node Challenge to encourage the submission of small scale results. The new ranked lists will be announced via live-stream at a virtual session. We hope to see many new results. The benchmark suite is designed to be easy to run and the community has multiple active support channels to help with any questions. Please note that submissions of all sizes are welcome; the site has customizable sorting so it is possible to submit on a small system and still get a very good per-client score for example. Additionally, the list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. More details below. Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 [1] was created in 2017, published its first list at SC17, and has grown exponentially since then. The need for such an initiative has long been known within High-Performance Computing; however, defining appropriate benchmarks had long been challenging. Despite this challenge, the community, after long and spirited discussion, finally reached consensus on a suite of benchmarks and a metric for resolving the scores into a single ranking. The multi-fold goals of the benchmark suite are as follows: * Maximizing simplicity in running the benchmark suite * Encouraging optimization and documentation of tuning parameters for performance * Allowing submitters to highlight their "hero run" performance numbers * Forcing submitters to simultaneously report performance for challenging IO patterns. Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes an IOR and mdtest run with highly constrained parameters forcing a difficult usage pattern in an attempt to determine a lower-bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Submitters are encouraged to share their tuning insights for publication. The goals of the community are also multi-fold: * Gather historical data for the sake of analysis and to aid predictions of storage futures * Collect tuning data to share valuable performance optimizations across the community * Encourage vendors and designers to optimize for workloads beyond "hero runs" * Establish bounded expectations for users, procurers, and administrators 10 NODE I/O CHALLENGE The 10 Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with, e.g., any number of servers. When submitting for the IO500 list, you can opt-in for "Participate in the 10 compute node challenge only", then we will not include the results into the ranked list. Other 10-node node submissions will be included in the full list and in the ranked list. We will announce the result in a separate derived list and in the full list but not on the ranked IO500 list at https://io500.org/. This information and rules for ISC20 submissions are available here: https://www.vi4io.org/io500/rules/submission Thanks, The IO500 Committee Links: ------ [1] http://io500.org/

1 0

rbd-mirror with snapshot, not doing any actaul data sync
by Hans van den Bogert 09 Jun '20

09 Jun '20

Hi list, I've awaited octopus for a along time to be able to use mirror with snapshotting, since my setup does not allow for journal based mirroring. (K8s/Rook 1.3.x with ceph 15.2.2) However, I seem to be stuck, i've come to the point where on the cluster on which the (non-active) replicas should reside I get this: ``` rbd mirror pool status -p replicapool --verbose ... pvc-f7ca0b55-ed38-4d9f-b306-7db6a0157e2e: global_id: d3a301f2-4f54-4e9e-b251-c55ddbb67dc6 state: up+starting_replay description: starting replay service: a on nldw1-6-26-1 last_update: 2020-06-07 11:54:54 ... ``` That seems good, right? But I don't see any actual data being copied into the failover cluster. Anybody any ideas what to check? Also, is it correct, you won't see mirror snapshots with the 'normal' `rbd snap` commands? Thanks in advance, Hans

2 3

poor cephFS performance on Nautilus 14.2.9 deployed by ceph_ansible
by Derrick Lin 09 Jun '20

09 Jun '20

Hi guys, We just deployed a CEPH 14.2.9 cluster with the following hardware: MDSS x 1 Xeon Gold 5122 3.6Ghz 192GB Mellanox ConnectX-4 Lx 25GbE MON x 3 Xeon Bronze 3103 1.7Ghz 48GB Mellanox ConnectX-4 Lx 25GbE 6 x 600GB 10K SAS OSD x 5 Xeon Silver 4110 2.1Ghz x 2 192GB Mellanox ConnectX-4 Lx 25GbE 16 x 10TB 7.2K NLSAS (block) 2 x 2TB Intel P4600 NVMe (block.db) Network is all Mellanox SN2410/SN2700 configured at 25GbE for both front and back network. Just for POC at this stage, the cluster was deployed by ceph_ansible without much customization and the initial test on its cephFS FUSE mount performance seems to be very low. We did some test with iozone the result as follow: ]# /opt/iozone/bin/iozone -i 0 -i 1-r 128k -s 5G -t 20 Iozone: Performance Test of File I/O Version $Revision: 3.465 $ Compiled for 64 bit mode. Build: linux-AMD64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, Vangel Bojaxhi, Ben England, Vikentsi Lapa, Alexey Skidanov. Run began: Tue Jun 2 16:40:53 2020 File size set to 5242880 kB Command line used: /opt/iozone/bin/iozone -i 0 -i 1-r -s 5G -t 20 128k Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Throughput test with 20 processes Each process writes a 5242880 kByte file in 4 kByte records Children see throughput for 20 initial writers = 35001.12 kB/sec Parent sees throughput for 20 initial writers = 34967.65 kB/sec Min throughput per process = 1748.22 kB/sec Max throughput per process = 1751.62 kB/sec Avg throughput per process = 1750.06 kB/sec Min xfer = 5232724.00 kB Children see throughput for 20 rewriters = 35704.79 kB/sec Parent sees throughput for 20 rewriters = 35704.30 kB/sec Min throughput per process = 1783.44 kB/sec Max throughput per process = 1786.29 kB/sec Avg throughput per process = 1785.24 kB/sec Min xfer = 5234532.00 kB Children see throughput for 20 readers = 49368539.50 kB/sec Parent sees throughput for 20 readers = 49317231.38 kB/sec Min throughput per process = 2414424.00 kB/sec Max throughput per process = 2599996.25 kB/sec Avg throughput per process = 2468426.98 kB/sec Min xfer = 4868708.00 kB Children see throughput for 20 re-readers = 48675891.50 kB/sec Parent sees throughput for 20 re-readers = 48617335.67 kB/sec Min throughput per process = 2316395.25 kB/sec Max throughput per process = 2703868.75 kB/sec Avg throughput per process = 2433794.58 kB/sec Min xfer = 4491704.00 kB We also did some dd tests, the write speed on a single test on our standard server is ~50MB/s but on a very big memory server, the speed is double ~ 80-90MB/s. We have zero experience on ceph and as said we haven't done more tuning at this stage. But if this sort of performance is way too low from those hardware spec? Any hints will be appreciated. Cheers D

3 4

ceph mgr prometheus
by Frank R 09 Jun '20

09 Jun '20

Hi all, Does the ceph manager prometheus module export bluestore rocksdb compaction times per OSD? I couldn't find anything. thx Frank

1 0

RDB Performance / High IOWaits.
by jameslipski＠protonmail.com 09 Jun '20

09 Jun '20

Greetings, I'm using ceph (14.2.2); in conjunction with Proxmox. Currently I'm just doing tests and ran into an issue relating to high I/O waits. Just to give a little bit of a background, specifically relating to my current ceph configurations; we have 6 nodes, each consisting of 2 osds (each node has 2x Intel SSDSC2KG019T8) OSD Type is bluestore. Global configurations (at least as shown on the proxmox interface) is as follows: [global] auth_client_required = xxxx auth_cluster_required = xxxx auth_service_required = xxxx cluster_network = 10.125.0.0/24 fsid = f64d2a67-98c3-4dbc-abfd-906ea7aaf314 mon_allow_pool_delete = true mon_host = 10.125.0.101 10.125.0.102 10.125.0.103 10.125.0.105 10.125.0.106 10.125.0.104 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 10.125.0.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring If I'm missing any relevant information relating to my ceph setup (I'm still learning this), please let me know. Each node consists of 2x Xeon E5-2660 v3. Where I ran into high I/O waits is when running a VM. VM is a mysql replication server (using 8 cores), and is performing mostly writes. When trying to narrow down, it was pointing to disk writes. The only thing I'm seeing in the ceph logs are the following: 2020-06-08 02:43:01.062082 mgr.node01 (mgr.2914449) 8009571 : cluster [DBG] pgmap v8009574: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 2.4 MiB/s wr, 274 op/s 2020-06-08 02:43:03.063137 mgr.node01 (mgr.2914449) 8009572 : cluster [DBG] pgmap v8009575: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 3.0 MiB/s wr, 380 op/s 2020-06-08 02:43:05.064125 mgr.node01 (mgr.2914449) 8009573 : cluster [DBG] pgmap v8009576: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 2.9 MiB/s wr, 332 op/s 2020-06-08 02:43:07.065373 mgr.node01 (mgr.2914449) 8009574 : cluster [DBG] pgmap v8009577: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 2.7 MiB/s wr, 313 op/s 2020-06-08 02:43:09.066210 mgr.node01 (mgr.2914449) 8009575 : cluster [DBG] pgmap v8009578: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 2.9 MiB/s wr, 350 op/s 2020-06-08 02:43:11.066913 mgr.node01 (mgr.2914449) 8009576 : cluster [DBG] pgmap v8009579: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.1 MiB/s wr, 346 op/s 2020-06-08 02:43:13.067926 mgr.node01 (mgr.2914449) 8009577 : cluster [DBG] pgmap v8009580: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.5 MiB/s wr, 408 op/s 2020-06-08 02:43:15.068834 mgr.node01 (mgr.2914449) 8009578 : cluster [DBG] pgmap v8009581: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.0 MiB/s wr, 320 op/s 2020-06-08 02:43:17.069627 mgr.node01 (mgr.2914449) 8009579 : cluster [DBG] pgmap v8009582: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 2.5 MiB/s wr, 285 op/s 2020-06-08 02:43:19.070507 mgr.node01 (mgr.2914449) 8009580 : cluster [DBG] pgmap v8009583: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.0 MiB/s wr, 349 op/s 2020-06-08 02:43:21.071241 mgr.node01 (mgr.2914449) 8009581 : cluster [DBG] pgmap v8009584: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 2.8 MiB/s wr, 319 op/s 2020-06-08 02:43:23.072286 mgr.node01 (mgr.2914449) 8009582 : cluster [DBG] pgmap v8009585: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 2.7 MiB/s wr, 329 op/s 2020-06-08 02:43:25.073369 mgr.node01 (mgr.2914449) 8009583 : cluster [DBG] pgmap v8009586: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 2.8 MiB/s wr, 304 op/s 2020-06-08 02:43:27.074315 mgr.node01 (mgr.2914449) 8009584 : cluster [DBG] pgmap v8009587: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 2.2 MiB/s wr, 262 op/s 2020-06-08 02:43:29.075284 mgr.node01 (mgr.2914449) 8009585 : cluster [DBG] pgmap v8009588: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 2.9 MiB/s wr, 342 op/s 2020-06-08 02:43:31.076180 mgr.node01 (mgr.2914449) 8009586 : cluster [DBG] pgmap v8009589: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 2.4 MiB/s wr, 269 op/s 2020-06-08 02:43:33.077523 mgr.node01 (mgr.2914449) 8009587 : cluster [DBG] pgmap v8009590: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.4 MiB/s wr, 389 op/s 2020-06-08 02:43:35.078543 mgr.node01 (mgr.2914449) 8009588 : cluster [DBG] pgmap v8009591: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.1 MiB/s wr, 344 op/s 2020-06-08 02:43:37.079428 mgr.node01 (mgr.2914449) 8009589 : cluster [DBG] pgmap v8009592: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.0 MiB/s wr, 334 op/s 2020-06-08 02:43:39.080419 mgr.node01 (mgr.2914449) 8009590 : cluster [DBG] pgmap v8009593: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.3 MiB/s wr, 377 op/s I'm not sure what could be causing high I/O waits; or whether this is an issue relating to my ceph configurations. Any suggestions would be appreciated/ or if you need any additional information, let me know what you need and I'll post them. Thank you.

1 0

Trying to upgrade to octopus removes current version of ceph release and tries to install older version...
by Drew Weaver 08 Jun '20

08 Jun '20

Hi cluster is version 14.2.9 ceph-deploy v2.0.1 using command ceph-deploy install --release octopus mon0 mon1 mon2 result is this command being run: sudo yum remove -y ceph-release which removes this package: ceph-release noarch 1-1.el7 @/ceph-release-1-0.el7.noarch Then it tries to install this older version: Running command: sudo yum install -y https://download.ceph.com/rpm-octopus/el7/noarch/ceph-release-1-0.el7.noarc… Which does not actually exist. Am I doing this wrong?

1 1

Access RBD pools from clients
by Amjad Kotobi 08 Jun '20

08 Jun '20

Hi, I’m trying to access CEPH cluster from client machines via RBD for operations like listing pools, creating image, etc.. I put all needed keys/configuration in ceph directory and when I try to list pool rbd hung/wait forever Client machine in IP range of: 136.172.26.0/24 The command e.g. I’m trying is rbd -c /etc/ceph/ceph.conf -p vms ls -n client.cinder4 My CEPH cluster configuration file “ceph.conf” is ########## [global] fsid = e6d92380-4d77-4064-8797-2722fecfd1f4 mon initial members = ceph1,ceph2,ceph3 mon host = 10.60.1.21,10.60.1.22,10.60.1.23 public network = 10.60.1.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx mon_allow_pool_delete = true osd pool default size = 3 osd pool default min size = 2 [mgr.ceph1] mgr tick period = 10 mon mgr beacon grace = 60 [client.cinder4] keyring = /etc/ceph/ceph.client.cinder4.keyring ########## There is no firewall in between and accessing/telnet port is very much possible. I’m suspecting because of different IP subnet, MON CEPH is not allowing client. Any idea and help on this will be great. Thanks Amjad

2 6

Re: crashing OSD: ceph_assert(is_valid_io(off, len))
by Harald Staub 08 Jun '20

08 Jun '20

(really sorry for spamming, but it is still waiting for moderator, so trying with xz ...) On 08.06.20 17:21, Harald Staub wrote: > (and now with trimmed attachment because of size restriction: only the > debug log) > > On 08.06.20 16:53, Harald Staub wrote: >> (and now with attachment ...) >> >> On 08.06.20 16:51, Harald Staub wrote: >>> Hi Igor >>> >>> Thank you for looking into this! I attached the complete log of >>> today, with the preceding "ceph_assert(h->file->fnode.ino != 1)" at >>> 13:13:22.609, the first "FAILED ceph_assert(is_valid_io(off, len))" >>> at 13:44:52.059, the debug log starting at 16:42:20.883. >>> >>> Cheers >>> Harry >>> >>> On 08.06.20 16:37, Igor Fedotov wrote: >>>> Hi Harald, >>>> >>>> was this exact OSD suffering from "ceph_assert(h->file->fnode.ino != >>>> 1)"? >>>> >>>> Could you please collect extended log with debug-bluefs set ot 20? >>>> >>>> >>>> Thanks, >>>> >>>> Igor >>>> >>>> On 6/8/2020 4:48 PM, Harald Staub wrote: >>>>> This is again about our bad cluster, with far too many objects. Now >>>>> another OSD crashes immediately at startup: >>>>> >>>>> /build/ceph-14.2.8/src/os/bluestore/KernelDevice.cc: 944: FAILED >>>>> ceph_assert(is_valid_io(off, len)) >>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>>>> const*)+0x152) [0x5601938e0e92] >>>>> 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char >>>>> const*, char const*, ...)+0) [0x5601938e106d] >>>>> 3: (KernelDevice::read(unsigned long, unsigned long, >>>>> ceph::buffer::v14_2_0::list*, IOContext*, bool)+0x8e0) >>>>> [0x560193f2ae90] >>>>> 4: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, >>>>> unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, >>>>> char*)+0x4f6) [0x560193ee0506] >>>>> 5: (BlueFS::_replay(bool, bool)+0x489) [0x560193ee14e9] >>>>> 6: (BlueFS::mount()+0x219) [0x560193ef4319] >>>>> 7: (BlueStore::_open_bluefs(bool)+0x41) [0x560193de2281] >>>>> 8: (BlueStore::_open_db(bool, bool, bool)+0x88c) [0x560193de347c] >>>>> 9: (BlueStore::_open_db_and_around(bool)+0x44) [0x560193dfa134] >>>>> 10: (BlueStore::_mount(bool, bool)+0x584) [0x560193e4a804] >>>>> 11: (OSD::init()+0x3b7) [0x56019398f957] >>>>> 12: (main()+0x3cdb) [0x5601938e85cb] >>>>> 13: (__libc_start_main()+0xe7) [0x7f54cdf1bb97] >>>>> 14: (_start()+0x2a) [0x56019391b08a] >>>>> >>>>> 2020-06-08 13:44:52.063 7f54d169ec00 -1 *** Caught signal (Aborted) ** >>>>> in thread 7f54d169ec00 thread_name:ceph-osd >>>>> >>>>> ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) >>>>> nautilus (stable) >>>>> 1: (()+0x12890) [0x7f54cf286890] >>>>> 2: (gsignal()+0xc7) [0x7f54cdf38e97] >>>>> 3: (abort()+0x141) [0x7f54cdf3a801] >>>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>>>> const*)+0x1a3) [0x5601938e0ee3] >>>>> 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char >>>>> const*, char const*, ...)+0) [0x5601938e106d] >>>>> 6: (KernelDevice::read(unsigned long, unsigned long, >>>>> ceph::buffer::v14_2_0::list*, IOContext*, bool)+0x8e0) >>>>> [0x560193f2ae90] >>>>> 7: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, >>>>> unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, >>>>> char*)+0x4f6) [0x560193ee0506] >>>>> 8: (BlueFS::_replay(bool, bool)+0x489) [0x560193ee14e9] >>>>> 9: (BlueFS::mount()+0x219) [0x560193ef4319] >>>>> 10: (BlueStore::_open_bluefs(bool)+0x41) [0x560193de2281] >>>>> 11: (BlueStore::_open_db(bool, bool, bool)+0x88c) [0x560193de347c] >>>>> 12: (BlueStore::_open_db_and_around(bool)+0x44) [0x560193dfa134] >>>>> 13: (BlueStore::_mount(bool, bool)+0x584) [0x560193e4a804] >>>>> 14: (OSD::init()+0x3b7) [0x56019398f957] >>>>> 15: (main()+0x3cdb) [0x5601938e85cb] >>>>> 16: (__libc_start_main()+0xe7) [0x7f54cdf1bb97] >>>>> 17: (_start()+0x2a) [0x56019391b08a] >>>>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is >>>>> needed to interpret this. >>>>> >>>>> There was a preceding assert (earlier written about): >>>>> >>>>> /build/ceph-14.2.8/src/os/bluestore/BlueFS.cc: 2261: FAILED >>>>> ceph_assert(h->file->fnode.ino != 1) >>>>> >>>>> Any ideas that I could try to save this OSDs? >>>>> >>>>> Cheers >>>>> Harry >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Broken PG in cephfs data_pool (lost objects)
by Francois Legrand 08 Jun '20

08 Jun '20

Hi all, We have a cephfs with data_pool in erasure coding (3+2) ans 1024 pg (nautilus 14.2.8). One of the pgs is partially destroyed (we lost 3 osd thus 3 shards), it have 143 objects unfound and is stuck in state "active+recovery_unfound+undersized+degraded+remapped". We then lost some datas (we are using cephfs-data-scan pg_files... to identify files with data on the bad pg) . We thus created a new filesystem (this time with data_pool in replica 3) and we are copying all the datas from the broken FS to the new one. But we need to remove files from the broken FS after copy to free space (because there will not be enough space on the cluster). To avoid problems of strays we removed the snapshots on the broken FS before deleting files. The point is that the mds managing the broken FS is now "Behind on trimming (123036/128) max_segments: 128, num_segments: 123036" and have 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 83645 secs. The slow IO correspond to osd 27 which is acting_primary for the broken PG. and the broken pg have a long "snap_trimq": "[1e0c~1,1e0e~1,1e12~1,1e16~1,1e18~1,1e1a~1,........" and "snap_trimq_len": 460. It then seems that cephfs is not able to trim ops corresponding to the deletion of objects and snaps which have data on the broken PG, probably because the pg is not healty. Is there a way to tell ceph/cephfs to flush or forget about (only) lost objects on the broken pg and get this pg healty enough to perform trimming ? thanks for your help F.

1 0

crashing OSD: ceph_assert(is_valid_io(off, len))
by Harald Staub 08 Jun '20

08 Jun '20

This is again about our bad cluster, with far too many objects. Now another OSD crashes immediately at startup: /build/ceph-14.2.8/src/os/bluestore/KernelDevice.cc: 944: FAILED ceph_assert(is_valid_io(off, len)) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x5601938e0e92] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x5601938e106d] 3: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, IOContext*, bool)+0x8e0) [0x560193f2ae90] 4: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, char*)+0x4f6) [0x560193ee0506] 5: (BlueFS::_replay(bool, bool)+0x489) [0x560193ee14e9] 6: (BlueFS::mount()+0x219) [0x560193ef4319] 7: (BlueStore::_open_bluefs(bool)+0x41) [0x560193de2281] 8: (BlueStore::_open_db(bool, bool, bool)+0x88c) [0x560193de347c] 9: (BlueStore::_open_db_and_around(bool)+0x44) [0x560193dfa134] 10: (BlueStore::_mount(bool, bool)+0x584) [0x560193e4a804] 11: (OSD::init()+0x3b7) [0x56019398f957] 12: (main()+0x3cdb) [0x5601938e85cb] 13: (__libc_start_main()+0xe7) [0x7f54cdf1bb97] 14: (_start()+0x2a) [0x56019391b08a] 2020-06-08 13:44:52.063 7f54d169ec00 -1 *** Caught signal (Aborted) ** in thread 7f54d169ec00 thread_name:ceph-osd ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) 1: (()+0x12890) [0x7f54cf286890] 2: (gsignal()+0xc7) [0x7f54cdf38e97] 3: (abort()+0x141) [0x7f54cdf3a801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x5601938e0ee3] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x5601938e106d] 6: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, IOContext*, bool)+0x8e0) [0x560193f2ae90] 7: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, char*)+0x4f6) [0x560193ee0506] 8: (BlueFS::_replay(bool, bool)+0x489) [0x560193ee14e9] 9: (BlueFS::mount()+0x219) [0x560193ef4319] 10: (BlueStore::_open_bluefs(bool)+0x41) [0x560193de2281] 11: (BlueStore::_open_db(bool, bool, bool)+0x88c) [0x560193de347c] 12: (BlueStore::_open_db_and_around(bool)+0x44) [0x560193dfa134] 13: (BlueStore::_mount(bool, bool)+0x584) [0x560193e4a804] 14: (OSD::init()+0x3b7) [0x56019398f957] 15: (main()+0x3cdb) [0x5601938e85cb] 16: (__libc_start_main()+0xe7) [0x7f54cdf1bb97] 17: (_start()+0x2a) [0x56019391b08a] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. There was a preceding assert (earlier written about): /build/ceph-14.2.8/src/os/bluestore/BlueFS.cc: 2261: FAILED ceph_assert(h->file->fnode.ino != 1) Any ideas that I could try to save this OSDs? Cheers Harry

2 3