June 2020 - ceph-users - lists.ceph.io

by Eugen Block

Hi, IIUC those are standalone OSDs (no separate rocksDB on faster devices, I assume). Have you checked OSD saturation (e.g. iostat) on the nodes? Although I'd expect "slow requests" in the logs if the disks were saturated but that would be my first guess. Zitat von jameslipski(a)protonmail.com: > Greetings, > > I'm using ceph (14.2.2); in conjunction with Proxmox. Currently I'm > just doing tests and ran into an issue relating to high I/O waits. > Just to give a little bit of a background, specifically relating to > my current ceph configurations; we have 6 nodes, each consisting of > 2 osds (each node has 2x Intel SSDSC2KG019T8) OSD Type is bluestore. > Global configurations (at least as shown on the proxmox interface) > is as follows: > > [global] > auth_client_required = xxxx > auth_cluster_required = xxxx > auth_service_required = xxxx > cluster_network = 10.125.0.0/24 > fsid = f64d2a67-98c3-4dbc-abfd-906ea7aaf314 > mon_allow_pool_delete = true > mon_host = 10.125.0.101 10.125.0.102 10.125.0.103 10.125.0.105 > 10.125.0.106 10.125.0.104 > osd_pool_default_min_size = 2 > osd_pool_default_size = 3 > public_network = 10.125.0.0/24 > > [client] > keyring = /etc/pve/priv/$cluster.$name.keyring > > > > If I'm missing any relevant information relating to my ceph setup > (I'm still learning this), please let me know. > > Each node consists of 2x Xeon E5-2660 v3. Where I ran into high I/O > waits is when running a VM. VM is a mysql replication server (using > 8 cores), and is performing mostly writes. When trying to narrow > down, it was pointing to disk writes. The only thing I'm seeing in > the ceph logs are the following: > > 2020-06-08 02:43:01.062082 mgr.node01 (mgr.2914449) 8009571 : > cluster [DBG] pgmap v8009574: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 2.4 MiB/s wr, 274 op/s > 2020-06-08 02:43:03.063137 mgr.node01 (mgr.2914449) 8009572 : > cluster [DBG] pgmap v8009575: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 3.0 MiB/s wr, > 380 op/s > 2020-06-08 02:43:05.064125 mgr.node01 (mgr.2914449) 8009573 : > cluster [DBG] pgmap v8009576: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 2.9 MiB/s wr, > 332 op/s > 2020-06-08 02:43:07.065373 mgr.node01 (mgr.2914449) 8009574 : > cluster [DBG] pgmap v8009577: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 2.7 MiB/s wr, > 313 op/s > 2020-06-08 02:43:09.066210 mgr.node01 (mgr.2914449) 8009575 : > cluster [DBG] pgmap v8009578: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 2.9 MiB/s wr, > 350 op/s > 2020-06-08 02:43:11.066913 mgr.node01 (mgr.2914449) 8009576 : > cluster [DBG] pgmap v8009579: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.1 MiB/s wr, > 346 op/s > 2020-06-08 02:43:13.067926 mgr.node01 (mgr.2914449) 8009577 : > cluster [DBG] pgmap v8009580: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.5 MiB/s wr, > 408 op/s > 2020-06-08 02:43:15.068834 mgr.node01 (mgr.2914449) 8009578 : > cluster [DBG] pgmap v8009581: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.0 MiB/s wr, > 320 op/s > 2020-06-08 02:43:17.069627 mgr.node01 (mgr.2914449) 8009579 : > cluster [DBG] pgmap v8009582: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 2.5 MiB/s wr, > 285 op/s > 2020-06-08 02:43:19.070507 mgr.node01 (mgr.2914449) 8009580 : > cluster [DBG] pgmap v8009583: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.0 MiB/s wr, > 349 op/s > 2020-06-08 02:43:21.071241 mgr.node01 (mgr.2914449) 8009581 : > cluster [DBG] pgmap v8009584: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 2.8 MiB/s wr, > 319 op/s > 2020-06-08 02:43:23.072286 mgr.node01 (mgr.2914449) 8009582 : > cluster [DBG] pgmap v8009585: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 2.7 MiB/s wr, 329 op/s > 2020-06-08 02:43:25.073369 mgr.node01 (mgr.2914449) 8009583 : > cluster [DBG] pgmap v8009586: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 2.8 MiB/s wr, 304 op/s > 2020-06-08 02:43:27.074315 mgr.node01 (mgr.2914449) 8009584 : > cluster [DBG] pgmap v8009587: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 2.2 MiB/s wr, 262 op/s > 2020-06-08 02:43:29.075284 mgr.node01 (mgr.2914449) 8009585 : > cluster [DBG] pgmap v8009588: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 2.9 MiB/s wr, > 342 op/s > 2020-06-08 02:43:31.076180 mgr.node01 (mgr.2914449) 8009586 : > cluster [DBG] pgmap v8009589: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 2.4 MiB/s wr, > 269 op/s > 2020-06-08 02:43:33.077523 mgr.node01 (mgr.2914449) 8009587 : > cluster [DBG] pgmap v8009590: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.4 MiB/s wr, > 389 op/s > 2020-06-08 02:43:35.078543 mgr.node01 (mgr.2914449) 8009588 : > cluster [DBG] pgmap v8009591: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.1 MiB/s wr, > 344 op/s > 2020-06-08 02:43:37.079428 mgr.node01 (mgr.2914449) 8009589 : > cluster [DBG] pgmap v8009592: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.0 MiB/s wr, > 334 op/s > 2020-06-08 02:43:39.080419 mgr.node01 (mgr.2914449) 8009590 : > cluster [DBG] pgmap v8009593: 512 pgs: 512 active+clean; 246 GiB > data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.3 MiB/s wr, > 377 op/s > > > I'm not sure what could be causing high I/O waits; or whether this > is an issue relating to my ceph configurations. Any suggestions > would be appreciated/ or if you need any additional information, let > me know what you need and I'll post them. > > Thank you. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 years, 10 months

1
0
0 0

Rebalancing after modifying CRUSH map

by Brett Randall

Hi all We are looking at implementing Ceph/CephFS for a project. Over time, we may wish to add additional replicas to our cluster. If we modify a CRUSH map, is there a way of then requesting Ceph to re-evaluate the placement of objects across the cluster according to the modified CRUSH map? Brett

3 years, 10 months

3
3
0 0

Maximum size of data in crush_choose_firstn Ceph CRUSH source code

by Bobby

Hi all, I have a question regarding a function called *crush_choose_firstn* in Ceph source code namely *mapper.c* This function has following pointer variables : - const struct crush_map *map, - struct crush_work *work, const struct crush_bucket *bucket, - int *out, - const __u32 *weight, - int *out2, - const struct crush_choose_arg *choose_args *- const struct crush_map *map.* What is the maximum size of data involved here? I mean what is the upper bound? BR Bobby

3 years, 10 months

1
0
0 0

IO500 Revised Call For Submissions Mid-2020 List

by committee＠io500.org

New Deadline: 13 July 2020 AoE NOTE: Given the short timeframe from the original announcement and complexities with some of the changes for this list, the deadline has been pushed out to give the community more time to participate. The BoF announcing the winners will be online 23 July 2020. Announcement: The IO500 [1] is now accepting and encouraging submissions for the upcoming 6th IO500 list. Once again, we are also accepting submissions to the 10 Node Challenge to encourage the submission of small scale results. The new ranked lists will be announced via live-stream at a virtual session. We hope to see many new results. The benchmark suite is designed to be easy to run and the community has multiple active support channels to help with any questions. Please note that submissions of all sizes are welcome; the site has customizable sorting so it is possible to submit on a small system and still get a very good per-client score for example. Additionally, the list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. More details below. Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 [1] was created in 2017, published its first list at SC17, and has grown exponentially since then. The need for such an initiative has long been known within High-Performance Computing; however, defining appropriate benchmarks had long been challenging. Despite this challenge, the community, after long and spirited discussion, finally reached consensus on a suite of benchmarks and a metric for resolving the scores into a single ranking. The multi-fold goals of the benchmark suite are as follows: * Maximizing simplicity in running the benchmark suite * Encouraging optimization and documentation of tuning parameters for performance * Allowing submitters to highlight their "hero run" performance numbers * Forcing submitters to simultaneously report performance for challenging IO patterns. Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes an IOR and mdtest run with highly constrained parameters forcing a difficult usage pattern in an attempt to determine a lower-bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Submitters are encouraged to share their tuning insights for publication. The goals of the community are also multi-fold: * Gather historical data for the sake of analysis and to aid predictions of storage futures * Collect tuning data to share valuable performance optimizations across the community * Encourage vendors and designers to optimize for workloads beyond "hero runs" * Establish bounded expectations for users, procurers, and administrators 10 NODE I/O CHALLENGE The 10 Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with, e.g., any number of servers. When submitting for the IO500 list, you can opt-in for "Participate in the 10 compute node challenge only", then we will not include the results into the ranked list. Other 10-node node submissions will be included in the full list and in the ranked list. We will announce the result in a separate derived list and in the full list but not on the ranked IO500 list at https://io500.org/. This information and rules for ISC20 submissions are available here: https://www.vi4io.org/io500/rules/submission Thanks, The IO500 Committee Links: ------ [1] http://io500.org/

3 years, 10 months

1
0
0 0

rbd-mirror with snapshot, not doing any actaul data sync

by Hans van den Bogert

Hi list, I've awaited octopus for a along time to be able to use mirror with snapshotting, since my setup does not allow for journal based mirroring. (K8s/Rook 1.3.x with ceph 15.2.2) However, I seem to be stuck, i've come to the point where on the cluster on which the (non-active) replicas should reside I get this: ``` rbd mirror pool status -p replicapool --verbose ... pvc-f7ca0b55-ed38-4d9f-b306-7db6a0157e2e: global_id: d3a301f2-4f54-4e9e-b251-c55ddbb67dc6 state: up+starting_replay description: starting replay service: a on nldw1-6-26-1 last_update: 2020-06-07 11:54:54 ... ``` That seems good, right? But I don't see any actual data being copied into the failover cluster. Anybody any ideas what to check? Also, is it correct, you won't see mirror snapshots with the 'normal' `rbd snap` commands? Thanks in advance, Hans

3 years, 10 months

2
3
0 0

poor cephFS performance on Nautilus 14.2.9 deployed by ceph_ansible

by Derrick Lin

Hi guys, We just deployed a CEPH 14.2.9 cluster with the following hardware: MDSS x 1 Xeon Gold 5122 3.6Ghz 192GB Mellanox ConnectX-4 Lx 25GbE MON x 3 Xeon Bronze 3103 1.7Ghz 48GB Mellanox ConnectX-4 Lx 25GbE 6 x 600GB 10K SAS OSD x 5 Xeon Silver 4110 2.1Ghz x 2 192GB Mellanox ConnectX-4 Lx 25GbE 16 x 10TB 7.2K NLSAS (block) 2 x 2TB Intel P4600 NVMe (block.db) Network is all Mellanox SN2410/SN2700 configured at 25GbE for both front and back network. Just for POC at this stage, the cluster was deployed by ceph_ansible without much customization and the initial test on its cephFS FUSE mount performance seems to be very low. We did some test with iozone the result as follow: ]# /opt/iozone/bin/iozone -i 0 -i 1-r 128k -s 5G -t 20 Iozone: Performance Test of File I/O Version $Revision: 3.465 $ Compiled for 64 bit mode. Build: linux-AMD64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, Vangel Bojaxhi, Ben England, Vikentsi Lapa, Alexey Skidanov. Run began: Tue Jun 2 16:40:53 2020 File size set to 5242880 kB Command line used: /opt/iozone/bin/iozone -i 0 -i 1-r -s 5G -t 20 128k Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Throughput test with 20 processes Each process writes a 5242880 kByte file in 4 kByte records Children see throughput for 20 initial writers = 35001.12 kB/sec Parent sees throughput for 20 initial writers = 34967.65 kB/sec Min throughput per process = 1748.22 kB/sec Max throughput per process = 1751.62 kB/sec Avg throughput per process = 1750.06 kB/sec Min xfer = 5232724.00 kB Children see throughput for 20 rewriters = 35704.79 kB/sec Parent sees throughput for 20 rewriters = 35704.30 kB/sec Min throughput per process = 1783.44 kB/sec Max throughput per process = 1786.29 kB/sec Avg throughput per process = 1785.24 kB/sec Min xfer = 5234532.00 kB Children see throughput for 20 readers = 49368539.50 kB/sec Parent sees throughput for 20 readers = 49317231.38 kB/sec Min throughput per process = 2414424.00 kB/sec Max throughput per process = 2599996.25 kB/sec Avg throughput per process = 2468426.98 kB/sec Min xfer = 4868708.00 kB Children see throughput for 20 re-readers = 48675891.50 kB/sec Parent sees throughput for 20 re-readers = 48617335.67 kB/sec Min throughput per process = 2316395.25 kB/sec Max throughput per process = 2703868.75 kB/sec Avg throughput per process = 2433794.58 kB/sec Min xfer = 4491704.00 kB We also did some dd tests, the write speed on a single test on our standard server is ~50MB/s but on a very big memory server, the speed is double ~ 80-90MB/s. We have zero experience on ceph and as said we haven't done more tuning at this stage. But if this sort of performance is way too low from those hardware spec? Any hints will be appreciated. Cheers D

3 years, 10 months

3
4
0 0

ceph mgr prometheus

by Frank R

Hi all, Does the ceph manager prometheus module export bluestore rocksdb compaction times per OSD? I couldn't find anything. thx Frank

3 years, 10 months

1
0
0 0

RDB Performance / High IOWaits.

by jameslipski＠protonmail.com

Greetings, I'm using ceph (14.2.2); in conjunction with Proxmox. Currently I'm just doing tests and ran into an issue relating to high I/O waits. Just to give a little bit of a background, specifically relating to my current ceph configurations; we have 6 nodes, each consisting of 2 osds (each node has 2x Intel SSDSC2KG019T8) OSD Type is bluestore. Global configurations (at least as shown on the proxmox interface) is as follows: [global] auth_client_required = xxxx auth_cluster_required = xxxx auth_service_required = xxxx cluster_network = 10.125.0.0/24 fsid = f64d2a67-98c3-4dbc-abfd-906ea7aaf314 mon_allow_pool_delete = true mon_host = 10.125.0.101 10.125.0.102 10.125.0.103 10.125.0.105 10.125.0.106 10.125.0.104 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 10.125.0.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring If I'm missing any relevant information relating to my ceph setup (I'm still learning this), please let me know. Each node consists of 2x Xeon E5-2660 v3. Where I ran into high I/O waits is when running a VM. VM is a mysql replication server (using 8 cores), and is performing mostly writes. When trying to narrow down, it was pointing to disk writes. The only thing I'm seeing in the ceph logs are the following: 2020-06-08 02:43:01.062082 mgr.node01 (mgr.2914449) 8009571 : cluster [DBG] pgmap v8009574: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 2.4 MiB/s wr, 274 op/s 2020-06-08 02:43:03.063137 mgr.node01 (mgr.2914449) 8009572 : cluster [DBG] pgmap v8009575: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 3.0 MiB/s wr, 380 op/s 2020-06-08 02:43:05.064125 mgr.node01 (mgr.2914449) 8009573 : cluster [DBG] pgmap v8009576: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 2.9 MiB/s wr, 332 op/s 2020-06-08 02:43:07.065373 mgr.node01 (mgr.2914449) 8009574 : cluster [DBG] pgmap v8009577: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 2.7 MiB/s wr, 313 op/s 2020-06-08 02:43:09.066210 mgr.node01 (mgr.2914449) 8009575 : cluster [DBG] pgmap v8009578: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 2.9 MiB/s wr, 350 op/s 2020-06-08 02:43:11.066913 mgr.node01 (mgr.2914449) 8009576 : cluster [DBG] pgmap v8009579: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.1 MiB/s wr, 346 op/s 2020-06-08 02:43:13.067926 mgr.node01 (mgr.2914449) 8009577 : cluster [DBG] pgmap v8009580: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.5 MiB/s wr, 408 op/s 2020-06-08 02:43:15.068834 mgr.node01 (mgr.2914449) 8009578 : cluster [DBG] pgmap v8009581: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.0 MiB/s wr, 320 op/s 2020-06-08 02:43:17.069627 mgr.node01 (mgr.2914449) 8009579 : cluster [DBG] pgmap v8009582: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 2.5 MiB/s wr, 285 op/s 2020-06-08 02:43:19.070507 mgr.node01 (mgr.2914449) 8009580 : cluster [DBG] pgmap v8009583: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 341 B/s rd, 3.0 MiB/s wr, 349 op/s 2020-06-08 02:43:21.071241 mgr.node01 (mgr.2914449) 8009581 : cluster [DBG] pgmap v8009584: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 0 B/s rd, 2.8 MiB/s wr, 319 op/s 2020-06-08 02:43:23.072286 mgr.node01 (mgr.2914449) 8009582 : cluster [DBG] pgmap v8009585: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 2.7 MiB/s wr, 329 op/s 2020-06-08 02:43:25.073369 mgr.node01 (mgr.2914449) 8009583 : cluster [DBG] pgmap v8009586: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 2.8 MiB/s wr, 304 op/s 2020-06-08 02:43:27.074315 mgr.node01 (mgr.2914449) 8009584 : cluster [DBG] pgmap v8009587: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 2.2 MiB/s wr, 262 op/s 2020-06-08 02:43:29.075284 mgr.node01 (mgr.2914449) 8009585 : cluster [DBG] pgmap v8009588: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 2.9 MiB/s wr, 342 op/s 2020-06-08 02:43:31.076180 mgr.node01 (mgr.2914449) 8009586 : cluster [DBG] pgmap v8009589: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 2.4 MiB/s wr, 269 op/s 2020-06-08 02:43:33.077523 mgr.node01 (mgr.2914449) 8009587 : cluster [DBG] pgmap v8009590: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.4 MiB/s wr, 389 op/s 2020-06-08 02:43:35.078543 mgr.node01 (mgr.2914449) 8009588 : cluster [DBG] pgmap v8009591: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.1 MiB/s wr, 344 op/s 2020-06-08 02:43:37.079428 mgr.node01 (mgr.2914449) 8009589 : cluster [DBG] pgmap v8009592: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.0 MiB/s wr, 334 op/s 2020-06-08 02:43:39.080419 mgr.node01 (mgr.2914449) 8009590 : cluster [DBG] pgmap v8009593: 512 pgs: 512 active+clean; 246 GiB data, 712 GiB used, 20 TiB / 21 TiB avail; 682 B/s rd, 3.3 MiB/s wr, 377 op/s I'm not sure what could be causing high I/O waits; or whether this is an issue relating to my ceph configurations. Any suggestions would be appreciated/ or if you need any additional information, let me know what you need and I'll post them. Thank you.

3 years, 10 months

1
0
0 0

Trying to upgrade to octopus removes current version of ceph release and tries to install older version...

by Drew Weaver

Hi cluster is version 14.2.9 ceph-deploy v2.0.1 using command ceph-deploy install --release octopus mon0 mon1 mon2 result is this command being run: sudo yum remove -y ceph-release which removes this package: ceph-release noarch 1-1.el7 @/ceph-release-1-0.el7.noarch Then it tries to install this older version: Running command: sudo yum install -y https://download.ceph.com/rpm-octopus/el7/noarch/ceph-release-1-0.el7.noarc… Which does not actually exist. Am I doing this wrong?

3 years, 10 months

1
1
0 0

Access RBD pools from clients

by Amjad Kotobi

Hi, I’m trying to access CEPH cluster from client machines via RBD for operations like listing pools, creating image, etc.. I put all needed keys/configuration in ceph directory and when I try to list pool rbd hung/wait forever Client machine in IP range of: 136.172.26.0/24 The command e.g. I’m trying is rbd -c /etc/ceph/ceph.conf -p vms ls -n client.cinder4 My CEPH cluster configuration file “ceph.conf” is ########## [global] fsid = e6d92380-4d77-4064-8797-2722fecfd1f4 mon initial members = ceph1,ceph2,ceph3 mon host = 10.60.1.21,10.60.1.22,10.60.1.23 public network = 10.60.1.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx mon_allow_pool_delete = true osd pool default size = 3 osd pool default min size = 2 [mgr.ceph1] mgr tick period = 10 mon mgr beacon grace = 60 [client.cinder4] keyring = /etc/ceph/ceph.client.cinder4.keyring ########## There is no firewall in between and accessing/telnet port is very much possible. I’m suspecting because of different IP subnet, MON CEPH is not allowing client. Any idea and help on this will be great. Thanks Amjad

3 years, 10 months

2
6
0 0

2024

2023

2022

2021

2020

2019

ceph-users June 2020