- ceph-users - lists.ceph.io

mgr+Prometheus/grafana (+consul)
by Jeremy Austin 20 May '21

20 May '21

I recently configured Prometheus to scrape mgr /metrics and add Grafana dashboards. All daemons at 15.2.11 I use Hashicorp consul to advertise the active mgr in DNS, and Prometheus points at a single DNS target. (Is anyone else using this method, or just statically pointing Prometheus at all potentially active managers?) All was working fine initially, and it's *mostly* still working fine. For the first couple of days, all went well, and then a few rate metrics stopped meaningfully increasing — essentially pegged at zero, which is implausible in a healthy cluster. Some cluster maintenance was occurring such as outing and recreating some OSDs, so I have a baseline for throughput and recovery. Metric graphs that stopped functioning: Throughput: ceph_osd_op_r_out_bytes, ceph_osd_op_w_in_bytes, ceph_osd_op_rw_in_bytes Recovery: ceph_osd_recovery_ops I can see that Grafana output is using this method of converting the counters to rates: sum(irate(ceph_osd_recovery_ops{job="$job"}[$interval])) The underlying counters appear to be sane, and reading the raw values from prometheus is also valid, so I'm guessing some failure either of the irate or sum functions? By inspection in Grafana, the queries return correct timestamps with zero values, so that leaves us with "sum(irate)" as the likely source of the problem. Does anyone have experience with this? I admit it is possibly tangential to ceph itself, but as the Prometheus/grafana integration is more or less supported, I thought I'd try here first. -- Jeremy Austin jhaustin(a)gmail.com

1 0

fsck error: found stray omap data on omap_head
by Pickett, Neale T 20 May '21

20 May '21

We just upgraded to pacific, and I'm trying to clear warnings about legacy bluestore omap usage stats by running 'ceph-bluestore-tool repair`, as instructed by the warning message. It's been going fine, but we are now getting this error: [root@vanilla bin]# ceph-bluestore-tool repair --path $osd_path 2021-05-19T19:25:26.485+0000 7f67ca3593c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) fsck error: found stray omap data on omap_head 12256434 0 0 repair status: remaining 1 error(s) and warning(s) [root@vanilla bin]# ceph-bluestore-tool fsck --path $osd_path -deep 2021-05-19T20:03:17.002+0000 7f4d1d6603c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) fsck error: found stray omap data on omap_head 12256434 0 0 fsck status: remaining 1 error(s) and warning(s) We're only 10% of the way through our OSDs, so I'd like to find some way to fix this other than destroying and rebuilding the OSD, in case it happens again. Fixing this error is especially attractive since we can't get out of HEALTH_WARN until we've run recover on all OSDs. Any suggestions? Neale Pickett <neale(a)lanl.gov> A-4: Advanced Research in Cyber Systems Los Alamos National Laboratory

2 2

"radosgw-admin bucket radoslist" loops when a multipart upload is happening
by Boris Behrens 20 May '21

20 May '21

Hi together, I still search for orphan objects and came across a strange bug: There is a huge multipart upload happening (around 4TB), and listing the rados objects in the bucket loops over the multipart upload. -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 2

Bucket index OMAP keys unevenly distributed among shards
by James, GleSYS 20 May '21

20 May '21

Hi, we're running 15.2.7 and our cluster is warning us about LARGE_OMAP_OBJECTS (1 large omap objects). Here is what the distribution looks like for the bucket in question, and as you can see all but 3 of the keys reside in shard 2. .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.0 1 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.8 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.9 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.7 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.1 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.4 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.3 1 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.2 262384 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.6 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.5 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.12 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.10 1 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.11 0 osd_deep_scrub_large_omap_object_key_threshold is set to 200000 by default, hence the warning observed for this bucket. Dynamic resharding is enabled, and the bucket is not in the process of being resharded. Versioning not in use for this bucket, so we're not affected by https://tracker.ceph.com/issues/46456. Can anyone help us understand why all the keys are getting mapped to a singe shard? Is there a bug here, or is this expected behaviour? Could it be related to the fact that the bucket contains large multipart uploads? (Object names look like this:) _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5900 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5901 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5902 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5903 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5904 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5905 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5906 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5907 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5908 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5909 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7152 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7153 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7154 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7155 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7156 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7157 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7158 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7159

1 0

ceph-ansible in Pacific and beyond?
by Matthew Vernon 20 May '21

20 May '21

Hi, I caught up with Sage's talk on what to expect in Pacific ( https://www.youtube.com/watch?v=PVtn53MbxTc ) and there was no mention of ceph-ansible at all. Is it going to continue to be supported? We use it (and uncontainerised packages) for all our clusters, so I'd be a bit alarmed if it was going to go away... Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

17 25

ceph orch status hangs forever
by Sebastian Luna Valero 20 May '21

20 May '21

Hi, After an unschedule power outage our Ceph (Octopus) cluster reports a healthy state with: "ceph status". However, when we run "ceph orch status" the command hangs forever. Are there other commands that we can run for a more thorough health check of the cluster? After looking at: https://docs.ceph.com/en/octopus/rados/operations/health-checks/ I also run "ceph crash ls-new" but it hangs forever as well. Any ideas? Our Ceph cluster is currently used as backend storage for our OpenStack cluster, and we are also having issues with storage volumes attached to VMs, but we don't know how to narrow down the root cause. Any feedback is highly appreciated. Best regards, Sebastian

2 3

Suitable 10G Switches for ceph storage - any recommendations?
by Hermann Himmelbauer 20 May '21

20 May '21

Dear Ceph users, I am currently constructing a small hyperconverged Proxmox cluster with ceph as storage. So far I always had 3 nodes, which I directly linked together via 2 bonded 10G network interfaces for the Ceph storage, so I never needed any switching devices. This new cluster has more nodes, so I am considering using a 10G switch for the storage network. As I have no experience with such a setup, I wonder if there are any specific issues that I should think of (latency...)? As the whole cluster should be not too expensive, I am currently thinking of the following solution: 2* CRS317-1G-16s+RM switches: https://mikrotik.com/product/crs317_1g_16s_rm#fndtn-testresults SFP+ Cables like these: https://www.fs.com/de/products/48883.html Some network interface for each node with two SFP+ ports, e.g.: https://ark.intel.com/content/www/de/de/ark/products/39776/intel-ethernet-c… Connect each port with each switch and configure master/slave configuration so that the switches are redundant. What do you think of this setup - or is there any information / recommendation for an optimized setup of a 10G storage network? Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

2 1

iSCSI - failed, gateway(s) unavailable UNKNOWN
by Paul Giralt (pgiralt) 19 May '21

19 May '21

I’m new to Ceph and have just deployed my first cluster using ceph-ansible and running into some issues I was hoping someone could point me in the right direction. I have 5 servers to start with. 5 OSD, 3 Monitors, 2 Manager, 4 iSCSI gateway. My intention is to use this environment as iSCSI storage for ESXi. When I went to add an iSCSI Target, I got an error that popped up in the browser (went away to fast to read it) but after this, now if I click on the “Targets” tab in Dashboard for iSCSI, I just continuously get errors popping up: 500 - Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request. I’m unable to add any targets now. The log files show this: 2021-05-19T12:58:05.294-0400 7f2db6115700 0 [dashboard ERROR exception] Internal Server Error Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 46, in dashboard_exception_handler return handler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 694, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 907, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 266, in list IscsiTarget._set_info(target) File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 990, in _set_info raise e File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 980, in _set_info target_iqn) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 531, in func_wrapper **kwargs) File "/usr/share/ceph/mgr/dashboard/services/iscsi_client.py", line 254, in get_targetinfo return request() File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 326, in __call__ data, raw_content, headers) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 449, in do_request resp.content) dashboard.rest_client.RequestException: iscsi REST API failed request with status code 503 (b'{\n "message": "failed, gateway(s) unavailable:cxcto-c240-j27-01(UNKNOWN' b' state)"\n}\n') 2021-05-19T12:58:05.294-0400 7f2db6115700 0 [dashboard ERROR request] [10.117.244.166:58270] [GET] [500] [0.398s] [admin] [513.0B] /api/iscsi/target 2021-05-19T12:58:05.294-0400 7f2db6115700 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "7bc691e8-e814-48c1-a69c-57ea1f3f3dbd"} ‘] Clearly it’s showing an error indicating that the gateway is “unavailable”. The output from gwcli shows this: [root@cxcto-c240-j27-01 ~]# ./gwcli 1 gateway is inaccessible - updates will be disabled /> ls o- / ......................................................................................................................... [...] o- cluster ......................................................................................................... [Clusters: 1] | o- ceph ............................................................................................................ [HEALTH_OK] | o- pools .......................................................................................................... [Pools: 4] | | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/15911599M (0%), Used: 0.00Y] | | o- iscsi ............................................................... [(x3), Commit: 0.00Y/15911599M (0%), Used: 262511b] | | o- rbd ................................................................... [(x3), Commit: 0.00Y/15911599M (0%), Used: 1650b] | | o- test .................................................................. [(x3), Commit: 0.00Y/15911599M (0%), Used: 0.00Y] | o- topology .............................................................................................. [OSDs: 110,MONs: 3] o- disks ....................................................................................................... [0.00Y, Disks: 0] o- iscsi-targets ............................................................................... [DiscoveryAuth: CHAP, Targets: 1] o- iqn.2001-07.com.ceph:1621437640904 ................................................................ [Auth: None, Gateways: 1] o- disks .......................................................................................................... [Disks: 0] o- gateways ............................................................................................ [Up: 0/1, Portals: 1] | o- cxcto-c240-j27-01 ....................................................................... [10.122.242.196 (UNAUTHORIZED)] o- host-groups .................................................................................................. [Groups : 0] o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 0] In this case it says “UNAUTHORIZED” When I created the target, I had selected all 4 iSCSI gateways, but it looks like something happened during the addition process that has left it in a weird state. The dashboard seems to think that all the gateways are up and running : [cid:8CA421EB-DFD1-43A9-87B6-7ADEE5B2DF69] Notice that it only shows the target on the one node which it is reporting an error. I’ve tried deleting the target from gwcli but that fails and I’m not really sure where to look next. The “UNAUTHORIZED” in the gwcli output makes me wonder if there is some kind of authorization issue, but I’m not sure what that would be. This is what I see if I navigate to the Targets tab on the iSCSI page: [cid:828103A0-C321-41F6-B018-AC8A883F6C7E] Any thoughts or guidance are greatly appreciated. -Paul

1 0

Re: Suitable 10G Switches for ceph storage - any recommendations?
by Max Vernimmen 19 May '21

19 May '21

Hermann, I think there was a discussion on recommended switches not too long ago. You should be able to find it in the mailing list archives. I think the latency of the network is usually very minor compared to ceph's dependency on cpu and disk latency, so for a simple cluster I wouldn't worry about it too much. I have found fs.com's dac cables to get stuck a lot, so I don't use them anymore. I usually buy dell or mellanox cables. Regarding network cards I've found the intel cards to be not that great due to bugs with lacp bonds, embedded lldp getting in the way and other issues. So I'm using mellanox cards instead, but broadcom should also work. hope it helps! best regards, Max On Wed, May 19, 2021 at 1:48 PM <ceph-users-request(a)ceph.io> wrote: > ---------- Forwarded message ---------- > From: Hermann Himmelbauer <hermann(a)qwer.tk> > To: ceph-users(a)ceph.com > Cc: > Bcc: > Date: Wed, 19 May 2021 11:22:26 +0200 > Subject: [ceph-users] Suitable 10G Switches for ceph storage - any > recommendations? > Dear Ceph users, > I am currently constructing a small hyperconverged Proxmox cluster with > ceph as storage. So far I always had 3 nodes, which I directly linked > together via 2 bonded 10G network interfaces for the Ceph storage, so I > never needed any switching devices. > > This new cluster has more nodes, so I am considering using a 10G switch > for the storage network. As I have no experience with such a setup, I > wonder if there are any specific issues that I should think of > (latency...)? > > As the whole cluster should be not too expensive, I am currently > thinking of the following solution: > > 2* CRS317-1G-16s+RM switches: > https://mikrotik.com/product/crs317_1g_16s_rm#fndtn-testresults > > SFP+ Cables like these: > https://www.fs.com/de/products/48883.html > > Some network interface for each node with two SFP+ ports, e.g.: > > https://ark.intel.com/content/www/de/de/ark/products/39776/intel-ethernet-c… > > Connect each port with each switch and configure master/slave > configuration so that the switches are redundant. > > What do you think of this setup - or is there any information / > recommendation for an optimized setup of a 10G storage network? > > Best Regards, > Hermann > > -- > hermann(a)qwer.tk > PGP/GPG: 299893C7 (on keyservers) > >

2 1

rbd-nbd crashes Error: failed to read nbd request header: (33) Numerical argument out of domain
by Zhi Zhang 19 May '21

19 May '21

Hi guys, We are recently testing rbd-nbd using ceph N version. After map rbd image, mkfs and mount the nbd device, the rbd-nbd and dmesg will show following errors when doing some read/write testing. rbd-nbd log: 2021-05-18 11:35:08.034 7efdb8ff9700 20 []rbd-nbd: reader_entry: waiting for nbd request ... 2021-05-18 11:35:08.066 7efdb8ff9700 -1 []rbd-nbd: failed to read nbd request header: (33) Numerical argument out of domain 2021-05-18 11:35:08.066 7efdb3fff700 20 []rbd-nbd: writer_entry: no io requests, terminating 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: 0x564a2be2b3c0 unregister_update_watcher: handle=0 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: handle=0 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: completing unregister 2021-05-18 11:35:08.066 7efdea8d1a00 10 []rbd-nbd: ~NBDServer: terminating 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: 0x564a2be2b3c0 close dmesg: [Tue May 18 11:35:07 2021] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: discard [Tue May 18 11:35:07 2021] block nbd0: shutting down sockets [Tue May 18 11:35:09 2021] blk_update_request: I/O error, dev nbd0, sector 75592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0 client host info: centos7.x kernel 5.4.109 It looks like the kernel nbd device shutdown its socket for some reason, but we haven't figured it out. BTW, we have tried to turn on/off rbd cache, use different fs ext4/xfs, use ec pool or replicated pool, but the error remains. It is more frequent for us to reproduce when batch map, mkfs and mount rbd-nbd on different hosts simultaneously. Thanks for any suggestions. Regards, Zhi Zhang (David) Contact: zhang.david2011(a)gmail.com zhangz.david(a)outlook.com

2 4