- ceph-users - lists.ceph.io

Accessing Ceph Storage Data via Ceph Block Storage
by Vaughan Beckwith 17 Nov '20

17 Nov '20

Hi All, I'm not sure if this is the correct place to ask this question, I have tried the channels, but have received very little help there. I am currently very new to Ceph and am investigating it as to a possible replacement for a legacy application which use to provide us with replication. At the moment my company has three servers, two primary servers running Ubuntu and a backup server also running Ubuntu, the two primary servers each host a virtual machine, and it is these virtual machines that the office workers use for shared folder access, email and as a domain server, the office workers are not aware of the underlying linux servers. In the past the legacy software would replicate the running VM files on both primary servers to the backup server. The replication is done at the underlying linux host level and not from within the guest VMs. I was hoping that I could get Ceph to do this as well. From what I have read and I speak under correction, the best Ceph client type for this would be the block access, whereby I would then mount the block and start up the VMs. As I would be running the VMs, as per normal routine, would Ceph then have to retrieve the large VM files from the storage nodes across the lan and bring the data back to the client to run in the VM. Is there an option to cache certain parts of the data on certain clients? Also none of the primary servers as they currently stand have the capacity to run both VMs together, so each primary has a dedicated VM which it runs, the backup server currently keeps replicated copies of both VM images from each primary, the replication is provided by the legacy application. I'm also wondering if I need to get a fourth server, so I have 2 clients and 2 storage nodes. Any suggestions or help would be greatly appreciated. Yours sincerely Vaughan Beckwith Bluesphere Technologies BSC I.T. (Honours) vaughan.beckwith(a)bluesphere.co.za<mailto:vaughan.beckwith@bluesphere.co.za> Telephone: 011 675 6354 Fax: (011) 675 6423

2 1

(Ceph Octopus) Repairing a neglected Ceph cluster - Degraded Data Reduncancy, all PGs degraded, undersized, not scrubbed in time
by seffyroff＠gmail.com 17 Nov '20

17 Nov '20

I've inherited a Ceph Octopus cluster that seems like it needs urgent maintenance before data loss begins to happen. I'm the guy with the most Ceph experience on hand and that's not saying much. I'm experiencing most of the ops and repair tasks for the first time here. Ceph health output looks like this: HEALTH_WARN Degraded data redundancy: 3640401/8801868 objects degraded (41.359%), 128 pgs degraded, 128 pgs undersized; 128 pgs not deep-scrubbed in time; 128 pgs not scrubbed in time Ceph -s output: https://termbin.com/i06u The crush rule 'cephfs.media' is here: https://termbin.com/2klmq So, it seems like all PGs are in a 'warning' state for the main pool, which is erasure coded and 11TiB across 4 OSDs, of which around 6.4TiB is used. The Ceph services themselves seem happy, they're stable and have Quorum. I'm able to access the web panel fine also. The block devices are of different sizes and types (2 large, different sized spinners, and 2 identical SSDs) I would welcome any pointers on what my steps to bring this up to full health may be. If it's undersized, can I simply add another block device/OSD? Or perhaps adjusting config somewhere will get it to rebalance successfully? (the rebalance jobs have been stuck at 0% for weeks) Thank you for your time reading this message.

8 19

Ceph RBD - High IOWait during the Writes
by athreyavc＠gmail.com 17 Nov '20

17 Nov '20

Hi, We have recently deployed a Ceph cluster with 12 OSD nodes(16 Core + 200GB RAM + 30 disks each of 14TB) Running CentOS 8 3 Monitoring Nodes (8 Core + 16GB RAM) Running CentOS 8 We are using Ceph Octopus and we are using RBD block devices. We have three Ceph client nodes(16core + 30GB RAM, Running CentOS 8) across which RBDs are mapped and mounted, 25 RBDs each on each client node. Each RBD size is 10TB. Each RBD is formatted as EXT4 file system. From network side, we have 10Gbps Active/Passive Bond on all the Ceph cluster nodes, including the clients. Jumbo frames enabled and MTU is 9000 This is a new cluster and cluster health reports Ok. But we see high IO wait during the writes. From one of the clients, 15:14:30 CPU %user %nice %system %iowait %steal %idle 15:14:31 all 0.06 0.00 1.00 45.03 0.00 53.91 15:14:32 all 0.06 0.00 0.94 41.28 0.00 57.72 15:14:33 all 0.06 0.00 1.25 45.78 0.00 52.91 15:14:34 all 0.00 0.00 1.06 40.07 0.00 58.86 15:14:35 all 0.19 0.00 1.38 41.04 0.00 57.39 Average: all 0.08 0.00 1.13 42.64 0.00 56.16 and the system load shows very high top - 15:19:15 up 34 days, 41 min, 2 users, load average: 13.49, 13.62, 13.83 From 'atop' one of the CPUs shows this CPU | sys 7% | user 1% | irq 2% | idle 1394% | wait 195% | steal 0% | guest 0% | ipc initial | cycl initial | curf 806MHz | curscal ?% On the OSD nodes, don't see much %utilization of the disks. RBD caching values are default. Are we overlooking some configuration item ? Thanks and Regards, At

4 5

Reclassify crush map
by Seena Fallah 17 Nov '20

17 Nov '20

Hi all, I want to reclassify my crushmap. I have two roots, one hiops and one default. In hiops root I have one datacenter and in that I have three rack and in each rack I have 3 osds. When I run the command below it says "item -55 in bucket -54 is not also a reclassified bucket". I see the new crushmap before reclassify-bucket command and item -55 was for my datacenter! What should I do? If I reclassify my datacenter I will lose my datacenter! crushtool -i crush-map --reclassify \ --reclassify-root default hdd \ --set-subtree-class default hdd \ --reclassify-root hiops ssd \ --reclassify-bucket hiops ssd default \ -o new-crush-map Thanks

1 1

Bucket notification is working strange
by Krasaev 17 Nov '20

17 Nov '20

Hi everyone, I asked the same question in stackoverflow, but will repeat here. I configured bucket notification using a bucket owner creds and when the owner does actions I can see new events in a configured endpoint(kafka actually). However, when I try to do actions in the bucket, but with another user creds I do not see events in the configured notification topic. Is it expected behavior and each user has to configure own topic(is it possible if a user is not system at all)? Or I have missed something? Thank you. https://stackoverflow.com/questions/64384060/enable-bucket-notifications-fo…

2 1

CephFS: Recovering from broken Mount
by Julian Fölsch 17 Nov '20

17 Nov '20

Hello, We are running a Octopus cluster however we still have some older Ubuntu 16.04 clients connecting using libcephfs2 version 14.2.13-1xenial. From time to time it happened that the network was having issues so the clients lost the connection to the cluster. But the system still thinks the mount is running and it has to be restarted manually. In some cases we even had to restart the complete machine because it would refuse to unmount. I also attached a dmesg log so you can see the systems behaivior. How do you deal with such issues? Kind regards, Julian Fölsch -- Julian Fölsch Arbeitsgemeinschaft Dresdner Studentennetz (AG DSN) Telefon: +49 351 271816 69 Mobil: +49 152 22915871 Fax: +49 351 46469685 Email: julian.foelsch(a)agdsn.de Studierendenrat der TU Dresden Helmholtzstr. 10 01069 Dresden

1 0

How to configure restful cert/key under nautilus
by Gary Molenkamp 16 Nov '20

16 Nov '20

I am attempting to configure the restful api for integration with zabbix agent2, but I am unable to configure the ssl cert and key based on the documentation here: https://docs.ceph.com/en/nautilus/mgr/restful/ #> ceph config-key set mgr/restful/controller02/crt -i ./mycert.crt WARNING: it looks like you might be trying to set a ceph-mgr module configuration key. Since Ceph 13.0.0 (Mimic), mgr module configuration is done with `config set`, and new values set using `config-key set` will be ignored. I found this issue tracker https://tracker.ceph.com/issues/40013 that also includes the alternate methods that I tried. While the tracker has be closed as resolved, its not clear to me what steps are needed to properly configure ssl on the restful api. Any assistance/clarification would be appreciated. Thanks. NOTE: ceph version 14.2.13

1 0

Re: Octopus OSDs dropping out of cluster: _check_auth_rotating possible clock skew, rotating keys expired way too early
by Lazuardi Nasution 16 Nov '20

16 Nov '20

Hi, I have the same situation with some OSD on Octopus 15.2.5 (Ubuntu 20,04). But, I have no problem with MGR. Any clue about this? Best regards, Date: Tue, 9 Jun 2020 23:47:24 +0200 > From: Wido den Hollander <wido(a)42on.com> > Subject: [ceph-users] Octopus OSDs dropping out of cluster: > _check_auth_rotating possible clock skew, rotating keys expired way > too early > To: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <be7aadc4-2142-ea31-caa8-28ca6db03d15(a)42on.com> > Content-Type: text/plain; charset=utf-8 > > Hi, > > On a recently deployed Octopus (15.2.2) cluster (240 OSDs) we are seeing > OSDs randomly drop out of the cluster. > > Usually it's 2 to 4 OSDs spread out over different nodes. Each node has > 16 OSDs and not all the failing OSDs are on the same node. > > The OSDs are marked as down and all they keep print in their logs: > > monclient: _check_auth_rotating possible clock skew, rotating keys > expired way too early (before 2020-06-04T07:57:17.706529-0400) > > Looking at their status through the admin socket: > > { > "cluster_fsid": "68653193-9b84-478d-bc39-1a811dd50836", > "osd_fsid": "87231b5d-ae5f-4901-93c5-18034381e5ec", > "whoami": 206, > "state": "active", > "oldest_map": 73697, > "newest_map": 75795, > "num_pgs": 19 > } > > The message brought me to my own ticket I created 2 years ago: > https://tracker.ceph.com/issues/23460 > > The first thing I've checked is NTP/time. Double, triple check this. All > the times are in sync on the cluster. Nothing wrong there. > > Again, it's not all the OSDs on a node failing. Just 1 or 2 dropping out. > > Restarting them brings them back right away and then within 24h some > other OSDs will drop out. > > Has anybody seen this behavior with Octopus as well? > > Wido >

1 0

OSD memory leak?
by Frank Schilder 16 Nov '20

16 Nov '20

Hi all, on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load. What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats: Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } } Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Am I looking at a memory leak here or are these heap stats expected? I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 25

Re: set rbd metadata 'conf_rbd_qos_bps_limit', make 'mkfs.xfs /dev/nbdX ' blocked
by Jason Dillaman 16 Nov '20

16 Nov '20

(CCing ceph-users since that is the appropriate forum). On Mon, Nov 16, 2020 at 4:44 AM liugangbiao(a)cmss.chinamobile.com < liugangbiao(a)cmss.chinamobile.com> wrote: > Hi dillaman， > > After I set rbd metadata 'conf_rbd_qos_bps_limit' as blow (600MB/s), the > image size is 1TB > > Then I use 'mkfs.xfs' to format this image, It's stuck for a long time > and never return. > > After unset 'conf_rbd_qos_bps_limit', 'mkfs.xfs' can return > successfully, > The *Ceph version is 14.2.10*, Any suggestion to reslove this porblem? > > Have you tried using the "-K" option to "mkfs.xfs" to disable discard at mkfs time? Perhaps it's issuing a discard region bigger than 600MiB? > Beset Regards， > LiuGangbiao > -- Jason

1 0