- ceph-users - lists.ceph.io

multiple nvme per osd
by Frank R 22 Oct '19

22 Oct '19

Hi all, Has anyone successfully created multiple partitions on an NVME device using ceph-disk? If so, which commands were used?

4 3

Ubuntu Disco with most recent Kernel 5.0.0-32 seems to be instable
by Ranjan Ghosh 21 Oct '19

21 Oct '19

Hi all, it seems Ceph on Ubuntu Disco (19.04) with the most recent kernel 5.0.0-32 is instable. It crashes sometimes after a few hours, sometimes even after a few minutes. I found this bug here on CoreOS: https://github.com/coreos/bugs/issues/2616 Which is exactly also the error message I get ("cache_from_obj: Wrong slab cache. inode_cache but object is from ceph_inode_info") and the problem seems fairly recent. The problem vanished immediately after downgrading the kernel again. Could it be that this isn't exclusively a CoreOS issue? Where should I file this? Ubuntu Launchpad? Seems quite serious issue to me... Thank you / BR Ranjan

2 4

Occasionally ceph.dir.rctime is incorrect (14.2.4 nautilus)
by Toby Darling 21 Oct '19

21 Oct '19

Hi Occasionally, a directory's ceph.dir.rctime isn't updated when it is copied (rsync'd) to ceph, it reads the same as from stat, not the time it was created on ceph. The cluster is all Scientific Linux release 7.7 with ceph 14.2.4 nautilus (stable), the problem happened with kernel client running on one of the mon nodes. The tree that was rsync'd in was about 20TB, 340k files, 3000 directories. In all, I've found just 5 directories suffering this - possibly of interest, they are all empty (although there were 472 empty dirs in total). * dir1 is reported correctly: [root@ceph ~]# ls -ld path/to/dir1 drwxr-xr-x 1 thisuser thisuser 0 Jun 18 2018 path/to/dir1 [root@ceph ~]# getfattr --only-values -n ceph.dir.rctime path/to/dir1 1571035073.09398156844 [root@ceph ~]# stat path/to/dir1 File: ‘path/to/dir1’ Size: 0 Blocks: 0 IO Block: 65536 directory Device: 0h/0d Inode: 1099637142596 Links: 1 Access: (0755/drwxr-xr-x) Uid: (26089/thisuser) Gid: (26089/thisuser) Access: 2019-10-14 07:37:53.397156853 +0100 Modify: 2018-06-18 18:19:12.001114388 +0100 Change: 2019-10-14 07:37:53.398156844 +0100 Birth: - * But dir2's rctime is wrong: [root@ceph ~]# ls -ld path/to/dir1/dir2 drwxr-xr-x 1 thisuser thisuser 0 Jun 18 2018 path/to/dir1/dir2 [root@ceph ~]# getfattr --only-values -n ceph.dir.rctime path/to/dir1/dir2 1529342352.091114388 [root@ceph ~]# stat path/to/dir1/dir2 File: ‘path/to/dir1/dir2’ Size: 0 Blocks: 0 IO Block: 65536 directory Device: 0h/0d Inode: 1099637142960 Links: 1 Access: (0755/drwxr-xr-x) Uid: (26089/thisuser) Gid: (26089/thisuser) Access: 2019-10-14 07:37:53.398156844 +0100 Modify: 2018-06-18 18:19:12.001114388 +0100 Change: 2019-10-14 07:37:53.398156844 +0100 Birth: - Cheers Toby

1 1

RBD Mirror, Clone non primary Image
by yveskretzschmar＠web.de 21 Oct '19

21 Oct '19

Hi, I have a working RBD Mirror Setup using ceph version 14.2.4 on both sides. I want to have a clone of a non primay image. I do it this way: 1. create snapshot of primary image 2. wait for the snapshot to appear on the backup cluster 3. create a clone in backup cluster (using Simplified RBD Image Cloning - no snap protect) 4. delete the snapshot of the primary image - this is to save space because the clone will last a long time 5. if I do rbd --cluster backup snap ls --all IMAGENAME it presents the deleted snapshot in the namespace trash 6. delete the clone 7. I expected that deleting the clone would purge the snapshot from trash, but it doesn't 8. trash purge or trash remove is impossible (read only filesystem) Only possbility for me to purge the trash is to disable und reenable the image mirror. Is there something wrong in this order. Essential for me is the snaphot removal of the primary image. Flatten the clone is no option because of space. Regards, Yves

2 1

Dashboard doesn't respond after failover
by Matthew Stroud 21 Oct '19

21 Oct '19

For some reason the active MGR process just resets the connection after failover. Nothing really sticks out in the logs to explain this. However if I restart the MGR process, it will start responding just fine. Thoughts? Thanks, Matt ________________________________ CONFIDENTIALITY NOTICE: This message is intended only for the use and review of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message solely to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately by telephone or return email. Thank you.

2 2

Ceph BlueFS Superblock Lost
by Winger Cheng 21 Oct '19

21 Oct '19

Hello Everyone， My osd is broken recently, the first 8M size block have been clean. Since I use ceph-volume create bluestore with all wal, db, slow on one disk, I lost the superblock. Thanks to the lvm backup, I save the superblock of Bluestore, but I can't get the bluefs superblock back. What can I do to save the bluefs superblock?

1 0

rgw index large omap
by Frank R 21 Oct '19

21 Oct '19

I have an rgw index pool that is alerting as "large" in 2 of the 3 osds on the PG. The primary has a large omap. The index is definitely in use by the bucket. Any opinions on the best way to solve this? 1. Remove the 2 osds with large index from cluster and rebalance? 2. Delete 2 of the 3 and deep-scrub? 3. something else?

1 0

Ceph Tech Talk October 2019: Ceph at Nasa
by Mike Perez 21 Oct '19

21 Oct '19

Hi Cephers, I'm pleased to announce that we're starting Ceph Tech Talks back up (unfortunately probably the last one for this year due to holidays). On October 24th at 17:00 UTC Kevin Hrpcek will be presenting on Ceph at Nasa, and why they use librados instead of higher-level features. For information to join the live recorded stream, see our tech talk page: https://ceph.com/ceph-tech-talks/ If you're interested in giving your own tech talk, please let me know. -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

Re: RadosGW cant list objects when there are too many of them
by Paul Emmerich 21 Oct '19

21 Oct '19

On Mon, Oct 21, 2019 at 11:20 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > Yes listing v2 is not supported yet. I checked metadata osds and all of them are 600gb 10k hdd I dont think this was the issue. > I will test the --allow-unordered 5 million objects in a single bucket and metadata on HDD is a disaster waiting to happen if this continues to grow. rgw setups with large buckets need SSDs (or better NVMe) for metadata if you value availability. Recovering after a node failure will be horrible if you keep this on HDDs. Paul > > > Regards > ________________________________ > From: Paul Emmerich <paul.emmerich(a)croit.io> > Sent: Thursday, October 17, 2019 10:00 AM > To: Arash Shams <ara4sh(a)hotmail.com> > Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: Re: [ceph-users] RadosGW cant list objects when there are too many of them > > Listing large buckets is slow due to S3 ordering requirements, it's > approximately O(n^2). > However, I wouldn't consider 5M to be a large bucket, it should go to > only ~50 shards which should still perform reasonable. How fast are > your metadata OSDs? > > Try --allow-unordered in radosgw-admin to get an unordered result > which is only O(n) as you'd expect. > > For boto3: I'm not sure if v2 object listing is available yet (I think > it has only been merged into master but has not yet made it into a > release?). It doesn't support unordered listing but there has been > some work to implement it there, not sure about the current state. > > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Thu, Oct 17, 2019 at 9:19 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > > > > Dear All > > > > I have a bucket with 5 million Objects and I cant list objects with > > radosgw-admin bucket list --bucket=bucket | jq .[].name > > or listing files using boto3 > > > > s3 = boto3.client('s3', > > endpoint_url=credentials['endpoint_url'], > > aws_access_key_id=credentials['access_key'], > > aws_secret_access_key=credentials['secret_key']) > > > > response = s3.list_objects_v2(Bucket=bucket_name) > > for item in response['Contents']: > > print(item['Key']) > > > > what is the solution ? how can I find list of my objects ? > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Install error
by masud parvez 21 Oct '19

21 Oct '19

I am trying to install ceph in ubuntu 16.04 by this link https://www.supportsages.com/ceph-part-5-ceph-configuration-on-ubuntu/ but when I am run this command #ceph-deploy install ceph-deploy monnode1 osd0 osd1 I am facing this error. [ceph-deploy][WARNIN] E: Sub-process /usr/bin/dpkg returned an error code (1) [ceph-deploy][ERROR ] RuntimeError: command returned non-zero exit status: 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-osd ceph-mds ceph-mon radosgw $ sudo apt-get install ceph-common Reading package lists... Done Building dependency tree Reading state information... Done ceph-common is already the newest version (12.2.12-1xenial). ceph-common set to manually installed. 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 8 not fully installed or removed. After this operation, 0 B of additional disk space will be used. Do you want to continue? [Y/n] y Setting up ceph-common (12.2.12-1xenial) ... Setting system user ceph properties..usermod: user ceph is currently used by process 1648 dpkg: error processing package ceph-common (--configure): subprocess installed post-installation script returned error exit status 8 dpkg: dependency problems prevent configuration of ceph-base: ceph-base depends on ceph-common (= 12.2.12-1xenial); however: Package ceph-common is not configured yet. dpkg: error processing package ceph-base (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of ceph-mds: ceph-mds depends on ceph-base (= 12.2.12-1xenial); however: Package ceph-base is not configured yet. dpkg: error processing package ceph-mds (--configure): dependency problems - leaving unconfigured No apport report written because the error message indicates its a followup error from a previous failure. No apport report written because the error message indicates its a followup error from a previous failure. dpkg: dependency problems prevent configuration of ceph-osd: ceph-osd depends on ceph-base (= 12.2.12-1xenial); however: Package ceph-base is not configured yet. dpkg: error processing package ceph-osd (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of ceph-mon: ceph-mon depends on ceph-base (= 12.2.12-1xenial); however: Package ceph-base is not configured yet. dpkg: error processing package ceph-mon (--configure): dependency problems - leaving unconfigured No apport report written because MaxReports is reached already No apport report written because MaxReports is reached already dpkg: dependency problems prevent configuration of ceph-mgr: ceph-mgr depends on ceph-base (= 12.2.12-1xenial); however: Package ceph-base is not configured yet. dpkg: error processing package ceph-mgr (--configure): dependency problems - leaving unconfigured No apport report written because MaxReports is reached already dpkg: dependency problems prevent configuration of ceph: ceph depends on ceph-mgr (= 12.2.12-1xenial); however: Package ceph-mgr is not configured yet. ceph depends on ceph-mon (= 12.2.12-1xenial); however: Package ceph-mon is not configured yet. ceph depends on ceph-osd (= 12.2.12-1xenial); however: Package ceph-osd is not configured yet. dpkg: error processing package ceph (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of radosgw: radosgw depends on ceph-common (= 12.2.12-1xenial); however: Package ceph-common is not configured yet. dpkg: error processing package radosgw (--configure): dependency problems - leaving unconfigured No apport report written because MaxReports is reached already No apport report written because MaxReports is reached already Processing triggers for libc-bin (2.23-0ubuntu11) ... Errors were encountered while processing: ceph-common ceph-base ceph-mds ceph-osd ceph-mon ceph-mgr ceph radosgw E: Sub-process /usr/bin/dpkg returned an error code (1) [ceph-deploy][ERROR ] RuntimeError: command returned non-zero exit status: 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https I have wasted a lot of time with this error. Could anyone give me a solution?

2 1