- ceph-users - lists.ceph.io

Ceph Trace System
by Liu, Changcheng 29 Mar '20

29 Mar '20

Hi all, Does anyone know how to trace the write/read request from client to rbd/rados/OSD? Is there any useful document besides the doc/dev/blkin? I'm wondering how to trace the large distributed storage system e.g. Ceph to observer/monitor it. B.R. Changcheng

1 0

Re: samba ceph-vfs and scrubbing interval
by Marco Savoca 28 Mar '20

28 Mar '20

Hi Jeff, i have digged a bit deeper and I was able to get write access on the ceph-vfs share. But the file and directory permissions are a mess. It seems that ceph-VfL does not evaluate the secondary group permissions. > Am 27.03.2020 um 18:14 schrieb Dr. Marco Savoca <quaternionma(a)gmail.com>: > > > „No. I haven't tested it in some time, but it does allow clients to > write. When you say you can't get write access, what are you doing to > test this, and what error are you getting back?“ > > For example: after successfull connection via smbclient it says „NT_STATUS_ACCESS_DENIED making remote Directory“ or > > „NT_STATUS_ACCESS_DENIED deleting remote file“ when I try to make a directory or to delete a file. If the same samba user connects to a share with a mounted path, everything works as expeted, so that there should not be some ACL Errors. > > > > > > Von: Jeff Layton > Gesendet: Freitag, 27. März 2020 13:05 > An: Marco Savoca; ceph-users(a)ceph.io > Betreff: Re: [ceph-users] samba ceph-vfs and scrubbing interval > > On Fri, 2020-03-27 at 12:00 +0100, Marco Savoca wrote: > > Hi all, > > > > i‘m running a 3 node ceph cluster setup with collocated mons and mds > > for actually 3 filesystems at home since mimic. I’m planning to > > downgrade to one FS and use RBD in the future, but this is another > > story. I’m using the cluster as cold storage on spindles with EC-pools > > for archive purposes. The cluster usually does not run 24/7. I > > actually managed to upgrade to octopus without problems yesterday. So > > first of all: great job with the release. > > > > Now I have a little problem and a general question to address. > > > > I have tried to share the CephFS via samba and the ceph-vfs module but > > I could not manage to get write access (read access is not a problem) > > to the share (even with the admin key). When I share the mounted path > > (kernel module or fuser mount) instead as usual there are no problems > > at all. Is ceph-vfs generally read only and I missed this point? > > No. I haven't tested it in some time, but it does allow clients to > write. When you say you can't get write access, what are you doing to > test this, and what error are you getting back? > > > Furthermore I suppose, that there is no possibility to choose between > > the different mds namespaces, right? > > > > Yeah, doesn't look like anyone has added that. That would probably be > pretty easy to add, though it would take a little while to trickle out > to the distros. > > > Now the general question. Since the cluster does not run 24/7 as > > stated and is turned on perhaps once a week for a couple of hours on > > demand, what are reasonable settings for the scrubbing intervals? As I > > said, the storage is cold and there is mostly read i/o. The archiving > > process adds approximately 0.5 % of new data of the cluster’s total > > storage capacity. > > -- > Jeff Layton <jlayton(a)redhat.com> > >

1 0

Leave of absence...
by Sage Weil 28 Mar '20

28 Mar '20

Hi everyone, I am taking time off from the Ceph project and from Red Hat, starting in April and extending through the US election in November. I will initially be working with an organization focused on voter registration and turnout and combating voter suppression and disinformation campaigns. During this time I will maintain some involvement in the Ceph community, primarily around strategic planning for Pacific and the Ceph Foundation, but most of my time will be focused elsewhere. Most decision making around Ceph will remain in the capable hands of the Ceph Leadership Team and component leads--I have the utmost confidence in their judgement and abilities. Yehuda Sadeh and Josh Durgin will be filling in to provide high-level guidance where needed. I’ll be participating in the Pacific planning meetings planned for next week, which will be important in kicking off development for Pacific: https://ceph.io/cds/ceph-developer-summit-pacific/ I am extremely proud of what we have accomplished with the Octopus release, and I believe the Ceph community will continue to do great things with Pacific! I look forward to returning at the end of the year to help wrap up the release and (hopefully) get things ready for Cephalocon next March. Most of all, I am excited to become engaged in another effort that I feel strongly about--one that will have a very real impact on my kids’ futures--and that will be easier to explain to lay people! :) Thanks! sage

2 1

Space leak in Bluestore
by vitalif＠yourcmc.ru 27 Mar '20

27 Mar '20

Hi. I'm experiencing some kind of a space leak in Bluestore. I use EC, compression and snapshots. First I thought that the leak was caused by "virtual clones" (issue #38184). However, then I got rid of most of the snapshots, but continued to experience the problem. I suspected something when I added a new disk to the cluster and free space in the cluster didn't increase (!). So to track down the issue I moved one PG (34.1a) using upmaps from osd11,6,0 to osd6,0,7 and then back to osd11,6,0. It ate +59 GB after the first move and +51 GB after the second. As I understand this proves that it's not #38184. Devirtualizaton of virtual clones couldn't eat additional space after SECOND rebalance of the same PG. The PG has ~39000 objects, it is EC 2+1 and the compression is enabled. Compression ratio is about ~2.7 in my setup, so the PG should use ~90 GB raw space. Before and after moving the PG I stopped osd0, mounted it with ceph-objectstore-tool with debug bluestore = 20/20 and opened the 34.1a***/all directory. It seems to dump all object extents into the log in that case. So now I have two logs with all allocated extents for osd0 (I hope all extents are there). I parsed both logs and added all compressed blob sizes together ("get_ref Blob ... 0x20000 -> 0x... compressed"). But they add up to ~39 GB before first rebalance (34.1as2), ~22 GB after it (34.1as1) and ~41 GB again after the second move (34.1as2) which doesn't indicate a leak. But the raw space usage still exceeds initial by a lot. So it's clear that there's a leak somewhere. What additional details can I provide for you to identify the bug? I posted the same message in the issue tracker, https://tracker.ceph.com/issues/44731 -- Vitaliy Filippov

5 9

Identify slow ops
by Thomas Schneider 27 Mar '20

27 Mar '20

Hi, the current output of ceph -s reports a warning: 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops This time is increasing. root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d) data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops There's no error on services (mgr, mon, osd). Can you please advise how to identify the root cause of this slow ops? THX

3 6

Help: corrupt pg
by Jake Grimmett 27 Mar '20

27 Mar '20

Dear All, We are "in a bit of a pickle"... No reply to my message (23/03/2020), subject "OSD: FAILED ceph_assert(clone_size.count(clone))" So I'm presuming it's not possible to recover the crashed OSD This is bad news, as one pg may be lost, (we are using EC 8+2, pg dump shows [NONE,NONE,NONE,388,125,25,427,226,77,154] ) Without this pg we have 1.8PB of broken cephfs. I could rebuild the cluster from scratch, but this means no user backups for a couple of weeks. The cluster has 10 nodes, uses an EC 8:2 pool for cephfs data (replicated NVMe metdata pool) and is running Nautilus 14.2.8 Clearly, it would be nicer if we could fix the OSD, but if this isn't possible, can someone confirm that the right procedure to recover from a corrupt pg is: 1) Stop all client access 2) find all files that store data on the bad pg, with: # cephfs-data-scan pg_files /backup 5.750 2> /dev/null > /root/bad_files 3) delete all of these bad files - presumably using truncate? or is "rm" fine? 4) destroy the bad pg # ceph osd force-create-pg 5.750 5) Copy the missing files back with rsync or similar... a better "recipe" or other advice gratefully received, best regards, Jake **** Note: I am working from home until further notice. For help, contact unixadmin(a)mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

3 4

fast luminous -> nautilus -> octopus upgrade could lead to assertion failure on OSD
by kefu chai 27 Mar '20

27 Mar '20

hi folks, if you are upgrading from luminous to octopus, or you plan to do so, please read on. in octopus, osd will crash if it processes an osdmap whose require_osd_release flag is still luminous. this only happens if a cluster upgrades very quickly from luminous to nautilus and to octopus. in this case, there are good chances that an octopus OSD will need to consume osdmaps which were created back in luminous. because we assumed that ceph did not ugprade across major releases, in octopus, OSD will panic at seeing osdmap from luminous. this is a known bug[0], and already fixed in master. and the next octopus release will include the fix to address this issue. as a workaround, you need to wait a while after running ceph osd require-osd-release nautilus and optionally inject lots of osdmaps into cluster to ensure that the old luminous osd maps are trimmed: for i in `seq 500`; do ceph osd blacklist add 192.168.0.1 ceph osd blacklist rm 192.168.0.1 done after the whole cluster are active+clean, then upgrade to octopus. happy upgrading! cheers, -- [0] https://tracker.ceph.com/issues/44759 -- Regards Kefu Chai

1 0

[ceph][nautilus] error initalizing secondary zone
by Ignazio Cassano 27 Mar '20

27 Mar '20

Hello , I am trying to initializing the secondary zone pulling the realm define in the primary zone: radosgw-admin realm pull --rgw-realm=nivola --url=http://10.102.184.190:8080 --access-key=access --secret=secret The following errors appears: request failed: (16) Device or resource busy Could you help me please ? Ignazio

1 1

Re: Combining erasure coding and replication?
by Eugen Block 27 Mar '20

27 Mar '20

Hi Brett, > Our concern with Ceph is the cost of having three replicas. Storage > may be cheap but I’d rather not buy ANOTHER 5pb for a third replica > if there are ways to do this more efficiently. Site-level redundancy > is important to us so we can’t simply create an erasure-coded volume > across two buildings – if we lose power to a building, the entire > array would become unavailable. can you elaborate on that? Why is EC not an option? We have installed several clusters with two datacenters resilient to losing a whole dc (and additional disks if required). So it's basically the choice of the right EC profile. Or did I misunderstand something? Zitat von Brett Randall <brett.randall(a)gmail.com>: > Hi all > > Had a fun time trying to join this list, hopefully you don’t get > this message 3 times! > > On to Ceph… We are looking at setting up our first ever Ceph cluster > to replace Gluster as our media asset storage and production system. > The Ceph cluster will have 5pb of usable storage. Whether we use it > as object-storage, or put CephFS in front of it, is still TBD. > > Obviously we’re keen to protect this data well. Our current Gluster > setup utilises RAID-6 on each of the nodes and then we have a single > replica of each brick. The Gluster bricks are split between > buildings so that the replica is guaranteed to be in another > premises. By doing it this way, we guarantee that we can have a > decent number of disk or node failures (even an entire building) > before we lose both connectivity and data. > > Our concern with Ceph is the cost of having three replicas. Storage > may be cheap but I’d rather not buy ANOTHER 5pb for a third replica > if there are ways to do this more efficiently. Site-level redundancy > is important to us so we can’t simply create an erasure-coded volume > across two buildings – if we lose power to a building, the entire > array would become unavailable. Likewise, we can’t simply have a > single replica – our fault tolerance would drop way down on what it > is right now. > > Is there a way to use both erasure coding AND replication at the > same time in Ceph to mimic the architecture we currently have in > Gluster? I know we COULD just create RAID6 volumes on each node and > use the entire volume as a single OSD, but that this is not the > recommended way to use Ceph. So is there some other way? > > Apologies if this is a nonsensical question, I’m still trying to > wrap my head around Ceph, CRUSH maps, placement rules, volume types, > etc etc! > > TIA > > Brett > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 3

Re: Combining erasure coding and replication?
by Lars Täuber 27 Mar '20

27 Mar '20

Hi Brett, I'm far from being an expert, but you may consider rbd-mirroring between EC-pools. Cheers, Lars Am Fri, 27 Mar 2020 06:28:02 +0000 schrieb Brett Randall <brett.randall(a)gmail.com>: > Hi all > > Had a fun time trying to join this list, hopefully you don’t get this message 3 times! > > On to Ceph… We are looking at setting up our first ever Ceph cluster to replace Gluster as our media asset storage and production system. The Ceph cluster will have 5pb of usable storage. Whether we use it as object-storage, or put CephFS in front of it, is still TBD. > > Obviously we’re keen to protect this data well. Our current Gluster setup utilises RAID-6 on each of the nodes and then we have a single replica of each brick. The Gluster bricks are split between buildings so that the replica is guaranteed to be in another premises. By doing it this way, we guarantee that we can have a decent number of disk or node failures (even an entire building) before we lose both connectivity and data. > > Our concern with Ceph is the cost of having three replicas. Storage may be cheap but I’d rather not buy ANOTHER 5pb for a third replica if there are ways to do this more efficiently. Site-level redundancy is important to us so we can’t simply create an erasure-coded volume across two buildings – if we lose power to a building, the entire array would become unavailable. Likewise, we can’t simply have a single replica – our fault tolerance would drop way down on what it is right now. > > Is there a way to use both erasure coding AND replication at the same time in Ceph to mimic the architecture we currently have in Gluster? I know we COULD just create RAID6 volumes on each node and use the entire volume as a single OSD, but that this is not the recommended way to use Ceph. So is there some other way? > > Apologies if this is a nonsensical question, I’m still trying to wrap my head around Ceph, CRUSH maps, placement rules, volume types, etc etc! > > TIA > > Brett > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de

1 0