- ceph-users - lists.ceph.io

Combining erasure coding and replication?
by Brett Randall 27 Mar '20

27 Mar '20

Hi all Had a fun time trying to join this list, hopefully you don’t get this message 3 times! On to Ceph… We are looking at setting up our first ever Ceph cluster to replace Gluster as our media asset storage and production system. The Ceph cluster will have 5pb of usable storage. Whether we use it as object-storage, or put CephFS in front of it, is still TBD. Obviously we’re keen to protect this data well. Our current Gluster setup utilises RAID-6 on each of the nodes and then we have a single replica of each brick. The Gluster bricks are split between buildings so that the replica is guaranteed to be in another premises. By doing it this way, we guarantee that we can have a decent number of disk or node failures (even an entire building) before we lose both connectivity and data. Our concern with Ceph is the cost of having three replicas. Storage may be cheap but I’d rather not buy ANOTHER 5pb for a third replica if there are ways to do this more efficiently. Site-level redundancy is important to us so we can’t simply create an erasure-coded volume across two buildings – if we lose power to a building, the entire array would become unavailable. Likewise, we can’t simply have a single replica – our fault tolerance would drop way down on what it is right now. Is there a way to use both erasure coding AND replication at the same time in Ceph to mimic the architecture we currently have in Gluster? I know we COULD just create RAID6 volumes on each node and use the entire volume as a single OSD, but that this is not the recommended way to use Ceph. So is there some other way? Apologies if this is a nonsensical question, I’m still trying to wrap my head around Ceph, CRUSH maps, placement rules, volume types, etc etc! TIA Brett

1 0

Combining erasure coding and replication?
by Brett Randall 27 Mar '20

27 Mar '20

Hi all We are looking at setting up our first ever Ceph cluster to replace Gluster as our media asset storage and production system. The Ceph cluster will have 5pb of usable storage. Whether we use it as object-storage, or put CephFS in front of it, is still TBD. Obviously we’re keen to protect this data well. Our current Gluster setup utilises RAID-6 on each of the nodes and then we have a single replica of each brick. The Gluster bricks are split between buildings so that the replica is guaranteed to be in another premises. By doing it this way, we guarantee that we can have a decent number of disk or node failures (even an entire building) before we lose both connectivity and data. Our concern with Ceph is the cost of having three replicas. Storage may be cheap but I’d rather not buy ANOTHER 5pb for a third replica if there are ways to do this more efficiently. Site-level redundancy is important to us so we can’t simply create an erasure-coded volume across two buildings – if we lose power to a building, the entire array would become unavailable. Likewise, we can’t simply have a single replica – our fault tolerance would drop way down on what it is right now. Is there a way to use both erasure coding AND replication at the same time in Ceph to mimic the architecture we currently have in Gluster? I know we COULD just create RAID6 volumes on each node and use the entire volume as a single OSD, but that this is not the recommended way to use Ceph. So is there some other way? Apologies if this is a nonsensical question, I’m still trying to wrap my head around Ceph, CRUSH maps, placement rules, volume types, etc etc! TIA Brett

1 0

Ceph rbd mirror and object storage multisite
by Ignazio Cassano 26 Mar '20

26 Mar '20

Hello All, I am going to test rbd mirroring and object storage multisite. I would like to know which network is used by rbd mirror ( ceph public or cluster network ?) Same question for object storage multisite.... What about firewall ? What about bandwidth ? Our sites are connected with 1Gbs network ? How many volumes can I mirror in this scenario? Could anyone help me ? Best Regards Ignazio

1 0

Performance characteristics of ‘if-none-match’ on rgw
by akmd＠metaswitch.com 26 Mar '20

26 Mar '20

Hello, I am observing non-intuitive results for a performance test using the S3 API to RGW. I am wondering if others have similar experiences or knowledge here. Our application is using the “if-none-match” header on S3-API requests. This header is set by the application if it already has a copy of the object in question but wishes to check if there is a newer version. If the etag of the current object matches then RGW sends a 304 response, and if it doesn’t it sends the updated content of the object. We’re observing that the response time of requests resulting in “304 Not Modified” is typically slower than those for normal object retrieval. This wasn’t intuitive to me – in the 304 case there is no content to transfer over the network and I would expect the request can be satisfied just by looking at the RGW index (I was under the impression that metadata including etag is in the index). Anecdotally, HEAD requests see similar results but I haven't yet analysed in full. Does anyone else have data or experience about expected performance of this scenario? Are there any potential avenues for optimization of configuration ? What kind of commands can I use to debug this further ? Some details of the current setup: => ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) => Objects are typically 80-100KB. => Versioning is enabled on the bucket. => Our requests specify a Range header (hence will generate 206 not 200). => Multisite features are enabled. => Bucket has 20 shards – I’ve put a dump of "bucket limits" below. Performance results Response, Request Count, Median, 75th percentile, 90th percentile, 95th percentile, 206 Partial, 20473, 3, 3, 16, 129, 1200 304 Not Modified, 15644, 9, 16, 46, 212, 1192 Bucket details { "bucket": "albansstack-scsdata", "tenant": "", "num_objects": 465780, "num_shards": 20, "objects_per_shard": 23289, "fill_status": "OK" }, Many thanks, Alistair.

1 0

octopus upgrade stuck: Assertion `map->require_osd_release >= ceph_release_t::mimic' failed.
by Ml Ml 26 Mar '20

26 Mar '20

Hello List, i followed: https://ceph.io/releases/v15-2-0-octopus-released/ I came from a healthy nautilus and i am stuck at: 5.) Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD host When i try to start an osd like this, i get: /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph ... 2020-03-25T20:11:03.292+0100 7f2762874e00 -1 osd.32 57223 log_to_monitors {default=true} ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion `map->require_osd_release >= ceph_release_t::mimic' failed. ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion `map->require_osd_release >= ceph_release_t::mimic' failed. *** Caught signal (Aborted) ** in thread 7f274854f700 thread_name:tp_osd_tp Aborted My current status: root@ceph03:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 60.70999 root default -2 20.25140 host ceph01 0 hdd 1.71089 osd.0 up 1.00000 1.00000 8 hdd 2.67029 osd.8 up 1.00000 1.00000 11 hdd 1.59999 osd.11 up 1.00000 1.00000 12 hdd 1.59999 osd.12 up 1.00000 1.00000 14 hdd 2.79999 osd.14 up 1.00000 1.00000 18 hdd 1.59999 osd.18 up 1.00000 1.00000 22 hdd 2.79999 osd.22 up 1.00000 1.00000 23 hdd 2.79999 osd.23 up 1.00000 1.00000 26 hdd 2.67029 osd.26 up 1.00000 1.00000 -3 23.05193 host ceph02 2 hdd 2.67029 osd.2 up 1.00000 1.00000 3 hdd 2.00000 osd.3 up 1.00000 1.00000 7 hdd 2.67029 osd.7 up 1.00000 1.00000 9 hdd 2.67029 osd.9 up 1.00000 1.00000 13 hdd 2.00000 osd.13 up 1.00000 1.00000 16 hdd 1.59999 osd.16 up 1.00000 1.00000 19 hdd 2.38409 osd.19 up 1.00000 1.00000 24 hdd 2.67020 osd.24 up 1.00000 1.00000 25 hdd 1.71649 osd.25 up 1.00000 1.00000 28 hdd 2.67029 osd.28 up 1.00000 1.00000 -4 17.40666 host ceph03 5 hdd 1.71660 osd.5 down 1.00000 1.00000 6 hdd 1.71660 osd.6 down 1.00000 1.00000 10 hdd 2.67029 osd.10 down 1.00000 1.00000 15 hdd 2.00000 osd.15 down 1.00000 1.00000 17 hdd 1.20000 osd.17 down 1.00000 1.00000 20 hdd 1.71649 osd.20 down 1.00000 1.00000 21 hdd 2.00000 osd.21 down 1.00000 1.00000 27 hdd 1.71649 osd.27 down 1.00000 1.00000 32 hdd 2.67020 osd.32 down 1.00000 1.00000 root@ceph03:~# ceph osd dump | grep require_osd_release require_osd_release nautilus root@ceph03:~# ceph osd versions { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19 } oot@ceph03:~# ceph mon dump | grep min_mon_release dumped monmap epoch 12 min_mon_release 15 (octopus) ceph versions { "mon": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "mgr": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "osd": { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19 }, "mds": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "overall": { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19, "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 9 } } Why does it complain about map->require_osd_release >= ceph_release_t::mimic ? Cheers, Michael

2 2

14.2.7 MDS High Virtual Memory
by Dave Hall 26 Mar '20

26 Mar '20

Hello. I have a cluster with 3 MDSs - 2 active and 1 standby. Looking at 'top' I see that the 2 active MDSs have relatively low resident memory - less than 3 g, but very high virtual memory - 21 g and 41 g. Is it safe to just restart these processes one at a time to release the virtual memory? Also, is there something that needs to be tweaked to prevent the MDSs from accumulating so much memory? Thanks. -Dave -- Dave Hall Binghamton University

2 1

Re: Exporting
by Eugen Block 26 Mar '20

26 Mar '20

Hi, have you tried to create a snapshot of that image and export the snapshot? Or maybe clone the snapshot first and then export it? Does it also fail? Zitat von Rhian Resnick <rresnick(a)fau.edu>: > Evening, > > We are running into issues exporting a disk image from ceph rbd. > When we attempt to export an rbd image in a cache tiered > erasure-coded pool on Luminus. > > All the other disks are working fine but this one is acting up. We > have a bit of important data on other disks so obviously want to > make sure this doesn't happen to those. > > [root@ceph-p-mon1 home]# rbd export one/one-177-588-0 one-177-588-0 > Exporting image: 8% complete...rbd: error reading from source image > at offset 5456789504: (5) Input/output error > 2020-03-23 20:11:29.210718 7f2f3effd700 -1 > librbd::io::ObjectRequest: 0x7f2f2c128f90 handle_read_object: failed > to read from object: (5) Input/output error > 2020-03-23 20:11:29.565184 7f2f3e7fc700 -1 > librbd::io::ObjectRequest: 0x7f2f280c84d0 handle_read_cache: failed > to read from cache: (5) Input/output error > Exporting image: 8% complete...failed. > rbd: export error: (5) Input/output error > > > Any thoughts would be appreciated. > > > Some info: > > [root@ceph-p-mon1 home]# rbd info one/one-177-588-0 > rbd image 'one-177-588-0': > size 58.6GiB in 15000 objects > order 22 (4MiB objects) > block_name_prefix: rbd_data.84a01279e2a9e3 > format: 2 > features: layering, exclusive-lock, object-map, fast-diff, deep-flatten > flags: > create_timestamp: Fri Apr 20 17:06:09 2018 > parent: one/one-177@snap > overlap: 2.20GiB > > [root@ceph-p-mon1 home]# ceph status > cluster: > id: 6a2e8f21-bca2-492b-8869-eecc995216cc > health: HEALTH_OK > > services: > mon: 3 daemons, quorum ceph-p-mon2,ceph-p-mon1,ceph-p-mon3 > mgr: ceph-p-mon2(active) > mds: cephfsec-1/1/1 up {0=ceph-p-mon2=up:active}, 6 up:standby > osd: 155 osds: 154 up, 154 in > > data: > pools: 6 pools, 5904 pgs > objects: 145.53M objects, 192TiB > usage: 253TiB used, 290TiB / 543TiB avail > pgs: 5896 active+clean > 8 active+clean+scrubbing+deep > > io: > client: 921KiB/s rd, 5.68MiB/s wr, 110op/s rd, 29op/s wr > cache: 5.65MiB/s flush, 0op/s promote > > > [root@ceph-p-mon1 home]# rpm -qa | grep ceph > ceph-common-12.2.9-0.el7.x86_64 > ceph-mds-12.2.9-0.el7.x86_64 > ceph-radosgw-12.2.9-0.el7.x86_64 > ceph-mgr-12.2.9-0.el7.x86_64 > ceph-12.2.9-0.el7.x86_64 > collectd-ceph-5.8.1-1.el7.x86_64 > ceph-deploy-2.0.1-0.noarch > libcephfs2-12.2.9-0.el7.x86_64 > python-cephfs-12.2.9-0.el7.x86_64 > ceph-selinux-12.2.9-0.el7.x86_64 > ceph-osd-12.2.9-0.el7.x86_64 > ceph-base-12.2.9-0.el7.x86_64 > ceph-mon-12.2.9-0.el7.x86_64 > ceph-release-1-1.el7.noarch > > > Rhian Resnick > > Associate Director Research Computing > > Enterprise Systems > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road, CM22, Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: OSDs wont mount on Debian 10 (Buster) with Nautilus
by Ml Ml 25 Mar '20

25 Mar '20

i brought them up manually. Decided to upgrade to octopus, but i am stuck there now. I will open a new thread for it. On Wed, Mar 25, 2020 at 6:23 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > Still down? > > > -----Original Message----- > Cc: ceph-users > Subject: [ceph-users] Re: OSDs wont mount on Debian 10 (Buster) with > Nautilus > > What does the osd error log say? I already have bluestore, if you have > file store maybe you should inspect the mounted fs of /dev/sdd2 eg. > maybe there permissions need to be changed. But first check the errors > of one osd. > > ( You did you reset the failed service with somethig like this systemctl > reset-failed ceph-osd@0 not? ) > > > -----Original Message----- > Cc: ceph-users > Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > Nautilus > > Still no luck. > > But the working OSDs have no partition: > OSD.1 => /dev/sdj > OSD.5 => /dev/sdb > OSD.6 => /dev/sdbc > OSD.10 => /dev/sdl > > > Where as the rest has: > > root@ceph03:~# ls -l /dev/sd* > brw-rw---- 1 root disk 8, 0 Mar 25 16:23 /dev/sda > brw-rw---- 1 root disk 8, 1 Mar 25 16:23 /dev/sda1 > brw-rw---- 1 root disk 8, 2 Mar 25 16:23 /dev/sda2 > brw-rw---- 1 root disk 8, 3 Mar 25 16:23 /dev/sda3 > brw-rw---- 1 root disk 8, 16 Mar 25 16:23 /dev/sdb > brw-rw---- 1 root disk 8, 32 Mar 25 16:23 /dev/sdc > brw-rw---- 1 root disk 8, 48 Mar 25 16:23 /dev/sdd > brw-rw---- 1 root disk 8, 49 Mar 25 16:23 /dev/sdd1 > brw-rw---- 1 ceph ceph 8, 50 Mar 25 16:23 /dev/sdd2 > brw-rw---- 1 root disk 8, 64 Mar 25 16:23 /dev/sde > brw-rw---- 1 root disk 8, 65 Mar 25 16:23 /dev/sde1 > brw-rw---- 1 ceph ceph 8, 66 Mar 25 16:23 /dev/sde2 > brw-rw---- 1 root disk 8, 80 Mar 25 16:23 /dev/sdf > brw-rw---- 1 root disk 8, 81 Mar 25 16:23 /dev/sdf1 > brw-rw---- 1 ceph ceph 8, 82 Mar 25 16:23 /dev/sdf2 > brw-rw---- 1 root disk 8, 96 Mar 25 16:23 /dev/sdg > brw-rw---- 1 root disk 8, 97 Mar 25 16:23 /dev/sdg1 > brw-rw---- 1 ceph ceph 8, 98 Mar 25 16:23 /dev/sdg2 > brw-rw---- 1 root disk 8, 112 Mar 25 16:23 /dev/sdh > brw-rw---- 1 root disk 8, 113 Mar 25 16:23 /dev/sdh1 > brw-rw---- 1 ceph ceph 8, 114 Mar 25 16:23 /dev/sdh2 > brw-rw---- 1 root disk 8, 128 Mar 25 16:23 /dev/sdi > brw-rw---- 1 root disk 8, 129 Mar 25 16:23 /dev/sdi1 > brw-rw---- 1 ceph ceph 8, 130 Mar 25 16:23 /dev/sdi2 > brw-rw---- 1 root disk 8, 144 Mar 25 16:23 /dev/sdj > brw-rw---- 1 root disk 8, 160 Mar 25 16:23 /dev/sdk > brw-rw---- 1 root disk 8, 161 Mar 25 16:23 /dev/sdk1 > brw-rw---- 1 ceph ceph 8, 162 Mar 25 16:23 /dev/sdk2 > brw-rw---- 1 root disk 8, 176 Mar 25 16:23 /dev/sdl > brw-rw---- 1 root disk 8, 192 Mar 25 16:23 /dev/sdm > brw-rw---- 1 root disk 8, 193 Mar 25 16:23 /dev/sdm1 > brw-rw---- 1 ceph ceph 8, 194 Mar 25 16:23 /dev/sdm2 > > Did i miss to convert to bluestore or something? > > > On Wed, Mar 25, 2020 at 4:23 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> > wrote: > > > > Try this > > > > chown ceph.ceph /dev/sdc2 > > chown ceph.ceph /dev/sdd2 > > chown ceph.ceph /dev/sde2 > > chown ceph.ceph /dev/sdf2 > > chown ceph.ceph /dev/sdg2 > > chown ceph.ceph /dev/sdh2 > > > > > > > > -----Original Message----- > > From: Ml Ml [mailto:mliebherr99@googlemail.com] > > Sent: 25 March 2020 16:22 > > To: Marc Roos > > Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > > Nautilus > > > > They where indeed disabled. I enabled them with: > > > > systemctl enable ceph-osd@4 > > systemctl enable ceph-osd@15 > > systemctl enable ceph-osd@17 > > systemctl enable ceph-osd@20 > > systemctl enable ceph-osd@21 > > systemctl enable ceph-osd@27 > > systemctl enable ceph-osd@32 > > > > But they still wont start. > > > > On Wed, Mar 25, 2020 at 4:09 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> > > wrote: > > > > > > > > > I had something similar. My osd were disabled, maybe this installer > > > of > > > > > nautilus does that check > > > > > > systemctl is-enabled ceph-osd@0 > > > > > > https://tracker.ceph.com/issues/44102 > > > >

2 1

Luminous upgrade question
by Shain Miley 25 Mar '20

25 Mar '20

Hi, We are thinking about upgrading our cluster currently running ceph version 12.2.12. I am wondering if we should be looking at upgrading to the latest version of Mimic or the latest version Nautilus. Can anyone here please provide a suggestion…I continue to be a little bit confused about the correct Ceph upgrade path given all the various versions that are out there currently. Thank you in advance. Shain

3 2

Re: OSDs wont mount on Debian 10 (Buster) with Nautilus
by Ml Ml 25 Mar '20

25 Mar '20

Still no luck. But the working OSDs have no partition: OSD.1 => /dev/sdj OSD.5 => /dev/sdb OSD.6 => /dev/sdbc OSD.10 => /dev/sdl Where as the rest has: root@ceph03:~# ls -l /dev/sd* brw-rw---- 1 root disk 8, 0 Mar 25 16:23 /dev/sda brw-rw---- 1 root disk 8, 1 Mar 25 16:23 /dev/sda1 brw-rw---- 1 root disk 8, 2 Mar 25 16:23 /dev/sda2 brw-rw---- 1 root disk 8, 3 Mar 25 16:23 /dev/sda3 brw-rw---- 1 root disk 8, 16 Mar 25 16:23 /dev/sdb brw-rw---- 1 root disk 8, 32 Mar 25 16:23 /dev/sdc brw-rw---- 1 root disk 8, 48 Mar 25 16:23 /dev/sdd brw-rw---- 1 root disk 8, 49 Mar 25 16:23 /dev/sdd1 brw-rw---- 1 ceph ceph 8, 50 Mar 25 16:23 /dev/sdd2 brw-rw---- 1 root disk 8, 64 Mar 25 16:23 /dev/sde brw-rw---- 1 root disk 8, 65 Mar 25 16:23 /dev/sde1 brw-rw---- 1 ceph ceph 8, 66 Mar 25 16:23 /dev/sde2 brw-rw---- 1 root disk 8, 80 Mar 25 16:23 /dev/sdf brw-rw---- 1 root disk 8, 81 Mar 25 16:23 /dev/sdf1 brw-rw---- 1 ceph ceph 8, 82 Mar 25 16:23 /dev/sdf2 brw-rw---- 1 root disk 8, 96 Mar 25 16:23 /dev/sdg brw-rw---- 1 root disk 8, 97 Mar 25 16:23 /dev/sdg1 brw-rw---- 1 ceph ceph 8, 98 Mar 25 16:23 /dev/sdg2 brw-rw---- 1 root disk 8, 112 Mar 25 16:23 /dev/sdh brw-rw---- 1 root disk 8, 113 Mar 25 16:23 /dev/sdh1 brw-rw---- 1 ceph ceph 8, 114 Mar 25 16:23 /dev/sdh2 brw-rw---- 1 root disk 8, 128 Mar 25 16:23 /dev/sdi brw-rw---- 1 root disk 8, 129 Mar 25 16:23 /dev/sdi1 brw-rw---- 1 ceph ceph 8, 130 Mar 25 16:23 /dev/sdi2 brw-rw---- 1 root disk 8, 144 Mar 25 16:23 /dev/sdj brw-rw---- 1 root disk 8, 160 Mar 25 16:23 /dev/sdk brw-rw---- 1 root disk 8, 161 Mar 25 16:23 /dev/sdk1 brw-rw---- 1 ceph ceph 8, 162 Mar 25 16:23 /dev/sdk2 brw-rw---- 1 root disk 8, 176 Mar 25 16:23 /dev/sdl brw-rw---- 1 root disk 8, 192 Mar 25 16:23 /dev/sdm brw-rw---- 1 root disk 8, 193 Mar 25 16:23 /dev/sdm1 brw-rw---- 1 ceph ceph 8, 194 Mar 25 16:23 /dev/sdm2 Did i miss to convert to bluestore or something? On Wed, Mar 25, 2020 at 4:23 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > Try this > > chown ceph.ceph /dev/sdc2 > chown ceph.ceph /dev/sdd2 > chown ceph.ceph /dev/sde2 > chown ceph.ceph /dev/sdf2 > chown ceph.ceph /dev/sdg2 > chown ceph.ceph /dev/sdh2 > > > > -----Original Message----- > From: Ml Ml [mailto:mliebherr99@googlemail.com] > Sent: 25 March 2020 16:22 > To: Marc Roos > Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > Nautilus > > They where indeed disabled. I enabled them with: > > systemctl enable ceph-osd@4 > systemctl enable ceph-osd@15 > systemctl enable ceph-osd@17 > systemctl enable ceph-osd@20 > systemctl enable ceph-osd@21 > systemctl enable ceph-osd@27 > systemctl enable ceph-osd@32 > > But they still wont start. > > On Wed, Mar 25, 2020 at 4:09 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> > wrote: > > > > > > I had something similar. My osd were disabled, maybe this installer of > > > nautilus does that check > > > > systemctl is-enabled ceph-osd@0 > > > > https://tracker.ceph.com/issues/44102 > > > > > > > > > > -----Original Message----- > > From: Ml Ml [mailto:mliebherr99@googlemail.com] > > Sent: 25 March 2020 16:05 > > To: ceph-users > > Subject: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > > Nautilus > > > > Hello list, > > > > i upgraded to Debian 10, after that i upgraded from luminous to > > nautilus. > > I restarted the mons, then the OSDs. > > > > Everything was up and healthy. > > After rebooting a node, only 3/10 OSD start up: > > > > -4 20.07686 host ceph03 > > 4 hdd 2.67020 osd.4 down 1.00000 1.00000 > > 5 hdd 1.71660 osd.5 up 1.00000 1.00000 > > 6 hdd 1.71660 osd.6 up 1.00000 1.00000 > > 10 hdd 2.67029 osd.10 up 1.00000 1.00000 > > 15 hdd 2.00000 osd.15 down 1.00000 1.00000 > > 17 hdd 1.20000 osd.17 down 1.00000 1.00000 > > 20 hdd 1.71649 osd.20 down 1.00000 1.00000 > > 21 hdd 2.00000 osd.21 down 1.00000 1.00000 > > 27 hdd 1.71649 osd.27 down 1.00000 1.00000 > > 32 hdd 2.67020 osd.32 down 1.00000 1.00000 > > > > root@ceph03:~# /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser > > ceph --setgroup ceph > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > > > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x56531c50a140) > > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > > > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x7ffd04120468) > > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > > > failed to fetch mon config (--no-mon-config to skip) > > > > root@ceph03:~# df > > Filesystem 1K-blocks Used Available Use% Mounted on > > udev 24624580 0 24624580 0% /dev > > tmpfs 4928216 9544 4918672 1% /run > > /dev/sda3 47930248 5209760 40262684 12% / > > tmpfs 24641068 0 24641068 0% /dev/shm > > tmpfs 5120 0 5120 0% /run/lock > > tmpfs 24641068 0 24641068 0% /sys/fs/cgroup > > /dev/sda1 944120 144752 734192 17% /boot > > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-1 > > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-6 > > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-5 > > tmpfs 24641068 24 24641044 1% > > /var/lib/ceph/osd/ceph-10 > > tmpfs 4928212 0 4928212 0% /run/user/0 > > > > root@ceph03:~# ceph-volume lvm list > > > > > > ====== osd.1 ======= > > > > [block] > > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > > -4 > > 7bd-8ddc-102b98444067 > > > > block device > > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > > -4 > > 7bd-8ddc-102b98444067 > > block uuid HSK6Da-elP2-CFYz-s0RH-UNiw-bey0-dVcml1 > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid b4987093-4fa5-47bd-8ddc-102b98444067 > > osd id 1 > > type block > > vdo 0 > > devices /dev/sdj > > > > ====== osd.10 ====== > > > > [block] > > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > > -4 > > b85-9799-988eee3b2b3f > > > > block device > > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > > -4 > > b85-9799-988eee3b2b3f > > block uuid 440fNG-guO2-l1WJ-m5cR-GUkz-ZTUd-Fcz5Ml > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid fa241441-1758-4b85-9799-988eee3b2b3f > > osd id 10 > > type block > > vdo 0 > > devices /dev/sdl > > > > ====== osd.5 ======= > > > > [block] > > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > > -4 > > e50-9bb5-775bacd854af > > > > block device > > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > > -4 > > e50-9bb5-775bacd854af > > block uuid Z6VeNx-S9sg-ZOsh-HTw9-ykTc-YBrh-qFwz5i > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid 112e0c75-f61b-4e50-9bb5-775bacd854af > > osd id 5 > > type block > > vdo 0 > > devices /dev/sdb > > > > ====== osd.6 ======= > > > > [block] > > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > > -4 > > 2ab-aeb0-f2915a389e48 > > > > block device > > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > > -4 > > 2ab-aeb0-f2915a389e48 > > block uuid G9qHxC-dN0b-XBes-QVss-Bzwa-7Xtw-ikksgM > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid 35288829-c1f6-42ab-aeb0-f2915a389e48 > > osd id 6 > > type block > > vdo 0 > > devices /dev/sdc > > > > I would mount it and run the osd deamon manually, but ceph-disk list > > seems to be gone in Nautilus. Therefore i dont know where to mount > what. > > > > Any ideas on that? > > > > > > Cheers, > > Michael > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > > > > >

2 2