December 2019 - ceph-users

by fleg＠lpnhe.in2p3.fr

Hi, We have a ceph+cephfs cluster runing nautilus version 14.2.4 We have debian buster/ubuntu bionic clients mounting cephfs in kernel mode without problems. We now want to mount cephfs from our new centos 8 clients. Unfortunately, ceph-common is needed but there are no packages available for el8 (only el7). And no way to install the el7 packages on centos 8 (missing deps). Thus, despite the fact that centos 8 have a 4.18 kernel (required to use quota, snapshots etc...), it seems impossible to mount in kernel mode (good perfs) and we still have to use the so slow fuse mode. Is it possible to workaround this problem ? Or when is it planned to provides (even as beta) the ceph packages for centos 8 ? Thanks.

4 years, 4 months

4
5
0 0

MGR log reports error related to Ceph Dashboard: SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:727)

by Thomas Schneider

Hello, I have enabled MGR module "Dashboard" and created a self-signed cert: root@ld5506:~# ceph dashboard create-self-signed-cert Self-signed certificate created Checking the MGR log I get multiple errors for any action in the dashboard with Ceph version 14.2.4.1 (596a387fb278758406deabf997735a1f706660c9) nautilus (stable): 2019-12-17 08:32:30.203 7f70942eb700 0 mgr[dashboard] [17/Dec/2019:08:32:30] ENGINE Error in HTTPServer.tick Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start self.tick() File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick s, ssl_env = self.ssl_adapter.wrap(s) File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap server_side=True) File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket _context=self) File "/usr/lib/python2.7/ssl.py", line 599, in __init__ self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake self._sslobj.do_handshake() SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:727) What is causing this error? How can I fix it? THX

4 years, 4 months

1
0
0 0

bluestore worries

by Frank R

Hi all, I am thinking about converting a Filestore cluster to Bluestore. The OSD nodes have 16X4TB 7200 SATA OSDs with NVME write journals. The NVME drives should be large enough to house ~30G DB/WAL OSDs. I am worried that I will see a significant performance hit when the deferred writes to the NVME journals are eliminated with Bluestore. Has anyone converted a similar setup to Bluestore? If so, what was the performance impact. thx Frank

4 years, 4 months

3
2
0 0

Re: ceph osd pool ls detail 'removed_snaps' on empty pool?

by Gregory Farnum

On Mon, Dec 16, 2019 at 11:34 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > Hi Gregory, > > I saw ceph -s showing 'snaptrim'(?), but I have still have these > 'removed_snaps' listed on this pool (also on other pools, I don't > remember creating/deleting them) A 'ceph tell mds.c scrub start /test/ > recursive repair' did not remove those. Can/should/how I remove these? > > > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] ceph osd pool ls detail 'removed_snaps' on empty > pool? > > > I have removed_snaps listed on pools that I am not using. They are > mostly for doing some performance testing, so I cannot imagine ever > creating snapshots in them. > > > pool 33 'fs_data.ssd' replicated size 3 min_size 1 crush_rule 5 > object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn > last_change 68413 lfor 0/0/65853 flags hashpspool,selfmanaged_snaps > stripe_width 0 application cephfs > removed_snaps > [567~1,56b~1,56f~1,57f~1,583~1,587~1,58b~1,58f~1,591~1,593~1,595~1,597~1 > ,599~1,59b~1,59d~1,59f~1,5a1~1,5a3~1,5a5~1,5a7~1, These are RADOS snapshots, so CephFS may have removed them but the OSDs still need to trim all the data; cephfs commands won't do anything to or with them. Once the OSD have removed the snapshot (it can take a while, depending on settings and other cluster activity) they'll report it back to the monitor, and it will remove them from the list of removed snaps. (It may not bother removing them if there aren't enough other live snapshots, I forget.) Either way, this list is basically harmless so you shouldn't worry about it. > > > [@ ]# rados df| egrep '^POOL|fs_data.ssd' > POOL_NAME USED OBJECTS CLONES COPIES > MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS > WR USED COMPR UNDER COMPR > fs_data.ssd 0 B 0 0 0 > > 0 0 0 0 0 B 0 0 B 0 > B 0 B > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

4 years, 4 months

1
0
0 0

How safe is k=2, m=1, min_size=2?

by Dan van der Ster

Hi all, We have said in the past that an EC pool should have min_size=k+1, for the same reasons that a replica 3 pool needs min_size=2. And we've heard several stories about replica 3, min_size=1 leading to incomplete PGs. Taking a quick poll -- did anyone ever suffer an outage on a pool with k=2, m=1, min_size=2? Thanks! Dan

4 years, 4 months

2
1
0 0

ceph osd pool ls detail 'removed_snaps' on empty pool?

by Marc Roos

I have removed_snaps listed on pools that I am not using. They are mostly for doing some performance testing, so I cannot imagine ever creating snapshots in them. pool 33 'fs_data.ssd' replicated size 3 min_size 1 crush_rule 5 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 68413 lfor 0/0/65853 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs removed_snaps [567~1,56b~1,56f~1,57f~1,583~1,587~1,58b~1,58f~1,591~1,593~1,595~1,597~1 ,599~1,59b~1,59d~1,59f~1,5a1~1,5a3~1,5a5~1,5a7~1, [@ ]# rados df| egrep '^POOL|fs_data.ssd' POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR fs_data.ssd 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B

4 years, 4 months

1
1
0 0

v13.2.7 mimic released

by Sage Weil

This is the seventh bugfix release of the Mimic v13.2.x long term stable release series. We recommend all Mimic users upgrade. For the full release notes, see https://ceph.io/releases/v13-2-7-mimic-released/ Notable Changes MDS: - Cache trimming is now throttled. Dropping the MDS cache via the “ceph tell mds.<foo> cache drop” command or large reductions in the cache size will no longer cause service unavailability. - Behavior with recalling caps has been significantly improved to not attempt recalling too many caps at once, leading to instability. MDS with a large cache (64GB+) should be more stable. - MDS now provides a config option “mds_max_caps_per_client” (default: 1M) to limit the number of caps a client session may hold. Long running client sessions with a large number of caps have been a source of instability in the MDS when all of these caps need to be processed during certain session events. It is recommended to not unnecessarily increase this value. - The “mds_recall_state_timeout” config parameter has been removed. Late client recall warnings are now generated based on the number of caps the MDS has recalled which have not been released. The new config parameters “mds_recall_warning_threshold” (default: 32K) and “mds_recall_warning_decay_rate” (default: 60s) set the threshold for this warning. - The “cache drop” admin socket command has been removed. The “ceph tell mds.X cache drop” remains. OSD: - A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option “mon_warn_on_slow_ping_ratio” specifies a percentage of “osd_heartbeat_grace” to determine the threshold. A value of zero disables the warning. A new configuration option “mon_warn_on_slow_ping_time”, specified in milliseconds, overrides the computed value, causing a warning when OSD heartbeat pings take longer than the specified amount. A new admin command “ceph daemon mgr.# dump_osd_network [threshold]” lists all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. A new admin command ceph daemon osd.# dump_osd_network [threshold]” does the same but only including heartbeats initiated by the specified OSD. - The default value of the “osd_deep_scrub_large_omap_object_key_threshold” parameter has been lowered to detect an object with large number of omap keys more easily. RGW: - radosgw-admin introduces two subcommands that allow the managing of expire-stale objects that might be left behind after a bucket reshard in earlier versions of RGW. One subcommand lists such objects and the other deletes them. Read the troubleshooting section of the dynamic resharding docs for details.

4 years, 4 months

4
7
0 0

any way to read magic number like #1018a1b3c14?

by Sang, Oliver

we are using cephfs. in /sys/kernel/debug/ceph/*/mdsc sometimes we saw something like - 2015909 mds0 setfilelock #1018a1b3c14 I know this #1018a1b3c14 is a file or a folder, but how to know what it really is in cephfs tree? Thanks!

4 years, 4 months

1
0
0 0

Re: ceph-volume sizing osds

by Jan Fajerski

On Fri, Dec 13, 2019 at 11:49:50PM +0100, Oscar Segarra wrote: > Hi, > I have recently started working with Ceph Nautilus release and I have > realized that you have to start working with LVM to create OSD instead > of the "old fashioned" ceph-disk. > In terms of performance and best practices, as far as I must use LVM I > can create volume groups that joins or extends two or more physical > disks. In this scenario (many disks for server) where ceph-volume is > manatory, It still remains the rule of one OSD for each physical > device? or I can reduce the number of OSDs? That is hard to answer generically. ceph-volume will not create multi-disk volumes. One disk, one OSD is still the recommendation afaik. However there are certainly scenarios where multi-disk OSDs are sensible. For example if you have a machine that has _a lot_ of disks attached but lacks CPU and RAM to run as many OSDs as disks, it can make sense to deploy one OSD on multiple disks. > Thanks in advance, > Óscar >_______________________________________________ >ceph-users mailing list >ceph-users(a)lists.ceph.com >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

4 years, 4 months

1
0
0 0

Re: Can't create new OSD

by Gregory Farnum

[ Re-adding users list ] On Thu, Dec 12, 2019 at 10:12 AM Rodrigo Severo - Fábrica <rodrigo(a)fabricadeideias.com> wrote: > > Em qui., 12 de dez. de 2019 às 12:55, Gregory Farnum > <gfarnum(a)redhat.com> escreveu: > > > > On Wed, Dec 11, 2019 at 12:04 PM Rodrigo Severo - Fábrica <rodrigo(a)fabricadeideias.com> wrote: > >> > >> Hi, > >> > >> > >> Trying to create a new OSD following the instructions available at > >> https://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/ > >> > >> On step 3 I'm instructed to run "ceph-osd -i {osd-num} --mkfs > >> --mkkey". Unfortunately it doesn't work: > >> > >> # ceph-osd -i 3 --mkfs --mkkey > >> 2019-12-11 16:59:58.257 7fac4597fc00 -1 auth: unable to find a keyring > >> on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > >> 2019-12-11 16:59:58.257 7fac4597fc00 -1 AuthRegistry(0x55ad976ea140) > >> no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > >> 2019-12-11 16:59:58.261 7fac4597fc00 -1 auth: unable to find a keyring > >> on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > >> 2019-12-11 16:59:58.261 7fac4597fc00 -1 AuthRegistry(0x7fffac4075e8) > >> no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > >> failed to fetch mon config (--no-mon-config to skip) > > I'm full of questions. > > > This is the important bit. Since you’re running the command without a way to access the Ceph monitors, > > Should I provide a way do access the Ceph monitors? I was just > following the docs and there is no mention to it in the above page. > > Anyway I tried to do it but the results were exactly the same: > > # ceph-osd -m 192.168.109.233:3300 -i 3 --mkfs --mkkey > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 AuthRegistry(0x564545484140) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 AuthRegistry(0x7ffccdef0958) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > failed to fetch mon config (--no-mon-config to skip) > > I also tried in the 6789 port and without providing port at all. > Always the same results. Tried also on the other monitors. Always the > same results. > > > it can’t use the cluster default configuration. You need to move the client.admin keyring onto the node (or another one, if you configure the ceph.conf or pass in the name on each command invocation). I haven't been through the docs in a while so maybe it's missing or lost but it should describe that somewhere. This is necessary whenever you are making changes to the cluster, such as adding an OSD — without a keyring, the cluster has no idea if the user making the change is authorized! > > Shouldn't it automatically read /etc/ceph/ceph.conf and figure everything out? > > I also tried explicitly setting the conf path but, again, same results: It's reading the ceph.conf but still needs cluster access, which you haven't given it. > > # ceph-osd -i 3 --mkfs --mkkey -c /etc/ceph/ceph.conf > 2019-12-12 14:57:35.740 7fee29b61c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:57:35.740 7fee29b61c00 -1 AuthRegistry(0x56457c784140) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > 2019-12-12 14:57:35.740 7fee29b61c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:57:35.740 7fee29b61c00 -1 AuthRegistry(0x7ffc6eccb0b8) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > failed to fetch mon config (--no-mon-config to skip) > > > You can pass the given option to not worry about that, > > Tried that. Apparently it worked, despite several disturbing messages: > > # ceph-osd -i 3 --mkfs --mkkey --no-mon-config > 2019-12-12 14:57:50.256 7f31d77b2c00 -1 auth: error reading file: > /var/lib/ceph/osd/ceph-3/keyring: can't open > /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 created new key in keyring > /var/lib/ceph/osd/ceph-3/keyring > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory > 2019-12-12 14:57:50.629 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid > # ll /var/lib/ceph/osd/ceph-3 > total 172 > drwxr-xr-x 2 root root 4096 Dec 12 14:57 ./ > drwxr-xr-x 4 ceph ceph 4096 Dec 11 16:19 ../ > -rw-r----- 1 root root 10737418240 Dec 12 14:57 block > -rw------- 1 root root 2 Dec 12 14:57 bluefs > -rw------- 1 root root 37 Dec 12 14:57 ceph_fsid > -rw-r----- 1 root root 37 Dec 12 14:57 fsid > -rw------- 1 root root 56 Dec 12 14:57 keyring > -rw------- 1 root root 8 Dec 12 14:57 kv_backend > -rw------- 1 root root 21 Dec 12 14:57 magic > -rw------- 1 root root 4 Dec 12 14:57 mkfs_done > -rw------- 1 root root 6 Dec 12 14:57 ready > -rw------- 1 root root 10 Dec 12 14:57 type > -rw------- 1 root root 2 Dec 12 14:57 whoami > > Is it how it should be? With all these "failed to open" messages? Yeah, those are BlueStore et al attempting various opens that are allowed to fail. I agree they're not great to have show up as standard. > > > or you can run the command with a user key ring that’s allowed to read the monitor configs. (You’ll need that key access next to add the OSD’s key to the cluster, anyway!) > > Haven't tried that. How is it done? I can't find any option to provide > a user key ring in ceph-osd's man page. It's not ceph-osd specific, just anywhere in ceph. -n client.[name], -k [keyring_file] etc > > > Many thanks for your help. > > Rodrigo >

4 years, 4 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users December 2019