- ceph-users - lists.ceph.io

Cephfs corruption(?) causing nfs-ganesha to "clients failing to respond to capability release" / "MDSs report slow requests"
by Marc Roos 26 Sep '19

26 Sep '19

Can it be that there is something wrong with the cephfs, that causes the nfs-ganesha to fail? I have the impression that when I do an rsync on only a specific nfs mount/cephfs tree. I am getting this error. All nfs mounts are inaccessible on that client. Other clients (with other mounts) are fine.

1 0

Re: Balancer active plan
by Massimo Sgaravatto 26 Sep '19

26 Sep '19

I think you should see this info in the mgr log (with debug mgr = 4/5) Cheers, Massimo On Thu, Sep 26, 2019 at 3:43 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > Suddenly I have objects misplaced, I assume this is because of the > balancer being active. But > > 1. how can I see the currently/last executed plan of the balancer? > 2. when it was activated? > > > 7 active+remapped+backfill_wait > 7 active+remapped+backfilling > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. > F1 Outsourcing Development Sp. z o.o. > Poland > > t: +48 (0)124466845 > f: +48 (0)124466843 > e: marc(a)f1-outsourcing.eu > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Balancer active plan
by Marc Roos 26 Sep '19

26 Sep '19

Suddenly I have objects misplaced, I assume this is because of the balancer being active. But 1. how can I see the currently/last executed plan of the balancer? 2. when it was activated? 7 active+remapped+backfill_wait 7 active+remapped+backfilling - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. F1 Outsourcing Development Sp. z o.o. Poland t: +48 (0)124466845 f: +48 (0)124466843 e: marc(a)f1-outsourcing.eu

1 0

CephFS deleted files' space not reclaimed
by Josh Haft 26 Sep '19

26 Sep '19

Hi, I've been migrating data from one EC pool to another EC pool: two directories are mounted with ceph.dir.layout.pool file attribute set appropriately, then rsync from old to new and finally, delete the old files. I'm using the kernel client to do this. While the removed files are no longer present on the filesystem, they still appear to be accounted for via "ceph df". When I tally up the sizes reported by "ls -lh" on all subdirectories under the root CephFS using a FUSE client mount (except for those on the new EC pool), it totals just under 2PiB. However, "ceph df" shows the original EC pool as 2.5PiB used. I've copied + deleted approximately 545TiB so far, so it seems like the unlinked files aren't being fully released/purged. I've only observed the num_strays counter from "ceph daemon mds.$name perf dump" for a few days now since I first suspected an issue, but I've never seen it drop below roughly 310k. From other ML postings I've gathered that stat has something to do with files pending deletion, but I'm not positive. So far all I've done is restart the mds and mon daemons, which hasn't helped. What are the next steps for troubleshooting? I can turn up mds debug logging, but am not sure what to look for. Thanks for your help! Josh

2 3

Luminous 12.2.12 "clients failing to respond to capability release"
by Marc Roos 26 Sep '19

26 Sep '19

I am getting this error I have in two sessions[0] num_caps high ( I assume the error is about num_caps ). I am using a default luminous and a default centos7 with default kernel 3.10. Do I really still need to change to a not stock kernel to resolve this? I read this in posts of 2016 and 2019, yet I also saw some post on tuning the mds. [0] ceph daemon mds.c session ls { "id": 3634861, "num_leases": 0, "num_caps": 2183, "state": "open", "request_load_avg": 181, "uptime": 2545057.775869, "replay_requests": 0, "completed_requests": 0, "reconnecting": false, { "id": 3644756, "num_leases": 0, "num_caps": 2080, "state": "open", "request_load_avg": 32, "uptime": 2545057.775869, "replay_requests": 0, "completed_requests": 0, "reconnecting": false,

1 4

Cephfs + docker
by Alex L 26 Sep '19

26 Sep '19

Hi, I am trying to figure out why my portainer and pi-hole in docker keeps getting broken databases. All other docker applications are working flawlessly but not these. I am running Ubuntu 18.04 + kernel ceph mount for the data directory. Have looked at how others do it, and they seem to all use the docker plugin for getting an RBD image but looking at those plugins many seem to not have been updated for years.. Thanks in advance, A

3 2

Nautilus dashboard: MDS performance graph doesn't refresh
by Eugen Block 26 Sep '19

26 Sep '19

Hi all, I regularly check the MDS performance graphs in the dashboard, especially the requests per second is interesting in my case. Since our upgrade to Nautilus the values in the activity column are still refreshed every 5 seconds (I believe), but the graphs are not refreshed since that upgrade anymore. I couldn't find anything in the tracker or the mailing list, can anyone comment on this? Thank you & best regards, Eugen

2 3

Re: Nautilus : ceph dashboard ssl not working
by Lenz Grimmer 26 Sep '19

26 Sep '19

Hi Miha, interesting observation, I don't think we've noticed this before. Would you mind submitting a bug report about this on our tracker, including these logs? https://tracker.ceph.com/projects/mgr/issues/new Thanks in advance! Lenz On 9/26/19 10:01 AM, Miha Verlic wrote: > On 24. 09. 19 14:53, Lenz Grimmer wrote: >> On 9/24/19 1:37 PM, Miha Verlic wrote: >> >>> I've got slightly different problem. After a few days of running fine, >>> dashboard stops working because it is apparently seeking for wrong >>> certificate file in /tmp. If I restart ceph-mgr it starts to work again. >> >> Does the restart trigger the creation of a similar-looking file in /tmp? >> I wonder if there's some kind of cron job that cleans up the /tmp >> directory every now and then... > > There is systemd-tmpfiles-clean.timer, but it ignores PrivateTmp folders. > > But as said, crt and key files do exist in tmp and according to > timestamps they were created when ceph-mgr daemon was started; it seems > to me like ceph-mgr simply starts looking for wrong filenames after a while. > > Going through ceph-mgr logs I'd say the problem is internal webserver is > restarted, but ceph-mgr is not notified about new crt/key files: > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Bus STOPPING > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 8443)) shut down > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Stopped thread '_TimeoutMonitor'. > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Bus STOPPED > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Bus STARTING > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Started monitor thread '_TimeoutMonitor'. > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: 2019-09-19 12:54:16.618 > 7f3a796e4700 -1 client.0 error registering admin socket command: (17) > File exists > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: 2019-09-19 12:54:16.618 > 7f3a796e4700 -1 client.0 error registering admin socket command: (17) > File exists > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: 2019-09-19 12:54:16.618 > 7f3a796e4700 -1 client.0 error registering admin socket command: (17) > File exists > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: 2019-09-19 12:54:16.618 > 7f3a796e4700 -1 client.0 error registering admin socket command: (17) > File exists > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: 2019-09-19 12:54:16.618 > 7f3a796e4700 -1 client.0 error registering admin socket command: (17) > File exists > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Serving on :::8443 > > Sep 19 12:54:16 cephtest01 ceph-mgr[2247]: [19/Sep/2019:12:54:16] ENGINE > Bus STARTED -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 247165 (AG Nürnberg)

1 0

how many monitor should to deploy in a 1000+ osd cluster
by 展荣臻（信泰） 26 Sep '19

26 Sep '19

hi all: I have a production cluster, and it had 24 hosts (528 osds,3mons) at a former. Now we want to add 36 hosts so the osd increase to 1320 . does the monitor need to increase?how many numbers of monitor node is recommended? Another question is which monitor does monclient commnuicate with? And how it decide? Any suggestions are welcome!

7 7

Re: download.ceph.com repository changes
by Alfredo Deza 25 Sep '19

25 Sep '19

Reviving this old thread. I still think this is something we should consider as users still experience problems: * Impossible to 'pin' to a version. User installs 14.2.0 and 4 months later they add other nodes but version moved to 14.2.2 * Impossible to use a version that is not what the latest is (e.g. if someone doesn't need the release from Monday, but wants the one from 6 months ago), similar to the above * When a release is underway, the repository breaks because syncing packages takes hours. The operation is not atomic. * It is not currently possible to "remove" a bad release, in the past, this means cutting a new release as soon as possible, which can take days The latest issue (my fault!) was to cut a release and get the packages out without communicating with the release manager, which caused users to note there is a new version *as soon as it was up* vs, a process that could've not touched the 'latest' url until the announcement goes out. If you have been affected by any of these issues (or others I didn't come up with), please let us know in this thread so that we can find some common ground and try to improve the process. Thanks! On Tue, Jul 24, 2018 at 10:38 AM Alfredo Deza <adeza(a)redhat.com> wrote: > > Hi all, > > After the 12.2.6 release went out, we've been thinking on better ways > to remove a version from our repositories to prevent users from > upgrading/installing a known bad release. > > The way our repos are structured today means every single version of > the release is included in the repository. That is, for Luminous, > every 12.x.x version of the binaries is in the same repo. This is true > for both RPM and DEB repositories. > > However, the DEB repos don't allow pinning to a given version because > our tooling (namely reprepro) doesn't construct the repositories in a > way that this is allowed. For RPM repos this is fine, and version > pinning works. > > To remove a bad version we have to proposals (and would like to hear > ideas on other possibilities), one that would involve symlinks and the > other one which purges the known bad version from our repos. > > *Symlinking* > When releasing we would have a "previous" and "latest" symlink that > would get updated as versions move forward. It would require > separation of versions at the URL level (all versions would no longer > be available in one repo). > > The URL structure would then look like: > > debian/luminous/12.2.3/ > debian/luminous/previous/ (points to 12.2.5) > debian/luminous/latest/ (points to 12.2.7) > > Caveats: the url structure would change from debian-luminous/ to > prevent breakage, and the versions would be split. For RPMs it would > mean a regression if someone is used to pinning, for example pinning > to 12.2.2 wouldn't be possible using the same url. > > Pros: Faster release times, less need to move packages around, and > easier to remove a bad version > > > *Single version removal* > Our tooling would need to go and remove the known bad version from the > repository, which would require to rebuild the repository again, so > that the metadata is updated with the difference in the binaries. > > Caveats: time intensive process, almost like cutting a new release > which takes about a day (and sometimes longer). Error prone since the > process wouldn't be the same (one off, just when a version needs to be > removed) > > Pros: all urls for download.ceph.com and its structure are kept the same.

9 11