- ceph-users - lists.ceph.io

mgr's stop responding, dropping out of cluster with _check_auth_rotating
by Welby McRoberts 21 Dec '20

21 Dec '20

Hi Folks We've noticed that in a cluster of 21 nodes (5 mgrs&mons & 504 OSDs with 24 per node) that the mgr's are, after a non specific period of time, dropping out of the cluster. The logs only show the following: debug 2020-12-10T02:02:50.409+0000 7f1005840700 0 log_channel(cluster) log [DBG] : pgmap v14163: 4129 pgs: 4129 active+clean; 10 GiB data, 31 TiB used, 6.3 PiB / 6.3 PiB avail debug 2020-12-10T03:20:59.223+0000 7f10624eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-12-10T02:20:59.226159+0000) debug 2020-12-10T03:21:00.223+0000 7f10624eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-12-10T02:21:00.226310+0000) The _check_auth_rotating repeats approximately every second. The instances are all syncing their time with NTP and have no issues on that front. A restart of the mgr fixes the issue. It appears that this may be related to https://tracker.ceph.com/issues/39264. The suggestion seems to be to disable prometheus metrics, however, this obviously isn't realistic for a production environment where metrics are critical for operations. Please let us know what additional information we can provide to assist in resolving this critical issue. Cheers Welby

4 6

Re: Setting up NFS with Octopus
by Jens Hyllegaard (Soft Design A/S) 21 Dec '20

21 Dec '20

I am sorry, but I am not sure how to do that? We have just started working with Ceph. -----Original Message----- From: Eugen Block <eblock(a)nde.ag> Sent: 18. december 2020 12:06 To: Jens Hyllegaard (Soft Design A/S) <jens.hyllegaard(a)softdesign.dk> Subject: Re: [ceph-users] Re: Setting up NFS with Octopus Oh you're right, it worked for me, I just tried that with a new path and it was created for me. Can you share the client keyrings? I have two nfs daemons running and they have these permissions: client.nfs.ses7-nfs.host2 key: AQClNNJf5KHVERAAAzhpp9Mclh5wplrcE9VMkQ== caps: [mon] allow r caps: [osd] allow rw pool=nfs-test namespace=ganesha client.nfs.ses7-nfs.host3 key: AQCqNNJf4rlqBhAARGTMkwXAldeprSYgmPEmJg== caps: [mon] allow r caps: [osd] allow rw pool=nfs-test namespace=ganesha Zitat von "Jens Hyllegaard (Soft Design A/S)" <jens.hyllegaard(a)softdesign.dk>: > On the Create NFS export page it says the directory will be created. > > Regards > > Jens > > > -----Original Message----- > From: Eugen Block <eblock(a)nde.ag> > Sent: 18. december 2020 11:52 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Setting up NFS with Octopus > > Hi, > > is the path (/objstore) present within your CephFS? If not you need to > mount the CephFS root first and create your directory to have NFS > access it. > > > Zitat von "Jens Hyllegaard (Soft Design A/S)" > <jens.hyllegaard(a)softdesign.dk>: > >> Hi. >> >> We are completely new to Ceph, and are exploring using it as an NFS >> server at first and expand from there. >> >> However we have not been successful in getting a working solution. >> >> I have set up a test environment with 3 physical servers, each with >> one OSD using the guide at: >> https://docs.ceph.com/en/latest/cephadm/install/ >> >> I created a new replicated pool: >> ceph osd pool create objpool replicated >> >> And then I deployed the gateway: >> ceph orch apply nfs objstore objpool nfs-ns >> >> I then created a new CephFS volume: >> ceph fs volume create objstore >> >> So far so good 😊 >> >> My problem is when I try to create the NFS export The settings are as >> follows: >> Cluster: objstore >> Daemons: nfs.objstore >> Storage Backend: CephFS >> CephFS User ID: admin >> CephFS Name: objstore >> CephFS Path: /objstore >> NFS Protocol: NFSV3 >> Access Type: RW >> Squash: all_squash >> Transport protocol: both UDP & TCP >> Client: Any client can access >> >> However when I click on Create NFS export, I get: >> Failed to create NFS 'objstore:/objstore' >> >> error in mkdirs /objstore: Permission denied [Errno 13] >> >> Has anyone got an idea as to why this is not working? >> >> If you need any further information, do not hesitate to say so. >> >> >> Best regards, >> >> Jens Hyllegaard >> Senior consultant >> Soft Design >> Rosenkaeret 13 | DK-2860 Søborg | Denmark | +45 39 66 02 00 | >> softdesign.dk<http://www.softdesign.dk/> | synchronicer.com >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

2 8

Is there a command to update a client with a new generated key?
by Marc Roos 21 Dec '20

21 Dec '20

Is there a command to update a client with a new generated key? Something like: ceph auth new-key client.rbd Could be usefull if you accidentaly did a ceph auth ls, because that still displays keys ;)

2 2

python API
by Vladimir Prokofev 21 Dec '20

21 Dec '20

Hello. I'm trying to write some Python code for analysis of my RBD images storage usage, rbd and rados package versions are 14.2.16. Basically I want the same data that I can acquire from shell 'rbd du <image>' and 'rbd info <image>' commands, but through Python API. At the moment I can connect to the cluster, get an image list from the pool, but that's about it. As far as I can read, there's no API method to obtain image info such as provisioned/used storage, etc. Is this really not possible with the current Python API, or am I missing something? Documentation used: https://docs.ceph.com/en/latest/rbd/api/librbdpy/ https://docs.ceph.com/en/latest/rados/api/python/

1 0

Issues upgrading from ceph 15.2.7 to 15.2.8 related to cephadm pull
by David Orman 20 Dec '20

20 Dec '20

Hi, We have an Octopus installation using cephadm/containers in podman. We're trying to upgrade from 15.2.7 to 15.2.8. The mgrs upgrade successfully, but the we then get a failure that cephadm failed to pull an image: [WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image Failed to pull docker.io/ceph/ceph:v15.2.8 on ceph02: Looking at this server, and trying to run the cephadm pull manually: root@ceph02:~# cephadm pull Using recent ceph image docker.io/ceph/ceph:v15.2.8 Pulling container image docker.io/ceph/ceph:v15.2.8... Traceback (most recent call last): File "/usr/sbin/cephadm", line 6111, in <module> r = args.func() File "/usr/sbin/cephadm", line 1381, in _infer_image return func() File "/usr/sbin/cephadm", line 2676, in command_pull return command_inspect_image() File "/usr/sbin/cephadm", line 1381, in _infer_image return func() File "/usr/sbin/cephadm", line 2716, in command_inspect_image info_from = get_image_info_from_inspect(out.strip(), args.image) File "/usr/sbin/cephadm", line 2727, in get_image_info_from_inspect image_id, digests = out.split(',', 1) ValueError: not enough values to unpack (expected 2, got 1) root@ceph02:~# cephadm version Using recent ceph image docker.io/ceph/ceph:v15.2.8 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable) root@ceph02:~# podman images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/ceph/ceph v15.2.8 5553b0cb212c 3 days ago 965 MB docker.io/ceph/ceph v15.2.7 2bc420ddb175 2 weeks ago 979 MB docker.io/ceph/ceph v15 2bc420ddb175 2 weeks ago 979 MB docker.io/prom/node-exporter v0.18.1 e5a616e4b9cf 18 months ago 24.3 MB root@ceph02:~# We've upgraded this cluster multiple times prior without issue, so this appear to be unique to the 15.2.7 -> 15.2.8 upgrade. If we pass --image with docker.io/ceph/ceph:v15.2.8, the same behavior occurs. It seems like cephadm in the 15.2.8 image isn't playing nicely with the pull command, for some reason. Interestingly, I did a fresh install of v15.2.8 on one server, and after the installation, trying a 'cephadm pull' also fails on that 15.2.8 setup, so it seems like the pull functionality may be broken on these images.

1 1

New cephadm install, OSD down after a few hours
by Jie Zhang 20 Dec '20

20 Dec '20

Hello, I'm new to ceph and setting up my first cluster and playing around with it. I followed the steps in cephadm guide ( https://docs.ceph.com/en/latest/cephadm/install/) Here is my implementation of the instructions, from a Ubuntu Server 20.04.1 install: sudo apt-get update && sudo apt-get dist-upgrade -y sudo apt-get install curl git nano docker docker-compose docker.io attr ntp bash-completion -y sudo usermod -aG docker $USER curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm chmod +x cephadm sudo ./cephadm add-repo --release octopus sudo ./cephadm install sudo mkdir -p /etc/ceph sudo cephadm install ceph-common ceph sudo cephadm bootstrap --mon-ip <IP> ceph orch daemon add server-node1:/dev/sda ceph orch daemon add server-node1:/dev/sdc ceph orch daemon add server-node1:/dev/sdd ceph orch daemon add server-node1:/dev/sde ceph orch daemon add server-node1:/dev/sdf Then I log into the ceph dashboard, change password, click around. Take a break... ...HOURS Later... My OSDs are down. I notice this in ceph-volume.log: [ceph_volume.main][INFO ] Running command: ceph-volume lvm deactivate 1 7a0fb8df-2d12-4e96-9def-5b2c195f6af4 So I run: ceph-volume lvm activate --all And my OSDs are back and 'cephadm ls' shows them as legacy. However, running 'cephadm adopt --style legacy --name osd.0' causes the osd to go down again. What is going on? PS: The only two other issues I in the logs see are /usr/bin/docker:stderr Error: No such object: ceph-<ID>-osd.0 and [ceph_volume.util.system][INFO ] /var/lib/ceph/osd/ceph-0 does not appear to be a tmpfs mount Jie

1 0

failed to process reshard logs
by Seena Fallah 19 Dec '20

19 Dec '20

Hi, I used radosgw-admin reshard process to process a manual bucket resharding after it completes it logs an error below ERROR: failed to process reshard logs, error=(2) No such file or directory I've added a bucket to resharding queue with radosgw-admin reshard add --bucket bucket-tmp --num-shards 2053 Is anything wrong with it? Using nautilus 14.2.14. Thanks.

1 0

Clearing contents of OSDs without removing them?
by Dallas Jones 19 Dec '20

19 Dec '20

Stumbling closer toward a usable production cluster with Ceph, but I have yet another stupid n00b question I'm hoping you all will tolerate. I have 38 OSDs up and in across 4 hosts. I (maybe prematurely) removed my test filesystem as well as the metadata and data pools used by the deleted filesystem. This leaves me with 38 OSDs with a bunch of data on them. Is there a simple way to just whack all of the data on all of those OSDs before I create new pools and a new filesystem? Version: ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) As you can see from the partial output of ceph -s, I left a bunch of crap spread across the OSDs... pools: 8 pools, 32 pgs objects: 219 objects, 1.2 KiB usage: 45 TiB used, 109 TiB / 154 TiB avail pgs: 32 active+clean Thanks in advance for a shove in the right direction. -Dallas

3 2

Fwd: ceph-fuse false passed X_OK check
by Alex Taylor 19 Dec '20

19 Dec '20

cc to the list On Thu, Dec 17, 2020 at 11:39 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote: > > On Wed, Dec 16, 2020 at 5:46 PM Alex Taylor <alexu4993(a)gmail.com> wrote: > > > > Hi Cephers, > > > > I'm using VSCode remote development with a docker server. It worked OK > > but fails to start the debugger after /root mounted by ceph-fuse. The > > log shows that the binary passes access X_OK check but cannot be > > actually executed. see: > > > > ``` > > strace_log: access("/root/.vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7", > > X_OK) = 0 > > > > root@develop:~# ls -alh > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > -rw-r--r-- 1 root root 978 Dec 10 13:06 > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > ``` > > > > I also test the access syscall on ext4, xfs and even cephfs kernel > > client, all of them return -EACCES, which is expected (the extension > > will then explicitly call chmod +x). > > > > After some digging in the code, I found it is probably caused by > > https://github.com/ceph/ceph/blob/master/src/client/Client.cc#L5549-L5550. > > So here come two questions: > > 1. Is this a bug or is there any concern I missed? > > I tried reproducing it with the master branch and could not. It might > be due to an older fuse/ceph. I suggest you upgrade! > I tried the master(332a188d9b3c4eb5c5ad2720b7299913c5a772ee) as well and the issue still exists. My test program is: ``` #include <stdio.h> #include <unistd.h> int main() { int r; const char path[] = "test"; r = access(path, F_OK); printf("file exists: %d\n", r); r = access(path, X_OK); printf("file executable: %d\n", r); return 0; } ``` And the test result: ``` # local filesystem: ext4 root@f626800a6e85:~# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:13 test root@f626800a6e85:~# ./a.out file exists: 0 file executable: -1 root@f626800a6e85:~# findmnt -t fuse.ceph-fuse TARGET SOURCE FSTYPE OPTIONS /root/mnt ceph-fuse fuse.ceph-fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other root@f626800a6e85:~# cd mnt # ceph-fuse root@f626800a6e85:~/mnt# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:10 test root@f626800a6e85:~/mnt# ./a.out file exists: 0 file executable: 0 root@f626800a6e85:~/mnt# ./test bash: ./test: Permission denied ``` Again, ceph-fuse says file `test` is executable but in fact it can't be executed. The kernel version I'm testing on is: ``` root@f626800a6e85:~/mnt# uname -ar Linux f626800a6e85 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64 GNU/Linux ``` Please try the program above and make sure you're running it as root user, thank you. And if the reproduction still fails, please let me know the kernel version. > > 2. It works again with fuse_default_permissions=true, any drawbacks if > > this option is set? > > Correctness (ironically, for you) and performance. > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Principal Software Engineer > Red Hat Sunnyvale, CA > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >

1 0

OSD spec db_devices: rotational: 0 not working in 15.2.7
by David Orman 18 Dec '20

18 Dec '20

Hi, We deployed a cluster with an OSD spec like this: service_type: osd service_id: osd_spec_test1 placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 This worked on an earlier version of 15 (I think we deployed with 15.2.5). However, this same drive spec results in the following in ceph orch ls --export: --- service_type: osd service_id: osd_spec_test1 service_name: osd.osd_spec_test1 placement: host_pattern: '*' spec: data_devices: rotational: 1 db_slots: 12 filter_logic: AND objectstore: bluestore Now, if we change rotational to 1 for db_devices: service_type: osd service_id: osd_spec_test1 service_name: osd.osd_spec_test1 placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 1 db_slots: 12 filter_logic: AND objectstore: bluestore For some reason, db_devices is ignored if rotational: 0 (our intended use case). This definitely worked before, our entire cluster was provisioned this way. We ran into this when attempting to replace a broken OSD. Any ideas what might be going on?

1 0