- ceph-users - lists.ceph.io

by Torben Hørup

Hi A colleague and I are talking about making an event in Denmark for the danish ceph community, and we would like to get a feeling of how many ceph users are there in Denmark and hereof who would be interested in a Danish ceph event ? Regards, Torben

4 years, 8 months

4
3
0 0

Re: Ceph + SAMBA (vfs_ceph)

by David Disseldorp

[resend to ceph.io addr] Hi Salsa, The vfs_ceph "ceph: user_id" parameter specifies the CephX user ID which Samba (via libcephfs) uses to authenticate with the Ceph cluster. These credentials are completely separate to the SMB user credentials which you provide with smbclient, etc. It seems that you might be confusing them in your setup. As documented in the Samba vfs_ceph manpage, "kernel share modes = no" must currently be configured for CephFS shares. Also, the "cephfs" share name in your smb.conf doesn't match the "cephfs_a" name you're giving to smbclient. I'd suggest configuring a locally backed Samba share first and get that working with your Windows clients before moving over to the CephFS backed share. Cheers, David

4 years, 8 months

1
0
0 0

Failure to start ceph-mon in docker

by Frank Schilder

Hi Robert, this sounds a bit worse than I thought. I remember that RedHat stopped packaging Ubuntu containers. The default for the public containers is now Centos, which I'm running on all my hosts (I also use ceph containers). If Ubuntu indeed reserves a different default UID for ceph and UID 167 is available, then you should probably create/change the ceph user with GID=UID=167 on all hosts before installing anything ceph related. Otherwise there might be more to re-own than just /var/lib/ceph (e.g /var/log/ceph). The ansible tasks I included earlier will do this user creation. Everything should work fine after that. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Robert LeBlanc <robert(a)leblancnet.us> Sent: 29 August 2019 16:41 To: Frank Schilder Cc: ceph-users Subject: Re: [ceph-users] Failure to start ceph-mon in docker Frank, Thank you for the explanation, these are freshly installed machines and did not have ceph on them. I checked one of the other OSD nodes and there is no ceph user in /etc/passwd, nor is UID 167 allocated to any user. I did install ceph-common from the 18.04 repos before realizing that deploying ceph in containers did not update the host's /etc/apt/sources.list (or add an entry in /etc/apt/sources.list.d/). I manually added the repo for nautilus and upgraded the packages. So, I don't know if that had anything to do with it. Maybe Ubuntu packages ceph under UID 64045 and upgrading to the Ceph distributed packages didn't change the UID. Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Aug 29, 2019 at 12:33 AM Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> wrote: Hi Robert, this is a bit less trivial than it might look right now. The ceph user is usually created by installing the package ceph-common. By default it will use id 167. If the ceph user already exists, I would assume it will use the existing user to allow an operator to avoid UID collisions (if 167 is used already). If you use docker, the ceph UID on the host and inside the container should match (or need to be translated). If they don't, you will have a lot of fun re-owning stuff all the time, because deployments will use the symbolic name ceph, which has different UIDs on the host and inside the container in your case. I would recommend removing this discrepancy as soon as possible: 1) Find out why there was a ceph user with UID different from 167 before installation of ceph-common. Did you create it by hand? Was UID 167 allocated already? 2) If you can safely change the GID and UID of ceph to 167, just do groupmod+usermod with new GID and UID. 3) If 167 is used already by another service, you will have to map the UIDs between host and container. To prevent ansible from deploying dockerized ceph with mismatching user ID for ceph, add these tasks to an appropriate part of your deployment (general host preparation or so): - name: "Create group 'ceph'." group: name: ceph gid: 167 local: yes state: present system: yes - name: "Create user 'ceph'." user: name: ceph password: "!" comment: "ceph-container daemons" uid: 167 group: ceph shell: "/sbin/nologin" home: "/var/lib/ceph" create_home: no local: yes state: present system: yes This should err if a group and user ceph already exist with IDs different from 167. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: ceph-users <ceph-users-bounces(a)lists.ceph.com<mailto:ceph-users-bounces@lists.ceph.com>> on behalf of Robert LeBlanc <robert(a)leblancnet.us<mailto:robert@leblancnet.us>> Sent: 28 August 2019 23:23:06 To: ceph-users Subject: Re: [ceph-users] Failure to start ceph-mon in docker Turns out /var/lib/ceph was ceph.ceph and not 167.167, chowning it made things work. I guess only monitor needs that permission, rgw,mgr,osd are all happy without needing it to be 167.167. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Aug 28, 2019 at 1:45 PM Robert LeBlanc <robert(a)leblancnet.us<mailto:robert@leblancnet.us><mailto:robert@leblancnet.us<mailto:robert@leblancnet.us>>> wrote: We are trying to set up a new Nautilus cluster using ceph-ansible with containers. We got things deployed, but I couldn't run `ceph s` on the host so decided to `apt install ceph-common and installed the Luminous version from Ubuntu 18.04. For some reason the docker container that was running the monitor restarted and won't restart. I added the repo for Nautilus and upgraded ceph-common, but the problem persists. The Manager and OSD docker containers don't seem to be affected at all. I see this in the journal: Aug 28 20:40:55 sun-gcs02-osd01 systemd[1]: Starting Ceph Monitor... Aug 28 20:40:55 sun-gcs02-osd01 docker[2926]: Error: No such container: ceph-mon-sun-gcs02-osd01 Aug 28 20:40:55 sun-gcs02-osd01 systemd[1]: Started Ceph Monitor. Aug 28 20:40:55 sun-gcs02-osd01 docker[2949]: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. Aug 28 20:40:56 sun-gcs02-osd01 docker[2949]: 2019-08-28 20:40:56 /opt/ceph-container/bin/entrypoint.sh: Existing mon, trying to rejoin cluster... Aug 28 20:40:56 sun-gcs02-osd01 docker[2949]: warning: line 41: 'osd_memory_target' in section 'osd' redefined Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: 2019-08-28 20:41:03 /opt/ceph-container/bin/entrypoint.sh: /etc/ceph/ceph.conf is already memory tuned Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: 2019-08-28 20:41:03 /opt/ceph-container/bin/entrypoint.sh: SUCCESS Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: exec: PID 368: spawning /usr/bin/ceph-mon --cluster ceph --default-log-to-file=false --default-mon-cluster-log-to-file=false --setuser ceph --setgroup ceph -d --mon-cluster-log-to-stderr --log-stderr-prefix=debug -i sun-gcs02-osd01 --mon-data /var/lib/ceph/mon/ceph-sun-gcs02-osd01 --public-addr 10.65.101.21 Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: exec: Waiting 368 to quit Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: warning: line 41: 'osd_memory_target' in section 'osd' redefined Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: debug 2019-08-28 20:41:03.835 7f401283c180 0 set uid:gid to 167:167 (ceph:ceph) Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: debug 2019-08-28 20:41:03.835 7f401283c180 0 ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable), process ceph-mon, pid 368 Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: debug 2019-08-28 20:41:03.835 7f401283c180 -1 stat(/var/lib/ceph/mon/ceph-sun-gcs02-osd01) (13) Permission denied Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: debug 2019-08-28 20:41:03.835 7f401283c180 -1 error accessing monitor data directory at '/var/lib/ceph/mon/ceph-sun-gcs02-osd01': (13) Permission denied Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: teardown: managing teardown after SIGCHLD Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: teardown: Waiting PID 368 to terminate Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: teardown: Process 368 is terminated Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: teardown: Bye Bye, container will die with return code -1 Aug 28 20:41:03 sun-gcs02-osd01 docker[2949]: teardown: if you don't want me to die and have access to a shell to debug this situation, next time run me with '-e DEBUG=stayalive' Aug 28 20:41:04 sun-gcs02-osd01 systemd[1]: ceph-mon(a)sun-gcs02-osd01.service: Main process exited, code=exited, status=255/n/a Aug 28 20:41:04 sun-gcs02-osd01 systemd[1]: ceph-mon(a)sun-gcs02-osd01.service: Failed with result 'exit-code'. The directories for the monitor are owned by 167.167 and matches the UID.GID that the container reports. oot@sun-gcs02-osd01:~# ls -lhd /var/lib/ceph/ drwxr-x--- 14 ceph ceph 4.0K Jul 30 22:15 /var/lib/ceph/ root@sun-gcs02-osd01:~# ls -lh /var/lib/ceph/ total 56K drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-mds drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-mgr drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-osd drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-rbd drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-rbd-mirror drwxr-xr-x 2 167 167 4.0K Jul 30 22:16 bootstrap-rgw drwxr-xr-x 3 167 167 4.0K Jul 30 22:15 mds drwxr-xr-x 3 167 167 4.0K Jul 30 22:15 mgr drwxr-xr-x 3 167 167 4.0K Jul 30 22:15 mon drwxr-xr-x 14 167 167 4.0K Jul 30 22:28 osd drwxr-xr-x 4 167 167 4.0K Aug 1 23:36 radosgw drwxr-xr-x 254 167 167 12K Aug 28 20:44 tmp root@sun-gcs02-osd01:~# ls -lh /var/lib/ceph/mon/ total 4.0K drwxr-xr-x 3 167 167 4.0K Jul 30 22:16 ceph-sun-gcs02-osd01 root@sun-gcs02-osd01:~# ls -lh /var/lib/ceph/mon/ceph-sun-gcs02-osd01/ total 16K -rw------- 1 167 167 77 Jul 30 22:15 keyring -rw-r--r-- 1 167 167 8 Jul 30 22:15 kv_backend -rw-r--r-- 1 167 167 3 Jul 30 22:16 min_mon_release drwxr-xr-x 2 167 167 4.0K Aug 28 19:16 store.db root@sun-gcs02-osd01:~# ls -lh /var/lib/ceph/mon/ceph-sun-gcs02-osd01/store.db/ total 149M -rw-r--r-- 1 167 167 1.7M Aug 28 19:16 050225.log -rw-r--r-- 1 167 167 65M Aug 28 19:16 050227.sst -rw-r--r-- 1 167 167 45M Aug 28 19:16 050228.sst -rw-r--r-- 1 167 167 16 Aug 16 07:40 CURRENT -rw-r--r-- 1 167 167 37 Jul 30 22:15 IDENTITY -rw-r--r-- 1 167 167 0 Jul 30 22:15 LOCK -rw-r--r-- 1 167 167 1.3M Aug 28 19:16 MANIFEST-027846 -rw-r--r-- 1 167 167 4.7K Aug 1 23:38 OPTIONS-002825 -rw-r--r-- 1 167 167 4.7K Aug 16 07:40 OPTIONS-027849 ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

4 years, 8 months

1
0
0 0

How to map 2 different Openstack users belonging to the same project to 2 distinct radosgw users ?

by Massimo Sgaravatto

I have a question regarding ceph-openstack integration for object storage. Is there some configuration/hack/workaround that allows to map 2 different users belonging to the same OpenStack project to 2 distinct radosgw users ? I saw this old bug: https://tracker.ceph.com/issues/20570 but it looks like it isn't going to be addressed in the near future Thanks, Massimo

4 years, 8 months

2
2
0 0

Ceph pool snapshot mechanism

by Yannick.Martin＠cdiscount.com

Hello Can you tell me if Ceph snapshot at pool level rely on copy-on-write mechanism ? I found a RedHat help page that said that this is incompatible with RBD snapshot (I don't use it) and taks about "irreversible" ? Can you tell me about the last term ? Regards,

4 years, 8 months

1
0
0 0

modifying "osd_memory_target"

by Amudhan P

Hi, How do i change "osd_memory_target" in ceph command line. regards Amudhan

4 years, 8 months

2
1
0 0

How to customize object size

by tapas＠wolk.com

Hi, I am new to ceph ... i am trying to increase object file size .. i can upload file size upto 128MB .. how can i upload more than 128MB file . i can upload file using this rados --pool z10 put testfile-128M.txt testfile-128M.txt Thats ok when the file size is upto 128MB But its not ok when the file size is >128MB Got this error error putting z13/testfile-129MB.txt: (27) File too large Please help Tapas Jana

4 years, 8 months

2
1
0 0

"ceph-users" mailing list!

by Tapas Jana

tapas@wolkcom

4 years, 8 months

1
0
0 0

Ceph Ansible - - name: set grafana_server_addr fact - ipv4

by Lee Norvall

Hi Ceph: nautilus (14.2.2) NFS-Ganesha v 2.8 ceph-ansible stable 4.0 << git checkout 28th Aug CentOS 7 I am trying to do a fresh installation using Ceph Ansible and I am getting the following error when running the playbook. I have not enabled or config dashboard/grafana/prometheus yet. fatal: [stor1]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: No first item, sequence was empty. The error appears to be in '/usr/share/ceph-ansible/roles/ceph-facts/tasks/facts.yml': line 314, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: set grafana_server_addr fact - ipv4 ^ here If I remove [grafana-server] from the /etc/ansible/hosts file I get fatal: [stor1]: FAILED! => changed=false msg: you must add a [grafana-server] group and add at least one node. Any idea what might be causing this?

4 years, 8 months

3
2
0 0

Re: FileStore OSD, journal direct symlinked, permission troubles.

by Marco Gaiarin

Mandi! Alwin Antreich In chel di` si favelave... > > There's something i can do? Thanks. > Did you go through our upgrade guide(s)? Sure! > See the link [0] below, for the > permission changes. They are needed when an upgrade from Hammer to Jewel > is done. Sure! The problem arise in the 'Set partition type' section, because: root@deadpool:~# for l in $(readlink -f /var/lib/ceph/osd/ceph-*/journal); do echo $l; blkid -o udev -p $l; echo ""; done /dev/sda5 ID_PART_ENTRY_SCHEME=dos ID_PART_ENTRY_UUID=9c277a97-05 ID_PART_ENTRY_TYPE=0xfd ID_PART_ENTRY_NUMBER=5 ID_PART_ENTRY_OFFSET=546877440 ID_PART_ENTRY_SIZE=97654784 ID_PART_ENTRY_DISK=8:0 /dev/sda6 ID_PART_ENTRY_SCHEME=dos ID_PART_ENTRY_UUID=9c277a97-06 ID_PART_ENTRY_TYPE=0xfd ID_PART_ENTRY_NUMBER=6 ID_PART_ENTRY_OFFSET=644534272 ID_PART_ENTRY_SIZE=97654784 ID_PART_ENTRY_DISK=8:0 /dev/sdb7 ID_PART_ENTRY_SCHEME=dos ID_PART_ENTRY_UUID=802474ca-07 ID_PART_ENTRY_TYPE=0xfd ID_PART_ENTRY_NUMBER=7 ID_PART_ENTRY_OFFSET=742191104 ID_PART_ENTRY_SIZE=97654784 ID_PART_ENTRY_DISK=8:16 /dev/sdb8 ID_PART_ENTRY_SCHEME=dos ID_PART_ENTRY_UUID=802474ca-08 ID_PART_ENTRY_TYPE=0xfd ID_PART_ENTRY_NUMBER=8 ID_PART_ENTRY_OFFSET=839847936 ID_PART_ENTRY_SIZE=97853440 ID_PART_ENTRY_DISK=8:16 As stated, partitions are 'DOS'... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

4 years, 8 months

2
2
0 0

2024

2023

2022

2021

2020

2019

ceph-users