- ceph-users - lists.ceph.io

rbd device name reuse frequency
by Void Star Nill 21 Apr '20

21 Apr '20

Hello, How frequently do RBD device names get reused? For instance, when I map a volume on a client and it gets mapped to /dev/rbd0 and when it is unmapped, does a subsequent map reuse this name right away? I ask this question, because in our use case, we try to unmap a volume and we are thinking about adding some retries in case unmap fails for any reason. But I am concerned about race conditions such as the following: 1. thread 1 calls unmap, but the call times out and returns in the process, but in the background unmap request does go through and the device gets removed 2. thread 1 does a retry based on the device name. If between 1 and 2, another thread tries to map another volume and if it gets mapped to same device right after previous unmap was successful, then in step 2, we will be trying to unmap a device that doesn't belong to previous map. So I want to know how frequently do the device names get reused, and if there is a way to keep them using new names until they round back after a max limit. Thanks, Shridhar

4 5

How to migrate ceph-xattribs?
by Frank Schilder 20 Apr '20

20 Apr '20

De all, we are in the process of migrating a ceph file system from a 2-pool layout (rep meta+ec data) to the recently recommended 3-pool layout (rep meta, per primary data, ec data). As part of this, we need to migrate any ceph xattribs set on files and directories. As these are no longer discoverable, how would one go about this? Special cases: How to migrate quota settings? How to migrate dir- and file-layouts? Ideally, at least quota attributes should be transferable on the fly with tools like rsync. If automatic migration is not possible, is there at least an efficient way to *find* everything with special ceph attributes? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 5

ceph-ansible osd sizing and configuration
by Hemant Sonawane 20 Apr '20

20 Apr '20

Hi All I am facing some issue related to osd config in ceph-ansible I have 2 nvme disks where i have wal and db configure and i configure lvm like this in osd.yamL osd.yaml devices: - /dev/sdb - /dev/sdc - /dev/sdd - /dev/sde - /dev/sdf - /dev/sdg - /dev/sdh - /dev/sdi - /dev/sdj - /dev/sdk - /dev/sdl - /dev/sdm lvm_volumes: - data: /dev/sdb wal: wal-lv1 wal_vg: vg1 db: journal-lv1 db_vg: vg1 - data: /dev/sdb wal: wal-lv2 wal_vg: vg1 db: journal-lv2 db_vg: vg1 - data: /dev/sdb wal: wal-lv3 wal_vg: vg1 db: journal-lv3 db_vg: vg1 - data: /dev/sdb wal: wal-lv4 wal_vg: vg1 db: journal-lv4 db_vg: vg1 - data: /dev/sdb wal: wal-lv5 wal_vg: vg1 db: journal-lv5 db_vg: vg1 - data: /dev/sdc wal: wal-lv1 wal_vg: vg2 db: journal-lv1 db_vg: vg2 - data: /dev/sdc wal: wal-lv2 wal_vg: vg2 db: journal-lv2 db_vg: vg2 - data: /dev/sdc wal: wal-lv3 wal_vg: vg2 db: journal-lv3 db_vg: vg2 - data: /dev/sdc wal: wal-lv4 wal_vg: vg2 db: journal-lv4 db_vg: vg2 - data: /dev/sdc wal: wal-lv5 wal_vg: vg2 db: journal-lv5 db_vg: vg2 - data: /dev/sdd - data: /dev/sde - data: /dev/sdf - data: /dev/sdg - data: /dev/sdh - data: /dev/sdi - data: /dev/sdj - data: /dev/sdk - data: /dev/sdl - data: /dev/sdm I have also setup wal size and db size respectively in all.yaml all.yaml ntp_service_enabled: false mon_group_name: mons osd_group_name: osds mgr_group_name: mgrs ceph_origin: repository ceph_repository: community ceph_stable_release: octopus public_network: 10.18.0.0/22 cluster_network: 10.25.0.0/24 monitor_interface: ens3 configure_firewall: False ceph_test: true cephx: true ip_version: ipv4 osd_objectstore: bluestore dashboard_enabled: False grafana_server_group_name: grafana-server ceph_conf_overrides: global: bluestore block db size: 85899345920 (80GB) bluestore_block_wal_size: 1073741824 (1GB) i have also tried to to set values of db_size and journal_size in OSD options journal_size: 1024 # OSD journal size in MB block_db_size: 85899345920 But my osd partitions are not creating as I want and taking full disk space sdb 8:16 0 447.1G 0 disk └─ceph--3e81a883--86cd--4ddb--b0b0--be5892f6afe8-osd--block--b981ea8d--410d--4e2f--86f5--509112ff5ebf 253:0 0 447.1G 0 lvm sdc 8:32 0 447.1G 0 disk └─ceph--d3d82887--a7ad--40a4--9c3d--440cd9057fd9-osd--block--de6cae7d--daef--4144--a4bc--077f6044ef22 253:1 0 447.1G 0 lvm sdd 8:48 0 5.5T 0 disk └─ceph--e5aa3641--263c--45df--90d0--cd7e26772e69-osd--block--989fd7b2--d7b7--4170--8b33--cf4c8a29f5c1 253:2 0 5.5T 0 lvm sde 8:64 0 5.5T 0 disk └─ceph--286034b5--1c55--4fb6--8983--4ec6ef482dbd-osd--block--7cdc5ef4--8a88--4ff8--a08e--278985f4764c 253:3 0 5.5T 0 lvm sdf 8:80 0 5.5T 0 disk └─ceph--3bd8b866--b99e--41c4--a993--d69e85c1b658-osd--block--d32e42f4--1133--4c0d--aaf6--821b724fd220 253:4 0 5.5T 0 lvm sdg 8:96 0 5.5T 0 disk └─ceph--972ae2d1--2098--436f--a448--9015edfc4dc5-osd--block--79411ef4--2469--4243--9545--89cd266719f1 253:5 0 5.5T 0 lvm sdh 8:112 0 5.5T 0 disk └─ceph--92cd1c5f--3c67--420e--8a4d--c9ab0b75705b-osd--block--e643c7b3--cb10--49ab--86a8--31b60f4525dc 253:6 0 5.5T 0 lvm sdi 8:128 0 5.5T 0 disk └─ceph--0a27a900--0e81--4ef9--9b2c--692d0a609e7e-osd--block--30eddf48--5e12--4c2f--aed0--e06fb403ed33 253:7 0 5.5T 0 lvm sdj 8:144 0 5.5T 0 disk └─ceph--90156988--e0a3--4699--ae18--ad1f743521bb-osd--block--eb879f94--f289--4dd9--ba2e--48ff06a14708 253:8 0 5.5T 0 lvm sdk 8:160 0 5.5T 0 disk └─ceph--cdcabf84--e713--42c2--b2f0--f1bd6bd54924-osd--block--067418cf--1bd3--46a3--bdb9--b23216d6e8e3 253:9 0 5.5T 0 lvm sdl 8:176 0 5.5T 0 disk └─ceph--60829d44--ca66--44b3--b4be--05e8fe0c154e-osd--block--49a34fa3--c8ba--4fd9--ba3e--a6c4a889ecd5 253:10 0 5.5T 0 lvm sdm 8:192 0 5.5T 0 disk └─ceph--bd0a95e3--4d83--4c9e--a5fe--9f28414015fd-osd--block--abf5c032--eaf8--47b4--9966--67526009f68f 253:11 0 5.5T 0 lvm so after running playbook i get this result of osds with error failed: [10.11.2.202] (item={u'wal_vg': u'vg2', u'data': u'/dev/sdc', u'wal': u'wal-lv5', u'db': u'journal-lv5', u'db_vg': u'vg2'}) => changed=true ansible_loop_var: item cmd: - ceph-volume - --cluster - ceph - lvm - create - --bluestore - --data - /dev/sdc - --block.db - vg2/journal-lv5 - --block.wal - vg2/wal-lv5 delta: '0:00:03.260879' end: '2020-04-20 15:49:51.407784' item: data: /dev/sdc db: journal-lv5 db_vg: vg2 wal: wal-lv5 wal_vg: vg2 msg: non-zero return code rc: 1 start: '2020-04-20 15:49:48.146905' stderr: |- Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b26655b0-0859-4f8b-9471-0bafddfc36ef Running command: /sbin/lvcreate --yes -l 100%FREE -n osd-block-b26655b0-0859-4f8b-9471-0bafddfc36ef ceph-b7e18568-6735-4a1a-9aa1-95094189e859 stderr: Calculated size of logical volume is 0 extents. Needs to be larger. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: purged osd.0 Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 77, in main self.create(args) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 246, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 327, in prepare block_lv = self.prepare_data_device('block', osd_fsid) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 223, in prepare_data_device **kwargs) File "/usr/lib/python3/dist-packages/ceph_volume/api/lvm.py", line 1185, in create_lv process.run(command) File "/usr/lib/python3/dist-packages/ceph_volume/process.py", line 153, in run raise RuntimeError(msg) RuntimeError: command returned non-zero exit status: 5 So some one could please help me to figure solution for this issue . May be i am doing somewhere something wrong. If yes then please tell me. Also how sizing can be done for osds i mean how we can allow ceph-ansible to not to use default sizing? -- Thanks and Regards, Hemant Sonawane

1 0

Putting objects in Buckets
by jhamster＠chem.byu.edu 20 Apr '20

20 Apr '20

I have successfully installed a ceph storage cluster with a radosgw ceph object gateway daemon. I have been able to create an s3 bucket using a python boto3 script and a radosgw user, using the key and secret. I cannot find very much documentation concerning the usage of buckets to store objects, and since this is a new concept for me I was wondering if anyone with experience with this could help me out. I am hoping to use a frontend to manage ceph that requires s3 compatibility.

2 1

cephadm: how to override dashboard container image ?
by Dimitri Savineau 20 Apr '20

20 Apr '20

Hi, Is there a way to override the default dashboard container image hardcoded in cephadm [1] ? For the Ceph container image it possible to change the default value [2] either: - via cephadm cli and CEPHADM_IMAGE environment variable - via cephadm cli and the --image parameter - via container_image orchestrator parameter (ie: ceph config set global container_image xxx/yyy:zzz) But I'm not able to find something similar for the dashboard stack (alertmanager, grafana, node-exporter and prometheus) Any idea ? Regards, Dimitri [1] https://github.com/ceph/ceph/blob/master/src/cephadm/cephadm#L114-L160 [2] https://github.com/ceph/ceph/blob/master/src/cephadm/cephadm#L3

1 0

Goodbye and thanks for all the cuttlefish!
by Christopher Kunz 20 Apr '20

20 Apr '20

Hello ceph-{users,devel}, most of you might not know me, but I've been around a while. I've been on these lists since probably 2012, when my company, Filoo, started operating one of the early Ceph clusters for VM images. Much has changed since then, and today we shut down our biggest Ceph cluster. This was not for technical reasons, but purely for operational and commercial reasons. I still think Ceph is a superior concept and I'm sure it will continue to evolve and mature. The interactions on and off the Ceph lists, especially by Sage and many of the original Inktank team, have helped us tremendously, thanks a lot for that! I wish you all the best for the future of Ceph and will unsubscribe now. See you around! Best regads, Chris

1 0

Ceph 14.2.8 radosgw start failed due to handle_auth_bad_method
by Amit Ghadge 19 Apr '20

19 Apr '20

Hi All, I seen one issue while deploying radosgw on CentOS 7. monitor and data node deploy successfully but in radosgw seen below issue. I set ceph config set mon auth_cluster_required cephx ceph config set mon auth_service_required cephx ceph config set mon auth_client_required cephx Error: $ /usr/bin/radosgw -f --cluster ceph --name client.radosgw.masifgw01 --setuser ceph --setgroup ceph --debug_rgw 20 2020-04-19 09:05:05.421 7fe2eeea9700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [1] but i only support [2,1] failed to fetch mon config (--no-mon-config to skip mon dump: $ ceph mon dump dumped monmap epoch 4 epoch 4 fsid 10122b0b-8375-4a5a-9769-ff25a2d1d8b4 last_changed 2020-04-18 20:48:29.497461 created 2020-04-18 19:42:20.346011 min_mon_release 14 (nautilus) 0: [v2:10.44.127.xxx:3300/0,v1:10.44.127.xxx:6789/0] mon.node01 1: [v2:10.44.127.xxx:3300/0,v1:10.44.127.xxx:6789/0] mon.node02 2: [v2:10.44.127.xxx:3300/0,v1:10.44.127.xxx:6789/0] mon.node03 Anyone can face same issue, please help here. Thanks, Amit G

1 0

Re: Ghost usage on pool and unable to reclaim free space.
by Kári Bertilsson 19 Apr '20

19 Apr '20

Hi Khodayar That rados purge command did not actually seem to delete anything. It is still using 90GiB like before. I have 4 pools on these OSD's. But the "fast" pool still has 90GB but should be empty. Looking at the PG's in `ceph pg ls` LOG is using very little space on the pool. Are there any other logs ? # ceph osd pool ls detail pool 42 'cephfs_metadata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 186707 lfor 0/124366/185624 flags hashpspool stripe_width 0 application cephfs pool 51 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 186709 lfor 0/186698/186696 flags hashpspool,selfmanaged_snaps stripe_width 0 compression_algorithm lz4 compression_mode aggressive compression_required_ratio 0.9 application rbd removed_snaps [1~9] pool 53 'kube' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 186711 lfor 0/185831/185829 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~3] pool 54 'fast' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 186713 flags hashpspool,selfmanaged_snaps stripe_width 0 compression_algorithm lz4 compression_mode aggressive compression_required_ratio 0.9 application cephfs removed_snaps [52~2,55~2,59~2] # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 175 nvme 1.81898 1.00000 1.8 TiB 361 GiB 342 GiB 9.7 GiB 9.8 GiB 1.5 TiB 19.38 0.23 160 up osd.175 176 nvme 1.81898 1.00000 1.8 TiB 361 GiB 342 GiB 9.5 GiB 9.6 GiB 1.5 TiB 19.36 0.23 160 up osd.176 # ceph df POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR cephfs_metadata 42 320 MiB 2.82M 320 MiB 0.01 1.4 TiB N/A N/A 2.82M 0 B 0 B rbd 51 417 GiB 106.93k 417 GiB 12.32 1.4 TiB N/A N/A 106.93k 83 GiB 167 GiB kube 53 38 B 3 38 B 0 1.4 TiB N/A N/A 3 0 B 0 B fast 54 90 GiB 5.25M 90 GiB 2.93 1.4 TiB N/A N/A 5.25M 0 B 0 B # ceph pg ls PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP... ... 54.3a 81396 0 0 0 1512202599 0 0 3020 active+clean 9m 186704'339296 186722:615571 54.3b 82612 0 0 0 1556478139 0 0 3073 active+clean 9m 186706'340432 186722:609005 54.3c 82526 0 0 0 1553816829 0 0 3031 active+clean 9m 186704'806269 186722:1175811 54.3d 82144 0 0 0 1454278881 0 0 3016 active+clean 9m 186706'339167 186722:743880 54.3e 82740 0 0 0 1501081654 0 0 3056 active+clean 9m 186704'343212 186722:621323 54.3f 80846 0 0 0 1474357106 0 0 3048 active+clean 9m 186706'333741 186722:733149 On Sun, Apr 19, 2020 at 4:00 AM Khodayar Doustar <doustar(a)rayanexon.ir> wrote: > Hi Kári, > > You are purging only 2.6M objects with your purge command, so I guess that > 5.25M objects would be something else like logs. > Have you checked the osd df detail and osd metadata? > I have a case which osds bluestore logs are eating my whole space up, > maybe you are facing a similar one. > > Regards, > Khodayar > > On Sun, Apr 19, 2020 at 6:47 AM Kári Bertilsson <karibertils(a)gmail.com> > wrote: > >> Hello >> >> Running ceph v14.2.8 on everything. Pool is using replicated_rule with >> size/min_size 2 with 2 OSD's. I have scrubbed and deep scrubbed the OSDs. >> >> This pool was attached as data pool to cephfs containing alot of small >> files. I have since removed all files and detached the pool from the fs. >> Somehow the pool is still using 90GB. >> >> POOLS: >> POOL ID STORED OBJECTS USED %USED >> MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR >> UNDER COMPR >> fast 54 90 GiB 5.25M 90 GiB 2.93 >> 1.4 TiB N/A N/A 5.25M 0 B >> 0 B >> >> # rados -p fast ls|wc -l >> 1312030 >> >> # rados -p fast stat 100073acce3.00000000 >> error stat-ing fast/100073acce3.00000000: (2) No such file or directory >> >> # rados -p fast rm 100073acce3.00000000 >> error removing fast>100073acce3.00000000: (2) No such file or directory >> >> # rados -p fast get 100073acce3.00000000 test >> error getting fast/100073acce3.00000000: (2) No such file or directory >> >> I get the same error for every single object in the pool >> >> # rados purge fast --yes-i-really-really-mean-it >> Warning: using slow linear search >> Removed 2625749 objects >> successfully purged pool fast >> >> There was 5.25M objects in the pool before and after running this command. >> No change in ceph df. >> >> Any ideas how to reclaim the free space ? I can remove & recreate the >> pool, >> but i would like to know why and how to deal with this situation when i >> don't have that privilege >> >> Best regards >> Kári Bertilsson >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

1 0

Ghost usage on pool and unable to reclaim free space.
by Kári Bertilsson 19 Apr '20

19 Apr '20

Hello Running ceph v14.2.8 on everything. Pool is using replicated_rule with size/min_size 2 with 2 OSD's. I have scrubbed and deep scrubbed the OSDs. This pool was attached as data pool to cephfs containing alot of small files. I have since removed all files and detached the pool from the fs. Somehow the pool is still using 90GB. POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR fast 54 90 GiB 5.25M 90 GiB 2.93 1.4 TiB N/A N/A 5.25M 0 B 0 B # rados -p fast ls|wc -l 1312030 # rados -p fast stat 100073acce3.00000000 error stat-ing fast/100073acce3.00000000: (2) No such file or directory # rados -p fast rm 100073acce3.00000000 error removing fast>100073acce3.00000000: (2) No such file or directory # rados -p fast get 100073acce3.00000000 test error getting fast/100073acce3.00000000: (2) No such file or directory I get the same error for every single object in the pool # rados purge fast --yes-i-really-really-mean-it Warning: using slow linear search Removed 2625749 objects successfully purged pool fast There was 5.25M objects in the pool before and after running this command. No change in ceph df. Any ideas how to reclaim the free space ? I can remove & recreate the pool, but i would like to know why and how to deal with this situation when i don't have that privilege Best regards Kári Bertilsson

1 0

some ceph general questions about osd and pg
by harald.freidhof＠gmail.com 18 Apr '20

18 Apr '20

Hello togehter we want to implement a 3 nodes ceph cluster wirh nautilus. i already tested some ceph installations in our test enviroment and i have some generall questions. end of this month we will have three physical server with 256gb ram and two cpus and nerly 48 x 6tb disks. iam a little bit confused to calculate the pgs with the pgcalc on the ceph side. in the field osds what exactly meands that? the 3 physicaly osd nodes or the disks that will be used for the osds? what can you recomment us? we want later connect the ceph with rados and openstack and lvm thx in advance hfreidhof

3 3