- ceph-users - lists.ceph.io

[cinder] Cinder & Ceph Integration Error: No Valid Backend
by SSelf＠performair.com 07 Jan '21

07 Jan '21

All; We're having problems with our Openstack/Ceph integration. The versions we're using are Ussuri & Nautilus. When trying to create a volume, the volume is created, though the status is stuck at 'ERROR'. This appears to be the most relevant line from the Cinder scheduler.log: 2021-01-07 14:00:38.473 140686 ERROR cinder.scheduler.flows.create_volume [req-f86556b5-cb2e-4b2d-b556-ed07e632289d 824c26c133b34d8b8e84a7acabbe6f91 a983323b5ffc47e18660794cd9344869 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available Here is the 'cinder.conf' from our Controller Node: [DEFAULT] # define own IP address my_ip = 10.0.80.40 log_dir = /var/log/cinder state_path = /var/lib/cinder auth_strategy = keystone enabled_backends = ceph glance_api_version = 2 debug = true # RabbitMQ connection info transport_url = rabbit://openstack:<password>@10.0.80.40:5672 enable_v3_api = True # MariaDB connection info [database] connection = mysql+pymysql://cinder:<password>@10.0.80.40/cinder # Keystone auth info [keystone_authtoken] www_authenticate_uri = http://10.0.80.40:5000 auth_url = http://10.0.80.40:5000 memcached_servers = 10.0.80.40:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = cinder password = <password> [oslo_concurrency] lock_path = $state_path/tmp [ceph] volume_driver = cinder.volume.drivers.rbd.RBDDriver volume_backend_name = ceph rbd_pool = rbd_os_volumes rbd_ceph_conf = /etc/ceph/463/ceph.conf rbd_flatten_volume_from_snapshot = false rbd_max_clone_depth = 5 rbd_store_chunk_size = 4 rados_connect_timeout = -1 rbd_user = cinder rbd_exclusive_cinder_pool = true backup_driver = cinder.backup.drivers.ceph backup_ceph_conf = /etc/ceph/300/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = rbd_os_backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true Does anyone have any ideas as to what is going wrong? Thank you, Stephen Self IT Manager Perform Air International sself(a)performair.com www.performair.com

1 1

safest way to remove a host from Mimic
by Adrian Nicolae 07 Jan '21

07 Jan '21

Hi guys, I need to remove a host (osd server) from my Ceph Mimic. First I started to remove every OSD drive one by one with 'ceph osd out' and then 'ceph osd purge'. After all the drives will be removed from the crush map, I will still have the host empty, without any drive, and with the crush weight = 0. What is the recommend way to remove the host from Ceph : - ceph-deploy purge <hostname> - ceph-deploy purgedata <hostname> - ceph osd crush remove <hostname> ? Thanks.

1 0

Storage down due to MON sync very slow
by Frank Schilder 06 Jan '21

06 Jan '21

Dear all, I had to restart one out of 3 MONs on an empty MON DB dir. It is in state syncing right now, but I'm not sure if there is any progress. The cluster is completely unresponsive even though I have 2 healthy MONs. Is there any way to sync the DB directory faster and/or without downtime? Thanks a lot! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 4

radosgw sync using more capacity on secondary than on master zone
by Scheurer François 06 Jan '21

06 Jan '21

Dear Ceph contributors While our (new) rgw secondary zone is doing the initial data sync from our master zone, we noticed that the reported capacity usage was getting higher than on primary zone: Master Zone: ceph version 14.2.5 zone parameters: "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "false", "sync_from": [], "redirect_zone": "" bucket stats: => "size_actual GiB": 269868.9823989868, "size_utilized GiB": 17180102071.868008, <= (cf. other issue below) "num_objects": 100218788, "Compression Rate": 63660.899148714125 pool stats: => "stored": 403834132234240, "objects": 191530416, "kb_used": 692891724288, "bytes_used": 709521125670912, "percent_used": 0.7570806741714478, "max_avail": 136595529269248 erasure-code-profile: crush-device-class=nvme crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 osd parameters: bluefs_alloc_size 1048576 default bluefs_shared_alloc_size 65536 default bluestore_extent_map_inline_shard_prealloc_size 256 default bluestore_max_alloc_size 0 default bluestore_min_alloc_size 0 default bluestore_min_alloc_size_hdd 65536 default bluestore_min_alloc_size_ssd 16384 default rgw compression: no Secondary Zone: ceph version 14.2.12 zone parameters: "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "true", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" bucket stats: => "size_actual GiB": 65282.37313461304, "size_utilized GiB": 60779.72828538809, "num_objects": 23074921, "Compression Rate": 0.9310281683550253 pool stats: => "stored": 407816305115136, "objects": 118637638, "kb_used": 497822635396, "bytes_used": 509770378645504, "percent_used": 0.7275146245956421, "max_avail": 152744706965504 erasure-code-profile: crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 osd parameters: EC k=8 m=2 bluefs_alloc_size 1048576 default bluefs_shared_alloc_size 65536 default bluestore_extent_map_inline_shard_prealloc_size 256 default bluestore_max_alloc_size 0 default bluestore_min_alloc_size 0 default bluestore_min_alloc_size_hdd 65536 default bluestore_min_alloc_size_ssd 4096 default rgw compression: yes As you see, the secondary zone is using 408 TB vs 404 TB on master zone. Summing size_actual for each buckets gives only 65 TB vs 270 TB on master zone. Any idea about what could cause such a difference? Is it a known issue? The are some known issue with Space Overhead with EC Pools for alloc_size > 4 KiB, cf. : https://www.mail-archive.com/ceph-users@ceph.io/msg06191.html https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/OHPO43J54TPBEUISYC… https://www.spinics.net/lists/ceph-users/msg59587.html But our secondary zone is on nvme osd's with bluestore_min_alloc_size_ssd=4096, so that should be fine. I will also investigate further with "radosgw-admin bucket radoslist" and rgw-orphan-list. Thank you in advance for any help. And Happy New Year very soon ;-) Best Regards Francois PS: The value on master zone "size_utilized GiB": 17180102071.868008 is wrong. This is due to a bucket with wrong stats: { "bucket": "XXX", "tenant": "", "zonegroup": "d29ea82c-4f77-40af-952b-2ab0705ad268", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "cb1594b3-a782-49d0-a19f-68cd48870a63.3119460.1", "marker": "cb1594b3-a782-49d0-a19f-68cd48870a63.63841.1481", "index_type": "Normal", "owner": "e13d054f6e9c4eea881f687923d7d380", "ver": "0#373075,1#286613,2#290913,3#341069,4#360862,5#341416,6#279526,7#352172,8#255944,9#314797,10#317650,11#305698,12#289557,13#345344,14#345273,15#294708,16#241001,17#298577,18#274866,19#293952,20#307635,21#334606,22#265355,23#302567,24#277505,25#307278,26#266963,27#297452,28#332274,29#319133,30#361027,31#314294,32#282887,33#324849,34#278560,35#307506,36#287269,37#344789,38#345389,39#323814,40#386483,41#280319,42#358072,43#336651,44#339176,45#248079,46#356784,47#381496,48#295152,49#251661,50#318661,51#330530,52#263564,53#332005,54#332937,55#320163,56#300485,57#296138,58#343271,59#359351,60#295711,61#275751,62#332264,63#351532", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0", "mtime": "2020-06-03 12:13:01.207610Z", "max_marker": "0#,1#,2#,3#00000341068.137793799.5,4#00000360861.156871075.5,5#00000341415.97799619.5,6#00000279525.155275260.5,7#,8#,9#00000314796.95564320.5,10#,11#,12#00000289556.137784091.5,13#,14#,15#,16#00000241000.126242121.5,17#,18#00000274865.124884405.5,19#,20#00000307634.137793798.5,21#00000334605.93836734.5,22#,23#00000302566.125226103.5,24#,25#,26#,27#00000297451.125229375.5,28#,29#00000319132.155275278.5,30#00000361026.98341455.5,31#,32#,33#00000324848.126242117.5,34#00000278559.124884417.5,35#,36#,37#00000344788.125945123.5,38#00000345388.97796715.5,39#,40#00000386482.98341457.5,41#,42#00000358071.124884415.5,43#,44#00000339175.135084366.5,45#00000248078.155263912.5,46#00000356783.98341461.5,47#00000381495.94538350.5,48#00000295151.138701826.5,49#00000251660.137793803.5,50#00000318660.93848186.5,51#,52#,53#00000332004.126242119.5,54#00000332936.138701824.5,55#,56#00000300484.156871073.5,57#,58#00000343270.98341459.5,59#00000359350.94257302.5,60#,61#,62#,63#00000351531.135084368.5", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551604 }, "rgw.main": { "size": 22615692324233, "size_actual": 22617004142592, "size_utilized": 18446736471689615156, "size_kb": 22085637036, "size_kb_actual": 22086918108, "size_kb_utilized": 18014391085634390, "num_objects": 521927 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 977 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } This is another issue, probably not related to our sync issue. -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer(a)everyware.ch web: http://www.everyware.ch

2 2

Updating Git Submodules -- a documentation question
by John Zachary Dover 06 Jan '21

06 Jan '21

I'm working on some improvements to the Developer Guide. One of those is a section that explains more clearly to people new to Ceph how to know that your submodules are out of date and what to do when they are. The following location (inb4 I know that this is not the optimal place for this information, which is the whole point of this exercise) explains how to update submodules: https://docs.ceph.com/en/latest/install/clone-source/?highlight=update%20su… Sometimes in my local repo, I get the following messages after I run "$ git reset --hard ceph/master", and I'd like to understand what's going on here, and whether I should expect the submodule-updating procedure to remedy it: zdover@zdover-ThinkPad-T530:~/Documents/ceph/src$ git status On branch master Your branch is up to date with 'remotes/origin/master'. Untracked files: (use "git add <file>..." to include in what will be committed) lua/ nothing added to commit but untracked files present (use "git add" to track) Any insight into this matter is welcome. Thanks, Zac Dover Upstream Docs Ceph Foundation

1 0

logging to stdout/stderr causes huge container log file
by Tony Liu 05 Jan '21

05 Jan '21

Hi, With ceph 15.2.5 octopus, mon, mgd and rgw dump loggings on debug level to stdout/stderr. It causes huge container log file (/var/lib/docker/containers/<ID>/<ID>-json.log). Is there any way to stop dumping logs or change the logging level? BTW, I tried "ceph config set <service> log_to_stderr false". It doesn't help. Thanks! Tony

2 2

Timeout ceph rbd-nbd mounted image
by Wissem MIMOUNA 05 Jan '21

05 Jan '21

Hello , Looking for information about a timeout which occur once a week for a ceph rbd image mounted on a machine using rbd-nbd (Linux Ubuntu machine). The error found in 'dmseg' is below : [798016.401469] block nbd0: Connection timed out [798016.401506] block nbd0: shutting down sockets Many Thanks

2 4

RGW SSL key in config database
by Glen Baars 05 Jan '21

05 Jan '21

Hello Ceph Users, Is anyone successful in getting the SSL cert for Beast frontend into the config database? Octopus 15.2.8 tail -f /var/log/ceph/ceph-client.rgw.*.log 2021-01-05T18:38:35.008+1100 7f7cd6ac9100 1 radosgw_Main not setting numa affinity 2021-01-05T18:38:35.321+1100 7f7cd6ac9100 0 framework: beast 2021-01-05T18:38:35.321+1100 7f7cd6ac9100 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt 2021-01-05T18:38:35.321+1100 7f7cd6ac9100 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key 2021-01-05T18:38:35.321+1100 7f7cd6ac9100 0 starting handler: beast 2021-01-05T18:38:35.322+1100 7f7c9826d700 0 RGWReshardLock::lock failed to acquire lock on reshard.0000000000 ret=-16 2021-01-05T18:38:35.333+1100 7f7cd6ac9100 -1 ssl_private_key was not found: rgw/cert/OURREALM/OURZONE.key 2021-01-05T18:38:35.334+1100 7f7cd6ac9100 -1 ssl_private_key was not found: rgw/cert/OURREALM/OURZONE.crt 2021-01-05T18:38:35.334+1100 7f7cd6ac9100 -1 no ssl_certificate configured for ssl_port 2021-01-05T18:38:35.334+1100 7f7cd6ac9100 -1 ERROR: failed initializing frontend Certs are in the key store: ceph config-key ls | grep rgw/cert/OURREALM/OURZONE.key "rgw/cert/OURREALM/OURZONE ", ceph config-key ls | grep rgw/cert/OURREALM/OURZONE.crt "rgw/cert/OURREALM/OURZONE ", I don't seem to have any luck with many different options set. Glen This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.

1 0

Bluestore migration: per-osd device copy
by Chris Dunlop 04 Jan '21

04 Jan '21

Hi, The docs have scant detail on doing a migration to bluestore using a per-osd device copy: https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-o… This mentions "using the copy function of ceph-objectstore-tool", but ceph-objectstore-tool doesn't have a copy function (all the way from v9 to current). Has anyone actually tried doing this? Is there any further detail available on what is involved, e.g. a broad outline of the steps? Of course, detailed instructions would be even better, even if accompanied by "here be dragons!" warnings. Cheers, Chris

3 6

Compression of data in existing cephfs EC pool
by Paul Mezzanini 04 Jan '21

04 Jan '21

Hey everyone, I've got an EC pool as part of our cephfs for colder data. When we started using it, compression was still marked experimental. Since then it has become stable so I turned compression on to "aggressive". Using 'ceph df detail' I can see that new data is getting compressed. We have a good chunk of data (~1P) that hasn't been through the compression side of things that I would love to get squished. I know there currently isn't a way to do this in place and requires a new copy to be made so I made a simple script to "wiggle" the data. If the file is older than the date I turned compression on, and the file size is >64k: rsync the file to a new file sitting right next to the old one, delete the old one then move the new file into it's place. New inode = new file = compressed data! In theory, this works awesome. In practice however I'm not seeing the needles move as my script goes through. Does anyone have any ideas into what I may have missed for this dance? -paul Side note: People with file names that include quotes, pipes and spaces should get their accounts disabled by default. -- Paul Mezzanini Sr Systems Administrator / Engineer, Research Computing Information & Technology Services Finance & Administration Rochester Institute of Technology o:(585) 475-3245 | pfmeec(a)rit.edu CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information. ------------------------

4 6