January 2021 - ceph-users

DocuBetter Meeting This Week -- 13 Jan 2021 1730 UTC
by John Zachary Dover 11 Jan '21

11 Jan '21

Unless an unforeseen crisis arises, the DocuBetter meetings for the next two months will focus on ensuring that we have a smooth and easy-to-understand docs suite for the release of Pacific. Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

1 0

[cephadm] Point release minor updates block themselves infinitely
by Paul Browne 11 Jan '21

11 Jan '21

Hello all, I've been having some real troubles in getting cephadm to apply some very minor point release updates cleanly, twice now applying the point update of 15.2.6 -> 15.2.7 and 15.2.7 to 15.2.8 has gotten blocked somewhere and ended up making no progress, requiring digging deep into internals to unblock things. In the most recent attempt of 15.2.7 -> 15.2.8, the Orchestrator cleanly replaced Mon and MGR containers in the first steps, but when it came to replacing Crash daemon containers the running 15.2.7 Crash container was purged but container update operations then seem to get blocked on trying to start it again on the older image, leading to an infinite loop of Podman trying to start a non-existent container in logging; https://pastebin.com/9zdMs1XU Forcing an `ceph orch daemon rm` of the Crash daemon affected for the host just repeats the loop again. I'd then tried removing the Crash service and all daemons through the Orchestrator API next. This purged all running Crash containers from all hosts, and then re-applyed a service spec to restart them, hopefully on the new image. The Orchestrator removal of the Crash containers seems to have left container state dangling on hosts however, as now we see the same issue of Crash containers not starting on *every* host in the cluster due to left-over container state ; https://pastebin.com/tjaegxqg At this point I'm not certain if Podman (v1.6.4 EPEL, CentOS7.9) or Orchestrator is to blame for leaving this state dangling and blocking new container creation, but it's proving a real problem in applying even simple minor version point updates. Has anyone else been seeing similar behaviour in applying minor version updates via cephadm+Orchestrator? Are there any good workarounds to clean up the dangling container state? -- ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk Tel: 0044-1223-746548 *******************

1 1

osd gradual reweight question
by mj 11 Jan '21

11 Jan '21

Hi, We are replacing HDD with SSD, and we first (gradually) drain (reweight) the HDDs with 0.5 steps until 0 = empty. Works perfectly. Then (just for kicks) I tried reducing HDD weight from 3.6 to 0 in one large step. That seemed to have had more impact on the cluster, and we even noticed some OSD's temporarily go down after a few minutes. It all worked out, but the impact seemed much larger. We never had OSDs go down when gradually reducing the weight step by step. This surprised us. Is it expected that the impact of a sudden reweight from 3.6 to 0 is bigger than a gradual step-by-step decrease? I would assume the impact to be similar, only the time it takes to reach HEALTH_OK to be longer. Thanks, MJ

3 3

performance impact by pool deletion?
by Scheurer François 11 Jan '21

11 Jan '21

Hi everybody Does somebody had experience with important performance degradations during a pool deletion? We are asking because we are going to delete a 370 TiB with 120 M objects and have never done this in the past. The pool is using erasure coding 8+2 on nvme ssd's with rocksdb/wal on nvme optane disks. Openstack VM's are running on the other rbd pools. Thank you in advance for your feedback! Cheers Francois PS: this is no option, as it take about 100 year to complete ;-) : rados -p ch-zh1-az1.rgw.buckets.data ls | while read i; do rados -p ch-zh1-az1.rgw.buckets.data rm "$i"; done -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer(a)everyware.ch web: http://www.everyware.ch

4 5

Snaptrim making cluster unusable
by Pascal Ehlert 11 Jan '21

11 Jan '21

Hi all, We are running a small cluster with three nodes and 6-8 OSDs each. The OSDs are SSDs with sizes from 2 to 4 TB. Crush map is configured so all data is replicated to each node. The Ceph version is Ceph 15.2.6. Today I deleted 4 Snapshots of the same two 400GB and 500GB rbd volumes. Shortly after issuing the delete, I noticed the cluster became unresponsive to an extend where almost all our services went down due high IO latency. After a while, I noticed about 20 active snaptrim tasks + another 200 or so snaptrim_wait. I tried setting osd_snap_trim_sleep to 3, osd_pg_max_concurrent_snap_trims to 1 rbd_balance_snap_reads to true, rbd_localize_snap_reads to true Still the only way to make the cluster responsive again was to set osd_pg_max_concurrent_snap_trims to 0 and thus disable snaptrimming. I tried a few other options, but whenever snaptrims are running for a significant number of PGs, the cluster becomes completely unusable. Are there any other options to throttle snaptrimming for that I haven't tried, yet? Thank you, Pascal

3 6

Nautils BlueStore OSDs will not start
by William Law 08 Jan '21

08 Jan '21

We have several OSDs that are crashing on start. We are running nautilus 14.2.16; here is the relevant bit of the log: -10> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _read left 0xf1000 len 0x1000 -9> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _read got 4096 -8> 2021-01-08 14:52:38.800 7feec5f27c00 10 bluefs _replay 0x10f000: txn(seq 5972194 len 0x27 crc 0x9783dfc6) -7> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _replay 0x10f000: op_file_update file(ino 68757 size 0x6041f mtime 2021-01-07 21:25:57.793664 allocated 100000 extents [1:0x17e00000~100000]) -6> 2021-01-08 14:52:38.800 7feec5f27c00 10 bluefs _read h 0x55abfb20a3c0 0x110000~1000 from file(ino 1 size 0x110000 mtime 0.000000 allocated 500000 extents [1:0x481700000~100000,1:0x7ad00000~400000]) -5> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _read left 0xf0000 len 0x1000 -4> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _read got 4096 -3> 2021-01-08 14:52:38.800 7feec5f27c00 10 bluefs _replay 0x110000: txn(seq 5972195 len 0x116 crc 0xec6cec7) -2> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _replay 0x110000: op_dir_link db/109668.log to 68759 -1> 2021-01-08 14:52:38.804 7feec5f27c00 -1 /build/ceph-14.2.16/src/os/bluestore/BlueFS.cc<http://BlueFS.cc>: In function 'int BlueFS::_replay(bool, bool)' thread 7feec5f27c00 time 2021-01-08 14:52:38.802560 /build/ceph-14.2.16/src/os/bluestore/BlueFS.cc<http://BlueFS.cc>: 1029: FAILED ceph_assert(file->fnode.ino) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55abefa51fba] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55abefa52195] 3: (BlueFS::_replay(bool, bool)+0x4fa5) [0x55abf006f8f5] 4: (BlueFS::mount()+0x229) [0x55abf006fd79] 5: (BlueStore::_open_bluefs(bool)+0x78) [0x55abeff57958] 6: (BlueStore::_open_db(bool, bool, bool)+0x8a3) [0x55abeff58e63] 7: (BlueStore::_open_db_and_around(bool)+0x44) [0x55abeff6a1a4] 8: (BlueStore::_mount(bool, bool)+0x584) [0x55abeffc0b64] 9: (OSD::init()+0x3f3) [0x55abefb01db3] 10: (main()+0x5214) [0x55abefa5acf4] 11: (__libc_start_main()+0xe7) [0x7feec279cbf7] 12: (_start()+0x2a) [0x55abefa8c72a] 0> 2021-01-08 14:52:38.808 7feec5f27c00 -1 *** Caught signal (Aborted) ** in thread 7feec5f27c00 thread_name:ceph-osd ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) It seems like a version of this: https://tracker.ceph.com/issues/45519 and maybe this https://tracker.ceph.com/issues/21087I haven't been able to get it to start with the stupid allocator. Here is the ceph.conf I've been using to try to get it to start; a thread seemed to indicate that increasing the bluefs_max_log_runway would help. [global] fsid =<redacted> mon initial members = <redacted> public_network = 10.210.20.0/22 mon_host = 10.210.20.21,10.210.20.22,10.210.20.23 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx #kernel doesn't support all features, so disable this #rbd default features = 3 [osd] #set snap trim priority to lowest, 1 osd_snap_trim_priority = 1 osd_recovery_op_priority = 8 osd-max-backfills = 16 osd-max-backfills = 1 #keep allocator commented out normally; defaults to bitmap but #may need stupid #inspired by https://tracker.ceph.com/issues/45519 bluestore_allocator = stupid bluefs_allocator = stupid debug_bluefs = 20/20 #set to 3x default 4194304 bluefs_max_log_runway = 12582912 The cluster is having issues and it is urgent we get these OSDs up. The DB is on a shared NVMe device (81G) and the disk is a 2.2TB 2.5 inch enterprise disk. I'll be very grateful for any assistance. Best, Will

1 0

Ceph orch syntax to create OSD on a partition?
by Marc Spencer 08 Jan '21

08 Jan '21

All, I acknowledge it’s not a fantastic idea, but I wish to create an OSD on a partition rather than an entire device. I can’t seem to find the answer in the documentation. Is there a way using ceph orch daemon add osd to identify targets other than a whole device? Thanks in advance. - m

2 1

Larger number of OSDs, cheroot, cherrypy, limits + containers == broken
by David Orman 08 Jan '21

08 Jan '21

Hi, We have a ceph 15.2.7 deployment using cephadm under podman w/ systemd. We've run into what we believe is: https://github.com/ceph/ceph-container/issues/1748 https://tracker.ceph.com/issues/47875 In our case, eventually the mgr container stops emitting output/logging. We are polling with external prometheus clusters, which is likely what triggers the issue, as it appears some amount of time after the container is spawned. Unfortunately, setting limits in the systemd service file for the mgr service on the host OS doesn't work, nor does modifying the unit.run file which is used to start the container under podman to include the --ulimit settings as suggested. Looking inside the container: lib/systemd/system/ceph-mgr@.service:LimitNOFILE=1048576 This prevents us from deploying medium to large ceph clusters, so I would argue it's a high priority bug that should not be closed, unless there is a workaround that works until EPEL 8 contains the fixed version of cheroot and the ceph containers include it. My understanding is this was fixed in cheroot 8.4.0: https://github.com/cherrypy/cheroot/issues/249 https://github.com/cherrypy/cheroot/pull/301 Thank you in advance for any suggestions, David

4 10

[cinder] Cinder & Ceph Integration Error: No Valid Backend
by SSelf＠performair.com 07 Jan '21

07 Jan '21

All; We're having problems with our Openstack/Ceph integration. The versions we're using are Ussuri & Nautilus. When trying to create a volume, the volume is created, though the status is stuck at 'ERROR'. This appears to be the most relevant line from the Cinder scheduler.log: 2021-01-07 14:00:38.473 140686 ERROR cinder.scheduler.flows.create_volume [req-f86556b5-cb2e-4b2d-b556-ed07e632289d 824c26c133b34d8b8e84a7acabbe6f91 a983323b5ffc47e18660794cd9344869 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available Here is the 'cinder.conf' from our Controller Node: [DEFAULT] # define own IP address my_ip = 10.0.80.40 log_dir = /var/log/cinder state_path = /var/lib/cinder auth_strategy = keystone enabled_backends = ceph glance_api_version = 2 debug = true # RabbitMQ connection info transport_url = rabbit://openstack:<password>@10.0.80.40:5672 enable_v3_api = True # MariaDB connection info [database] connection = mysql+pymysql://cinder:<password>@10.0.80.40/cinder # Keystone auth info [keystone_authtoken] www_authenticate_uri = http://10.0.80.40:5000 auth_url = http://10.0.80.40:5000 memcached_servers = 10.0.80.40:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = cinder password = <password> [oslo_concurrency] lock_path = $state_path/tmp [ceph] volume_driver = cinder.volume.drivers.rbd.RBDDriver volume_backend_name = ceph rbd_pool = rbd_os_volumes rbd_ceph_conf = /etc/ceph/463/ceph.conf rbd_flatten_volume_from_snapshot = false rbd_max_clone_depth = 5 rbd_store_chunk_size = 4 rados_connect_timeout = -1 rbd_user = cinder rbd_exclusive_cinder_pool = true backup_driver = cinder.backup.drivers.ceph backup_ceph_conf = /etc/ceph/300/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = rbd_os_backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true Does anyone have any ideas as to what is going wrong? Thank you, Stephen Self IT Manager Perform Air International sself(a)performair.com www.performair.com

1 1

safest way to remove a host from Mimic
by Adrian Nicolae 07 Jan '21

07 Jan '21

Hi guys, I need to remove a host (osd server) from my Ceph Mimic. First I started to remove every OSD drive one by one with 'ceph osd out' and then 'ceph osd purge'. After all the drives will be removed from the crush map, I will still have the host empty, without any drive, and with the crush weight = 0. What is the recommend way to remove the host from Ceph : - ceph-deploy purge <hostname> - ceph-deploy purgedata <hostname> - ceph osd crush remove <hostname> ? Thanks.

1 0

Storage down due to MON sync very slow
by Frank Schilder 06 Jan '21

06 Jan '21

Dear all, I had to restart one out of 3 MONs on an empty MON DB dir. It is in state syncing right now, but I'm not sure if there is any progress. The cluster is completely unresponsive even though I have 2 healthy MONs. Is there any way to sync the DB directory faster and/or without downtime? Thanks a lot! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 4

radosgw sync using more capacity on secondary than on master zone
by Scheurer François 06 Jan '21

06 Jan '21

Dear Ceph contributors While our (new) rgw secondary zone is doing the initial data sync from our master zone, we noticed that the reported capacity usage was getting higher than on primary zone: Master Zone: ceph version 14.2.5 zone parameters: "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "false", "sync_from": [], "redirect_zone": "" bucket stats: => "size_actual GiB": 269868.9823989868, "size_utilized GiB": 17180102071.868008, <= (cf. other issue below) "num_objects": 100218788, "Compression Rate": 63660.899148714125 pool stats: => "stored": 403834132234240, "objects": 191530416, "kb_used": 692891724288, "bytes_used": 709521125670912, "percent_used": 0.7570806741714478, "max_avail": 136595529269248 erasure-code-profile: crush-device-class=nvme crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 osd parameters: bluefs_alloc_size 1048576 default bluefs_shared_alloc_size 65536 default bluestore_extent_map_inline_shard_prealloc_size 256 default bluestore_max_alloc_size 0 default bluestore_min_alloc_size 0 default bluestore_min_alloc_size_hdd 65536 default bluestore_min_alloc_size_ssd 16384 default rgw compression: no Secondary Zone: ceph version 14.2.12 zone parameters: "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "true", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" bucket stats: => "size_actual GiB": 65282.37313461304, "size_utilized GiB": 60779.72828538809, "num_objects": 23074921, "Compression Rate": 0.9310281683550253 pool stats: => "stored": 407816305115136, "objects": 118637638, "kb_used": 497822635396, "bytes_used": 509770378645504, "percent_used": 0.7275146245956421, "max_avail": 152744706965504 erasure-code-profile: crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 osd parameters: EC k=8 m=2 bluefs_alloc_size 1048576 default bluefs_shared_alloc_size 65536 default bluestore_extent_map_inline_shard_prealloc_size 256 default bluestore_max_alloc_size 0 default bluestore_min_alloc_size 0 default bluestore_min_alloc_size_hdd 65536 default bluestore_min_alloc_size_ssd 4096 default rgw compression: yes As you see, the secondary zone is using 408 TB vs 404 TB on master zone. Summing size_actual for each buckets gives only 65 TB vs 270 TB on master zone. Any idea about what could cause such a difference? Is it a known issue? The are some known issue with Space Overhead with EC Pools for alloc_size > 4 KiB, cf. : https://www.mail-archive.com/ceph-users@ceph.io/msg06191.html https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/OHPO43J54TPBEUISYC… https://www.spinics.net/lists/ceph-users/msg59587.html But our secondary zone is on nvme osd's with bluestore_min_alloc_size_ssd=4096, so that should be fine. I will also investigate further with "radosgw-admin bucket radoslist" and rgw-orphan-list. Thank you in advance for any help. And Happy New Year very soon ;-) Best Regards Francois PS: The value on master zone "size_utilized GiB": 17180102071.868008 is wrong. This is due to a bucket with wrong stats: { "bucket": "XXX", "tenant": "", "zonegroup": "d29ea82c-4f77-40af-952b-2ab0705ad268", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "cb1594b3-a782-49d0-a19f-68cd48870a63.3119460.1", "marker": "cb1594b3-a782-49d0-a19f-68cd48870a63.63841.1481", "index_type": "Normal", "owner": "e13d054f6e9c4eea881f687923d7d380", "ver": "0#373075,1#286613,2#290913,3#341069,4#360862,5#341416,6#279526,7#352172,8#255944,9#314797,10#317650,11#305698,12#289557,13#345344,14#345273,15#294708,16#241001,17#298577,18#274866,19#293952,20#307635,21#334606,22#265355,23#302567,24#277505,25#307278,26#266963,27#297452,28#332274,29#319133,30#361027,31#314294,32#282887,33#324849,34#278560,35#307506,36#287269,37#344789,38#345389,39#323814,40#386483,41#280319,42#358072,43#336651,44#339176,45#248079,46#356784,47#381496,48#295152,49#251661,50#318661,51#330530,52#263564,53#332005,54#332937,55#320163,56#300485,57#296138,58#343271,59#359351,60#295711,61#275751,62#332264,63#351532", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0", "mtime": "2020-06-03 12:13:01.207610Z", "max_marker": "0#,1#,2#,3#00000341068.137793799.5,4#00000360861.156871075.5,5#00000341415.97799619.5,6#00000279525.155275260.5,7#,8#,9#00000314796.95564320.5,10#,11#,12#00000289556.137784091.5,13#,14#,15#,16#00000241000.126242121.5,17#,18#00000274865.124884405.5,19#,20#00000307634.137793798.5,21#00000334605.93836734.5,22#,23#00000302566.125226103.5,24#,25#,26#,27#00000297451.125229375.5,28#,29#00000319132.155275278.5,30#00000361026.98341455.5,31#,32#,33#00000324848.126242117.5,34#00000278559.124884417.5,35#,36#,37#00000344788.125945123.5,38#00000345388.97796715.5,39#,40#00000386482.98341457.5,41#,42#00000358071.124884415.5,43#,44#00000339175.135084366.5,45#00000248078.155263912.5,46#00000356783.98341461.5,47#00000381495.94538350.5,48#00000295151.138701826.5,49#00000251660.137793803.5,50#00000318660.93848186.5,51#,52#,53#00000332004.126242119.5,54#00000332936.138701824.5,55#,56#00000300484.156871073.5,57#,58#00000343270.98341459.5,59#00000359350.94257302.5,60#,61#,62#,63#00000351531.135084368.5", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551604 }, "rgw.main": { "size": 22615692324233, "size_actual": 22617004142592, "size_utilized": 18446736471689615156, "size_kb": 22085637036, "size_kb_actual": 22086918108, "size_kb_utilized": 18014391085634390, "num_objects": 521927 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 977 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } This is another issue, probably not related to our sync issue. -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer(a)everyware.ch web: http://www.everyware.ch

2 2

Updating Git Submodules -- a documentation question
by John Zachary Dover 06 Jan '21

06 Jan '21

I'm working on some improvements to the Developer Guide. One of those is a section that explains more clearly to people new to Ceph how to know that your submodules are out of date and what to do when they are. The following location (inb4 I know that this is not the optimal place for this information, which is the whole point of this exercise) explains how to update submodules: https://docs.ceph.com/en/latest/install/clone-source/?highlight=update%20su… Sometimes in my local repo, I get the following messages after I run "$ git reset --hard ceph/master", and I'd like to understand what's going on here, and whether I should expect the submodule-updating procedure to remedy it: zdover@zdover-ThinkPad-T530:~/Documents/ceph/src$ git status On branch master Your branch is up to date with 'remotes/origin/master'. Untracked files: (use "git add <file>..." to include in what will be committed) lua/ nothing added to commit but untracked files present (use "git add" to track) Any insight into this matter is welcome. Thanks, Zac Dover Upstream Docs Ceph Foundation

1 0

logging to stdout/stderr causes huge container log file
by Tony Liu 05 Jan '21

05 Jan '21

Hi, With ceph 15.2.5 octopus, mon, mgd and rgw dump loggings on debug level to stdout/stderr. It causes huge container log file (/var/lib/docker/containers/<ID>/<ID>-json.log). Is there any way to stop dumping logs or change the logging level? BTW, I tried "ceph config set <service> log_to_stderr false". It doesn't help. Thanks! Tony

2 2

Timeout ceph rbd-nbd mounted image
by Wissem MIMOUNA 05 Jan '21

05 Jan '21

Hello , Looking for information about a timeout which occur once a week for a ceph rbd image mounted on a machine using rbd-nbd (Linux Ubuntu machine). The error found in 'dmseg' is below : [798016.401469] block nbd0: Connection timed out [798016.401506] block nbd0: shutting down sockets Many Thanks

2 4

RGW SSL key in config database
by Glen Baars 05 Jan '21

05 Jan '21

Hello Ceph Users, Is anyone successful in getting the SSL cert for Beast frontend into the config database? Octopus 15.2.8 tail -f /var/log/ceph/ceph-client.rgw.*.log 2021-01-05T18:38:35.008+1100 7f7cd6ac9100 1 radosgw_Main not setting numa affinity 2021-01-05T18:38:35.321+1100 7f7cd6ac9100 0 framework: beast 2021-01-05T18:38:35.321+1100 7f7cd6ac9100 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt 2021-01-05T18:38:35.321+1100 7f7cd6ac9100 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key 2021-01-05T18:38:35.321+1100 7f7cd6ac9100 0 starting handler: beast 2021-01-05T18:38:35.322+1100 7f7c9826d700 0 RGWReshardLock::lock failed to acquire lock on reshard.0000000000 ret=-16 2021-01-05T18:38:35.333+1100 7f7cd6ac9100 -1 ssl_private_key was not found: rgw/cert/OURREALM/OURZONE.key 2021-01-05T18:38:35.334+1100 7f7cd6ac9100 -1 ssl_private_key was not found: rgw/cert/OURREALM/OURZONE.crt 2021-01-05T18:38:35.334+1100 7f7cd6ac9100 -1 no ssl_certificate configured for ssl_port 2021-01-05T18:38:35.334+1100 7f7cd6ac9100 -1 ERROR: failed initializing frontend Certs are in the key store: ceph config-key ls | grep rgw/cert/OURREALM/OURZONE.key "rgw/cert/OURREALM/OURZONE ", ceph config-key ls | grep rgw/cert/OURREALM/OURZONE.crt "rgw/cert/OURREALM/OURZONE ", I don't seem to have any luck with many different options set. Glen This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.

1 0

Bluestore migration: per-osd device copy
by Chris Dunlop 05 Jan '21

05 Jan '21

Hi, The docs have scant detail on doing a migration to bluestore using a per-osd device copy: https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-o… This mentions "using the copy function of ceph-objectstore-tool", but ceph-objectstore-tool doesn't have a copy function (all the way from v9 to current). Has anyone actually tried doing this? Is there any further detail available on what is involved, e.g. a broad outline of the steps? Of course, detailed instructions would be even better, even if accompanied by "here be dragons!" warnings. Cheers, Chris

3 6

Compression of data in existing cephfs EC pool
by Paul Mezzanini 04 Jan '21

04 Jan '21

Hey everyone, I've got an EC pool as part of our cephfs for colder data. When we started using it, compression was still marked experimental. Since then it has become stable so I turned compression on to "aggressive". Using 'ceph df detail' I can see that new data is getting compressed. We have a good chunk of data (~1P) that hasn't been through the compression side of things that I would love to get squished. I know there currently isn't a way to do this in place and requires a new copy to be made so I made a simple script to "wiggle" the data. If the file is older than the date I turned compression on, and the file size is >64k: rsync the file to a new file sitting right next to the old one, delete the old one then move the new file into it's place. New inode = new file = compressed data! In theory, this works awesome. In practice however I'm not seeing the needles move as my script goes through. Does anyone have any ideas into what I may have missed for this dance? -paul Side note: People with file names that include quotes, pipes and spaces should get their accounts disabled by default. -- Paul Mezzanini Sr Systems Administrator / Engineer, Research Computing Information & Technology Services Finance & Administration Rochester Institute of Technology o:(585) 475-3245 | pfmeec(a)rit.edu CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information. ------------------------

4 6

What is the specific meaning "total_time" in RGW ops log
by opengers 04 Jan '21

04 Jan '21

Hello everyone，I enabled rgw ops log by setting "rgw_enable_ops_log = true"，There is a "total_time" field in rgw ops log But I want to figure out whether "total_time" includes the period of time when rgw returns a response to the client?

2 2

Sequence replacing a failed OSD disk?
by Rainer Krienke 04 Jan '21

04 Jan '21

Hello, hope you had a nice Xmas and I wish all of you a good and happy new year in advance... Yesterday my ceph nautilus 14.2.15 cluster had a disk with unreadable sectors, after several tries the OSD was marked down and rebalancing started and has also finished successfully. ceph osd stat shows the osd now as "autoout,exists". Usually the steps to replace a failed disk are: 1. Destroy the failed OSD: ceph osd destroy {id} 2. run ceph-volume lvm create --bluestore --osd-id {id} --data /dev/sdX ... with a new disk in place to recreate a OSD with the same id without the need to change the crushmap or auth info etc. Now I still wait for a new disk and I am a unsure if I should run the destroy-command already now to keep ceph from trying to reactivate the broken osd? Then I would wait until the disk has arrived in a day or so and then use ceph volume to create a new osd? Or should I leave the state as it is now until the disk has arrived and then run both steps (destroy, volume ceph-lvm-create) one right after the other? Do the two slightly different ways make any difference if for example a power failure would result in a reboot of the node with the failed OSD before I could replace the broken disk? Any comments on this? Thanks Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312

2 1

Re: kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production)
by Marc Roos 04 Jan '21

04 Jan '21

Hi Eugen, Indeed some really useful tips explaining what goes wrong, yet this thread[1] is about cephfs directly mounted on the osd node. I was having this also quite some time without any problems until suddenly I ran into the same issue as they had. I think I did not have any issues with the kernel client cephfs mount with luminous until I enabled the cephfs snapshots. Then I had to switch to the fuse client. In my case I am running a vm on the the osd node, which I thought would be different. I have been able to reproduce this stale mount just 2 times now. I have been testing with 10x more clients and it still works. Anyway I decided to move everything to rbd. I have been running vm's with rbd images without problems colocated on osd nodes for quite some time. I really would like to use the hosts because they have each 16c/32t and an average load of just 2-3. Unfortunately I did not document precisely how I recovered from the stale mount. I would like to see if I can reduce the amount of steps to take. Things started happening for me after I did the mds failover. Then I got blocked clients that I could unblock and fix the mount with mount -l Thanks for the pointers, I have linked them in my docs ;) -----Original Message----- To: ceph-users(a)ceph.io Subject: [ceph-users] Re: kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production) Hi, there have been several threads about hanging cephfs mounts, one quite long thread [1] describes a couple of debugging options but also mentions to avoid mounting cephfs on OSD nodes in a production environment. Do you see blacklisted clients with 'ceph osd blacklist ls'? If the answer is yes try to unblock that client [2]. The same option ('umount -l') is available on a cephfs client, you can try that, too. Other options described in [1] are to execute an MDS failover, but sometimes a reboot of that VM is the only solution left. Regards, Eugen [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028719.html [2] https://docs.ceph.com/en/latest/cephfs/eviction/#advanced-un-blocklisting-a… Zitat von Marc Roos <M.Roos(a)f1-outsourcing.eu>: > Is there not some genius out there that can shed a ligth on this? ;) > Currently I am not able to reproduce this. Thus it would be nice to > have some procedure at hand that resolves stale cephfs mounts nicely. > > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] kvm vm cephfs mount hangs on osd node (something > like umount -l available?) (help wanted going to production) > > > > I have a vm on a osd node (which can reach host and other nodes via > the macvtap interface (used by the host and guest)). I just did a > simple > bonnie++ test and everything seems to be fine. Yesterday however the > dovecot procces apparently caused problems (only using cephfs for an > archive namespace, inbox is on rbd ssd, fs meta also on ssd) > > How can I recover from such lock-up. If I have a similar situation > with an nfs-ganesha mount, I have the option to do a umount -l, and > clients recover quickly without any issues. > > Having to reset the vm, is not really an option. What is best way to > resolve this? > > > > Ceph cluster: 14.2.11 (the vm has 14.2.16) > > I have in my ceph.conf nothing special, these 2x in the mds section: > > mds bal fragment size max = 120000 > # maybe for nfs-ganesha problems? > # http://docs.ceph.com/docs/master/cephfs/eviction/ > #mds_session_blacklist_on_timeout = false > #mds_session_blacklist_on_evict = false mds_cache_memory_limit = > 17179860387 > > > All running: > CentOS Linux release 7.9.2009 (Core) > Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC > 2020 x86_64 x86_64 x86_64 GNU/Linux > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Data migration between clusters
by Szabo, Istvan (Agoda) 04 Jan '21

04 Jan '21

What is the easiest and best way to migrate bucket from an old cluster to a new one? Luminous to octopus not sure does it matter from the data perspective. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

Re: OSD reboot loop after running out of memory
by Stefan Wild 02 Jan '21

02 Jan '21

Of course, I forgot to mention that. Thank you for bringing it up! We made sure balancer and PG autoscaler were turned off for the (only) pool that uses those PGs shortly after we noticed the cycle of remapping/backfilling: # ceph balancer status { "active": false, "last_optimize_duration": "", "last_optimize_started": "", "mode": "none", "optimize_result": "", "plans": [] } # ceph osd pool get avl1.rgw.buckets.data pg_autoscale_mode pg_autoscale_mode: off Thanks, Stefan On 1/1/21, 11:23 PM, "Anthony D'Atri" <anthony.datri(a)gmail.com> wrote: I have to ask if this might be the balancer or the PG autoscaler at work > On Jan 1, 2021, at 7:15 PM, Stefan Wild <swild(a)tiltworks.com> wrote: > > Our setup is not using SSDs as the Bluestore DB devices. We only have 2 SSDs vs 12 HDDs, which is normally fine for the low workload of the cluster. The SSDs are serving a pool that is just used by RGW for index and meta. > > Since the compaction two weeks ago the OSDs have all been stable. However, besides some other minor issues the cluster now keeps remapping erasure coded PGs with identical OSDs just in different order. Ceph will remap 11 (out of 128) PGs, then slowly backfill them and the second it's done, it'll pick another 11 PGs and remap those. I had to set osd_max_backfills to 0 in order to get any scrubbing/repair in. Not sure how to stop the constant cycle of remapping/backfilling. > > Thanks, > Stefan > > > > > On 12/16/20, 4:24 AM, "Frédéric Nass" <frederic.nass(a)univ-lorraine.fr> wrote: > > Regarding RocksDB compaction, if you were in a situation were RocksDB > had overspilled to HDDs (if your cluster is using an hybrid setup), the > compaction should have move the bits back to fast devices. So it might > have helped in this situation too. > > Regards, > > Frédéric. > > Le 16/12/2020 à 09:57, Frédéric Nass a écrit : >> Hi Sefan, >> >> This has me thinking that the issue your cluster may be facing is >> probably with bluefs_buffered_io set to true, as this has been >> reported to induce excessive swap usage (and OSDs flapping or OOMing >> as consequences) in some versions starting from Nautilus I believe. >> >> Can you check the value of bluefs_buffered_io that OSDs are currently >> using ? : ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config >> show | grep bluefs_buffered_io >> >> Can you check the kernel value of vm.swappiness ? : sysctl >> vm.swappiness (default value is 30) >> >> And describe your OSD nodes ? # of HDDS and SSDs/NVMes and HDD/SSD >> ratio, and how much memory they have ? >> >> You should be able to avoid swap usage by setting bluefs_buffered_io >> to false but your cluster / workload might not allow that performance >> and stability wise. >> Or you may be able to workaround the excessive swap usage (when >> bluefs_buffered_io is set to true) by lowering vm.swappiness or >> disabling the swap. >> >> Regards, >> >> Frédéric. >> >> Le 14/12/2020 à 22:12, Stefan Wild a écrit : >>> Hi Frédéric, >>> >>> Thanks for the additional input. We are currently only running RGW on >>> the cluster, so no snapshot removal, but there have been plenty of >>> remappings with the OSDs failing (all of them at first during and >>> after the OOM incident, then one-by-one). I haven't had a chance to >>> look into or test the bluefs_buffered_io setting, but will do that >>> next. Initial results from compacting all OSDs' RocksDBs look >>> promising (thank you, Igor!). Things have been stable for the past >>> two hours, including the two OSDs with issues (one in reboot loop, >>> the other with some heartbeats missed), while 15 degraded PGs are >>> backfilling. >>> >>> The ballooning of each OSD to over 15GB memory right after the >>> initial crash was even with osd_memory_target set to 2GB. The only >>> thing that helped at that point was to temporarily add enough swap >>> space to fit 12 x 15GB and let them do their thing. Once they had all >>> booted, memory usage went back down to normal levels. >>> >>> I will report back here with more details when the cluster is >>> hopefully back to a healthy state. >>> >>> Thanks, >>> Stefan >>> >>> >>> >>> On 12/14/20, 3:35 PM, "Frédéric Nass" >>> <frederic.nass(a)univ-lorraine.fr> wrote: >>> >>> Hi Stefan, >>> >>> Initial data removal could also have resulted from a snapshot >>> removal >>> leading to OSDs OOMing and then pg remappings leading to more >>> removals >>> after OOMed OSDs rejoined the cluster and so on. >>> >>> As mentioned by Igor : "Additionally there are users' reports that >>> recent default value's modification for bluefs_buffered_io >>> setting has >>> negative impact (or just worsen existing issue with massive >>> removal) as >>> well. So you might want to switch it back to true." >>> >>> We're some of them. Our cluster suffered from a severe >>> performance drop >>> during snapshot removal right after upgrading to Nautilus, due to >>> bluefs_buffered_io being set to false by default, with slow >>> requests >>> observed around the cluster. >>> Once back to true (can be done with ceph tell osd.* injectargs >>> '--bluefs_buffered_io=true') snap trimming would be fast again >>> so as >>> before the upgrade, with no more slow requests. >>> >>> But of course we've seen the excessive memory swap usage >>> described here >>> : https://github.com/ceph/ceph/pull/34224 >>> So we lower osd_memory_target from 8MB to 4MB and haven't >>> observed any >>> swap usage since then. You can also have a look here : >>> https://github.com/ceph/ceph/pull/38044 >>> >>> What you need to look at to understand if your cluster would >>> benefit >>> from changing bluefs_buffered_io back to true is the %util of your >>> RocksDBD devices on an iostat. Run an iostat -dmx 1 /dev/sdX (if >>> you're >>> using SSD RocksDB devices) and look at the %util of the device with >>> bluefs_buffered_io=false and with bluefs_buffered_io=true. If with >>> bluefs_buffered_io=false, the %util is over 75% most of the >>> time, then >>> you'd better change it to true. :-) >>> >>> Regards, >>> >>> Frédéric. >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

OSD reboot loop after running out of memory
by Stefan Wild 02 Jan '21

02 Jan '21

Hi, We recently upgraded a cluster from 15.2.1 to 15.2.5. About two days later, one of the server ran out of memory for unknown reasons (normally the machine uses about 60 out of 128 GB). Since then, some OSDs on that machine get caught in an endless restart loop. Logs will just mention system seeing the daemon fail and then restarting it. Since the out of memory incident, we’ve have 3 OSDs fail this way at separate times. We resorted to wiping the affected OSD and re-adding it to the cluster, but it seems as soon as all PGs have moved to the OSD, the next one fails. This is also keeping us from re-deploying RGW, which was affected by the same out of memory incident, since cephadm runs a check and won’t deploy the service unless the cluster is in HEALTH_OK status. Any help would be greatly appreciated. Thanks, Stefan

5 17