- ceph-users - lists.ceph.io

Stop logging ceph-mgr every 2s
by Marc Roos 31 Mar '20

31 Mar '20

How to get rid of this logging?? Mar 31 13:40:03 c01 ceph-mgr: 2020-03-31 13:40:03.521 7f554edc8700 0 log_channel(cluster) log [DBG] : pgmap v672067: 384 pgs: 384 active+clean;

1 0

[ceph][radosgw] nautilus multisite problems
by Ignazio Cassano 31 Mar '20

31 Mar '20

Hello, I have configured a multisite ceph. The master zone has not changed but on the destination zone I had some problems. On the destination zone I cleaned and reinstalled the radosgw, but trying to assing the same zone name it had before reinstallation does not work (radosgw does not start). I changed its zone name and radosgw starts, but the master is using the old name when I try to execute: radosgw-admin period update commit Please, how can I solve this problem ? Thanks Ignazi

2 2

samba ceph-vfs and scrubbing interval
by Marco Savoca 31 Mar '20

31 Mar '20

Hi all, i‘m running a 3 node ceph cluster setup with collocated mons and mds for actually 3 filesystems at home since mimic. I’m planning to downgrade to one FS and use RBD in the future, but this is another story. I’m using the cluster as cold storage on spindles with EC-pools for archive purposes. The cluster usually does not run 24/7. I actually managed to upgrade to octopus without problems yesterday. So first of all: great job with the release. Now I have a little problem and a general question to address. I have tried to share the CephFS via samba and the ceph-vfs module but I could not manage to get write access (read access is not a problem) to the share (even with the admin key). When I share the mounted path (kernel module or fuser mount) instead as usual there are no problems at all. Is ceph-vfs generally read only and I missed this point? Furthermore I suppose, that there is no possibility to choose between the different mds namespaces, right? Now the general question. Since the cluster does not run 24/7 as stated and is turned on perhaps once a week for a couple of hours on demand, what are reasonable settings for the scrubbing intervals? As I said, the storage is cold and there is mostly read i/o. The archiving process adds approximately 0.5 % of new data of the cluster’s total storage capacity. Stay healthy and regards, Marco Savoca

3 3

Unable to use iscsi gateway with https | iscsi-gateway-add returns errors
by givemeone 31 Mar '20

31 Mar '20

Hi all, I am installing ceph Nautilus and getting constantly errors while adding iscsi gateways It was working using http schema but after moving to https with wildcard certs gives API errors Below some of my configurations Thanks for your help Command: ceph --cluster ceph dashboard iscsi-gateway-add https://myadmin:admin.01@1.2.3.4:5050 Error: Error EINVAL: iscsi REST API cannot be reached. Please check your configuration and that the API endpoint is accessible Tried also disabling ssl verify # ceph dashboard set-rgw-api-ssl-verify False Option RGW_API_SSL_VERIFY updated "/etc/ceph/iscsi-gateway.cfg" 23L, 977C # Ansible managed [config] api_password = admin.01 api_port = 5050 # API settings. # The API supports a number of options that allow you to tailor it to your # local environment. If you want to run the API under https, you will need to # create cert/key files that are compatible for each iSCSI gateway node, that is # not locked to a specific node. SSL cert and key files *must* be called # 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory # on *each* gateway node. With the SSL files in place, you can use 'api_secure = true' # to switch to https mode. # To support the API, the bear minimum settings are: api_secure = True # Optional settings related to the CLI/API service api_user = myadmin cluster_name = ceph loop_delay = 1 trusted_ip_list = 1.2.3.3,1.2.3.4 Log file ====== ceph-rgw-cnode04.rgw0.log 2020-03-30 10:24:20.392 7f6a2dc1b700 1 ====== req done req=0x561d9ce465f0 op status=0 http_status=200 latency=0.0119993s ====== 2020-03-30 10:24:20.394 7f6a2cc19700 1 ====== starting new request req=0x561d9ce465f0 ===== 2020-03-30 10:24:20.396 7f6a2cc19700 1 ====== req done req=0x561d9ce465f0 op status=0 http_status=404 latency=0.00199988s ====== 2020-03-30 10:24:20.397 7f6a2bc17700 1 ====== starting new request req=0x561d9ce465f0 ===== 2020-03-30 10:24:20.410 7f6a2bc17700 1 ====== req done req=0x561d9ce465f0 op status=0 http_status=200 latency=0.0129992s ====== 2020-03-30 10:24:20.499 7f6a27c0f700 1 ====== starting new request req=0x561d9cec25f0 ===== 2020-03-30 10:24:20.502 7f6a27c0f700 1 ====== req done req=0x561d9cec25f0 op status=0 http_status=200 latency=0.00299982s ====== 2020-03-30 10:24:20.504 7f6a2740e700 1 ====== starting new request req=0x561d9cec25f0 ===== 2020-03-30 10:24:20.506 7f6a2740e700 1 ====== req done req=0x561d9cec25f0 op status=0 http_status=200 latency=0.00199988s ====== 2020-03-30 10:24:30.516 7f6a22404700 1 ====== starting new request req=0x561d9cf825f0 ===== 2020-03-30 10:24:30.518 7f6a22404700 1 ====== req done req=0x561d9cf825f0 op status=0 http_status=200 latency=0.00199988s ====== 2020-03-30 10:24:30.620 7f6a1ebfd700 1 ====== starting new request req=0x561d9cf925f0 ===== 2020-03-30 10:24:30.622 7f6a1ebfd700 1 ====== req done req=0x561d9cf925f0 op status=0 http_status=200 latency=0.00199988s ====== 2020-03-30 10:24:30.708 7f6a19bf3700 1 ====== starting new request req=0x561d9cfd45f0 ===== 2020-03-30 10:24:30.708 7f6a193f2700 1 ====== starting new request req=0x561d9cfaa5f0 ===== 2020-03-30 10:24:30.710 7f6a19bf3700 1 ====== req done req=0x561d9cfd45f0 op status=0 http_status=200 latency=0.00199988s ====== 2020-03-30 10:24:30.711 7f6a193f2700 1 ====== req done req=0x561d9cfaa5f0 op status=0 http_status=200 latency=0.00299982s ====== /ceph-rgw-cnode05.rgw0.log 2020-03-30 10:07:41.309 7fb79d31c700 1 ====== req done http_status=400 ====== 2020-03-30 10:07:41.505 7fb798312700 1 ====== starting new request req=0x5565d88b45f0 ===== 2020-03-30 10:07:41.508 7fb798312700 1 ====== req done req=0x5565d88b45f0 op status=0 http_status=200 latency=0.00299982s ====== 2020-03-30 10:07:41.531 7fb79430a700 1 failed to read header: bad method 2020-03-30 10:07:41.531 7fb79430a700 1 ====== req done http_status=400 ====== 2020-03-30 10:07:41.552 7fb791304700 1 failed to read header: bad method 2020-03-30 10:07:41.552 7fb791304700 1 ====== req done http_status=400 ====== (END)

3 2

ceph cephadm generate-key => No such file or directory: '/tmp/tmp4ejhr7wh/key'
by Ml Ml 30 Mar '20

30 Mar '20

Hello List, is this a bug? root@ceph02:~# ceph cephadm generate-key Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 1413, in _generate_key with open(path, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp4ejhr7wh/key' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 110, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 308, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 72, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/cephadm/module.py", line 1418, in _generate_key os.unlink(path) FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp4ejhr7wh/key' root@ceph02:~# dpkg -l |grep ceph ii ceph-base 15.2.0-1~bpo10+1 amd64 common ceph daemon libraries and management tools ii ceph-common 15.2.0-1~bpo10+1 amd64 common utilities to mount and interact with a ceph storage cluster ii ceph-deploy 2.0.1 all Ceph-deploy is an easy to use configuration tool ii ceph-mds 15.2.0-1~bpo10+1 amd64 metadata server for the ceph distributed file system ii ceph-mgr 15.2.0-1~bpo10+1 amd64 manager for the ceph distributed storage system ii ceph-mgr-cephadm 15.2.0-1~bpo10+1 all cephadm orchestrator module for ceph-mgr ii ceph-mgr-dashboard 15.2.0-1~bpo10+1 all dashboard module for ceph-mgr ii ceph-mgr-diskprediction-cloud 15.2.0-1~bpo10+1 all diskprediction-cloud module for ceph-mgr ii ceph-mgr-diskprediction-local 15.2.0-1~bpo10+1 all diskprediction-local module for ceph-mgr ii ceph-mgr-k8sevents 15.2.0-1~bpo10+1 all kubernetes events module for ceph-mgr ii ceph-mgr-modules-core 15.2.0-1~bpo10+1 all ceph manager modules which are always enabled ii ceph-mgr-rook 15.2.0-1~bpo10+1 all rook module for ceph-mgr ii ceph-mon 15.2.0-1~bpo10+1 amd64 monitor server for the ceph storage system ii ceph-osd 15.2.0-1~bpo10+1 amd64 OSD server for the ceph storage system ii cephadm 15.2.0-1~bpo10+1 amd64 cephadm utility to bootstrap ceph daemons with systemd and containers ii libcephfs1 10.2.11-2 amd64 Ceph distributed file system client library ii libcephfs2 15.2.0-1~bpo10+1 amd64 Ceph distributed file system client library ii python-ceph-argparse 14.2.8-1 all Python 2 utility libraries for Ceph CLI ii python3-ceph-argparse 15.2.0-1~bpo10+1 all Python 3 utility libraries for Ceph CLI ii python3-ceph-common 15.2.0-1~bpo10+1 all Python 3 utility libraries for Ceph ii python3-cephfs 15.2.0-1~bpo10+1 amd64 Python 3 libraries for the Ceph libcephfs library root@ceph02:~# cat /etc/debian_version 10.3 Thanks, Michael

2 1

Re: How do I get a sector marked bad?
by Dan van der Ster 30 Mar '20

30 Mar '20

Hi, I have a feeling that the pg repair didn't actually run yet. Sometimes if the OSDs are busy scrubbing, the repair doesn't start when you ask it to. You can force it through with something like: ceph osd set noscrub ceph osd set nodeep-scrub ceph config set osd_max_scrubs 3 ceph pg repair <the pg> ceph status # and check that the repair really started ceph config set osd_max_scrubs 1 ceph osd unset nodeep-scrub ceph osd unset noscrub Once repair runs/completes, it will rewrite the inconsistent object replica (to a new place on the disk). Check your ceph.log to see when this happens. From my experience, the PendingSectors counter will not be decremented until that sector is written again (which will happen at some random point in the future when bluestore allocates some new data there). Hope that helps, Dan On Mon, Mar 30, 2020 at 9:00 AM David Herselman <dhe(a)syrex.co> wrote: > > Hi, > > We have a single inconsistent placement group where I then subsequently triggered a deep scrub and tried doing a 'pg repair'. The placement group remains in an inconsistent state. > > How do I discard the objects for this placement group only on the one OSD and get Ceph to essentially write the data out new. Drives will only mark a sector as remapped when asked to overwrite the problematic sector or repeated reads of the failed sector eventually succeed (this is my limited understanding). > > Nothing useful in the 'ceph pg 1.35 query' output that I could decipher. Then ran 'ceph pg deep-scrub 1.35' and 'rados list-inconsistent-obj 1.35' thereafter indicates a read error on one of the copies: > {"epoch":25776,"inconsistents":[{"object":{"name":"rbd_data.746f3c94fb3a42.000000000001e48d","nspace":"","locator":"","snap":"head","version":34866184},"errors":[],"union_shard_errors":["read_error"],"selected_object_info":{"oid":{"oid":"rbd_data.746f3c94fb3a42.000000000001e48d","key":"","snapid":-2,"hash":3814100149,"max":0,"pool":1,"namespace":""},"version":"22845'1781037","prior_version":"22641'1771494","last_reqid":"client.136837683.0:124047","user_version":34866184,"size":4194304,"mtime":"2020-03-08 17:59:00.159846","local_mtime":"2020-03-08 17:59:00.159670","lost":0,"flags":["dirty","data_digest","omap_digest"],"truncate_seq":0,"truncate_size":0,"data_digest":"0x031cb17c","omap_digest":"0xffffffff","expected_object_size":4194304,"expected_write_size":4194304,"alloc_hint_flags":0,"manifest":{"type":0},"watchers":{}},"shards":[{"osd":51,"primary":false,"errors":["read_error"],"size":4194304},{"osd":60,"primary":false,"errors":[],"size":4194304,"omap_digest":"0xffffffff","data_dig > est":"0x031cb17c"},{"osd":82,"primary":true,"errors":[],"size":4194304,"omap_digest":"0xffffffff","data_digest":"0x031cb17c"}]}]} > > /var/log/syslog: > Mar 30 08:40:40 kvm1e kernel: [74792.229021] ata2.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x0 > Mar 30 08:40:40 kvm1e kernel: [74792.230416] ata2.00: irq_stat 0x40000008 > Mar 30 08:40:40 kvm1e kernel: [74792.231715] ata2.00: failed command: READ FPDMA QUEUED > Mar 30 08:40:40 kvm1e kernel: [74792.233071] ata2.00: cmd 60/00:08:00:7a:50/04:00:c9:00:00/40 tag 1 ncq dma 524288 in > Mar 30 08:40:40 kvm1e kernel: [74792.233071] res 43/40:00:10:7b:50/00:04:c9:00:00/00 Emask 0x409 (media error) <F> > Mar 30 08:40:40 kvm1e kernel: [74792.235736] ata2.00: status: { DRDY SENSE ERR } > Mar 30 08:40:40 kvm1e kernel: [74792.237045] ata2.00: error: { UNC } > Mar 30 08:40:40 kvm1e ceph-osd[450777]: 2020-03-30 08:40:40.240 7f48a41f3700 -1 bluestore(/var/lib/ceph/osd/ceph-51) _do_read bdev-read failed: (5) Input/output error > Mar 30 08:40:40 kvm1e kernel: [74792.244914] ata2.00: configured for UDMA/133 > Mar 30 08:40:40 kvm1e kernel: [74792.244938] sd 1:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > Mar 30 08:40:40 kvm1e kernel: [74792.244942] sd 1:0:0:0: [sdb] tag#1 Sense Key : Medium Error [current] > Mar 30 08:40:40 kvm1e kernel: [74792.244945] sd 1:0:0:0: [sdb] tag#1 Add. Sense: Unrecovered read error - auto reallocate failed > Mar 30 08:40:40 kvm1e kernel: [74792.244949] sd 1:0:0:0: [sdb] tag#1 CDB: Read(16) 88 00 00 00 00 00 c9 50 7a 00 00 00 04 00 00 00 > Mar 30 08:40:40 kvm1e kernel: [74792.244953] blk_update_request: I/O error, dev sdb, sector 3377494800 op 0x0:(READ) flags 0x0 phys_seg 94 prio class 0 > Mar 30 08:40:40 kvm1e kernel: [74792.246238] ata2: EH complete > > > Regards > David Herselman > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Terrible IOPS performance
by Jarett DeAngelis 30 Mar '20

30 Mar '20

Hi folks, I have a three-node cluster on a 10G network with very little traffic. I have a six-OSD flash-only pool with two devices — a 1TB NVMe drive and a 256GB SATA SSD — on each node, and here’s how it benchmarks: Oof. How can I troubleshoot this? Anthony mentioned that I might be able to run more than one OSD on the NVMe — how is that done, and can I do it “on the fly” with the system already up and running like this? And, will more OSDs give me better IOPS? Thanks, Jarett

2 1

Move on cephfs not O(1)?
by Frank Schilder 30 Mar '20

30 Mar '20

Some time ago I made a surprising observation. I reorganised a directory structure and needed to move a folder one level up with a command like mv A/B/ B B contained something like 9TB in very large files. To my surprise, this command didn't return for a couple of minutes and I started to look what was going on. What I discovered was, that the mv command actually performed a full copy with a subsequent remove. I had to wait for several hours for the move to complete. I tried to reproduce this today to collect further information. However, this behaviour seems not reproducible. No matter what I try, mv completes almost instantly. I was running the original mv on mimic 13.2.2 and retried now with mimic 13.2.8. In addition, there was an OS upgrade from Centos 7.6 to 7.7. I'm using the kernel-ml versions (5.xxx). Only one cephfs mount was present at all times. My questions are: 1) Was there a change from 13.2.2 to 13.2.8 explaining this? 2) Are there (rare) conditions under which an mv on cephfs becomes a cp+rm? 3) Am I seeing ghosts? Thanks for clues and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

5 11

How do I get a sector marked bad?
by David Herselman 30 Mar '20

30 Mar '20

Hi, We have a single inconsistent placement group where I then subsequently triggered a deep scrub and tried doing a 'pg repair'. The placement group remains in an inconsistent state. How do I discard the objects for this placement group only on the one OSD and get Ceph to essentially write the data out new. Drives will only mark a sector as remapped when asked to overwrite the problematic sector or repeated reads of the failed sector eventually succeed (this is my limited understanding). Nothing useful in the 'ceph pg 1.35 query' output that I could decipher. Then ran 'ceph pg deep-scrub 1.35' and 'rados list-inconsistent-obj 1.35' thereafter indicates a read error on one of the copies: {"epoch":25776,"inconsistents":[{"object":{"name":"rbd_data.746f3c94fb3a42.000000000001e48d","nspace":"","locator":"","snap":"head","version":34866184},"errors":[],"union_shard_errors":["read_error"],"selected_object_info":{"oid":{"oid":"rbd_data.746f3c94fb3a42.000000000001e48d","key":"","snapid":-2,"hash":3814100149,"max":0,"pool":1,"namespace":""},"version":"22845'1781037","prior_version":"22641'1771494","last_reqid":"client.136837683.0:124047","user_version":34866184,"size":4194304,"mtime":"2020-03-08 17:59:00.159846","local_mtime":"2020-03-08 17:59:00.159670","lost":0,"flags":["dirty","data_digest","omap_digest"],"truncate_seq":0,"truncate_size":0,"data_digest":"0x031cb17c","omap_digest":"0xffffffff","expected_object_size":4194304,"expected_write_size":4194304,"alloc_hint_flags":0,"manifest":{"type":0},"watchers":{}},"shards":[{"osd":51,"primary":false,"errors":["read_error"],"size":4194304},{"osd":60,"primary":false,"errors":[],"size":4194304,"omap_digest":"0xffffffff","data_digest":"0x031cb17c"},{"osd":82,"primary":true,"errors":[],"size":4194304,"omap_digest":"0xffffffff","data_digest":"0x031cb17c"}]}]} /var/log/syslog: Mar 30 08:40:40 kvm1e kernel: [74792.229021] ata2.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x0 Mar 30 08:40:40 kvm1e kernel: [74792.230416] ata2.00: irq_stat 0x40000008 Mar 30 08:40:40 kvm1e kernel: [74792.231715] ata2.00: failed command: READ FPDMA QUEUED Mar 30 08:40:40 kvm1e kernel: [74792.233071] ata2.00: cmd 60/00:08:00:7a:50/04:00:c9:00:00/40 tag 1 ncq dma 524288 in Mar 30 08:40:40 kvm1e kernel: [74792.233071] res 43/40:00:10:7b:50/00:04:c9:00:00/00 Emask 0x409 (media error) <F> Mar 30 08:40:40 kvm1e kernel: [74792.235736] ata2.00: status: { DRDY SENSE ERR } Mar 30 08:40:40 kvm1e kernel: [74792.237045] ata2.00: error: { UNC } Mar 30 08:40:40 kvm1e ceph-osd[450777]: 2020-03-30 08:40:40.240 7f48a41f3700 -1 bluestore(/var/lib/ceph/osd/ceph-51) _do_read bdev-read failed: (5) Input/output error Mar 30 08:40:40 kvm1e kernel: [74792.244914] ata2.00: configured for UDMA/133 Mar 30 08:40:40 kvm1e kernel: [74792.244938] sd 1:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Mar 30 08:40:40 kvm1e kernel: [74792.244942] sd 1:0:0:0: [sdb] tag#1 Sense Key : Medium Error [current] Mar 30 08:40:40 kvm1e kernel: [74792.244945] sd 1:0:0:0: [sdb] tag#1 Add. Sense: Unrecovered read error - auto reallocate failed Mar 30 08:40:40 kvm1e kernel: [74792.244949] sd 1:0:0:0: [sdb] tag#1 CDB: Read(16) 88 00 00 00 00 00 c9 50 7a 00 00 00 04 00 00 00 Mar 30 08:40:40 kvm1e kernel: [74792.244953] blk_update_request: I/O error, dev sdb, sector 3377494800 op 0x0:(READ) flags 0x0 phys_seg 94 prio class 0 Mar 30 08:40:40 kvm1e kernel: [74792.246238] ata2: EH complete Regards David Herselman

1 0

BlueStore and checksum
by Priya Sehgal 30 Mar '20

30 Mar '20

Hi, I am trying to find out whether Ceph has a method to detect silent corruption such as bit rot. I came across this text in a book - "Mastering Ceph : Infrastructure Storage Solution with the latest Ceph release" by Nick Fisk - Luminous release of Ceph employs ZFS-like ability to checksum data at every read. BlueStore calculates and stores the crc32 checksum of any data that is written. On each read request, BlueStore reads the checksum and compares it with the data read from the device. If there is a mismatch, BlueStore will report read error and repair damage. Ceph will then retry the read from another OSD holding the object. I have two questions: 1. If there is a mismatch will the ceph user (who initiated GET) receive any error message and he has to retry OR will this error be auto-corrected by Ceph and the data would be returned from the other OSD (user will never know what went wrong underneath? 2. Does this feature work for Multi-part upload objects also? -- Regards, Priya

2 2