Hi.
How do I find out if the MDS is "busy" - being the one limiting CephFS
metadata throughput. (12.2.8).
$ time find . | wc -l
1918069
real 8m43.008s
user 0m2.689s
sys 0m7.818s
or 3.667ms per file.
In the light of "potentially batching" and a network latency of ~0.20ms to
the MDS - I have a feeling that this could be significantly improved.
Then I additionally tried to do the same through the NFS -ganesha gateway.
For reference:
Same - but on "local DAS - xfs".
$ time find . | wc -l
1918061
real 0m4.848s
user 0m2.360s
sys 0m2.816s
Same but "above local DAS over NFS":
$ time find . | wc -l
1918061
real 5m56.546s
user 0m2.903s
sys 0m34.381s
jk@ceph-mon1:~$ sudo ceph fs status
[sudo] password for jk:
cephfs - 84 clients
======
+------+----------------+-----------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+----------------+-----------+---------------+-------+-------+
| 0 | active | ceph-mds2 | Reqs: 1369 /s | 11.3M | 11.3M |
| 0-s | standby-replay | ceph-mds1 | Evts: 0 /s | 0 | 0 |
+------+----------------+-----------+---------------+-------+-------+
+------------------+----------+-------+-------+
| Pool | type | used | avail |
+------------------+----------+-------+-------+
| cephfs_metadata | metadata | 226M | 16.4T |
| cephfs_data | data | 164T | 132T |
| cephfs_data_ec42 | data | 180T | 265T |
+------------------+----------+-------+-------+
+-------------+
| Standby MDS |
+-------------+
+-------------+
MDS version: ceph version 12.2.5-45redhat1xenial
(d4b9f17b56b3348566926849313084dd6efc2ca2) luminous (stable)
How can we asses where the bottleneck is and what to do to speed it up?
Hi all,
*** Short version ***
Is there a way to repair a rocksdb from errors "Encountered error while
reading data from compression dictionary block Corruption: block
checksum mismatch" and "_open_db erroring opening db" ?
*** Long version ***
We operate a nautilus ceph cluster (with 100 disks of 8TB in 6 servers +
4 mons/mgr + 3 mds).
We recently (Monday 20) upgraded from 14.2.7 to 14.2.8. This triggered a
rebalancing of some data.
Two days later (Wednesday 22) we had a very short power outage. Only one
of the osd servers went down (and unfortunately died).
This triggered a reconstruction of the losts osds. Operations went fine
until Saturday 25 where some osds in the 5 remaining servers started to
crash apparently with no reasons.
We tryed to restart them, but they crashed again. We ended with 18 osd
down (+ 16 in the dead server so 34 osd downs out of 100).
Looking at the logs we found for all the crashed osd :
-237> 2020-04-25 16:32:51.835 7f1f45527a80 3 rocksdb:
[table/block_based_table_reader.cc:1117] Encountered error while reading
data from compression dictionary block Corruption: block checksum
mismatch: expected 0, got 2729370997 in db/181355.sst offset
18446744073709551615 size 18446744073709551615
and
2020-04-25 16:05:47.251 7fcbd1e46a80 -1
bluestore(/var/lib/ceph/osd/ceph-3) _open_db erroring opening db:
We also noticed that the "Encountered error while reading data from
compression dictionary block Corruption: block checksum mismatch" was
present few days before the crash.
We also have some osd with this error but still up.
We tryed to repair with :
ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-3 repair
But no success (it ends with _open_db erroring opening db).
Thus does somebody have an idea to fix this or at least know if it's
possible to repair and correct the "Encountered error while reading data
from compression dictionary block Corruption: block checksum mismatch"
and "_open_db erroring opening db" !
Thanks for your help (we are desperate because we will loose datas and
are fighting to save something) !!!
F.
Hello All,
I just installed ceph iscsi target following "manual installation" at
https://docs.ceph.com/docs/nautilus/rbd/iscsi-target-cli-manual-install/
When the api server starts it seems to work but in service status it
reports:
apr 30 07:22:51 lab-ceph-01 rbd-target-api[58740]: Processing osd blacklist
entries for this node
apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: No OSD blacklist entries
found
apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: Reading the
configuration object to update local LIO configuration
apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: Configuration does not
have an entry for this host(lab-ceph-01) - nothing to define to LIO
apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: * Serving Flask app
"rbd-target-api" (lazy loading)
apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: * Environment: production
apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: WARNING: This is a
development server. Do not use it in a production deployment.
apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: Use a production WSGI
server instead.
apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: * Debug mode: on
apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: * Running on
http://[::]:5000/
(Press CTRL+C to quit)
My configuration contains my hostname :
cat /etc/ceph/iscsi-gateway.cfg
[config]
# Name of the Ceph storage cluster. A suitable Ceph configuration file
allowing
# access to the Ceph storage cluster from the gateway node is required, if
not
# colocated on an OSD node.
cluster_name = ceph
# Place a copy of the ceph cluster's admin keyring in the gateway's
/etc/ceph
# drectory and reference the filename here
gateway_keyring = ceph.client.admin.keyring
# API settings.
# The API supports a number of options that allow you to tailor it to your
# local environment. If you want to run the API under https, you will need
to
# create cert/key files that are compatible for each iSCSI gateway node,
that is
# not locked to a specific node. SSL cert and key files *must* be called
# 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the
'/etc/ceph/' directory
# on *each* gateway node. With the SSL files in place, you can use
'api_secure = true'
# to switch to https mode.
debug = true
# To support the API, the bear minimum settings are:
api_secure = false
#prometheus_exporter = false
# Additional API configuration options are as follows, defaults shown.
# api_user = admin
# api_password = admin
# api_port = 5001
trusted_ip_list = lab-ceph-01
When I run gwcli command it reports:
[root@lab-ceph-01 ansible]# gwcli -d
Adding ceph cluster 'ceph' to the UI
Fetching ceph osd information
Querying ceph for state information
Unable to access the configuration object : REST API failure, code : 504
Traceback (most recent call last):
File "/bin/gwcli", line 4, in <module>
__import__('pkg_resources').run_script('ceph-iscsi==3.4', 'gwcli')
File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line
654, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line
1441, in run_script
exec(script_code, namespace, namespace)
File
"/usr/local/lib/python3.6/site-packages/ceph_iscsi-3.4-py3.6.egg/EGG-INFO/scripts/gwcli",
line 194, in <module>
File
"/usr/local/lib/python3.6/site-packages/ceph_iscsi-3.4-py3.6.egg/EGG-INFO/scripts/gwcli",
line 105, in main
File
"/usr/local/lib/python3.6/site-packages/ceph_iscsi-3.4-py3.6.egg/gwcli/gateway.py",
line 84, in refresh
gwcli.utils.GatewayError
I installed the same on different server 2 or 3 weeks ago and and it is
working,
Any change in the code in last weeks ?
Please, anyone could help me ?
Ignazio
Shutdown nautilus cluster, start stuck in peering
I followed some manual here to do before turning all nodes off:
ceph osd set norecover ; ceph osd set norebalance ; ceph osd set
nobackfill ; ceph osd set nodown ; ceph osd set pause
And after reboot almost al pg's were stuck in peering after unsetting
these.
What to do now????
Hi,
I just installed a new cluster with cephadm (https://docs.ceph.com/docs/master/cephadm/install/), all was working fine until I disabled cephx using the following commands:
ceph config set global auth_cluster_required none
ceph config set global auth_service_required none
ceph config set global auth_client_required none
Now I keep running into:
[root@cph1]# ceph health
[errno 13] RADOS permission denied (error connecting to the cluster)
However, in my /etc/ceph/ceph.conf there is no mention of the configuration keys mentioned above. Same goes for the cluster folder in:
/var/lib/ceph/d5d206f4-8a4f-11ea-9d04-ac1f6bf8b2d3/mgr.cph1.srvnl.nl.qwnssx/config
How can I remove this setting now? Obviously all ceph * related commands run into the same RADOS permission denied.
Thanks,
Tom
I am trying to manually create a radosgw instance for a small development
installation. I was able to muddle through and get a working mon, mgr, and
osd (x2), but the docs for radosgw are based on ceph-deploy which is not
part of the octopus release.
The host systems are all lxc/lxd containers with centos 7 and the
http://download.ceph.com/rpm-octopus/el7/$basearch repos.
The radosgw setip that I did was essentially creating a keyring following
the steps in
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.2.3/ht…
as far as I could, but I have to admit that even after getting mon, mgr,
and a couple of osds working I still don't grok the keyring usage... is
there a more conceptual doc that explains how they are used and what has to
be on each host?
** start rgw0 (systemctl start ceph-radosgw@rgw0)
** The failure on rgw0 host (journalctl -f -u ceph-radosgw(a)rgw0.service)
Apr 27 17:00:20 rgw0 systemd[1]: Started Ceph rados gateway.
Apr 27 17:00:20 rgw0 radosgw[8668]: 2020-04-27T17:00:20.791+0000
7f330173d700 -1 monclient(hunting): handle_auth_bad_method server
allowed_methods [2] but i only support [2]
Apr 27 17:00:20 rgw0 radosgw[8668]: failed to fetch mon config
(--no-mon-config to skip)
Apr 27 17:00:20 rgw0 systemd[1]: ceph-radosgw(a)rgw0.service: main process
exited, code=exited, status=1/FAILURE
[root@cephadm ~]# ceph mon dump
dumped monmap epoch 2
epoch 2
fsid f286f47e-7043-4dca-aeb0-03fda1910378
last_changed 2020-04-20T16:23:49.311252+0000
created 2020-04-20T16:18:21.389527+0000
min_mon_release 15 (octopus)
0: [v2:10.121.149.102:3300/0,v1:10.121.149.102:6789/0] mon.mon0
** on mon0 I also see a lot of this in the logs:
2020-04-22T00:29:41.293+0000 7f35cb1da700 0 cephx server client.rgw0:
handle_request failed to decode CephXAuthenticate: buffer::end_of_buffer
Any help would be appreciated.
--
Patrick Dowler
Canadian Astronomy Data Centre
Victoria, BC, Canada
Hi,
It is a newbie question. I would be really thankful if you can answer it
please. I want to compile the Ceph source code. Because I want to profile
*Librados* and *CRUSH* function stacks. Please verify if this is the right
track I am following:
- I have cloned the Ceph from Ceph git repository
- I have installed the build code dependencies from script *install-deps.sh*
- Because I would like to use the* gdb debug* client program later, the
client program will depend on the librados library, so I must compile ceph
in debug mode. Therefore I would modify the parameters of ceph cmake in
*do_cmake.sh* script accordingly.
- Then I compile *do_cmake*
*- *In build I run* make - j 32*
*- *To start the developer mode, I run *make vstart.*
*- *In the developer mode I can write *READ* and *WRITE* tests...compile
these tests and then use some profiling tool to call the compiled
executable to profile the function stacks.
Is this the correct way for *CPU profiling*? Please let me know if it is
fine or is there something more also.
Bobby !
Hi everyone!
I have been working for the past week or so trying to get ceph-iscsi to work - Octopus release. Even just getting a single node working would be a major victory in this battle but so far, victory has proven elusive.
My setup: a pair of Dell Optiplex 7010 desktops, each with 16 gig of memory and 1 boot drive (USB 3) and 3 SATA drives (500 Gb SSHD drives). No RAID controllers anywhere. Yes, I know that 3 nodes is the recommended minimum number for a production system - this isn't production (this is just seeing if the darned thing will even work).
I am using Centos 8.1.1911 for the OS (4.18.0 kernel) with a basic or minimal installation (no X-Window). Single Gigabit ethernet per node. I have 2 MON and 2 Mgr installed and working, and I have a total of 6 OSDs working. I created the RBD pool (named "rbd" per the published instructions), creating it initially with 256 PGs (autoscale decided that 32 was a better choice - whatever). The cluster is green and all 6 OSDs are green (up and in). All deployment is via cephadm and all containers are running via podman.
Here is where things start to fall apart.
I was able to find RPM packages for targetcli and python-rtslib (called python3-rtslib) but was not able to find tcmu-runner nor ceph-iscsi packages. OK, no big deal. Time to head over to the manual install guide.
I was able to build tcmu-runner, install it and apparently it is running (systemctl says it is active) so that appears to be OK.
The problem is getting rbd-target-gw and rbd-target-api to work. They appear to build OK and of course, I am able to get them registered with systemd. They universally fail when trying to run them (systemctl start rbd-target-gw or systemctl start rbd-target-api). Both report failure. Looking in journalctl -xe shows no hints at all regarding why they failed (only that they did). Looking in /var/log/rbd-target-api/ show nothing at all (no files). Likewise in /var/log/rbd-target-gw/ (no files).
HELP!!
Now, some possibly germane questions:
1) are any other Ceph services required for ceph-iscsi to work like RADOSgw?
2) since there are no apparent packages available for ceph-iscsi, can anything be inferred to the production-readiness of the subsystem?
3) are there any known errata or missing steps in the instructions for getting ceph-iscsi to work?
Thanks!
Ron Gage