I have two rbd images.
running fio on them, one performs very well, and one performs poorly. I'd like to get more insight on why.
I know that
rbd info xx/yyy
gives me a small amount of information. but I cant seem to find a detailed info dump, in the way you can do things like
ceph daemon osd.9 config show
I cant even get useful google hits on things like
"ceph find rbd image stripe-unit"
all the pages seem to detail how to set the value, but not how to query it.
Can anyone point me in the right direction?
--
Philip Brown| Sr. Linux System Administrator | Medata, Inc.
5 Peters Canyon Rd Suite 250
Irvine CA 92606
Office 714.918.1310| Fax 714.918.1325
pbrown(a)medata.com| www.medata.com
Hello Everyone!
I'm trying to mount the ceph, with the fuse client under debian 9 (ceph-fuse 10.2.11-2).
Ceph is on the latest Octopus release.
The direct command is working, but writing it in fstab does not.
Command I use:
ceph-fuse --id dev.wsc -k /etc/ceph/ceph.clinet.dev.wsc.keyring -r /testing/dev.wsc/userdata /mnt/ceph
/etc/ceph.conf:
[global]
fsid = [fsid]
mon_host = 10.99.15.1 10.99.15.2 10.99.15.3 10.99.15.4 10.99.15.5
/etc/ceph/ceph.clinet.dev.wsc.keyring:
[client.dev.wsc]
key = [key]
fstab:
none /mnt/ceph fuse.ceph ceph.id=dev.wsc,ceph.conf=/etc/ceph/ceph.conf,ceph.client_mountpoint=/testing/dev.wsc/userdata,_netdev,defaults 0 0
With mount -a it tells me "ceph mount failed with (1) Operation not permitted".
Where can I find authentication logs on the cluster?
Thanks in advance and kind regards,
Simon
Hello, Ceph users,
How can I figure out why it is not possible to unprotect a snapshot
in a RBD image? I use this RBD pool for OpenNebula, and somehow there
is a snapshot in one image, which OpenNebula does not see. So I wanted
to delete the snapshot:
# rbd info one/one-1312
rbd image 'one-1312':
size 8 GiB in 2048 objects
order 22 (4 MiB objects)
snapshot_count: 1
id: 6732dccd50fa75
block_name_prefix: rbd_data.6732dccd50fa75
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Wed Jun 30 10:41:59 2021
access_timestamp: Wed Jun 30 16:48:30 2021
modify_timestamp: Wed Jun 30 15:52:18 2021
# rbd snap ls one/one-1312
SNAPID NAME SIZE PROTECTED TIMESTAMP
1727 snap 8 GiB yes Wed Jun 30 16:11:39 2021
# rbd snap rm one/one-1312@snap
Removing snap: 0% complete...failed.
rbd: snapshot 'snap' is protected from removal.
2021-07-01 08:33:41.489 7f79c6ffd700 -1 librbd::Operations: snapshot is protected
# rbd snap unprotect one/one-1312@snap
2021-07-01 08:28:40.747 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [68ba8e7bace188] in pool 'one'
2021-07-01 08:28:40.749 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy
2021-07-01 08:28:40.749 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: 0x56522f10e830 should_complete_error: ret_val=-16
rbd: unprotecting snap failed: 2021-07-01 08:28:40.751 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: 0x56522f10e830 should_complete_error: ret_val=-16
As far as I can see neither the snapshot nor the RBD image itself is
used by a running qemu in my cluster. How can I delete the snapshot
or debug the problem further?
This is Ceph 14.2.21 on CentOS (mons are CentOS 8 now, and OSDs are CentOS 7).
Thanks,
-Yenya
--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 |
We all agree on the necessity of compromise. We just can't agree on
when it's necessary to compromise. --Larry Wall
Hello
We did try to use Cephadm with Podman to start 44 OSDs per host which consistently stop after adding 24 OSDs per host.
We did look into the cephadm.log on the problematic host and saw that the command `cephadm ceph-volume lvm list --format json` did stuck.
We were the output of the command wasn't complete. Therefore, we tried to use compacted JSON and we could increase the number to 36 OSDs per host.
If you need more information just ask.
Podman version: 3.2.1
Ceph version: 16.2.4
OS version: Suse Leap 15.3
Greetings,
Jan
Hi,
Today while debugging something we had a few questions that might lead
to improving the cephfs forward scrub docs:
https://docs.ceph.com/en/latest/cephfs/scrub/
tldr:
1. Should we document which sorts of issues that the forward scrub is
able to fix?
2. Can we make it more visible (in docs) that scrubbing is not
supported with multi-mds?
3. Isn't the new `ceph -s` scrub task status misleading with multi-mds?
Details here:
1) We found a CephFS directory with a number of zero sized files:
# ls -l
...
-rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58
upload_fc501199e3e7abe6b574101cf34aeefb.png
-rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23
upload_fce4f55348185fefa0abdd8d11095ba8.gif
-rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54
upload_fd95b8358851f0dac22fb775046a6163.png
...
The user claims that those files were non-zero sized last week. The
sequence of zero sized files includes *all* files written between Nov
2 and 9.
The user claims that his client was running out of memory, but this is
now fixed. So I suspect that his ceph client (kernel
3.10.0-1127.19.1.el7.x86_64) was not behaving well.
Anyway, I noticed that even though the dentries list 0 bytes, the
underlying rados objects have data, and the data looks good. E.g.
# rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx
200212e68b5.00000000
# file 200212e68b5.00000000
200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA,
non-interlaced
So I managed to recover the files doing something like this (using an
input file mapping inode to filename) [see PS 0].
But I'm wondering if a forward scrub is able to fix this sort of
problem directly?
Should we document which sorts of issues that the forward scrub is able to fix?
I anyway tried to scrub it, which led to:
# ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx
recursive repair
Scrub is not currently supported for multiple active MDS. Please
reduce max_mds to 1 and then scrub.
So ...
2) Shouldn't we update the doc to mention loud and clear that scrub is
not currently supported for multiple active MDS?
3) I was somehow surprised by this, because I had thought that the new
`ceph -s` multi-mds scrub status implied that multi-mds scrubbing was
now working:
task status:
scrub status:
mds.x: idle
mds.y: idle
mds.z: idle
Is it worth reporting this task status for cephfs if we can't even scrub them?
Thanks!!
Dan
[0]
mkdir -p recovered
while read -r a b; do
for i in {0..9}
do
echo "rados stat --cluster=flax --pool=cephfs_data
--namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get
--cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x"
$a).0000000$i $(printf "%x" $a).0000000$i
done
echo cat $(printf "%x" $a).* ">" $(printf "%x" $a)
echo mv $(printf "%x" $a) recovered/$b
done < inones_fnames.txt
Dear all,
I'm sorry if I'm asking for the obvious or missing a previous discussion of
this but I could not find the answer to my question online. I'd be happy to
be pointed to the right direction only.
The cephfs-mirror tool in pacific looks extremely promising. How does it
work exactly? Is it based on files and (recursive) ctime or rather based on
object information? Does it handle incremental changes (only) between
snapshots?
There is an issue related to this that mentions recursive ctime. But that
would mean that users could "rsync -a" data to the file system and this
would not get synchronized.
I have good experience with ZFS which is able to identify changes between
two snapshots A and B and then only transfer these changes (using a
sub-file level, on the ZFS equivalent of blocks to my understanding) to
another server with the same file system that is in the exact state as
snapshot A. Does cephfs-mirror work the same?
Best wishes,
Manuel