Hello everyone and sorry. Maybe someone has already faced this problem.
A day ago, we restored our Openshift cluster, however, at the moment, the PVCs cannot connect to the pod. We looked at the status of the ceph and found that our MDS were in standby mode, then found that the metadata was corrupted. After some manipulations, we were able to turn on our MDS daemons, but there is still no record on the cluster, the ceph status command shows the following.
sh-4.4$ ceph -s
cluster:
id: 9213604e-b0b6-49d5-bcb3-f55ab3d79119
health: HEALTH_ERR
1 MDSs report damaged metadata
1 MDSs are read only
6 daemons have recently crashed
services:
mon: 5 daemons, quorum bd,bj,bm,bn,bo (age 26h)
mgr: a(active, since 25h)
mds: 1/1 daemons up, 1 hot standby
osd: 9 osds: 9 up (since 41h), 9 in (since 42h)
rgw: 1 daemon active (1 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 10 pools, 225 pgs
objects: 1.60M objects, 234 GiB
usage: 606 GiB used, 594 GiB / 1.2 TiB avail
pgs: 225 active+clean
io:
client: 852 B/s rd, 1 op/s rd, 0 op/s wr
Now we trying to follow this instructions:
https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#recovery-…
What else have we tried:
cephfs-journal-tool --rank=1:0 event recover_dentries summary
cephfs-journal-tool --rank=1:0 journal reset
cephfs-table-tool all reset session
ceph tell mds.gml--cephfs-a scrub start / recursive repair force
ceph tell mds.gml--cephfs-b scrub start / recursive repair force
ceph mds repaired 0
ceph tell mds.gml--cephfs-a damage ls
[
{
"damage_type": "dir_frag",
"id": 26851730,
"ino": 1100162409473,
"frag": "*",
"path": "/volumes/csi/csi-vol-5ad18c03-3205-11ed-9ba7-0a580a810206/e5664004-51e0-4bff-85c8-029944b431d8/store/096/096a1497-78ab-4802-a5a7-d09e011fd3a5/202301_1027796_1027796_0"
},
………
{
"damage_type": "dir_frag",
"id": 118336643,
"ino": 1100162424469,
"frag": "*",
"path": "/volumes/csi/csi-vol-5ad18c03-3205-11ed-9ba7-0a580a810206/e5664004-51e0-4bff-85c8-029944b431d8/store/096/096a1497-78ab-4802-a5a7-d09e011fd3a5/202301_1027832_1027832_0"
},
Now we trying:
# Session table
cephfs-table-tool 0 reset session
# SnapServer
cephfs-table-tool 0 reset snap
# InoTable
cephfs-table-tool 0 reset inode
# Journal
cephfs-journal-tool --rank=0 journal reset
# Root inodes ("/" and MDS directory)
cephfs-data-scan init
cephfs-data-scan scan_extents <data pool>
cephfs-data-scan scan_inodes <data pool>
cephfs-data-scan scan_links
Is it right way and cant it be our salvation?
Thank you!
Can anyone please point me at a doc that explains the most efficient procedure to rename a ceph node WITHOUT causing a massive misplaced objects churn?
When my node came up with a new name, it properly joined the cluster and owned the OSDs, but the original node with no devices remained. I expect this affected the crush map such that a large qty of objects got reshuffled. I want no object movement, if possible.
BTW this old cluster is on luminous. ☹
Hello!
I stuck with a problem, while trying to create cluster of 3 nodes (AWS EC2 instancies):
fa11 ~ # ceph orch host ls
HOST ADDR LABELS STATUS
fa11 172.16.24.67 _admin
fa12 172.16.23.159 _admin
fa13 172.16.25.119 _admin
3 hosts in cluster
Each of them have 2 disks (all accepted by CEPH):
fa11 ~ # ceph orch device ls
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
fa11 /dev/nvme1n1 ssd Amazon_Elastic_Block_Store_vol016651cf7f3b9c9dd 8589M Yes 7m ago
fa11 /dev/nvme2n1 ssd Amazon_Elastic_Block_Store_vol034082d7d364dfbdb 5368M Yes 7m ago
fa12 /dev/nvme1n1 ssd Amazon_Elastic_Block_Store_vol0ec193fa3f77fee66 8589M Yes 3m ago
fa12 /dev/nvme2n1 ssd Amazon_Elastic_Block_Store_vol018736f7eeab725f5 5368M Yes 3m ago
fa13 /dev/nvme1n1 ssd Amazon_Elastic_Block_Store_vol0443a031550be1024 8589M Yes 84s ago
fa13 /dev/nvme2n1 ssd Amazon_Elastic_Block_Store_vol0870412d37717dc2c 5368M Yes 84s ago
fa11 is first host, where from I manage cluster.
Adding OSD from fa11 itself works fine:
fa11 ~ # ceph orch daemon add osd fa11:/dev/nvme1n1
Created osd(s) 0 on host 'fa11'
But it doesn't work for other 2 hosts (it hangs forever):
fa11 ~ # ceph orch daemon add osd fa12:/dev/nvme1n1
^CInterrupted
Logs on fa12 shows that it hangs at following step:
fa12 ~ # tail /var/log/ceph/a9ef6c26-ac38-11ed-9429-06e6bc29c1db/ceph-volume.log
...
[2023-02-14 07:38:20,942][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key
[2023-02-14 07:38:20,964][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a51506c2-e910-4763-9a0c-f6c2194944e2
I'm not sure what might be the reason for this hanging?
*Additional details:
1) cephadm installed, using curl (https://docs.ceph.com/en/quincy/cephadm/install/#curl-based-installation)
2) I use user "ceph", instead of "root" and port 2222 instead of 22. First node was bootstrapped, using below command:
cephadm bootstrap --mon-ip 172.16.24.67 --allow-fqdn-hostname --ssh-user ceph --ssh-config /home/anton/ceph/ssh_config --cluster-network 172.16.16.0/20 --skip-monitoring-stack
Content of /home/anton/ceph/ssh_config:
fa11 ~ # cat /home/anton/ceph/ssh_config
Host *
User ceph
Port 2222
IdentityFile /home/ceph/.ssh/id_rsa
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null
3) Hosts fa12 and fa13 were added, using commnds:
ceph orch host add fa12.testing.swiftserve.com 172.16.23.159 --labels _admin
ceph orch host add fa13.testing.swiftserve.com 172.16.25.119 --labels _admin
Thanks in advance!
BR/Anton
Greetings,
Regarding: https://tracker.ceph.com/issues/58019
And: https://github.com/ceph/ceph/pull/47341
We have completed upgrading our production pacific clusters to 16.2.11 and are still experiencing this bug. Can anyone confirm if this backport was included in 16.2.11. Could there be anything wrong with our configuration that is contributing?
Thank you,
Josh Beaman
Hi,
You can use smartctl_exporter [1] for all your media, not only the SSD
k
[1] https://github.com/prometheus-community/smartctl_exporter
Sent from my iPhone
> On 14 Feb 2023, at 23:01, Drew Weaver <drew.weaver(a)thenap.com> wrote:
> Hello,
>
> After upgrading a lot of iDRAC9 modules to version 6.10 in servers that are involved in a Ceph cluster we noticed that the iDRAC9 shows the write endurance as 0% on any non-certified disk.
>
> OMSA still shows the correct remaining write endurance but I am assuming that they are working feverishly to eliminate that too.
>
> I opened a support ticket with Dell once this was brought to my attention and they basically told me that I was lucky that it ever worked at all, which I thought was an odd response given that the iDRAC enterprise licenses cost several hundred dollars each.
>
> I know that the old Intel Datacenter Tool used to be able to reach through a MegaRAID controller and read the remaining write endurance but that tool is essentially defunct now.
>
> What are you folks using to monitor your write endurance on your SSDs that you couldn't buy from Dell because they had a 16 week lead time while the MFG could deliver the drives in 3 days?
>
> Thanks,
> -Drew
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
We are happy to announce another release of the go-ceph API library. This
is a
regular release following our every-two-months release cadence.
https://github.com/ceph/go-ceph/releases/tag/v0.20.0
Changes include additions to the rbd, rgw and cephfs packages. More details
are
available at the link above.
The library includes bindings that aim to play a similar role to the
"pybind"
python bindings in the ceph tree but for the Go language. The library also
includes additional APIs that can be used to administer cephfs, rbd, and rgw
subsystems.
There are already a few consumers of this library in the wild, including the
ceph-csi project.
Sven
Hello,
After upgrading a lot of iDRAC9 modules to version 6.10 in servers that are involved in a Ceph cluster we noticed that the iDRAC9 shows the write endurance as 0% on any non-certified disk.
OMSA still shows the correct remaining write endurance but I am assuming that they are working feverishly to eliminate that too.
I opened a support ticket with Dell once this was brought to my attention and they basically told me that I was lucky that it ever worked at all, which I thought was an odd response given that the iDRAC enterprise licenses cost several hundred dollars each.
I know that the old Intel Datacenter Tool used to be able to reach through a MegaRAID controller and read the remaining write endurance but that tool is essentially defunct now.
What are you folks using to monitor your write endurance on your SSDs that you couldn't buy from Dell because they had a 16 week lead time while the MFG could deliver the drives in 3 days?
Thanks,
-Drew
CALL FOR PROPOSALS
EXTENDED
[image: Logo]
CALL FOR PROPOSALS EXTENDED
<https://events.linuxfoundation.org/cephalocon/program/cfp/>
[image: Ceph23-LaunchAssets-v1_snackable-1200x627-A.png]
<https://events.linuxfoundation.org/cephalocon/>
We heard your feedback on the short notice for the Call for Proposals
<https://events.linuxfoundation.org/cephalocon/program/cfp/> and have
extended the deadline to Sunday, February 19, at 11:59 pm PST!
Cephalocon <https://events.linuxfoundation.org/cephalocon/> is the premier
yearly event that brings together the global community of operators,
developers, and researchers for Ceph, the open source distributed storage
system designed to provide excellent performance, reliability, and
scalability. Join new and existing community members worldwide to learn
more about Ceph and the project’s future from the developers writing the
code and the operators deploying it at scale.
SUBMIT YOUR PROPOSAL
<https://events.linuxfoundation.org/cephalocon/program/cfp/>
Important Dates
- *CFP Closes: *Sunday, February 19 at 11:59 pm PST
- *CFP Notifications:* Friday, February 24
- *Schedule Announcement:* Wednesday, March 1
- *Presentation Slide Due Date:* Wednesday, April 12
- *Event Dates:* Monday, April 17 – Tuesday, April 18 (Developer Summit
Sunday, April 16)
Sponsorship
Sponsoring Cephalocon
<https://events.linuxfoundation.org/cephalocon/sponsor/> is a unique
opportunity to gain valuable mindshare with an elite audience of engineers,
researchers, and end-users. Building on the success of Ceph Days and
virtual Developer Summits, Cephalocon brings together more than 400
attendees from across the globe to showcase Ceph’s history and its future,
real-world applications, and of course, highlight vendor solutions.
Cephalocon 2023 promises to make for incredible community building,
cross-company collaboration, and cutting-edge training.
Contact us at sponsorships(a)ceph.foundation to secure your sponsorship,
request additional details or discuss custom options.
See Sponsorship Prospectus
<https://events.linuxfoundation.org/cephalocon/sponsor/>
[image: Link icon] <https://ceph.io>
[image: Twitter icon] <https://twitter.com/ceph>
[image: YouTube icon] <https://youtube.com/c/cephstorage>
[image: LinkedIn icon] <https://www.linkedin.com/company/ceph/>
[image: Facebook icon] <https://www.facebook.com/cephstorage/>
f8xj3pa8x9xf
My ceph cluster became unstable yesterday after zincati (CoreOS's
auto-updater) updated one of my nodes from 37.20221225.3.0 to
37.20230110.3.1(*). The symptom was slow ops in my cephfs mds which
started immediately the OSDs on this node became in and up. Excluding
the OSDs on this node worked round the problem. Note that the node is
also running a mon and client workloads which use ceph. Also note that
the OSD came up and (IIUC) were participating in recovering their data
to other OSDs. The problem only started when I allowed them to be in.
I rolled back the OS update and the problem was immediately resolved.
Unfortunately I didn't keep the OSD logs, but they lead me to this
thread from ceph-users:
https://www.mail-archive.com/ceph-users@ceph.io/msg18474.html . I
wonder if we have an issue with a very recent kernel update.
I should be able to reproduce if it's likely to be of use to anybody,
but for now I've rolled back this OS update and disabled automatic
updating on my other nodes.
Matt
(*) The complete list of changes:
$ rpm-ostree db diff
d477f98d52bf707d4282f6835b85bed3d60e305a0cf6eb8effd4db4b89607f05
fc214c16d248686d4cf2bb3050b59c559f091692d7af3b07ef896f1b8ab2f161
ostree diff commit from:
d477f98d52bf707d4282f6835b85bed3d60e305a0cf6eb8effd4db4b89607f05
ostree diff commit to:
fc214c16d248686d4cf2bb3050b59c559f091692d7af3b07ef896f1b8ab2f161
Upgraded:
bash 5.2.9-3.fc37 -> 5.2.15-1.fc37
btrfs-progs 6.0.2-1.fc37 -> 6.1.2-1.fc37
clevis 18-12.fc37 -> 18-14.fc37
clevis-dracut 18-12.fc37 -> 18-14.fc37
clevis-luks 18-12.fc37 -> 18-14.fc37
clevis-systemd 18-12.fc37 -> 18-14.fc37
container-selinux 2:2.193.0-1.fc37 -> 2:2.198.0-1.fc37
containerd 1.6.12-1.fc37 -> 1.6.14-2.fc37
containers-common 4:1-73.fc37 -> 4:1-76.fc37
containers-common-extra 4:1-73.fc37 -> 4:1-76.fc37
coreutils 9.1-6.fc37 -> 9.1-7.fc37
coreutils-common 9.1-6.fc37 -> 9.1-7.fc37
crun 1.7.2-2.fc37 -> 1.7.2-3.fc37
curl 7.85.0-4.fc37 -> 7.85.0-5.fc37
dnsmasq 2.87-3.fc37 -> 2.88-1.fc37
ethtool 2:6.0-1.fc37 -> 2:6.1-1.fc37
fwupd 1.8.8-1.fc37 -> 1.8.9-1.fc37
git-core 2.38.1-1.fc37 -> 2.39.0-1.fc37
grub2-common 1:2.06-63.fc37 -> 1:2.06-72.fc37
grub2-efi-x64 1:2.06-63.fc37 -> 1:2.06-72.fc37
grub2-pc 1:2.06-63.fc37 -> 1:2.06-72.fc37
grub2-pc-modules 1:2.06-63.fc37 -> 1:2.06-72.fc37
grub2-tools 1:2.06-63.fc37 -> 1:2.06-72.fc37
grub2-tools-minimal 1:2.06-63.fc37 -> 1:2.06-72.fc37
kernel 6.0.15-300.fc37 -> 6.0.18-300.fc37
kernel-core 6.0.15-300.fc37 -> 6.0.18-300.fc37
kernel-modules 6.0.15-300.fc37 -> 6.0.18-300.fc37
libcurl-minimal 7.85.0-4.fc37 -> 7.85.0-5.fc37
libgpg-error 1.45-2.fc37 -> 1.46-1.fc37
libgusb 0.4.2-1.fc37 -> 0.4.3-1.fc37
libksba 1.6.2-1.fc37 -> 1.6.3-1.fc37
libpcap 14:1.10.1-4.fc37 -> 14:1.10.2-1.fc37
libpwquality 1.4.4-11.fc37 -> 1.4.5-1.fc37
libsmbclient 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37
libwbclient 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37
moby-engine 20.10.20-1.fc37 -> 20.10.21-1.fc37
ncurses 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37
ncurses-base 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37
ncurses-libs 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37
net-tools 2.0-0.63.20160912git.fc37 -> 2.0-0.64.20160912git.fc37
rpm-ostree 2022.16-2.fc37 -> 2022.19-2.fc37
rpm-ostree-libs 2022.16-2.fc37 -> 2022.19-2.fc37
samba-client-libs 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37
samba-common 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37
samba-common-libs 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37
selinux-policy 37.16-1.fc37 -> 37.17-1.fc37
selinux-policy-targeted 37.16-1.fc37 -> 37.17-1.fc37
tpm2-tss 3.2.0-3.fc37 -> 3.2.1-1.fc37
Removed:
cracklib-dicts-2.9.7-30.fc37.x86_64
--
Matthew Booth