March 2020 - ceph-users - lists.ceph.io

by Dave Hall

Hello. I have a cluster with 3 MDSs - 2 active and 1 standby. Looking at 'top' I see that the 2 active MDSs have relatively low resident memory - less than 3 g, but very high virtual memory - 21 g and 41 g. Is it safe to just restart these processes one at a time to release the virtual memory? Also, is there something that needs to be tweaked to prevent the MDSs from accumulating so much memory? Thanks. -Dave -- Dave Hall Binghamton University

4 years, 1 month

2
1
0 0

Re: Exporting

by Eugen Block

Hi, have you tried to create a snapshot of that image and export the snapshot? Or maybe clone the snapshot first and then export it? Does it also fail? Zitat von Rhian Resnick <rresnick(a)fau.edu>: > Evening, > > We are running into issues exporting a disk image from ceph rbd. > When we attempt to export an rbd image in a cache tiered > erasure-coded pool on Luminus. > > All the other disks are working fine but this one is acting up. We > have a bit of important data on other disks so obviously want to > make sure this doesn't happen to those. > > [root@ceph-p-mon1 home]# rbd export one/one-177-588-0 one-177-588-0 > Exporting image: 8% complete...rbd: error reading from source image > at offset 5456789504: (5) Input/output error > 2020-03-23 20:11:29.210718 7f2f3effd700 -1 > librbd::io::ObjectRequest: 0x7f2f2c128f90 handle_read_object: failed > to read from object: (5) Input/output error > 2020-03-23 20:11:29.565184 7f2f3e7fc700 -1 > librbd::io::ObjectRequest: 0x7f2f280c84d0 handle_read_cache: failed > to read from cache: (5) Input/output error > Exporting image: 8% complete...failed. > rbd: export error: (5) Input/output error > > > Any thoughts would be appreciated. > > > Some info: > > [root@ceph-p-mon1 home]# rbd info one/one-177-588-0 > rbd image 'one-177-588-0': > size 58.6GiB in 15000 objects > order 22 (4MiB objects) > block_name_prefix: rbd_data.84a01279e2a9e3 > format: 2 > features: layering, exclusive-lock, object-map, fast-diff, deep-flatten > flags: > create_timestamp: Fri Apr 20 17:06:09 2018 > parent: one/one-177@snap > overlap: 2.20GiB > > [root@ceph-p-mon1 home]# ceph status > cluster: > id: 6a2e8f21-bca2-492b-8869-eecc995216cc > health: HEALTH_OK > > services: > mon: 3 daemons, quorum ceph-p-mon2,ceph-p-mon1,ceph-p-mon3 > mgr: ceph-p-mon2(active) > mds: cephfsec-1/1/1 up {0=ceph-p-mon2=up:active}, 6 up:standby > osd: 155 osds: 154 up, 154 in > > data: > pools: 6 pools, 5904 pgs > objects: 145.53M objects, 192TiB > usage: 253TiB used, 290TiB / 543TiB avail > pgs: 5896 active+clean > 8 active+clean+scrubbing+deep > > io: > client: 921KiB/s rd, 5.68MiB/s wr, 110op/s rd, 29op/s wr > cache: 5.65MiB/s flush, 0op/s promote > > > [root@ceph-p-mon1 home]# rpm -qa | grep ceph > ceph-common-12.2.9-0.el7.x86_64 > ceph-mds-12.2.9-0.el7.x86_64 > ceph-radosgw-12.2.9-0.el7.x86_64 > ceph-mgr-12.2.9-0.el7.x86_64 > ceph-12.2.9-0.el7.x86_64 > collectd-ceph-5.8.1-1.el7.x86_64 > ceph-deploy-2.0.1-0.noarch > libcephfs2-12.2.9-0.el7.x86_64 > python-cephfs-12.2.9-0.el7.x86_64 > ceph-selinux-12.2.9-0.el7.x86_64 > ceph-osd-12.2.9-0.el7.x86_64 > ceph-base-12.2.9-0.el7.x86_64 > ceph-mon-12.2.9-0.el7.x86_64 > ceph-release-1-1.el7.noarch > > > Rhian Resnick > > Associate Director Research Computing > > Enterprise Systems > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road, CM22, Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 years, 1 month

1
0
0 0

Re: OSDs wont mount on Debian 10 (Buster) with Nautilus

by Ml Ml

i brought them up manually. Decided to upgrade to octopus, but i am stuck there now. I will open a new thread for it. On Wed, Mar 25, 2020 at 6:23 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > Still down? > > > -----Original Message----- > Cc: ceph-users > Subject: [ceph-users] Re: OSDs wont mount on Debian 10 (Buster) with > Nautilus > > What does the osd error log say? I already have bluestore, if you have > file store maybe you should inspect the mounted fs of /dev/sdd2 eg. > maybe there permissions need to be changed. But first check the errors > of one osd. > > ( You did you reset the failed service with somethig like this systemctl > reset-failed ceph-osd@0 not? ) > > > -----Original Message----- > Cc: ceph-users > Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > Nautilus > > Still no luck. > > But the working OSDs have no partition: > OSD.1 => /dev/sdj > OSD.5 => /dev/sdb > OSD.6 => /dev/sdbc > OSD.10 => /dev/sdl > > > Where as the rest has: > > root@ceph03:~# ls -l /dev/sd* > brw-rw---- 1 root disk 8, 0 Mar 25 16:23 /dev/sda > brw-rw---- 1 root disk 8, 1 Mar 25 16:23 /dev/sda1 > brw-rw---- 1 root disk 8, 2 Mar 25 16:23 /dev/sda2 > brw-rw---- 1 root disk 8, 3 Mar 25 16:23 /dev/sda3 > brw-rw---- 1 root disk 8, 16 Mar 25 16:23 /dev/sdb > brw-rw---- 1 root disk 8, 32 Mar 25 16:23 /dev/sdc > brw-rw---- 1 root disk 8, 48 Mar 25 16:23 /dev/sdd > brw-rw---- 1 root disk 8, 49 Mar 25 16:23 /dev/sdd1 > brw-rw---- 1 ceph ceph 8, 50 Mar 25 16:23 /dev/sdd2 > brw-rw---- 1 root disk 8, 64 Mar 25 16:23 /dev/sde > brw-rw---- 1 root disk 8, 65 Mar 25 16:23 /dev/sde1 > brw-rw---- 1 ceph ceph 8, 66 Mar 25 16:23 /dev/sde2 > brw-rw---- 1 root disk 8, 80 Mar 25 16:23 /dev/sdf > brw-rw---- 1 root disk 8, 81 Mar 25 16:23 /dev/sdf1 > brw-rw---- 1 ceph ceph 8, 82 Mar 25 16:23 /dev/sdf2 > brw-rw---- 1 root disk 8, 96 Mar 25 16:23 /dev/sdg > brw-rw---- 1 root disk 8, 97 Mar 25 16:23 /dev/sdg1 > brw-rw---- 1 ceph ceph 8, 98 Mar 25 16:23 /dev/sdg2 > brw-rw---- 1 root disk 8, 112 Mar 25 16:23 /dev/sdh > brw-rw---- 1 root disk 8, 113 Mar 25 16:23 /dev/sdh1 > brw-rw---- 1 ceph ceph 8, 114 Mar 25 16:23 /dev/sdh2 > brw-rw---- 1 root disk 8, 128 Mar 25 16:23 /dev/sdi > brw-rw---- 1 root disk 8, 129 Mar 25 16:23 /dev/sdi1 > brw-rw---- 1 ceph ceph 8, 130 Mar 25 16:23 /dev/sdi2 > brw-rw---- 1 root disk 8, 144 Mar 25 16:23 /dev/sdj > brw-rw---- 1 root disk 8, 160 Mar 25 16:23 /dev/sdk > brw-rw---- 1 root disk 8, 161 Mar 25 16:23 /dev/sdk1 > brw-rw---- 1 ceph ceph 8, 162 Mar 25 16:23 /dev/sdk2 > brw-rw---- 1 root disk 8, 176 Mar 25 16:23 /dev/sdl > brw-rw---- 1 root disk 8, 192 Mar 25 16:23 /dev/sdm > brw-rw---- 1 root disk 8, 193 Mar 25 16:23 /dev/sdm1 > brw-rw---- 1 ceph ceph 8, 194 Mar 25 16:23 /dev/sdm2 > > Did i miss to convert to bluestore or something? > > > On Wed, Mar 25, 2020 at 4:23 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> > wrote: > > > > Try this > > > > chown ceph.ceph /dev/sdc2 > > chown ceph.ceph /dev/sdd2 > > chown ceph.ceph /dev/sde2 > > chown ceph.ceph /dev/sdf2 > > chown ceph.ceph /dev/sdg2 > > chown ceph.ceph /dev/sdh2 > > > > > > > > -----Original Message----- > > From: Ml Ml [mailto:mliebherr99@googlemail.com] > > Sent: 25 March 2020 16:22 > > To: Marc Roos > > Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > > Nautilus > > > > They where indeed disabled. I enabled them with: > > > > systemctl enable ceph-osd@4 > > systemctl enable ceph-osd@15 > > systemctl enable ceph-osd@17 > > systemctl enable ceph-osd@20 > > systemctl enable ceph-osd@21 > > systemctl enable ceph-osd@27 > > systemctl enable ceph-osd@32 > > > > But they still wont start. > > > > On Wed, Mar 25, 2020 at 4:09 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> > > wrote: > > > > > > > > > I had something similar. My osd were disabled, maybe this installer > > > of > > > > > nautilus does that check > > > > > > systemctl is-enabled ceph-osd@0 > > > > > > https://tracker.ceph.com/issues/44102 > > > >

4 years, 1 month

2
1
0 0

Luminous upgrade question

by Shain Miley

Hi, We are thinking about upgrading our cluster currently running ceph version 12.2.12. I am wondering if we should be looking at upgrading to the latest version of Mimic or the latest version Nautilus. Can anyone here please provide a suggestion…I continue to be a little bit confused about the correct Ceph upgrade path given all the various versions that are out there currently. Thank you in advance. Shain

4 years, 1 month

3
2
0 0

Re: OSDs wont mount on Debian 10 (Buster) with Nautilus

by Ml Ml

Still no luck. But the working OSDs have no partition: OSD.1 => /dev/sdj OSD.5 => /dev/sdb OSD.6 => /dev/sdbc OSD.10 => /dev/sdl Where as the rest has: root@ceph03:~# ls -l /dev/sd* brw-rw---- 1 root disk 8, 0 Mar 25 16:23 /dev/sda brw-rw---- 1 root disk 8, 1 Mar 25 16:23 /dev/sda1 brw-rw---- 1 root disk 8, 2 Mar 25 16:23 /dev/sda2 brw-rw---- 1 root disk 8, 3 Mar 25 16:23 /dev/sda3 brw-rw---- 1 root disk 8, 16 Mar 25 16:23 /dev/sdb brw-rw---- 1 root disk 8, 32 Mar 25 16:23 /dev/sdc brw-rw---- 1 root disk 8, 48 Mar 25 16:23 /dev/sdd brw-rw---- 1 root disk 8, 49 Mar 25 16:23 /dev/sdd1 brw-rw---- 1 ceph ceph 8, 50 Mar 25 16:23 /dev/sdd2 brw-rw---- 1 root disk 8, 64 Mar 25 16:23 /dev/sde brw-rw---- 1 root disk 8, 65 Mar 25 16:23 /dev/sde1 brw-rw---- 1 ceph ceph 8, 66 Mar 25 16:23 /dev/sde2 brw-rw---- 1 root disk 8, 80 Mar 25 16:23 /dev/sdf brw-rw---- 1 root disk 8, 81 Mar 25 16:23 /dev/sdf1 brw-rw---- 1 ceph ceph 8, 82 Mar 25 16:23 /dev/sdf2 brw-rw---- 1 root disk 8, 96 Mar 25 16:23 /dev/sdg brw-rw---- 1 root disk 8, 97 Mar 25 16:23 /dev/sdg1 brw-rw---- 1 ceph ceph 8, 98 Mar 25 16:23 /dev/sdg2 brw-rw---- 1 root disk 8, 112 Mar 25 16:23 /dev/sdh brw-rw---- 1 root disk 8, 113 Mar 25 16:23 /dev/sdh1 brw-rw---- 1 ceph ceph 8, 114 Mar 25 16:23 /dev/sdh2 brw-rw---- 1 root disk 8, 128 Mar 25 16:23 /dev/sdi brw-rw---- 1 root disk 8, 129 Mar 25 16:23 /dev/sdi1 brw-rw---- 1 ceph ceph 8, 130 Mar 25 16:23 /dev/sdi2 brw-rw---- 1 root disk 8, 144 Mar 25 16:23 /dev/sdj brw-rw---- 1 root disk 8, 160 Mar 25 16:23 /dev/sdk brw-rw---- 1 root disk 8, 161 Mar 25 16:23 /dev/sdk1 brw-rw---- 1 ceph ceph 8, 162 Mar 25 16:23 /dev/sdk2 brw-rw---- 1 root disk 8, 176 Mar 25 16:23 /dev/sdl brw-rw---- 1 root disk 8, 192 Mar 25 16:23 /dev/sdm brw-rw---- 1 root disk 8, 193 Mar 25 16:23 /dev/sdm1 brw-rw---- 1 ceph ceph 8, 194 Mar 25 16:23 /dev/sdm2 Did i miss to convert to bluestore or something? On Wed, Mar 25, 2020 at 4:23 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > Try this > > chown ceph.ceph /dev/sdc2 > chown ceph.ceph /dev/sdd2 > chown ceph.ceph /dev/sde2 > chown ceph.ceph /dev/sdf2 > chown ceph.ceph /dev/sdg2 > chown ceph.ceph /dev/sdh2 > > > > -----Original Message----- > From: Ml Ml [mailto:mliebherr99@googlemail.com] > Sent: 25 March 2020 16:22 > To: Marc Roos > Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > Nautilus > > They where indeed disabled. I enabled them with: > > systemctl enable ceph-osd@4 > systemctl enable ceph-osd@15 > systemctl enable ceph-osd@17 > systemctl enable ceph-osd@20 > systemctl enable ceph-osd@21 > systemctl enable ceph-osd@27 > systemctl enable ceph-osd@32 > > But they still wont start. > > On Wed, Mar 25, 2020 at 4:09 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> > wrote: > > > > > > I had something similar. My osd were disabled, maybe this installer of > > > nautilus does that check > > > > systemctl is-enabled ceph-osd@0 > > > > https://tracker.ceph.com/issues/44102 > > > > > > > > > > -----Original Message----- > > From: Ml Ml [mailto:mliebherr99@googlemail.com] > > Sent: 25 March 2020 16:05 > > To: ceph-users > > Subject: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > > Nautilus > > > > Hello list, > > > > i upgraded to Debian 10, after that i upgraded from luminous to > > nautilus. > > I restarted the mons, then the OSDs. > > > > Everything was up and healthy. > > After rebooting a node, only 3/10 OSD start up: > > > > -4 20.07686 host ceph03 > > 4 hdd 2.67020 osd.4 down 1.00000 1.00000 > > 5 hdd 1.71660 osd.5 up 1.00000 1.00000 > > 6 hdd 1.71660 osd.6 up 1.00000 1.00000 > > 10 hdd 2.67029 osd.10 up 1.00000 1.00000 > > 15 hdd 2.00000 osd.15 down 1.00000 1.00000 > > 17 hdd 1.20000 osd.17 down 1.00000 1.00000 > > 20 hdd 1.71649 osd.20 down 1.00000 1.00000 > > 21 hdd 2.00000 osd.21 down 1.00000 1.00000 > > 27 hdd 1.71649 osd.27 down 1.00000 1.00000 > > 32 hdd 2.67020 osd.32 down 1.00000 1.00000 > > > > root@ceph03:~# /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser > > ceph --setgroup ceph > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > > > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x56531c50a140) > > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > > > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x7ffd04120468) > > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > > > failed to fetch mon config (--no-mon-config to skip) > > > > root@ceph03:~# df > > Filesystem 1K-blocks Used Available Use% Mounted on > > udev 24624580 0 24624580 0% /dev > > tmpfs 4928216 9544 4918672 1% /run > > /dev/sda3 47930248 5209760 40262684 12% / > > tmpfs 24641068 0 24641068 0% /dev/shm > > tmpfs 5120 0 5120 0% /run/lock > > tmpfs 24641068 0 24641068 0% /sys/fs/cgroup > > /dev/sda1 944120 144752 734192 17% /boot > > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-1 > > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-6 > > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-5 > > tmpfs 24641068 24 24641044 1% > > /var/lib/ceph/osd/ceph-10 > > tmpfs 4928212 0 4928212 0% /run/user/0 > > > > root@ceph03:~# ceph-volume lvm list > > > > > > ====== osd.1 ======= > > > > [block] > > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > > -4 > > 7bd-8ddc-102b98444067 > > > > block device > > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > > -4 > > 7bd-8ddc-102b98444067 > > block uuid HSK6Da-elP2-CFYz-s0RH-UNiw-bey0-dVcml1 > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid b4987093-4fa5-47bd-8ddc-102b98444067 > > osd id 1 > > type block > > vdo 0 > > devices /dev/sdj > > > > ====== osd.10 ====== > > > > [block] > > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > > -4 > > b85-9799-988eee3b2b3f > > > > block device > > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > > -4 > > b85-9799-988eee3b2b3f > > block uuid 440fNG-guO2-l1WJ-m5cR-GUkz-ZTUd-Fcz5Ml > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid fa241441-1758-4b85-9799-988eee3b2b3f > > osd id 10 > > type block > > vdo 0 > > devices /dev/sdl > > > > ====== osd.5 ======= > > > > [block] > > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > > -4 > > e50-9bb5-775bacd854af > > > > block device > > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > > -4 > > e50-9bb5-775bacd854af > > block uuid Z6VeNx-S9sg-ZOsh-HTw9-ykTc-YBrh-qFwz5i > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid 112e0c75-f61b-4e50-9bb5-775bacd854af > > osd id 5 > > type block > > vdo 0 > > devices /dev/sdb > > > > ====== osd.6 ======= > > > > [block] > > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > > -4 > > 2ab-aeb0-f2915a389e48 > > > > block device > > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > > -4 > > 2ab-aeb0-f2915a389e48 > > block uuid G9qHxC-dN0b-XBes-QVss-Bzwa-7Xtw-ikksgM > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid 35288829-c1f6-42ab-aeb0-f2915a389e48 > > osd id 6 > > type block > > vdo 0 > > devices /dev/sdc > > > > I would mount it and run the osd deamon manually, but ceph-disk list > > seems to be gone in Nautilus. Therefore i dont know where to mount > what. > > > > Any ideas on that? > > > > > > Cheers, > > Michael > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > > > > >

4 years, 1 month

2
2
0 0

Re: OSDs wont mount on Debian 10 (Buster) with Nautilus

by Marc Roos

Try this chown ceph.ceph /dev/sdc2 chown ceph.ceph /dev/sdd2 chown ceph.ceph /dev/sde2 chown ceph.ceph /dev/sdf2 chown ceph.ceph /dev/sdg2 chown ceph.ceph /dev/sdh2 -----Original Message----- From: Ml Ml [mailto:mliebherr99@googlemail.com] Sent: 25 March 2020 16:22 To: Marc Roos Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with Nautilus They where indeed disabled. I enabled them with: systemctl enable ceph-osd@4 systemctl enable ceph-osd@15 systemctl enable ceph-osd@17 systemctl enable ceph-osd@20 systemctl enable ceph-osd@21 systemctl enable ceph-osd@27 systemctl enable ceph-osd@32 But they still wont start. On Wed, Mar 25, 2020 at 4:09 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I had something similar. My osd were disabled, maybe this installer of > nautilus does that check > > systemctl is-enabled ceph-osd@0 > > https://tracker.ceph.com/issues/44102 > > > > > -----Original Message----- > From: Ml Ml [mailto:mliebherr99@googlemail.com] > Sent: 25 March 2020 16:05 > To: ceph-users > Subject: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > Nautilus > > Hello list, > > i upgraded to Debian 10, after that i upgraded from luminous to > nautilus. > I restarted the mons, then the OSDs. > > Everything was up and healthy. > After rebooting a node, only 3/10 OSD start up: > > -4 20.07686 host ceph03 > 4 hdd 2.67020 osd.4 down 1.00000 1.00000 > 5 hdd 1.71660 osd.5 up 1.00000 1.00000 > 6 hdd 1.71660 osd.6 up 1.00000 1.00000 > 10 hdd 2.67029 osd.10 up 1.00000 1.00000 > 15 hdd 2.00000 osd.15 down 1.00000 1.00000 > 17 hdd 1.20000 osd.17 down 1.00000 1.00000 > 20 hdd 1.71649 osd.20 down 1.00000 1.00000 > 21 hdd 2.00000 osd.21 down 1.00000 1.00000 > 27 hdd 1.71649 osd.27 down 1.00000 1.00000 > 32 hdd 2.67020 osd.32 down 1.00000 1.00000 > > root@ceph03:~# /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser > ceph --setgroup ceph > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x56531c50a140) > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x7ffd04120468) > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > failed to fetch mon config (--no-mon-config to skip) > > root@ceph03:~# df > Filesystem 1K-blocks Used Available Use% Mounted on > udev 24624580 0 24624580 0% /dev > tmpfs 4928216 9544 4918672 1% /run > /dev/sda3 47930248 5209760 40262684 12% / > tmpfs 24641068 0 24641068 0% /dev/shm > tmpfs 5120 0 5120 0% /run/lock > tmpfs 24641068 0 24641068 0% /sys/fs/cgroup > /dev/sda1 944120 144752 734192 17% /boot > tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-1 > tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-6 > tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-5 > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-10 > tmpfs 4928212 0 4928212 0% /run/user/0 > > root@ceph03:~# ceph-volume lvm list > > > ====== osd.1 ======= > > [block] > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > -4 > 7bd-8ddc-102b98444067 > > block device > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > -4 > 7bd-8ddc-102b98444067 > block uuid HSK6Da-elP2-CFYz-s0RH-UNiw-bey0-dVcml1 > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid b4987093-4fa5-47bd-8ddc-102b98444067 > osd id 1 > type block > vdo 0 > devices /dev/sdj > > ====== osd.10 ====== > > [block] > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > -4 > b85-9799-988eee3b2b3f > > block device > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > -4 > b85-9799-988eee3b2b3f > block uuid 440fNG-guO2-l1WJ-m5cR-GUkz-ZTUd-Fcz5Ml > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid fa241441-1758-4b85-9799-988eee3b2b3f > osd id 10 > type block > vdo 0 > devices /dev/sdl > > ====== osd.5 ======= > > [block] > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > -4 > e50-9bb5-775bacd854af > > block device > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > -4 > e50-9bb5-775bacd854af > block uuid Z6VeNx-S9sg-ZOsh-HTw9-ykTc-YBrh-qFwz5i > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid 112e0c75-f61b-4e50-9bb5-775bacd854af > osd id 5 > type block > vdo 0 > devices /dev/sdb > > ====== osd.6 ======= > > [block] > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > -4 > 2ab-aeb0-f2915a389e48 > > block device > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > -4 > 2ab-aeb0-f2915a389e48 > block uuid G9qHxC-dN0b-XBes-QVss-Bzwa-7Xtw-ikksgM > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid 35288829-c1f6-42ab-aeb0-f2915a389e48 > osd id 6 > type block > vdo 0 > devices /dev/sdc > > I would mount it and run the osd deamon manually, but ceph-disk list > seems to be gone in Nautilus. Therefore i dont know where to mount what. > > Any ideas on that? > > > Cheers, > Michael > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

4 years, 1 month

1
0
0 0

OSDs wont mount on Debian 10 (Buster) with Nautilus

by Ml Ml

Hello list, i upgraded to Debian 10, after that i upgraded from luminous to nautilus. I restarted the mons, then the OSDs. Everything was up and healthy. After rebooting a node, only 3/10 OSD start up: -4 20.07686 host ceph03 4 hdd 2.67020 osd.4 down 1.00000 1.00000 5 hdd 1.71660 osd.5 up 1.00000 1.00000 6 hdd 1.71660 osd.6 up 1.00000 1.00000 10 hdd 2.67029 osd.10 up 1.00000 1.00000 15 hdd 2.00000 osd.15 down 1.00000 1.00000 17 hdd 1.20000 osd.17 down 1.00000 1.00000 20 hdd 1.71649 osd.20 down 1.00000 1.00000 21 hdd 2.00000 osd.21 down 1.00000 1.00000 27 hdd 1.71649 osd.27 down 1.00000 1.00000 32 hdd 2.67020 osd.32 down 1.00000 1.00000 root@ceph03:~# /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x56531c50a140) no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x7ffd04120468) no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx failed to fetch mon config (--no-mon-config to skip) root@ceph03:~# df Filesystem 1K-blocks Used Available Use% Mounted on udev 24624580 0 24624580 0% /dev tmpfs 4928216 9544 4918672 1% /run /dev/sda3 47930248 5209760 40262684 12% / tmpfs 24641068 0 24641068 0% /dev/shm tmpfs 5120 0 5120 0% /run/lock tmpfs 24641068 0 24641068 0% /sys/fs/cgroup /dev/sda1 944120 144752 734192 17% /boot tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-1 tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-6 tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-5 tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-10 tmpfs 4928212 0 4928212 0% /run/user/0 root@ceph03:~# ceph-volume lvm list ====== osd.1 ======= [block] /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5-47bd-8ddc-102b98444067 block device /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5-47bd-8ddc-102b98444067 block uuid HSK6Da-elP2-CFYz-s0RH-UNiw-bey0-dVcml1 cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid b4987093-4fa5-47bd-8ddc-102b98444067 osd id 1 type block vdo 0 devices /dev/sdj ====== osd.10 ====== [block] /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758-4b85-9799-988eee3b2b3f block device /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758-4b85-9799-988eee3b2b3f block uuid 440fNG-guO2-l1WJ-m5cR-GUkz-ZTUd-Fcz5Ml cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid fa241441-1758-4b85-9799-988eee3b2b3f osd id 10 type block vdo 0 devices /dev/sdl ====== osd.5 ======= [block] /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b-4e50-9bb5-775bacd854af block device /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b-4e50-9bb5-775bacd854af block uuid Z6VeNx-S9sg-ZOsh-HTw9-ykTc-YBrh-qFwz5i cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid 112e0c75-f61b-4e50-9bb5-775bacd854af osd id 5 type block vdo 0 devices /dev/sdb ====== osd.6 ======= [block] /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6-42ab-aeb0-f2915a389e48 block device /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6-42ab-aeb0-f2915a389e48 block uuid G9qHxC-dN0b-XBes-QVss-Bzwa-7Xtw-ikksgM cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid 35288829-c1f6-42ab-aeb0-f2915a389e48 osd id 6 type block vdo 0 devices /dev/sdc I would mount it and run the osd deamon manually, but ceph-disk list seems to be gone in Nautilus. Therefore i dont know where to mount what. Any ideas on that? Cheers, Michael

4 years, 1 month

2
1
0 0

March Ceph Science User Group Virtual Meeting

by Kevin Hrpcek

Hello, We will be having a Ceph science/research/big cluster call on Wednesday March 25th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20200325 Due to the worldwide effects of covid-19 I'm thinking it won't hurt to try to host this call. If only a few people join the call we can then decide to continue or cancel it. Ceph calendar event details: March 25, 2020 9am US Central 4pm Central Eurpean We try to keep it to an hour or less. Description:Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index <https://www.google.com/url?q=https://pad.ceph.com/p/Ceph_Science_User_Group…> Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink <https://www.google.com/url?q=https://bluejeans.com/908675367?src%3Dcalendar…> To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc <https://www.google.com/url?q=http://bjn.vc&sa=D&ust=1579363980705000&usg=AO…> 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers <https://www.google.com/url?q=https://www.redhat.com/en/conference-numbers&s…> 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 <https://www.google.com/url?q=https://bluejeans.com/111&sa=D&ust=15793639807…> Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

4 years, 1 month

1
1
0 0

Re: How can I recover PGs in state 'unknown', where OSD location seems to be lost?

by Mark S. Holliman

So I've managed to use ceph-objectstore-tool to locate the pgs in 'unknown' state on the OSDs, but how do I tell the rest of the system where to find them? Is there a command for setting a the OSDs associated with a PG? Or, less ideally, is there a table somewhere I can hack to do this by hand? Mark Holliman Wide Field Astronomy Unit Institute for Astronomy University of Edinburgh The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

4 years, 1 month

1
0
0 0

Re: Ceph fully crash and we unable to recovery

by Parker Lau

Hello Sir/Madam, We are facing the serious problem for our proxmox with Ceph. I have already submitted the ticket to Proxmox but they said that only option trying to recover the mondb.We would like to know any some suggestion in our situation. So far the only option that I see would be in trying to recover the mondb from an OSD. But this action is usually a last resort. Since I don't know the outcome, the cluster could then very well be dead and thus all data lost. https://docs.ceph.com/docs/luminous/rados/troubleshooting/troubleshooting-m… --- > > I would like to set the nodown on the cluster to see if the OSDs are kept in the cluster. > > The OSDs are joining the cluster but are set as down shortly after. > > > ok . Please go ahead. Sadly this didn't have any effect either. But I think I found a clue to what might be going on. # ceph-osd.0.log 2020-03-24 21:22:06.462100 7fb33aab0e00 10 osd.0 0 read_superblock sb(e8e81549-91e5-4370-b091-9500f406a2b2 osd.0 0bb2b9bb-9a70-4d6f-8d4e-3fc5049d63d6 e14334 [13578,14334] lci=[0,14334]) # ceph-mon.cccs01.log 2020-03-24 21:26:48.038345 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 e14299: 48 total, 13 up, 35 in 2020-03-24 21:26:48.038351 7f7ef791a700 5 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 can_mark_out current in_ratio 0.729167 < min 0.75, will not mark osds out 2020-03-24 21:26:48.038360 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 tick NOOUT flag set, not checking down osds 2020-03-24 21:26:48.038364 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 min_last_epoch_clean 0 # ceph-mon.cccs06.log 2020-03-22 22:26:57.056939 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14333 e14333: 48 total, 48 up, 48 in 2020-03-22 22:27:04.113054 7f3c1993a700 0 mon.cccs06@5(peon)<mailto:mon.cccs06@5(peon)> e31 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1 2020-03-22 22:27:04.113086 7f3c1993a700 0 log_channel(audit) log [DBG] : from='client.? 10.1.14.8:0/4265796352' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch 2020-03-22 22:27:09.752027 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14334 e14334: 48 total, 48 up, 48 in ... 2020-03-23 10:42:51.891722 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891729 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891730 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 1009089991638532096, adjusting msgr requires 2020-03-23 10:42:51.891732 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires It seems that the OSD have a epoch of e14334 but the MONs seems to have e14269 for the OSDs. I could only find the e14334 on the ceph-mon.cccs06. The cccs06 has been the last MON standing (the node also reset). But when the cluster came back the MONs with the old epoch came up first and joined. The syslog showed that these were the last log entries written. So the nodes reset shortly after. This would fit to cccs06 being the last MON alive. # cccs01 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: leader is 1/2502 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: synced members: 1/2502, 5/2219 # cccs02 Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] notice: we (3/2514) left the process group Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] crit: leaving CPG group # cccs06 Mar 22 22:31:16 cccs06 pmxcfs[2662]: [status] noMar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'iscsi_tcp' Mar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'ib_iser' There must have been some issue prior to the reset as I found those error messages. They could explain why no older epoch was written anymore. # cccs01 2020-03-22 22:22:15.661060 7fc09c85c100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs01/store.db/LOCK: Permission denied 2020-03-22 22:22:15.661067 7fc09c85c100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs01': (22) Invalid argument # cccs02 2020-03-22 22:31:11.209524 7fd034786100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs02/store.db/LOCK: Permission denied 2020-03-22 22:31:11.209541 7fd034786100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs02': (22) Invalid argument # cccs06 no such entries. From my point of view, the last question is now: How to get the epoch from the OSD into the MON DB. I have no answer to this yet. Best Regards, Parker Lau ReadySpace Ltd - Cloud and Managed Hosting Professionals Direct: +852 3726 1120 Hotiline: +852 3568 3372<tel:+852%203568%203372> Fax: +852 3568 3376<tel:+852%203568%203376> Website: www.readyspace.com.hk<http://www.readyspace.com.hk/> Helpdesk: helpdesk.readyspace.com<http://helpdesk.readyspace.com/> Get the latest update and promotion here: twitter.com/readyspace<http://twitter.com/readyspace> | facebook.com/readyspace<http://www.facebook.com/readyspace> [cid:image001.jpg@01D5CC61.CF90E050]Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. From: Parker Lau Sent: Wednesday, March 25, 2020 1:00 AM To: ceph-users(a)ceph.io Subject: 回覆: Ceph fully crash and we unable to recovery Hello Sir/Madam, We are facing the serious problem for our proxmox with Ceph. I have already submitted the ticket to Proxmox but they said that only option trying to recover the mondb.We would like to know any some suggestion in our situation. So far the only option that I see would be in trying to recover the mondb from an OSD. But this action is usually a last resort. Since I don't know the outcome, the cluster could then very well be dead and thus all data lost. https://docs.ceph.com/docs/luminous/rados/troubleshooting/troubleshooting-m… --- > > I would like to set the nodown on the cluster to see if the OSDs are kept in the cluster. > > The OSDs are joining the cluster but are set as down shortly after. > > > ok . Please go ahead. Sadly this didn't have any effect either. But I think I found a clue to what might be going on. # ceph-osd.0.log 2020-03-24 21:22:06.462100 7fb33aab0e00 10 osd.0 0 read_superblock sb(e8e81549-91e5-4370-b091-9500f406a2b2 osd.0 0bb2b9bb-9a70-4d6f-8d4e-3fc5049d63d6 e14334 [13578,14334] lci=[0,14334]) # ceph-mon.cccs01.log 2020-03-24 21:26:48.038345 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 e14299: 48 total, 13 up, 35 in 2020-03-24 21:26:48.038351 7f7ef791a700 5 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 can_mark_out current in_ratio 0.729167 < min 0.75, will not mark osds out 2020-03-24 21:26:48.038360 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 tick NOOUT flag set, not checking down osds 2020-03-24 21:26:48.038364 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 min_last_epoch_clean 0 # ceph-mon.cccs06.log 2020-03-22 22:26:57.056939 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14333 e14333: 48 total, 48 up, 48 in 2020-03-22 22:27:04.113054 7f3c1993a700 0 mon.cccs06@5(peon)<mailto:mon.cccs06@5(peon)> e31 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1 2020-03-22 22:27:04.113086 7f3c1993a700 0 log_channel(audit) log [DBG] : from='client.? 10.1.14.8:0/4265796352' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch 2020-03-22 22:27:09.752027 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14334 e14334: 48 total, 48 up, 48 in ... 2020-03-23 10:42:51.891722 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891729 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891730 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 1009089991638532096, adjusting msgr requires 2020-03-23 10:42:51.891732 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires It seems that the OSD have a epoch of e14334 but the MONs seems to have e14269 for the OSDs. I could only find the e14334 on the ceph-mon.cccs06. The cccs06 has been the last MON standing (the node also reset). But when the cluster came back the MONs with the old epoch came up first and joined. The syslog showed that these were the last log entries written. So the nodes reset shortly after. This would fit to cccs06 being the last MON alive. # cccs01 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: leader is 1/2502 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: synced members: 1/2502, 5/2219 # cccs02 Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] notice: we (3/2514) left the process group Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] crit: leaving CPG group # cccs06 Mar 22 22:31:16 cccs06 pmxcfs[2662]: [status] noMar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'iscsi_tcp' Mar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'ib_iser' There must have been some issue prior to the reset as I found those error messages. They could explain why no older epoch was written anymore. # cccs01 2020-03-22 22:22:15.661060 7fc09c85c100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs01/store.db/LOCK: Permission denied 2020-03-22 22:22:15.661067 7fc09c85c100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs01': (22) Invalid argument # cccs02 2020-03-22 22:31:11.209524 7fd034786100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs02/store.db/LOCK: Permission denied 2020-03-22 22:31:11.209541 7fd034786100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs02': (22) Invalid argument # cccs06 no such entries. From my point of view, the last question is now: How to get the epoch from the OSD into the MON DB. I have no answer to this yet. Best Regards, Parker Lau ReadySpace Ltd - Cloud and Managed Hosting Professionals Direct: +852 3726 1120 Hotiline: +852 3568 3372<tel:+852%203568%203372> Fax: +852 3568 3376<tel:+852%203568%203376> Website: www.readyspace.com.hk<http://www.readyspace.com.hk/> Helpdesk: helpdesk.readyspace.com<http://helpdesk.readyspace.com/> Get the latest update and promotion here: twitter.com/readyspace<http://twitter.com/readyspace> | facebook.com/readyspace<http://www.facebook.com/readyspace> [cid:image001.jpg@01D5CC61.CF90E050]Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. CONFIDENTIALITY: This message is intended for the recipient named above. It may contain confidential or privileged information. If you are not the intended recipient, please notify the sender immediately by replying to this message and then delete it from your system. Do not read, copy, use or circulate this communication.   SECURITY: Internet communications are not secure. While reasonable effort has been made to ensure that this communication has not been tampered with, the sender cannot be responsible for alterations made to the contents of this message without his/her express consent. The sender also cannot be responsible for any computer virus/worm/ trojan horse or other program(s) accompanying this message which he/she is unaware of. Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. 寄件者: Parker Lau 寄件日期: Wednesday, 25 March 2020 12:55 am 收件者: 'ceph-fs(a)ceph.io' <ceph-fs(a)ceph.io<mailto:ceph-fs@ceph.io>> 主旨: Ceph fully crash and we unable to recovery Hello Sir/Madam, We are facing the serious problem for our proxmox with Ceph. I have already submitted the ticket to Proxmox but they said that only option trying to recover the mondb.We would like to know any some suggestion in our situation. So far the only option that I see would be in trying to recover the mondb from an OSD. But this action is usually a last resort. Since I don't know the outcome, the cluster could then very well be dead and thus all data lost. https://docs.ceph.com/docs/luminous/rados/troubleshooting/troubleshooting-m… --- > > I would like to set the nodown on the cluster to see if the OSDs are kept in the cluster. > > The OSDs are joining the cluster but are set as down shortly after. > > > ok . Please go ahead. Sadly this didn't have any effect either. But I think I found a clue to what might be going on. # ceph-osd.0.log 2020-03-24 21:22:06.462100 7fb33aab0e00 10 osd.0 0 read_superblock sb(e8e81549-91e5-4370-b091-9500f406a2b2 osd.0 0bb2b9bb-9a70-4d6f-8d4e-3fc5049d63d6 e14334 [13578,14334] lci=[0,14334]) # ceph-mon.cccs01.log 2020-03-24 21:26:48.038345 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 e14299: 48 total, 13 up, 35 in 2020-03-24 21:26:48.038351 7f7ef791a700 5 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 can_mark_out current in_ratio 0.729167 < min 0.75, will not mark osds out 2020-03-24 21:26:48.038360 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 tick NOOUT flag set, not checking down osds 2020-03-24 21:26:48.038364 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 min_last_epoch_clean 0 # ceph-mon.cccs06.log 2020-03-22 22:26:57.056939 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14333 e14333: 48 total, 48 up, 48 in 2020-03-22 22:27:04.113054 7f3c1993a700 0 mon.cccs06@5(peon)<mailto:mon.cccs06@5(peon)> e31 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1 2020-03-22 22:27:04.113086 7f3c1993a700 0 log_channel(audit) log [DBG] : from='client.? 10.1.14.8:0/4265796352' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch 2020-03-22 22:27:09.752027 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14334 e14334: 48 total, 48 up, 48 in ... 2020-03-23 10:42:51.891722 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891729 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891730 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 1009089991638532096, adjusting msgr requires 2020-03-23 10:42:51.891732 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires It seems that the OSD have a epoch of e14334 but the MONs seems to have e14269 for the OSDs. I could only find the e14334 on the ceph-mon.cccs06. The cccs06 has been the last MON standing (the node also reset). But when the cluster came back the MONs with the old epoch came up first and joined. The syslog showed that these were the last log entries written. So the nodes reset shortly after. This would fit to cccs06 being the last MON alive. # cccs01 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: leader is 1/2502 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: synced members: 1/2502, 5/2219 # cccs02 Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] notice: we (3/2514) left the process group Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] crit: leaving CPG group # cccs06 Mar 22 22:31:16 cccs06 pmxcfs[2662]: [status] noMar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'iscsi_tcp' Mar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'ib_iser' There must have been some issue prior to the reset as I found those error messages. They could explain why no older epoch was written anymore. # cccs01 2020-03-22 22:22:15.661060 7fc09c85c100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs01/store.db/LOCK: Permission denied 2020-03-22 22:22:15.661067 7fc09c85c100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs01': (22) Invalid argument # cccs02 2020-03-22 22:31:11.209524 7fd034786100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs02/store.db/LOCK: Permission denied 2020-03-22 22:31:11.209541 7fd034786100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs02': (22) Invalid argument # cccs06 no such entries. From my point of view, the last question is now: How to get the epoch from the OSD into the MON DB. I have no answer to this yet. Best Regards, Parker Lau ReadySpace Ltd - Cloud and Managed Hosting Professionals Direct: +852 3726 1120 Hotiline: +852 3568 3372<tel:+852%203568%203372> Fax: +852 3568 3376<tel:+852%203568%203376> Website: www.readyspace.com.hk<http://www.readyspace.com.hk/> Helpdesk: helpdesk.readyspace.com<http://helpdesk.readyspace.com/> Get the latest update and promotion here: twitter.com/readyspace<http://twitter.com/readyspace> | facebook.com/readyspace<http://www.facebook.com/readyspace> [cid:image001.jpg@01D5CC61.CF90E050]Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. CONFIDENTIALITY: This message is intended for the recipient named above. It may contain confidential or privileged information. If you are not the intended recipient, please notify the sender immediately by replying to this message and then delete it from your system. Do not read, copy, use or circulate this communication.   SECURITY: Internet communications are not secure. While reasonable effort has been made to ensure that this communication has not been tampered with, the sender cannot be responsible for alterations made to the contents of this message without his/her express consent. The sender also cannot be responsible for any computer virus/worm/ trojan horse or other program(s) accompanying this message which he/she is unaware of. Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately.

4 years, 1 month

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users March 2020