- ceph-users - lists.ceph.io

Re: OSDs wont mount on Debian 10 (Buster) with Nautilus
by Marc Roos 25 Mar '20

25 Mar '20

Try this chown ceph.ceph /dev/sdc2 chown ceph.ceph /dev/sdd2 chown ceph.ceph /dev/sde2 chown ceph.ceph /dev/sdf2 chown ceph.ceph /dev/sdg2 chown ceph.ceph /dev/sdh2 -----Original Message----- From: Ml Ml [mailto:mliebherr99@googlemail.com] Sent: 25 March 2020 16:22 To: Marc Roos Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with Nautilus They where indeed disabled. I enabled them with: systemctl enable ceph-osd@4 systemctl enable ceph-osd@15 systemctl enable ceph-osd@17 systemctl enable ceph-osd@20 systemctl enable ceph-osd@21 systemctl enable ceph-osd@27 systemctl enable ceph-osd@32 But they still wont start. On Wed, Mar 25, 2020 at 4:09 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I had something similar. My osd were disabled, maybe this installer of > nautilus does that check > > systemctl is-enabled ceph-osd@0 > > https://tracker.ceph.com/issues/44102 > > > > > -----Original Message----- > From: Ml Ml [mailto:mliebherr99@googlemail.com] > Sent: 25 March 2020 16:05 > To: ceph-users > Subject: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > Nautilus > > Hello list, > > i upgraded to Debian 10, after that i upgraded from luminous to > nautilus. > I restarted the mons, then the OSDs. > > Everything was up and healthy. > After rebooting a node, only 3/10 OSD start up: > > -4 20.07686 host ceph03 > 4 hdd 2.67020 osd.4 down 1.00000 1.00000 > 5 hdd 1.71660 osd.5 up 1.00000 1.00000 > 6 hdd 1.71660 osd.6 up 1.00000 1.00000 > 10 hdd 2.67029 osd.10 up 1.00000 1.00000 > 15 hdd 2.00000 osd.15 down 1.00000 1.00000 > 17 hdd 1.20000 osd.17 down 1.00000 1.00000 > 20 hdd 1.71649 osd.20 down 1.00000 1.00000 > 21 hdd 2.00000 osd.21 down 1.00000 1.00000 > 27 hdd 1.71649 osd.27 down 1.00000 1.00000 > 32 hdd 2.67020 osd.32 down 1.00000 1.00000 > > root@ceph03:~# /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser > ceph --setgroup ceph > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x56531c50a140) > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x7ffd04120468) > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > failed to fetch mon config (--no-mon-config to skip) > > root@ceph03:~# df > Filesystem 1K-blocks Used Available Use% Mounted on > udev 24624580 0 24624580 0% /dev > tmpfs 4928216 9544 4918672 1% /run > /dev/sda3 47930248 5209760 40262684 12% / > tmpfs 24641068 0 24641068 0% /dev/shm > tmpfs 5120 0 5120 0% /run/lock > tmpfs 24641068 0 24641068 0% /sys/fs/cgroup > /dev/sda1 944120 144752 734192 17% /boot > tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-1 > tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-6 > tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-5 > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-10 > tmpfs 4928212 0 4928212 0% /run/user/0 > > root@ceph03:~# ceph-volume lvm list > > > ====== osd.1 ======= > > [block] > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > -4 > 7bd-8ddc-102b98444067 > > block device > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > -4 > 7bd-8ddc-102b98444067 > block uuid HSK6Da-elP2-CFYz-s0RH-UNiw-bey0-dVcml1 > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid b4987093-4fa5-47bd-8ddc-102b98444067 > osd id 1 > type block > vdo 0 > devices /dev/sdj > > ====== osd.10 ====== > > [block] > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > -4 > b85-9799-988eee3b2b3f > > block device > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > -4 > b85-9799-988eee3b2b3f > block uuid 440fNG-guO2-l1WJ-m5cR-GUkz-ZTUd-Fcz5Ml > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid fa241441-1758-4b85-9799-988eee3b2b3f > osd id 10 > type block > vdo 0 > devices /dev/sdl > > ====== osd.5 ======= > > [block] > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > -4 > e50-9bb5-775bacd854af > > block device > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > -4 > e50-9bb5-775bacd854af > block uuid Z6VeNx-S9sg-ZOsh-HTw9-ykTc-YBrh-qFwz5i > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid 112e0c75-f61b-4e50-9bb5-775bacd854af > osd id 5 > type block > vdo 0 > devices /dev/sdb > > ====== osd.6 ======= > > [block] > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > -4 > 2ab-aeb0-f2915a389e48 > > block device > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > -4 > 2ab-aeb0-f2915a389e48 > block uuid G9qHxC-dN0b-XBes-QVss-Bzwa-7Xtw-ikksgM > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid 35288829-c1f6-42ab-aeb0-f2915a389e48 > osd id 6 > type block > vdo 0 > devices /dev/sdc > > I would mount it and run the osd deamon manually, but ceph-disk list > seems to be gone in Nautilus. Therefore i dont know where to mount what. > > Any ideas on that? > > > Cheers, > Michael > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

1 0

OSDs wont mount on Debian 10 (Buster) with Nautilus
by Ml Ml 25 Mar '20

25 Mar '20

Hello list, i upgraded to Debian 10, after that i upgraded from luminous to nautilus. I restarted the mons, then the OSDs. Everything was up and healthy. After rebooting a node, only 3/10 OSD start up: -4 20.07686 host ceph03 4 hdd 2.67020 osd.4 down 1.00000 1.00000 5 hdd 1.71660 osd.5 up 1.00000 1.00000 6 hdd 1.71660 osd.6 up 1.00000 1.00000 10 hdd 2.67029 osd.10 up 1.00000 1.00000 15 hdd 2.00000 osd.15 down 1.00000 1.00000 17 hdd 1.20000 osd.17 down 1.00000 1.00000 20 hdd 1.71649 osd.20 down 1.00000 1.00000 21 hdd 2.00000 osd.21 down 1.00000 1.00000 27 hdd 1.71649 osd.27 down 1.00000 1.00000 32 hdd 2.67020 osd.32 down 1.00000 1.00000 root@ceph03:~# /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x56531c50a140) no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x7ffd04120468) no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx failed to fetch mon config (--no-mon-config to skip) root@ceph03:~# df Filesystem 1K-blocks Used Available Use% Mounted on udev 24624580 0 24624580 0% /dev tmpfs 4928216 9544 4918672 1% /run /dev/sda3 47930248 5209760 40262684 12% / tmpfs 24641068 0 24641068 0% /dev/shm tmpfs 5120 0 5120 0% /run/lock tmpfs 24641068 0 24641068 0% /sys/fs/cgroup /dev/sda1 944120 144752 734192 17% /boot tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-1 tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-6 tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-5 tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-10 tmpfs 4928212 0 4928212 0% /run/user/0 root@ceph03:~# ceph-volume lvm list ====== osd.1 ======= [block] /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5-47bd-8ddc-102b98444067 block device /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5-47bd-8ddc-102b98444067 block uuid HSK6Da-elP2-CFYz-s0RH-UNiw-bey0-dVcml1 cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid b4987093-4fa5-47bd-8ddc-102b98444067 osd id 1 type block vdo 0 devices /dev/sdj ====== osd.10 ====== [block] /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758-4b85-9799-988eee3b2b3f block device /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758-4b85-9799-988eee3b2b3f block uuid 440fNG-guO2-l1WJ-m5cR-GUkz-ZTUd-Fcz5Ml cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid fa241441-1758-4b85-9799-988eee3b2b3f osd id 10 type block vdo 0 devices /dev/sdl ====== osd.5 ======= [block] /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b-4e50-9bb5-775bacd854af block device /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b-4e50-9bb5-775bacd854af block uuid Z6VeNx-S9sg-ZOsh-HTw9-ykTc-YBrh-qFwz5i cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid 112e0c75-f61b-4e50-9bb5-775bacd854af osd id 5 type block vdo 0 devices /dev/sdb ====== osd.6 ======= [block] /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6-42ab-aeb0-f2915a389e48 block device /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6-42ab-aeb0-f2915a389e48 block uuid G9qHxC-dN0b-XBes-QVss-Bzwa-7Xtw-ikksgM cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid 35288829-c1f6-42ab-aeb0-f2915a389e48 osd id 6 type block vdo 0 devices /dev/sdc I would mount it and run the osd deamon manually, but ceph-disk list seems to be gone in Nautilus. Therefore i dont know where to mount what. Any ideas on that? Cheers, Michael

2 1

March Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 25 Mar '20

25 Mar '20

Hello, We will be having a Ceph science/research/big cluster call on Wednesday March 25th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20200325 Due to the worldwide effects of covid-19 I'm thinking it won't hurt to try to host this call. If only a few people join the call we can then decide to continue or cancel it. Ceph calendar event details: March 25, 2020 9am US Central 4pm Central Eurpean We try to keep it to an hour or less. Description:Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index <https://www.google.com/url?q=https://pad.ceph.com/p/Ceph_Science_User_Group…> Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink <https://www.google.com/url?q=https://bluejeans.com/908675367?src%3Dcalendar…> To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc <https://www.google.com/url?q=http://bjn.vc&sa=D&ust=1579363980705000&usg=AO…> 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers <https://www.google.com/url?q=https://www.redhat.com/en/conference-numbers&s…> 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 <https://www.google.com/url?q=https://bluejeans.com/111&sa=D&ust=15793639807…> Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

1 1

Re: How can I recover PGs in state 'unknown', where OSD location seems to be lost?
by Mark S. Holliman 25 Mar '20

25 Mar '20

So I've managed to use ceph-objectstore-tool to locate the pgs in 'unknown' state on the OSDs, but how do I tell the rest of the system where to find them? Is there a command for setting a the OSDs associated with a PG? Or, less ideally, is there a table somewhere I can hack to do this by hand? Mark Holliman Wide Field Astronomy Unit Institute for Astronomy University of Edinburgh The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: Ceph fully crash and we unable to recovery
by Parker Lau 25 Mar '20

25 Mar '20

Hello Sir/Madam, We are facing the serious problem for our proxmox with Ceph. I have already submitted the ticket to Proxmox but they said that only option trying to recover the mondb.We would like to know any some suggestion in our situation. So far the only option that I see would be in trying to recover the mondb from an OSD. But this action is usually a last resort. Since I don't know the outcome, the cluster could then very well be dead and thus all data lost. https://docs.ceph.com/docs/luminous/rados/troubleshooting/troubleshooting-m… --- > > I would like to set the nodown on the cluster to see if the OSDs are kept in the cluster. > > The OSDs are joining the cluster but are set as down shortly after. > > > ok . Please go ahead. Sadly this didn't have any effect either. But I think I found a clue to what might be going on. # ceph-osd.0.log 2020-03-24 21:22:06.462100 7fb33aab0e00 10 osd.0 0 read_superblock sb(e8e81549-91e5-4370-b091-9500f406a2b2 osd.0 0bb2b9bb-9a70-4d6f-8d4e-3fc5049d63d6 e14334 [13578,14334] lci=[0,14334]) # ceph-mon.cccs01.log 2020-03-24 21:26:48.038345 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 e14299: 48 total, 13 up, 35 in 2020-03-24 21:26:48.038351 7f7ef791a700 5 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 can_mark_out current in_ratio 0.729167 < min 0.75, will not mark osds out 2020-03-24 21:26:48.038360 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 tick NOOUT flag set, not checking down osds 2020-03-24 21:26:48.038364 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 min_last_epoch_clean 0 # ceph-mon.cccs06.log 2020-03-22 22:26:57.056939 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14333 e14333: 48 total, 48 up, 48 in 2020-03-22 22:27:04.113054 7f3c1993a700 0 mon.cccs06@5(peon)<mailto:mon.cccs06@5(peon)> e31 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1 2020-03-22 22:27:04.113086 7f3c1993a700 0 log_channel(audit) log [DBG] : from='client.? 10.1.14.8:0/4265796352' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch 2020-03-22 22:27:09.752027 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14334 e14334: 48 total, 48 up, 48 in ... 2020-03-23 10:42:51.891722 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891729 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891730 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 1009089991638532096, adjusting msgr requires 2020-03-23 10:42:51.891732 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires It seems that the OSD have a epoch of e14334 but the MONs seems to have e14269 for the OSDs. I could only find the e14334 on the ceph-mon.cccs06. The cccs06 has been the last MON standing (the node also reset). But when the cluster came back the MONs with the old epoch came up first and joined. The syslog showed that these were the last log entries written. So the nodes reset shortly after. This would fit to cccs06 being the last MON alive. # cccs01 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: leader is 1/2502 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: synced members: 1/2502, 5/2219 # cccs02 Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] notice: we (3/2514) left the process group Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] crit: leaving CPG group # cccs06 Mar 22 22:31:16 cccs06 pmxcfs[2662]: [status] noMar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'iscsi_tcp' Mar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'ib_iser' There must have been some issue prior to the reset as I found those error messages. They could explain why no older epoch was written anymore. # cccs01 2020-03-22 22:22:15.661060 7fc09c85c100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs01/store.db/LOCK: Permission denied 2020-03-22 22:22:15.661067 7fc09c85c100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs01': (22) Invalid argument # cccs02 2020-03-22 22:31:11.209524 7fd034786100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs02/store.db/LOCK: Permission denied 2020-03-22 22:31:11.209541 7fd034786100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs02': (22) Invalid argument # cccs06 no such entries. From my point of view, the last question is now: How to get the epoch from the OSD into the MON DB. I have no answer to this yet. Best Regards, Parker Lau ReadySpace Ltd - Cloud and Managed Hosting Professionals Direct: +852 3726 1120 Hotiline: +852 3568 3372<tel:+852%203568%203372> Fax: +852 3568 3376<tel:+852%203568%203376> Website: www.readyspace.com.hk<http://www.readyspace.com.hk/> Helpdesk: helpdesk.readyspace.com<http://helpdesk.readyspace.com/> Get the latest update and promotion here: twitter.com/readyspace<http://twitter.com/readyspace> | facebook.com/readyspace<http://www.facebook.com/readyspace> [cid:image001.jpg@01D5CC61.CF90E050]Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. From: Parker Lau Sent: Wednesday, March 25, 2020 1:00 AM To: ceph-users(a)ceph.io Subject: 回覆: Ceph fully crash and we unable to recovery Hello Sir/Madam, We are facing the serious problem for our proxmox with Ceph. I have already submitted the ticket to Proxmox but they said that only option trying to recover the mondb.We would like to know any some suggestion in our situation. So far the only option that I see would be in trying to recover the mondb from an OSD. But this action is usually a last resort. Since I don't know the outcome, the cluster could then very well be dead and thus all data lost. https://docs.ceph.com/docs/luminous/rados/troubleshooting/troubleshooting-m… --- > > I would like to set the nodown on the cluster to see if the OSDs are kept in the cluster. > > The OSDs are joining the cluster but are set as down shortly after. > > > ok . Please go ahead. Sadly this didn't have any effect either. But I think I found a clue to what might be going on. # ceph-osd.0.log 2020-03-24 21:22:06.462100 7fb33aab0e00 10 osd.0 0 read_superblock sb(e8e81549-91e5-4370-b091-9500f406a2b2 osd.0 0bb2b9bb-9a70-4d6f-8d4e-3fc5049d63d6 e14334 [13578,14334] lci=[0,14334]) # ceph-mon.cccs01.log 2020-03-24 21:26:48.038345 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 e14299: 48 total, 13 up, 35 in 2020-03-24 21:26:48.038351 7f7ef791a700 5 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 can_mark_out current in_ratio 0.729167 < min 0.75, will not mark osds out 2020-03-24 21:26:48.038360 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 tick NOOUT flag set, not checking down osds 2020-03-24 21:26:48.038364 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 min_last_epoch_clean 0 # ceph-mon.cccs06.log 2020-03-22 22:26:57.056939 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14333 e14333: 48 total, 48 up, 48 in 2020-03-22 22:27:04.113054 7f3c1993a700 0 mon.cccs06@5(peon)<mailto:mon.cccs06@5(peon)> e31 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1 2020-03-22 22:27:04.113086 7f3c1993a700 0 log_channel(audit) log [DBG] : from='client.? 10.1.14.8:0/4265796352' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch 2020-03-22 22:27:09.752027 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14334 e14334: 48 total, 48 up, 48 in ... 2020-03-23 10:42:51.891722 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891729 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891730 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 1009089991638532096, adjusting msgr requires 2020-03-23 10:42:51.891732 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires It seems that the OSD have a epoch of e14334 but the MONs seems to have e14269 for the OSDs. I could only find the e14334 on the ceph-mon.cccs06. The cccs06 has been the last MON standing (the node also reset). But when the cluster came back the MONs with the old epoch came up first and joined. The syslog showed that these were the last log entries written. So the nodes reset shortly after. This would fit to cccs06 being the last MON alive. # cccs01 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: leader is 1/2502 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: synced members: 1/2502, 5/2219 # cccs02 Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] notice: we (3/2514) left the process group Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] crit: leaving CPG group # cccs06 Mar 22 22:31:16 cccs06 pmxcfs[2662]: [status] noMar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'iscsi_tcp' Mar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'ib_iser' There must have been some issue prior to the reset as I found those error messages. They could explain why no older epoch was written anymore. # cccs01 2020-03-22 22:22:15.661060 7fc09c85c100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs01/store.db/LOCK: Permission denied 2020-03-22 22:22:15.661067 7fc09c85c100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs01': (22) Invalid argument # cccs02 2020-03-22 22:31:11.209524 7fd034786100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs02/store.db/LOCK: Permission denied 2020-03-22 22:31:11.209541 7fd034786100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs02': (22) Invalid argument # cccs06 no such entries. From my point of view, the last question is now: How to get the epoch from the OSD into the MON DB. I have no answer to this yet. Best Regards, Parker Lau ReadySpace Ltd - Cloud and Managed Hosting Professionals Direct: +852 3726 1120 Hotiline: +852 3568 3372<tel:+852%203568%203372> Fax: +852 3568 3376<tel:+852%203568%203376> Website: www.readyspace.com.hk<http://www.readyspace.com.hk/> Helpdesk: helpdesk.readyspace.com<http://helpdesk.readyspace.com/> Get the latest update and promotion here: twitter.com/readyspace<http://twitter.com/readyspace> | facebook.com/readyspace<http://www.facebook.com/readyspace> [cid:image001.jpg@01D5CC61.CF90E050]Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. CONFIDENTIALITY: This message is intended for the recipient named above. It may contain confidential or privileged information. If you are not the intended recipient, please notify the sender immediately by replying to this message and then delete it from your system. Do not read, copy, use or circulate this communication.   SECURITY: Internet communications are not secure. While reasonable effort has been made to ensure that this communication has not been tampered with, the sender cannot be responsible for alterations made to the contents of this message without his/her express consent. The sender also cannot be responsible for any computer virus/worm/ trojan horse or other program(s) accompanying this message which he/she is unaware of. Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. 寄件者: Parker Lau 寄件日期: Wednesday, 25 March 2020 12:55 am 收件者: 'ceph-fs(a)ceph.io' <ceph-fs(a)ceph.io<mailto:ceph-fs@ceph.io>> 主旨: Ceph fully crash and we unable to recovery Hello Sir/Madam, We are facing the serious problem for our proxmox with Ceph. I have already submitted the ticket to Proxmox but they said that only option trying to recover the mondb.We would like to know any some suggestion in our situation. So far the only option that I see would be in trying to recover the mondb from an OSD. But this action is usually a last resort. Since I don't know the outcome, the cluster could then very well be dead and thus all data lost. https://docs.ceph.com/docs/luminous/rados/troubleshooting/troubleshooting-m… --- > > I would like to set the nodown on the cluster to see if the OSDs are kept in the cluster. > > The OSDs are joining the cluster but are set as down shortly after. > > > ok . Please go ahead. Sadly this didn't have any effect either. But I think I found a clue to what might be going on. # ceph-osd.0.log 2020-03-24 21:22:06.462100 7fb33aab0e00 10 osd.0 0 read_superblock sb(e8e81549-91e5-4370-b091-9500f406a2b2 osd.0 0bb2b9bb-9a70-4d6f-8d4e-3fc5049d63d6 e14334 [13578,14334] lci=[0,14334]) # ceph-mon.cccs01.log 2020-03-24 21:26:48.038345 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 e14299: 48 total, 13 up, 35 in 2020-03-24 21:26:48.038351 7f7ef791a700 5 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 can_mark_out current in_ratio 0.729167 < min 0.75, will not mark osds out 2020-03-24 21:26:48.038360 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 tick NOOUT flag set, not checking down osds 2020-03-24 21:26:48.038364 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 min_last_epoch_clean 0 # ceph-mon.cccs06.log 2020-03-22 22:26:57.056939 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14333 e14333: 48 total, 48 up, 48 in 2020-03-22 22:27:04.113054 7f3c1993a700 0 mon.cccs06@5(peon)<mailto:mon.cccs06@5(peon)> e31 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1 2020-03-22 22:27:04.113086 7f3c1993a700 0 log_channel(audit) log [DBG] : from='client.? 10.1.14.8:0/4265796352' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch 2020-03-22 22:27:09.752027 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14334 e14334: 48 total, 48 up, 48 in ... 2020-03-23 10:42:51.891722 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891729 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891730 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 1009089991638532096, adjusting msgr requires 2020-03-23 10:42:51.891732 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires It seems that the OSD have a epoch of e14334 but the MONs seems to have e14269 for the OSDs. I could only find the e14334 on the ceph-mon.cccs06. The cccs06 has been the last MON standing (the node also reset). But when the cluster came back the MONs with the old epoch came up first and joined. The syslog showed that these were the last log entries written. So the nodes reset shortly after. This would fit to cccs06 being the last MON alive. # cccs01 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: leader is 1/2502 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: synced members: 1/2502, 5/2219 # cccs02 Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] notice: we (3/2514) left the process group Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] crit: leaving CPG group # cccs06 Mar 22 22:31:16 cccs06 pmxcfs[2662]: [status] noMar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'iscsi_tcp' Mar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'ib_iser' There must have been some issue prior to the reset as I found those error messages. They could explain why no older epoch was written anymore. # cccs01 2020-03-22 22:22:15.661060 7fc09c85c100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs01/store.db/LOCK: Permission denied 2020-03-22 22:22:15.661067 7fc09c85c100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs01': (22) Invalid argument # cccs02 2020-03-22 22:31:11.209524 7fd034786100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs02/store.db/LOCK: Permission denied 2020-03-22 22:31:11.209541 7fd034786100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs02': (22) Invalid argument # cccs06 no such entries. From my point of view, the last question is now: How to get the epoch from the OSD into the MON DB. I have no answer to this yet. Best Regards, Parker Lau ReadySpace Ltd - Cloud and Managed Hosting Professionals Direct: +852 3726 1120 Hotiline: +852 3568 3372<tel:+852%203568%203372> Fax: +852 3568 3376<tel:+852%203568%203376> Website: www.readyspace.com.hk<http://www.readyspace.com.hk/> Helpdesk: helpdesk.readyspace.com<http://helpdesk.readyspace.com/> Get the latest update and promotion here: twitter.com/readyspace<http://twitter.com/readyspace> | facebook.com/readyspace<http://www.facebook.com/readyspace> [cid:image001.jpg@01D5CC61.CF90E050]Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. CONFIDENTIALITY: This message is intended for the recipient named above. It may contain confidential or privileged information. If you are not the intended recipient, please notify the sender immediately by replying to this message and then delete it from your system. Do not read, copy, use or circulate this communication.   SECURITY: Internet communications are not secure. While reasonable effort has been made to ensure that this communication has not been tampered with, the sender cannot be responsible for alterations made to the contents of this message without his/her express consent. The sender also cannot be responsible for any computer virus/worm/ trojan horse or other program(s) accompanying this message which he/she is unaware of. Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately.

1 0

multi-node NFS Ganesha + libcephfs caching
by Maged Mokhtar 24 Mar '20

24 Mar '20

Hello all, For multi-node NFS Ganesha over CephFS, is it OK to leave libcephfs write caching on, or should it be configured off for failover ? Cheers /Maged

3 8

rbd-mirror -> how far behind_master am i time wise?
by Ml Ml 24 Mar '20

24 Mar '20

Hello List, i use rbd-mirror and i asynchronously mirror to my backup cluster. My backup cluster only has "spinnung rust" and wont be able to always perform like the live cluster. Thats is fine for me, as far as it´s not further behind than 12h. vm-194-disk-1: global_id: 7a95730f-451c-4973-8038-2a59e29ac5ad state: up+replaying description: replaying, master_position=[object_number=1046, tag_tid=4, entry_tid=936210], mirror_position=[object_number=911, tag_tid=4, entry_tid=815131], entries_behind_master=121079 last_update: 2020-03-24 08:43:43 I learned, that the entries_behind_master are single transactions. But what i am really interested in is: How far am i behind time wise? Is there a way to tell his? Thanks, Michael

2 1

French Classes in Bangalore
by Ria institute 24 Mar '20

24 Mar '20

RIA Institute of Technology Provides easy to understand French Language Classes in Bangalore. Learning French is considered to be a part of the curriculum for many professional jobs in major Multinational companies. We offer French Training in Marathahalli, Bangalore to aspiring Professionals and Students who are looking forward to upgrade their skills in Foreign Language. We offer French Training Classes to Educational Segments such as First and Second PUC College Students, Working Professionals, Business Travellers and Language learning enthusiasts. Structured Course content and individual focused approach has enabled us to be one of the Best Training Institutes in Bangalore when it comes to French Language Courses. Our Trainers are Professionals in their field with relative experience. https://www.riainstitute.co.in/French-Language-Training-in-Bangalore.html

1 0

Fwd: RGW failing to create bucket
by Abhinav Singh 24 Mar '20

24 Mar '20

---------- Forwarded message --------- From: Abhinav Singh <singhabhinav0796(a)gmail.com> Date: Mon, Mar 23, 2020 at 7:43 PM Subject: RGW failing to create bucket To: <dev(a)ceph.io> ceph : octopus JaegerTracing : master ubuntu : 18.04 When I implementing jaeger tracing it is unable to create a bucket. (I m using swif to perform testing.) /src/librados/IoCtxImpl.cc ``` void librados::IoCtxImpl::queue_aio_write(AioCompletionImpl *c) { std::cout<<"yes"<<std::endl; JTracer tracer; tracer.initTracer("Writing Started", "/home/abhinav/Desktop/GSOC/deepika/ceph/src/librados/tracerConfig.yaml"); Span span=tracer.newSpan("writing started"); span->Finish(); try{ auto yaml = YAML::LoadFile("tracerConfig.yaml"); }catch(const YAML::ParserException& pe){ // ldout<<pe.what()<<dendl; std::cout<<pe.what()<<std::endl; ofstream f; f.open("/home/abhinav/Desktop/err.txt"); f<<pe.what(); f.close(); } // auto config = jaegertracing::Config::parse(yaml); // auto tracer=jaegertracing::Tracer::make( // "Writing", // config, // jaegertracing::logging::consoleLogger() // ); // opentracing::Tracer::InitGlobal( // static_pointer_cast<opentracing::Tracer>(tracer) // ); // auto span = opentracing::Tracer::Global()->StartSpan("Span1"); get(); ofstream file; file.open("/home/abhinav/Desktop/write.txt",std::ios::out | std::ios::app); file<<"Writing /src/librados/IoCtxImpl.cc 310.\n"; file.close(); std::scoped_lock l{aio_write_list_lock}; ceph_assert(c->io == this); c->aio_write_seq = ++aio_write_seq; ldout(client->cct, 20) << "queue_aio_write " << this << " completion " << c << " write_seq " << aio_write_seq << dendl; aio_write_list.push_back(&c->aio_write_list_item); // opentracing::Tracer::Global()->Close(); } ``` /include/tracer.h ``` typedef std::unique_ptr<opentracing::Span> Span; class JTracer{ public: JTracer(){} ~JTracer(){ opentracing::Tracer::Global()->Close(); } void static inline loadYamlConfigFile(const char* path){ return; } void initTracer(const char* tracerName,const char* filePath){ auto yaml = YAML::LoadFile(filePath); auto configuration = jaegertracing::Config::parse(yaml); auto tracer = jaegertracing::Tracer::make( tracerName, configuration, jaegertracing::logging::consoleLogger()); opentracing::Tracer::InitGlobal( std::static_pointer_cast<opentracing::Tracer>(tracer)); Span s=opentracing::Tracer::Global()->StartSpan("Testing"); s->Finish(); } Span newSpan(const char* spanName){ Span span=opentracing::Tracer::Global()->StartSpan(spanName); return std::move(span); } Span childSpan(const char* spanName,const Span& parentSpan){ Span span = opentracing::Tracer::Global()->StartSpan(spanName, {opentracing ::ChildOf(&parentSpan->context())}); return std::move(span); } Span followUpSpan(const char *spanName, const Span& parentSpan){ Span span = opentracing::Tracer::Global()->StartSpan(spanName, {opentracing ::FollowsFrom(&parentSpan->context())}); return std::move(span); } }; ``` Output when trying to create new container ``` errno 111 connection refused ``` But when I remove the tracer part in IoCtxImpl.cc it is workng fine. I m new to ceph, and dont know what information to share to correctly track down the problem, if any extra informtion is needed I will share it instantly. Been stuck into this issue for one week. Please someone help me! Thank you.

2 3

Re: Newbie to Ceph jacked up his monitor
by Jarett DeAngelis 24 Mar '20

24 Mar '20

So, I thought I’d post with what I learned re: what to do with this problem. This system is a 3-node Proxmox cluster, and each node had: 1 x 1TB NVMe 2 x 512GB HDD I had maybe 100GB of data in this system total. Then I added: 2 x 256GB SSD 1 x 1TB HDD To each system, and let it start rebalancing. When it started the management interface showed the storage as being out of order in various ways, but it was clear that Ceph was rebalancing PGs across the 3 nodes and the “broken” part of the graphic display was shrinking as it spread data across the added OSDs. In the process, however, the monitors racked up ENORMOUS amounts of files. On one machine, the boot drive only has 64GB of space total so the partition where /var/lib/ceph/somethingsomething.db lived was only 27GB. This filled up very, very fast, and eventually killed the monitor on that node. I figured out you can `ceph-monstore-tool compact` or `ceph-kvstore-tool rocksdb /path compact` to get the system to truncate the files in there, but even when I scheduled those jobs to run on each monitor every minute the amount of space being taken up by those rocksdb files grew and grew until they threatened to kill the monitors on the nodes with larger amounts of space too. Other, dumber measures I tried taking to give the system more space for these files ended up screwing up my Proxmox system, so now I have to reinstall. What can be done about this problem so that I don’t have this issue when I try to implement again? Thanks!

2 1