- ceph-users - lists.ceph.io

MDS blocked ops; kernel: Workqueue: ceph-pg-invalid ceph_invalidate_work [ceph]
by Frank Schilder 03 Sep '19

03 Sep '19

Hi, I encountered a problem with blocked MDS operations and a client becoming unresponsive. I dumped the MDS cache, ops, blocked ops and some further log information here: https://files.dtu.dk/u/peQSOY1kEja35BI5/2010-09-03-mds-blocked-ops?l A user of our HPC system was running a job that creates a somewhat stressful MDS load. This workload tends to lead to MDS warnings like "slow metadata ops" and "client does not respond to caps release", which usually disappear without intervantion after a while. He cancelled the job and one operation from one of the clients remained stuck in the MDS. We had a health warning about 1 blocked meta data operation and one client failing to respond to caps release. I should mention that we execute "echo 3 > /proc/sys/vm/drop_caches" in the epilogue script executed after every job, which usually cleans up all unused caps without problems. So, at the time I was looking at the number of client caps, these were down to below 100 for the client in question due to epilogue script execution. Looks like there might be a race condition with the drop caches and MDS requests. In addition, while this happened, there was backfill going on. All PGs were active+other stuff. All storage was r/w-accessible. On the client side, this was in the logs: Sep 3 09:15:57 sn110 kernel: INFO: task kworker/0:1:79782 blocked for more than 120 seconds. Sep 3 09:15:57 sn110 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 3 09:15:57 sn110 kernel: kworker/0:1 D ffff995cf4614100 0 79782 2 0x00000000 Sep 3 09:15:57 sn110 kernel: Workqueue: ceph-pg-invalid ceph_invalidate_work [ceph] Sep 3 09:15:57 sn110 kernel: Call Trace: [... see link above ...] I did not see slow ops on any of the OSDs. All other information in the link above. We had to reboot the client to resolve this problem. It seems like the MDS does not clean up blocked requests in certain situations when it ought to be possible. I hope the cache and ops dumps help pinpoint the reason. Best regards, Frank

2 1

TASK_UNINTERRUPTIBLE kernel client threads
by Toby Darling 03 Sep '19

03 Sep '19

Hi We have a couple of RHEL 7.6 (3.10.0-957.21.3.el7.x86_64) clients that have a number of uninterruptible threads and I'm wondering if we're looking at the issue fixed by https://www.spinics.net/lists/ceph-devel/msg45467.html (the fix hasn't made it into RHEL 7.7 3.10.0-1062). Stack traces of the hung threads are at http://p.ip.fi/9pQA There are a number of entries listed in /sys/kernel/debug/ceph/*/{osdc,mdsc} at http://p.ip.fi/VVzx Unfortunately, the issue isn't consistently reproducible Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology Francis Crick Avenue Cambridge Biomedical Campus Cambridge CB2 0QH Phone 01223 267070

2 1

Manual pg repair help
by Marc Roos 03 Sep '19

03 Sep '19

Is there no ceph wiki page with examples of manual repairs with the ceph-objectstore-tool (eg. where pg repair and pg scrub don’t work) I am having this issue for quite some time. 2019-09-02 14:17:34.175139 7f9b3f061700 -1 log_channel(cluster) log [ERR] : deep-scrub 17.36 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing And tried to resolve it according to this procedure[0] but now I am getting the message ceph-objectstore-tool --dry-run --type bluestore --data-path /var/lib/ceph/osd/ceph-29 --pgid 17.36 '{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","snapid":-2, "hash":1357874486,"pool":17,"namespace":"","max":0}' remove Snapshots are present, use removeall to delete everything I am not sure about this removeall, but I do not want to start deleting snapshots hoping it will amount to something. Besides if only maybe 4mb block is damaged, do you really need to purge snapshots of 40GB. I rather have a snapshot of 40GB missing 4MB than having no snapshot at all. [0] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg47218.html PS. Is there a record of who is having the longest unhealthy cluster state? Because I would not like it to be me ;)

1 0

Bug: ceph-objectstore-tool ceph version 12.2.12
by Marc Roos 03 Sep '19

03 Sep '19

Max 2x listed ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]

1 0

Placement Groups - default.rgw.metadata pool.
by Jaroslaw Owsiewski 03 Sep '19

03 Sep '19

Hi, we have cluster which running Ceph Luminous 12.2.12. Rados Gateway Only (S3). Pool with data is placed on SAS HDDs (1430 pcs) and the rest of pools is placed on the SSD (72 pcs) disks - 72 hosts with OSD role (3 rows, 2 racks per row, and 12 hosts per rack). BlueStore of course. The question is: how many PG we need for default.rgw.metadata? Any ideas? Example statistics from this cluster: pool default.rgw.buckets.data id 15 189946/9416348469 objects misplaced (0.002%) recovery io 13.5MiB/s, 83objects/s client io 531MiB/s rd, 51.4MiB/s wr, 13.24kop/s rd, 4.84kop/s wr pool .rgw.root id 16 nothing is going on pool default.rgw.control id 17 nothing is going on pool default.rgw.meta id 18 client io 47.0MiB/s rd, 0B/s wr, 57.95kop/s rd, 450op/s wr pool default.rgw.log id 19 nothing is going on pool default.rgw.buckets.index id 20 client io 3.12MiB/s rd, 0B/s wr, 3.19kop/s rd, 1.92kop/s wr Regards, Jarek

1 0

Re: pg 17.36 is active+clean+inconsistent head expected clone 1 missing?
by Marc Roos 03 Sep '19

03 Sep '19

Hi Steve, I was just about to follow your steps[0] with the ceph-objectstore-tool, (I do not want to remove more snapshots) So I have this error pg 17.36 is active+clean+inconsistent, acting [7,29,12] 2019-09-02 14:17:34.175139 7f9b3f061700 -1 log_channel(cluster) log [ERR] : deep-scrub 17.36 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing I removed the snapshot with snapshot id 4, did a pg repair without any result. I am trying to understand this command of yours, ceph-objectstore-tool --type bluestore --data-path /var/lib/ceph/osd/ceph-229/ --pgid 2.9a6 '{"oid":"rb.0.2479b45.238e1f29","snapid":-2,"hash":2320771494,"max":0,"p ool":2,"namespace":"","max":0}' I think you are getting this info from the --op list not? And grep for the "rbd_data.1f114174b0dc51.0000000000000974" occurance? I have these entries on osd.29 ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}] ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}] So I guess snapid's with -2 are bad? I have noticed actually quite a few -2 listings in these op list output, and do not understand why there are so many and the cluster is healthy except for this pg 17.36. [0] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg47212.html -----Original Message----- From: Steve Anthony [mailto:sma310@lehigh.edu] Sent: vrijdag 16 november 2018 17:44 To: ceph-users(a)lists.ceph.com Subject: Re: [ceph-users] pg 17.36 is active+clean+inconsistent head expected clone 1 missing? Looks similar to a problem I had after a several OSDs crashed while trimming snapshots. In my case, the primary OSD thought the snapshot was gone, but some of the replicas are still there, so scrubbing flags it. First I purged all snapshots and then ran ceph pg repair on the problematic placement groups. The first time I encountered this, that action was sufficient to repair the problem. The second time however, I ended up having to manually remove the snapshot objects. http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-June/027431.html Once I had done that, repair the placement group fixed the issue. -Steve On 11/16/2018 04:00 AM, Marc Roos wrote: > > > I am not sure that is going to work, because I have this error quite > some time, from before I added the 4th node. And on the 3 node cluster > it was: > > osdmap e18970 pg 17.36 (17.36) -> up [9,0,12] acting [9,0,12] > > If I understand correctly what you intent to do, moving the data around. > This was sort of accomplished by adding the 4th node. > > > > -----Original Message----- > From: Frank Yu [mailto:flyxiaoyu@gmail.com] > Sent: vrijdag 16 november 2018 3:51 > To: Marc Roos > Cc: ceph-users > Subject: Re: [ceph-users] pg 17.36 is active+clean+inconsistent head > expected clone 1 missing? > > try to restart osd.29, then use pg repair. If this doesn't work or it > appear again after a while, scan your HDD which used for osd.29, maybe > there is bad sector of your disks, just replace the disk with new one. > > > > On Thu, Nov 15, 2018 at 5:00 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> > wrote: > > > > Forgot, these are bluestore osds > > > > -----Original Message----- > From: Marc Roos > Sent: donderdag 15 november 2018 9:59 > To: ceph-users > Subject: [ceph-users] pg 17.36 is active+clean+inconsistent head > expected clone 1 missing? > > > > I thought I will give it another try, asking again here since there > is > another thread current. I am having this error since a year or so. > > This I of course already tried: > ceph pg deep-scrub 17.36 > ceph pg repair 17.36 > > > [@c01 ~]# rados list-inconsistent-obj 17.36 > {"epoch":24363,"inconsistents":[]} > > > [@c01 ~]# ceph pg map 17.36 > osdmap e24380 pg 17.36 (17.36) -> up [29,12,6] acting [29,12,6] > > > [@c04 ceph]# zgrep ERR ceph-osd.29.log*gz > ceph-osd.29.log-20181114.gz:2018-11-13 14:19:55.766604 7f25a05b1700 > -1 > log_channel(cluster) log [ERR] : deep-scrub 17.36 > 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head > expected > clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 > missing > ceph-osd.29.log-20181114.gz:2018-11-13 14:24:55.943454 7f25a05b1700 > -1 > log_channel(cluster) log [ERR] : 17.36 deep-scrub 1 errors > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- Steve Anthony LTS HPC Senior Analyst Lehigh University sma310(a)lehigh.edu _______________________________________________ ceph-users mailing list ceph-users(a)lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

BACKPORT #21481 - jewel: "FileStore.cc: 2930: FAILED assert(0 == "unexpected error")" in fs
by Reddi Prasad Yendluri 02 Sep '19

02 Sep '19

Hi, I have come across this issue on one of ceph cluster storage nodes. (in the production OpenStack). Now, I would like to redploy the CEPH and start OSD on this node again with the same version. I will try to upgrade to a newer version, after increasing the capacity of cluster storage and bring the ceph health status to "OK" status. Has any one used, kolla ansible to deploy or add new node the existing ceph cluster storage. let me know the steps. What I have tried: 1. Ran "ceph playbook from kolla-ansible $ kolla-ansible bootstrap-servers -i <path/multinode> --limit nodeXX.maas Output: success 2. Unable to go through the "prechecks" as docker deamon is not running. 3. Could not start the docker, Uninstalled docker and during installation of docker using the docker-install/site.yml the docker could not start docker neither install nor start the docker because it is looking for dependencies of OSD device (disk) 4. Trying to uninstall the docker completely from physical node and insgtall docker from scratch and reset up the CEPH on this node? Has anyone tried resetup CEPH OSD on the existing ceph cluster storge. Thanks, Reddi Prasad YENDLURI Cloud Specialist M +65 8345 9599 | D +65 6220 9908 Office: *51B Circular Road Singapore 049406* [image: PALO IT]

1 0

ceph-volume 'ascii' codec can't decode byte 0xe2
by changcheng.liu 02 Sep '19

02 Sep '19

Hi Alfredo Deza I see your below PR in ceph PR 23289 "ceph-volume ensure encoded bytes are always used" I meet with one problem when using ceph-deploy to deploy ceph cluster. During running ceph-volume command to create OSD through ceph-deploy, it always hit below problem: nstcc1@nstcloudcc1:deploy$ PYTHONIOENCODING=UTF8 ceph-deploy osd create --data /dev/sdc nstcloudcc1 [nstcloudcc1][WARNIN] stderr: purged osd.0 [nstcloudcc1][WARNIN] --> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 132: ordinal not in range(128) [nstcloudcc1][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdc [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs Do you know how to solve the problem? Below is the deploy host environment: nstcc1@nstcloudcc1:deploy$ cat /etc/os-release NAME="Ubuntu" VERSION="18.04.1 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.1 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic nstcc1@nstcloudcc1:deploy$ env | grep 'LC_\|PYTHON' LC_ALL=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 PYTHONIOENCODING=UTF8 LC_NAME=en_US.UTF-8 LC_ADDRESS=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_TELEPHONE=en_US.UTF-8 LC_IDENTIFICATION=en_US.UTF-8 LC_TIME=en_US.UTF-8 nstcc1@nstcloudcc1:deploy$ python --version Python 2.7.15rc1 B.R. Changcheng

1 0

EC Compression
by Ashley Merrick 02 Sep '19

02 Sep '19

I have an EC RBD pool I want to add compression on. I have the meta pool and the data pool, do I need to enable compression on both for it to function correctly or only on one pool? Thanks

2 1

Which network is used for recovery / rebalancing
by Thomas Schneider 02 Sep '19

02 Sep '19

Hi, my ceph cluster is recovering / rebalancing PGs since some OSDs are marked out. Question: Which network is used for this task be default? Is there any configuration for the network to be used in //etc/ceph/ceph.conf/? THX

2 1