May 2020 - ceph-users - lists.ceph.io

Upgrade Luminous to Nautilus on a Debian system

by Herve Ballans

Hi all, I'm planning to upgrade one on my Ceph Cluster currently on Luminous 12.2.13 / Debian Stretch (updated). On this cluster, Luminous is packaged from the official Ceph repo (deb https://download.ceph.com/debian-luminous/ stretch main) I would like to upgrade it with Debian Buster and Nautilus using the croit.io repository (deb https://mirror.croit.io/debian-nautilus/ buster main) I already prepared the steps procedure but I just want to verify one step regarding the upgrade of the ceph packages. Do I have to upgrade ceph in the same time than Debian or do i have to upgrade ceph after the Debian upgrade from Stretch to Buster ? 1) In the first case : * Replace stretch by buster in /etc/apt/sources.list * Modify the ceph.list repo by croit.io one * Upgrade the entire nodes 2) In the second case (upgrade Debian then Ceph) * Replace stretch by buster in /etc/apt/sources.list * keep the /etc/apt/sources.list.d/ceph.list as it is * Upgrade and reboot the nodes * replace the ceph.list file by croit.io * upgrade the ceph packages * restarting the Ceph services (in the right order MON -> MGR -> OSD -> MDS) Thanks a lot for your advices Regards, Hervé

3 years, 11 months

3
6
0 0

pg incomplete blocked by destroyed osd

by Francois Legrand

Hi all, During a crash disaster we had destroyed and reinstalled with a different number a few osds. As an example osd 3 was destroyed and recreated with id 101 by command ceph osd purge 3 --yes-i-really-mean-it + ceph osd create (to block id 3) + ceph-deploy osd create --data /dev/sdxx <server> and finally ceph osd rm 3). Some of our pgs are now incomplet (which can be understood) but blocked by some of the removed osd : ex: here is an part of the ceph pg 30.3 query { "state": "incomplete", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 384075, "up": [ 103, 43, 29, 2, 66 ], "acting": [ 103, 43, 29, 2, 66 ], ........ "peer_info": [ { "peer": "2(3)", "pgid": "30.3s3", "last_update": "373570'105925965", "last_complete": "373570'105925965", ....... }, "up": [ 103, 43, 29, 2, 66 ], "acting": [ 103, 43, 29, 2, 66 ], "avail_no_missing": [], "object_location_counts": [], *"blocked_by": [** ** 3,** ** 49** ** ],* ............ "down_osds_we_would_probe": [ *3* ], "peering_blocked_by": [], "peering_blocked_by_detail": [ { * "detail": "peering_blocked_by_history_les_bound"* } ] I don't understand why the removed osd are still considered and present in the pg infos. Is there a way to get rid of that ? Moreover, we have tons of slow ops (more than 15 000) but I guess that the two problems are linked. Thanks for your help. F.

3 years, 11 months

2
1
0 0

Re: upmap balancer and consequences of osds briefly marked out

by Anthony D'Atri

Do I misunderstand this script, or does it not _quite_ do what’s desired here? I fully get the scenario of applying a full-cluster map to allow incremental topology changes. To be clear, if this is run to effectively freeze backfill during / following a traumatic event, it will freeze that adapted state, not strictly return to the pre-event state? And thus the pg-upmap balancer would still need to be run to revert to the prior state? And this would also hold true for a failed/replaced OSD? > On May 1, 2020, at 7:37 AM, Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: > > Thanks Dan, that looks like a really neat method & script for a few use-cases. We've actually used several of the scripts in that repo over the years, so, many thanks for sharing. > > That method will definitely help in the scenario in which a set of unnecessary pg remaps have been triggered and can be caught early and reverted. I'm still a little concerned about the possibility of, for example, a brief network glitch occurring at night and then waking up to a full unbalanced cluster. Especially with NVMe clusters that can rapidly remap and rebalance (and for which we also have a greater impetus to squeeze out as much available capacity as possible with upmap due to cost per TB). It's just a risk I hadn't previously considered and was wondering if others have either run into it or felt any need to plan around it. > > Cheers, > Dylan > > >> From: Dan van der Ster <dan(a)vanderster.com> >> Sent: Friday, 1 May 2020 5:53 PM >> To: Dylan McCulloch <dmc(a)unimelb.edu.au> >> Cc: ceph-users <ceph-users(a)ceph.io> >> >> Subject: Re: [ceph-users] upmap balancer and consequences of osds briefly marked out >> >> Hi, >> >> You're correct that all the relevant upmap entries are removed when an >> OSD is marked out. >> You can try to use this script which will recreate them and get the >> cluster back to HEALTH_OK quickly: >> https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-rema… >> >> Cheers, Dan >> >> >> On Fri, May 1, 2020 at 9:36 AM Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: >>> >>> Hi all, >>> >>> We're using upmap balancer which has made a huge improvement in evenly distributing data on our osds and has provided a substantial increase in usable capacity. >>> >>> Currently on ceph version: 12.2.13 luminous >>> >>> We ran into a firewall issue recently which led to a large number of osds being briefly marked 'down' & 'out'. The osds came back 'up' & 'in' after about 25 mins and the cluster was fine but had to perform a significant amount of backfilling/recovery despite >> there being no end-user client I/O during that period. >>> >>> Presumably the large number of remapped pgs and backfills were due to pg_upmap_items being removed from the osdmap when osds were marked out and subsequently those pgs were redistributed using the default crush algorithm. >>> As a result of the brief outage our cluster became significantly imbalanced again with several osds very close to full. >>> Is there any reasonable mitigation for that scenario? >>> >>> The auto-balancer will not perform optimizations while there are degraded pgs, so it would only start reapplying pg upmap exceptions after initial recovery is complete (at which point capacity may be dangerously reduced). >>> Similarly, as admins, we normally only apply changes when the cluster is in a healthy state, but if the same issue were to occur again would it be advisable to manually apply balancer plans while initial recovery is still taking place? >>> >>> I guess my concern from this experience is that making use of the capacity gained by using upmap balancer appears to carry some risk. i.e. it's possible for a brief outage to remove those space efficiencies relatively quickly and potentially result in full >> osds/cluster before the automatic balancer is able to resume and redistribute pgs using upmap. >>> >>> Curious whether others have any thoughts or experience regarding this. >>> >>> Cheers, >>> Dylan >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 years, 11 months

2
1
0 0

Re: upmap balancer and consequences of osds briefly marked out

by Dan van der Ster

Hi Dylan, The backfillfull_ratio, which defaults to 0.9, prevents backfilling into an osd which is getting too full. So, worst case scenario is that your cluster will have some osds getting up to 90% full, after which case the upmap balancer should starting putting things back into place. Also, check that your "mon osd down out subtree limit" is set appropriately for your cluster. In our case, we set it to "host" -- we don't want to automatically "out" all the osds from an entire host, because this is normally something that we can quickly fix with a manual intervention. But I fear that wouldn't have helped in your case, because the firewall issue probably downed a random subset of osds from several hosts all at once. We've also had happen a couple times, and now set "mon osd down out interval = 3600" so that we have time to notice the network outage and set noout on the cluster to prevent lots of rebalancing carnage. Hope it helps, Dan On Fri, May 1, 2020 at 4:37 PM Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: > > Thanks Dan, that looks like a really neat method & script for a few use-cases. We've actually used several of the scripts in that repo over the years, so, many thanks for sharing. > > That method will definitely help in the scenario in which a set of unnecessary pg remaps have been triggered and can be caught early and reverted. I'm still a little concerned about the possibility of, for example, a brief network glitch occurring at night and then waking up to a full unbalanced cluster. Especially with NVMe clusters that can rapidly remap and rebalance (and for which we also have a greater impetus to squeeze out as much available capacity as possible with upmap due to cost per TB). It's just a risk I hadn't previously considered and was wondering if others have either run into it or felt any need to plan around it. > > Cheers, > Dylan > > > >From: Dan van der Ster <dan(a)vanderster.com> > >Sent: Friday, 1 May 2020 5:53 PM > >To: Dylan McCulloch <dmc(a)unimelb.edu.au> > >Cc: ceph-users <ceph-users(a)ceph.io> > > > >Subject: Re: [ceph-users] upmap balancer and consequences of osds briefly marked out > > > >Hi, > > > >You're correct that all the relevant upmap entries are removed when an > >OSD is marked out. > >You can try to use this script which will recreate them and get the > >cluster back to HEALTH_OK quickly: > >https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-rema… > > > >Cheers, Dan > > > > > >On Fri, May 1, 2020 at 9:36 AM Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: > >> > >> Hi all, > >> > >> We're using upmap balancer which has made a huge improvement in evenly distributing data on our osds and has provided a substantial increase in usable capacity. > >> > >> Currently on ceph version: 12.2.13 luminous > >> > >> We ran into a firewall issue recently which led to a large number of osds being briefly marked 'down' & 'out'. The osds came back 'up' & 'in' after about 25 mins and the cluster was fine but had to perform a significant amount of backfilling/recovery despite > > there being no end-user client I/O during that period. > >> > >> Presumably the large number of remapped pgs and backfills were due to pg_upmap_items being removed from the osdmap when osds were marked out and subsequently those pgs were redistributed using the default crush algorithm. > >> As a result of the brief outage our cluster became significantly imbalanced again with several osds very close to full. > >> Is there any reasonable mitigation for that scenario? > >> > >> The auto-balancer will not perform optimizations while there are degraded pgs, so it would only start reapplying pg upmap exceptions after initial recovery is complete (at which point capacity may be dangerously reduced). > >> Similarly, as admins, we normally only apply changes when the cluster is in a healthy state, but if the same issue were to occur again would it be advisable to manually apply balancer plans while initial recovery is still taking place? > >> > >> I guess my concern from this experience is that making use of the capacity gained by using upmap balancer appears to carry some risk. i.e. it's possible for a brief outage to remove those space efficiencies relatively quickly and potentially result in full > > osds/cluster before the automatic balancer is able to resume and redistribute pgs using upmap. > >> > >> Curious whether others have any thoughts or experience regarding this. > >> > >> Cheers, > >> Dylan > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

3 years, 11 months

1
0
0 0

large difference between "STORED" and "USED" size of ceph df

by Lee, H. (Hurng-Chun)

Hello, We use purely cephfs in out ceph cluster (version 14.2.7). The cephfs data is an EC pool (k=4, m=2) with hdd OSDs using bluestore. The default file layout (i.e. 4MB object size) is used. We see the following output of ceph df: --- RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 951 TiB 888 TiB 63 TiB 63 TiB 6.58 ssd 9.6 TiB 9.6 TiB 1.4 GiB 16 GiB 0.17 TOTAL 961 TiB 898 TiB 63 TiB 63 TiB 6.52 POOLS: POOL ID STORED OBJECTS USED %USE D MAX AVAIL cephfs-data 2 34 TiB 12.51M 52 TiB 5.93 553 TiB cephfs-metadata 4 994 MiB 98.61k 1.5 GiB 0.02 3.0 TiB --- What triggered my attention is the discrepency between the reported size of "USED" (52 TiB) and "STORED" (34 TiB) on the cephfs-data pool. From this document ( https://docs.ceph.com/docs/master/releases/nautilus/#upgrade-compatibility-… ), it says that - "USED" represents amount of space allocated purely for data by all OSD nodes in KB - "STORED" represents amount of data stored by the user. I seem to undersand that the "USED" size can be roughly taken as the number of objects (12.51M) times the object size (4MB) of the file layout; and since there are many files with size smaller than 4 MB in our system, the actual stored data is less. Is my interpretation correct? If so, does it mean that we will be wasting a lot of space when we have a lot files smaller than the object size of 4MB in the system? Thanks for the help! Cheers, Hong -- Hurng-Chun (Hong) Lee, PhD ICT manager Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging Radboud University Nijmegen e-mail: h.lee(a)donders.ru.nl tel: +31(0) 243610977 web: http://www.ru.nl/donders/

3 years, 11 months

2
1
0 0

14.2.9 MDS Failing

by Marco Pizzolo

Hello, Hoping you can help me. Ceph had been largely problem free for us for the better part of a year. We have a high file count in a single CephFS filesystem, and are seeing this error in the logs: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/mds/OpenFileTable.cc: 777: FAILED ceph_assert(omap_num_objs == num_objs) The issued seemed to occur this morning, and restarting the MDS as well as rebooting the servers doesn't correct the problem. Not really sure where to look next as the MDS daemons crash. Appreciate any help you can provide Marco

3 years, 11 months

4
9
0 0

repairing osd rocksdb

by Francois Legrand

Hi, We had a major crash which ended with ~1/3 of our osd downs. Trying to fix it we reinstalled a few down osd (that was a mistake, I agree) and destroy the datas on it. Finally, we could fix the problem (thanks to Igor Fedotov) and restart almost all of our osds except one for which the rocksdb seems corrupted (at least for one file). Unfortunately, we now have 4 pgs down (all involving the dead osd) and 8 pg incompletes (some of them also involving the down osd). Before considering data loss, we would like to try to restart the down osds hopping to recover the down pgs and maybe some of the incomplete ones. Does someone have an idea on how to do that (maybe by removing the file corrupting the rocksdb or forcing to ignore the data in error) ? If it's not possible, how can we fix (even with dataloss) the downs and incomplete pgs ? Thanks for your advices. F.

3 years, 11 months

2
1
0 0

dashboard module missing dependencies in 15.2.1 Octopus

by Duncan Bellamy

Hi, I have installed ceph on Ubuntu Focal Fossa using the ubuntu repo, instead of ceph-deploy (as ceph-deploy install does not work for Focal Fossa yet) install I used: sudo apt-get install -y ceph ceph-mds radosgw ceph-mgr-dashboard The rest of the setup was the same as the quickstart on ceph.io <http://ceph.io/> with ceph-deploy. It installed ceph version 15.2.1 (octopus). If I do a 'ceph -s' I get the warning: health: HEALTH_WARN 2 mgr modules have failed dependencies If I run 'ceph mgr module ls', for enabled and active modules I get: "always_on_modules": [ "balancer", "crash", "devicehealth", "orchestrator", "osd_support", "pg_autoscaler", "progress", "rbd_support", "status", "telemetry", "volumes" ], "enabled_modules": [ "iostat", "restful” Then when I run 'ceph mgr module enable dashboard’ I get the error: Error ENOENT: module 'dashboard' reports that it cannot run on the active manager daemon: No module named 'yaml' (pass --force to force enablement) I have tried searching, and searching with apt but cannot find any ‘yaml’ package that might be used by ceph. Duncan

3 years, 11 months

2
1
0 0

Re: 回复: Re: OSDs continuously restarting under load

by David Turner

I have 2 filestore OSDs in a cluster facing "Caught signal (Bus error)" as well and can't find anything about it. Ceph 12.2.12. The disks are less than 50% full and basic writes have been successful. Both disks are on different nodes. The other 14 disks on each node are unaffected. Restarting the node doesn't change the behavior. The affected OSD still crashes and the other 14 start fine (which likely rules out the controller and other shared components along those lines). I've attempted [1] these commands on the OSDs to see how much of the disk I could access cleanly. The first is just to flush the journal to disk and it crashed out with the same error. The second command is to compact the DB which also crashed with the same error. On one of the OSDs I was able to make it a fair bit into compacting the DB before it crashed the first time and now it crashes instantly. That leads me to think that it might have gotten to a specific part of the disk and/or filesystem that is having problems. I'm currently running [2] xfs_repair on one of the disks to see if it might be the filesystem. On the other disk I'm running [3] badblocks to check for problems with underlying sectors. I'm assuming that if it's a bad block on the disk that is preventing the disk from starting that there's really nothing that I can do to recover the OSD and I'll just need to export any PGs on the disks that aren't active. Here's hoping I make it through this without data loss. Since I started this data migration I've already lost a couple disks (completely unreadable by the OS so I can't get copies of the PGs off of them). Luckily these ones seem like I might be able to access that part of the data at least. As well, I only have some unfound objects at the moment, but all of my PGs are active, which is an improvement. [1] sudo -u ceph ceph-osd -i 285 --flush-journal sudo -u ceph ceph-kvstore-tool leveldb /var/lib/ceph/osd/ceph-285/current/omap compact [2] xfs_repair -n /dev/sdi1 [3] badblocks -b 4096 -v /dev/sdn On Thu, Mar 19, 2020 at 9:04 AM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > Hi, Igor, > > thanks for the tip. Dmesg does not say any suspicious information. > > I will investigate whether hardware has any problem or not. > > best regards, > > samuel > > > > > > huxiaoyu(a)horebdata.cn > > 发件人： Igor Fedotov > 发送时间： 2020-03-19 12:07 > 收件人： huxiaoyu(a)horebdata.cn; ceph-users; ceph-users > 主题： Re: [ceph-users] OSDs continuously restarting under load > Hi, Samuel, > > I've never seen that sort of signal in the real life: > > 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** > > > I suppose this has some hardware roots. Have you checked dmesg output? > > > Just in case, here is some info on "Bus Error" signal, may be it will > provide some insight: https://en.wikipedia.org/wiki/Bus_error > > > Thanks, > > Igor > > > On 3/18/2020 5:06 PM, huxiaoyu(a)horebdata.cn wrote: > > Hello， folks, > > > > I am trying to add a ceph node into an existing ceph cluster. Once the > reweight of newly-added OSD on the new node exceed 0.4 somewhere, the osd > becomes unresponsive and restarting, eventually go down. > > > > What could be the problem? Any suggestion would be highly appreciated. > > > > best regards, > > > > samuel > > > > **************************************************** > > root@node81:/var/log/ceph# > > root@node81:/var/log/ceph# > > root@node81:/var/log/ceph# > > root@node81:/var/log/ceph# ceph osd df > > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > > 12 hybrid 1.00000 1.00000 3.81TiB 38.3GiB 3.77TiB 0.98 1.32 316 > > 13 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 308 > > 14 hybrid 1.00000 1.00000 3.81TiB 36.9GiB 3.77TiB 0.95 1.27 301 > > 15 hybrid 1.00000 1.00000 3.81TiB 37.1GiB 3.77TiB 0.95 1.28 297 > > 0 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 305 > > 1 hybrid 1.00000 1.00000 3.81TiB 38.2GiB 3.77TiB 0.98 1.31 309 > > 2 hybrid 1.00000 1.00000 3.81TiB 37.4GiB 3.77TiB 0.96 1.29 296 > > 3 hybrid 1.00000 1.00000 3.81TiB 37.9GiB 3.77TiB 0.97 1.30 303 > > 4 hdd 0.20000 1.00000 3.42TiB 10.5GiB 3.41TiB 0.30 0.40 0 > > 5 hdd 0.20000 1.00000 3.42TiB 9.63GiB 3.41TiB 0.28 0.37 87 > > 6 hdd 0.20000 1.00000 3.42TiB 1.91GiB 3.42TiB 0.05 0.07 0 > > 7 hdd 0.20000 1.00000 3.42TiB 11.3GiB 3.41TiB 0.32 0.43 83 > > 16 hdd 0.39999 1.00000 1.79TiB 16.3GiB 1.78TiB 0.89 1.19 142 > > TOTAL 45.9TiB 351GiB 45.6TiB 0.75 > > > > > ------------------------------------------------------------------------------------ > 日志 > > > > root@node81:/var/log/ceph# cat ceph-osd.6.log | grep load_pgs > > 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs > > 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs > > -1324> 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs > > -1323> 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs > opened 0 pgs > > 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs > > 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 > pgs > > -466> 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs > > -465> 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs > opened 202 pgs > > 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs > > 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 > pgs > > -422> 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs > > -421> 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs > opened 177 pgs > > 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > > 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 > pgs > > -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > > -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs > opened 170 pgs > > > > > > 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected > need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have > 3291'557 flags = none tried to add 3291'557 flags = none > > 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected > need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have > 4738'731 flags = none tried to add 4738'731 flags = none > > 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected > need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have > 1789'169 flags = delete tried to add 1789'169 flags = delete > > 2020-03-18 18:39:20.985370 2000fc61b40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 > conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 > conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > 2020-03-18 18:39:21.495101 2000fc61b40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 > conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 > conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) > ** > > in thread 201e35fdb40 thread_name:tp_osd_tp > > > > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) > luminous (stable) > > 1: (()+0x145882c) [0x2000245882c] > > 2: (()+0x19890) [0x2000c54b890] > > 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, > std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] > > 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, > std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] > > 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, > std::vector<ObjectStore::Transaction, > std::allocator<ObjectStore::Transaction> >&, > boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) > [0x200022c7a6c] > > 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, > ObjectStore::Transaction&&, Context*, Context*, Context*, > boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) > [0x20001c19ee0] > > 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, > boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] > > 8: > (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) > [0x2000202e970] > > 9: > (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) > [0x200020442bc] > > 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) > [0x20001ecea74] > > 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, > ThreadPool::TPHandle&)+0x814) [0x20001de1384] > > 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, > boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) > [0x20001b817d4] > > 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> > const&)+0xb8) [0x20001f98968] > > 14: (OSD::ShardedOpWQ::_process(unsigned int, > ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] > > 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) > [0x200024d60a4] > > 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) > [0x200024da278] > > 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] > > 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] > > 19: (()+0x80fc) [0x2000c53a0fc] > > 20: (()+0x119854) [0x2000f2ad854] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > > > --- begin dump of recent events --- > > -147> 2020-03-18 18:37:51.039443 2000f374000 5 asok(0x2000cd7f230) > register_command perfcounters_dump hook 0x2000ce09e40 > > -146> 2020-03-18 18:37:51.039716 2000f374000 5 asok(0x2000cd7f230) > register_command 1 hook 0x2000ce09e40 > > -145> 2020-03-18 18:37:51.039736 2000f374000 5 asok(0x2000cd7f230) > register_command perf dump hook 0x2000ce09e40 > > -144> 2020-03-18 18:37:51.039769 2000f374000 5 asok(0x2000cd7f230) > register_command perfcounters_schema hook 0x2000ce09e40 > > -143> 2020-03-18 18:37:51.039789 2000f374000 5 asok(0x2000cd7f230) > register_command perf histogram dump hook 0x2000ce09e40 > > -142> 2020-03-18 18:37:51.039807 2000f374000 5 asok(0x2000cd7f230) > register_command 2 hook 0x2000ce09e40 > > -141> 2020-03-18 18:37:51.039823 2000f374000 5 asok(0x2000cd7f230) > register_command perf schema hook 0x2000ce09e40 > > -140> 2020-03-18 18:37:51.039843 2000f374000 5 asok(0x2000cd7f230) > register_command perf histogram schema hook 0x2000ce09e40 > > -139> 2020-03-18 18:37:51.039863 2000f374000 5 asok(0x2000cd7f230) > register_command perf reset hook 0x2000ce09e40 > > -138> 2020-03-18 18:37:51.039881 2000f374000 5 asok(0x2000cd7f230) > register_command config show hook 0x2000ce09e40 > > -137> 2020-03-18 18:37:51.039899 2000f374000 5 asok(0x2000cd7f230) > register_command config help hook 0x2000ce09e40 > > -136> 2020-03-18 18:37:51.039928 2000f374000 5 asok(0x2000cd7f230) > register_command config set hook 0x2000ce09e40 > > -135> 2020-03-18 18:37:51.039949 2000f374000 5 asok(0x2000cd7f230) > register_command config get hook 0x2000ce09e40 > > -134> 2020-03-18 18:37:51.039967 2000f374000 5 asok(0x2000cd7f230) > register_command config diff hook 0x2000ce09e40 > > -133> 2020-03-18 18:37:51.039985 2000f374000 5 asok(0x2000cd7f230) > register_command config diff get hook 0x2000ce09e40 > > -132> 2020-03-18 18:37:51.040005 2000f374000 5 asok(0x2000cd7f230) > register_command log flush hook 0x2000ce09e40 > > -131> 2020-03-18 18:37:51.040021 2000f374000 5 asok(0x2000cd7f230) > register_command log dump hook 0x2000ce09e40 > > -130> 2020-03-18 18:37:51.040038 2000f374000 5 asok(0x2000cd7f230) > register_command log reopen hook 0x2000ce09e40 > > -129> 2020-03-18 18:37:51.040189 2000f374000 5 asok(0x2000cd7f230) > register_command dump_mempools hook 0x2000ce0d038 > > -128> 2020-03-18 18:37:51.099580 2000f374000 -1 WARNING: the > following dangerous and experimental features are enabled: bluestore,rocksdb > > -127> 2020-03-18 18:37:51.102046 2000f374000 -1 WARNING: the > following dangerous and experimental features are enabled: bluestore,rocksdb > > -126> 2020-03-18 18:37:51.102148 2000f374000 0 ceph version 12.2.7 > (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable), process > ceph-osd, pid 1159687 > > -125> 2020-03-18 18:37:51.109860 2000f374000 0 pidfile_write: ignore > empty --pid-file > > -124> 2020-03-18 18:37:51.130256 2000f374000 -1 WARNING: the > following dangerous and experimental features are enabled: bluestore,rocksdb > > -123> 2020-03-18 18:37:51.183798 2000f374000 0 load: jerasure load: > lrc > > -122> 2020-03-18 18:37:51.184330 2000f374000 1 bdev create path > /var/lib/ceph/osd/ceph-6/block type kernel > > -121> 2020-03-18 18:37:51.184492 2000f374000 1 bdev(0x2000cf49800 > /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > > -120> 2020-03-18 18:37:51.184741 2000f374000 1 bdev(0x2000cf49800 > /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize > 8192, using bdev_block_size 4096 anyway > > -119> 2020-03-18 18:37:51.185841 2000f374000 1 bdev(0x2000cf49800 > /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, > 3500 GB) block_size 4096 (4096 B) rotational > > -118> 2020-03-18 18:37:51.186232 2000f374000 1 bdev(0x2000cf49800 > /var/lib/ceph/osd/ceph-6/block) close > > -117> 2020-03-18 18:37:51.498718 2000f374000 1 bdev create path > /var/lib/ceph/osd/ceph-6/block type kernel > > -116> 2020-03-18 18:37:51.498748 2000f374000 1 bdev(0x2000cf5d560 > /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > > -115> 2020-03-18 18:37:51.498811 2000f374000 1 bdev(0x2000cf5d560 > /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize > 8192, using bdev_block_size 4096 anyway > > -114> 2020-03-18 18:37:51.499234 2000f374000 1 bdev(0x2000cf5d560 > /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, > 3500 GB) block_size 4096 (4096 B) rotational > > -113> 2020-03-18 18:37:51.500074 2000f374000 1 bdev create path > /var/lib/ceph/osd/ceph-6/block.db type kernel > > -112> 2020-03-18 18:37:51.500096 2000f374000 1 bdev(0x2000cf5e6a0 > /var/lib/ceph/osd/ceph-6/block.db) open path > /var/lib/ceph/osd/ceph-6/block.db > > -111> 2020-03-18 18:37:51.500170 2000f374000 1 bdev(0x2000cf5e6a0 > /var/lib/ceph/osd/ceph-6/block.db) open backing device/file reports > st_blksize 8192, using bdev_block_size 4096 anyway > > -110> 2020-03-18 18:37:51.500815 2000f374000 1 bdev(0x2000cf5e6a0 > /var/lib/ceph/osd/ceph-6/block.db) open size 39998980096 (0x950200000, > 38146 MB) block_size 4096 (4096 B) rotational > > -109> 2020-03-18 18:37:51.502625 2000f374000 1 bdev create path > /var/lib/ceph/osd/ceph-6/block type kernel > > -108> 2020-03-18 18:37:51.502651 2000f374000 1 bdev(0x2000cf5ed80 > /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > > -107> 2020-03-18 18:37:51.502718 2000f374000 1 bdev(0x2000cf5ed80 > /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize > 8192, using bdev_block_size 4096 anyway > > -106> 2020-03-18 18:37:51.503137 2000f374000 1 bdev(0x2000cf5ed80 > /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, > 3500 GB) block_size 4096 (4096 B) rotational > > -105> 2020-03-18 18:37:51.549269 2000f374000 0 set rocksdb option > compaction_readahead_size = 2MB > > -104> 2020-03-18 18:37:51.549349 2000f374000 0 set rocksdb option > compaction_style = kCompactionStyleLevel > > -103> 2020-03-18 18:37:51.552610 2000f374000 0 set rocksdb option > compaction_threads = 32 > > -102> 2020-03-18 18:37:51.552652 2000f374000 0 set rocksdb option > compression = kNoCompression > > -101> 2020-03-18 18:37:51.553442 2000f374000 0 set rocksdb option > flusher_threads = 8 > > -100> 2020-03-18 18:37:51.553508 2000f374000 0 set rocksdb option > level0_file_num_compaction_trigger = 64 > > -99> 2020-03-18 18:37:51.553536 2000f374000 0 set rocksdb option > level0_slowdown_writes_trigger = 128 > > -98> 2020-03-18 18:37:51.553559 2000f374000 0 set rocksdb option > level0_stop_writes_trigger = 256 > > -97> 2020-03-18 18:37:51.553579 2000f374000 0 set rocksdb option > max_background_compactions = 64 > > -96> 2020-03-18 18:37:51.553601 2000f374000 0 set rocksdb option > max_bytes_for_level_base = 2GB > > -95> 2020-03-18 18:37:51.553624 2000f374000 0 set rocksdb option > max_write_buffer_number = 64 > > -94> 2020-03-18 18:37:51.553646 2000f374000 0 set rocksdb option > min_write_buffer_number_to_merge = 32 > > -93> 2020-03-18 18:37:51.553665 2000f374000 0 set rocksdb option > recycle_log_file_num = 64 > > -92> 2020-03-18 18:37:51.553687 2000f374000 0 set rocksdb option > target_file_size_base = 4MB > > -91> 2020-03-18 18:37:51.553708 2000f374000 0 set rocksdb option > write_buffer_size = 4MB > > -90> 2020-03-18 18:37:51.553892 2000f374000 0 set rocksdb option > compaction_readahead_size = 2MB > > -89> 2020-03-18 18:37:51.553923 2000f374000 0 set rocksdb option > compaction_style = kCompactionStyleLevel > > -88> 2020-03-18 18:37:51.553948 2000f374000 0 set rocksdb option > compaction_threads = 32 > > -87> 2020-03-18 18:37:51.553973 2000f374000 0 set rocksdb option > compression = kNoCompression > > -86> 2020-03-18 18:37:51.553994 2000f374000 0 set rocksdb option > flusher_threads = 8 > > -85> 2020-03-18 18:37:51.554016 2000f374000 0 set rocksdb option > level0_file_num_compaction_trigger = 64 > > -84> 2020-03-18 18:37:51.554043 2000f374000 0 set rocksdb option > level0_slowdown_writes_trigger = 128 > > -83> 2020-03-18 18:37:51.554065 2000f374000 0 set rocksdb option > level0_stop_writes_trigger = 256 > > -82> 2020-03-18 18:37:51.554084 2000f374000 0 set rocksdb option > max_background_compactions = 64 > > -81> 2020-03-18 18:37:51.554106 2000f374000 0 set rocksdb option > max_bytes_for_level_base = 2GB > > -80> 2020-03-18 18:37:51.554133 2000f374000 0 set rocksdb option > max_write_buffer_number = 64 > > -79> 2020-03-18 18:37:51.554154 2000f374000 0 set rocksdb option > min_write_buffer_number_to_merge = 32 > > -78> 2020-03-18 18:37:51.554174 2000f374000 0 set rocksdb option > recycle_log_file_num = 64 > > -77> 2020-03-18 18:37:51.554196 2000f374000 0 set rocksdb option > target_file_size_base = 4MB > > -76> 2020-03-18 18:37:51.554232 2000f374000 0 set rocksdb option > write_buffer_size = 4MB > > -75> 2020-03-18 18:37:56.382110 2000f374000 0 <cls> > /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/hello/cls_hello.cc:296: > loading cls_hello > > -74> 2020-03-18 18:37:56.383845 2000f374000 0 _get_class not > permitted to load lua > > -73> 2020-03-18 18:37:56.386594 2000f374000 0 _get_class not > permitted to load sdk > > -72> 2020-03-18 18:37:56.395800 2000f374000 0 _get_class not > permitted to load kvs > > -71> 2020-03-18 18:37:56.398226 2000f374000 0 <cls> > /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/cephfs/cls_cephfs.cc:197: > loading cephfs > > -70> 2020-03-18 18:37:56.433293 2000f374000 0 osd.6 5247 crush map > has features 432629239337189376, adjusting msgr requires for clients > > -69> 2020-03-18 18:37:56.433330 2000f374000 0 osd.6 5247 crush map > has features 432629239337189376 was 8705, adjusting msgr requires for mons > > -68> 2020-03-18 18:37:56.433357 2000f374000 0 osd.6 5247 crush map > has features 1009089991640629248, adjusting msgr requires for osds > > -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > > -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs > opened 170 pgs > > -65> 2020-03-18 18:39:03.377040 2000f374000 0 osd.6 5247 using > weightedpriority op queue with priority op cut off at 196. > > -64> 2020-03-18 18:39:03.413901 2000f374000 -1 osd.6 5247 > log_to_monitors {default=true} > > -63> 2020-03-18 18:39:03.663128 2000f374000 0 osd.6 5247 done with > init, starting boot process > > -62> 2020-03-18 18:39:03.663856 201d65fdb40 4 mgrc handle_mgr_map > Got map version 34 > > -61> 2020-03-18 18:39:03.663947 201d65fdb40 4 mgrc handle_mgr_map > Active mgr is now 192.168.230.120:6808/44007 > > -60> 2020-03-18 18:39:03.663972 201d65fdb40 4 mgrc reconnect > Starting new session with 192.168.230.120:6808/44007 > > -59> 2020-03-18 18:39:03.667814 201d65fdb40 4 mgrc > handle_mgr_configure stats_period=5 > > -58> 2020-03-18 18:39:03.667831 201d65fdb40 4 mgrc > handle_mgr_configure updated stats threshold: 5 > > -57> 2020-03-18 18:39:03.752253 2000dae5b40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.121:6802/33592 > conn(0x20014018040 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -56> 2020-03-18 18:39:03.754040 2000fc61b40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.121:6800/33864 > conn(0x2001403ee60 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -55> 2020-03-18 18:39:03.762441 2000ec1fb40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.120:6804/13410 > conn(0x20014079940 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -54> 2020-03-18 18:39:03.762860 2000ec1fb40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.120:6806/13743 > conn(0x20014083980 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -53> 2020-03-18 18:39:03.765775 2000fc61b40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.121:6806/12423 > conn(0x2001407e030 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -52> 2020-03-18 18:39:03.767284 2000ec1fb40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.121:6804/11599 > conn(0x2001408c660 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -51> 2020-03-18 18:39:03.769434 2000dae5b40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.120:6800/12205 > conn(0x20014087ff0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -50> 2020-03-18 18:39:03.775190 2000ec1fb40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.120:6802/12991 > conn(0x2001409d300 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -49> 2020-03-18 18:39:04.009755 2000dae5b40 0 -- > 192.168.240.122:6806/1159687 >> 192.168.240.122:6804/1159466 > conn(0x200140b3420 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=0).handle_connect_msg: challenging authorizer > > -48> 2020-03-18 18:39:05.177544 2000ec1fb40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.202:0/3091162658 > conn(0x200140cd900 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > -47> 2020-03-18 18:39:05.402465 2000dae5b40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4289863819 > conn(0x200140d8500 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > -46> 2020-03-18 18:39:09.483237 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b442b93f:::rbd_data.911772ae8944a.0000000000002aa7:head have 3097'452 > flags = none tried to add 3097'452 flags = none > > -45> 2020-03-18 18:39:09.483318 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b445877b:::rbd_data.2fa7e6b8b4567.000000000000002d:head have 1915'212 > flags = none tried to add 1915'212 flags = none > > -44> 2020-03-18 18:39:09.483336 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44674a4:::rbd_data.110706b8b4567.0000000000000659:head have 1915'213 > flags = none tried to add 1915'213 flags = none > > -43> 2020-03-18 18:39:09.483351 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44757d1:::rbd_data.cd282238e1f29.0000000000009ea2:head have 5165'734 > flags = none tried to add 5165'734 flags = none > > -42> 2020-03-18 18:39:09.483366 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4490029:::rbd_data.8d8146b8b4567.00000000000080a0:head have 2855'272 > flags = none tried to add 2855'272 flags = none > > -41> 2020-03-18 18:39:09.483381 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44950ae:::rbd_data.5f15f625558ec.0000000000009fd2:head have 1915'214 > flags = none tried to add 1915'214 flags = none > > -40> 2020-03-18 18:39:09.483395 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b449ce0d:::rbd_data.9117a327b23c6.00000000000074a6:head have 3798'560 > flags = none tried to add 3798'560 flags = none > > -39> 2020-03-18 18:39:09.483409 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44a69ad:::rbd_data.8b4a76b8b4567.000000000000017a:head have 2197'242 > flags = none tried to add 2197'242 flags = none > > -38> 2020-03-18 18:39:09.483423 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44bbb34:::rbd_data.8922074b0dc51.00000000000098a5:head have 3099'543 > flags = delete tried to add 3099'543 flags = delete > > -37> 2020-03-18 18:39:09.483438 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44be196:::rbd_data.cd8e219495cff.00000000000192a0:head have 5169'1101 > flags = delete tried to add 5169'1101 flags = delete > > -36> 2020-03-18 18:39:09.483454 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44cd1c7:::rbd_data.5f15f625558ec.000000000000820b:head have 1915'215 > flags = none tried to add 1915'215 flags = none > > -35> 2020-03-18 18:39:09.483469 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44e04c0:::rbd_data.cd66c46e87ccd.0000000000015aa4:head have 5175'1280 > flags = none tried to add 5175'1280 flags = none > > -34> 2020-03-18 18:39:09.483483 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b44fa767:::rbd_data.ccf5e625558ec.00000000000150a0:head have 5166'844 > flags = delete tried to add 5166'844 flags = delete > > -33> 2020-03-18 18:39:09.483497 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4518cb0:::rbd_data.589572ae8944a.0000000000000433:head have 1805'174 > flags = delete tried to add 1805'174 flags = delete > > -32> 2020-03-18 18:39:09.483511 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4539e4a:::rbd_data.5f15f625558ec.000000000000031a:head have 1915'216 > flags = none tried to add 1915'216 flags = none > > -31> 2020-03-18 18:39:09.483525 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4540dfd:::rbd_data.cd66c46e87ccd.000000000001d6a6:head have 5175'1283 > flags = none tried to add 5175'1283 flags = none > > -30> 2020-03-18 18:39:09.483539 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45474ee:::rbd_data.2fa7e6b8b4567.000000000000003a:head have 1915'217 > flags = none tried to add 1915'217 flags = none > > -29> 2020-03-18 18:39:09.483553 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45506f1:::rbd_data.9177446e87ccd.00000000000036c3:head have 5175'1284 > flags = none tried to add 5175'1284 flags = none > > -28> 2020-03-18 18:39:09.483567 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4576e29:::rbd_data.589572ae8944a.0000000000001207:head have 1805'179 > flags = delete tried to add 1805'179 flags = delete > > -27> 2020-03-18 18:39:09.483582 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4578057:::rbd_data.ccf5e625558ec.00000000000096a5:head have 5166'785 > flags = delete tried to add 5166'785 flags = delete > > -26> 2020-03-18 18:39:09.483595 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b457fe94:::rbd_data.589572ae8944a.0000000000004e27:head have 1805'200 > flags = delete tried to add 1805'200 flags = delete > > -25> 2020-03-18 18:39:09.483610 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4592563:::rbd_data.9117a327b23c6.000000000000b2a0:head have 3894'562 > flags = none tried to add 3894'562 flags = none > > -24> 2020-03-18 18:39:09.483625 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45bb3ff:::rbd_data.c6c99507ed7ab.000000000000aea0:head have 5172'1141 > flags = none tried to add 5172'1141 flags = none > > -23> 2020-03-18 18:39:09.483639 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45d7c66:::rbd_data.110706b8b4567.0000000000000e07:head have 1915'218 > flags = none tried to add 1915'218 flags = none > > -22> 2020-03-18 18:39:09.483654 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45e1a62:::rbd_data.589572ae8944a.0000000000004733:head have 1805'198 > flags = delete tried to add 1805'198 flags = delete > > -21> 2020-03-18 18:39:09.483667 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b45f03a3:::rbd_data.914c22ae8944a.00000000000058a8:head have 3908'593 > flags = delete tried to add 3908'593 flags = delete > > -20> 2020-03-18 18:39:09.483681 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4604fdd:::rbd_data.c6c99507ed7ab.00000000000004ca:head have 5176'1523 > flags = none tried to add 5176'1523 flags = none > > -19> 2020-03-18 18:39:09.483695 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4636d5b:::rbd_data.cd282238e1f29.000000000000b6a1:head have 5165'735 > flags = none tried to add 5165'735 flags = none > > -18> 2020-03-18 18:39:09.483710 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b463cd0a:::rbd_data.9117a327b23c6.000000000000c0a8:head have 3908'563 > flags = none tried to add 3908'563 flags = none > > -17> 2020-03-18 18:39:09.483724 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b468ca37:::rbd_data.9177446e87ccd.000000000000040a:head have 4427'730 > flags = none tried to add 4427'730 flags = none > > -16> 2020-03-18 18:39:09.483738 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4696438:::rbd_data.9177446e87ccd.0000000000000c53:head have 4743'732 > flags = none tried to add 4743'732 flags = none > > -15> 2020-03-18 18:39:09.483752 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b46be061:::rbd_data.c70aa2eb141f2.0000000000001ea6:head have 5175'1287 > flags = none tried to add 5175'1287 flags = none > > -14> 2020-03-18 18:39:09.483766 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b46bfc4e:::rbd_data.cda772ae8944a.00000000000020a4:head have 5172'1144 > flags = none tried to add 5172'1144 flags = none > > -13> 2020-03-18 18:39:09.483782 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b46e7efd:::rbd_data.110706b8b4567.0000000000001c01:head have 1915'219 > flags = none tried to add 1915'219 flags = none > > -12> 2020-03-18 18:39:09.483796 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b46ece7b:::rbd_data.8ff1766334873.00000000000000ae:head have 3092'344 > flags = delete tried to add 3092'344 flags = delete > > -11> 2020-03-18 18:39:09.483811 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b4786571:::rbd_data.8f6b3643c9869.00000000000040a0:head have 3092'381 > flags = delete tried to add 3092'381 flags = delete > > -10> 2020-03-18 18:39:09.483825 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b47888b8:::rbd_data.589572ae8944a.0000000000004534:head have 1805'197 > flags = delete tried to add 1805'197 flags = delete > > -9> 2020-03-18 18:39:09.483840 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b478b833:::rbd_data.589572ae8944a.00000000000048ff:head have 1805'199 > flags = delete tried to add 1805'199 flags = delete > > -8> 2020-03-18 18:39:09.483854 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b47ec645:::rbd_data.8b4a76b8b4567.00000000000001d2:head have 2197'271 > flags = none tried to add 2197'271 flags = none > > -7> 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 > flags = none tried to add 3291'557 flags = none > > -6> 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 > flags = none tried to add 4738'731 flags = none > > -5> 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d > unexpected need for > 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 > flags = delete tried to add 1789'169 flags = delete > > -4> 2020-03-18 18:39:20.985370 2000fc61b40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 > conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > -3> 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 > conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > -2> 2020-03-18 18:39:21.495101 2000fc61b40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 > conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > -1> 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- > 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 > conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 > l=1).handle_connect_msg: challenging authorizer > > 0> 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal > (Bus error) ** > > in thread 201e35fdb40 thread_name:tp_osd_tp > > > > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) > luminous (stable) > > 1: (()+0x145882c) [0x2000245882c] > > 2: (()+0x19890) [0x2000c54b890] > > 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, > std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] > > 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, > std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] > > 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, > std::vector<ObjectStore::Transaction, > std::allocator<ObjectStore::Transaction> >&, > boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) > [0x200022c7a6c] > > 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, > ObjectStore::Transaction&&, Context*, Context*, Context*, > boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) > [0x20001c19ee0] > > 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, > boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] > > 8: > (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) > [0x2000202e970] > > 9: > (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) > [0x200020442bc] > > 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) > [0x20001ecea74] > > 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, > ThreadPool::TPHandle&)+0x814) [0x20001de1384] > > 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, > boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) > [0x20001b817d4] > > 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> > const&)+0xb8) [0x20001f98968] > > 14: (OSD::ShardedOpWQ::_process(unsigned int, > ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] > > 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) > [0x200024d60a4] > > 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) > [0x200024da278] > > 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] > > 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] > > 19: (()+0x80fc) [0x2000c53a0fc] > > 20: (()+0x119854) [0x2000f2ad854] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > > > > > > > huxiaoyu(a)horebdata.cn > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

3 years, 11 months

1
1
0 0

Re: upmap balancer and consequences of osds briefly marked out

by Dan van der Ster

Hi, You're correct that all the relevant upmap entries are removed when an OSD is marked out. You can try to use this script which will recreate them and get the cluster back to HEALTH_OK quickly: https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-rema… Cheers, Dan On Fri, May 1, 2020 at 9:36 AM Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: > > Hi all, > > We're using upmap balancer which has made a huge improvement in evenly distributing data on our osds and has provided a substantial increase in usable capacity. > > Currently on ceph version: 12.2.13 luminous > > We ran into a firewall issue recently which led to a large number of osds being briefly marked 'down' & 'out'. The osds came back 'up' & 'in' after about 25 mins and the cluster was fine but had to perform a significant amount of backfilling/recovery despite there being no end-user client I/O during that period. > > Presumably the large number of remapped pgs and backfills were due to pg_upmap_items being removed from the osdmap when osds were marked out and subsequently those pgs were redistributed using the default crush algorithm. > As a result of the brief outage our cluster became significantly imbalanced again with several osds very close to full. > Is there any reasonable mitigation for that scenario? > > The auto-balancer will not perform optimizations while there are degraded pgs, so it would only start reapplying pg upmap exceptions after initial recovery is complete (at which point capacity may be dangerously reduced). > Similarly, as admins, we normally only apply changes when the cluster is in a healthy state, but if the same issue were to occur again would it be advisable to manually apply balancer plans while initial recovery is still taking place? > > I guess my concern from this experience is that making use of the capacity gained by using upmap balancer appears to carry some risk. i.e. it's possible for a brief outage to remove those space efficiencies relatively quickly and potentially result in full osds/cluster before the automatic balancer is able to resume and redistribute pgs using upmap. > > Curious whether others have any thoughts or experience regarding this. > > Cheers, > Dylan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 years, 11 months

2
1
0 0

2024

2023

2022

2021

2020

2019

ceph-users May 2020