- ceph-users - lists.ceph.io

OSD disk usage
by Kristof Coucke 27 Oct '20

27 Oct '20

Hi, My cluster is in a warning state as it is rebalancing after I've added a bunch of disks. (no issue here!) Though, there are a few things which I just cannot understand... I hope someone can help me... I'm getting hopeless finding the answers... If you can answer any question (even one), it will be greatly appreciated. Environment: Ceph Nautilus 14.2.11, 282 OSDs, mainly erasure coding. 1. Ceph dashboard shows more PGs for an OSD then I can extract from the pg dump information. I assume that the value in the dashboard is coming from Prometheus, though I'm not sure. Querying Prometheus thru Grafana also gives me the same figures as the one I see in the dashboard (which is incorrect). I can't find out how this value is calculated... All the information I can find regarding calculating the pg's stored on an osd are derived from pg dump :-( Help help help 2. As the cluster is rebalancing and there is a huge gap between the acting and the up osd's for a bunch of pg's, some disks sometimes have slow responses due to the backfilling and are from time to time marked as down. (I know I can go around this one by setting "nodown" temporarily). If the disk which is going down is in an acting set of a specific pg (not in the upmap for that pg), then the pg will be marked as degraded as the system will try to rebuild the missing data towards an osd in the up set. This I understand... What I don't understand is that when the disk is restarted and thus marked back as "up" (or simply being patient...), it doesn't add that osd back to the acting set... Restarting other osds (and thus causing more peering operations again) results in the disk being added again to the acting set... I don't understand how this is happening. 3. Addition to question 2: If an osd which was in an acting set went down and is removed from the acting set (replaced with -1), which process will remove the obsolete data from the osd that went down once it is back up? Which process is cleaning up the obsolete copy in the end? Does scrubbing take this process into account? I assume that this might be related to my first question too. 4. I'm data mining the pg dump output... If I get an answer on all the previous questions, this will probably be answered automatically: When I look at all the acting pg's for a specific osd, and I look at the numbytes of that pg and I calculate the size that should be stored on that osd (taking into account the erasure coding process) I get a difference between disk space which should be used and effectively used. E.g. for a disk the system is saying 11.3 TiB is being used... Calculating it using pg dump gives me +/- 10TiB which should be used. I know that a delta might occur due to the block size etc, but it doesn't seem correct as the usage is too high... I've tried to search on processes which clean up the osds, garbage collection, etc. but no good information is available. You can find tons of information related to garbage collection in combination with RGW, but not for the RADOS mechanism... I really can't find any clue how the pg's of a disk are removed after it went down and is not being used again in the acting/up set of those pg's... Another question which I probably should post in the dev group, but which IDE is recommended for developing in the Ceph project? I'm working on a Mac... Don't know if there are any recommendations. I really hope I can get some help on these questions. Many thanks! Regards, Kristof

2 1

Re: Ceph Octopus and Snapshot Schedules
by Adam Boyhan 27 Oct '20

27 Oct '20

Benji is independent from Ceph. It utilizes Ceph snapshots to do the backups, but it has nothing to do with managing Ceph snapshots. I am simply looking for the ability to manage Ceph snapshots. For example. Take a snapshot every 30 minutes, keep 8 of those 30 minute snapshots.

1 0

Switching to a private repository
by Liam MacKenzie 26 Oct '20

26 Oct '20

Hi, What is the correct procedure to change a running cluster deployed with cephadm to a custom container from a private repository that requires a login? I would have thought something like this would have been the right way, but the second command fails with an authentication error. $ cephadm registry-login --registry-url myregistry.com/ceph/ --registry-username 'robot$myuser --registry-password 'mysecret' $ ceph orch upgrade start --image myregistry.com/ceph/ceph:v15.2.5 Thanks, Liam

2 1

Cephadm: module not found
by Marco Venuti 26 Oct '20

26 Oct '20

Hi, I'm experimenting ceph on a (small) test cluster. I'm using version 15.2.5 deployed with cephadm. I was trying to do some "disaster" testing, such as wiping a disk in order to simulate a hardware failure, destroy the osd and recreate it, all of which I managed to do successfully. However, a few hours after this test, the orchestrator failed with no apparent reason. I tried to disable and reenable cephadm, but with no luck: # ceph orch ls Error ENOENT: No orchestrator configured (try `ceph orch set backend`) # ceph orch set backend cephadm Error ENOENT: Module not found What could have happened? Is there some way to reenable cephadm? Thanks, Marco

3 6

Ceph not showing full capacity
by Amudhan P 26 Oct '20

26 Oct '20

Hi, I have created a test Ceph cluster with Ceph Octopus using cephadm. Cluster total RAW disk capacity is 262 TB but it's allowing to use of only 132TB. I have not set quota for any of the pool. what could be the issue? Output from :- ceph -s cluster: id: f8bc7682-0d11-11eb-a332-0cc47a5ec98a health: HEALTH_WARN clock skew detected on mon.strg-node3, mon.strg-node2 2 backfillfull osd(s) 4 pool(s) backfillfull 1 pools have too few placement groups services: mon: 3 daemons, quorum strg-node1,strg-node3,strg-node2 (age 7m) mgr: strg-node3.jtacbn(active, since 7m), standbys: strg-node1.gtlvyv mds: cephfs-strg:1 {0=cephfs-strg.strg-node1.lhmeea=up:active} 1 up:standby osd: 48 osds: 48 up (since 7m), 48 in (since 5d) task status: scrub status: mds.cephfs-strg.strg-node1.lhmeea: idle data: pools: 4 pools, 289 pgs objects: 17.29M objects, 66 TiB usage: 132 TiB used, 130 TiB / 262 TiB avail pgs: 288 active+clean 1 active+clean+scrubbing+deep mounted volume shows node1:/ 67T 66T 910G 99% /mnt/cephfs

5 13

Re: Huge HDD ceph monitor usage
by Ing. Luis Felipe Domínguez Vega 26 Oct '20

26 Oct '20

Only 9M ------------------------------------------------------------------------------ root@fond-beagle:/var/lib/ceph/mon/ceph-fond-beagle/store.db# ls -lh *.log -rw------- 1 ceph ceph 9.9M Oct 26 10:57 1443554.log ------------------------------------------------------------------------------- El 2020-10-26 10:55, Anthony D'Atri escribió: > See if you have big LOG* files inside misleading you > >> On Oct 26, 2020, at 7:13 AM, Ing. Luis Felipe Domínguez Vega >> <luis.dominguez(a)desoft.cu> wrote: >> >> How can i free the store of ceph monitor?: >> >> ------------------------------------------------------------------------ >> root@fond-beagle:/var/lib/ceph/mon/ceph-fond-beagle# du -h -d1 >> 542G ./store.db >> 542G . >> ------------------------------------------------------------------------ >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Hardware for new OSD nodes.
by Dave Hall 26 Oct '20

26 Oct '20

Hello, (BTW, Nautilus 14.2.7 on Debian non-container.) We're about to purchase more OSD nodes for our cluster, but I have a couple questions about hardware choices. Our original nodes were 8 x 12TB SAS drives and a 1.6TB Samsung NVMe card for WAL, DB, etc. We chose the NVMe card for performance since it has an 8 lane PCIe interface. However, we're currently BlueFS spillovers. The Tyan chassis we are considering has the option of 4 x U.2 NVMe bays - each with 4 PCIe lanes, (and 8 SAS bays). It has occurred to me that I might stripe 4 1TB NVMe drives together to get much more space for WAL/DB and a net performance of 16 PCIe lanes. Any thoughts on this approach? Also, any thoughts/recommendations on 12TB OSD drives? For price/capacity this is a good size for us, but I'm wondering if my BlueFS spillovers are resulting from using drives that are too big. I also thought I might have seen some comments about cutting large drives into multiple OSDs - could that be? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

4 15

Question about expansion existing Ceph cluster - adding OSDs
by Kristof Coucke 26 Oct '20

26 Oct '20

Hi, I have a cluster with 182 OSDs, this has been expanded towards 282 OSDs. Some disks were near full. The new disks have been added with initial weight = 0. The original plan was to increase this slowly towards their full weight using the gentle reweight script. However, this is going way too slow and I'm also having issues now with "backfill_toofull". Can I just add all the OSDs with their full weight, or will I get a lot of issues when I'm doing that? I know that a lot of PGs will have to be replaced, but increasing the weight slowly will take a year at the current speed. I'm already playing with the max backfill to increase the speed, but every time I increase the weight it will take a lot of time again... I can face the fact that there will be a performance decrease. Looking forward to your comments! Regards, Kristof

3 6

Issues with the ceph-bluestore-tool during cluster upgrade from Mimic to Nautilus
by Jean-Philippe Méthot 26 Oct '20

26 Oct '20

Hi, We’re upgrading our cluster OSD node per OSD node to Nautilus from Mimic. From some release notes, it was recommended to run the following command to fix stats after an upgrade : ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-0 However, running that command gives us the following error message: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: In > function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 time 2020-09-10 14:40:25.872353 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: 53 > : FAILED ceph_assert(r == 0) > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f1a5a823025] > 2: (()+0x25c1ed) [0x7f1a5a8231ed] > 3: (()+0x3c7a4f) [0x55b33537ca4f] > 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 8: (main()+0x10b3) [0x55b335187493] > 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 10: (()+0x1f9b5f) [0x55b3351aeb5f] > 2020-09-10 14:40:25.873 7f1a6467eec0 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: In function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 time 2020-09-10 14:40:25.872353 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: 53: FAILED ceph_assert(r == 0) > > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f1a5a823025] > 2: (()+0x25c1ed) [0x7f1a5a8231ed] > 3: (()+0x3c7a4f) [0x55b33537ca4f] > 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 8: (main()+0x10b3) [0x55b335187493] > 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 10: (()+0x1f9b5f) [0x55b3351aeb5f] > *** Caught signal (Aborted) ** > in thread 7f1a6467eec0 thread_name:ceph-bluestore- > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (()+0xf630) [0x7f1a58cf0630] > 2: (gsignal()+0x37) [0x7f1a574be387] > 3: (abort()+0x148) [0x7f1a574bfa78] > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x7f1a5a823074] > 5: (()+0x25c1ed) [0x7f1a5a8231ed] > 6: (()+0x3c7a4f) [0x55b33537ca4f] > 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 9: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 11: (main()+0x10b3) [0x55b335187493] > 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 13: (()+0x1f9b5f) [0x55b3351aeb5f] > 2020-09-10 14:40:25.874 7f1a6467eec0 -1 *** Caught signal (Aborted) ** > in thread 7f1a6467eec0 thread_name:ceph-bluestore- > > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (()+0xf630) [0x7f1a58cf0630] > 2: (gsignal()+0x37) [0x7f1a574be387] > 3: (abort()+0x148) [0x7f1a574bfa78] > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x7f1a5a823074] > 5: (()+0x25c1ed) [0x7f1a5a8231ed] > 6: (()+0x3c7a4f) [0x55b33537ca4f] > 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 9: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 11: (main()+0x10b3) [0x55b335187493] > 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 13: (()+0x1f9b5f) [0x55b3351aeb5f] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. What could be the source of this error? I haven’t found much of anything about it online. Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc. 4414-4416 Louis B Mayer Laval, QC, H7P 0G1, Canada TEL : +1.514.802.1644 - Poste : 2644 FAX : +1.514.612.0678 CA/US : 1.855.774.4678 FR : 01 76 60 41 43 UK : 0808 189 0423

3 7

Re: Issues with the ceph-bluestore-tool during cluster upgrade from Mimic to Nautilus
by Igor Fedotov 26 Oct '20

26 Oct '20

On 9/25/2020 6:07 PM, Saber(a)PlanetHoster.info wrote: > Hi Igor, > > The only thing abnormal about this osdstore is that it was created by > Mimic 13.2.8 and I can see that the OSDs size of this osdstore are not > the same as the others in the cluster (while they should be exactly > the same size). > > Can it be https://tracker.ceph.com/issues/39151 ? hmm, may be... Did you change H/W at some point for this OSD's node as it happened in the ticket? And it's still unclear to me if the issue is reproducible for you. Could you please also run fsck (at first) and then repair for this OSD and collect log(s). Thanks, Igor > > Thanks! > Saber > CTO @PlanetHoster > >> On Sep 25, 2020, at 5:46 AM, Igor Fedotov <ifedotov(a)suse.de >> <mailto:ifedotov@suse.de>> wrote: >> >> Hi Saber, >> >> I don't think this is related. New assertion happens along the write >> path while the original one occurred on allocator shutdown. >> >> >> Unfortunately there are not much information to troubleshoot this... >> Are you able to reproduce the case? >> >> >> Thanks, >> >> Igor >> >> On 9/25/2020 4:21 AM, Saber(a)PlanetHoster.info wrote: >>> Hi Igor, >>> >>> We had an osd crash a week after running Nautilus. I have attached >>> the logs, is it related to the same bug? >>> >>> >>> >>> >>> Thanks, >>> Saber >>> CTO @PlanetHoster >>> >>>> On Sep 14, 2020, at 10:22 AM, Igor Fedotov <ifedotov(a)suse.de >>>> <mailto:ifedotov@suse.de>> wrote: >>>> >>>> Thanks! >>>> >>>> Now got the root cause. The fix is on its way... >>>> >>>> Meanwhile you might want to try to workaround the issue via setting >>>> "bluestore_hybrid_alloc_mem_cap" to 0 or using different allocator, >>>> e.g. avl for bluestore_allocator (and optionally for >>>> bluefs_allocator too). >>>> >>>> >>>> Hope this helps, >>>> >>>> Igor. >>>> >>>> >>>> >>>> On 9/14/2020 5:02 PM, Jean-Philippe Méthot wrote: >>>>> Alright, here’s the full log file. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Jean-Philippe Méthot >>>>> Senior Openstack system administrator >>>>> Administrateur système Openstack sénior >>>>> PlanetHoster inc. >>>>> 4414-4416 Louis B Mayer >>>>> Laval, QC, H7P 0G1, Canada >>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>> FAX : +1.514.612.0678 >>>>> CA/US : 1.855.774.4678 >>>>> FR : 01 76 60 41 43 >>>>> UK : 0808 189 0423 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Le 14 sept. 2020 à 06:49, Igor Fedotov <ifedotov(a)suse.de >>>>>> <mailto:ifedotov@suse.de>> a écrit : >>>>>> >>>>>> Well, I can see duplicate admin socket command >>>>>> registration/de-registration (and the second de-registration >>>>>> asserts) but don't understand how this could happen. >>>>>> >>>>>> Would you share the full log, please? >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Igor >>>>>> >>>>>> On 9/11/2020 7:26 PM, Jean-Philippe Méthot wrote: >>>>>>> Here’s the out file, as requested. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Jean-Philippe Méthot >>>>>>> Senior Openstack system administrator >>>>>>> Administrateur système Openstack sénior >>>>>>> PlanetHoster inc. >>>>>>> 4414-4416 Louis B Mayer >>>>>>> Laval, QC, H7P 0G1, Canada >>>>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>>>> FAX : +1.514.612.0678 >>>>>>> CA/US : 1.855.774.4678 >>>>>>> FR : 01 76 60 41 43 >>>>>>> UK : 0808 189 0423 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Le 11 sept. 2020 à 10:38, Igor Fedotov <ifedotov(a)suse.de >>>>>>>> <mailto:ifedotov@suse.de>> a écrit : >>>>>>>> >>>>>>>> Could you please run: >>>>>>>> >>>>>>>> CEPH_ARGS="--log-file log --debug-asok 5" ceph-bluestore-tool >>>>>>>> repair --path <...> ; cat log | grep asok > out >>>>>>>> >>>>>>>> and share 'out' file. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Igor >>>>>>>> >>>>>>>> On 9/11/2020 5:15 PM, Jean-Philippe Méthot wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> We’re upgrading our cluster OSD node per OSD node to Nautilus >>>>>>>>> from Mimic. From some release notes, it was recommended to run >>>>>>>>> the following command to fix stats after an upgrade : >>>>>>>>> >>>>>>>>> ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-0 >>>>>>>>> >>>>>>>>> However, running that command gives us the following error >>>>>>>>> message: >>>>>>>>> >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: In >>>>>>>>>> function 'virtual Allocator::SocketHook::~SocketHook()' >>>>>>>>>> thread 7f1a6467eec0 time 2020-09-10 14:40:25.872353 >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: 53 >>>>>>>>>> : FAILED ceph_assert(r == 0) >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x14a) [0x7f1a5a823025] >>>>>>>>>> 2: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 3: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 6: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 8: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 10: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> 2020-09-10 14:40:25.873 7f1a6467eec0 -1 >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: In function 'virtual >>>>>>>>>> Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 >>>>>>>>>> time 2020-09-10 14:40:25.872353 >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: 53: FAILED ceph_assert(r == 0) >>>>>>>>>> >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x14a) [0x7f1a5a823025] >>>>>>>>>> 2: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 3: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 6: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 8: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 10: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> *** Caught signal (Aborted) ** >>>>>>>>>> in thread 7f1a6467eec0 thread_name:ceph-bluestore- >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (()+0xf630) [0x7f1a58cf0630] >>>>>>>>>> 2: (gsignal()+0x37) [0x7f1a574be387] >>>>>>>>>> 3: (abort()+0x148) [0x7f1a574bfa78] >>>>>>>>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x199) [0x7f1a5a823074] >>>>>>>>>> 5: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 6: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 9: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 11: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 13: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> 2020-09-10 14:40:25.874 7f1a6467eec0 -1 *** Caught signal >>>>>>>>>> (Aborted) ** >>>>>>>>>> in thread 7f1a6467eec0 thread_name:ceph-bluestore- >>>>>>>>>> >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (()+0xf630) [0x7f1a58cf0630] >>>>>>>>>> 2: (gsignal()+0x37) [0x7f1a574be387] >>>>>>>>>> 3: (abort()+0x148) [0x7f1a574bfa78] >>>>>>>>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x199) [0x7f1a5a823074] >>>>>>>>>> 5: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 6: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 9: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 11: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 13: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> NOTE: a copy of the executable, or `objdump -rdS >>>>>>>>>> <executable>` is needed to interpret this. >>>>>>>>> >>>>>>>>> What could be the source of this error? I haven’t found much >>>>>>>>> of anything about it online. >>>>>>>>> >>>>>>>>> >>>>>>>>> Jean-Philippe Méthot >>>>>>>>> Senior Openstack system administrator >>>>>>>>> Administrateur système Openstack sénior >>>>>>>>> PlanetHoster inc. >>>>>>>>> 4414-4416 Louis B Mayer >>>>>>>>> Laval, QC, H7P 0G1, Canada >>>>>>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>>>>>> FAX : +1.514.612.0678 >>>>>>>>> CA/US : 1.855.774.4678 >>>>>>>>> FR : 01 76 60 41 43 >>>>>>>>> UK : 0808 189 0423 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>> <mailto:ceph-users@ceph.io> >>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>> <mailto:ceph-users-leave@ceph.io> >>>>>>> >>>>> >>> >

2 1