- ceph-users - lists.ceph.io

how to "undelete" a pool
by Peter Sarossy 26 Sep '20

26 Sep '20

hey folks, I have managed to fat finger a config apply command and accidentally deleted the CRD for one of my pools. The operator went ahead and tried to purge it, but fortunately since it's used by CephFS it was unable to. Redeploying the exact same CRD does not make the operator stop trying to delete it though. Any hints on how to make the operator forget about the deletion request and leave it be? -- Cheers, Peter Sarossy Technical Program Manager Data Center Data Security - Google LLC.

4 6

Re: Issues with the ceph-bluestore-tool during cluster upgrade from Mimic to Nautilus
by Igor Fedotov 25 Sep '20

25 Sep '20

Hi Saber, I don't think this is related. New assertion happens along the write path while the original one occurred on allocator shutdown. Unfortunately there are not much information to troubleshoot this... Are you able to reproduce the case? Thanks, Igor On 9/25/2020 4:21 AM, Saber(a)PlanetHoster.info wrote: > Hi Igor, > > We had an osd crash a week after running Nautilus. I have attached the > logs, is it related to the same bug? > > > > > Thanks, > Saber > CTO @PlanetHoster > >> On Sep 14, 2020, at 10:22 AM, Igor Fedotov <ifedotov(a)suse.de >> <mailto:ifedotov@suse.de>> wrote: >> >> Thanks! >> >> Now got the root cause. The fix is on its way... >> >> Meanwhile you might want to try to workaround the issue via setting >> "bluestore_hybrid_alloc_mem_cap" to 0 or using different allocator, >> e.g. avl for bluestore_allocator (and optionally for bluefs_allocator >> too). >> >> >> Hope this helps, >> >> Igor. >> >> >> >> On 9/14/2020 5:02 PM, Jean-Philippe Méthot wrote: >>> Alright, here’s the full log file. >>> >>> >>> >>> >>> >>> Jean-Philippe Méthot >>> Senior Openstack system administrator >>> Administrateur système Openstack sénior >>> PlanetHoster inc. >>> 4414-4416 Louis B Mayer >>> Laval, QC, H7P 0G1, Canada >>> TEL : +1.514.802.1644 - Poste : 2644 >>> FAX : +1.514.612.0678 >>> CA/US : 1.855.774.4678 >>> FR : 01 76 60 41 43 >>> UK : 0808 189 0423 >>> >>> >>> >>> >>> >>> >>>> Le 14 sept. 2020 à 06:49, Igor Fedotov <ifedotov(a)suse.de >>>> <mailto:ifedotov@suse.de>> a écrit : >>>> >>>> Well, I can see duplicate admin socket command >>>> registration/de-registration (and the second de-registration >>>> asserts) but don't understand how this could happen. >>>> >>>> Would you share the full log, please? >>>> >>>> >>>> Thanks, >>>> >>>> Igor >>>> >>>> On 9/11/2020 7:26 PM, Jean-Philippe Méthot wrote: >>>>> Here’s the out file, as requested. >>>>> >>>>> >>>>> >>>>> >>>>> Jean-Philippe Méthot >>>>> Senior Openstack system administrator >>>>> Administrateur système Openstack sénior >>>>> PlanetHoster inc. >>>>> 4414-4416 Louis B Mayer >>>>> Laval, QC, H7P 0G1, Canada >>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>> FAX : +1.514.612.0678 >>>>> CA/US : 1.855.774.4678 >>>>> FR : 01 76 60 41 43 >>>>> UK : 0808 189 0423 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Le 11 sept. 2020 à 10:38, Igor Fedotov <ifedotov(a)suse.de >>>>>> <mailto:ifedotov@suse.de>> a écrit : >>>>>> >>>>>> Could you please run: >>>>>> >>>>>> CEPH_ARGS="--log-file log --debug-asok 5" ceph-bluestore-tool >>>>>> repair --path <...> ; cat log | grep asok > out >>>>>> >>>>>> and share 'out' file. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Igor >>>>>> >>>>>> On 9/11/2020 5:15 PM, Jean-Philippe Méthot wrote: >>>>>>> Hi, >>>>>>> >>>>>>> We’re upgrading our cluster OSD node per OSD node to Nautilus >>>>>>> from Mimic. From some release notes, it was recommended to run >>>>>>> the following command to fix stats after an upgrade : >>>>>>> >>>>>>> ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-0 >>>>>>> >>>>>>> However, running that command gives us the following error message: >>>>>>> >>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>> <http://allocator.cc/>: In >>>>>>>> function 'virtual Allocator::SocketHook::~SocketHook()' thread >>>>>>>> 7f1a6467eec0 time 2020-09-10 14:40:25.872353 >>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>> <http://allocator.cc/>: 53 >>>>>>>> : FAILED ceph_assert(r == 0) >>>>>>>> ceph version 14.2.11 >>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>> char const*)+0x14a) [0x7f1a5a823025] >>>>>>>> 2: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>> 3: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>> 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>> 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>> 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] >>>>>>>> 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>> [0x55b3352749a1] >>>>>>>> 8: (main()+0x10b3) [0x55b335187493] >>>>>>>> 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>> 10: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>> 2020-09-10 14:40:25.873 7f1a6467eec0 -1 >>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>> <http://allocator.cc/>: In function 'virtual >>>>>>>> Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 time >>>>>>>> 2020-09-10 14:40:25.872353 >>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>> <http://allocator.cc/>: 53: FAILED ceph_assert(r == 0) >>>>>>>> >>>>>>>> ceph version 14.2.11 >>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>> char const*)+0x14a) [0x7f1a5a823025] >>>>>>>> 2: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>> 3: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>> 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>> 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>> 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] >>>>>>>> 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>> [0x55b3352749a1] >>>>>>>> 8: (main()+0x10b3) [0x55b335187493] >>>>>>>> 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>> 10: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>> *** Caught signal (Aborted) ** >>>>>>>> in thread 7f1a6467eec0 thread_name:ceph-bluestore- >>>>>>>> ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) >>>>>>>> nautilus (stable) >>>>>>>> 1: (()+0xf630) [0x7f1a58cf0630] >>>>>>>> 2: (gsignal()+0x37) [0x7f1a574be387] >>>>>>>> 3: (abort()+0x148) [0x7f1a574bfa78] >>>>>>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>> char const*)+0x199) [0x7f1a5a823074] >>>>>>>> 5: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>> 6: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>> 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>> 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>> 9: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] >>>>>>>> 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>> [0x55b3352749a1] >>>>>>>> 11: (main()+0x10b3) [0x55b335187493] >>>>>>>> 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>> 13: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>> 2020-09-10 14:40:25.874 7f1a6467eec0 -1 *** Caught signal >>>>>>>> (Aborted) ** >>>>>>>> in thread 7f1a6467eec0 thread_name:ceph-bluestore- >>>>>>>> >>>>>>>> ceph version 14.2.11 >>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>> 1: (()+0xf630) [0x7f1a58cf0630] >>>>>>>> 2: (gsignal()+0x37) [0x7f1a574be387] >>>>>>>> 3: (abort()+0x148) [0x7f1a574bfa78] >>>>>>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>> char const*)+0x199) [0x7f1a5a823074] >>>>>>>> 5: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>> 6: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>> 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>> 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>> 9: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] >>>>>>>> 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>> [0x55b3352749a1] >>>>>>>> 11: (main()+0x10b3) [0x55b335187493] >>>>>>>> 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>> 13: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>> NOTE: a copy of the executable, or `objdump -rdS <executable>` >>>>>>>> is needed to interpret this. >>>>>>> >>>>>>> What could be the source of this error? I haven’t found much of >>>>>>> anything about it online. >>>>>>> >>>>>>> >>>>>>> Jean-Philippe Méthot >>>>>>> Senior Openstack system administrator >>>>>>> Administrateur système Openstack sénior >>>>>>> PlanetHoster inc. >>>>>>> 4414-4416 Louis B Mayer >>>>>>> Laval, QC, H7P 0G1, Canada >>>>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>>>> FAX : +1.514.612.0678 >>>>>>> CA/US : 1.855.774.4678 >>>>>>> FR : 01 76 60 41 43 >>>>>>> UK : 0808 189 0423 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> <mailto:ceph-users@ceph.io> >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>> <mailto:ceph-users-leave@ceph.io> >>>>> >>> >

1 0

Epson printer making odd sound? Arrive at technical support by reaching Epson Customer Service.
by mary smith 25 Sep '20

25 Sep '20

Now and then an issue can emerge when you may hear peculiar sounds originating from the printer because of some tech glitch. In the event that that occurs, at that point you can get the assistance by heading off to the tech help locales or you can call the Epson Customer Service to get the issue settled. https://www.epsonprintersupportpro.net/

1 0

Incapable to see companions on Facebook? Get to Facebook Customer Service Toll Free number.
by mary smith 25 Sep '20

25 Sep '20

The companions on Facebook tells you the number of individuals you're associated with, on the web-based media website. Subsequently, on the off chance that you can't see your companions, at that point this may be because of the helpless web association. On the off chance that you accept that there is some other explanation, at that point you can get to Facebook Customer Service Toll Free Number to know the clear arrangements. https://www.customercare-email.com/facebook-customer-service.html

1 0

NVMe's
by Brent Kennedy 25 Sep '20

25 Sep '20

We currently run a SSD cluster and HDD clusters and are looking at possibly creating a cluster for NVMe storage. For spinners and SSDs, it seemed the max recommended per osd host server was 16 OSDs ( I know it depends on the CPUs and RAM, like 1 cpu core and 2GB memory ). Questions: 1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers were bought with 48 cores and 100+ GB of RAM, would this make sense? 2. Should we just raid 5 by groups of NVMe drives instead ( and buy less CPU/RAM )? There is a reluctance to waste even a single drive on raid because redundancy is basically cephs job. 3. The plan was to build this with octopus ( hopefully there are no issues we should know about ). Though I just saw one posted today, but this is a few months off. 4. Any feedback on max OSDs? 5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking this would need at least 40Gb links to every node ( the hope is to use these to speed up image processing at the application layer locally in the DC ). I haven't spoken to the Dell engineers yet but my concern with NVMe is that the raid controller would end up being the bottleneck ( next in line after network connectivity ). Regards, -Brent Existing Clusters: Test: Nautilus 14.2.11 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi gateways ( all virtual on nvme ) US Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 3 gateways, 2 iscsi gateways

12 33

Unable to restart OSD assigned to LVM partition on Ceph 15.1.2?
by Matt Larson 24 Sep '20

24 Sep '20

Hi, I recently restarted a storage node for our Ceph cluster and had an issue bringing one of the OSDs back online. This storage node has multiple HDs each as a devoted OSD for a data pool, and a single nVME drive with an LVM partition assigned as an OSD in a metadata pool. After rebooting the host, the OSD using an LVM partition did not restart. When trying to manually start the OSD using systemctl, I can follow the launch of a podman container and see an error message prior to the container shutting down again: Sep 23 14:02:06 X bash[30318]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/boot/cephfs_meta --path /var/lib/ceph/osd/ceph-165 --no-mon-config Sep 23 14:02:06 X bash[30318]: stderr: failed to read label for /dev/boot/cephfs_meta: (2) No such file or directory Sep 23 14:02:06 X bash[30318]: --> RuntimeError: command returned non-zero exit status: 1 1. I can see the existence of the /dev/boot/cephfs_meta symlink to a device ../dm-3 2. `lsblk` shows the lvm partition 'boot-cephfs_meta' under nvme0n1p3 3. `sudo lvscan --all` shows the it as activated: ` ACTIVE '/dev/boot/cephfs_meta' [3.42 TiB] inherit` This is on a CentOS 8 system, with ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable) Related issues I have found include: 1. https://github.com/rook/rook/issues/2591 2. https://github.com/rook/rook/issues/3289 There were indicated solutions for these involving installing the LVM2 package, which I completed with `sudo dnf install lvm2`, then tried a restart of the system and restart of the container. This was not able to resolve the problem for LVM-partition based OSD. This LVM-based OSD was initially created with a `ceph-volume` command: `ceph-volume lvm create --bluestore --data /dev/sd<x> --block.db /dev/nvme0n1<partition-nr>` Is there a workaround for this problem where the container process is unable to read the label of the LVM partition and fails to start the OSD? Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

1 0

Feature highlight: CephFS network restriction
by Stefan Kooman 24 Sep '20

24 Sep '20

Hi, While reading documentation on CephFS I came accross the "network restriction" feature (supported since Nautilus) [1]. I have never seen a reference to / blog post on this feature and didn't even know it was possible. This might be true for more people, hence this mail to the list :-). So, for improved security you can restrict on ip (ranges) in Ceph (besides / instead) of regular network filtering. I have tested this on Nautilus with IPv6 and it works as documented. Kudos for CephFS team. FYI, Gr. Stefan [1] https://docs.ceph.com/en/latest/cephfs/client-auth/#network-restriction

1 0

RBD quota per namespace
by Eugen Block 24 Sep '20

24 Sep '20

Hi *, I'm curious if this idea [1] of quotas on namespace level for rbd will be implemented. I couldn't find any existing commands in my lab Octopus cluster so I guess it's still just an idea, right? If there's any information I missed please point me to it. Thanks! Eugen [1] https://trello.com/c/Y1n75zpy/396-rbd-per-pool-per-namespace-provision-quot…

3 2

Re: Vitastor, a fast Ceph-like block storage for VMs
by vitalif＠yourcmc.ru 24 Sep '20

24 Sep '20

Hi Roman, Yes, you're right - OSDs list all objects during peering and take the latest full version of each object. Full version is a version that has at least min_size parts for XOR/EC, or any version for replicated setups which is OK because writes are atomic. If there is a newer "committed" ("stabilized") version of the object, but it doesn't have sufficient number of parts, the object is marked as incomplete. By the way, did you receive my reply that I sent directly to your email? - it could've ended up in spam because yourcmc.ru is my own mail server. > Hi Vitaliy, > > Awesome what you did. I like the minimalistic approach focused on > performance very much. A long way you've passed implementing the > whole stack from client to storage. > > Could you please describe peering process and object recovery in more > details? You write in the README that there is no PGLog in your > storage. After a brief glance into osd_ops.h and blockstore_impl.cpp I > see there is a 'version' field. I can assume that every object is > marked with a version and recovery happens taking the whole up-to-date > object, right? > > -- > Roman > > On 2020-09-23 00:44, vitalif(a)yourcmc.ru wrote: > >> Hi! >> After almost a year of development in my spare time I present my own >> software-defined block storage system: Vitastor - https://vitastor.io >> I designed it similar to Ceph in many ways, it also has Pools, PGs, >> OSDs, different coding schemes, rebalancing and so on. However it's >> much simpler and much faster. In a test cluster with SATA SSDs it >> achieved Q1T1 latency of 0.14ms which is especially great compared to >> Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops >> saturation" parallel load benchmark it reached 895k read / 162k write >> iops, compared to Ceph's 480k / 100k on the same hardware, but the >> most interesting part was CPU usage: Ceph OSDs were using 40 CPU cores >> out of 64 on each node and Vitastor was only using 4. >> Of course it's an early pre-release which means that, for example, it >> lacks snapshot support and other useful features. However the base is >> finished - it works and runs QEMU VMs. I like the design and I plan to >> develop it further. >> There are more details in the README file which currently opens from >> the domain https://vitastor.io >> Sorry if it was a bit off-topic, I just thought it could be >> interesting for you :)

1 0

Remove separate WAL device from OSD
by Michael Fladischer 24 Sep '20

24 Sep '20

Hi, Is it possible to remove an existing WAL device from an OSD? I saw that ceph-bluestore-tool has a command bluefs-bdev-migrate, but it's not clear to me if this can only move a WAL device or if it can be used to remove it ... Regards, Michael

4 7