- ceph-users - lists.ceph.io

Global Recovery Event
by Jeremy Hansen 08 Jun '21

08 Jun '21

I’m seeing this in my health status: progress: Global Recovery Event (13h) [............................] (remaining: 5w) I’m not sure how this was initiated but this is a cluster with almost zero objects. Is there a way to halt this process? Why would it estimate 5 weeks to recover a cluster with almost zero data? [ceph: root@cn01 /]# ceph -s -w cluster: id: bfa2ad58-c049-11eb-9098-3c8cf8ed728d health: HEALTH_OK services: mon: 2 daemons, quorum cn02,cn05 (age 13h) mgr: cn01.ceph.la1.clx.corp.xnkoft(active, since 13h), standbys: cn02.arszct mds: 1/1 daemons up, 1 standby osd: 27 osds: 27 up (since 13h), 27 in (since 16h) data: volumes: 1/1 healthy pools: 3 pools, 65 pgs objects: 22.09k objects, 86 GiB usage: 261 GiB used, 98 TiB / 98 TiB avail pgs: 65 active+clean progress: Global Recovery Event (13h) [............................] (remaining: 5w) Thanks -jeremy

2 2

Failed OSD has 29 Slow MDS Ops.
by Dave Hall 07 Jun '21

07 Jun '21

Hello, Nautilus 14.2.16 I had an OSD go bad about 10 days ago. Apparently as it was going down some MDS ops got hung up waiting for it to come back. I was out of town for a couple days and found the OSD 'Down and Out' when I checked in. (Also, oddly, the cluster did not appear to initiate recovery right away - it took until I rebooted the OSD node.) As of right now, the damaged OSD is 'safe-to-destroy' but the slow ops are still hanging around. Earlier today I quiesced the clients that were accessing the CephFS, then unmounted and re-mounted it. However, this did not clear the lingering ops. When I had the node offline I verified that the HDD and NVMe associated with the OSD seem to actually be healthy, so I plan to zap and re-deploy using the same hardware. I would also like to upgrade to 14.2.20 (latest Ceph for debian 10), but I'm hesitant to do any of this until I get rid of these 29 slow ops. Can anybody suggest a path forward? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

1 0

Connect ceph to proxmox
by Szabo, Istvan (Agoda) 07 Jun '21

07 Jun '21

Hi, Is there a way to connect from my nautilus ceph setup the pool that I created in ceph to proxmox? Or need a totally different ceph install? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

4 5

Re: Why you might want packages not containers for Ceph deployments
by Marc 07 Jun '21

07 Jun '21

Do you use rbd images in containers that are residing on osd nodes? Does this give any problems? I used to have kernel mounted cephfs on a osd node, after a specific luminous release this was giving me problems. > -----Original Message----- > From: Eneko Lacunza <elacunza(a)binovo.es> > Sent: Friday, 4 June 2021 15:49 > To: ceph-users(a)ceph.io > Subject: *****SPAM***** [ceph-users] Re: Why you might want packages > not containers for Ceph deployments > > Hi, > > We operate a few Ceph hyperconverged clusters with Proxmox, that > provides a custom ceph package repository. They do a great work; and > deployment is a brezee. > > So, even as currently we would rely on Proxmox packages/distribution and > not upstream, we have a number of other projects deployed with > containers and we even distribute some of our own development in deb and > container packages, so I will comment on our view: > > El 2/6/21 a las 23:26, Oliver Freyermuth escribió: > [...] > > > > If I operate services in containers built by developers, of course > > this ensures the setup works, and dependencies are well tested, and > > even upgrades work well — but it also means that, > > at the end of the day, if I run 50 services in 50 different containers > > from 50 different upstreams, I'll have up to 50 different versions of > > OpenSSL floating around my production servers. > > If a security issue is found in any of the packages used in all the > > container images, I now need to trust the security teams of all the 50 > > developer groups building these containers > > (and most FOSS projects won't have the ressources, understandably...), > > instead of the one security team of the disto I use. And then, I also > > have to re-pull all these containers, after finding out that a > > security fix has become available. > > Or I need to build all these containers myself, and effectively take > > over the complete job, and have my own security team. > > > > This may scale somewhat well, if you have a team of 50 people, and > > every person takes care of one service. Containers are often your > > friend in this case[1], > > since it allows to isolate the different responsibilities along with > > the service. > > > > But this is rarely the case outside of industry, and especially not in > > academics. > > So the approach we chose for us is to have one common OS everywhere, > > and automate all of our deployment and configuration management with > > Puppet. > > Of course, that puts is in one of the many corners out there, but it > > scales extremely well to all services we operate, > > and I can still trust the distro maintainers to keep the base OS safe > > on all our servers, automate reboots etc. > > > > For Ceph, we've actually seen questions about security issues already > > on the list[0] (never answered AFAICT). > > These are the two main issues I find with containers really: > > - Keeping hosts uptodate is more complex (apt-get update+apt-get > dist-upgrade and also some kind of docker pull+docker > restart/docker-compose up ...). Much of the time the second part is not > standard (just deployed a Harbor service, upgrade is quite simple but I > have to know how to do it as it's speciffic, maintenance would be much > easier if it was packaged in Debian). I won't say it's more difficult, > but it will be more diverse and complex. > > - Container image quality and security support quality, that will vary > from upstream to upstream. You have to research each of them to know > were they stand. A distro (specially a good one like Debian, Ubuntu, > RHEL or SUSE) has known, quality security support for the repositories. > They will even fix issues not fixed by upstream (o backport them to > distro's version...). This is more an upstream vs distro issue, really. > > About debugging issues reported with Ceph containers, I think those are > things waiting for a fix: why are logs writen in container image (or an > ephemeral volume, I don't know really how is that done right now) > instead of an external name volume o a local mapped dir in /var/log/ceph ? > > All that said, I think that it makes sense for an upstream project like > Ceph, to distribute container images, as it is the most generic way to > distribute (you can deploy on any system/distro supporting container > images) and eases development. But only distributing container images > could make more users depend on third party distribution (global or > specific distros), which would delay feeback/bugreport to upstream. > > Cheers and thanks for the great work! > > Eneko Lacunza > Zuzendari teknikoa | Director técnico > Binovo IT Human Project > > Tel. +34 943 569 206 | https://www.binovo.es > Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun > > https://www.youtube.com/user/CANALBINOVO > https://www.linkedin.com/company/37269706/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Turning on "compression_algorithm" old pool with 500TB usage
by mhnx 07 Jun '21

07 Jun '21

Hello. I have a erasure pool and I didn't turn on compression at the beginning. Now I'm writing new type of very small data and overhead is becoming an issue. I'm thinking to turn on compression on the pool but in most filesystems it will effect only the new data. What is the behavior in ceph?

3 2

Cephfs root/boot?
by Harry G. Coin 07 Jun '21

07 Jun '21

Has anyone added the 'conf.d' modules (and on the centos/rhel/fedora world done the selinux work) so that initramfs/dracut can 'direct kernel boot' cephfs as a guest image root file system? It took some work for the nfs folks to manage being the root filesystem. Harry

1 0

Running ISCSI with Ubuntu 18.04 OS
by Michel Niyoyita 07 Jun '21

07 Jun '21

Dear all , IS it possible to configure and run ISCSI when you are deploying ceph using ansible running on ubuntu 18.04 OS? please help me to know and if possible provide helpful links on that. Best Regard Michel

1 0

cephfs objets without 'parent' xattr?
by Burkhard Linke 07 Jun '21

07 Jun '21

Hi, during an OS upgrade from Ubuntu 18.04 to 20.04 we seem to have triggered a bcache bug on three OSD hosts. These hosts are used with a 6+2 EC pool used with CephFS, so a number of PGs are affected by the bug. We were able to restart two of the three hosts (and will run some extra scrubs on all PGs), but at least 7 PGs have unfound objects now. I'm currently trying to find out which files are affected to restore them from backup or inform the users about data corruption for files in dedicated scratch directories. This works quite well for most of the files. The 'parent' xattr attached to the file's first chunks contains the complete path of the file within the filesystem, so locating the files should be easy with a little help from ceph-dencoder. But there are some files that do not have the 'parent' xattr: for pg in $(ceph health detail | grep active\+recovery_unfound | cut -d' ' -f 6); do echo $pg; for obj in $(ceph pg $pg list_unfound | jq -r '.objects | .[] | .oid.oid'| cut -c1-11); do rados -p bcf_fs_data_rep getxattr $obj.00000000 parent > $obj.parent; done; done gives 77.1 error getting xattr bcf_fs_data_rep/1002a143927.00000000/parent: (2) No such file or directory ..... 'bcf_fs_data_rep' is the first data pool of the filesystem which should contain the xattr data. But for a number of objects (58 of 928) the above command is not able to retrieve the information. Questions: 1. If the 'parent' xattr is not available in the first data pool (and neither in the meta data pool), what might be the state of these objects? Can they be deleted by using 'mark_unfound delete'? 2. the list_unfound command only prints 256 objects; how can this limit be lifted, since some pools have more unfound objects? Regards, Burkhard

1 0

Debian buster nautilus 14.2.21 missing?
by Szabo, Istvan (Agoda) 07 Jun '21

07 Jun '21

Hi, Am I doing something wrong or the 21 update is missing for buster? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
by Jeremy Hansen 07 Jun '21

07 Jun '21

What’s the proper way to track down where this error is coming from? Thanks. 6/7/21 12:40:00 AM [WRN] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 6/7/21 12:40:00 AM [WRN] Health detail: HEALTH_WARN 1 failed cephadm daemon(s)

2 4