- ceph-users - lists.ceph.io

Ceph Community Infrastructure Outage
by Mike Perez 18 Jan '23

18 Jan '23

Hi everyone, From November into January, we experienced a series of outages with the Ceph Community Infrastructure and its services: - Mailing lists - https://lists.ceph.io - Sepia (testing infrastructure) - https://wiki.sepia.ceph.com - https://pulpito.ceph.com - https://chacra.ceph.com - https://shaman.ceph.com - VPN to access testing services - Etherpad - https://pad.ceph.com - Images: - https://quay.ceph.io - Git mirror - https://git.ceph.com - https://ceph.io - Telemetry <https://telemetry-public.ceph.com/> These services are now mostly restored, but we did experience some data loss, notably in our mailing lists. We have restored them from backups, but subscription changes after July 2021 need to be repeated. If you subscribed or unsubscribed since then, please check your settings with the appropriate list at https://lists.ceph.io. If your posts to our mailing lists are now needing approval, that is also an indication that you need to re-subscribe to the appropriate lists. Keep an eye out for emails with subject lines such as “Your message to ceph-users(a)ceph.io awaits moderator approval”. When the community infrastructure was first created in late 2014, the VM cluster management software selected by the team came with the benefit of being widely entrenched and familiar to the lab administrators but didn't support Ceph as a storage backend at the time. As services grew, we relied more and more on its legacy storage solution, which was never migrated to Ceph. Over the last few months, this legacy storage solution had several instances of silent data corruption, rendering the VMs unbootable, taking down various services, and requiring restoration from backups in many cases. We are moving these services to a more reliable, mostly container-based, infrastructure backed by Ceph, and planning for longer-term improvements to monitoring, backups, deployment, and other pieces of the project infrastructure. This event highlights the need to better support the infrastructure. A handful of contributors have stepped up to restore these services, but we need an invested team focused. If you or your company is looking for a great way to contribute to the Ceph community, this could be your opportunity. Please contact council(a)ceph.io if you can provide time to contribute to the Ceph Community Infrastructure and would like to join the team. You can also join the upstream #sepia slack channel to participate in these discussions using this link: https://join.slack.com/t/ceph-storage/shared_invite/zt-1n1eh6po5-PF9sokUSoo… Unfortunately, these events have slowed down our upstream development and releases. We are currently working on publishing the next Pacific point release. The development freeze and release deadline for the Reef release will likely be pushed out, and more discussions to follow in the Ceph Leadership Team meetings. - The Ceph Leadership Team

2 1

Stable erasure coding CRUSH rule for multiple hosts?
by aschmitz 18 Jan '23

18 Jan '23

Hi folks, I have a small cluster of three Ceph hosts running on Pacific. I'm trying to balance resilience and disk usage, so I've set up a k=4 m=2 pool for some bulk storage on HDD devices. With the correct placement of PGs this should allow me to take any one host offline for maintenance. I've written this CRUSH rule for that purpose: rule erasure_k4_m2_hdd_rule { id 3 type erasure step take default class hdd step choose indep 3 type host step chooseleaf indep 2 type osd step emit } This should pick three hosts, and then two OSDs from each, which at least ensures that no host has more than two OSDs. This appears to work correctly, but I'm running into an odd situation when adding additional OSDs to the cluster: sometimes the hosts flip order in a PG's set, resulting in unnecessary remapping work. For example, I have one PG that changed from OSDs [0,13,7,9,3,5] to [0,13,3,5,7,9]. (Note that the middle two and last two sets of OSDs have swapped with one another.) From a quick perusal of other PGs that are being moved, the two OSDs within a host never appear to be rearranged, but the set of hosts that are chosen may be shuffled. Is there something I'm missing that would make this rule more stable in the face of OSD addition? (I'm wondering if the host choosing component should be "firstn" rather than "indep", even though the discussion at https://docs.ceph.com/en/latest/rados/operations/crush-map-edits/#crushmapr… implies indep is preferable in EC pools.) I don't have current plans to expand beyond a three-host cluster, but if there's an alternative way to express "not more than two OSDs per host", that could be helpful as well. Any insights or suggestions would be appreciated. Thanks, aschmitz

2 1

bidirectional rbd-mirroring
by Aielli, Elia 18 Jan '23

18 Jan '23

Hi all, I've a working couple of cluster configured with rbd mirror, Master cluster is production, Backup cluster is DR. Right now all is working good with Master configured in "tx-only" and Backup in "rx-tx". I'd like to modify Master direction to rx-tx so I'm already prepared for a failover after a disaster has happened, but while I'm doin so, I face this error and i'm stuck: ceph version 15.2.17 (694d03a6f6c6e9f814446223549caf9a9f60dba0) octopus (stable) Ceph user able to operate on Master is rbd-mirror.master, while on Backup is rbd-mirror.backup On Master cluster I've my ceph.conf and backup.conf, and on Backup cluster I've ceph.conf and master.conf Keyrings has been copied correctly. I've change direction without any problem, but when i try to configure the peer with this command, i receive following error: root@master# rbd mirror pool peer add <my_pool> client.rbd-mirror.backup@backup rbd: multiple RX peers are not currently supported And when I check my pool info, i have the "Client:" section empty (while the one on my DR is populated with client.rbd-mirror.master" Can someone lend me a hand? Is this something I can't do or simply I'm using the wrong commands? Thanks in advance! Elia

2 2

Ceph User + Dev Monthly January Meetup
by Neha Ojha 17 Jan '23

17 Jan '23

Hi everyone, This month's Ceph User + Dev Monthly meetup is on January 19, 15:00-16:00 UTC. There are some topics in the agenda regarding RGW backports, please feel free to add other topics to https://pad.ceph.com/p/ceph-user-dev-monthly-minutes. Hope to see you there! Thanks, Neha

1 0

Dashboard access to CephFS snapshots
by Robert Sander 17 Jan '23

17 Jan '23

Hi, The dashboard has a simple CephFS browser where we can set quota and snapshots for the directories. When a directory has the "other" permission bits unset, i.e. only access for user and group, the dashboard displays an error: Failed to execute CephFS opendir failed at /path/to/dir/.snap: Permission denied [Errno 13] It can be reproduced in Ceph 17.2.5 by creating the directory and using "chmod o= /path/to/dir" to not allow "other". How does the dashboard access the contents of the CephFS? It looks like the MGR uses something like the nobody account. Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 1

Re: opnesuse rpm repos
by Eugen Block 17 Jan '23

17 Jan '23

Hi, the last RPMs I'm aware of for openSUSE are for Pacific, and we get them from software.opensuse.org. We used Pacific for the switch to cephadm and containers in our openSUSE based ceph cluster. I don't see any opensuse directory underneath download.ceph.com, were they actually present in the past? I've never used that repo. But the general development is directed to containers. Regards, Eugen Zitat von Mazzystr <mazzystr(a)gmail.com>: > Hello, > I ran the following on OpenSuse > > cephadm add-repo --release quincy > > > The command landed /etc/zypp/repos.d/ceph.repo and contents > > [Ceph] > name=Ceph $basearch > baseurl=https://download.ceph.com/rpm-quincy/opensuse/$basearch > enabled=1 > gpgcheck=1 > gpgkey=https://download.ceph.com/keys/release.gpg > > [Ceph-noarch] > name=Ceph noarch > baseurl=https://download.ceph.com/rpm-quincy/opensuse/noarch > enabled=1 > gpgcheck=1 > gpgkey=https://download.ceph.com/keys/release.gpg > > [Ceph-source] > name=Ceph SRPMS > baseurl=https://download.ceph.com/rpm-quincy/opensuse/SRPMS > enabled=1 > gpgcheck=1 > gpgkey=https://download.ceph.com/keys/release.gpg > > Apparently these repos don't exist. > > Do we plan on building for OpenSuse or isd the plan to migrate to > containers? > > Thanks, > /Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Mysterious HDD-Space Eating Issue
by matthew＠peregrineit.net 17 Jan '23

17 Jan '23

Hi Guys, I've got a funny one I'm hoping someone can point me in the right direction with: We've got three identical(?) Ceph nodes running 4 OSDs, Mon, Man, and iSCSI G/W each (we're only a small shop) on Rocky Linux 8 / Ceph Quincy. Everything is running fine, no bottle-necks (as far as we can see) and the Cluster is holding up very well. However, one of the boxes is constantly running out of space on the /var mount. Its 16 GiB in size, and it only takes a day or three to fill up, thus taking it's monitor service out of quorum. The thing is, I can't find *what's* taking up all the space. At first we thought it was an overly large log file, but I've done searches to find the largest files, etc, and nothing is showing up (that I can find) - ie the log files on this box are comparable with the log files on the other two boxes and the other two boxes are sitting at around 10% full (via a df-H), while the problem box is at around 85% and growing (at time of posting). Another interesting point is that the problem box, unrelated to this issue, was rebooted recently and when it came back on-line the space-issue was gone ie the /var mount was back down to around the 10% mark. This suggests to me its some sort of "temporary" journal/log/dump/whatever/? that was "reset" (cleaned-up?) via the reboot. I've had a look at the logs but I'm not sure what I should be looking for - so I don't even know if I'm looking in the *correct* logs... Anyone got any ideas? I mean, rebooting the server every couple of days is not really a practical solution, and neither is turning off the monitor service on the box, and increasing the size of the /var mount just seems like it'll postpone the issue. Any help would be greatly appreciated. Cheers Dulux-Oz

4 4

._handle_peer_banner peer [v2:***,v1:***] is using msgr V1 protocol
by Frank Schilder 17 Jan '23

17 Jan '23

Hi all, on an octopus latest cluster I see a lot of these log messages: Jan 13 20:00:25 ceph-21 journal: 2023-01-13T20:00:25.366+0100 7f47702b8700 -1 --2- [v2:192.168.16.96:6826/5724,v1:192.168.16.96:6827/5724] >> [v2:192.168.16.93:6928/3503064,v1:192.168.16.93:6929/3503064] conn(0x55c867624400 0x55c7e9dfa800 unknown :-1 s=BANNER_CONNECTING pgs=22826 cs=73364 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.168.16.93:6928/3503064,v1:192.168.16.93:6929/3503064] is using msgr V1 protocol These addresses are on the replication network and both hosts are OSD hosts. What is the reason for these messages and how can I fix it? Thanks a lot! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 3

PG_BACKFILL_FULL
by Iztok Gregori 17 Jan '23

17 Jan '23

Hi to all! We are in a situation where we have 3 PG in "active+remapped+backfill_toofull". It happened when we executed a "gentle-reweight" to zero of one OSD (osd.77) to swap it with a new one (the current one registered some read errors and it's to be replaced just-in-case). > # ceph health detail: > [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull > pg 10.46c is active+remapped+backfill_toofull, acting [77,607,96] > pg 10.8ad is active+remapped+backfill_toofull, acting [577,152,77] > pg 10.b15 is active+remapped+backfill_toofull, acting [483,348,77] Our cluster is a little unbalanced and we have 7 OSD nearfull (I think it's because we have 4 nodes with 6 TB disks and the other 19 have 10 TB, but should be unrelated, why is the cluster unbalanced I mean, to the backfill_toofull ) not by too much (less then 88%). I'm not too much worried about it, we will add new storage this month (if the servers will arrive) and we will get rid of the old 6 TB servers. If I dump the PGs I see, if I'm not mistaken, that the osd.77 will be "replaced" by the osd.60, which is one of the nearfull ones (the top one with 87.53% used). > # ceph pg dump: > > 10.b15 37236 0 0 37236 0 155249620992 0 0 5265 5265 active+remapped+backfill_toofull 2023-01-16T14:45:46.155801+0100 305742'144106 305742:901513 [483,348,60] 483 [483,348,77] 483 305211'144056 2023-01-11T10:20:56.600135+0100 305211'144056 2023-01-11T10:20:56.600135+0100 0 > 10.8ad 37518 0 0 37518 0 156345024512 0 0 5517 5517 active+remapped+backfill_toofull 2023-01-16T14:45:38.510038+0100 305213'142117 305742:937228 [577,60,152] 577 [577,152,77] 577 303828'142043 2023-01-06T17:52:02.523104+0100 303334'141645 2022-12-20T17:39:22.668083+0100 0 > 10.46c 36710 0 0 36710 0 153023443456 0 0 8172 8172 active+remapped+backfill_toofull 2023-01-16T14:45:29.284223+0100 305298'141437 305741:877331 [60,607,96] 60 [77,607,96] 77 304802'141358 2023-01-08T21:39:23.469198+0100 304363'141349 2023-01-01T18:13:45.645494+0100 0 > # ceph osd df: > 60 hdd 5.45999 1.00000 5.5 TiB 4.8 TiB 697 GiB 128 MiB 0 B 697 GiB 87.53 1.29 37 up In this situation was the correct way to address the problem? reweight-by-utilization the osd.60 to free up space (the OSD is a 6 TB disk, and other OSD on the same host are nearfull)? There is other way to manually map a PG to a different OSD? Thank you for your attention Iztok Gregori

3 4

Filesystem is degraded, offline, mds daemon damaged
by bpurvis＠kfpl.ca 17 Jan '23

17 Jan '23

I am really hoping you can help. THANKS in advance. I have inherited a Docker swarm running CEPH but I know very little about it. Current I have an unhealthy ceph environment that will not mount my data drive. Its a cluster of 4 vm servers. docker01,docker02, docker03, docker-cloud CL has the /data that is on a separate drive, currently failing to mount. how can I recover this without loosing the data? on server docker-cloud, mount /data returns: mount error 113 = No route to host docker ps is healthy on all nodes. bc81d14dde92 ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 34 minutes ceph-mds d4fecec5e0e8 ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 34 minutes ceph-osd 482ba41803af ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 34 minutes ceph-mgr d6a5c44179c7 ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 32 minutes ceph-mon ceph -s: cluster: id: 7a5b2243-8e92-4e03-aee7-aa64cea666ec health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged noout,noscrub,nodeep-scrub flag(s) set clock skew detected on mon.docker02, mon.docker03, mon.docker-cloud mons docker-cloud,docker01,docker02,docker03 are low on available space services: mon: 4 daemons, quorum docker01,docker02,docker03,docker-cloud mgr: docker01(active), standbys: docker02, docker03, docker-cloud mds: cephfs-0/1/1 up , 4 up:standby, 1 damaged osd: 4 osds: 4 up, 4 in flags noout,noscrub,nodeep-scrub data: pools: 2 pools, 256 pgs objects: 194.2 k objects, 241 GiB usage: 499 GiB used, 1.5 TiB / 2.0 TiB avail pgs: 256 active+clean

2 1