- ceph-users - lists.ceph.io

ceph-exporter binds to IPv4 only
by Stefan Kooman 23 Nov '23

23 Nov '23

Hi, In a IPv6 only deployment the ceph-exporter daemons are not listening on IPv6 address(es). This can be fixed by editing the unit.run file of the ceph-exporter by changing "--addrs=0.0.0.0" to "--addrs=::". Is this configurable? So that cephadm deploys ceph-exporter with proper unit.run arguments? Gr. Stefan ... who really thinks the Ceph test lab should have an IPv6 only test environment to catch these things

1 1

Service Discovery issue in Reef 18.2.0 release ( upgrading )
by Brent Kennedy 22 Nov '23

22 Nov '23

Greetings group! We recently reloaded a cluster from scratch using cephadm and reef. The cluster came up, no issues. We then decided to upgrade two existing cephadm clusters that were on quincy. Those two clusters came up just fine but there is an issue with the Grafana graphs on both clusters ( which were working before the upgrade ). They are now blank. There is an error in the Prometheus alerts (PrometheusJobMissing) that is alerting and it states the following: The prometheus job that scrapes from Ceph is no longer defined, this will effectively mean you'll have no metrics or alerts for the cluster. Please review the job definitions in the prometheus.yml file of the prometheus instance. summary: The scrape job for Ceph is missing from Prometheus When I look at the Prometheus.yml file on the performance monitoring node, this is what is there( I replaced ip with x.x.x.x ): global: scrape_interval: 10s evaluation_interval: 10s rule_files: - /etc/prometheus/alerting/* alerting: alertmanagers: - scheme: http http_sd_configs: - url: http://x.x.x.x:8765/sd/prometheus/sd-config?service=alertmanager scrape_configs: - job_name: 'ceph' honor_labels: true http_sd_configs: - url: http://x.x.x.x:8765/sd/prometheus/sd-config?service=mgr-prometheus - job_name: 'node' http_sd_configs: - url: http://x.x.x.x:8765/sd/prometheus/sd-config?service=node-exporter - job_name: 'ceph-exporter' honor_labels: true http_sd_configs: - url: http://x.x.x.x:8765/sd/prometheus/sd-config?service=ceph-exporter When I open a run "netstat -ntlp" on the active mgr node, I see the 8765 port being used by docker. However, when I try to use the chrome browser to access the URLs listed in the Prometheus.yml file, the page times out. However, if I do this with the active manager on the cluster that was installed from scratch ( and not upgraded ), the URL for that cluster returns output( different for each URL ). So it appears to me that the service discovery function is not working for upgrades from quincy. Also, the ceph-exporter service was not installed on the cluster during the upgrade process. I manually added the service when I noticed that it was not there ( when comparing the from scratch cluster to the upgraded cluster ). Not sure if this will help or is even related, but I saw it in the cephadm log: 2023-11-15T04:22:30.789998+0000 mgr. CEPH-MON-01.mlmups (mgr.144601) 753 : cephadm 4 host CEPH-MON-02 `cephadm gather-facts` failed: Cannot decode JSON: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1425, in _run_cephadm_json return json.loads(''.join(out)) File "/lib64/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Is there any way to fix the service discovery? Thanks! -Brent Existing Clusters: Test: Reef 18.2.0 ( all virtual on nvme ) US Production(HDD): Reef 18.2.0 with 11 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.22 with 18 osd servers, 3 mons, 4 gateways, 2 iscsi gateways US Production(SSD): Reef 18.2.0 Cephadm with 6 osd servers, 5 mons, 4 gateways UK Production(SSD): Reef 18.2.0 cephadm with 7 osd servers, 5 mons, 4 gateways

2 2

CloudStack and Ceph Day 2024
by 42on - Michiel Manten 22 Nov '23

22 Nov '23

Hello Ceph users, Together with ShapeBlue and Adyen, 42on is organizing a CloudStack and Ceph Day; this time in Amsterdam, The Netherlands. We are planning this for February 8 | 2024. We want to create a technical event that shares updates on both technologies, as well as 'use cases', stories, challenges and perhaps some crazy ideas or configurations. Let’s share information and make Ceph even bigger and better! I am still looking for some speakers who would love to share something about their Ceph infrastructure or configuration. As small as it might seem to you, any ideas are welcome and if you are in doubt about the subject, please message or call me to discuss. I would love to hear about your ideas. To RSVP and more information can be found here: https://www.eventbrite.nl/e/cloudstack-and-ceph-day-netherlands-2024-ticket… We hope to see you in Amsterdam once more. Sincerely, Michiel Manten

1 0

No SSL Dashboard working after installing mgr crt|key with RSA/4096 secp384r1
by Ackermann, Christoph 22 Nov '23

22 Nov '23

Hello all, today i got a new certificate for our internal domain based on RSA/4096 secp384r1. After inserting CRT and Key i got both "...updated" messages. After checking the dashboard i got an empty page and this error: health: HEALTH_ERR Module 'dashboard' has failed: key type unsupported So we tried to go back to the original state by removing CRT anf KEY but without success. The new key seems to stuck into the config [root@cephxxxx ~]# ceph config-key get mgr/dashboard/crt -----BEGIN CERTIFICATE----- MIIFqTCCBJGgAwIBAgIMB5tjLSz264Ic8zeHMA0GCSqGSIb3DQEBCwUAMEwxCzAJ [...] ItzkEzq4SZ6V1Jhuf4bFlOMBVAKgAwZ90gXlguoiFFQu5+NIqNljZ8Jz7d0jhH43 e3zhm5sn21+eIqRbiQ== -----END CERTIFICATE----- [root@cephxxxx ~]# ceph config-key get mgr/dashboard/key *Error ENOENT: * We tried to generate a self signed Cert but no luck. It looks like manger stays in an intermediate state. The only way to get back the dashboard is to disable SSL and use plain http. Can somebody explain this behaviour? Maybe secp384r1 elliptic curves aren't supported? How can we clean up SSL configuration? Thanks, Christoph Ackermann Ps we checked some Information like https://tracker.ceph.com/issues/57924#change-227744 and others but no luck...

2 3

[RGW][STS] How to use Roles to limit access to only buckets of one user?
by Rudenko Aleksandr 21 Nov '23

21 Nov '23

Hi, I have setup with one default tenant and next user/bucket structure: user1 bucket1 bucket11 user2 bucket2 user3 bucket3 IAM and STS APIs are enabled, user1 has roles=* capabilities. When user1 permit user2 to assume role with next permission policy: {"Version":"2012-10-17", "Statement":[ {"Effect":"Allow", "Action":["s3:*"], "Resource":"arn:aws:s3:::*"}] } user2 can use temporary credentials (after AssumeRole action) which give him access to ALL buckets of ALL users in this tenant (bucket3 for example). But I expect that access should be limited by user1 own buckets. I understand that roles=* caps it’s some kind of global admin permissions, but this caps are so powerful and give access to all buckets through roles. How can I use Role to limit access to only all own buckets? Now I can specify one or few exact buckets or ALL (*) in tenant. In AWS, we can give access to all buckets (*) but it will be all buckets in our account, not all buckets in S3.

1 0

RadosGW public HA traffic - best practices?
by Boris Behrens 21 Nov '23

21 Nov '23

Hi, I am looking for some experience on how people make their RGW public. Currently we use the follow: 3 IP addresses that get distributed via keepalived between three HAproxy instances, which then balance to three RGWs. The caveat is, that keepalived is PITA to get working in distributing a set of IP addresses, and it doesn't scale very well (up and down). The upside is, that it is really stable and customer nearly never have an availability problem. And we have 3 IPs that make some sort of LB. It serves up to 24Gbit in peak times, when all those backup jobs are running at night. But today I thought, what will happen if I just ditch the keepalived and configure thos addresses static to the haproxy hosts? How bad will the impact to a customer if I reboot one haproxy? Is there an easier, more scalable way if I want to spread the load even further without having an ingress HW LB (what I don't have)? I have a lot of hosts that would be able to host some POD with a haproxy and a RGW as container together, or even host the RGW alone in a container. It would just need to bridge two networks. But I currently do not have a way to use BGP to have one IP address split between a set of RGW instances. So long story short: What are your easy setups to serve public RGW traffic with some sort of HA and LB (without using a big HW LB that is capable of 100GBit traffic)? And have you experienced problems when you do not shift around IP addresses. Cheers Boris

4 4

really need help how to save old client out of hang?
by zxcs 21 Nov '23

21 Nov '23

Hi, Experts, we have an cephfs cluster 16.2.* run with multi active mds, and we have some old machine run with ubuntu 16.04 , so we mount these client using ceph-fuse. After a full mds process restart, all of these old ubuntu 16.04 clients cannot connect to ceph , `ls -lrth` or `df -hT` hang on the client node. and we can see mds log said `evicting unresponsive client *** after waiting ** seconds during restart` also checking the client node, no ceph-fuse process, (ps -ef | grep ceph or ps -ef | grep fuse). but there is a remount process (ps -ef | grep mount) in the client node. root 176516 1 0 01:38 ? 00:00:00 mount -i -o remount /data cannot kill this process using `sudo kill -9 176516`. we really need experts help us how to save this client out of hang. we cannot restart this client node due to there are some other critical service. Thanks a ton. zx

2 1

Previously synced bucket resharded after sync removed
by Szabo, Istvan (Agoda) 21 Nov '23

21 Nov '23

Hi, I had a multisite bucket which I've removed from sync completely and resharded on the master zone the bucket which was successful. On the 2nd site (which was expected) can't list anything inside that bucket anymore which is okay, the issue is how I can delete the data somehow? It was 50TB data there which I'd like to cleanup but at the moment everything shows 0, if I check the user space usage or the bucket space usage, all 0, however the data I'm sure still there because the used space is still the same as before in my cluster. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Re: CFP closing soon: Everything Open 2024 (Gladstone, Queensland, Australia, April 16-18)
by Tim Serong 21 Nov '23

21 Nov '23

Update: the CFP has been extended 'til November 30 (see http://lists.linux.org.au/pipermail/eo-announce/2023-November/000011.html) On 11/17/23 14:09, Tim Serong wrote: > Everything Open (auspiced by Linux Australia) is happening again in > 2024. The CFP closes at the end of this weekend (November 19): > > https://2024.everythingopen.au/programme/proposals/ > > More details below. > > -------- Forwarded Message -------- > Date: Sun, 15 Oct 2023 09:16:31 +1000 > From: Everything Open <contact(a)everythingopen.au> > To: eo-announce(a)lists.linux.org.au, announce(a)lists.linux.org.au > Subject: [Announce] Everything Open 2024: Call for Sessions Now Open > User-Agent: Roundcube Webmail/1.1.1 > > Submit your session proposals today - the Everything Open 2024 Call for > Sessions is now open. > > > ## Call for Sessions > > We invite you to submit a session proposal on a topic you are familiar > with via our proposals portal at > https://2024.everythingopen.au/programme/proposals/. > The Call for Sessions will remain open until 11:59pm on Sunday 19 > November 2023 anywhere on earth (AoE). > > There will be multiple streams catering for a wide range of interest > areas across the many facets of open technology, including Linux, open > source software, open hardware, standards, formats and documentation, > and our communities. > In keeping with the conference’s aim to be inclusive to all community > members, presentations can be aimed at any level, ranging from technical > deep-dives through to beginner and intermediate level presentations for > those who are newer to the subject. > > There will be two types of sessions at Everything Open: talks and > tutorials. Talks will nominally be 45 minutes long on a single topic > presented in lecture format. We will also have a few short talk slots of > 25 minutes available, which are perfect for people new to presenting at > a conference. Tutorials are interactive and hands-on in nature, > presented in classroom format. > Each accepted session will receive one Professional level ticket to > attend the conference. > > The Session Selection Committee is looking forward to reading your > submissions. We would also like to thank them for coming together and > volunteering their time to help put this conference together. > > > ## Sponsor Early > > As usual, we have a range of sponsorship opportunities available, for > the conference overall as well as the ability to contribute towards > specific parts of the event. > We encourage you to sponsor the conference early, to get the maximum > promotion during the lead up to the event. > If you or your organisation is interested in sponsoring Everything Open, > please get in touch via > https://2024.everythingopen.au/sponsors/prospectus/. > > ---- > Read this online at > https://2024.everythingopen.au/news/call-for-sessions-open/ > > > _______________________________________________ > announce mailing list > announce(a)lists.linux.org.au > http://lists.linux.org.au/mailman/listinfo/announce > > ----- End forwarded message ----- > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

1 0

Why is min_size of erasure pools set to k+1
by Vladimir Brik 20 Nov '23

20 Nov '23

Could someone help me understand why it's a bad idea to set min_size of erasure-coded pools to k? From what I've read, the argument for k+1 is that if min_size is k and you lose an OSD during recovery after a failure of m OSDs, data will become unavailable. But how does setting min_size to k+1 help? If m=2, if you experience a double failure followed by another failure during recovery you still lost 3 OSDs and therefore your data because the pool wasn't set up to handle 3 concurrent failures, and the value of min_size is irrelevant. https://github.com/ceph/ceph/pull/8008 mentions inability to peer if min_size = k, but I don't understand why. Does that mean that if min_size=k and I lose m OSDs, and then an OSD is restarted during recovery, PGs will not peer even after the restarted OSD comes back online? Vlad

2 1