Hi everyone,
On behalf of the Ceph Foundation Board, I would like to announce the
creation of, and cordially invite you to, the first of a recurring series
of meetings focused solely on gathering feedback from the users of
Ceph. The overarching goal of these meetings is to elicit feedback from the
users, companies, and organizations who use Ceph in their production
environments. You can find more details about the motivation behind this
effort in our user survey [1] that we highly encourage all of you to take.
This is an extension of the Ceph User Dev Meeting with concerted focus on
Performance (led by Vincent Hsu, IBM) and Orchestration/Deployment (led by
Matt Leonard, Bloomberg), to start off with. We would like to kick off this
series of meetings on March 21, 2024. The survey will be open until March
18, 2024.
Looking forward to hearing from you!
Thanks,
Neha
[1]
https://docs.google.com/forms/d/15aWxoG4wSQz7ziBaReVNYVv94jA0dSNQsDJGqmHCLM…
Hi everyone,
I’d like to extend a warm thank you to Mike Perez for his years of service
as community manager for Ceph. He is changing focuses now to engineering.
The Ceph Foundation board decided to use services from the Linux Foundation
to fulfill some community management responsibilities, rather than rely on
a single member organization employing a community manager. The Linux
Foundation will assist with Ceph Foundation membership and governance
matters.
Please welcome Noah Lehman (cc’d) as our social media and marketing point
person - for anything related to this area, including the Ceph YouTube
channel, please reach out to him.
Ceph days will continue to be organized and funded by organizations around
the world, with the help of the Ceph Ambassadors (
https://ceph.io/en/community/ambassadors/). Gaurav Sitlani (cc’d) will help
organize the ambassadors going forward.
For other matters, please contact council(a)ceph.io and we’ll direct the
matter to the appropriate people.
Thanks,
Neha Ojha, Dan van der Ster, Josh Durgin
Ceph Executive Council
We are happy to announce another release of the go-ceph API library. This is a
regular release following our every-two-months release cadence.
https://github.com/ceph/go-ceph/releases/tag/v0.27.0
The library includes bindings that aim to play a similar role to the "pybind"
python bindings in the ceph tree but for the Go language. The library also
includes additional APIs that can be used to administer cephfs, rbd, rgw, and
other subsystems.
There are already a few consumers of this library in the wild, including the
ceph-csi project.
--
John Mulligan
phlogistonjohn(a)asynchrono.us
jmulligan(a)redhat.com
Hello! I've installed my 5-node CEPH cluster next install NFS server by command:
ceph nfs cluster create nfshacluster 5 --ingress --virtual_ip 192.168.171.48/26 --ingress-mode haproxy-protocol.
I don't understand fully how this must be works but when i stop NFS daemon even on one of this nodes I've see that writing on NFS shares is disappear (testing via vdbench).
As i understand it is wrong and IO from stopped daemon must switching to another NFS daemon without any impact on IO.
Can someone help me with troubleshoot this issue? Or explain how done full-fledged Active-Active HA NFS Cluster for production use.
Thanks!
Руслан Нурабаев
Старший инженер
Сектор ИТ платформы
Отдел развития опорной сети
Департамент развития сети
+77012119272
Ruslan.Nurabayev(a)kcell.kz
-----Original Message-----
From: ceph-users-request(a)ceph.io <ceph-users-request(a)ceph.io>
Sent: Thursday, April 11, 2024 15:07
To: Ruslan Nurabayev <Ruslan.Nurabayev(a)kcell.kz>
Subject: Welcome to the "ceph-users" mailing list
[You don't often get email from ceph-users-request(a)ceph.io. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Welcome to the "ceph-users" mailing list!
To post to this list, send your email to:
ceph-users(a)ceph.io
You can unsubscribe or make adjustments to your options via email by sending a message to:
ceph-users-request(a)ceph.io
with the word 'help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You will need your password to change your options, but for security purposes, this password is not included here. If you have forgotten your password you will need to reset it via the web UI.
________________________________
****************************************************************************************
Осы хабарлама және онымен берілетін кез келген файлдар құпия болып
табылады және олар мекенжайда көрсетілген жеке немесе заңды тұлғалардың
пайдалануына ғана арналған. Егер сіз болжамды алушы болып табылмайтын
болсаңыз, осы арқылы осындай ақпаратты кез келген таратуға, жіберуге,
көшіруге немесе пайдалануға қатаң тыйым салынатыны және осы электрондық
хабарлама дереу жойылуға тиіс екендігін хабарлаймыз.
KCELL осы хабарламадағы кез келген ақпараттың дәлдігіне немесе
толықтығына қатысты ешқандай кепілдік бермейді және сол арқылы онда
қамтылған ақпарат үшін немесе оны беру, қабылдау, сақтау немесе қандай да
бір түрде пайдалану үшін кез келген жауапкершілікті болдырмайды. Осы
хабарламада айтылған пікірлер тек жіберушіге ғана тиесілі және KCELL
пікірін де білдіруі міндетті емес. Бұл электрондық хабарлама барлық
танымал компьютерлік вирустарға тексерілді.
****************************************************************************************
Данное сообщение и любые передаваемые с ним файлы являются
конфиденциальными и предназначены исключительно для использования
физическими или юридическими лицами, которым они адресованы. Если вы не
являетесь предполагаемым получателем, настоящим уведомляем о том,
что любое распространение, пересылка, копирование или использование такой
информации строго запрещено, и данное электронное сообщение должно
быть немедленно удалено.
KCELL не дает никаких гарантий относительно точности или полноты любой
информации, содержащейся в данном сообщении, и тем самым исключает
любую ответственность за информацию, содержащуюся в нем, или за ее
передачу, прием, хранение или использование каким-либо образом. Мнения,
выраженные в данном сообщении, принадлежат только отправителю и
не обязательно отражают мнение KCELL.
Данное электронное сообщение было проверено на наличие всех известных
компьютерных вирусов.
****************************************************************************************
This e-mail and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you are not the intended recipient you are hereby notified
that any dissemination, forwarding, copying or use of any of the
information is strictly prohibited, and the e-mail should immediately be
deleted.
KCELL makes no warranty as to the accuracy or completeness of any
information contained in this message and hereby excludes any liability
of any kind for the information contained therein or for the information
transmission, reception, storage or use of such in any way
whatsoever. The opinions expressed in this message belong to sender alone
and may not necessarily reflect the opinions of KCELL.
This e-mail has been scanned for all known computer viruses.
****************************************************************************************
Is there a how-to document available on how to setup Hashicorp's Vault for Ceph, preferably in a HA state?
Due to some encryption needs, we need to set up LUKS, OSD encryption AND Ceph bucket encryption as well. Yes, we know there will be a performance hit, but the encrypt-everything is a hard requirement for our business needs since we have government and healthcare-related contracts.
-- Michael
This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.
Hello Ceph User Community,
I currently have a large Amazon S3 environment with terabytes of data
spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an
on-site Ceph cluster using the RGW. I'm trying to figure out the
most efficient way to achieve this. Looking through the documentation, I
found articles related to the cloud sync module, released in Mimic (
https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/). I also watched
a video on the cloud sync module as well. It *sounds* like this is the
functionality I'm looking for.
Given I'm moving away from Amazon S3, I'm really just looking for a one-way
replication between the buckets (i.e. Provide an Amazon S3 access
key/secret which is associated to the buckets and the same for the Ceph
environment, so object data can be replicated one-to-one, without creating
ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on
modifying my boto connection objects to use the new Ceph environment. Is
what I'm describing feasible with the cloud sync module? Just looking for
some affirmation, given I'm not well versed in Ceph's RGW, especially
around multi-site configurations.
Thanks,
Jimmy
Hi,
I'm running Ceph Quincy (17.2.6) with a rados-gateway. I have muti tenants,
for example:
- Tenant1$manager
- Tenant1$readwrite
I would like to set a policy on a bucket (backups for example) owned by
*Tenant1$manager* to allow *Tenant1$readwrite* access to that bucket. I
can't find any documentation that discusses this scenario.
Does anyone know how to specify the Principle and Resource section of a
policy.json file? Or any other configuration that I might be missing?
I've tried some variations on Principal and Resource including and
excluding tenant information, but not no luck yet.
For example:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["arn:aws:iam:::user/*Tenant1$readwrite*"]},
"Action": ["s3:ListBucket","s3:GetObject", ,"s3:PutObject"],
"Resource": [
"arn:aws:s3:::*Tenant1/backups*"
]
}]
}
I'm using s3cmd for testing, so:
s3cmd --config s3cfg.manager setpolicy policy.json s3://backups/
Returns:
s3://backups/: Policy updated
And then testing:
s3cmd --config s3cfg.readwrite ls s3://backups/
ERROR: Access to bucket 'backups' was denied
ERROR: S3 error: 403 (AccessDenied)
Thanks,
Tom
Hi All,
I've been battling this for a while and I'm not sure where to go from
here. I have a Ceph health warning as such:
# ceph -s
cluster:
id: 58bde08a-d7ed-11ee-9098-506b4b4da440
health: HEALTH_WARN
1 MDSs report slow requests
1 MDSs behind on trimming
services:
mon: 5 daemons, quorum
pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d)
mgr: pr-md-01.jemmdf(active, since 3w), standbys: pr-md-02.emffhz
mds: 1/1 daemons up, 2 standby
osd: 46 osds: 46 up (since 9h), 46 in (since 2w)
data:
volumes: 1/1 healthy
pools: 4 pools, 1313 pgs
objects: 260.72M objects, 466 TiB
usage: 704 TiB used, 424 TiB / 1.1 PiB avail
pgs: 1306 active+clean
4 active+clean+scrubbing+deep
3 active+clean+scrubbing
io:
client: 123 MiB/s rd, 75 MiB/s wr, 109 op/s rd, 1.40k op/s wr
And the specifics are:
# ceph health detail
HEALTH_WARN 1 MDSs report slow requests; 1 MDSs behind on trimming
[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
mds.slugfs.pr-md-01.xdtppo(mds.0): 99 slow requests are blocked >
30 secs
[WRN] MDS_TRIM: 1 MDSs behind on trimming
mds.slugfs.pr-md-01.xdtppo(mds.0): Behind on trimming (13884/250)
max_segments: 250, num_segments: 13884
That "num_segments" number slowly keeps increasing. I suspect I just
need to tell the MDS servers to trim faster but after hours of googling
around I just can't figure out the best way to do it. The best I could
come up with was to decrease "mds_cache_trim_decay_rate" from 1.0 to .8
(to start), based on this page:
https://www.suse.com/support/kb/doc/?id=000019740
But it doesn't seem to help, maybe I should decrease it further? I am
guessing this must be a common issue...? I am running Reef on the MDS
servers, but most clients are on Quincy.
Thanks for any advice!
cheers,
erich
Hi everyone.
I got a warning with
root@cthulhu1:/etc/ceph# ceph -s
cluster:
id: 9c5bb196-c212-11ee-84f3-c3f2beae892d
health: HEALTH_ERR
1 scrub errors
Possible data damage: 1 pg inconsistent
So I find the pg with the issue, and launch a pg repair (still waiting)
But I try to find «why» so I check all the OSD related on this pg and
didn't find anything, no error from osd daemon, no errors from smartctl, no
error from the kernel message.
So I just like to know if that's «normal» or should I scratch deeper.
JAS
--
Albert SHIH 🦫 🐸
France
Heure locale/Local time:
ven. 12 avril 2024 11:51:37 CEST
Hi,
I'm trying to estimate the possible impact when large PGs are
splitted. Here's one example of such a PG:
PG_STAT OBJECTS BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG UP
86.3ff 277708 414403098409 0 0 3092
3092
[187,166,122,226,171,234,177,163,155,34,81,239,101,13,117,8,57,111]
Their main application is RGW on EC (currently 1024 PGs on 240 OSDs),
8TB HDDs backed by SSDs. There are 6 RGWs running behind HAProxies. It
took me a while to convince them to do a PG split and now they're
trying to assess how big the impact could be. The fullest OSD is
already at 85% usage, the least filled one at 59%, so there is
definitely room for a better balancing which, will be necessary until
the new hardware arrives. The current distribution is around 100 PGs
per OSD which usually would be fine, but since the PGs are that large
only a few PGs difference have a huge impact on the OSD utilization.
I'm targeting 2048 PGs for that pool for now, probably do another
split when the new hardware has been integrated.
Any comments are appreciated!
Eugen
