Hi,
This seems not very relevant since all ceph components are running in
containers. Any ideas to get over this issue? Any other ideas or
suggestions on this kind of deployment?
sudo ./cephadm --image 10.21.22.1:5000/ceph:v17.2.5-20230316 --docker
bootstrap --mon-ip 10.21.22.1 --skip-monitoring-stack
Creating directory /etc/ceph for ceph.conf
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
No time sync service is running; checked for ['chrony.service',
'chronyd.service', 'systemd-timesyncd.service', 'ntpd.service',
'ntp.service', 'ntpsec.service', 'openntpd.service']
ERROR: Distro uos version 20 not supported
uname -a
Linux aaaa 4.19.0-91.82.42.uelc20.x86_64 #1 SMP Sat May 15 13:50:04 CST
2021 x86_64 x86_64 x86_64 GNU/Linux
Thank you in advance
Ben
Details of this release are summarized here:
https://tracker.ceph.com/issues/59542#note-1
Release Notes - TBD
Seeking approvals for:
smoke - Radek, Laura
rados - Radek, Laura
rook - Sébastien Han
cephadm - Adam K
dashboard - Ernesto
rgw - Casey
rbd - Ilya
krbd - Ilya
fs - Venky, Patrick
upgrade/octopus-x (pacific) - Laura (look the same as in 16.2.8)
upgrade/pacific-p2p - Laura
powercycle - Brad (SELinux denials)
ceph-volume - Guillaume, Adam K
Thx
YuriW
Hi all,
our monitors have enjoyed democracy since the beginning. However, I don't share a sudden excitement about voting:
2/9/23 4:42:30 PM[INF]overall HEALTH_OK
2/9/23 4:42:30 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4)
2/9/23 4:42:26 PM[INF]mon.ceph-01 calling monitor election
2/9/23 4:42:26 PM[INF]mon.ceph-26 calling monitor election
2/9/23 4:42:26 PM[INF]mon.ceph-25 calling monitor election
2/9/23 4:42:26 PM[INF]mon.ceph-02 calling monitor election
2/9/23 4:40:00 PM[INF]overall HEALTH_OK
2/9/23 4:30:00 PM[INF]overall HEALTH_OK
2/9/23 4:24:34 PM[INF]overall HEALTH_OK
2/9/23 4:24:34 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4)
2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election
2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election
2/9/23 4:24:29 PM[INF]mon.ceph-03 calling monitor election
2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election
2/9/23 4:24:29 PM[INF]mon.ceph-26 calling monitor election
2/9/23 4:24:29 PM[INF]mon.ceph-25 calling monitor election
2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election
2/9/23 4:24:04 PM[INF]overall HEALTH_OK
2/9/23 4:24:03 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4)
2/9/23 4:23:59 PM[INF]mon.ceph-01 calling monitor election
2/9/23 4:23:59 PM[INF]mon.ceph-02 calling monitor election
2/9/23 4:20:00 PM[INF]overall HEALTH_OK
2/9/23 4:10:00 PM[INF]overall HEALTH_OK
2/9/23 4:00:00 PM[INF]overall HEALTH_OK
2/9/23 3:50:00 PM[INF]overall HEALTH_OK
2/9/23 3:43:13 PM[INF]overall HEALTH_OK
2/9/23 3:43:13 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4)
2/9/23 3:43:08 PM[INF]mon.ceph-01 calling monitor election
2/9/23 3:43:08 PM[INF]mon.ceph-26 calling monitor election
2/9/23 3:43:08 PM[INF]mon.ceph-25 calling monitor election
We moved a switch from one rack to another and after the switch came beck up, the monitors frequently bitch about who is the alpha. How do I get them to focus more on their daily duties again?
Thanks for any help!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers
have started failing with the error message
In function 'void Server::handle_client_open(MDRequestRef&)' thread
7f0ca9908700 time 2021-06-28T09:21:11.484768+0200
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el7/BUILD/ceph-15.2.13/src/mds/Server.cc:
4149: FAILED ceph_assert(cur->is_auth())
Complete log is:
https://gist.github.com/pvanheus/4da555a6de6b5fa5e46cbf74f5500fbd
ceph status output is:
# ceph status
cluster:
id: ed7b2c16-b053-45e2-a1fe-bf3474f90508
health: HEALTH_WARN
30 OSD(s) experiencing BlueFS spillover
insufficient standby MDS daemons available
1 MDSs report slow requests
2 mgr modules have failed dependencies
4347046/326505282 objects misplaced (1.331%)
6 nearfull osd(s)
23 pgs not deep-scrubbed in time
23 pgs not scrubbed in time
8 pool(s) nearfull
services:
mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 22m)
mgr: ceph-mon1(active, since 11w), standbys: ceph-mon2, ceph-mon3
mds: SANBI_FS:2 {0=ceph-mon1=up:active(laggy or
crashed),1=ceph-mon2=up:stopping}
osd: 54 osds: 54 up (since 2w), 54 in (since 11w); 50 remapped pgs
data:
pools: 8 pools, 833 pgs
objects: 42.37M objects, 89 TiB
usage: 159 TiB used, 105 TiB / 264 TiB avail
pgs: 4347046/326505282 objects misplaced (1.331%)
782 active+clean
49 active+clean+remapped
1 active+clean+scrubbing+deep
1 active+clean+remapped+scrubbing
io:
client: 29 KiB/s rd, 427 KiB/s wr, 37 op/s rd, 48 op/s wr
When restarting a MDS it goes through states replace, reconnect, resolve
and finally sets itself to active before this crash happens.
Any advice on what to do?
Thanks,
Peter
P.S. apologies if you received this email more than once - I have had some
trouble figuring out the correct mailing list to use.
Hi,
When using rbd mirroring, the mirroring concerns the images only, not the
whole pool? So, we don't need to have a dedicated pool in the destination
site to be mirrored, the only obligation is that the mirrored pools must
have the same name.
In other words, We create two pools with the same name, one on the source
site the other on the destination site, we create the mirror link (one way
or two ways replication), then we choose what images to sync.
Both pools can be used simultaneously on both sites, it's the mirrored
images that cannot be used simultaneously, only promoted ones.
Is this correct?
Regards.
Hey guys and girls,
I'm working on a project to build storage for one of our departments,
and I want to ask you guys and girls for input on the high-level
overview part. It's a long one, I hope you read along and comment.
SUMMARY
I made a plan last year to build a 'storage solution' including ceph
and some windows VM's to expose the data over SMB to clients. A year
later I finally have the hardware, built a ceph cluster, and I'm doing
tests. Ceph itself runs great, but when I wanted to start exposing the
data using iscsi to our VMware farm, I ran into some issues. I know
the iscsi gateways will introduce some new performance bottlenecks,
but I'm seeing really slow performance, still working on that.
But then I ran into the warning on the iscsi gateway page: "The iSCSI
gateway is in maintenance as of November 2022. This means that it is
no longer in active development and will not be updated to add new
features.". Wait, what? Why!? What does this mean? Does this mean that
iSCSI is now 'feature complete' and will still be supported the next 5
years, or will it be deprecated in the future? I tried searching, but
couldn't find any info on the decision and the roadmap.
My goal is to build a future-proof setup, and using deprecated
components should not be part of that of course.
If the iscsi gateway will still be supported the next few years and I
can iron out the performance issues, I can still go on with my
original plan. If not, I have to go back to the drawing board. And
maybe you guys would advise me to take another route anyway.
GOALS
My goals/considerations are:
- we want >1PB of storage capacity for cheap (on a tight budget) for
research data. Most of it is 'store once, read sometimes'. <1% of the
data is 'hot'.
- focus is on capacity, but it would be nice to have > 200MB/s of
sequential write/read performance and not 'totally suck' on random
i/o. Yes, not very well quantified, but ah. Sequential writes are most
important.
- end users all run Windows computers (mostly VDI's) and a lot of
applications require SMB shares.
- security is a big thing, we want really tight ACL's, specific
monitoring agents, etc.
- our data is incredibly important to us, we still want the 3-2-1
backup rule. Primary storage solution, a second storage solution in a
different place, and some of the data that is not reproducible is also
written to tape. We also want to be protected from ransomware or user
errors (so no direct replication to the second storage).
- I like open source, reliability, no fork-lift upgrades, no vendor
lock-in, blah, well, I'm on the ceph list here, no need to convince
you guys ;)
- We're hiring a commercial company to do ceph maintenance and support
for when I'm on leave or leaving the company, but they won't support
clients, backup software, etc, so I want something as simple as
possible. We do have multiple Windows/VMware admins, but no other real
linux guru's.
THE INITIAL PLAN
Given these considerations, I ordered two identical clusters, each
consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2
ssd's and 10 capacity disks (EC 4:2 for the data), and each node is
connected using a 2x25Gbps bond. Ceph is running like a charm. Now I
just have to think about exposing the data to end users, and I've been
testing different setups.
My original plan was to expose for example 10x100TB rbd images using
iSCSI to our VMware farm, formatting the luns with VMFS6, and run for
example 2 Windows file servers per datastore on that with a single DFS
namespace to end users. Then backup the file servers using our
existing Veeam infrastructure to RGW running on the second cluster
with an immutable bucket. This way we would have easily defined
security boundaries: the clients can only reach the file servers, the
file servers only see their local VMDK's, ESX only sees the luns on
the iSCSI target, etc. When a file server would be compromised, it
would have no access to ceph. We have easy incremental backups,
immutability for ransomware protection, etc. And the best part is that
the ceph admin can worry about ceph, the vmware admin can focus on
ESX, VMFS and all the vmware stuff, and the Windows admins can focus
on the Windows boxes, Windows-specific ACLS and tools and Veeam
backups and stuff.
CURRENT SITUATION
I'm building out this plan now, but I'm running into issues with
iSCSI. Are any of you doing something similar? What is your iscsi
performance compared to direct rbd?
In regard to performance: If I take 2 test windows VM's, I put one on
an iSCSI datastore and another with direct rbd access using the
windows rbd driver, I create a share on those boxes and push data to
it, I see different results (of course). Copying some iso images over
SMB to the 'windows vm running direct rbd' I see around 800MB/s write,
and 200MB/s read, which is pretty okay. When I send data to the
'windows vm running on top of iscsi' it starts writing at around
350MB/s, but after like 10-20 seconds drops to 100MB/s and won't go
faster. Reads are anywhere from 40MB/s to 80MB/s, which is not really
acceptable.
Another really viable and performant scenario would be to have the
Windows file servers connect to rbd directly with the windows rbd
driver. It seems to work well, it's fast, and you don't have the
bottleneck that the iscsi gateway creates. But I see this driver is
still in beta. Is anyone using this in production? What are your
experiences? We would miss out on the separation of layers and thus
have less security, but at the same time, it really increases
efficiency and performance.
And if I use rbd, then vmware won't see the storage, and I cannot do
an image backup using veeam. I could of course do backups of the rbd
images, using tools like restic or backy to rgw running on the second
cluster with immutable buckets. What are your experiences? Is it easy
to do differential backups of lots of 50TB rbd images? Change rate is
usually like 0.005% per day or something ;)
By the way, we also thought about CephFS, but we have some complex
stuff going on with extended ACL's that I don't think will play nice
with CephFS, and I think it's a lot more complex to backup CephFS than
block images.
If you made it here, thank you for your time! I hope you can share
thoughts on my questions!
Angelo.
Dear all,
We are experiencing with Ceph after deploying it by PVE with the network backed by a 10G Cisco switch with VPC feature on. We are encountering a slow OSD heartbeat and have not been able to identify any network traffic issues.
Upon checking, we found that the ping is around 0.1ms, and there is occasional 2% packet loss when using flood ping, but not consistently. We also noticed a large number of UDP port 5405 packets and the 'corosync' process utilizing a significant amount of CPU.
When running the 'ceph -s' command, we observed a slow OSD heartbeat on the back and front, with the longest latency being 2250.54ms. We suspect that this may be a network issue, but we are unsure of how Ceph detects such long latency. Additionally, we are wondering if a 2% packet loss can significantly affect Ceph's performance and even cause the OSD process to fail sometimes.
We have heard about potential issues with rockdb 6 causing OSD process failures, and we are curious about how to check the rockdb version. Furthermore, we are wondering how severe traffic package loss and latency must be to cause OSD process crashes, and how the monitoring system determines that an OSD is offline.
We would greatly appreciate any assistance or insights you could provide on these matters.
Thanks,
Hi Venky,
> Also, at one point the kclient wasn't able to handle more than 400 snapshots (per file system), but we have come a long way from that and that is not a constraint right now.
Does it mean that there is no more limit to the number of snapshots per filesystem? And, if not, do you know what is the max number of snapshots now (per filesystem) ??
Cheers,
Arnaud
Hi,
When creating a ceph cluster, a failover domain is created, and by default
it uses host as a minimal domain, that domain can be modified to chassis,
or rack, ...etc.
My question is :
Suppose I have three osd nodes, my replication is 3 and my failover domain
is host, which means that each copy of data is stored on a different node.
What happens when one node crashes, does Ceph use the remaining free space
on the other two to create the third copy, or the ceph cluster will run in
degraded mode, like a RAID5
which lost a disk.
Regards.
