- ceph-users - lists.ceph.io

RBD mirroring, asking for clarification
by wodel youchi 03 May '23

03 May '23

Hi, When using rbd mirroring, the mirroring concerns the images only, not the whole pool? So, we don't need to have a dedicated pool in the destination site to be mirrored, the only obligation is that the mirrored pools must have the same name. In other words, We create two pools with the same name, one on the source site the other on the destination site, we create the mirror link (one way or two ways replication), then we choose what images to sync. Both pools can be used simultaneously on both sites, it's the mirrored images that cannot be used simultaneously, only promoted ones. Is this correct? Regards.

2 5

architecture help (iscsi, rbd, backups?)
by Angelo Höngens 02 May '23

02 May '23

Hey guys and girls, I'm working on a project to build storage for one of our departments, and I want to ask you guys and girls for input on the high-level overview part. It's a long one, I hope you read along and comment. SUMMARY I made a plan last year to build a 'storage solution' including ceph and some windows VM's to expose the data over SMB to clients. A year later I finally have the hardware, built a ceph cluster, and I'm doing tests. Ceph itself runs great, but when I wanted to start exposing the data using iscsi to our VMware farm, I ran into some issues. I know the iscsi gateways will introduce some new performance bottlenecks, but I'm seeing really slow performance, still working on that. But then I ran into the warning on the iscsi gateway page: "The iSCSI gateway is in maintenance as of November 2022. This means that it is no longer in active development and will not be updated to add new features.". Wait, what? Why!? What does this mean? Does this mean that iSCSI is now 'feature complete' and will still be supported the next 5 years, or will it be deprecated in the future? I tried searching, but couldn't find any info on the decision and the roadmap. My goal is to build a future-proof setup, and using deprecated components should not be part of that of course. If the iscsi gateway will still be supported the next few years and I can iron out the performance issues, I can still go on with my original plan. If not, I have to go back to the drawing board. And maybe you guys would advise me to take another route anyway. GOALS My goals/considerations are: - we want >1PB of storage capacity for cheap (on a tight budget) for research data. Most of it is 'store once, read sometimes'. <1% of the data is 'hot'. - focus is on capacity, but it would be nice to have > 200MB/s of sequential write/read performance and not 'totally suck' on random i/o. Yes, not very well quantified, but ah. Sequential writes are most important. - end users all run Windows computers (mostly VDI's) and a lot of applications require SMB shares. - security is a big thing, we want really tight ACL's, specific monitoring agents, etc. - our data is incredibly important to us, we still want the 3-2-1 backup rule. Primary storage solution, a second storage solution in a different place, and some of the data that is not reproducible is also written to tape. We also want to be protected from ransomware or user errors (so no direct replication to the second storage). - I like open source, reliability, no fork-lift upgrades, no vendor lock-in, blah, well, I'm on the ceph list here, no need to convince you guys ;) - We're hiring a commercial company to do ceph maintenance and support for when I'm on leave or leaving the company, but they won't support clients, backup software, etc, so I want something as simple as possible. We do have multiple Windows/VMware admins, but no other real linux guru's. THE INITIAL PLAN Given these considerations, I ordered two identical clusters, each consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2 ssd's and 10 capacity disks (EC 4:2 for the data), and each node is connected using a 2x25Gbps bond. Ceph is running like a charm. Now I just have to think about exposing the data to end users, and I've been testing different setups. My original plan was to expose for example 10x100TB rbd images using iSCSI to our VMware farm, formatting the luns with VMFS6, and run for example 2 Windows file servers per datastore on that with a single DFS namespace to end users. Then backup the file servers using our existing Veeam infrastructure to RGW running on the second cluster with an immutable bucket. This way we would have easily defined security boundaries: the clients can only reach the file servers, the file servers only see their local VMDK's, ESX only sees the luns on the iSCSI target, etc. When a file server would be compromised, it would have no access to ceph. We have easy incremental backups, immutability for ransomware protection, etc. And the best part is that the ceph admin can worry about ceph, the vmware admin can focus on ESX, VMFS and all the vmware stuff, and the Windows admins can focus on the Windows boxes, Windows-specific ACLS and tools and Veeam backups and stuff. CURRENT SITUATION I'm building out this plan now, but I'm running into issues with iSCSI. Are any of you doing something similar? What is your iscsi performance compared to direct rbd? In regard to performance: If I take 2 test windows VM's, I put one on an iSCSI datastore and another with direct rbd access using the windows rbd driver, I create a share on those boxes and push data to it, I see different results (of course). Copying some iso images over SMB to the 'windows vm running direct rbd' I see around 800MB/s write, and 200MB/s read, which is pretty okay. When I send data to the 'windows vm running on top of iscsi' it starts writing at around 350MB/s, but after like 10-20 seconds drops to 100MB/s and won't go faster. Reads are anywhere from 40MB/s to 80MB/s, which is not really acceptable. Another really viable and performant scenario would be to have the Windows file servers connect to rbd directly with the windows rbd driver. It seems to work well, it's fast, and you don't have the bottleneck that the iscsi gateway creates. But I see this driver is still in beta. Is anyone using this in production? What are your experiences? We would miss out on the separation of layers and thus have less security, but at the same time, it really increases efficiency and performance. And if I use rbd, then vmware won't see the storage, and I cannot do an image backup using veeam. I could of course do backups of the rbd images, using tools like restic or backy to rgw running on the second cluster with immutable buckets. What are your experiences? Is it easy to do differential backups of lots of 50TB rbd images? Change rate is usually like 0.005% per day or something ;) By the way, we also thought about CephFS, but we have some complex stuff going on with extended ACL's that I don't think will play nice with CephFS, and I think it's a lot more complex to backup CephFS than block images. If you made it here, thank you for your time! I hope you can share thoughts on my questions! Angelo.

6 8

[multisite] "bucket sync status" takes a while
by Yixin Jin 02 May '23

02 May '23

Hi folks, With a multi-site environment, when I create a bucket-level sync policy with a symmetric flow between the master zone and another zone, "bucket sync status" immediately shows that the sync is now enabled in the master zone. But it takes a while for it to show that in the other zone. I tried "period pull" at the other zone and "period push" at the master zone. Neither seem to make a difference. Is there a way to speed up this process? Thanks, Yixin

1 0

17.2.6 fs 'ls' ok, but 'cat' 'operation not permitted' puzzle
by Harry G Coin 02 May '23

02 May '23

In 17.2.6 is there a security requirement that pool names supporting a ceph fs filesystem match the filesystem name.data for the data and name.meta for the associated metadata pool? (multiple file systems are enabled) I have filesystems from older versions with the data pool name matching the filesystem and appending _metadata for that, and even older filesystems with the pool name as in 'library' and 'library_metadata' supporting a filesystem called 'libraryfs' The pools all have the cephfs tag. But using the documented: ceph fs authorize libraryfs client.basicuser / rw command allows the root user to mount and browse the library directory tree, but fails with 'operation not permitted' when even reading any file. However, changing the client.basicuser osd auth to 'allow rw' instead of 'allow rw tag...' allows normal operations. So: [client.basicuser] key = <key stuff>== caps mds = "allow rw fsname=libraryfs" caps mon = "allow r fsname=libraryfs" caps osd = "allow rw" works, but the same with caps osd = "allow rw tag cephfs data=libraryfs" leads to the 'operation not permitted' on read, or write or any actual access. It remains a puzzle. Help appreciated! Were there upgrade instructions about that, any help pointing me to them? Thanks Harry Coin Rock Stable Systems

1 0

Memory leak in MGR after upgrading to pacific.
by Gary Molenkamp 02 May '23

02 May '23

Good morning, After upgrading from Octopus (15.2.17) to Pacific (16.2.12) two days ago, I'm noticing that the MGR daemons keep failing over to standby and then back every 24hrs. Watching the output of 'ceph orch ps' I can see that the memory consumption of the mgr is steadily growing until it becomes unresponsive. When the mgr becomes unresponsive, tasks such as RESTful calls start to fail, and the standby eventually takes over after ~20 minutes. I've included a log of memory consumption (in 10 minute intervals) at the end of this message. While the cluster recovers during this issue, the loss of usage data during the outage, and the fact its occurring is problematic. Any assistance would be appreciated. Note, this is a cluster that has been upgraded from an original jewel based ceph using filestore, through bluestore conversion, container conversion, and now to Pacific. The data below shows memory use with three mgr modules enabled: cephadm, restful, iostat. By disabling iostat, I can reduce the rate of memory consumption increasing to about 200MB/hr. Thanks Gary. Wed Apr 26 12:50:08 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 7m ago 3h 1099M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 7m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:00:09 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 6m ago 3h 1161M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:10:10 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 6m ago 3h 1230M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:20:11 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 6m ago 4h 1250M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:30:12 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1318M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 5m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:40:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1379M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:50:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1442M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:00:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 4m ago 4h 1498M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:10:15 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 4m ago 4h 1554M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:20:16 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 4m ago 5h 1617M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:30:17 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 3m ago 5h 1677M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:40:18 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 3m ago 5h 1735M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 3m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:50:19 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1806M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:00:20 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1873M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:10:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1920M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:20:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 114s ago 6h 1981M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 113s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:30:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 95s ago 6h 2048M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 95s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:40:23 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 80s ago 6h 2121M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 80s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:50:24 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 53s ago 6h 2170M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 53s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:00:25 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 35s ago 6h 2238M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 34s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:10:26 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 19s ago 6h 2266M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 19s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:20:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 10m ago 7h 2266M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 10m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:30:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2321M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 9m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:40:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2382M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:50:29 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2449M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:00:30 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 8m ago 7h 2505M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:10:31 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 8m ago 7h 2562M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:20:32 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 8m ago 8h 2628M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:30:33 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2692M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 7m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:40:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2761M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:50:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2811M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:00:35 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 6m ago 8h 2873M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:10:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 6m ago 8h 2923M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:20:37 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 6m ago 9h 2995M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:30:38 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3055M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 5m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:40:39 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3110M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:50:40 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3166M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:00:41 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 4m ago 9h 3244M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:10:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 4m ago 9h 3289M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:20:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 4m ago 10h 3368M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:30:43 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3437M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 3m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:40:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3496M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:50:45 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3546M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:00:46 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 2m ago 10h 3627M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:10:47 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 2m ago 10h 3637M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:20:48 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 92s ago 11h 492M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 91s ago 23h 418M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:31:55 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 51s ago 11h 373M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 51s ago 23h 520M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:41:56 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 32s ago 11h 374M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 32s ago 0h 585M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:51:57 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 17s ago 11h 376M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 17s ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:01:58 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 10m ago 11h 376M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 10m ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:11:58 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 9m ago 11h 378M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 698M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:21:59 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 9m ago 12h 379M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 758M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:32:00 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 9m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 825M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:42:01 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 895M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:52:02 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 948M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:02:03 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 1010M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:12:04 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 7m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1071M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:22:05 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 7m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1132M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:32:06 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 7m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1183M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:42:06 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1244M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:52:07 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1311M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:02:08 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1380M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:12:09 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 5m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1441M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:22:10 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1508M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:32:11 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1541M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:42:12 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1610M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:52:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1674M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:02:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1725M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:12:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 3m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1794M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:22:15 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1844M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:32:16 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1917M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:42:17 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2002M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:52:18 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2056M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:02:19 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2113M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:12:20 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 92s ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 91s ago 4h 2173M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:22:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 73s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 72s ago 4h 2201M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:32:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 57s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 57s ago 4h 2264M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:42:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 29s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 29s ago 5h 2324M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:52:23 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 10s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 10s ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:02:24 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 10m ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 10m ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:12:25 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 9m ago 16h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2442M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:22:26 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2502M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:32:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2552M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:42:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2614M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:52:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2681M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:02:29 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2748M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:12:30 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 7m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2810M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:22:31 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2867M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:32:32 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2916M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:42:33 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3024M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:52:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3070M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:02:35 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3127M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:12:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 5m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 5m ago 7h 3183M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:22:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 5m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 5m ago 7h 3215M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:32:37 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 4m ago 7h 3290M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:42:38 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 4m ago 8h 3342M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:52:39 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 4m ago 8h 3392M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:02:40 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3451M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:12:41 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3516M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:22:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 3m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3566M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:32:43 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 2m ago 8h 3632M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:42:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 2m ago 9h 3696M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:52:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 2m ago 9h 3760M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:02:45 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 115s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 114s ago 9h 3813M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:12:46 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 99s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 99s ago 9h 3869M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:22:47 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 72s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 72s ago 9h 3923M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:32:48 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 54s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 53s ago 9h 3990M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:42:49 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 38s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 38s ago 10h 4040M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:52:50 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 11s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 10s ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:02:51 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 10m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 10m ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:12:51 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 9m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 9m ago 10h 4157M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:22:52 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (22h) 9m ago 22h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 9m ago 10h 4213M - 16.2.12 921c4e969fff 6fed006895e8 -- Gary Molenkamp Science Technology Services Systems Administrator University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

1 1

Balancing Reads in Ceph
by Alan Nair 02 May '23

02 May '23

Hi. I am currently using Ceph for replicated storage to store many objects across 5 nodes with 3x replication. When I generate ~1000 read requests to a single object, they all get serviced by the same primary OSD. I would like to balance the reads across the replicas. So I use the following: auto read_op = rados_create_read_op(); rados_read_op_read(read_op, offset, outSize, buffer, &bytes_read, &prval); err = rados_read_op_operate(read_op, pool->ioctx, keyName.c_str(), LIBRADOS_OPERATION_BALANCE_READS); However, this does not seem to balance the reads across replicas. I do not see what I am doing wrong in the above code. Could you please guide me on this? ceph-mon and ceph-osd are run on Ubuntu 22.04 installed via apt-get update ceph ceph-mds ceph-volume If I should ask this question somewhere else, please point me in the right direction. Thanks and regards, Alan. The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th' ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

1 0

PVE CEPH OSD heartbeat show
by Peter 02 May '23

02 May '23

Dear all, We are experiencing with Ceph after deploying it by PVE with the network backed by a 10G Cisco switch with VPC feature on. We are encountering a slow OSD heartbeat and have not been able to identify any network traffic issues. Upon checking, we found that the ping is around 0.1ms, and there is occasional 2% packet loss when using flood ping, but not consistently. We also noticed a large number of UDP port 5405 packets and the 'corosync' process utilizing a significant amount of CPU. When running the 'ceph -s' command, we observed a slow OSD heartbeat on the back and front, with the longest latency being 2250.54ms. We suspect that this may be a network issue, but we are unsure of how Ceph detects such long latency. Additionally, we are wondering if a 2% packet loss can significantly affect Ceph's performance and even cause the OSD process to fail sometimes. We have heard about potential issues with rockdb 6 causing OSD process failures, and we are curious about how to check the rockdb version. Furthermore, we are wondering how severe traffic package loss and latency must be to cause OSD process crashes, and how the monitoring system determines that an OSD is offline. We would greatly appreciate any assistance or insights you could provide on these matters. Thanks,

3 4

Re: cephfs - max snapshot limit?
by MARTEL Arnaud 01 May '23

01 May '23

Hi Venky, > Also, at one point the kclient wasn't able to handle more than 400 snapshots (per file system), but we have come a long way from that and that is not a constraint right now. Does it mean that there is no more limit to the number of snapshots per filesystem? And, if not, do you know what is the max number of snapshots now (per filesystem) ?? Cheers, Arnaud

2 1

[multisite] Resetting an empty bucket
by Yixin Jin 01 May '23

01 May '23

Hi folks, Armed with bucket-specific sync policy feature, I found that we could move objects of a bucket between zones. It is migration via sync followed by object removal at the source. This allows us to better utilize available capacities in different clusters/zones. However, to achieve this, we need a way to reset an empty bucket so that it can serve as a destination for a migration after it serves as a source before. ceph/rgw currently doesn't seem to be able to do that. So I create a feature request for it https://tracker.ceph.com/issues/59593 My own prototype shows that this feature is fairly simply to implement and works well for bucket migration. Cheers, Yixin

2 1

Ceph recovery
by wodel youchi 01 May '23

01 May '23

Hi, When creating a ceph cluster, a failover domain is created, and by default it uses host as a minimal domain, that domain can be modified to chassis, or rack, ...etc. My question is : Suppose I have three osd nodes, my replication is 3 and my failover domain is host, which means that each copy of data is stored on a different node. What happens when one node crashes, does Ceph use the remaining free space on the other two to create the third copy, or the ceph cluster will run in degraded mode, like a RAID5 which lost a disk. Regards.

2 2