June 2023 - ceph-users - lists.ceph.io

libceph: mds1 IP+PORT wrong peer at address

by Frank Schilder

Hi all, we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what action to take. We get the error "wrong peer at address" in dmesg and some jobs on that server seem to get stuck in fs access; log extract below. I found these 2 tracker items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519, which don't seem to have fixes. My questions: - Is this harmless or does it indicate invalid/corrupted client cache entries? - How to resolve, ignore, umount+mount or reboot? Here an extract from the dmesg log, the error has survived a couple of MDS restarts already: [Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [...] [Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3 [Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

7 months, 3 weeks

4
6
0 0

RGW multisite logs (data, md, bilog) not being trimmed automatically?

by Christian Rohmann

Hey ceph-users, I am running two (now) Quincy clusters doing RGW multi-site replication with only one actually being written to by clients. The other site is intended simply as a remote copy. On the primary cluster I am observing an ever growing (objects and bytes) "sitea.rgw.log" pool, not so on the remote "siteb.rgw.log" which is only 300MiB and around 15k objects with no growth. Metrics show that the growth of pool on primary is linear for at least 6 months, so not sudden spikes or anything. Also sync status appears to be totally happy. There are also no warnings in regards to large OMAPs or anything similar. I was under the impression that RGW will trim its three logs (md, bi, data) automatically and only keep data that has not yet been replicated by the other zonegroup members? The config option "ceph config get mgr rgw_sync_log_trim_interval" is set to 1200, so 20 Minutes. So I am wondering if there might be some inconsistency or how I can best analyze what the cause for the accumulation of log data is? There are older questions on the ML, such as [1], but there was not really a solution or root cause identified. I know there is manual trimming, but I'd rather want to analyze the current situation and figure out what the cause for the lack of auto-trimming is. * Do I need to go through all buckets and count logs and look at their timestamps? Which queries do make sense here? * Is there usually any logging of the log trimming activity that I should expect? Or that might indicate why trimming does not happen? Regards Christian [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/WZCFOAMLWV…

7 months, 3 weeks

1
2
0 0

Critical Information: DELL/Toshiba SSDs dying after 70,000 hours of operation

by Frédéric Nass

Hello, This message does not concern Ceph itself but a hardware vulnerability which can lead to permanent loss of data on a Ceph cluster equipped with the same hardware in separate fault domains. The DELL / Toshiba PX02SMF020, PX02SMF040, PX02SMF080 and PX02SMB160 SSD drives of the 13G generation of DELL servers are subject to a vulnerability which renders them unusable after 70,000 hours of operation, i.e. approximately 7 years and 11 months of activity. This topic has been discussed here: https://www.dell.com/community/PowerVault/TOSHIBA-PX02SMF080-has-lost-commu… The risk is all the greater since these disks may die at the same time in the same server leading to the loss of all data in the server. To date, DELL has not provided any firmware fixing this vulnerability, the latest firmware version being "A3B3" released on Sept. 12, 2016: https://www.dell.com/support/home/en-us/ drivers/driversdetails?driverid=hhd9k If your have servers running these drives, check their uptime. If they are close to the 70,000 hour limit, replace them immediately. The smartctl tool does not report the uptime for these SSDs, but if you have HDDs in the server, you can query their SMART status and get their uptime, which should be about the same as the SSDs. The smartctl command is: smartctl -a -d megaraid,XX /dev/sdc (where XX is the iSCSI bus number). We have informed DELL about this but have no information yet on the arrival of a fix. We have lost 6 disks, in 3 different servers, in the last few weeks. Our observation shows that the drives don't survive full shutdown and restart of the machine (power off then power on in iDrac), but they may also die during a single reboot (init 6) or even while the machine is running. Fujitsu released a corrective firmware in June 2021 but this firmware is most certainly not applicable to DELL drives: https://www.fujitsu.com/us/imagesgig5/PY-CIB070-00.pdf Regards, Frederic Sous-direction Infrastructure and Services Direction du Numérique Université de Lorraine

8 months, 1 week

2
2
1 0

ceph quincy repo update to debian bookworm...?

by Christian Peters

Hi ceph users/maintainers, I installed ceph quincy on debian bullseye as a ceph client and now want to update to bookworm. I see that there is at the moment only bullseye supported. https://download.ceph.com/debian-quincy/dists/bullseye/ Will there be an update of deb https://download.coeh.com/debian-quincy/ bullseye main to deb https://download.coeh.com/debian-quincy/ boowkworm main in the near future!? Regards, Christian

9 months, 2 weeks

3
2
0 0

cephfs - unable to create new subvolume

by karon karon

Hello, I recently use cephfs in version 17.2.6 I have a pool named "*data*" and a fs "*kube*" it was working fine until a few days ago, now i can no longer create a new subvolume*, *it gives me the following error: Error EINVAL: invalid value specified for ceph.dir.subvolume > here is the command used: ceph fs subvolume create kube newcsivol --pool_layout data > from what I understand it seems that it creates the subvolume but immediately puts it in the trash !? here is the log : 2023-06-23T08:30:53.307+0000 7f2b929d2700 0 log_channel(audit) log [DBG] : > from='client.86289 -' entity='client.admin' cmd=[{"prefix": "fs subvolume > create", "vol_name": "kube", "sub_name": "newcsivol", "group_name": "csi", > "pool_layout": "data", "target": ["mon-mgr", ""]}]: dispatch > 2023-06-23T08:30:53.307+0000 7f2b8a1d1700 0 [volumes INFO volumes.module] > Starting _cmd_fs_subvolume_create(group_name:csi, pool_layout:data, > prefix:fs subvolume create, sub_name:newcsivol, target:['mon-mgr', ''], > vol_name:kube) < "" > 2023-06-23T08:30:53.327+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.operations.versions.subvolume_v2] cleaning up subvolume with > path: newcsivol > 2023-06-23T08:30:53.331+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.operations.versions.subvolume_base] subvolume path > 'b'/volumes/csi/newcsivol'' moved to trashcan > 2023-06-23T08:30:53.331+0000 7f2b8a1d1700 0 [volumes INFO > volumes.fs.async_job] queuing job for volume 'kube' > 2023-06-23T08:30:53.335+0000 7f2b8a1d1700 0 [volumes INFO volumes.module] > Finishing _cmd_fs_subvolume_create(group_name:csi, pool_layout:data, > prefix:fs subvolume create, sub_name:newcsivol, target:['mon-mgr', ''], > vol_name:kube) < "" > 2023-06-23T08:30:53.335+0000 7f2b8a1d1700 -1 mgr.server reply reply (22) > Invalid argument invalid value specified for ceph.dir.subvolume > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.339+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.363+0000 7f2b461bf700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.383+0000 7f2b479c2700 -1 client.0 error registering > admin socket command: (17) File exists > 2023-06-23T08:30:53.507+0000 7f2b3ff33700 0 [prometheus INFO > cherrypy.access.139824530773776] 192.168.240.231 - - [23/Jun/2023:08:30:53] > "GET /metrics HTTP/1.1" 200 194558 "" "Prometheus/2.33.4" > 2023-06-23T08:30:54.219+0000 7f2b3ddaf700 0 [dashboard INFO request] [ > 172.29.2.142:33040] [GET] [200] [0.003s] [admin] [22.0B] > /api/prometheus/notifications > 2023-06-23T08:30:54.223+0000 7f2b929d2700 0 log_channel(audit) log [DBG] > : from='mon.0 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": > "json"}]: dispatch > 2023-06-23T08:30:54.227+0000 7f2b3a5a8700 0 [dashboard INFO request] [ > 172.29.2.142:49348] [GET] [200] [0.019s] [admin] [22.0B] /api/prometheus > 2023-06-23T08:30:54.227+0000 7f2b929d2700 0 log_channel(audit) log [DBG] > : from='mon.0 -' entity='mon.' cmd=[{"prefix": "balancer status", "format": > "json"}]: dispatch > 2023-06-23T08:30:54.231+0000 7f2b3d5ae700 0 [dashboard INFO request] [ > 172.29.2.142:39414] [GET] [200] [0.022s] [admin] [9.3K] > /api/prometheus/rules > 2023-06-23T08:30:54.275+0000 7f2ba39d4700 0 log_channel(cluster) log > [DBG] : pgmap v2116480: 145 pgs: 145 active+clean; 2.8 GiB data, 12 GiB > used, 1.5 TiB / 1.5 TiB avail; 5.5 KiB/s wr, 0 op/s > my fs info : # ceph fs ls > name: kube, metadata pool: metadata, data pools: [data ] > thank for your help best regards Karim

9 months, 2 weeks

3
3
0 0

quincy 17.2.6 - write performance continuously slowing down until OSD restart needed

by Nikola Ciprich

Hello dear CEPH users and developers, we're dealing with strange problems.. we're having 12 node alma linux 9 cluster, initially installed CEPH 15.2.16, then upgraded to 17.2.5. It's running bunch of KVM virtual machines accessing volumes using RBD. everything is working well, but there is strange and for us quite serious issue - speed of write operations (both sequential and random) is constantly degrading drastically to almost unusable numbers (in ~1week it drops from ~70k 4k writes/s from 1 VM to ~7k writes/s) When I restart all OSD daemons, numbers immediately return to normal.. volumes are stored on replicated pool of 4 replicas, on top of 7*12 = 84 INTEL SSDPE2KX080T8 NVMEs. I've updated cluster to 17.2.6 some time ago, but the problem persists. This is especially annoying in connection with https://tracker.ceph.com/issues/56896 as restarting OSDs is quite painfull when half of them crash.. I don't see anything suspicious, nodes load is quite low, no logs errors, network latency and throughput is OK too Anyone having simimar issue? I'd like to ask for hints on what should I check further.. we're running lots of 14.2.x and 15.2.x clusters, none showing similar issue, so I'm suspecting this is something related to quincy thanks a lot in advance with best regards nikola ciprich -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz -------------------------------------

9 months, 3 weeks

7
16
0 0

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards

by Christian Rohmann

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

9 months, 3 weeks

3
4
0 0

1 PG stucked in "active+undersized+degraded for long time

by siddhit.renake＠nxtgen.com

Hello All, Ceph version: 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable) Issue: 1 PG stucked in "active+undersized+degraded for long time Degraded data redundancy: 44800/8717052637 objects degraded (0.001%), 1 pg degraded, 1 pg undersized #ceph pg dump_stuck PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.28f0 active+undersized+degraded [2147483647,343,355,415,426,640,302,392,78,202,607] 343 [2147483647,343,355,415,426,640,302,392,78,202,607] 343 PG Query: #ceph pg 15.28f0 query { "state": "active+undersized+degraded", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 303362, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting_recovery_backfill": [ "78(8)", "202(9)", "302(6)", "343(1)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "info": { "pgid": "15.28f0s1", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598853, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "303161'598853", "reported_seq": "3594551", "reported_epoch": "303362", "state": "active+undersized+degraded", "last_fresh": "2023-06-20 19:03:59.135295", "last_change": "2023-06-20 15:11:12.569114", "last_active": "2023-06-20 19:03:59.135295", "last_peered": "2023-06-20 19:03:59.135295", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2023-06-20 15:11:12.569114", "last_became_peered": "2023-06-20 15:11:12.569114", "last_unstale": "2023-06-20 19:03:59.135295", "last_undegraded": "2023-06-20 15:11:10.430426", "last_fullsized": "2023-06-20 15:11:10.430154", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3028, "ondisk_log_size": 3028, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54989065178, "num_objects": 44800, "num_object_clones": 0, "num_object_copies": 492800, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44800, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44800, "num_whiteouts": 0, "num_read": 201078, "num_read_kb": 30408632, "num_write": 219335, "num_write_kb": 59459782, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44800 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "78(8)", "pgid": "15.28f0s8", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "202(9)", "pgid": "15.28f0s9", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "302(6)", "pgid": "15.28f0s6", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "355(2)", "pgid": "15.28f0s2", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "392(7)", "pgid": "15.28f0s7", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "415(3)", "pgid": "15.28f0s3", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "426(4)", "pgid": "15.28f0s4", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "607(10)", "pgid": "15.28f0s10", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "640(5)", "pgid": "15.28f0s5", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2023-06-20 15:11:10.430274", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "recovery_ops": [], "read_ops": [] } }, "scrub": { "scrubber.epoch_start": "0", "scrubber.active": false, "scrubber.state": "INACTIVE", "scrubber.start": "MIN", "scrubber.end": "MIN", "scrubber.max_end": "MIN", "scrubber.subset_last_update": "0'0", "scrubber.deep": false, "scrubber.waiting_on_whom": [] } }, { "name": "Started", "enter_time": "2023-06-20 15:11:09.294876" } ], "agent_state": {} } Require assistance to to make this PG active+clean

9 months, 3 weeks

3
4
0 0

Re: RBD with PWL cache shows poor performance compared to cache device

by Matthew Booth

On Wed, 28 Jun 2023 at 22:44, Ilya Dryomov <idryomov(a)redhat.com> wrote: >> ** TL;DR >> >> In testing, the write latency performance of a PWL-cache backed RBD >> disk was 2 orders of magnitude worse than the disk holding the PWL >> cache. >> >> ** Summary >> >> I was hoping that PWL cache might be a good solution to the problem of >> write latency requirements of etcd when running a kubernetes control >> plane on ceph. Etcd is extremely write latency sensitive and becomes >> unstable if write latency is too high. The etcd workload can be >> characterised by very small (~4k) writes with a queue depth of 1. >> Throughput, even on a busy system, is normally very low. As etcd is >> distributed and can safely handle the loss of un-flushed data from a >> single node, a local ssd PWL cache for etcd looked like an ideal >> solution. > > > Right, this is exactly the use case that the PWL cache is supposed to address. Good to know! >> My expectation was that adding a PWL cache on a local SSD to an >> RBD-backed would improve write latency to something approaching the >> write latency performance of the local SSD. However, in my testing >> adding a PWL cache to an rbd-backed VM increased write latency by >> approximately 4x over not using a PWL cache. This was over 100x more >> than the write latency performance of the underlying SSD. >> >> My expectation was based on the documentation here: >> https://docs.ceph.com/en/quincy/rbd/rbd-persistent-write-log-cache/ >> >> “The cache provides two different persistence modes. In >> persistent-on-write mode, the writes are completed only when they are >> persisted to the cache device and will be readable after a crash. In >> persistent-on-flush mode, the writes are completed as soon as it no >> longer needs the caller’s data buffer to complete the writes, but does >> not guarantee that writes will be readable after a crash. The data is >> persisted to the cache device when a flush request is received.” >> >> ** Method >> >> 2 systems, 1 running single-node Ceph Quincy (17.2.6), the other >> running libvirt and mounting a VM’s disk with librbd (also 17.2.6) >> from the first node. >> >> All performance testing is from the libvirt system. I tested write >> latency performance: >> >> * Inside the VM without a PWL cache >> * Of the PWL device directly from the host (direct to filesystem, no VM) >> * Inside the VM with a PWL cache >> >> I am testing with fio. Specifically I am running a containerised test, >> executed with: >> podman run --volume .:/var/lib/etcd:Z quay.io/openshift-scale/etcd-perf >> >> This container runs: >> fio --rw=write --ioengine=sync --fdatasync=1 >> --directory=/var/lib/etcd --size=100m --bs=8000 --name=etcd_perf >> --output-format=json --runtime=60 --time_based=1 >> >> And extracts sync.lat_ns.percentile["99.000000"] > > > Matthew, do you have the rest of the fio output captured? It would be interesting to see if it's just the 99th percentile that is bad or the PWL cache is worse in general. Sure. With PWL cache: https://paste.openstack.org/show/820504/ Without PWL cache: https://paste.openstack.org/show/b35e71zAwtYR2hjmSRtR/ With PWL cache, 'rbd_cache'=false: https://paste.openstack.org/show/byp8ZITPzb3r9bb06cPf/ >> ** Results >> >> All results were stable across multiple runs within a small margin of error. >> >> * rbd no cache: 1417216 ns >> * pwl cache device: 44288 ns >> * rbd with pwl cache: 5210112 ns >> >> Note that by adding a PWL cache we increase write latency by >> approximately 4x, which is more than 100x than the underlying device. >> >> ** Hardware >> >> 2 x Dell R640s, each with Xeon Silver 4216 CPU @ 2.10GHz and 192G RAM >> Storage under test: 2 x SAMSUNG MZ7KH480HAHQ0D3 SSDs attached to PERC >> H730P Mini (Embedded) >> >> OS installed on rotational disks >> >> N.B. Linux incorrectly detects these disks as rotational, which I >> assume relates to weird behaviour by the PERC controller. I remembered >> to manually correct this on the ‘client’ machine for the PWL cache, >> but at OSD configuration time ceph would have detected them as >> rotational. They are not rotational. >> >> ** Ceph Configuration >> >> CentOS Stream 9 >> >> # ceph version >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> Single node installation with cephadm. 2 OSDs, one on each SSD. >> 1 pool with size 2 >> >> ** Client Configuration >> >> Fedora 38 >> Librbd1-17.2.6-3.fc38.x86_64 >> >> PWL cache is XFS filesystem with 4k block size, matching the >> underlying device. The filesystem uses the whole block device. There >> is no other load on the system. >> >> ** RBD Configuration >> >> # rbd config image list libvirt-pool/pwl-test | grep cache >> rbd_cache true > > > I wonder if rbd_cache should have been set to false here to disable the default volatile cache. Other than that, I don't see anything obviously wrong with the configuration at first sight. I added some full output for this above. > > -- > Ilya > >> config >> rbd_cache_block_writes_upfront false >> config >> rbd_cache_max_dirty 25165824 >> config >> rbd_cache_max_dirty_age 1.000000 >> config >> rbd_cache_max_dirty_object 0 >> config >> rbd_cache_policy writeback >> pool >> rbd_cache_size 33554432 >> config >> rbd_cache_target_dirty 16777216 >> config >> rbd_cache_writethrough_until_flush true >> pool >> rbd_parent_cache_enabled false >> config >> rbd_persistent_cache_mode ssd >> pool >> rbd_persistent_cache_path /var/lib/libvirt/images/pwl >> pool >> rbd_persistent_cache_size 1073741824 >> config >> rbd_plugins pwl_cache >> pool >> >> # rbd status libvirt-pool/pwl-test >> Watchers: >> watcher=10.1.240.27:0/1406459716 client.14475 >> cookie=140282423200720 >> Persistent cache state: >> host: dell-r640-050 >> path: >> /var/lib/libvirt/images/pwl/rbd-pwl.libvirt-pool.37e947fd216b.pool >> size: 1 GiB >> mode: ssd >> stats_timestamp: Mon Jun 26 11:29:21 2023 >> present: true empty: false clean: true >> allocated: 180 MiB >> cached: 135 MiB >> dirty: 0 B >> free: 844 MiB >> hits_full: 1 / 0% >> hits_partial: 3 / 0% >> misses: 21952 >> hit_bytes: 6 KiB / 0% >> miss_bytes: 349 MiB -- Matthew Booth

9 months, 3 weeks

4
16
0 0

Re: Ceph Mgr/Dashboard Python depedencies: a new approach

by Casey Bodley

hi Ernesto and lists, > [1] https://github.com/ceph/ceph/pull/47501 are we planning to backport this to quincy so we can support centos 9 there? enabling that upgrade path on centos 9 was one of the conditions for dropping centos 8 support in reef, which i'm still keen to do if not, can we find another resolution to https://tracker.ceph.com/issues/58832? as i understand it, all of those python packages exist in centos 8. do we know why they were dropped for centos 9? have we looked into making those available in epel? (cc Ken and Kaleb) On Fri, Sep 2, 2022 at 12:01 PM Ernesto Puerta <epuertat(a)redhat.com> wrote: > > Hi Kevin, > >> >> Isn't this one of the reasons containers were pushed, so that the packaging isn't as big a deal? > > > Yes, but the Ceph community has a strong commitment to provide distro packages for those users who are not interested in moving to containers. > >> Is it the continued push to support lots of distros without using containers that is the problem? > > > If not a problem, it definitely makes it more challenging. Compiled components often sort this out by statically linking deps whose packages are not widely available in distros. The approach we're proposing here would be the closest equivalent to static linking for interpreted code (bundling). > > Thanks for sharing your questions! > > Kind regards, > Ernesto > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

9 months, 4 weeks

4
10
0 0

2024

2023

2022

2021

2020

2019

ceph-users June 2023