- ceph-users - lists.ceph.io

100 % Genuine Call Girls In Mahipalpur Delhi 8448079011 Escort Service In Delhi
by Vicky Singh 11 Jun '24

11 Jun '24

Call Vicky 8448079011 Book Call Girls In Mahipalpur Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr

1 0

Announcing go-ceph v0.28.0
by Anoop C S 11 Jun '24

11 Jun '24

Hi, We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.28.0 More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S.

1 0

[community] Announcing Ceph Days London 2024 - July 17th
by Danny Abu-Kalam (BLOOMBERG/ LONDON) 11 Jun '24

11 Jun '24

Hi All, Ceph Days is coming to London on July 17th 2024, co-hosted by IBM and Canonical. We're planning a full day of talks, lighting talks, and panel discussions all relating to various parts of the Ceph project. The CFP is now open and will close on the 30th June, so get your talks in ASAP! Event registration is also open now and space is limited, so book now to reserve your seat! Links to both registration and CFP are here: https://ceph.io/en/community/events/2024/ceph-days-london/ Looking forward to seeing you there! Thanks, Danny

1 0

100 % Genuine Call Girls In Paharganj Delhi 8448079011 Escort In Delhi
by Vicky Singh 11 Jun '24

11 Jun '24

Call Vicky 8448079011 Book Call Girls In Paharganj Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr

1 0

degraded objects when setting different CRUSH rule on a pool, why?
by Stefan Kooman 11 Jun '24

11 Jun '24

Hi, TL;DR: Selecting a different CRUSH rule (stretch_rule, no device class) for pool SSD results in degraded objects (unexpected) and misplaced objects (expected). Why would Ceph drop up to two healthy copies? Consider this two data center cluster: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.78400 root default -10 0.39200 datacenter DC1 -3 0.39200 host pve1 0 hdd 0.09799 osd.0 up 1.00000 1.00000 1 hdd 0.09799 osd.1 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -11 0.39200 datacenter DC2 -5 0.39200 host pve2 2 hdd 0.09799 osd.2 up 1.00000 1.00000 3 hdd 0.09799 osd.3 up 1.00000 1.00000 6 ssd 0.09799 osd.6 up 1.00000 1.00000 7 ssd 0.09799 osd.7 up 1.00000 1.00000 Pools available: device_health_metrics HDD SSD Let's focus on SSD for now. Crush rule in use for SSD pool: rule SSD { id 2 type replicated min_size 1 max_size 10 step take default class ssd step choose firstn 0 type host step chooseleaf firstn 0 type osd step emit } SSD pool replication settings: min_size=2, size=4 The new stretch rule to use: rule stretch_rule { id 3 type replicated min_size 1 max_size 10 step take default step take DC1 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit step take default step take DC2 step choose firstn 0 type host step chooseleaf firstn 2 type osd step emit } ceph pg ls for pool 3 (SSD): > 3.0 184 0 0 0 738070528 0 0 7458 active+clean 9m 926'7458 951:18695 [7,6,5,4]p7 [7,6,5,4]p7 2024-06-05T08:52:51.951936+0200 2024-06-05T08:52:51.951936+0200 > 3.1 169 0 0 0 667156655 0 0 3242 active+clean 18m 926'3242 951:14151 [4,5,6,7]p4 [4,5,6,7]p4 2024-06-05T08:54:42.948682+0200 2024-06-05T08:54:42.948682+0200 > 3.2 221 0 0 0 885641968 0 0 6989 active+clean 9m 926'6989 951:17645 [4,5,7,6]p4 [4,5,7,6]p4 2024-06-05T08:53:27.981787+0200 2024-06-05T08:53:27.981787+0200 > 3.3 180 0 0 0 716509184 0 0 3194 active+clean 9m 926'3194 951:14191 [4,5,6,7]p4 [4,5,6,7]p4 2024-06-05T08:54:29.584216+0200 2024-06-05T08:54:29.584216+0200 > 3.4 189 0 0 0 754417698 137 8 3616 active+clean 9m 926'3616 951:19245 [6,7,4,5]p6 [6,7,4,5]p6 2024-06-05T08:54:02.307323+0200 2024-06-05T08:54:02.307323+0200 > 3.5 188 0 0 0 742543377 0 0 5992 active+clean 9m 926'5992 951:18862 [6,7,5,4]p6 [6,7,5,4]p6 2024-06-05T08:53:09.483136+0200 2024-06-05T08:53:09.483136+0200 > 3.6 191 0 0 0 769482752 150 16 6810 active+clean 9m 926'6810 951:30043 [7,6,4,5]p7 [7,6,4,5]p7 2024-06-05T08:53:46.646517+0200 2024-06-05T08:53:46.646517+0200 > 3.7 170 0 0 0 681587379 0 0 10081 active+clean 9m 926'21581 951:28473 [4,5,7,6]p4 [4,5,7,6]p4 2024-06-05T08:54:16.047967+0200 2024-06-05T08:54:16.047967+0200 ceph pg ls when the new crush rule is selected for pool SSD: > 3.0 184 372 186 0 738070528 0 0 7458 active+recovery_wait+undersized+degraded+remapped 14s 926'7458 955:18688 [1,0,3,7]p1 [4,7]p4 2024-06-05T08:52:51.951936+0200 2024-06-05T08:52:51.951936+0200 > 3.1 169 0 0 0 667156655 0 0 3242 active+clean 19m 926'3242 954:14154 [4,5,6,7]p4 [4,5,6,7]p4 2024-06-05T08:54:42.948682+0200 2024-06-05T08:54:42.948682+0200 > 3.2 221 444 0 0 885641968 0 0 6989 active+recovery_wait+undersized+degraded+remapped 14s 926'6989 955:17657 [4,5,3,2]p4 [4,5]p4 2024-06-05T08:53:27.981787+0200 2024-06-05T08:53:27.981787+0200 > 3.3 180 0 540 0 716509184 0 0 3194 active+recovering+undersized+degraded+remapped 15s 926'3194 955:14204 [4,0,2,3]p4 [4,0]p4 2024-06-05T08:54:29.584216+0200 2024-06-05T08:54:29.584216+0200 > 3.4 189 378 0 0 754417698 0 0 3616 active+recovery_wait+undersized+degraded+remapped 15s 926'3616 955:19220 [1,4,6,2]p1 [4,6]p4 2024-06-05T08:54:02.307323+0200 2024-06-05T08:54:02.307323+0200 > 3.5 188 189 0 0 742543377 0 0 5992 active+recovery_wait+undersized+degraded+remapped 15s 926'5992 955:18845 [5,4,6,3]p5 [5,4,6]p5 2024-06-05T08:53:09.483136+0200 2024-06-05T08:53:09.483136+0200 > 3.6 191 390 195 0 769482752 0 0 6810 active+recovery_wait+undersized+degraded+remapped 14s 926'6810 955:30016 [0,1,2,7]p0 [4,7]p4 2024-06-05T08:53:46.646517+0200 2024-06-05T08:53:46.646517+0200 > 3.7 170 0 170 0 681587379 0 0 10081 active+remapped+backfill_wait 14s 926'21581 955:28486 [4,5,3,6]p4 [4,5,6,7]p4 2024-06-05T08:54:16.047967+0200 2024-06-05T08:54:16.047967+0200 So CRUSH is able to find a suitable mapping just fine, but somehow Ceph decides to drop up to two healthy copies from its acting se,t and I do not understand why. I would expect only misplaced objects at this point. Ceph version is 15.2.17. Latest CRUSH tunables (ceph osd crush tunables optimal). Do I miss something obvious here? If so, would you please point it out to me? :D. Gr. Stefan

2 1

Ceph RBD, MySQL write IOPs - what is possible?
by Mark Lehrer 11 Jun '24

11 Jun '24

I've been using MySQL on Ceph forever, and have been down this road before but it's been a couple of years so I wanted to see if there is anything new here. So the TL:DR version of this email - is there a good way to improve 16K write IOPs with a small number of threads? The OSDs themselves are idle so is this just a weakness in the algorithms or do ceph clients need some profiling? Or "other"? Basically, this is one of the worst possible Ceph workloads so it is fun to try to push the limits. I also happen have a MySQL instance that is reaching the write IOPs limit so this is also a last-ditch effort to keep it on Ceph. This cluster is as straightforward as it gets... 6 servers with 10 SSDs each, 100 Gb networking. I'm using size=3. During operations, the OSDs are more or less idle so I don't suspect any hardware limitations. MySQL has no parallelism so the number of threads and effective queue depth stay pretty low. Therefore, as a proxy for MySQL I use rados bench with 16K writes and 8 threads. The RBD actually gets about 2x this level - still not so great. I get about 2000 IOPs with this test: # rados bench -p volumes 10 write -t 8 -b 16K hints = 1 Maintaining 8 concurrent writes of 16384 bytes to objects of size 16384 for up to 10 seconds or 0 objects Object prefix: benchmark_data_fstosinfra-5_3652583 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 8 2050 2042 31.9004 31.9062 0.00247633 0.00390848 2 8 4306 4298 33.5728 35.25 0.00278488 0.00371784 3 8 6607 6599 34.3645 35.9531 0.00277546 0.00363139 4 7 8951 8944 34.9323 36.6406 0.00414908 0.00357249 5 8 11292 11284 35.257 36.5625 0.00291434 0.00353997 6 8 13588 13580 35.3588 35.875 0.00306094 0.00353084 7 7 15933 15926 35.5432 36.6562 0.00308388 0.0035123 8 8 18361 18353 35.8399 37.9219 0.00314996 0.00348327 9 8 20629 20621 35.7947 35.4375 0.00352998 0.0034877 10 5 23010 23005 35.9397 37.25 0.00395566 0.00347376 Total time run: 10.003 Total writes made: 23010 Write size: 16384 Object size: 16384 Bandwidth (MB/sec): 35.9423 Stddev Bandwidth: 1.63433 Max bandwidth (MB/sec): 37.9219 Min bandwidth (MB/sec): 31.9062 Average IOPS: 2300 Stddev IOPS: 104.597 Max IOPS: 2427 Min IOPS: 2042 Average Latency(s): 0.0034737 Stddev Latency(s): 0.00163661 Max latency(s): 0.115932 Min latency(s): 0.00179735 Cleaning up (deleting benchmark objects) Removed 23010 objects Clean up completed and total clean up time :7.44664 Are there any good options to improve this? It seems like the client side is the bottleneck since the OSD servers are at like 15% utilization. Thanks, Mark

6 12

About disk disk iops and ultil peak
by Phong Tran Thanh 11 Jun '24

11 Jun '24

Hi ceph user! I am encountering a problem with IOPS and disk utilization of OSD. Sometimes, my disk peaks in IOPS and utilization become too high, which affects my cluster and causes slow operations to appear in the logs. 6/6/24 9:51:46 AM[WRN]Health check update: 0 slow ops, oldest one blocked for 36 sec, osd.268 has slow ops (SLOW_OPS) 6/6/24 9:51:37 AM[WRN]Health check update: 0 slow ops, oldest one blocked for 31 sec, osd.268 has slow ops (SLOW_OPS) [image: image.png] This is config tu reduce it, but its not resolve my problem global advanced osd_mclock_profile custom global advanced osd_mclock_scheduler_background_best_effort_lim 0.100000 global advanced osd_mclock_scheduler_background_best_effort_res 0.100000 global advanced osd_mclock_scheduler_background_best_effort_wgt 1 global advanced osd_mclock_scheduler_background_recovery_lim 0.100000 global advanced osd_mclock_scheduler_background_recovery_res 0.100000 global advanced osd_mclock_scheduler_background_recovery_wgt 1 global advanced osd_mclock_scheduler_client_lim 0.400000 global advanced osd_mclock_scheduler_client_res 0.400000 global advanced osd_mclock_scheduler_client_wgt 4 Hope someone can help me Thanks so much! -- Email: tranphong079(a)gmail.com Skype: tranphong079

2 3

multipart uploads in reef 18.2.2
by Christopher Durham 10 Jun '24

10 Jun '24

We have a reef 18.2.2 cluster with 6 radosgw servers on Rocky 8.9. The radosgw servers are not fronted by anything like HAProxy as the clients connect directly to a DNS name via a round-robin DNS. Each of the radosgw servers have a certificate using SAN entries for all 6 radosgw servers as well as the primary DNS name. This has worked wonderfully for four years in distributing our load. This is an RPM install. Since our update to 18.2.2, we have had some issues with a specific set of clients (spark). They *always* create multipart uploads, both before and after the ceph update from 17.2.6 to 18.2.2, even when there is a single multipart part less than what would otherwise be the threshold. The single multipart part is the norm. This works fine for a time after restarting the radosgws. This is what happens: 1. The single PUT of a single multipart part with a given uploadId 2. The single POST to otherwise complete the multipart, with the same uploadId used in the PUT But when the problem occurs, the PUT works, but The POST fails with a "500 302" error, and the client will continue to try this, eventually returning with a reported error of: "Status code; 500 Error-Code: Internal error, ... The multipart completion is already in progress."While this error makes sense when multiple POSTs happen for the completion of the multipart upload, the first one should not fail. Sometimes the PUT and the POST happen from different clients or to different servers, even when things are working. But, when things begin to fail, just before the failed POST in the radosgw logs, I get: s3:complete_multipart failed to acquire lock Then the multiple "500 302" errors happen. Note that after the multiple "500 302" errors, the spark client DELETEs the object, terminating the multipart upload (no 'leftover' multipart uploads). There are about 50-60 multiparts happening when this occurs, and as time goes on, I get less and less successful multipart uploads, and eventually I have to restart the rados gateways. There are some other minor GETs and PUTs happening as well unrelated to spark When this happens, we get multiple sockets on the radosgw servers that are in CLOSE-WAIT state. While I cannot prove it, these appear to be related to the issue at hand as the CLOSE-WAIT states are only coming from IPs associated withthe spark jobs. After I restart the radosgw servers, things are good for a time, and all CLOSE-WAITs disappear until the problem starts. I have set: rgw thread pool size = 1024rgw max concurrent requests = 2048 and restarted both the mons and the radosgw servers to no avail. The problem takes maybe 6 hours to start No object versioning is in effect. Any ideas would be appreciated, thanks. -Chris

1 0

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

Ceph Leadership Team Weekly Minutes 2024-06-10
by Casey Bodley 10 Jun '24

10 Jun '24

# quincy now past estimated 2024-06-01 end-of-life will 17.2.8 be the last point release? maybe not, depending on timing # centos 8 eol * Casey tried to summarize the fallout in https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/H7I4Q4RAIT6UZQNPPZ… * c8 builds were disabled with https://github.com/ceph/ceph-build/pull/2235 * Patrick points out that we'll no longer be able to test upgrades from octopus/pacific to quincy/reef ## reef 18.2.3 validation delayed * need to remove references to centos 8 in the qa suites ** reef backport started in https://github.com/ceph/ceph/pull/57932 ** still blocked on fs upgrade suite (for main/squid also) * the plan is to move upgrade suites to cephadm where possible ** concern about lack of package-based upgrade testing ## alternatives to centos for container base distro? to be discussed in public on the mailing list # Cephalocon program committee - volunteers? * Josh Durgin * Patrick Donnelly * Joseph Mundackal (first time - so happy to help in anyway i can) * Matt Benjamin (talk review) # Crimson Tech lead change * congrats to Matan Breizman # docs backports to Reef and to Quincy fail * Zac would like to learn about the doc build infrastructure in order to fix issues like this

1 0