- ceph-users - lists.ceph.io

Deleting millions of objects
by Rok Jaklič 18 May '23

18 May '23

Hi, I would like to delete millions of objects in RGW instance with: mc rm --recursive --force ceph/archive/veeam but it seems it allows only 1000 (or 1002 exactly) removals per command. How can I delete/remove all objects with some prefix? Kind regards, Rok

7 9

Discussion thread for Known Pacific Performance Regressions
by Mark Nelson 17 May '23

17 May '23

Hi Everyone, This email was originally posted to dev(a)ceph.io, but Marc mentioned that he thought this would be useful to post on the user list so I'm re-posting here as well. David Orman mentioned in the CLT meeting this morning that there are a number of people on the mailing list asking about performance regressions in Pacific+ vs older releases. I want to document a couple of the bigger ones that we know about for the community's benefit. I want to be clear that Pacific does have a number of performance improvements over previous releases, and we do have tests showing improvement relative to nautilus (especially RBD on NVMe drives). Some of these regressions are going to have a bigger effect for some users than others. Having said that, let's get into them. ********** Regression #1: RocksDB Log File Recycling ********** Effects: More metadata updates to the underlying FS, higher write-amplification (Observed by Digital Ocean), Slower performance especially when the WAL device is saturated. When bluestore was created back in 2015 Sage implemented an optimization in RocksDB that allowed WAL log files to be recycled. The idea is that instead of deleting logs when they are flushed, rocksdb can simply reuse them. The benefit here is that it allows records to be written and fadatasync can be called without touching the inode for every IO. Sage did a pretty good job of explaining the benefit in the PR available here: https://github.com/facebook/rocksdb/pull/746 After much discussion, that PR was merged and received a couple of bug fixes over the years: Locking bug fix from Somnath back in 2016: https://github.com/facebook/rocksdb/pull/1313 Another bug fix from ajkr in 2020: https://github.com/facebook/rocksdb/pull/5900 In 2020, the RocksDB folks discovered there is a fundamental flaw in the way that the original PR works. It turns out that the feature to recycle log files is incompatible with RocksDB's kPointInTimeRecovery, kAbsoluteConsistency, and kTolerateCorruptedTailRecords recovery modes. One of the later PR's included a very good and concise description of the problem: "The two features are naturally incompatible. WAL recycling expects the recovery to succeed upon encountering a corrupt record at the point where new data ends and recycled data remains at the tail. However, WALRecoveryMode::kTolerateCorruptedTailRecords must fail upon encountering any such corrupt record, as it cannot differentiate between this and a real corruption, which would cause committed updates to be truncated." More background discussion on the RocksDB side available in these PRs and comments: https://github.com/facebook/rocksdb/pull/6351 https://github.com/facebook/rocksdb/pull/6351#issuecomment-672838284 https://github.com/facebook/rocksdb/pull/7252 https://github.com/facebook/rocksdb/pull/7271 On the Ceph side, there was a PR to try to re-enable the old behavior which we rejected as unsafe based on the analysis by the RocksDB folks (which we agree with): https://github.com/ceph/ceph/pull/36579 Sage also commented about a potential way forward: https://github.com/ceph/ceph/pull/36579#issuecomment-870884583 "tbh I think the best approach would be to create a new WAL file format that (1) is 4k block aligned and (2) has a header for each block that indicates the generation # for that log file (so we can see whether what we read is from a previous pass or corruption). That would be a fair bit of effort, though." On a side note, Igor tried to also disable WAL file recycling as a backport to Octopus but was thwarted by a BlueFS bug. That PR was eventually reverted leaving the old (dangerous!) behavior being left in place: https://github.com/ceph/ceph/pull/45040 https://github.com/ceph/ceph/pull/47053 The gist of it is that releases of Ceph older than Pacific are benefiting from the speed improvement of log file recycling but may be vulnerable to the issue as described above. This is likely one of the more impactful regressions that people upgrading to Pacific or later releases are seeing. Josh Baergen from Digital Ocean followed up that there is a slew of additional information on this issue in the following tracker as well: https://tracker.ceph.com/issues/58530 ********** Regression #1 Potential Fixes *********** Josh Baergen also mentioned that the write-amplification effect that was observed due to this issue is mitigated in by https://github.com/ceph/ceph/pull/48915 which was merged into 16.2.11 back in December. That however does not improve write IOPS amplification. Beyond that, we could follow Sage's idea and try to implement a new WAL file format. The risks here are that it could be a lot of work and we don't know if there is really any appetite on the RocksDB side to merge something like this upstream. My personal take is that we're already kind of abusing the RocksDB WAL for short lived PG log updates and I'm not thrilled about trying to add further code into RocksDB to try and support our use cases (though there is benefit here that goes beyond Ceph). We already maintain a custom version of RocksDB's LRU cache in our code to tie into our memory autotuning system but it would be really nice to avoid custom code like that in the future. One alternative: Igor Fedetov implemented a prototype WAL inside bluestore itself and we saw very good initial results from it with the RocksDB WAL disabled. These can be seen on slide 24 of my performance deck from Cephalocon 2023: https://www.linkedin.com/in/markhpc/overlay/experience/2113859303/multiple-… If Igor (or others) want to continue this work, I personally would be in favor of trying to move the WAL into Bluestore itself. I suspect we can make better decisions about PG log life cycles and have better BlueFS integration than what RocksDB provides us. Igor probably has a better idea of the pitfalls here though so I think we should hear out his thoughts on whether this is the right path forward. Igor also mentioned that he is continuing to work on his Bluestore WAL prototype with promising results, but that PG Log will (as expected) likely require a different solution that looks more like a specialized ring buffer. I think moving the WAL out of RocksDB is a good step toward that eventual goal. ********** Regression #2: (re-)Enabling BlueFS Buffered IO ********** Effects: Works around unexpected readahead behavior in RocksDB by utilizing underlying kernel page cache. Hurts write performance on fast devices. We're stuck between a bit of a rock and a hard place here. Over the years we have sea-sawed back and forth regarding when we should or should not use buffered IO: https://github.com/ceph/ceph/pull/11012 https://github.com/ceph/ceph/pull/11059 https://github.com/ceph/ceph/pull/18172 https://github.com/ceph/ceph/pull/20542 https://github.com/ceph/ceph/pull/34224 https://github.com/ceph/ceph/pull/38044 <-- lots of discussion here The gist of it is that there are upsides and downside to having bluefs_buffered_io=true. Direct IO is faster in some scenarios, especially more recent write tests on NVMe drives. The trade off is that RocksDB really seems to benefit from kernel buffer cache and there are other scenarios where bluefs_buffered_io is a big win. 2 years ago Adam and I did a walkthrough of the RocksDB code to try to understand the behavior regarding RocksDB readahead and we couldn't understand why it was re-reading data from the file system so often (or in the case of buffered IO the page cache!). I wrote up our walkthrough of the code here: https://github.com/ceph/ceph/pull/38044#issuecomment-790157415 ********** Regression #2 Potential Fixes ********** In a recent discussion with Mark Callaghan (of MyRocks/RocksDB performance tuning fame), he pointed out that RocksDB has an option to pre-populate the block cache with the data from SSTs created by memtable flush and that might help when O_DIRECT is used: https://github.com/facebook/rocksdb/blob/main/include/rocksdb/table.h#L600 We may want to experiment to see if this helps keep the block cache pre-populated after compaction and avoid (re)reads from the disk during iteration. We also might want to revisit this topic in general with the compact on iteration feature that was recently added and backported to pacific in 16.2.13. I'm still a little concerned however that we were seeing repeated overlapping reads for the same ranges during iteration that I would have expected to be cached by RocksDB on a previous read. Ultimately I think many of us would prefer to move entirely to directIO but there's more work to do to figure this one out. Josh Baergen provided further advice here: They have had good luck enabling buffered IO for rgw bucket index OSDs and disabling it everywhere else. This assumes that bucket indexes are on their own dedicated OSDs though, and personally I am a bit wary of hitting slow cases in RocksDB even on "regular" OSDs, but this might be something to consider as they've had good luck with this configuration for over a year. ********** Regression #3: RadosGW Coroutine and Request Timeout Changes ********** Effects: Higher RadosGW CPU usage, lower performance, especially for small object workloads Back when Pacific was released it was observed that RadosGW was showing much higher CPU usage and lower performance vs Nautilus for small (4KB) objects. It's likely that larger objects may be affected, though to a lesser degree A git bisection was performed and the results are summarized in the introduction section of the folliowing RGW performance analysis blog post here: https://ceph.io/en/news/blog/2023/reef-freeze-rgw-performance/ The bisection uncovered two primary PRs that were causing performance regression: https://github.com/ceph/ceph/pull/31580 https://github.com/ceph/ceph/pull/35355 The good news is that once those PRs were identified, the RGW team started working to improve things, especially for #35355: https://github.com/ceph/ceph/pull/43761 <-- Fixes issues introduced in #35355, backported to Pacific in 2022 *********** Regression #3 Potential Fixes ********** Quincy (and due to the backport likely Pacific) is showing significantly better behavior in recent tests due to PR #43761. The effects of #31580 are still present, but are considered a necessary trade-off. Other improvements since then may be helping, but we'll need to continue to make up the difference in other areas and start really investigating where we are spending cycles/time, especially in Reef. ********** Regression #4: Gradually slowing down OSDs ********** Effects: Significant slowdown after 1-2 weeks of OSD runtime Igor Fedetov pointed this one out in discussion earlier today: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/OXAUOK7CQXW… This one is pretty new and there is not much there yet other than perhaps low memory (and cache?) usage despite regular IO workload. Onode misses can absolutely cause performance degradation, but it's not clear yet whether this is memory related issue or something else. More investigation needed. Hopefully we'll get perf data from the users who encountered it to help diagnose what's going on here. ********** Conclusion ********** There may be other performance issues that I'm not remembering, but these are the big ones I can think of off the top of my head at the moment. Hopefully this helps clarify what's going on if people are seeing a regression, what to look for, and if they are hitting it, the why behind it. Thanks, Mark -- Best Regards, Mark Nelson Head of R&D (USA) Clyso GmbH p: +49 89 21552391 12 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: mark.nelson(a)clyso.com We are hiring: https://www.clyso.com/jobs/

2 2

Re: Enable Centralized Logging in Dashboard.
by Vladimir Brik 17 May '23

17 May '23

How do I create a user name and password that I could use to log in to grafana? Vlad On 11/16/22 08:42, E Taka wrote: > Thank you, Nizam. I wasn't aware that the Dashboard login is not the same > as the grafana login. Now I have accass to the logfiles. > > Am Mi., 16. Nov. 2022 um 15:06 Uhr schrieb Nizamudeen A <nia(a)redhat.com>: > >> Hi, >> >> Did you login to the grafana dashboard? For centralized logging you'll >> need to login to the >> grafana using your grafana username and password. If you do that and >> refresh the dashboard, >> I think the Loki page should be visible from the Daemon Logs page. >> >> Regards, >> Nizam >> >> On Wed, Nov 16, 2022 at 7:31 PM E Taka <0etaka0(a)gmail.com> wrote: >> >>> Ceph: 17.2.5, dockerized with Ubuntu 20.04 >>> >>> Hi all, >>> >>> I try to enable the Centralized Logging in Dashboard as described in >>> >>> >>> https://docs.ceph.com/en/quincy/cephadm/services/monitoring/#cephadm-monito… >>> >>> Logging inti files is enabled: >>> ceph config set global log_to_file true >>> ceph config set global mon_cluster_log_to_file true >>> >>> Loki is deployed at one host, promtail on every host: >>> >>> >>> service_type: loki >>> service_name: loki >>> placement: >>> hosts: >>> - ceph00 >>> --- >>> service_type: promtail >>> service_name: promtail >>> placement: >>> host_pattern: '*' >>> >>> >>> After applying the YAML above the log messages in »ceph -W cephadm« look >>> good (deploying loki+promtail and reconfiguring grafana). But the >>> Dashboard >>> "Cluster → Logs → Daemon Logs" just shows a standard grafana page without >>> any buttons for the Ceph Cluster. Its URL is https://ceph00. >>> >>> [mydomain]:3000/explore?orgId=1&left=["now-1h","now","Loki",{"refId":"A"}]&kiosk. >>> >>> Did I miss something for the centralized Logging in the Dashboard? >>> >>> Thank! >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

new install or change default registry to private registry
by farhad kh 17 May '23

17 May '23

i try deploy cluster from private registry and used this command {cephadm bootstrap ---mon-ip 10.10.128.68 --registry-url my.registry.xo --registry-username myuser1 --registry-password mypassword1 --dashboard-password-noupdate --initial-dashboard-password P@ssw0rd } even i changed section Default container images in cephadm script # Default container images ----------------------------------------------------- DEFAULT_IMAGE = ' my.registry.xo/ceph/ceph:v17' DEFAULT_IMAGE_IS_MASTER = False DEFAULT_IMAGE_RELEASE = 'quincy' DEFAULT_PROMETHEUS_IMAGE = ' my.registry.xo/ceph/prometheus:v2.33.4' DEFAULT_LOKI_IMAGE = ' my.registry.xo/ceph/loki:2.4.0' DEFAULT_PROMTAIL_IMAGE = my.registry.xo/ceph/promtail:2.4.0' DEFAULT_NODE_EXPORTER_IMAGE = ' my.registry.xo/ceph/node-exporter:v1.3.1' DEFAULT_ALERT_MANAGER_IMAGE = ' my.registry.xo/ceph/alertmanager:v0.23.0' DEFAULT_GRAFANA_IMAGE = ' my.registry.xo/ceph/ceph-grafana:8.3.5' DEFAULT_HAPROXY_IMAGE = ' my.registry.xo/ceph/haproxy:2.3' DEFAULT_KEEPALIVED_IMAGE = ' my.registry.xo/ceph//keepalived:2.1.5' DEFAULT_SNMP_GATEWAY_IMAGE = ' my.registry.xo/ceph/ snmp-notifier:v1.2.1' DEFAULT_REGISTRY = ' my.registry.xo' # normalize unqualified digests to this # ------------------------------------------------------------------------------ to my private registy but when i deploy new cluster cephadm try pull prometheus , node-exporter and ceph-grafana from quay.io . Why does it happen and what should I do to change it? Even the default registry is not changed #ceph config get mgr mgr/cephadm/default_registry docker.io Can someone guide me, what should I do?

2 1

how to enable multisite resharding feature?
by Alexander Mamonov 17 May '23

17 May '23

https://docs.ceph.com/en/latest/radosgw/multisite/#feature-resharding When I try this I get: root@ceph-m-02:~# radosgw-admin zone modify --rgw-zone=sel --enable-feature=resharding ERROR: invalid flag --enable-feature=resharding root@ceph-m-02:~# ceph version ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)

2 1

CEPH Version choice
by Tino Todino 17 May '23

17 May '23

Hi all, I've been reading through this email list for a while now, but one thing that I'm curious about is why a lot of installations out there aren't upgraded to the latest version of CEPH (Quincy). What are the main reasons for not upgrading to the latest and greatest? Thanks. Tino This E-mail is intended solely for the person or organisation to which it is addressed. It may contain privileged or confidential information and, if you are not the intended recipient, you must not copy, distribute or take any action in reliance upon it. Any views or opinions presented are solely those of the author and do not necessarily represent those of Marlan Maritime Technologies Ltd. If you have received this E-mail in error, please notify us as soon as possible and delete it from your computer. Marlan Maritime Technologies Ltd Registered in England & Wales 323 Mariners House, Norfolk Street, Liverpool. L1 0BG Company No. 08492427.

10 14

Best practice for expanding Ceph cluster
by huxiaoyu＠horebdata.cn 17 May '23

17 May '23

Dear Ceph folks, I am writing to ask for advice on best practice of expanding ceph cluster. We are running an 8-node Ceph cluster and RGW, and would like to add another 10 node, each of which have 10x 12TB HDD. The current 8-node has ca. 400TB user data. I am wondering whether to add 10 nodes at one shot and let the cluster to rebalance, or divide into 5 steps, each of which add 2 nodes and rebalance step by step? I do not know what would be the advantages or disadvantages with the one shot scheme vs 5 bataches of adding 2 nodes step-by-step. Any suggestions, experience sharing or advice are highly appreciated. thanks a lot in advance, Samuel huxiaoyu(a)horebdata.cn

4 6

CFP NOW OPEN: Ceph Days Vancouver Co-located with OpenInfra Summit
by Mike Perez 17 May '23

17 May '23

Hi everyone, I invite you to Ceph Days Vancouver on June 15, co-located with the OpenInfra Summit. Ceph Days are one-day events dedicated to multiple breakout and BoF sessions with a wide range of topics around Ceph. You can receive a nice discount to attend Ceph Days and the OpenInfra Summit if you register before May 12! https://ceph.io/en/community/events/2023/ceph-days-vancouver/ Our Call for proposals is open until May 17, and we have the following suggested topics: - Ceph operations, management, and development - New and proposed Ceph features, development status - Ceph development roadmap - Best practices - Ceph use-cases, solution architectures, and user experiences - Ceph performance and optimization - Platform Integrations - Kubernetes, OpenShift - OpenStack (Cinder, Manila, etc.) - Spark - Multi-site and multi-cluster data services - Persistent memory, ZNS SSDs, SMR HDDs, DPUs, and other new hardware technologies - Storage management, monitoring, and deployment automation - Experiences deploying and operating Ceph in production and/or at scale - Small-scale or edge deployments - Long-term, archival storage - Data compression, deduplication, and storage optimization - Developer processes, tools, challenges - Ceph testing infrastructure, tools - Ceph community issues, outreach, and project governance - Ceph documentation, training, and learner experience CFP: https://survey.zohopublic.com/zs/TVCCCQ See you all there soon! -- Mike Perez Community Manager Ceph Foundation

1 1

FW: Public Access URL returns "NoSuchBucket" when rgw_swift_account_in_url is True
by Beaman, Joshua 17 May '23

17 May '23

Thought I might forward this to the users list in case anyone else is experiencing or knows how to resolve. Thank you, Josh Beaman From: Beaman, Joshua <Joshua_Beaman(a)comcast.com> Date: Tuesday, May 16, 2023 at 3:23 PM To: dev(a)ceph.io <dev(a)ceph.io> Subject: Public Access URL returns "NoSuchBucket" when rgw_swift_account_in_url is True We have a continuing problem with Openstack Swift Public URLs returning “NoSuchBucket” even though the S3 equivalent URL works fine. This was not a problem in our Nautilus 14.2.22 clusters, but after upgrading to 16.2.9 or 16.2.10, this regression bug has returned. All 13 of our prod clusters were upgraded to 16.2.11 and still experience the problem, and now I’ve upgraded 1 prod cluster to Quincy/17.2.6 and it is still present. I’ve submitted the following bug report: https://tracker.ceph.com/issues/58019 Thanks in advance, Josh Beaman

1 0

Bucket-level redirect_zone
by Yixin Jin 16 May '23

16 May '23

Hi folks, I created a feature request ticket to call for bucket-level redirect_zone (https://tracker.ceph.com/issues/61199) which is basically an extension from zone-level redirect_zone. I found it helpful in realizing CopyObject with (x-copy-source) in multisite environments where bucket content don't exist in all zones. This feature is similar to what Matt Benjamin suggested about the concept of "bucket residence". In my own prototyping affect, redirecting feature at bucket level is fairly straightforward. Making use of it with CopyObject is trickier because I haven't found a good way to get the source object policy when RGWCopyObj::verify_permission() is called. Anyway, I will continue to explore this idea and hope to get more support of it. Thanks, Yixin

2 1