- ceph-users - lists.ceph.io

Recover UUID from a partition
by Szabo, Istvan (Agoda) 29 May '20

29 May '20

Hi, Is there a way to recover UUID from the partition? Someone mapped in fstab to /sd* not UUID and all the metadata is gone. The data is there , just can't mount it to access the data. Any idea how to get it back or determine what was it before? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Ceph Tech Talk: What's New In Octopus
by Mike Perez 29 May '20

29 May '20

Hi everyone, We had a Ceph Tech Talk on May 28th at 17:00 UTC from Josh Durgin and Lenz Grimmer on a summary of the new features and enhancements with the new Octopus release. You can watch the recording here: https://www.youtube.com/watch?list=PLrBUGiINAakM36YJiTT0qYepZTVncFDdc&v=UGU… You can see our archive here: https://ceph.io/ceph-tech-talks/ We have live streams of Ceph Tech Talks every last Thursday of each month at 17:00 UTC. If you're interested in giving a talk next month, please reply to me directly! -- Mike Perez He/Him Ceph Community Manager Red Hat Los Angeles <https://www.redhat.com> thingee(a)redhat.com <mailto:thingee@redhat.com> M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> <https://www.redhat.com> <https://redhat.com/summit>

1 0

15.2.2 bluestore issue
by Josh Durgin 29 May '20

29 May '20

Hi folks, at this time we recommend pausing OSD upgrades to 15.2.2. There have been a couple reports of OSDs crashing due to rocksdb corruption after upgrading to 15.2.2 [1] [2]. It's safe to upgrade monitors and mgr, but OSDs and everything else should wait. We're investigating and will get a fix out as soon as we can. You can follow progress on this tracker: https://tracker.ceph.com/issues/45613 Josh [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/CX5PRFGL6UB… [2] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/CWN7BNPGSRB…

3 4

PGs degraded after osd restart
by Vytenis A 29 May '20

29 May '20

Hi cephists, We have a 10 node cluster running Nautilus 14.2.9 All objects are on EC pool. We have mgr balancer plugin in upmap mode doing it's rebalancing: health: HEALTH_OK pgs: 1985 active+clean 190 active+remapped+backfilling 65 active+remapped+backfill_wait io: client: 0 B/s wr, 0 op/s rd, 0 op/s wr recovery: 770 MiB/s, 463 objects/s We have restarted osd.0 on one of our OSD nodes, and this was the status immediately after: ``` health: HEALTH_WARN 1 osds down Degraded data redundancy: 4531479/531067647 objects degraded (0.853%), 109 pgs degraded ``` Then OSD became UP again: ``` health: HEALTH_WARN Degraded data redundancy: 4963207/531067545 objects degraded (0.935%), 120 pgs degraded ``` And after a minute or so has passed it settled on: ``` health: HEALTH_WARN Degraded data redundancy: 295515/531067347 objects degraded (0.056%), 10 pgs degraded, 10 pgs undersized ``` upmap balancer was running during osd.0 restart, the restart was successfull, without any issues. This left us wondering - how could a simple osd restart cause degraded PGs? Could this be related to the upmap balancer running? Thanks! -- Vyteni

1 0

snapshot-based mirroring explanation in docs
by Hans van den Bogert 28 May '20

28 May '20

Hi list, When reading the documentation for the new way of mirroring [1], some questions arose, especially with the following sentence: > Since this mode is not point-in-time consistent, the full snapshot delta will need to be synced prior to use during a failover scenario. 1) I'm not sure I follow. Why are snapshots not point-in-time, whereas using the journal-based mirroring is? 2) Also, I am not sure what the implications are for the second part of the referenced sentence, what significance does the word "full" bear in "full snapshot delta" ? Further, Isn't it obvious that in order to do a failover, the other side should have the latest snapshot? - I am probably missing what the documentation is trying to tell me. All in all, I think some background concepts are missing in order to read this chapter of the documentation. It might well be that this is explained somewhere else in the docs, but maybe we should link to that. If someone can explain the above 2 points, I'm willing to take a shot at making a patch if I can make this clearer in the docs. Thanks, Hans [1] https://docs.ceph.com/docs/octopus/rbd/rbd-mirroring/

2 1

cephfs file layouts, empty objects in first data pool
by Håkan T Johansson 28 May '20

28 May '20

Hi, running 14.2.6, debian buster (backports). Have set up a cephfs with 3 data pools and one metadata pool: myfs_data, myfs_data_hdd, myfs_data_ssd, and myfs_metadata. The data of all files are with the use of ceph.dir.layout.pool either stored in the pools myfs_data_hdd or myfs_data_ssd. This has also been checked by dumping the ceph.file.layout.pool attributes of all files. The filesystem has 1617949 files and 36042 directories. There are however approximately as many objects in the first pool created for the cephfs, myfs_data, as there are files. They also becomes more or fewer as files are created or deleted (so cannot be some leftover from earlier exercises). Note how the USED size is reported as 0 bytes, correctly reflecting that no file data is stored in them. POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR myfs_data 0 B 1618229 0 4854687 0 0 0 2263590 129 GiB 23312479 124 GiB 0 B 0 B myfs_data_hdd 831 GiB 136309 0 408927 0 0 0 106046 200 GiB 269084 277 GiB 0 B 0 B myfs_data_ssd 43 GiB 1552412 0 4657236 0 0 0 181468 2.3 GiB 4661935 12 GiB 0 B 0 B myfs_metadata 1.2 GiB 36096 0 108288 0 0 0 4828623 82 GiB 1355102 143 GiB 0 B 0 B Is this expected? I was assuming that in this scenario, all objects, both their data and any keys would be either in the metadata pool, or the two pools where the objects are stored. Is it some additional metadata keys that are stored in the first created data pool for cephfs? This would not be so nice in case the osd selection rules for it are using worse disks than the data itself... Btw: is there any tool to see the amount of key value data size associated with a pool? 'ceph osd df' gives omap and meta for osds, but not broken down per pool. Best regards, Håkan

4 6

Re: Reducing RAM usage on production MDS
by Dan van der Ster 28 May '20

28 May '20

Hi Dylan, It looks like you have 10GB of heap to be release -- try `ceph tell mds.$(hostname) heap release` to free that up. Otherwise, I've found it safe to incrementally inject decreased mds_cache_memory_limit's on prod mds's running v12.2.12. I'd start by decreasing the size just a few hundred MBs at a time while tailing the mds log with `debug mds = 2` or `watch --color ceph fs status` to see the cache sizes decrease and stabilize after each change. (In my case I've decreased from ~16GB caches to ~4GB across 9 active MDSs -- I moved at around 500MB per injection and there were no slow req's or client issues.) BTW, we also increase `mds cache trim threshold` to allow the MDS to trim more caps per 5s tick -- if you find the lru is not trimming quickly enough you could try 1.5 or 2x the default value. If things get hairy you could increase the mds_beacon_grace (on mon and or mds) to tolerate longer missed heartbeats rather than failing the mds. Cheers, Dan On Thu, May 28, 2020 at 7:09 AM Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: > > Hi all, > > The single active MDS on one of our Ceph clusters is close to running out of RAM. > > MDS total system RAM = 528GB > MDS current free system RAM = 4GB > mds_cache_memory_limit = 451GB > current mds cache usage = 426GB > > Presumably we need to reduce our mds_cache_memory_limit and/or mds_max_caps_per_client, but would like some guidance on whether it’s possible to do that safely on a live production cluster when the MDS is already pretty close to running out of RAM. > > Cluster is Luminous - 12.2.12 > Running single active MDS with two standby. > 890 clients > Mix of kernel client (4.19.86) and ceph-fuse. > Clients are 12.2.12 (398) and 12.2.13 (3) > > The kernel clients have stayed under “mds_max_caps_per_client”: “1048576". But the ceph-fuse clients appear to hold very large numbers according to the ceph-fuse asok. > e.g. > “num_caps”: 1007144398, > “num_caps”: 1150184586, > “num_caps”: 1502231153, > “num_caps”: 1714655840, > “num_caps”: 2022826512, > > Dropping caches on the clients appears to reduce their cap usage but does not free up RAM on the MDS. > What is the safest method to free cache and reduce RAM usage on the MDS in this situation (without having to evict or remount clients)? > I’m concerned that reducing mds_cache_memory_limit even in very small increments may trigger a large recall of caps and overwhelm the MDS. > We also considered setting a reduced mds_cache_memory_limit on both the standby MDS. Would a subsequent failover to an MDS with a lower cache limit be safe? > Some more details below and I’d be more than happy to provide additional logs. > > Thanks, > Dylan > > > # free -b > total used free shared buff/cache available > Mem: 540954992640 535268749312 4924698624 438284288 761544704 3893182464 > Swap: 0 0 0 > > # ceph daemon mds.$(hostname -s) config get mds_cache_memory_limit > { > "mds_cache_memory_limit": "450971566080" > } > > # ceph daemon mds.$(hostname -s) cache status > { > "pool": { > "items": 10593257843, > "bytes": 425176150288 > } > } > > # ceph daemon mds.$(hostname -s) dump_mempools | grep -A2 "mds_co\|anon" > "buffer_anon": { > "items": 3935, > "bytes": 4537932 > -- > "mds_co": { > "items": 10595391186, > "bytes": 425255456209 > > # ceph daemon mds.$(hostname -s) perf dump | jq '.mds_mem.rss' > 520100552 > > # ceph tell mds.$(hostname) heap stats > tcmalloc heap stats:------------------------------------------------ > MALLOC: 496040753720 (473061.3 MiB) Bytes in use by application > MALLOC: + 11085479936 (10571.9 MiB) Bytes in page heap freelist > MALLOC: + 22568895888 (21523.4 MiB) Bytes in central cache freelist > MALLOC: + 31744 ( 0.0 MiB) Bytes in transfer cache freelist > MALLOC: + 34186296 ( 32.6 MiB) Bytes in thread cache freelists > MALLOC: + 2802057216 ( 2672.2 MiB) Bytes in malloc metadata > MALLOC: ------------ > MALLOC: = 532531404800 (507861.5 MiB) Actual memory used (physical + swap) > MALLOC: + 1315700736 ( 1254.8 MiB) Bytes released to OS (aka unmapped) > MALLOC: ------------ > MALLOC: = 533847105536 (509116.3 MiB) Virtual address space used > MALLOC: > MALLOC: 44496459 Spans in use > MALLOC: 22 Thread heaps in use > MALLOC: 8192 Tcmalloc page size > ------------------------------------------------ > Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). > Bytes released to the OS take up virtual address space but no physical memory. > > > # ceph fs status > hpc_projects - 890 clients > ============ > +------+--------+----------------+---------------+-------+-------+ > | Rank | State | MDS | Activity | dns | inos | > +------+--------+----------------+---------------+-------+-------+ > | 0 | active | mds1-ceph2-qh2 | Reqs: 304 /s | 167M | 167M | > +------+--------+----------------+---------------+-------+-------+ > +--------------------+----------+-------+-------+ > | Pool | type | used | avail | > +--------------------+----------+-------+-------+ > | hpcfs_metadata | metadata | 17.4G | 1893G | > | hpcfs_data | data | 1014T | 379T | > | test_nvmemeta | data | 0 | 1893G | > | hpcfs_data_sandisk | data | 312T | 184T | > +--------------------+----------+-------+-------+ > > +----------------+ > | Standby MDS | > +----------------+ > | mds3-ceph2-qh2 | > | mds2-ceph2-qh2 | > +----------------+ > MDS version: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Reducing RAM usage on production MDS
by Dylan McCulloch 28 May '20

28 May '20

Hi all, The single active MDS on one of our Ceph clusters is close to running out of RAM. MDS total system RAM = 528GB MDS current free system RAM = 4GB mds_cache_memory_limit = 451GB current mds cache usage = 426GB Presumably we need to reduce our mds_cache_memory_limit and/or mds_max_caps_per_client, but would like some guidance on whether it’s possible to do that safely on a live production cluster when the MDS is already pretty close to running out of RAM. Cluster is Luminous - 12.2.12 Running single active MDS with two standby. 890 clients Mix of kernel client (4.19.86) and ceph-fuse. Clients are 12.2.12 (398) and 12.2.13 (3) The kernel clients have stayed under “mds_max_caps_per_client”: “1048576". But the ceph-fuse clients appear to hold very large numbers according to the ceph-fuse asok. e.g. “num_caps”: 1007144398, “num_caps”: 1150184586, “num_caps”: 1502231153, “num_caps”: 1714655840, “num_caps”: 2022826512, Dropping caches on the clients appears to reduce their cap usage but does not free up RAM on the MDS. What is the safest method to free cache and reduce RAM usage on the MDS in this situation (without having to evict or remount clients)? I’m concerned that reducing mds_cache_memory_limit even in very small increments may trigger a large recall of caps and overwhelm the MDS. We also considered setting a reduced mds_cache_memory_limit on both the standby MDS. Would a subsequent failover to an MDS with a lower cache limit be safe? Some more details below and I’d be more than happy to provide additional logs. Thanks, Dylan # free -b total used free shared buff/cache available Mem: 540954992640 535268749312 4924698624 438284288 761544704 3893182464 Swap: 0 0 0 # ceph daemon mds.$(hostname -s) config get mds_cache_memory_limit { "mds_cache_memory_limit": "450971566080" } # ceph daemon mds.$(hostname -s) cache status { "pool": { "items": 10593257843, "bytes": 425176150288 } } # ceph daemon mds.$(hostname -s) dump_mempools | grep -A2 "mds_co\|anon" "buffer_anon": { "items": 3935, "bytes": 4537932 -- "mds_co": { "items": 10595391186, "bytes": 425255456209 # ceph daemon mds.$(hostname -s) perf dump | jq '.mds_mem.rss' 520100552 # ceph tell mds.$(hostname) heap stats tcmalloc heap stats:------------------------------------------------ MALLOC: 496040753720 (473061.3 MiB) Bytes in use by application MALLOC: + 11085479936 (10571.9 MiB) Bytes in page heap freelist MALLOC: + 22568895888 (21523.4 MiB) Bytes in central cache freelist MALLOC: + 31744 ( 0.0 MiB) Bytes in transfer cache freelist MALLOC: + 34186296 ( 32.6 MiB) Bytes in thread cache freelists MALLOC: + 2802057216 ( 2672.2 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 532531404800 (507861.5 MiB) Actual memory used (physical + swap) MALLOC: + 1315700736 ( 1254.8 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 533847105536 (509116.3 MiB) Virtual address space used MALLOC: MALLOC: 44496459 Spans in use MALLOC: 22 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. # ceph fs status hpc_projects - 890 clients ============ +------+--------+----------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+----------------+---------------+-------+-------+ | 0 | active | mds1-ceph2-qh2 | Reqs: 304 /s | 167M | 167M | +------+--------+----------------+---------------+-------+-------+ +--------------------+----------+-------+-------+ | Pool | type | used | avail | +--------------------+----------+-------+-------+ | hpcfs_metadata | metadata | 17.4G | 1893G | | hpcfs_data | data | 1014T | 379T | | test_nvmemeta | data | 0 | 1893G | | hpcfs_data_sandisk | data | 312T | 184T | +--------------------+----------+-------+-------+ +----------------+ | Standby MDS | +----------------+ | mds3-ceph2-qh2 | | mds2-ceph2-qh2 | +----------------+ MDS version: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)

1 0

Nautilus to Octopus Upgrade mds without downtime
by Andreas Schiefer 28 May '20

28 May '20

Hello, if I understand correctly: if we upgrade from an running nautilus cluster to octopus we have a downtime on an update of MDS. Is this correct? Mit freundlichen Grüßen / Kind regards Andreas Schiefer Leiter Systemadministration / Head of systemadministration --- HOME OF LOYALTY CRM- & Customer Loyalty Solution by UW Service Gesellschaft für Direktwerbung und Marketingberatung mbH Alter Deutzer Postweg 221 51107 Koeln (Rath/Heumar) Deutschland Telefon : +49 221 98696 0 Telefax : +49 221 98696 5222 info(a)uw-service.de www.hooloy.de Amtsgericht Koeln HRB 24 768 UST-ID: DE 164 191 706 Geschäftsführer: Ralf Heim ---

2 1

Fwd: [IO-500] IO500 ISC20 Call for Submission
by John Bent 28 May '20

28 May '20

FYI. Hope to see some awesome CephFS submissions for our virtual IO500 BoF! Thanks, John ---------- Forwarded message --------- From: committee--- via IO-500 <io-500(a)vi4io.org> Date: Fri, May 22, 2020 at 1:53 PM Subject: [IO-500] IO500 ISC20 Call for Submission To: <io-500(a)vi4io.org> *Deadline*: 08 June 2020 AoE The IO500 <http://io500.org/> is now accepting and encouraging submissions for the upcoming 6th IO500 list. Once again, we are also accepting submissions to the 10 Node Challenge to encourage the submission of small scale results. The new ranked lists will be announced via live-stream at a virtual session. We hope to see many new results. The benchmark suite is designed to be easy to run and the community has multiple active support channels to help with any questions. Please note that submissions of all sizes are welcome; the site has customizable sorting so it is possible to submit on a small system and still get a very good per-client score for example. Additionally, the list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. More details below. Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 <http://io500.org/> was created in 2017, published its first list at SC17, and has grown exponentially since then. The need for such an initiative has long been known within High-Performance Computing; however, defining appropriate benchmarks had long been challenging. Despite this challenge, the community, after long and spirited discussion, finally reached consensus on a suite of benchmarks and a metric for resolving the scores into a single ranking. The multi-fold goals of the benchmark suite are as follows: 1. Maximizing simplicity in running the benchmark suite 2. Encouraging optimization and documentation of tuning parameters for performance 3. Allowing submitters to highlight their “hero run” performance numbers 4. Forcing submitters to simultaneously report performance for challenging IO patterns. Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes an IOR and mdtest run with highly constrained parameters forcing a difficult usage pattern in an attempt to determine a lower-bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Submitters are encouraged to share their tuning insights for publication. The goals of the community are also multi-fold: 1. Gather historical data for the sake of analysis and to aid predictions of storage futures 2. Collect tuning data to share valuable performance optimizations across the community 3. Encourage vendors and designers to optimize for workloads beyond “hero runs” 4. Establish bounded expectations for users, procurers, and administrators *10 Node I/O Challenge* The 10 Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly *10 client nodes* must be used to run the benchmark. You may use any shared storage with, e.g., any number of servers. When submitting for the IO500 list, you can opt-in for “Participate in the 10 compute node challenge only”, then we will not include the results into the ranked list. Other 10-node node submissions will be included in the full list and in the ranked list. We will announce the result in a separate derived list and in the full list but not on the ranked IO500 list at https://io500.org/. This information and rules for ISC20 submissions are available here: https://www.vi4io.org/io500/rules/submission Thanks, The IO500 Committee _______________________________________________ IO-500 mailing list IO-500(a)vi4io.org https://www.vi4io.org/mailman/listinfo/io-500

1 0