- ceph-users - lists.ceph.io

Re: Cephfs metadta pool suddenly full (100%) !
by Sebastian Knust 01 Jun '21

01 Jun '21

Hi Hervé, On 01.06.21 13:15, Hervé Ballans wrote: > # ceph status > cluster: > id: 838506b7-e0c6-4022-9e17-2d1cf9458be6 > health: HEALTH_ERR > 1 filesystem is degraded > 3 full osd(s) > 1 pool(s) full > 1 daemons have recently crashed You have full OSDs and therefore a full pool. The "fullness" of a pool is limited by the fullest OSD, i.e. a single full OSDs can block your pool. Take a look at `ceph osd df` and you will notice a very non-uniform osd usage (both with numbers of PG / size as well as usage %). > osd: 126 osds: 126 up (since 5m), 126 in (since 5M) > pgs: 1662 active+clean The PG/osd ratio seems to be very low for me. The general recommendation is 100 PG / osd post-replication (and power of 2 for each pool). In my cluster I actually run with ~200 PG / osd for my SSD which contain the cephfs metadata. > > Thanks a lot if you have some ways for trying to solve this... You have to get your OSDs to rebalance, which probably includes increasing the number of PGs in some pools. Details depend on which Ceph version you are running and your CRUSH rules (maybe your cephfs metadata pool is residing only on NVMe?). Take a look at the balancer module [1] and the autoscaler [2] (`ceph osd pool autoscale-status` is most interesting). Theoretically, you could (temporarilly!) increase the full_ratio. However, this is a very dangerous operation which you should not do unless you know *exactly* what you are doing. Cheers & Best of luck Sebastian [1] https://docs.ceph.com/en/latest/rados/operations/balancer/ [2] https://docs.ceph.com/en/latest/rados/operations/placement-groups/ Replace latest in the URIs with your Ceph version string (i.e. octopus, nautilus) for version specific documentation

2 1

local mirror from quay.ceph.io
by Seba chanel 01 Jun '21

01 Jun '21

hi, Can you show me how to mirror the ceph docker images that are available from quay.ceph.io Our CEPH clusters are on a private network, and they are installed from a local repository because they cannot directly access the internet network. With Octopus, I need to be able to create a local mirror of the different packages and images. What do you recommend as a product for creating a local registry? thanks for your advices Sébastien.

1 0

The always welcomed large omap
by Szabo, Istvan (Agoda) 01 Jun '21

01 Jun '21

Hi, Any way to clean up large-omap in the index pool? PG deep_scrub didn't help. I know how to clean in the log pool, but no idea in the index pool :/ It's an octopus deployment 15.2.10. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

Re: The always welcomed large omap
by Szabo, Istvan (Agoda) 01 Jun '21

01 Jun '21

So the bucket has been deleted on the master zone which has been removed from the other zones as well. On the master zone after deep scrub the omap disappeared but on the secondary zone it's still there. It was 3 at the beginning after I scrubbed the affected OSDs (not just pgs) I have 6 omap. The bucket which I assumed caused the issue is not there anymore. This is the log: 2021-06-01T08:48:11.560449+0700 osd.39 (osd.39) 1 : cluster [DBG] 20.6 deep-scrub starts 2021-06-01T08:48:13.218814+0700 osd.39 (osd.39) 2 : cluster [WRN] Large omap object found. Object: 20:62d24dc9:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.5:head PG: 20.93b24b46 (20.6) Key count: 244664 Size (bytes): 69659918 2021-06-01T08:48:15.245975+0700 osd.39 (osd.39) 3 : cluster [DBG] 20.6 deep-scrub ok 2021-06-01T08:48:16.623097+0700 osd.39 (osd.39) 4 : cluster [DBG] 20.e deep-scrub starts 2021-06-01T08:48:20.201926+0700 osd.39 (osd.39) 5 : cluster [WRN] Large omap object found. Object: 20:77cbb881:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.9:head PG: 20.811dd3ee (20.e) Key count: 244363 Size (bytes): 69582418 2021-06-01T08:48:20.275906+0700 osd.39 (osd.39) 6 : cluster [DBG] 20.e deep-scrub ok 2021-06-01T08:48:21.560212+0700 osd.39 (osd.39) 7 : cluster [DBG] 20.15 deep-scrub starts 2021-06-01T08:48:22.456133+0700 osd.39 (osd.39) 8 : cluster [WRN] Large omap object found. Object: 20:a8a0840e:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.3:head PG: 20.70210515 (20.15) Key count: 244169 Size (bytes): 69508179 2021-06-01T08:48:25.202051+0700 osd.39 (osd.39) 9 : cluster [DBG] 20.15 deep-scrub ok 2021-06-01T08:48:26.019422+0700 osd.37 (osd.37) 4 : cluster [DBG] 20.f deep-scrub starts 2021-06-01T08:48:28.370919+0700 osd.37 (osd.37) 5 : cluster [WRN] Large omap object found. Object: 20:f59fa8d7:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.6:head PG: 20.eb15f9af (20.f) Key count: 244243 Size (bytes): 69539068 2021-06-01T08:48:29.010877+0700 osd.37 (osd.37) 6 : cluster [DBG] 20.f deep-scrub ok 2021-06-01T08:48:29.573160+0700 osd.39 (osd.39) 10 : cluster [DBG] 20.18 deep-scrub starts 2021-06-01T08:48:32.682416+0700 osd.39 (osd.39) 11 : cluster [WRN] Large omap object found. Object: 20:1e2579ff:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.4:head PG: 20.ff9ea478 (20.18) Key count: 243858 Size (bytes): 69426645 2021-06-01T08:48:33.383843+0700 osd.39 (osd.39) 12 : cluster [DBG] 20.18 deep-scrub ok 2021-06-01T08:48:35.021940+0700 osd.37 (osd.37) 7 : cluster [DBG] 20.14 deep-scrub starts 2021-06-01T08:48:36.139892+0700 osd.37 (osd.37) 8 : cluster [WRN] Large omap object found. Object: 20:291b1669:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.1:head PG: 20.9668d894 (20.14) Key count: 244533 Size (bytes): 69611691 2021-06-01T08:48:38.843235+0700 osd.37 (osd.37) 9 : cluster [DBG] 20.14 deep-scrub ok Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com --------------------------------------------------- -----Original Message----- From: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> Sent: Monday, May 31, 2021 10:53 PM To: Matt Vandermeulen <storage(a)reenigne.net> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: [Suspicious newsletter] Re: The always welcomed large omap Yeah, I found a bucket at the moment in progress the deleting, will preshard it. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com --------------------------------------------------- -----Original Message----- From: Matt Vandermeulen <storage(a)reenigne.net> Sent: Monday, May 31, 2021 9:43 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> Cc: ceph-users(a)ceph.io Subject: [Suspicious newsletter] [ceph-users] Re: The always welcomed large omap All the index data will be in OMAP, which you can see a listing of with `ceph osd df tree` Do you have large buckets (many, many objects in a single bucket) with few shards? You may have to reshard one (or some) of your buckets. It'll take some reading if you're using multisite, in order to coordinate it (though I'm unfamiliar with how it works with multisite in Octopus). On 2021-05-31 02:25, Szabo, Istvan (Agoda) wrote: > Hi, > > Any way to clean up large-omap in the index pool? > PG deep_scrub didn't help. > I know how to clean in the log pool, but no idea in the index pool :/ > It's an octopus deployment 15.2.10. > > Thank you > > ________________________________ > This message is confidential and is for the sole use of the intended > recipient(s). It may also be privileged or otherwise protected by > copyright or other legal rules. If you have received it by mistake > please let us know by reply email and delete it from your system. It > is prohibited to copy this message or disclose its content to anyone. > Any confidentiality or privilege is not waived or lost by any mistaken > delivery or unauthorized disclosure of the message. All messages sent > to and from Agoda may be monitored to ensure compliance with company > policies, to protect the company's interests and to remove potential > malware. Electronic messages may be intercepted, amended, lost or > deleted, or contain viruses. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Nautilus CentOS-7 rpm dependencies
by Andreas Haupt 31 May '21

31 May '21

Dear all, ceph-mgr-dashboard-15.2.13-0.el7.noarch contains three rpm dependencies that cannot be resolved here (not part of CentOS & EPEL 7): python3-cherrypy python3-routes python3-jwt Does anybody know where they are expected to come from? Thanks, Andreas -- | Andreas Haupt | E-Mail: andreas.haupt(a)desy.de | DESY Zeuthen | WWW: http://www-zeuthen.desy.de/~ahaupt | Platanenallee 6 | Phone: +49/33762/7-7359 | D-15738 Zeuthen | Fax: +49/33762/7-7216

3 2

Cephadm/docker or install from packages
by Stanislav Datskevych 31 May '21

31 May '21

Hi all, I want to ask your opinion which Ceph deploy version is better: using cephadm(docker) or installing from packages? Cephadm brings lots of convenience: easy upgrade (which sometimes stucks but still), easy add new OSDs, ability to make placement policies etc. In contrary I seem to lose some transparency on how Ceph works as well as ability to apply fine-grained control, namely setting up daemon configs myself. Even running ceph-volume lvm list to see which osd uses a particular block devices requires entering a docker container. So which option is better to use in production cluster?

1 0

SSD recommendations for RBD and VM's
by by morphin 30 May '21

30 May '21

Hello. I have virtualization env and I'm looking new SSD for HDD replacement. What are the best Performance / Price SSDs in the market right now? I'm looking 1TB, 512GB, 480GB, 256GB, 240GB. Is there a SSD recommendation list for ceph?

3 3

nomenclature: ceph or cephfs (initramfs-tools)
by Markus Kienast 30 May '21

30 May '21

Hi, I have created initramfs scripts for booting from RBD and CephFS: https://launchpad.net/~trickkiste/+archive/ubuntu/ceph-initramfs https://gitlab.com/trickkiste-at/ceph-initramfs I noticed that the CephFS kernel module seems to be named "ceph" not "cephfs", so I would like to to inquire whether I should also name my initramfs-tools scripts and hooks accordingly - ergo "ceph" (not cephfs). My best regards Markus

1 0

HEALTH_WARN Reduced data availability: 33 pgs inactive
by Jeremy Hansen 29 May '21

29 May '21

I’m trying to understand this situation: ceph health detail HEALTH_WARN Reduced data availability: 33 pgs inactive [WRN] PG_AVAILABILITY: Reduced data availability: 33 pgs inactive pg 1.0 is stuck inactive for 20h, current state unknown, last acting [] pg 2.0 is stuck inactive for 20h, current state unknown, last acting [] pg 2.1 is stuck inactive for 20h, current state unknown, last acting [] pg 2.2 is stuck inactive for 20h, current state unknown, last acting [] pg 2.3 is stuck inactive for 20h, current state unknown, last acting [] pg 2.4 is stuck inactive for 20h, current state unknown, last acting [] pg 2.5 is stuck inactive for 20h, current state unknown, last acting [] pg 2.6 is stuck inactive for 20h, current state unknown, last acting [] pg 2.7 is stuck inactive for 20h, current state unknown, last acting [] pg 2.8 is stuck inactive for 20h, current state unknown, last acting [] pg 2.9 is stuck inactive for 20h, current state unknown, last acting [] pg 2.a is stuck inactive for 20h, current state unknown, last acting [] pg 2.b is stuck inactive for 20h, current state unknown, last acting [] pg 2.c is stuck inactive for 20h, current state unknown, last acting [] pg 2.d is stuck inactive for 20h, current state unknown, last acting [] pg 2.e is stuck inactive for 20h, current state unknown, last acting [] pg 2.f is stuck inactive for 20h, current state unknown, last acting [] pg 2.10 is stuck inactive for 20h, current state unknown, last acting [] pg 2.11 is stuck inactive for 20h, current state unknown, last acting [] pg 2.12 is stuck inactive for 20h, current state unknown, last acting [] pg 2.13 is stuck inactive for 20h, current state unknown, last acting [] pg 2.14 is stuck inactive for 20h, current state unknown, last acting [] pg 2.15 is stuck inactive for 20h, current state unknown, last acting [] pg 2.16 is stuck inactive for 20h, current state unknown, last acting [] pg 2.17 is stuck inactive for 20h, current state unknown, last acting [] pg 2.18 is stuck inactive for 20h, current state unknown, last acting [] pg 2.19 is stuck inactive for 20h, current state unknown, last acting [] pg 2.1a is stuck inactive for 20h, current state unknown, last acting [] pg 2.1b is stuck inactive for 20h, current state unknown, last acting [] pg 2.1c is stuck inactive for 20h, current state unknown, last acting [] pg 2.1d is stuck inactive for 20h, current state unknown, last acting [] pg 2.1e is stuck inactive for 20h, current state unknown, last acting [] pg 2.1f is stuck inactive for 20h, current state unknown, last acting [] [ceph: root@cn01 /]# date Sat May 29 01:28:37 UTC 2021 [ceph: root@cn01 /]# ceph pg dump_stuck inactive PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 2.1f unknown [] -1 [] -1 2.1e unknown [] -1 [] -1 2.1d unknown [] -1 [] -1 2.1c unknown [] -1 [] -1 2.1b unknown [] -1 [] -1 2.1a unknown [] -1 [] -1 2.19 unknown [] -1 [] -1 2.18 unknown [] -1 [] -1 2.17 unknown [] -1 [] -1 2.16 unknown [] -1 [] -1 2.15 unknown [] -1 [] -1 2.14 unknown [] -1 [] -1 2.13 unknown [] -1 [] -1 2.12 unknown [] -1 [] -1 2.11 unknown [] -1 [] -1 2.10 unknown [] -1 [] -1 2.f unknown [] -1 [] -1 2.9 unknown [] -1 [] -1 2.b unknown [] -1 [] -1 2.c unknown [] -1 [] -1 2.e unknown [] -1 [] -1 2.a unknown [] -1 [] -1 2.d unknown [] -1 [] -1 2.8 unknown [] -1 [] -1 2.7 unknown [] -1 [] -1 2.6 unknown [] -1 [] -1 2.5 unknown [] -1 [] -1 2.0 unknown [] -1 [] -1 1.0 unknown [] -1 [] -1 2.3 unknown [] -1 [] -1 2.1 unknown [] -1 [] -1 2.2 unknown [] -1 [] -1 2.4 unknown [] -1 [] -1 ok [ceph: root@cn01 /]# ceph pg 2.4 query Couldn't parse JSON : Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/bin/ceph", line 1310, in <module> retval = main() File "/usr/bin/ceph", line 1230, in main sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli') File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 966, in parse_json_funcsigs raise e File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 963, in parse_json_funcsigs overall = json.loads(s) File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) -jeremy

1 0

Remapping OSDs under a PG
by Jeremy Hansen 29 May '21

29 May '21

I’m very new to Ceph so if this question makes no sense, I apologize. Continuing to study but I thought an answer to this question would help me understand Ceph a bit more. Using cephadm, I set up a cluster. Cephadm automatically creates a pool for Ceph metrics. It looks like one of my ssd osd’s was allocated for the PG. I’d like to understand how to remap this PG so it’s not using the SSD OSDs. ceph pg map 1.0 osdmap e205 pg 1.0 (1.0) -> up [28,33,10] acting [28,33,10] OSD 28 is the SSD. Is this possible? Does this make any sense? I’d like to reserve the SSDs for their own pool. Thank you! -jeremy

3 5