- ceph-users - lists.ceph.io

Hardware needs for MDS for HPC/OpenStack workloads?
by Matthew Vernon 23 Oct '20

23 Oct '20

Hi, We're considering the merits of enabling CephFS for our main Ceph cluster (which provides object storage for OpenStack), and one of the obvious questions is what sort of hardware we would need for the MDSs (and how many!). These would be for our users scientific workloads, so they would need to provide reasonably high performance. For reference, we have 3060 6TB OSDs across 51 OSD hosts, and 6 dedicated RGW nodes. The minimum specs are very modest (2-3GB RAM, a tiny amount of disk, similar networking to the OSD nodes), but I'm not sure how much going beyond that is likely to be useful in production. I've also seen it suggested that an SSD-only pool is sensible for the CephFS metadata pool; how big is that likely to get? I'd be grateful for any pointers :) Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

4 3

Re: ceph octopus centos7, containers, cephadm
by Dan van der Ster 23 Oct '20

23 Oct '20

I'm not sure I understood the question. If you're asking if you can run octopus via RPMs on el7 without the cephadm and containers orchestration, then the answer is yes. -- dan On Fri, Oct 23, 2020 at 9:47 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > No clarity on this? > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] ceph octopus centos7, containers, cephadm > > > I am running Nautilus on centos7. Does octopus run similar as nautilus > thus: > > - runs on el7/centos7 > - runs without containers by default > - runs without cephadm by default > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Ceph Octopus and Snapshot Schedules
by Adam Boyhan 23 Oct '20

23 Oct '20

Care to provide anymore detail?

1 0

ceph octopus centos7, containers, cephadm
by Marc Roos 23 Oct '20

23 Oct '20

I am running Nautilus on centos7. Does octopus run similar as nautilus thus: - runs on el7/centos7 - runs without containers by default - runs without cephadm by default

2 2

RFC: Seeking your input on some design documents related to cephadm / Dashboard
by Lenz Grimmer 22 Oct '20

22 Oct '20

Hi all, FYI, we're currently reviewing and discussing several design documents that cover some topics related to "Day 1 installation" and "Day 2 operations" using the functionality provided by cephadm in Ceph Dashboard. We'd like to solicit your input on these plans, so feel free to review and comment on these drafts here (many thanks to Paul Cuzner for kicking this off): docs: Dashboard host management — https://github.com/ceph/ceph/pull/37292 doc/dev/cephadm: Doc defining the design for host maintenance — https://github.com/ceph/ceph/pull/37607 doc/dev/cephadm: high level design for a compliance check feature — https://github.com/ceph/ceph/pull/37519 Thank you for your input! Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

1 0

Huge RAM Ussage on OSD recovery
by Ing. Luis Felipe Domínguez Vega 22 Oct '20

22 Oct '20

Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed. Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3 In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that.

4 9

Need help integrating radosgw with keystone for openstack swift
by Bujack, Stefan 22 Oct '20

22 Oct '20

Hello, I am struggling to integrate ceph radosgw as obejctstore in openstack swift via keystone. Could someone please have a look at my configs and help finding the issue? Many thanks ins advance. ceph version 14.2.11 nautilus (stable) [root@ciosmon06 ~]# cat /etc/ceph/ceph.conf [global] fsid = someid mon_initial_members = ciosmon06, ciosmon07, ciosmon08, ciosmon09, ciosmon10 mon_host = 10.254.32.32,10.254.32.33,10.254.32.34,10.254.32.35,10.254.32.36 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = 10.254.32.0/24 cluster_network = 10.254.36.0/24 [client.radosgw.gateway] debug ms = 1 debug rgw = 20 rgw host = 10.254.32.32 rgw keystone verify ssl = false rgw keystone api version = 3 rgw keystone url = https://keystone-intern.desy.de:5000 rgw keystone admin domain = default rgw keystone admin project = service rgw keystone admin user = swift rgw keystone admin password = password rgw s3 auth use keystone = true rgw swift account in url = true rgw keystone implicit tenants = true rgw swift versioning enabled = true rgw enable apis = swift, s3 rgw keystone accepted roles = admin, _member_, member, swiftoperator rgw keystone accepted admin roles = admin, _member_ keyring = /etc/ceph/client.rgw.ciosmon06.keyring [root@ciosmon06 ~]# systemctl status ceph-radosgw(a)rgw.ciosmon06.service ● ceph-radosgw(a)rgw.ciosmon06.service - Ceph rados gateway Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-10-21 17:22:04 CEST; 3min 36s ago Main PID: 3372096 (radosgw) CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw(a)rgw.ciosmon06.service └─3372096 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ciosmon06 --setuser ceph --setgroup ceph Oct 21 17:22:04 ciosmon06.desy.de systemd[1]: Started Ceph rados gateway. [root@ciosmon06 ~]# netstat -ntlup Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:7480 0.0.0.0:* LISTEN 3372096/radosgw tcp6 0 0 :::7480 :::* LISTEN 3372096/radosgw [root@ciosmon06 ceph]# ls -la total 32 drwxr-xr-x 2 root root 4096 Oct 21 18:18 . drwxr-xr-x. 93 root root 12288 Sep 24 05:25 .. -rw------- 1 root root 63 Feb 24 2020 ceph.client.admin.keyring -rw-r--r-- 1 root root 1036 Oct 21 18:18 ceph.conf -rw------- 1 root root 71 Oct 21 16:38 client.rgw.ciosmon06.keyring [root@ciosmon06 ~]# cat /etc/ceph/client.rgw.ciosmon06.keyring [client.rgw.ciosmon06] key = somekey [root@ciosmon06 ~]# ceph auth ls client.rgw.ciosmon06 key: somekey caps: [mon] allow rw caps: [osd] allow rwx Openstack ussuri on ubuntu 20.04 root@keystone:~# openstack endpoint list +----------------------------------+-----------+--------------+---------------------+---------+-----------+------------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+---------------------+---------+-----------+------------------------------------------------------------+ | 0ee9c91af2424e33a91a4c118b693301 | RegionOne | swift | object-store | True | internal | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | 1fe9f91db7f942d2a8cb2163bd09772d | RegionOne | keystone | identity | True | admin | https://keystone-intern.desy.de:5000/v3/ | | 4719a266432f45bda380c52486421e62 | RegionOne | swift | object-store | True | public | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | 95a292a5584d4e2c9f8177af3a312494 | RegionOne | keystone | identity | True | public | https://keystone-intern.desy.de:5000/v3/ | | e68b3990e74447bfa35a5d6aa66ca2aa | RegionOne | swift | object-store | True | admin | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | efc1df3cb21344bfb74d8e345e267606 | RegionOne | keystone | identity | True | internal | https://keystone-intern.desy.de:5000/v3/ | root@keystone:~# openstack service list +----------------------------------+-----------+---------------------+ | ID | Name | Type | +----------------------------------+-----------+---------------------+ | 4881557156f844df87d13e427d02b74a | swift | object-store | | 6d928ad49d954a179e37dc17f96c4b1e | keystone | identity | root@keystone:~# openstack user list +----------------------------------+-----------+ | ID | Name | +----------------------------------+-----------+ | ca94085253944ba3803f9d3202172b80 | admin | | ea025c992c224b5f8ca5fea0334eaf15 | swift | +----------------------------------+-----------+ root@keystone:~# openstack role assignment list --names --project service +-------+-------------------+-------+-----------------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +-------+-------------------+-------+-----------------+--------+--------+-----------+ | admin | swift@Default | | service@Default | | | False | +-------+-------------------+-------+-----------------+--------+--------+-----------+ Tests: root@it-build:~# openstack ec2 credentials create +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | access | bc6f1ad9064d4097ad7a323c8368780b | | links | {'self': 'https://keystone-intern.desy.de:5000/v3/users/926c750033e668f0af2330b1c7c72…'} | | project_id | 286f5d2b38ae4595ba9ff8129e754f54 | | secret | c6d9a58220fd4491ba8654d6d3b3c4dc | | trust_id | None | | user_id | 926c750033e668f0af2330b1c7c723a05b86fa393655369fdb1a5622ae65dac8 | +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ root@it-build:~# /usr/local/bin/aws configure AWS Access Key ID [****************e90c]: bc6f1ad9064d4097ad7a323c8368780b AWS Secret Access Key [****************6ba4]: c6d9a58220fd4491ba8654d6d3b3c4dc Default region name [default]: Default output format [None]: root@it-build:~# /usr/local/bin/aws --endpoint='http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s' s3 ls s3:// expected string or bytes-like object [root@ciosmon06 ~]# tail -f /var/log/ceph/ceph-client.rgw.ciosmon06.log 2020-10-21 18:24:24.710 7efeabf69700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:24:24.713 7efeabf69700 1 ====== req done req=0x562f3dbb08f0 op status=-2024 http_status=301 latency=0.003s ====== root@keystone:~# openstack role assignment list --names --project 286f5d2b38ae4595ba9ff8129e754f54 | grep swiftoperator | swiftoperator | myusername@closedbeta | | closedbeta@closedbeta | | | False | root@it-build:~# cat myusername-openrc-closedbeta export OS_PROJECT_DOMAIN_NAME=closedbeta export OS_USER_DOMAIN_NAME=closedbeta export OS_PROJECT_NAME=closedbeta export OS_USERNAME=myusername export OS_PASSWORD=password export OS_AUTH_URL=https://keystone-intern.desy.de:5000 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 export OS_VOLUME_API_VERSION=3 root@it-build:~# openstack container list Unrecognized schema in response body. (HTTP 401) (Request-ID: tx000000000000000000005-005f9060f7-26173f-default) [root@ciosmon06 ~]# tail -f /var/log/ceph/ceph-client.rgw.ciosmon06.log 2020-10-21 18:25:26.868 7efea9f65700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:25:26.868 7efea9f65700 1 ====== req done req=0x562f3dbb08f0 op status=0 http_status=401 latency=0s ====== 2020-10-21 18:25:27.088 7efeadf6d700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:25:27.088 7efeadf6d700 1 ====== req done req=0x562f3dbb08f0 op status=0 http_status=401 latency=0s ====== Greets Stefan Bujack

2 2

6 PG's stuck not-active, remapped
by Mac Wynkoop 22 Oct '20

22 Oct '20

We recently did some work on the Ceph cluster, and a few disks ended up offline at the same time. There are now 6 PG's that are stuck in a "remapped" state, and this is all of their recovery states: *recovery_state: 0: name: Started/Primary/WaitActingChangeenter_time: 2020-10-21 18:48:02.034430comment: waiting for pg acting set to change1: name: Startedenter_time: 2020-10-21 18:48:01.752957* Any ideas? Mac Wynkoop

2 2

Re: Huge RAM Ussage on OSD recovery
by Ing. Luis Felipe Domínguez Vega 21 Oct '20

21 Oct '20

El 2020-10-21 10:08, Mark Nelson escribió: > On 10/21/20 7:54 AM, Ing. Luis Felipe Domínguez Vega wrote: >> El 2020-10-21 08:43, Mark Nelson escribió: >>> Theoretically we shouldn't be spiking memory as much these days >>> during >>> recovery, but the code is complicated and it's tough to reproduce >>> these kinds of issues in-house. If you happen to catch it in the >>> act, >>> do you see the pglog mempool stats also spiking up? >>> >>> >>> Mark >>> >>> >>> On 10/21/20 2:34 AM, Dan van der Ster wrote: >>>> Hi, >>>> >>>> This might be the pglog issue which has been coming up a few times >>>> on the list. >>>> If the OSD cannot boot without going OOM, you might have success by >>>> trimming the pglog, e.g. search this list for "ceph-objectstore-tool >>>> --op trim-pg-log" for some recipes. The thread "OSDs taking too much >>>> memory, for pglog" in particular might help. >>>> >>>> Cheers, Dan >>>> >>>> >>>> >>>> On Tue, Oct 20, 2020 at 11:57 PM Ing. Luis Felipe Domínguez Vega >>>> <luis.dominguez(a)desoft.cu> wrote: >>>>> Hi, today mi Infra provider has a blackout, then the Ceph was try >>>>> to >>>>> recover but are in an inconsistent state because many OSD can >>>>> recover >>>>> itself because the kernel kill it by OOM. Even now one OSD that was >>>>> OK, >>>>> go down by OOM killed. >>>>> >>>>> Even in a server with 32GB RAM the OSD use ALL that and never >>>>> recover, i >>>>> think that can be a memory leak, ceph version octopus 15.2.3 >>>>> >>>>> In: https://pastebin.pl/view/59089adc >>>>> You can see that buffer_anon get 32GB, but why?? all my cluster is >>>>> down >>>>> because that. >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> this https://pastebin.pl/view/59089adc is almost the OSD going to be >> killed by OOM >> > > Ok, that is very interesting! The OSD memory autotuning code shrank > the caches to be almost nothing to try and compensate for the huge > growth in buffer_anon (and to a lesser extent osd_pglog) usage but > obviously couldn't do anything with that much memory being used. Any > chance you could create a tracker ticket and paste the memory pool > info in along with ceph version/etc? > > > https://tracker.ceph.com/ > > > Mark Thanks, https://tracker.ceph.com/issues/47929

1 0

Difference between node exporter and ceph exporter data
by Seena Fallah 21 Oct '20

21 Oct '20

Hi all, There is a huge difference between node exporter and ceph exporter (prometheus mgr module) data. For example I see there is a 120MB/s write on my disk from node exporter but ceph exporter says it is 22MB! Also for latency and IOPS and... Which one is reliable? Thanks.

1 0