October 2020 - ceph-users

by Amudhan P

Hi, I have installed Ceph Octopus cluster using cephadm with a single network now I want to add a second network and configure it as a cluster address. How do I configure ceph to use second Network as cluster network?. Amudhan

3 years, 6 months

3
6
0 0

TOO_FEW_PGS warning and pg_autoscale

by Peter Eisch

Hi, # ceph health detail HEALTH_WARN too few PGs per OSD (24 < min 30) TOO_FEW_PGS too few PGs per OSD (24 < min 30) ceph version 14.2.9 This warning popped up when autoscale shrunk a pool from pg_num and pgp_num from 512 to 256 on its own. The hdd35 storage is only used by this pool. I have three different storage classes and the pools use the different classes as appropriate. How can I convert the warning into something useful which then helps me make the appropriate change to the right class of storage? I'm guessing it's referring to hdd35. RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd25 129 TiB 83 TiB 46 TiB 46 TiB 35.87 hdd35 269 TiB 220 TiB 49 TiB 49 TiB 18.12 ssd 256 TiB 164 TiB 92 TiB 92 TiB 35.84 TOTAL 655 TiB 468 TiB 186 TiB 187 TiB 28.56 If I follow: https://docs.ceph.com/en/latest/rados/operations/health-checks/#too-few-pgs Which then links to: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#choosing… The math for this would want the pool to have pg/p_num of 2048 -- where autoscale just recently shrunk the count. Which is more right? Thanks! peter Peter Eisch Senior Site Reliability Engineer T1.612.445.5135 virginpulse.com Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.66

3 years, 6 months

1
0
0 0

Hardware needs for MDS for HPC/OpenStack workloads?

by Matthew Vernon

Hi, We're considering the merits of enabling CephFS for our main Ceph cluster (which provides object storage for OpenStack), and one of the obvious questions is what sort of hardware we would need for the MDSs (and how many!). These would be for our users scientific workloads, so they would need to provide reasonably high performance. For reference, we have 3060 6TB OSDs across 51 OSD hosts, and 6 dedicated RGW nodes. The minimum specs are very modest (2-3GB RAM, a tiny amount of disk, similar networking to the OSD nodes), but I'm not sure how much going beyond that is likely to be useful in production. I've also seen it suggested that an SSD-only pool is sensible for the CephFS metadata pool; how big is that likely to get? I'd be grateful for any pointers :) Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

3 years, 6 months

4
3
0 0

Re: ceph octopus centos7, containers, cephadm

by Dan van der Ster

I'm not sure I understood the question. If you're asking if you can run octopus via RPMs on el7 without the cephadm and containers orchestration, then the answer is yes. -- dan On Fri, Oct 23, 2020 at 9:47 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > No clarity on this? > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] ceph octopus centos7, containers, cephadm > > > I am running Nautilus on centos7. Does octopus run similar as nautilus > thus: > > - runs on el7/centos7 > - runs without containers by default > - runs without cephadm by default > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 years, 6 months

2
1
0 0

Re: Ceph Octopus and Snapshot Schedules

by Adam Boyhan

Care to provide anymore detail?

3 years, 6 months

1
0
0 0

ceph octopus centos7, containers, cephadm

by Marc Roos

I am running Nautilus on centos7. Does octopus run similar as nautilus thus: - runs on el7/centos7 - runs without containers by default - runs without cephadm by default

3 years, 6 months

2
2
0 0

RFC: Seeking your input on some design documents related to cephadm / Dashboard

by Lenz Grimmer

Hi all, FYI, we're currently reviewing and discussing several design documents that cover some topics related to "Day 1 installation" and "Day 2 operations" using the functionality provided by cephadm in Ceph Dashboard. We'd like to solicit your input on these plans, so feel free to review and comment on these drafts here (many thanks to Paul Cuzner for kicking this off): docs: Dashboard host management — https://github.com/ceph/ceph/pull/37292 doc/dev/cephadm: Doc defining the design for host maintenance — https://github.com/ceph/ceph/pull/37607 doc/dev/cephadm: high level design for a compliance check feature — https://github.com/ceph/ceph/pull/37519 Thank you for your input! Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

3 years, 6 months

1
0
0 0

Huge RAM Ussage on OSD recovery

by Ing. Luis Felipe Domínguez Vega

Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed. Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3 In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that.

3 years, 6 months

4
9
0 0

Need help integrating radosgw with keystone for openstack swift

by Bujack, Stefan

Hello, I am struggling to integrate ceph radosgw as obejctstore in openstack swift via keystone. Could someone please have a look at my configs and help finding the issue? Many thanks ins advance. ceph version 14.2.11 nautilus (stable) [root@ciosmon06 ~]# cat /etc/ceph/ceph.conf [global] fsid = someid mon_initial_members = ciosmon06, ciosmon07, ciosmon08, ciosmon09, ciosmon10 mon_host = 10.254.32.32,10.254.32.33,10.254.32.34,10.254.32.35,10.254.32.36 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = 10.254.32.0/24 cluster_network = 10.254.36.0/24 [client.radosgw.gateway] debug ms = 1 debug rgw = 20 rgw host = 10.254.32.32 rgw keystone verify ssl = false rgw keystone api version = 3 rgw keystone url = https://keystone-intern.desy.de:5000 rgw keystone admin domain = default rgw keystone admin project = service rgw keystone admin user = swift rgw keystone admin password = password rgw s3 auth use keystone = true rgw swift account in url = true rgw keystone implicit tenants = true rgw swift versioning enabled = true rgw enable apis = swift, s3 rgw keystone accepted roles = admin, _member_, member, swiftoperator rgw keystone accepted admin roles = admin, _member_ keyring = /etc/ceph/client.rgw.ciosmon06.keyring [root@ciosmon06 ~]# systemctl status ceph-radosgw(a)rgw.ciosmon06.service ● ceph-radosgw(a)rgw.ciosmon06.service - Ceph rados gateway Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-10-21 17:22:04 CEST; 3min 36s ago Main PID: 3372096 (radosgw) CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw(a)rgw.ciosmon06.service └─3372096 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ciosmon06 --setuser ceph --setgroup ceph Oct 21 17:22:04 ciosmon06.desy.de systemd[1]: Started Ceph rados gateway. [root@ciosmon06 ~]# netstat -ntlup Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:7480 0.0.0.0:* LISTEN 3372096/radosgw tcp6 0 0 :::7480 :::* LISTEN 3372096/radosgw [root@ciosmon06 ceph]# ls -la total 32 drwxr-xr-x 2 root root 4096 Oct 21 18:18 . drwxr-xr-x. 93 root root 12288 Sep 24 05:25 .. -rw------- 1 root root 63 Feb 24 2020 ceph.client.admin.keyring -rw-r--r-- 1 root root 1036 Oct 21 18:18 ceph.conf -rw------- 1 root root 71 Oct 21 16:38 client.rgw.ciosmon06.keyring [root@ciosmon06 ~]# cat /etc/ceph/client.rgw.ciosmon06.keyring [client.rgw.ciosmon06] key = somekey [root@ciosmon06 ~]# ceph auth ls client.rgw.ciosmon06 key: somekey caps: [mon] allow rw caps: [osd] allow rwx Openstack ussuri on ubuntu 20.04 root@keystone:~# openstack endpoint list +----------------------------------+-----------+--------------+---------------------+---------+-----------+------------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+---------------------+---------+-----------+------------------------------------------------------------+ | 0ee9c91af2424e33a91a4c118b693301 | RegionOne | swift | object-store | True | internal | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | 1fe9f91db7f942d2a8cb2163bd09772d | RegionOne | keystone | identity | True | admin | https://keystone-intern.desy.de:5000/v3/ | | 4719a266432f45bda380c52486421e62 | RegionOne | swift | object-store | True | public | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | 95a292a5584d4e2c9f8177af3a312494 | RegionOne | keystone | identity | True | public | https://keystone-intern.desy.de:5000/v3/ | | e68b3990e74447bfa35a5d6aa66ca2aa | RegionOne | swift | object-store | True | admin | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | efc1df3cb21344bfb74d8e345e267606 | RegionOne | keystone | identity | True | internal | https://keystone-intern.desy.de:5000/v3/ | root@keystone:~# openstack service list +----------------------------------+-----------+---------------------+ | ID | Name | Type | +----------------------------------+-----------+---------------------+ | 4881557156f844df87d13e427d02b74a | swift | object-store | | 6d928ad49d954a179e37dc17f96c4b1e | keystone | identity | root@keystone:~# openstack user list +----------------------------------+-----------+ | ID | Name | +----------------------------------+-----------+ | ca94085253944ba3803f9d3202172b80 | admin | | ea025c992c224b5f8ca5fea0334eaf15 | swift | +----------------------------------+-----------+ root@keystone:~# openstack role assignment list --names --project service +-------+-------------------+-------+-----------------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +-------+-------------------+-------+-----------------+--------+--------+-----------+ | admin | swift@Default | | service@Default | | | False | +-------+-------------------+-------+-----------------+--------+--------+-----------+ Tests: root@it-build:~# openstack ec2 credentials create +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | access | bc6f1ad9064d4097ad7a323c8368780b | | links | {'self': 'https://keystone-intern.desy.de:5000/v3/users/926c750033e668f0af2330b1c7c72…'} | | project_id | 286f5d2b38ae4595ba9ff8129e754f54 | | secret | c6d9a58220fd4491ba8654d6d3b3c4dc | | trust_id | None | | user_id | 926c750033e668f0af2330b1c7c723a05b86fa393655369fdb1a5622ae65dac8 | +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ root@it-build:~# /usr/local/bin/aws configure AWS Access Key ID [****************e90c]: bc6f1ad9064d4097ad7a323c8368780b AWS Secret Access Key [****************6ba4]: c6d9a58220fd4491ba8654d6d3b3c4dc Default region name [default]: Default output format [None]: root@it-build:~# /usr/local/bin/aws --endpoint='http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s' s3 ls s3:// expected string or bytes-like object [root@ciosmon06 ~]# tail -f /var/log/ceph/ceph-client.rgw.ciosmon06.log 2020-10-21 18:24:24.710 7efeabf69700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:24:24.713 7efeabf69700 1 ====== req done req=0x562f3dbb08f0 op status=-2024 http_status=301 latency=0.003s ====== root@keystone:~# openstack role assignment list --names --project 286f5d2b38ae4595ba9ff8129e754f54 | grep swiftoperator | swiftoperator | myusername@closedbeta | | closedbeta@closedbeta | | | False | root@it-build:~# cat myusername-openrc-closedbeta export OS_PROJECT_DOMAIN_NAME=closedbeta export OS_USER_DOMAIN_NAME=closedbeta export OS_PROJECT_NAME=closedbeta export OS_USERNAME=myusername export OS_PASSWORD=password export OS_AUTH_URL=https://keystone-intern.desy.de:5000 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 export OS_VOLUME_API_VERSION=3 root@it-build:~# openstack container list Unrecognized schema in response body. (HTTP 401) (Request-ID: tx000000000000000000005-005f9060f7-26173f-default) [root@ciosmon06 ~]# tail -f /var/log/ceph/ceph-client.rgw.ciosmon06.log 2020-10-21 18:25:26.868 7efea9f65700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:25:26.868 7efea9f65700 1 ====== req done req=0x562f3dbb08f0 op status=0 http_status=401 latency=0s ====== 2020-10-21 18:25:27.088 7efeadf6d700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:25:27.088 7efeadf6d700 1 ====== req done req=0x562f3dbb08f0 op status=0 http_status=401 latency=0s ====== Greets Stefan Bujack

3 years, 6 months

2
2
0 0

6 PG's stuck not-active, remapped

by Mac Wynkoop

We recently did some work on the Ceph cluster, and a few disks ended up offline at the same time. There are now 6 PG's that are stuck in a "remapped" state, and this is all of their recovery states: *recovery_state: 0: name: Started/Primary/WaitActingChangeenter_time: 2020-10-21 18:48:02.034430comment: waiting for pg acting set to change1: name: Startedenter_time: 2020-10-21 18:48:01.752957* Any ideas? Mac Wynkoop

3 years, 6 months

2
2
0 0

2024

2023

2022

2021

2020

2019

ceph-users October 2020