November 2020 - ceph-users

by Vladimir Prokofev

Hi. Just want to notice that if you google for ceph python lib examples it leads to 404 https://www.google.ru/search?hl=ru&q=ceph+python+rbd https://docs.ceph.com/en/latest/rbd/api/librbdpy/ Some 3rd party sites and chinese version works fine though http://docs.ceph.org.cn/rbd/librbdpy/ https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/b…

3 years, 4 months

1
0
0 0

high memory usage in osd_pglog

by Robert Brooks

We are seeing very high osd_pglog usage in mempools for ceph osds. For example... "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluestore_alloc_bytes": 41857200, "bluestore_alloc_items": 523215, "bluestore_cache_data_bytes": 50876416, "bluestore_cache_data_items": 1326, "bluestore_cache_onode_bytes": 6814080, "bluestore_cache_onode_items": 13104, "bluestore_cache_other_bytes": 57793850, "bluestore_cache_other_items": 2599669, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_txc_bytes": 29904, "bluestore_txc_items": 42, "bluestore_writing_deferred_bytes": 733191, "bluestore_writing_deferred_items": 96, "bluestore_writing_bytes": 0, "bluestore_writing_items": 0, "bluefs_bytes": 101400, "bluefs_items": 1885, "buffer_anon_bytes": 21505818, "buffer_anon_items": 14949, "buffer_meta_bytes": 1161512, "buffer_meta_items": 13199, "osd_bytes": 1962920, "osd_items": 167, "osd_mapbl_bytes": 825079, "osd_mapbl_items": 17, "osd_pglog_bytes": 14099381936, "osd_pglog_items": 134285429, "osdmap_bytes": 734616, "osdmap_items": 26508, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "mds_co_bytes": 0, "mds_co_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, Where roughly 14g is required for pg_logs. Cluster has 106 OSD and 2432 placement groups. The pg log count for placement groups is much less than 134285429 logs. Top counts are... 1486 1.41c 883 7.3 834 7.f 683 7.13 669 7.a 623 7.5 565 7.8 560 7.1c 546 7.16 544 7.19 Summing these gives 21594 pg logs. Overall the performance of the cluster is poor, OSD memory usage is high (20-30G resident), and with a moderate workload we are seeing iowait on OSD hosts. The memory allocated to caches appears to be low, I believe because osd_pglog is taking most of the available memory. Regards, Rob -- ******************************************************************* This message was sent from RiskIQ, and is intended only for the designated recipient(s). It may contain confidential or proprietary information and may be subject to confidentiality protections. If you are not a designated recipient, you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you. *******************************************************************

3 years, 4 months

2
2
0 0

CEPH-ISCSI fails when restarting rbd-target-api and won't work anymore

by Hamidreza Hosseini

I have an issue on ceph-iscsi ( ubuntu 20 LTS and Ceph 15.2.6) after I restart rbd-target-api, it fails and not starting again: ``` sudo systemctl status rbd-target-api.service ● rbd-target-api.service - Ceph iscsi target configuration API Loaded: loaded (/lib/systemd/system/rbd-target-api.service; enabled; vendor preset: enabled) Active: deactivating (stop-sigterm) since Sat 2020-11-28 17:01:40 +0330; 20s ago Main PID: 37651 (rbd-target-api) Tasks: 55 (limit: 9451) Memory: 141.4M CGroup: /system.slice/rbd-target-api.service ├─15289 /usr/bin/python3 /usr/bin/rbd-target-api └─37651 /usr/bin/python3 /usr/bin/rbd-target-api Nov 28 14:36:53 dev11 systemd[1]: Started Ceph iscsi target configuration API. Nov 28 14:36:54 dev11 rbd-target-api[37651]: Started the configuration object watcher Nov 28 14:36:54 dev11 rbd-target-api[37651]: Processing osd blacklist entries for this node Nov 28 14:36:54 dev11 rbd-target-api[37651]: Checking for config object changes every 1s Nov 28 14:36:55 dev11 rbd-target-api[37651]: Reading the configuration object to update local LIO configuration Nov 28 14:36:55 dev11 rbd-target-api[37651]: Processing Gateway configuration Nov 28 14:36:55 dev11 rbd-target-api[37651]: Setting up iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 14:36:55 dev11 rbd-target-api[37651]: (Gateway.load_config) successfully loaded existing target definition Nov 28 17:01:40 dev11 systemd[1]: Stopping Ceph iscsi target configuration API... ``` journalctl: ``` Nov 28 17:00:01 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:01 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:04 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:04 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:06 dev11 ceph-mgr[3184]: [172.16.1.3:57002] [GET] [500] [45.074s] [admin] [513.0B] /api/health/minimal Nov 28 17:00:06 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "68eed46b-3ece-4e60-bc17-a172358f2d76"} '] Nov 28 17:00:06 dev11 ceph-mgr[3184]: [172.16.1.3:60128] [GET] [500] [45.070s] [admin] [513.0B] /api/health/minimal Nov 28 17:00:06 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "5b6fdaa2-dc70-48a7-b01f-ca554ecfec41"} '] Nov 28 17:00:07 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:07 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:11 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:11 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:11 dev11 ceph-mgr[3184]: ::ffff:127.0.0.1 - - [28/Nov/2020:17:00:11] "GET /metrics HTTP/1.1" 200 151419 "" "Prometheus/2.7.2" Nov 28 17:00:14 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:14 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:17 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:17 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:20 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:20 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:22 dev11 ceph-mgr[3184]: [172.16.1.3:59834] [GET] [500] [45.062s] [admin] [513.0B] /api/health/minimal Nov 28 17:00:22 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "1ba61331-1dfd-43e7-8ced-9f28aeb8a39c"} '] Nov 28 17:00:23 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:23 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:26 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:26 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:26 dev11 ceph-mgr[3184]: ::ffff:127.0.0.1 - - [28/Nov/2020:17:00:26] "GET /metrics HTTP/1.1" 200 151420 "" "Prometheus/2.7.2" Nov 28 17:00:27 dev11 ceph-mgr[3184]: [172.16.1.3:60132] [GET] [500] [45.081s] [admin] [513.0B] /api/health/minimal Nov 28 17:00:27 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "9c1dd49b-07fb-4c49-a033-f5d8d82d9cbe"} '] Nov 28 17:00:29 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:29 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:32 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:32 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:35 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:35 dev11 kernel: iSCSI Login negotiation failed. ``` What should I do for this problem?

3 years, 4 months

2
1
0 0

RESTful manager module deprecation

by Ernesto Puerta

Hi all, Since Ceph-Dashboard RESTful API <https://docs.ceph.com/en/latest/mgr/ceph_api/> is becoming the official RESTful API for Ceph (starting Pacific release), the proposal is to mark RESTful Module <https://docs.ceph.com/en/latest/mgr/restful/> for deprecation by Pacific and for removal on Q-release. You may find a detailed feature-gap analysis in this tracker issue <https://tracker.ceph.com/issues/47066>. We'd like to know about existing users of the RESTful module and their concerns and suggestions for this proposal. Thank you! Kind regards, Ernesto

3 years, 4 months

1
0
0 0

Re: rbd image backup best practice

by Janne Johansson

Den fre 27 nov. 2020 kl 23:21 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: > Is there a best practice or guide for backuping rbd images? > One would think that most things that apply to an iscsi mounted device would be equally valid for an RBD mount, so you might look into that for hints and tips on how to backup remote network data devices if you are seeing this from the mounting client point of view. If now, it probably matters what you are aiming for since "backup" is quite a wide concept apart from "copy of my data". Is it best practice for storing sparse images for conserving backup space, quick restores, valid images, legal archiving demands, or "want to try this weird update and be able to move backwards 30 minutes" ? The solution will be quite different depending on what the problem is, more than perhaps "what the mountpoint type is". -- May the most significant bit of your life be positive.

3 years, 4 months

2
1
0 0

Re: Manual bucket resharding problem

by Mateusz Skała

Thank You for response, how I can upload this to metadata? Is this operation safe? Regards Mateusz Skała W dniu sob., 21.11.2020 o 18:01 Amit Ghadge <amitg.b14(a)gmail.com> napisał(a): > I go through this and you need to update bucket metadata, radosgw-admin > metadata get bucket.instance:bucket:xxx > bucket.json, update two parameter > I don't remember but it's look reshard: false and next_marker set empty. > > -AmitG > On Sat, 21 Nov, 2020, 2:04 PM Mateusz Skała, <mateusz.skala(a)gmail.com> > wrote: > >> Hello Community. >> I need Your help. Few days ago I started manual resharding of one bucket >> with large objects. Unfortunately I interrupted this by Ctrl+c. At now I >> can’t start this process again. >> There is message: >> # radosgw-admin bucket reshard --bucket objects --num-shards 2 >> ERROR: the bucket is currently undergoing resharding and cannot be added >> to the reshard list at this time >> >> But list of reshard process is empty: >> # radosgw-admin reshard list >> [] >> >> # radosgw-admin reshard status --bucket objects >> [ >> { >> "reshard_status": "not-resharding", >> "new_bucket_instance_id": "", >> "num_shards": -1 >> } >> ] >> >> How can I fix this situation ? How to restore possibility resharding this >> bucket? >> And BTW is resharding process locking writes/reads on bucket? >> Regards >> Mateusz Skała >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

3 years, 4 months

2
3
0 0

rbd image backup best practice

by Marc Roos

Is there a best practice or guide for backuping rbd images?

3 years, 4 months

2
1
0 0

DB sizing for lots of large files

by Richard Thornton

Hi, Sorry to bother you all. It’s a home server setup. Three nodes (ODROID-H2+ with 32GB RAM and dual 2.5Gbit NICs), two 14TB 7200rpm SATA drives and an Optane 118GB NVMe in each node (OS boots from eMMC). Only CephFS, I'm anticipating having 50-200K files when the 50TB (4+2 EC) is full. I'm trying to address the issue of really big OSD's, not my words, a Redditor: "When you write an object to a drive with collocated db and raw space, the disk has to read/write to both sections before acking a write. That's a lot to ask a 7200 disk to handle gracefully. I believe Red Hat only supports up to 8TB because of performance concerns with larger disks. I may be wrong, but once you are shuffling through 6-10TB of data I'd think those disks are gonna be bogged down in seek time." So I want to have my two DB's on the Optane to avoid the above, am I making sense? OK, so large files have lower metadata overhead than small files. This is for a media library this probably means super low overhead, one guy I spoke to had similar setup and for 48TB used he had a 2.6GB DB? Is there a rough CephFS calculation (each file uses x bytes of metadata), I think I should be safe with 30GB, now I read I should double that (you should allocate twice the size of the biggest layer to allow for compaction) but I only have 118GB and two OSDs so I will have to go for 59GB (or whatever will fit)? I'm thinking that I might not even use 3GB but to be safe I’ll make it 30GB, lets say it settles at 5GB, when compaction comes that means I will only need 20GB and therefore no spillover? I realise that if the Optane dies both OSD's go with it, do I have to configure anything special there (CRUSH would just handle it)? Being that this a home deployment will Ceph be OK with an occasional power outage, I mean if a bluray mkv gets corrupted ripping it again is an easy fix, also I will backup the clusters data once a month? Below are the commands I got from the website: Create the volume groups: $ vgcreate ceph-block-0 /dev/sda $ vgcreate ceph-block-1 /dev/sdb Create the logical volumes: $ lvcreate -l 100%FREE -n block-0 ceph-block-0 $ lvcreate -l 100%FREE -n block-1 ceph-block-1 Create db logical volumes (118GB Optane) $ vgcreate ceph-db-0 /dev/sdc $ lvcreate -L 59GB -n db-0 ceph-db-0 $ lvcreate -L 59GB -n db-1 ceph-db-0 Create the OSDs $ ceph-volume lvm create --bluestore --data ceph-block-0/block-0 --block.db ceph-db-0/db-0 $ ceph-volume lvm create --bluestore --data ceph-block-1/block-1 --block.db ceph-db-0/db-1 Thanks. Richard

3 years, 4 months

5
5
0 0

Access/Delete RGW user with leading whitespace

by Benjamin.Zieglmeier

Hello, In our environment we have a user that has a leading whitespace in the UID. I don’t know how it was created, however I am unable to GET or DELETE it either using `radosgw-admin` or the Admin API: # radosgw-admin user list | grep rgw " rgw-prometheus", "rgw-prometheus", When I try to get info of the user directly, I get: # radosgw-admin user info --uid=" rgw-prometheus" could not fetch user info: no user info saved I’ve tried using the admin API as well while URL encoding the whitespace to %20 and get “Invalid Argument”. This user is not important in any way, however it creates issues trying to monitor the rgw usage logs using https://github.com/blemmenes/radosgw_usage_exporter I’ve considered modifying the script to ignore that user, but clearly Ceph is having troubles addressing it as well, so I figured I’d try to get to the bottom of how to remove this user. We are running 12.2.11 currently, however this cluster was built on 12.2.5 and I’m 99% certain the user was created in 12.2.5 Thanks, Ben

3 years, 4 months

1
1
0 0

Tracing in ceph

by Seena Fallah

Hi all, Does this project work with the latest zipkin apis? https://github.com/ceph/babeltrace-zipkin Also what do you prefer to trace requests for rgw and rbd in ceph? Thanks.

3 years, 4 months

2
3
0 0

2024

2023

2022

2021

2020

2019

ceph-users November 2020