- ceph-users - lists.ceph.io

cephadm and VRF
by Karsten Nielsen 23 Apr '21

23 Apr '21

Hi cephers, I am a happy user of ceph and cephadm, we run it in a hyberconverged setup. We are moving into running bgp on to the node and have ECMP. My networking team is asking to run VRF on the nodes we will have a VRF for client traffic and a VRF for the storage network. Is that supported and will the daemons bind the right interfaces when specifying the client and cluster network ? Thanks, - Karsten

1 0

Re: s3 requires twice the space it should use
by Boris Behrens 23 Apr '21

23 Apr '21

Ah you are right. [root@s3db1 ~]# ceph daemon osd.23 config get bluestore_min_alloc_size_hdd { "bluestore_min_alloc_size_hdd": "65536" } But I also checked how many objects our s3 hold and the numbers just do not add up. There are only 26509200 objects, which would result in around 1TB "waste" if every object would be empty. I think the problem began when I updated the PG count from 1024 to 2048. Could there be an issue where the data is written twice? Am Do., 15. Apr. 2021 um 16:48 Uhr schrieb Amit Ghadge <amitg.b14(a)gmail.com >: > verify those two parameter values ,bluestore_min_alloc_size_hdd & > bluestore_min_alloc_size_sdd, If you are using hdd disk then > bluestore_min_alloc_size_hdd are applicable. > > On Thu, Apr 15, 2021 at 8:06 PM Boris Behrens <bb(a)kervyn.de> wrote: > >> So, I need to live with it? A value of zero leads to use the default? >> [root@s3db1 ~]# ceph daemon osd.23 config get bluestore_min_alloc_size >> { >> "bluestore_min_alloc_size": "0" >> } >> >> I also checked the fragmentation on the bluestore OSDs and it is around >> 0.80 - 0.89 on most OSDs. yikes. >> [root@s3db1 ~]# ceph daemon osd.23 bluestore allocator score block >> { >> "fragmentation_rating": 0.85906054329923576 >> } >> >> The problem I currently have is, that I barely keep up with adding OSD >> disks. >> >> Am Do., 15. Apr. 2021 um 16:18 Uhr schrieb Amit Ghadge < >> amitg.b14(a)gmail.com>: >> >>> size_kb_actual are actually bucket object size but on OSD level the >>> bluestore_min_alloc_size default 64KB and SSD are 16KB >>> >>> >>> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/a… >>> >>> -AmitG >>> >>> On Thu, Apr 15, 2021 at 7:29 PM Boris Behrens <bb(a)kervyn.de> wrote: >>> >>>> Hi, >>>> >>>> maybe it is just a problem in my understanding, but it looks like our s3 >>>> requires twice the space it should use. >>>> >>>> I ran "radosgw-admin bucket stats", and added all "size_kb_actual" >>>> values >>>> up and divided to TB (/1024/1024/1024). >>>> The resulting space is 135,1636733 TB. When I tripple it because of >>>> replication I end up with around 405TB which is nearly half the space of >>>> what ceph df tells me. >>>> >>>> Hope someone can help me. >>>> >>>> ceph df shows >>>> RAW STORAGE: >>>> CLASS SIZE AVAIL USED RAW USED %RAW >>>> USED >>>> hdd 1009 TiB 189 TiB 820 TiB 820 TiB >>>> 81.26 >>>> TOTAL 1009 TiB 189 TiB 820 TiB 820 TiB >>>> 81.26 >>>> >>>> POOLS: >>>> POOL ID PGS STORED >>>> OBJECTS >>>> USED %USED MAX AVAIL >>>> rbd 0 64 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> .rgw.root 1 64 99 KiB >>>> 119 >>>> 99 KiB 0 18 TiB >>>> eu-central-1.rgw.control 2 64 0 B >>>> 8 >>>> 0 B 0 18 TiB >>>> eu-central-1.rgw.data.root 3 64 1.0 MiB >>>> 3.15k >>>> 1.0 MiB 0 18 TiB >>>> eu-central-1.rgw.gc 4 64 71 MiB >>>> 32 >>>> 71 MiB 0 18 TiB >>>> eu-central-1.rgw.log 5 64 267 MiB >>>> 564 >>>> 267 MiB 0 18 TiB >>>> eu-central-1.rgw.users.uid 6 64 2.8 MiB >>>> 6.91k >>>> 2.8 MiB 0 18 TiB >>>> eu-central-1.rgw.users.keys 7 64 263 KiB >>>> 6.73k >>>> 263 KiB 0 18 TiB >>>> eu-central-1.rgw.meta 8 64 384 KiB >>>> 1k >>>> 384 KiB 0 18 TiB >>>> eu-central-1.rgw.users.email 9 64 40 B >>>> 1 >>>> 40 B 0 18 TiB >>>> eu-central-1.rgw.buckets.index 10 64 10 GiB >>>> 67.61k >>>> 10 GiB 0.02 18 TiB >>>> eu-central-1.rgw.buckets.data 11 2048 264 TiB >>>> 138.31M >>>> 264 TiB 83.37 18 TiB >>>> eu-central-1.rgw.buckets.non-ec 12 64 297 MiB >>>> 11.32k >>>> 297 MiB 0 18 TiB >>>> eu-central-1.rgw.usage 13 64 536 MiB >>>> 32 >>>> 536 MiB 0 18 TiB >>>> eu-msg-1.rgw.control 56 64 0 B >>>> 8 >>>> 0 B 0 18 TiB >>>> eu-msg-1.rgw.data.root 57 64 72 KiB >>>> 227 >>>> 72 KiB 0 18 TiB >>>> eu-msg-1.rgw.gc 58 64 300 KiB >>>> 32 >>>> 300 KiB 0 18 TiB >>>> eu-msg-1.rgw.log 59 64 835 KiB >>>> 242 >>>> 835 KiB 0 18 TiB >>>> eu-msg-1.rgw.users.uid 60 64 56 KiB >>>> 104 >>>> 56 KiB 0 18 TiB >>>> eu-msg-1.rgw.usage 61 64 37 MiB >>>> 25 >>>> 37 MiB 0 18 TiB >>>> eu-msg-1.rgw.users.keys 62 64 3.8 KiB >>>> 97 >>>> 3.8 KiB 0 18 TiB >>>> eu-msg-1.rgw.meta 63 64 607 KiB >>>> 1.60k >>>> 607 KiB 0 18 TiB >>>> eu-msg-1.rgw.buckets.index 64 64 71 MiB >>>> 119 >>>> 71 MiB 0 18 TiB >>>> eu-msg-1.rgw.users.email 65 64 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> eu-msg-1.rgw.buckets.data 66 64 2.9 TiB >>>> 1.16M >>>> 2.9 TiB 5.30 18 TiB >>>> eu-msg-1.rgw.buckets.non-ec 67 64 2.2 MiB >>>> 354 >>>> 2.2 MiB 0 18 TiB >>>> default.rgw.control 69 32 0 B >>>> 8 >>>> 0 B 0 18 TiB >>>> default.rgw.data.root 70 32 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> default.rgw.gc 71 32 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> default.rgw.log 72 32 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> default.rgw.users.uid 73 32 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> fra-1.rgw.control 74 32 0 B >>>> 8 >>>> 0 B 0 18 TiB >>>> fra-1.rgw.meta 75 32 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> fra-1.rgw.log 76 32 50 B >>>> 28 >>>> 50 B 0 18 TiB >>>> >>>> >>>> -- >>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >>>> groÃƒ¼en Saal. >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>> >> >> -- >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >> groÃƒ¼en Saal. >> > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 4

cephadm: how to create more than 1 rgw per host
by ivan＠z1storage.com 23 Apr '21

23 Apr '21

Hello, According to the documentation, there's count-per-host key to 'ceph orch', but it does not work for me: :~# ceph orch apply rgw z1 sa-1 --placement='label:rgw count-per-host:2' --port=8000 --dry-run Error EINVAL: Host and label are mutually exclusive Why it says anything about Host if I don't specify any hosts, just labels? ~# ceph orch host ls HOST ADDR LABELS STATUS s101 s101 mon rgw s102 s102 mgr mon rgw s103 s103 mon rgw s104 s104 mgr mon rgw s105 s105 mgr mon rgw s106 s106 mon rgw s107 s107 mon rgw

4 7

Zero Reclaim/Trim on RBD image
by Vikas Rana 23 Apr '21

23 Apr '21

Hi Friends, We have multiple RBD images/devices. We used fstrim to reclaim space on XFS filesystem on RBD images and it works great. Some of the RBD images, we share as block devices to Vmware using SCST. Any suggestion on how to reclaim space on these devices? The images are about 20TB but showing as 100TB used inside ceph pool. Thanks, -Vikas

2 1

Metrics for object sizes
by Szabo, Istvan (Agoda) 22 Apr '21

22 Apr '21

Hi, Is there any clusterwise metric regarding object sizes? I'd like to collect some information about the users what is the object sizes in their buckets. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Re: monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
by Ilya Dryomov 22 Apr '21

22 Apr '21

On Thu, Apr 22, 2021 at 10:16 PM Cem Zafer <cemzafer(a)gmail.com> wrote: > > This client ceph-common version is 16.2.0, here are the outputs. > > indiana@mars:~$ ceph -v > ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) > > indiana@mars:~$ dpkg -l | grep -i ceph-common > ii ceph-common 16.2.0-1focal amd64 common utilities to mount and interact with a ceph storage cluster > ii python3-ceph-common 16.2.0-1focal all Python 3 utility libraries for Ceph Is the problem local to "mars"? Are you able to "ceph -s" with 16.2.0 client from other hosts? I suspect there is something wrong with ceph.conf on mars. The error message is different: handle_auth_bad_method server allowed_methods [2] but i only support [1] instead of handle_auth_bad_method server allowed_methods [2] but i only support [2] Please run "ceph -s --debug_ms=1 --debug_monc=25 --debug_auth=25" and attach ceph.conf from mars. Thanks, Ilya

1 0

Configuring an S3 gateway
by Fabrice Bacchella 22 Apr '21

22 Apr '21

I'm trying to configure an S3 gateway with pacific and can't wrap my mind around. In the configuration file, my configuration is: [client.radosgw.fa41] rgw_data = /data/ceph/data/radosgw/$cluster.$id log_file = /data/ceph/logs/$cluster-radosgw.$id.log rgw_frontends = "beast ssl_endpoint=0.0.0.0:443 ssl_certificate=/data/ceph/conf/ceph.crt ssl_private_key=/data/ceph/conf/ceph.key" and radosgw ignore it: # /usr/bin/radosgw -d --cluster ngceph --name client.fa41 --setuser ceph --setgroup ceph 2021-04-22T20:19:44.362+0200 7fdf51416480 0 deferred set uid:gid to 167:167 (ceph:ceph) 2021-04-22T20:19:44.363+0200 7fdf51416480 0 ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable), process radosgw, pid 9780 2021-04-22T20:19:44.363+0200 7fdf51416480 0 framework: beast 2021-04-22T20:19:44.363+0200 7fdf51416480 0 framework conf key: port, val: 7480 2021-04-22T20:19:44.363+0200 7fdf51416480 1 radosgw_Main not setting numa affinity 2021-04-22T20:19:45.585+0200 7fdf51416480 0 framework: beast 2021-04-22T20:19:45.586+0200 7fdf51416480 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt 2021-04-22T20:19:45.586+0200 7fdf51416480 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key 2021-04-22T20:19:45.586+0200 7fdf51416480 0 starting handler: beast 2021-04-22T20:19:45.592+0200 7fdf51416480 0 WARNING: cannot open socket for endpoint=[::]:7480, Address family not supported by protocol 2021-04-22T20:19:45.627+0200 7fdf51416480 0 set uid:gid to 167:167 (ceph:ceph) 2021-04-22T20:19:45.811+0200 7fdf51416480 1 mgrc service_daemon_register rgw.134130 metadata {arch=x86_64,ceph_release=pacific,ceph_version=ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable),ceph_version_short=16.2.1,cpu=Intel Core Processor (Haswell, no TSX),distro=centos,distro_description=CentOS Linux 8,distro_version=8,frontend_config#0=beast port=7480,frontend_type#0=beast,hostname=fa41,id=fa41,kernel_description=#1 SMP Thu Apr 8 19:01:30 UTC 2021,kernel_version=4.18.0-240.22.1.el8_3.x86_64,mem_swap_kb=0,mem_total_kb=16211232,num_handles=1,os=Linux,pid=9780,zone_id=2dc75a54-8c59-42bc-98a8-35542fdc4e52,zone_name=default,zonegroup_id=d11b8d14-7608-4b1d-a548-09b5dd813a7a,zonegroup_name=default} I don't get what I'm missing. Is there any typo in the configuration that I'm missing ? I've verified using strace, and it reads the expected configuration file.

3 3

/ceph-osd-prestart.sh does use the configuration
by Fabrice Bacchella 22 Apr '21

22 Apr '21

I'm using a rpm build directly from ceph, ceph-osd-16.2.1-0.el8.x86_64 I tweaked the configuration with a custom cluster name and custom installation dir In the unit, there is the configuration: ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i And this script check the osd path with an hard coded value: data="/var/lib/ceph/osd/${cluster:-ceph}-$id" It doesn't use the key from the configuration file. But using ceph-conf would have solve the problem: $ CEPH_CONF=/data/ceph/conf/ngceph.conf CLUSTER=ngceph ceph-conf --name osd.1 --show-config-value osd_data /data/ceph/data/osd/ceph.1

1 0

Re: monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
by Ilya Dryomov 22 Apr '21

22 Apr '21

On Thu, Apr 22, 2021 at 9:24 PM Cem Zafer <cemzafer(a)gmail.com> wrote: > > Sorry to disturb you again but changing the value to yes doesnt affect anything. Executing simple ceph command from the client replies the following error, again. I'm not so sure it is related with that parameter. > Have you any idea what could cause the problem? > > indiana@mars:~$ ceph -s > 2021-04-22T22:19:51.305+0300 7f20ea249700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] > [errno 13] RADOS permission denied (error connecting to the cluster) This looks like a different host/client. What version of ceph-common is installed on mars (or just run "ceph -v" instead of "ceph -s")? Thanks, Ilya

1 0

RGW objects has same marker and bucket id in different buckets.
by by morphin 22 Apr '21

22 Apr '21

Hello. I have a rgw s3 user and the user have 2 bucket. I tried to copy objects from old.bucket to new.bucket with rclone. (in the rgw client server) After I checked the object with "radosgw-admin --bucket=new.bucket object stat $i" and I saw old.bucket id and marker id also old bucket name in the object stats. Is rgw doing this for deduplication or is it a bug? If it's not a bug then If I delete the old bucket what will happen to these objects???

2 4