- ceph-users - lists.ceph.io

Adding new OSDs - also adding PGs?
by Erich Weiler 04 Jun '24

04 Jun '24

Hi All, I'm going to be adding a bunch of OSDs to our cephfs cluster shortly (increasing the total size by 50%). We're on reef, and will be deploying using the cephadm method, and the OSDs are exactly the same size and disk type as the current ones. So, after adding the new OSDs, my understanding is that ceph will begin rebalancing the data. I will also probably want to increase my PGs to accommodate the new OSDs being added. My question is basically: should I wait for the rebalance to finish before increasing my PG count? Which would kick off another relabance action for the new PGs? Or, should I increase the PG count as soon as the rebalance action starts after adding the new OSDs, and it would then create new PGs and rebalance on the new OSDs at the same time? Thanks for any guidance! -erich

2 1

Setting hostnames for zonegroups via cephadm / rgw mgr module?
by Matthew Vernon 04 Jun '24

04 Jun '24

Hi, I'm using reef (18.2.2); the docs talk about setting up a multi-site setup with a spec file e.g. rgw_realm: apus rgw_zonegroup: apus_zg rgw_zone: eqiad placement: label: "rgw" but I don't think it's possible to configure the "hostnames" parameter of the zonegroup (and thus control what hostname(s) the rgws are expecting to serve)? Have I missed something, or do I need to set up the realm/zonegroup/zone, extract the zonegroup json and edit hostnames by hand? Thanks, Matthew

1 0

Update OS with clean install
by Sake Ceph 04 Jun '24

04 Jun '24

Hi all I'm working on a way to automate the OS upgrade of our hosts. This happens with a complete reinstall of the OS. What is the correct way to do this? At the moment I'm using the following: * Store host labels (we use labels to deploy the services) * Fail-over MDS and MGR services if running on the host * Set host in maintenance mode * Reinstall host with newer OS * Remove host from cluster * Configure host with correct settings (for example cephadm user SSH key etc.) * Add host to cluster again with correct labels * For OSD hosts run ceph cephadm osd activate If somebody has some advice I would gladly hear about it! KIND regards, Sake

2 2

stretched cluster new pool and second pool with nvme
by ronny.lippold 04 Jun '24

04 Jun '24

hi ... running against the wall, i need your help, again. our test stretched cluster is running fine. now i have 2 questions. whats the right way to add another pool? create pool with 4/2 and use the rule for the stretched mode, finished? the exsisting pools were automaticly set to 4/2 after "ceph mon enable_stretch_mode". the second question, we want to use ssd and nvme together. so, we need to have a second pool for class nvme. i don't know, how to setup a second crush rule for the nvme class. i thought, that i need to filter with 2 rules for the classes. is that correct? thanks for help, ronny

3 7

Re: Missing ceph data
by Eugen Block 04 Jun '24

04 Jun '24

Hi, if you can verify which data has been removed, and that client is still connected, you might find out who was responsible for that. Do you know which files in which directories are missing? Does that maybe already reveal one or several users/clients? You can query the mds daemons and inspect the session output, it shows which directories are mounted (if you use kernel client): quincy-1:~ # ceph tell mds.quincy-1.yrgpqm session ls I doubt that you'll find much in the logs if you don't have debug enabled, but it might be worth checking anyway. Zitat von dhivagar selvam <s.dhivagar.cse(a)gmail.com>: > Hi, > > We are not using cephfs snapshots. Is there any other way to find this out? > > On Thu, May 30, 2024 at 5:20 PM Eugen Block <eblock(a)nde.ag> wrote: > >> Hi, >> >> I've never heard of automatic data deletion. Maybe just some snapshots >> were removed? Or someone deleted data on purpose because of the >> nearfull state of some OSDs? And there's no trash function for cephfs >> (for rbd there is). Do you use cephfs snapshots? >> >> >> Zitat von Prabu GJ <gjprabu(a)zohocorp.com>: >> >> > Hi Team, >> > >> > >> > We are using Ceph Octopus version with a total disk size of 136 TB, >> > configured with two replicas. Currently, our usage is 57 TB, and the >> > available size is 5.3 TB. An incident occurred yesterday where >> > around 3 TB of data was deleted automatically. Upon analysis, we >> > couldn't find the reason for the deletion. All OSDs are functioning >> > properly and actively running. >> > >> > We have 3 MDS , we try to restarted all MDS services. Is there any >> > solution to recover those data. Can anyone please help us find the >> > issue? >> > >> > >> > >> > >> > >> > cluster: >> > >> > id: 0d605d58-5caf-4f76-b6bd-e12402a22296 >> > >> > health: HEALTH_WARN >> > >> > insufficient standby MDS daemons available >> > >> > 5 nearfull osd(s) >> > >> > 3 pool(s) nearfull >> > >> > 1 pool(s) have non-power-of-two pg_num >> > >> > >> > >> > services: >> > >> > mon: 4 daemons, quorum >> > download-mon3,download-mon4,download-mon1,download-mon2 (age 14h) >> > >> > mgr: download-mon2(active, since 14h), standbys: download-mon1, >> > download-mon3 >> > >> > mds: integdownload:2 >> > {0=download-mds3=up:active,1=download-mds1=up:active} >> > >> > osd: 39 osds: 39 up (since 16h), 39 in (since 4d) >> > >> > >> > >> > data: >> > >> > pools: 3 pools, 1087 pgs >> > >> > objects: 71.76M objects, 51 TiB >> > >> > usage: 105 TiB used, 31 TiB / 136 TiB avail >> > >> > pgs: 1087 active+clean >> > >> > >> > >> > io: >> > >> > client: 414 MiB/s rd, 219 MiB/s wr, 513 op/s rd, 1.22k op/s wr >> > >> > ================================================================ >> > >> > ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA >> > STATE >> > >> > 0 download-osd1 2995G 581G 14 4785k 6 6626k >> > exists,up >> > >> > 1 download-osd2 2578G 998G 84 3644k 18 10.1M >> > exists,up >> > >> > 2 download-osd3 3093G 483G 17 5114k 5 4152k >> > exists,nearfull,up >> > >> > 3 download-osd4 2757G 819G 12 996k 2 4107k >> > exists,up >> > >> > 4 download-osd5 2889G 687G 28 3355k 20 8660k >> > exists,up >> > >> > 5 download-osd6 2448G 1128G 183 3312k 10 9435k >> > exists,up >> > >> > 6 download-osd7 2814G 762G 7 1667k 4 6354k >> > exists,up >> > >> > 7 download-osd8 2872G 703G 14 1672k 15 10.5M >> > exists,up >> > >> > 8 download-osd9 2577G 999G 10 6615k 3 6960k >> > exists,up >> > >> > 9 download-osd10 2651G 924G 16 4736k 3 7378k >> > exists,up >> > >> > 10 download-osd11 2889G 687G 15 4810k 6 8980k >> > exists,up >> > >> > 11 download-osd12 2912G 664G 11 2516k 2 4106k >> > exists,up >> > >> > 12 download-osd13 2785G 791G 74 4643k 11 3717k >> > exists,up >> > >> > 13 download-osd14 3150G 426G 214 6133k 4 7389k >> > exists,nearfull,up >> > >> > 14 download-osd15 2728G 848G 11 4959k 4 6603k >> > exists,up >> > >> > 15 download-osd16 2682G 894G 13 3170k 3 2503k >> > exists,up >> > >> > 16 download-osd17 2555G 1021G 53 2183k 7 5058k >> > exists,up >> > >> > 17 download-osd18 3013G 563G 18 3497k 3 4427k >> > exists,up >> > >> > 18 download-osd19 2924G 651G 24 3534k 12 10.4M >> > exists,up >> > >> > 19 download-osd20 3003G 573G 19 5149k 3 2531k >> > exists,up >> > >> > 20 download-osd21 2757G 819G 16 3707k 9 9816k >> > exists,up >> > >> > 21 download-osd22 2576G 999G 15 2526k 8 7739k >> > exists,up >> > >> > 22 download-osd23 2758G 818G 13 4412k 16 7125k >> > exists,up >> > >> > 23 download-osd24 2862G 714G 18 4424k 6 5787k >> > exists,up >> > >> > 24 download-osd25 2792G 783G 16 1972k 9 9749k >> > exists,up >> > >> > 25 download-osd26 2397G 1179G 14 4296k 9 12.0M >> > exists,up >> > >> > 26 download-osd27 2308G 1267G 8 3149k 22 6280k >> > exists,up >> > >> > 27 download-osd29 2732G 844G 12 3357k 3 7372k >> > exists,up >> > >> > 28 download-osd28 2814G 761G 11 476k 5 3316k >> > exists,up >> > >> > 29 download-osd30 3069G 507G 15 9043k 17 5628k >> > exists,nearfull,up >> > >> > 30 download-osd31 2660G 916G 15 841k 14 7798k >> > exists,up >> > >> > 31 download-osd32 2037G 1539G 10 1153k 15 3719k >> > exists,up >> > >> > 32 download-osd33 3116G 460G 20 7704k 12 9041k >> > exists,nearfull,up >> > >> > 33 download-osd34 2847G 728G 19 5788k 4 9014k >> > exists,up >> > >> > 34 download-osd35 3088G 488G 17 7178k 7 5730k >> > exists,nearfull,up >> > >> > 35 download-osd36 2414G 1161G 27 2017k 14 7612k >> > exists,up >> > >> > 36 download-osd37 2760G 815G 17 4292k 5 10.6M >> > exists,up >> > >> > 37 download-osd38 2679G 897G 12 2610k 5 10.0M >> > exists,up >> > >> > 38 download-osd39 3013G 563G 18 1804k 7 9235k >> > exists,up >> > >> > >> > >> > >> > >> > Regards >> > >> > Prabu GJ >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 0

About placement group scrubbing state
by Phong Tran Thanh 03 Jun '24

03 Jun '24

Hi everyone, I want to know about the scrubbing state placement group, my cluster has too many pg in state scrubbing and it's increasing over time, maybe scrubbing takes too long. [image: image.png] Cluster is not problem I'm using reef version root@n1s1:~# ceph health detail HEALTH_OK I want to ask, if the scrubbing state of pg is too long, what is the matter with my cluster? scrubbing error or another problem? Thank to the community ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079(a)gmail.com Skype: tranphong079

2 1

Find PG mappings without upmap
by Xavier Trilla 03 Jun '24

03 Jun '24

Hi, I'm working on a script, and I would like to find where a specific PG would be mapped without the existing upmap . For what I can see, all the commands I can think of, will always report the current mapping, not the original mapping that crush without upmap exceptions would output. I've been trying to query ceph, or even use osdmaptool, but as osdmaptool requires the OSD map -and I don't see how I can remove the upmaps from a downloaded osdmap file-. Any ideas? Thanks! Xavier

1 0

Reef: RGW Multisite object fetch limits
by Jayanth Reddy 01 Jun '24

01 Jun '24

Hello Community, We've two zones with Reef (v18.2.1) and trying to sync over 2 billion RGW objects to the secondary zone. We've added a fresh secondary zone and each zones have 2 RGW dedicated daemons (behind LB) each and only for multisite; whereas others don't run sync threads. Strange thing is that during the sync, the master zone LB records only 1k to 1.5k HTTP requests per second from secondary zone RGWs and not more than that. This behaviour leaves us with less link utilization. I don't see any noticeable issues with the link, cluster or RGW daemons on both ends. We've already increased rgw_data_sync_spawn_window, rgw_bucket_sync_spawn_window, rgw_meta_sync_spawn_window values to more than the defaults and don't seem to improve. Has someone noticed this behaviour of secondary zone RGW daemons fetching only around 1k to 1.5k objects per second? Regards, Jayanth

2 3

rgw mgr module not shipped? (in reef at least)
by Matthew Vernon 31 May '24

31 May '24

Hi, As far as I can tell, the rgw mgr module is not shipped in the published reef Debian packages (nor, I suspect, the ubuntu ones, but I've not actually checked). Is there a reason why it couldn't just be added to ceph-mgr-modules-core ? That contains quite a large number of modules already, and the rgw one is effectively one small python file, I think... I'm using 18.2.2. Thanks, Matthew

2 1

How to create custom container that exposes a listening port?
by Robert Sander 31 May '24

31 May '24

Hi, with the custom container service it is possible to deploy other services with the cephadm orchestrator: https://docs.ceph.com/en/reef/cephadm/services/custom-container/ The application inside the container wants to listen to a port (say 8080/tcp). How do I tell podman to publish this port? I tried adding this to the service specification: extra_container_args: - "--publish 8080/tcp" but this results in this error: Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'extra_container_args' https://docs.ceph.com/en/reef/cephadm/services/#extra-container-arguments Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1