- ceph-users - lists.ceph.io

Adding osds to each nodes
by Szabo, Istvan (Agoda) 08 Feb '23

08 Feb '23

Hi, What is the safest way to add disk(s) to each of the node in the cluster? Should it be done 1 by 1 or can add all of them at once and let it rebalance? My concern is that if add all in one due to host based EC code it will block all the host. The other side if I add 1 by 1, one node will have more performance and more osds than the others which is also not a good setup, so wonder which is the safer way? (have 9 nodes with host based EC 4:2, 1 disk is going to have 4osds) Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

RGW archive zone lifecycle
by Ondřej Kukla 08 Feb '23

08 Feb '23

Hi, I have two Ceph clusters in a multi-zone setup. The first one (master zone) would be accessible to users for their interaction using RGW. The second one is set to sync from the master zone with the tier type of the zone set as an archive (to version all files). My question here is. Is there an option to set a lifecycle for the version files saved on the archive zone? For example, keep only 5 versions per file or delete version files older than one year? Thanks a lot.

2 1

Replacing OSD with containerized deployment
by mailing-lists 08 Feb '23

08 Feb '23

Dear Ceph-Users, i am struggling to replace a disk. My ceph-cluster is not replacing the old OSD even though I did: ceph orch osd rm 232 --replace The OSD 232 is still shown in the osd list, but the new hdd will be placed as a new OSD. This wouldnt mind me much, if the OSD was also placed on the bluestoreDB / NVME, but it doesn't. My steps: "ceph orch osd rm 232 --replace" remove the failed hdd. add the new one. Convert the disk within the servers bios, so that the node can have direct access on it. It shows up as /dev/sdt, enter maintenance mode reboot server drive is now /dev/sdm (which the old drive had) "ceph orch device zap node-x /dev/sdm " A new OSD is placed on the cluster. Can you give me a hint, where did I take a wrong turn? Why is the disk not being used as OSD 232? Best Ken

3 16

Cephalocon 2023 Amsterdam CFP ENDS in Less Than Five Days
by Mike Perez 07 Feb '23

07 Feb '23

Upcoming Events ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ [image: Logo] <https://ceph.io> Upcoming Events [image: Ceph23-LaunchAssets-v1_snackable-1200x627-A.png] <https://ceph.io/en/community/events/2023/cephalocon-amsterdam/> Cephalocon is the premier yearly event that brings together the global community of operators, developers, and researchers for Ceph, the open source distributed storage system designed to provide excellent performance, reliability, and scalability. Join new and existing community members worldwide to learn more about Ceph and the project's future from the developers writing the code and the operators deploying it at scale. You can take advantage of early bird registration <https://events.linuxfoundation.org/cephalocon/register/> pricing rates before *February 26th*. Does your company want to show support for the Ceph community, be recognized, and reach an audience of passionate Ceph users? You can review the sponsorship prospectus <https://events.linuxfoundation.org/cephalocon/sponsor/> and contact sponsorships(a)ceph.io to discuss your custom sponsorship package. *The CFP ends in 5 days - February 12th at 23:59 PST.* *All talks are in-person.* Cephalocon Event Page <https://ceph.io/en/community/events/2023/cephalocon-amsterdam/> Upcoming Ceph Days [image: ceph-days-socal.png] <https://ceph.io/en/community/events/2023/ceph-days-socal/> Ceph Days Southern California <https://ceph.io/en/community/events/2023/ceph-days-socal/> Ceph is coming to Pasadena, CA, co-located with SCALE, on *March 9th*. Registration is open, and the CFP has been extended to *February 9th | 2 DAYS LEFT.* [image: Ceph Days Webpage Banner.png] <https://ceph.io/en/community/events/2023/ceph-days-nyc/> Ceph Days New York City <https://ceph.io/en/community/events/2023/ceph-days-nyc/> Join us in the big city for a big event you want to attend on *February 21st*. Hear from Ceph users providing massively scalable storage clusters and how they manage them. Thank you to our sponsor Tech At Bloomberg <https://www.bloomberg.com/company/values/tech-at-bloomberg/>! [image: Link icon] <https://ceph.io> [image: Twitter icon] <https://twitter.com/ceph> [image: YouTube icon] <https://youtube.com/c/cephstorage> [image: LinkedIn icon] <https://www.linkedin.com/company/ceph/> [image: Facebook icon] <https://www.facebook.com/cephstorage/> z7bx26gewrsc

1 0

Nautilus to Octopus when RGW already on Octopus
by r.burrowes＠auckland.ac.nz 07 Feb '23

07 Feb '23

We are finally going to upgrade our Ceph from Nautilus to Octopus, before looking at moving onward. We are still on Ubuntu 18.04, so once on Octopus, we will then upgrade the OS to 20.04, ready for the next upgrade. Unfortunately, we have already upgraded our rados gateways to Ubuntu 20.04, last Sept, which had the side effect of upgrading the RGWs to Octopus. So I'm looking to downgrade the rados gateways, back to Nautilus, just to be safe. We can then do the upgrade in the right order. I have no idea if the newer Octopus rados gateways will have altered any metadata, that would affect a downgrade back to Nautilus. Any advise.

3 3

ceph-fuse in infinite loop reading objects without client requests
by Andras Pataki 07 Feb '23

07 Feb '23

We've been running into a strange issue with ceph-fuse on some nodes lately. After some job runs on the node (and finishes or gets killed), ceph-fuse gets stuck busy requesting objects from the OSDs without any processes on the node using cephfs. When this happens, ceph-fuse uses 2-3 cores, spinning in what seems like an infinite loop making objecter requests to various OSDs of files that were perhaps requested by some process that is long gone. The same object gets requested every few seconds cycling through a list of objects. This close to saturates the traffic in the 25Gbps NIC of the node. I have a one minute log file with debug_objecter/objectcacher/client/context/finisher set to 20 that traces this, but I don't see why this is happening exactly. Can someone with a better understanding on the client->cache->OSD flow share some insights into what is going wrong here? Here is an example: 2023-02-03T14:42:47.593-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42777727 osd.4349 2023-02-03T14:43:00.056-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42778921 osd.4349 2023-02-03T14:43:02.048-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42779104 osd.4349 2023-02-03T14:43:05.392-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42779408 osd.4349 2023-02-03T14:43:10.076-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42779847 osd.4349 2023-02-03T14:43:13.288-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42780156 osd.4349 2023-02-03T14:43:18.908-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42780697 osd.4349 2023-02-03T14:43:29.056-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42781660 osd.4349 2023-02-03T14:43:33.707-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002b '@9' '@9' [read 0~25165824] tid 42782079 osd.4349 2023-02-03T14:42:41.609-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42777194 osd.3251 2023-02-03T14:42:49.809-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42777954 osd.3251 2023-02-03T14:43:07.884-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42779646 osd.3251 2023-02-03T14:43:16.736-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42780500 osd.3251 2023-02-03T14:43:22.160-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42781009 osd.3251 2023-02-03T14:43:31.603-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42781892 osd.3251 2023-02-03T14:43:35.503-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 10097d5e07f.0000002c '@9' '@9' [read 0~25165824] tid 42782258 osd.3251 Taking a specific object as an example: 100dbad3fce.000000b3 2023-02-03T14:42:46.293-0500 7fffdbfff700 10 objectcacher readx extent(100dbad3fce.000000b3 (179) in @9 0~25165824 -> [729808896,25165824]) 2023-02-03T14:42:46.293-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) map_read 100dbad3fce.000000b3 0~25165824 2023-02-03T14:42:46.293-0500 7fffdbfff700 20 objectcacher.object(100dbad3fce.000000b3/head) map_read miss 25165824 left, bh[ 0x7fffb0f461a0 0~25165824 0x7fffb0e4b720 (0) v 0 missing] waiters = {} ... a few times the above ... ... then an OSD read ... 2023-02-03T14:42:48.557-0500 7fffdbfff700 10 objectcacher readx extent(100dbad3fce.000000b3 (179) in @9 0~25165824 -> [50331648,25165824]) 2023-02-03T14:42:48.557-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) map_read 100dbad3fce.000000b3 0~25165824 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 objectcacher.object(100dbad3fce.000000b3/head) map_read miss 25165824 left, bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (0) v 0 missing] waiters = {} 2023-02-03T14:42:48.557-0500 7fffdbfff700 7 objectcacher bh_read on bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (0) v 0 missing] waiters = {} outstanding reads 170 2023-02-03T14:42:48.557-0500 7fffdbfff700 10 client.151672236.objecter _op_submit op 0x7fffb12de170 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 client.151672236.objecter _calc_target epoch 2609365 base 100dbad3fce.000000b3 @9 precalc_pgid 0 pgid 0.0 is_read 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 client.151672236.objecter _calc_target target 100dbad3fce.000000b3 @9 -> pgid 9.502eec40 2023-02-03T14:42:48.557-0500 7fffdbfff700 10 client.151672236.objecter _calc_target raw pgid 9.502eec40 -> actual 9.6c40s0 acting [975,3712,536,3462,2276,1365,2726,1648,4053] primary 975 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 client.151672236.objecter _get_session s=0x7fff9824d360 osd=975 3 2023-02-03T14:42:48.557-0500 7fffdbfff700 10 client.151672236.objecter _op_submit oid 100dbad3fce.000000b3 '@9' '@9' [read 0~25165824] tid 42777830 osd.975 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 client.151672236.objecter get_session s=0x7fff9824d360 osd=975 3 2023-02-03T14:42:48.557-0500 7fffdbfff700 15 client.151672236.objecter _session_op_assign 975 42777830 2023-02-03T14:42:48.557-0500 7fffdbfff700 15 client.151672236.objecter _send_op 42777830 to 9.6c40s0 on osd.975 2023-02-03T14:42:48.557-0500 7fffdbfff700 20 client.151672236.objecter put_session s=0x7fff9824d360 osd=975 4 2023-02-03T14:42:48.557-0500 7fffdbfff700 5 client.151672236.objecter 170 in flight ... 2023-02-03T14:42:50.045-0500 7fffdbfff700 10 objectcacher readx extent(100dbad3fce.000000b3 (179) in @9 0~25165824 -> [729808896,25165824]) 2023-02-03T14:42:50.045-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) map_read 100dbad3fce.000000b3 0~25165824 2023-02-03T14:42:50.045-0500 7fffdbfff700 20 objectcacher.object(100dbad3fce.000000b3/head) map_read rx bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (0) v 0 rx] waiters = { 0->[0x7fffb0798310, ]} ... read reply ... 2023-02-03T14:42:51.653-0500 7fffeb7fe700 10 client.151672236.objecter ms_dispatch 0x55555e305810 osd_op_reply(42777830 100dbad3fce.000000b3 [read 0~25165824 out=25165824b] v0'0 uv471599 ondisk = 0) v8 2023-02-03T14:42:51.653-0500 7fffeb7fe700 10 client.151672236.objecter in handle_osd_op_reply 2023-02-03T14:42:51.653-0500 7fffeb7fe700 7 client.151672236.objecter handle_osd_op_reply 42777830 ondisk uv 471599 in 9.6c40 attempt 0 2023-02-03T14:42:51.653-0500 7fffeb7fe700 10 client.151672236.objecter op 0 rval 0 len 25165824 2023-02-03T14:42:51.653-0500 7fffeb7fe700 15 client.151672236.objecter handle_osd_op_reply completed tid 42777830 2023-02-03T14:42:51.653-0500 7fffeb7fe700 15 client.151672236.objecter _finish_op 42777830 2023-02-03T14:42:51.653-0500 7fffeb7fe700 20 client.151672236.objecter put_session s=0x7fff9824d360 osd=975 5 2023-02-03T14:42:51.653-0500 7fffeb7fe700 15 client.151672236.objecter _session_op_remove 975 42777830 2023-02-03T14:42:51.653-0500 7fffeb7fe700 5 client.151672236.objecter 169 in flight 2023-02-03T14:42:51.657-0500 7fffdbfff700 10 finisher(0x55555e43bf90) finisher_thread doing [0x7fffb0f2ad60,0] 2023-02-03T14:42:51.657-0500 7fffdbfff700 7 objectcacher bh_read_finish 100dbad3fce.000000b3/head tid 42770553 0~25165824 (bl is 25165824) returned 0 outstanding reads 170 2023-02-03T14:42:51.657-0500 7fffdbfff700 20 objectcacher checking bh bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (0) v 0 rx] waiters = { 0->[0x7fffb0798310, ]} 2023-02-03T14:42:51.657-0500 7fffdbfff700 10 objectcacher bh_read_finish read bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (25165824) v 0 clean firstbyte=5] waiters = {} 2023-02-03T14:42:51.657-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) try_merge_bh bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (25165824) v 0 clean firstbyte=5] waiters = {} 2023-02-03T14:42:51.657-0500 7fffdbfff700 20 objectcacher finishing waiters 0x7fffb0798310 ... 2023-02-03T14:42:51.657-0500 7fffdbfff700 10 objectcacher readx extent(100dbad3fce.000000b3 (179) in @9 0~25165824 -> [50331648,25165824]) 2023-02-03T14:42:51.657-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) map_read 100dbad3fce.000000b3 0~25165824 2023-02-03T14:42:51.657-0500 7fffdbfff700 20 objectcacher.object(100dbad3fce.000000b3/head) map_read hit bh[ 0x7fffb123acd0 0~25165824 0x7fffb0e4b720 (25165824) v 0 clean firstbyte=5] waiters = {} ... a few times the above ... 2023-02-03T14:42:55.085-0500 7fffacff9700 10 objectcacher trim trimming object[100dbad3fce.000000b3/head oset 0x7fffd2895870 wr 0/0] 2023-02-03T14:42:55.085-0500 7fffacff9700 10 objectcacher close_object object[100dbad3fce.000000b3/head oset 0x7fffd2895870 wr 0/0] ... 2023-02-03T14:42:55.293-0500 7fffdbfff700 10 objectcacher readx extent(100dbad3fce.000000b3 (179) in @9 0~25165824 -> [50331648,25165824]) 2023-02-03T14:42:55.293-0500 7fffdbfff700 10 objectcacher.object(100dbad3fce.000000b3/head) map_read 100dbad3fce.000000b3 0~25165824 2023-02-03T14:42:55.293-0500 7fffdbfff700 20 objectcacher.object(100dbad3fce.000000b3/head) map_read miss 25165824 left, bh[ 0x7fffb1c36480 0~25165824 0x7fffb0dfefa0 (0) v 0 missing] waiters = {} ... some more times ... ... then eventually another OSD read for the same object - and the cycle repeats ... The client is ceph-fuse Pacific 16.2.10, the back end ceph is Octopus 15.2.7. The cluster is healthy during this time, no down OSDs, recoveries, etc. Here is the tracker: https://tracker.ceph.com/issues/58640 Any insight/help would be appreciated, Andras

2 1

Re: Removing Rados Gateway in ceph cluster
by Eugen Block 07 Feb '23

07 Feb '23

Please send responses to the mailing-list. If the orchestrator is available, please share also this output (mask sensitive data): ceph orch ls rgw --export --format yaml Which ceph version is this? The command 'ceph dashboard get-rgw-api-host' was removed between Octopus and Pacific, that's why I asked for your ceph version. I also forgot that mgr/dashboard/RGW_API_HOST was used until Octopus, in Pacific it's not applied anymore. I'll need to check how it is determined now. Zitat von Michel Niyoyita <micou12(a)gmail.com>: > Hello Eugen, > > Thanks for your reply , > > I am trying the shared command but no output . > > root@ceph-mon1:~# ceph config dump | grep mgr/dashboard/RGW_API_HOST > root@ceph-mon1:~# ceph config dump | grep mgr/dashboard/ > mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST > http://10.10.110.196:9093 > * > mgr advanced mgr/dashboard/GRAFANA_API_PASSWORD XXXX > > * > mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false > > * > mgr advanced mgr/dashboard/GRAFANA_API_URL > https://10.10.110.198:3000 > * > mgr advanced mgr/dashboard/GRAFANA_API_USERNAME admin > > * > mgr advanced mgr/dashboard/PROMETHEUS_API_HOST > http://10.10.110.196:9092 > * > mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY > XXXX > * > mgr advanced mgr/dashboard/RGW_API_SECRET_KEY > XXXX > * > mgr advanced mgr/dashboard/RGW_API_SSL_VERIFY false > > * > mgr advanced mgr/dashboard/ceph-mon1/server_addr 10.10.110.196 > > * > mgr advanced mgr/dashboard/ceph-mon2/server_addr 10.10.110.197 > > * > mgr advanced mgr/dashboard/ceph-mon3/server_addr 10.10.110.198 > > * > mgr advanced mgr/dashboard/motd {"message": > "WELCOME TO AOS ZONE 3 STORAGE CLUSTER", "md5": > "87149a6798ce42a7e990bc8584a232cd", "severity": "info", "expires": ""} * > mgr advanced mgr/dashboard/server_port 8443 > > * > mgr advanced mgr/dashboard/ssl true > > * > mgr advanced mgr/dashboard/ssl_server_port 8443 > > > for the second one it seems is not valid > > root@ceph-mon1:~# ceph dashboard get-rgw-api-host > no valid command found; 10 closest matches: > dashboard set-jwt-token-ttl <seconds:int> > dashboard get-jwt-token-ttl > dashboard create-self-signed-cert > dashboard grafana dashboards update > dashboard get-account-lockout-attempts > dashboard set-account-lockout-attempts <value> > dashboard reset-account-lockout-attempts > dashboard get-alertmanager-api-host > dashboard set-alertmanager-api-host <value> > dashboard reset-alertmanager-api-host > Error EINVAL: invalid command > root@ceph-mon1:~# > > > Kindly check the output . > > Best Regards > > Michel > * > > On Mon, Feb 6, 2023 at 2:06 PM Eugen Block <eblock(a)nde.ag> wrote: > >> Hi, >> >> can you paste the output of: >> >> ceph config dump | grep mgr/dashboard/RGW_API_HOST >> >> Does it match your desired setup? Depending on the ceph version (and >> how ceph-ansible deploys the services) you could also check: >> >> ceph dashboard get-rgw-api-host >> >> I'm not familiar with ceph-ansible, but if you shared your rgw >> definitions and the respective ceph output we might be able to assist >> resolving this. >> >> Regards, >> Eugen >> >> Zitat von Michel Niyoyita <micou12(a)gmail.com>: >> >> > Hello team, >> > >> > I have a ceph cluster deployed using ceph-ansible , running on ubuntu >> 20.04 >> > OS which have 6 hosts , 3 hosts for OSD and 3 hosts used as monitors and >> > managers , I have deployed RGW on all those hosts and RGWLOADBALENCER on >> > top of them , for testing purpose , I have switched off one OSD , to >> check >> > if the rest can work properly , The test went well as expected, >> > unfortunately while coming back an OSD , the RGW failed to connect >> through >> > the dashboard. below is the message : >> > The Object Gateway Service is not configuredError connecting to Object >> > GatewayPlease consult the documentation >> > < >> https://docs.ceph.com/en/latest/mgr/dashboard/#enabling-the-object-gateway-… >> > >> > on >> > how to configure and enable the Object Gateway management functionality. >> > >> > would like to ask how to solve that issue or how can I proceed to remove >> > completely RGW and redeploy it after . >> > >> > >> > root@ceph-mon1:~# ceph -s >> > cluster: >> > id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 >> > health: HEALTH_OK >> > >> > services: >> > mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 72m) >> > mgr: ceph-mon2(active, since 71m), standbys: ceph-mon3, ceph-mon1 >> > osd: 48 osds: 48 up (since 79m), 48 in (since 3d) >> > rgw: 6 daemons active (6 hosts, 1 zones) >> > >> > data: >> > pools: 9 pools, 257 pgs >> > objects: 59.49k objects, 314 GiB >> > usage: 85 TiB used, 348 TiB / 433 TiB avail >> > pgs: 257 active+clean >> > >> > io: >> > client: 2.0 KiB/s wr, 0 op/s rd, 0 op/s wr >> > >> > Kindly help >> > >> > Best Regards >> > >> > Michel >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

4 9

Rotate lockbox keyring
by Zhongzhou Cai 07 Feb '23

07 Feb '23

Hi, I'm on Ceph 16.2.10, and I'm trying to rotate the ceph lockbox keyring. I used ceph-authtool to create a new keyring, and used `ceph auth import -i <new-keyring>` to update the lockbox keyring. I also updated the keyring file, which is /var/lib/ceph/osd/ceph-<osd-id>/lockbox.keyring. I tried `systemctl restart ceph-volume@lvm-<osd-id>-<osd-fsid>.service`, the command succeeded. Then I rebooted the node, ceph-volume failed because the lockbox.keyring file was overwritten with the old key, which doesn't match the lockbox keyring in `ceph auth get`. Does anyone know where it gets the lockbox.keyring during reboot? Thanks, Zhongzhou Cai

1 1

Rotate lockbox keyring
by Zhongzhou Cai 06 Feb '23

06 Feb '23

Hi, I'm on Ceph 16.2.10, and I'm trying to rotate the ceph lockbox keyring. I used ceph-authtool to create a new keyring, and used `ceph auth import -i <new-keyring>` to update the lockbox keyring. I also updated the keyring file, which is /var/lib/ceph/osd/ceph-<osd-id>/lockbox.keyring. Then I ran `systemctl restart ceph-volume@lvm-<osd-id>-<osd-fsid>.service`, the command succeeded. Then I rebooted the node, ceph-volume failed because the lockbox.keyring file was overwritten with the old key. Does anyone know where it gets the lockbox.keyring during reboot? Thanks, Zhongzhou Cai

1 0

Ceph Pacific 16.2.11 : ceph-volume does not like LV with the same name in different VG
by Gilles Mocellin 06 Feb '23

06 Feb '23

Hello ! It seems like ceph-volume from Ceph Pacific 16.2.11 has a problem with same LV names in different VG. I use ceph-ansible (stable-6), with a pre-existing LVM configuration. Here's the error : TASK [ceph-osd : include_tasks scenarios/lvm.yml] ************************************************************************************************************************************** Monday 06 February 2023 16:13:55 +0100 (0:00:00.065) 0:03:41.576 ******* included: /home/cephadmin/ceph-ansible/roles/ceph-osd/tasks/scenarios/lvm.yml for fidcllabs-sto-01.labs.fidcl.cloud, fidcllabs-sto-02.labs.fidcl.cloud TASK [ceph-osd : use ceph-volume to create bluestore osds] ***************************************************************************************************************************** Monday 06 February 2023 16:13:55 +0100 (0:00:00.121) 0:03:41.698 ******* failed: [fidcllabs-sto-01.labs.fidcl.cloud] (item={'data': 'data-lv1', 'data_vg': 'data-vg1', 'crush_device_class': 'sas15k'}) => changed=false ansible_loop_var: item item: crush_device_class: sas15k data: data-lv1 data_vg: data-vg1 msg: 'Could not decode json output: from the command [''ceph-volume'', ''--cluster'', ''ceph'', ''lvm'', ''list'', ''data-vg1/data-lv1'', ''--format=json'']' rc: 1 If I execute the ceph-volume command myself on one target host, I get : fcadmin@fidcllabs-sto-01:~$ sudo ceph-volume --cluster ceph lvm list data-vg1/data-lv1 --format=json --> RuntimeError: Filters {'lv_name': 'data-lv1'} matched more than 1 LV present on this host. My LVs : 1 fcadmin@fidcllabs-sto-01:~$ sudo lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert data-lv1 data-vg1 -wi-ao---- <1024.00g data-lv1 data-vg2 -wi-ao---- <1024.00g data-lv1 data-vg3 -wi-ao---- <2.00t data-lv1 data-vg4 -wi-ao---- <2.00t logs sys -wi-ao---- 3.81g root sys -wi-ao---- <15.26g swap sys -wi-ao---- <7.63g unused sys -wi-a----- <43.30g Yes, all data LVs have the same name, but under a different VG. I look at the tracker but don't find a clear corresponding issue. I will certainly open one if there's nothing known around here.

1 0