- ceph-users - lists.ceph.io

Issue creating LVs within cephadm shell
by Ed Krotee 02 Dec '24

02 Dec '24

Trying to create the Block and Block.DB devices for BlueStore. Within the cephadm shell able to run the vgcreate but get the following errors and we cannot see the vg device in /dev so doesn't seem to actually create the vg, but vgs within the cephadm shell shows it, but vgs at the o/s level does not. FYI - SE Linux is disabled. stdout: Physical volume "/dev/sda" successfully created. Not creating system devices file due to existing VGs. stdout: Volume group "ceph-b80b7206-0c2e-4770-9895-51077b1d59d4" successfully created Running command: lvcreate --yes -l 5245439 -n osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9 ceph-b80b7206-0c2e-4770-9895-51077b1d59d4 stderr: /dev/ceph-b80b7206-0c2e-4770-9895-51077b1d59d4/osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9: not found: device not cleared Aborting. Failed to wipe start of new LV. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: purged osd.0 --> RuntimeError: Unable to find any LV for zapping OSD: 0 [ceph: root@ritcephstrdata09 /]# lvcreate --yes -l 5245439 -n osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9 ceph-b80b7206-0c2e-4770-9895-51077b1d59d4 /dev/ceph-b80b7206-0c2e-4770-9895-51077b1d59d4/osd-block-b992b707-c77a-412d-9286-3b3ec1d8b3e9: not found: device not cleared Aborting. Failed to wipe start of new LV. Any thoughts?

2 1

[Cephadm] Bootstrap Ceph with alternative data directory
by Jinfeng Biao 02 Dec '24

02 Dec '24

We’d like to bootstrap and deploy Ceph into alternative directory. By keeping Ceph relevant data to a separate directory mounted from a separate disk, Ceph services will start right after operating system reprovision. While deploying with –data-dir and other directory parameters, cephadm --data-dir /pd/var/lib/ceph --unit-dir /pd/etc/systemd/system --log-dir /pd/var/log/ceph --logrotate-dir /pd/etc/logrotate.d bootstrap --mon-ip xx.xx.xx.xx The bootstrap failed at mon creation. Ceph version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Non-zero exit code 1 from systemctl disable ceph.target systemctl: stderr Failed to disable unit: Unit file ceph.target does not exist. RuntimeError: Failed command: systemctl disable ceph.target: Failed to disable unit: Unit file ceph.target does not exist. Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 10889, in <module> File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 10877, in main File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 6318, in _rollback File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 2626, in _default_image File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 6463, in command_bootstrap File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 5848, in create_mon File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 3964, in deploy_daemon File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 4191, in deploy_daemon_units File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 4459, in install_base_units File "/tmp/tmpp5btnpe6.cephadm.build/__main__.py", line 2266, in call_throws RuntimeError: Failed command: systemctl disable ceph.target: Failed to disable unit: Unit file ceph.target does not exist. Ceph.target does exist in the directory specified by –unit-dir $ pwd /pd/etc/systemd/system $ ls -al total 4 drwxr-x---. 2 root root 25 Dec 2 04:12 . drwxr-x---. 3 root root 20 Nov 29 06:39 .. -rw-r-----. 1 root root 88 Dec 2 04:12 ceph.target Is this a supported approach and is it a bug? regards James Biao ************** IMPORTANT MESSAGE ************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124 AFSL and Australian credit licence 234945) or its subsidiaries. We can be contacted through our web site: commbank.com.au. If you no longer wish to receive commercial electronic messages from us, please reply to this e-mail by typing Opt Out in the subject line. **************************************************

1 0

classes crush rules new cluster
by Marc 30 Nov '24

30 Nov '24

It looks like it is not possible to create crush rules when you don't have harddrives active in this class. I am testing with new squid and did not add ssd's yet, eventhough I added class like this. ceph osd crush class create ssd I can't execute this ceph osd crush rule create-replicated replicated_ssd default host ssd Is there any way around this?

4 8

new cluser ceph osd perf = 0
by Marc 30 Nov '24

30 Nov '24

My ceph osd perf are all 0, do I need to enable module for this? osd_perf_query? Where should I find this in manuals? Or do I just need to wait? [@ target]# ceph osd perf osd commit_latency(ms) apply_latency(ms) 25 0 0 24 0 0 23 0 0 22 0 0 21 0 0 20 0 0 19 0 0 18 0 0

4 3

Additional rgw pool
by Rok Jaklič 29 Nov '24

29 Nov '24

Hi, we are already running the "default" rgw pool with some users. Data is stored in pool: pool 9 'default.rgw.buckets.data' erasure profile ec-32-profile size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 309346 lfor 0/127784/214408 flags hashpspool,ec_overwrites,backfillfull stripe_width 12288 application rgw Is it possible to create another rgw pool with diferent ec profile and associate those pools to specific s3 users? Kind regards, Rok

3 2

nfs-ganesha 5 changes
by Marc 29 Nov '24

29 Nov '24

In my old environment I have simple nfs-ganesha export like this, which is sufficent and mounts. EXPORT { Export_Id = 200; Path = /backup; Pseudo = /backup; FSAL { Name = CEPH; Filesystem = "xxxx"; User_Id = "cephfs.xxxx.bakup"; Secret_Access_Key = "xxxxx=="; } Disable_ACL = FALSE; CLIENT { Clients = 192.168.11.200; access_type = "RW"; } CLIENT { Clients = *; Access_Type = NONE; } } In the new ganesha 5 I am getting these errors. Don't really get why it wants to create a pool rados_kv_connect :CLIENT ID :EVENT :Failed to create pool: -34 rados_ng_init :CLIENT ID :EVENT :Failed to connect to cluster: -34 main :NFS STARTUP :CRIT :Recovery backend initialization failed! cephfs kernel mount with this userid is ok. User only has access to this dir. Anyone an idea what config I need to update?

2 3

internal communication network
by Michel Niyoyita 29 Nov '24

29 Nov '24

Hello team , I am creating new cluster which will be created using CEPHADM, I will use 192.168.1.0/24 as public network and 10.10.90.0/24 as internal network for osd , mon communication . would like if this command is helpful as it is my first time to use caphadm . sudo cephadm bootstrap --mon-ip 192.168.1.23 --cluster-network 10.10.90.0/24 . Kindly help me if there is alternative. Best regards Michel

3 3

[CephFS] Completely exclude some MDS rank from directory processing
by Александр Руденко 29 Nov '24

29 Nov '24

Hi, I try to distribute all top level dirs in CephFS by different MDS ranks. I have two active MDS with rank *0* and *1 *and I have 2 top dirs like */dir1* and* /dir2*. After pinning: setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir1 setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir2 I can see next INOS and DNS distribution: RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active c Reqs: 127 /s 12.6k 12.5k 333 505 1 active b Reqs: 11 /s 21 24 19 1 When I write to dir1 I can see a small amount on Reqs: in rank 1. Events in journal of MDS with rank 1: cephfs-journal-tool --rank=fs1:1 event get list 2024-11-20T12:24:42.045056+0300 0xc5c1cb UPDATE: (scatter_writebehind) A2037D53 2024-11-20T12:24:46.935934+0300 0xc5c629 SESSION: () 2024-11-20T12:24:47.192012+0300 0xc5c7cd UPDATE: (lock inest accounted scatter stat update) 2024-11-20T12:24:47.904717+0300 0xc5ca0b SESSION: () 2024-11-20T12:26:46.912719+0300 0xc5ca98 SESSION: () 2024-11-20T12:26:47.910806+0300 0xc5cc3c SESSION: () 2024-11-20T12:27:35.746239+0300 0xc5ccc9 SESSION: () 2024-11-20T12:28:46.923812+0300 0xc5ce63 SESSION: () 2024-11-20T12:28:47.903066+0300 0xc5d007 SESSION: () 2024-11-20T12:29:08.063326+0300 0xc5d094 EXPORT: () di1/A2037D53 2024-11-20T12:30:46.909621+0300 0xc5d96f SESSION: () 2024-11-20T12:30:47.908050+0300 0xc5db13 SESSION: () 2024-11-20T12:32:46.907649+0300 0xc5dba0 SESSION: () 2024-11-20T12:32:47.905962+0300 0xc5dd44 SESSION: () 2024-11-20T12:34:44.349348+0300 0xc5ddd1 SESSIONS: () But the main problem, when I stop MDS rank 1 (without any kind of standby) - FS hangs for all actions. Is this correct? Is it possible to completely exclude rank 1 from processing dir1 and not stop io when rank 1 goes down?

4 19

2024-11-28 Perf Meeting Cancelled
by Matt Vandermeulen 28 Nov '24

28 Nov '24

Hi folks, the perf meeting for today will be cancelled for US thanksgiving! As a heads up, next week will also be cancelled for Cephalocon. Thanks, Matt

1 0

rgw multisite excessive data usage on secondary zone
by Adam Prycki 28 Nov '24

28 Nov '24

Hi, I've just configured a second zones for 2 of our ceph s3 deployments and I've noticed that after initial sync secondary zone data pool are much bigger than ones on master zones. My setup consists of main zone, archive zone and sync_policy which configure directional sync from main zone to archive. here is an example from the zone I've configured today. Master zone looks like this X.rgw.buckets.data 42 1024 361 GiB 254.26k 542 GiB 0.01 3.1 PiB secondary archive zone looks like this. X-archive.rgw.buckets.data 32 32 755 GiB 716.68k 1007 GiB 0.03 2.4 PiB This archive zone was created few hours ago. Users didn't overwrite so much data to double archive zone size. (object count is almost trippled) I've checked gc list --include-all on archive zones and it's empty. I'm not sure why zone is this big. Few days ago I've also configured archive zone for different deployment. I've configured archive-zone lifecycle policy to 1 day and tried to cleanup all the buckets on archive zone. It didn't help. My other archive zone is 150% of the size of it's master zone. I've tried to force sync with `radosgw-admin data sync init` Sync worked but didn't help with excess data on the pool. I suspect it's an error during initial multisite synchronization. I've restarted RGW daemon on archive zone during initial synchronization in both cases. What else could have caused this? Are RGW daemons in multisite setups sensitive to restarts? Could similar issues happen during normal rgw restart during multisite operations? Best regards Adam Prycki

1 0