- ceph-users - lists.ceph.io

Ceph/daemon container lvm tools don’t work
by Gaël THEROND 30 Nov '23

30 Nov '23

Is there anyone using containerized CEPH over CentOS Stream 9 Hosts already? I think there is a pretty big issue in here if CEPH images are built over CentOS but never tested against it.

1 0

Recommended architecture
by Francisco Arencibia Quesada 30 Nov '23

30 Nov '23

Hello again guys, Can you recommend me a book that explains best practices with Ceph, for example is it okay to have mon,mgr, osd in the same virtual machine, what is the recommended architecture according to your experience? Because by default is doing this: Cluster Ceph | +----------------------------+----------------------------+ | | | |10.0.0.52 |10.0.0.194 |10.0.0.229 +-----------+-----------+ +-----------+-----------+ +-----------+-----------+ |[node01.jotelulu.space]| [node02.jotelulu.space] |[node03.jotelulu.space]| | OSD +----+ OSD +----+ OSD | | Monitor Daemon | | Monitor Daemon | Monitor Daemon | | Manager Daemon | |Manager Daemon(standby) | | | +-----------------------+ +-----------------------+ +-----------------------+ -- Regards *Francisco Arencibia Quesada.* *DevOps Engineer*

3 2

Public/private network
by Albert Shih 30 Nov '23

30 Nov '23

Hi everyone. Status : Installing a ceph cluster Version : 17.2.7 Quincy OS : Debian 11. Each of my server got two ip address. One public and one private. When I'm trying to deploy my cluster with on a server server1 (the hostname) with cephadm bootstrap --mon-id hostname --mon-ip IP_PRIVATE --cluster-network PRIVATE_SUB I end up with private network for ceph config get mon public_network So I try to change it with ceph config set mon public_network PUBLIC_SUB still with lsof -i |grep -i listen I got ceph-mgr 31427 ceph 49u IPv4 119937 0t0 TCP server1-ceph.private.:7150 (LISTEN) node_expo 31572 nobody 3u IPv6 65495 0t0 TCP *:9100 (LISTEN) alertmana 31573 nobody 3u IPv6 21377 0t0 TCP *:9094 (LISTEN) alertmana 31573 nobody 8u IPv6 136298 0t0 TCP *:9093 (LISTEN) prometheu 31757 nobody 7u IPv6 109680 0t0 TCP *:9095 (LISTEN) grafana 31758 node-exporter 11u IPv6 100726 0t0 TCP *:3000 (LISTEN) ceph-mon 31850 ceph 27u IPv4 139664 0t0 TCP server1-ceph.private.:3300 (LISTEN) ceph-mon 31850 ceph 28u IPv4 139665 0t0 TCP server1-ceph.private.:6789 (LISTEN) So the ceph-mon listen on the private interface. Is this something normal ? Because according to https://access.redhat.com/documentation/fr-fr/red_hat_ceph_storage/5/html/c… only the OSD should listen on private network. Is they are anyway to configure booth public_network and private_network with cephadm bootstrap ? Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 30 nov. 2023 18:27:08 CET

2 1

MDS_DAMAGE in 17.2.7 / Cannot delete affected files
by Sebastian Knust 30 Nov '23

30 Nov '23

Hi, After updating from 17.2.6 to 17.2.7 with cephadm, our cluster went into MDS_DAMAGE state. We had some prior issues with faulty kernel clients not releasing capabilities, therefore the update might just be a coincidence. `ceph tell mds.cephfs:0 damage ls` lists 56 affected files all with these general details: { "damage_type": "dentry", "id": 123456, "ino": 1234567890, "frag": "*", "dname": "some-filename.ext", "snap_id": "head", "path": "/full/path/to/file" } The behaviour upon trying to access file information in the (Kernel mounted) filesystem is a bit inconsistent. Generally, the first `stat` call seems to result in "Input/output error", the next call provides all `stat` data as expected from an undamaged file. The file can be read with `cat` with full and correct content (verified with backup) once the stat call succeeds. Scrubbing the affected subdirectories with `ceph tell mds.cephfs:0 scrub start /path/to/dir/ recursive,repair,force` does not fix the issue. Trying to delete the file results in an "Input/output error". If the stat calls beforehand succeeded, this also crashes the active MDS with these messages in the system journal: > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: mds.0.cache.den(0x10012271195 DisplaySettings.json) newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 | inodepin=1 0x56413e1e2780] > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 state=1073741824 | inodepin=1 0x56413e1e2780] > Nov 24 14:21:15 iceph-18.servernet ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]: 2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 mds.0.cache.den(0x10012271195 DisplaySettings.json) newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x1001> > Nov 24 14:21:15 iceph-18.servernet ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]: 2023-11-24T13:21:15.654+0000 7f3fdcde0700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012> > Nov 24 14:21:15 iceph-18.servernet ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time 2023-11-24T13:21:15.655088+0000 > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 time 2023-11-24T13:21:15.655088+0000 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc: 937: ceph_abort_msg("abort() called") > > ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) > 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd7) [0x7f3fe5a1cb03] > 2: (MDSRank::abort(std::basic_string_view<char, std::char_traits<char> >)+0x7d) [0x5640f2e6fa2d] > 3: (CDentry::check_corruption(bool)+0x740) [0x5640f30e4820] > 4: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x5640f2f41877] > 5: (EOpen::add_clean_inode(CInode*)+0x121) [0x5640f2f49fc1] > 6: (Locker::adjust_cap_wanted(Capability*, int, int)+0x426) [0x5640f305e036] > 7: (Locker::process_request_cap_release(boost::intrusive_ptr<MDRequestImpl>&, client_t, ceph_mds_request_release const&, std::basic_string_view<char, std::char_traits<char> >)+0x599) [0x5640f307f7e9] > 8: (Server::handle_client_request(boost::intrusive_ptr<MClientRequest const> const&)+0xc06) [0x5640f2f2a7c6] > 9: (Server::dispatch(boost::intrusive_ptr<Message const> const&)+0x13c) [0x5640f2f2ef6c] > 10: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5db) [0x5640f2e7727b] > 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x5640f2e778bc] > 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1bf) [0x5640f2e60c2f] > 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7f3fe5c97ed8] > 14: (DispatchQueue::entry()+0x50f) [0x7f3fe5c9531f] > 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe5d5f381] > 16: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe4a0b1ca] > 17: clone() Deleting the file with cephfs-shell also does give Input/output error (5). Does anyone have an idea on how to proceed here? I am perfectly fine with loosing the affected files, they can all be easily restored from backup. Cheers Sebastian

3 5

rook-ceph RAW USE / DATA size difference reported after osd resize operation
by merp 30 Nov '23

30 Nov '23

Hi, I am set to resize OSDs in ceph cluster to extend overall cluster capacity, by adding 40GB to each of disk and noticed that after disk resize and OSD restart RAW USE size grows proportionally to new size, ex. by 20GB while DATA remains the same, which makes new space not readily available. Here is the osd output of cluster: kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1 hdd 0.09769 1.00000 100 GiB 83 GiB 82 GiB 164 MiB 891 MiB 17 GiB 82.79 1.00 77 up 3 hdd 0.09769 1.00000 100 GiB 83 GiB 82 GiB 355 MiB 772 MiB 17 GiB 82.74 1.00 84 up 2 hdd 0.09769 1.00000 100 GiB 84 GiB 82 GiB 337 MiB 1.3 GiB 16 GiB 83.88 1.01 82 up 4 hdd 0.09769 1.00000 140 GiB 125 GiB 84 GiB 148 MiB 919 MiB 15 GiB 89.24 1.07 80 up 6 hdd 0.09769 1.00000 140 GiB 106 GiB 104 GiB 333 MiB 1015 MiB 34 GiB 75.47 0.91 107 up 7 hdd 0.09769 1.00000 140 GiB 118 GiB 97 GiB 351 MiB 1.2 GiB 22 GiB 84.48 1.02 101 up TOTAL 720 GiB 598 GiB 531 GiB 1.6 GiB 6.1 GiB 122 GiB 83.10 MIN/MAX VAR: 0.91/1.07 STDDEV: 4.06 The OSDs I managed to extend are 7, 6 and 4. Only OSD number 6 detected new size and did not inflate RAW USE, OSD 7 and 4 have RAW USE vs DATA gap. kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 720 GiB 122 GiB 598 GiB 598 GiB 83.10 TOTAL 720 GiB 122 GiB 598 GiB 598 GiB 83.10 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 449 KiB 2 1.3 MiB 0 16 GiB cephfs-metadata 2 16 832 MiB 245.62k 2.4 GiB 4.80 16 GiB cephfs-replicated 3 128 176 GiB 545.23k 530 GiB 91.63 16 GiB replicapool 4 32 19 B 2 12 KiB 0 16 GiB This reports nearly 600GB used, while it should be more like 530GB as cephfs-replicated pool is reporting its data usage. Any ideas why is this happening? Should I continue with extension of all OSDs to 140GB to see if that makes a difference? Br, merp.

1 0

error deploying ceph
by Francisco Arencibia Quesada 30 Nov '23

30 Nov '23

Hello guys, This situation is driving me crazy, I have tried to deploy a ceph cluster, in all ways possible, even with ansible and at some point it breaks. I'm using Ubuntu 22.0.4. This is one of the errors I'm having, some problem with ceph-exporter. Please could you help me, I have been dealing with this for like 5 days. Kind regards root@node1-ceph:~# cephadm bootstrap --mon-ip 10.0.0.52 Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit systemd-timesyncd.service is enabled and running Repeating the final host check... docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit systemd-timesyncd.service is enabled and running Host looks OK Cluster fsid: 4ce3a92a-8ddd-11ee-9b23-6341187f70c1 Verifying IP 10.0.0.52 port 3300 ... Verifying IP 10.0.0.52 port 6789 ... Mon IP `10.0.0.52` is in CIDR network `10.0.0.0/24` Mon IP `10.0.0.52` is in CIDR network `10.0.0.0/24` Mon IP `10.0.0.52` is in CIDR network `10.0.0.1/32` Mon IP `10.0.0.52` is in CIDR network `10.0.0.1/32` Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Pulling container image quay.io/ceph/ceph:v17... Ceph version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting mon public_network to 10.0.0.1/32,10.0.0.0/24 Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 9283 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr not available, waiting (3/15)... mgr not available, waiting (4/15)... mgr not available, waiting (5/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Generating ssh key... Wrote public SSH key to /etc/ceph/ceph.pub Adding key to root@localhost authorized_keys... Adding host node1-ceph... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Deploying ceph-exporter service with default placement... Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=node1-ceph -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4ce3a92a-8ddd-11ee-9b23-6341187f70c1:/var/log/ceph:z -v /tmp/ceph-tmp6yz3vt5s:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfhd01qwu:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch apply ceph-exporter /usr/bin/ceph: stderr Error EINVAL: Usage: /usr/bin/ceph: stderr ceph orch apply -i <yaml spec> [--dry-run] /usr/bin/ceph: stderr ceph orch apply <service_type> [--placement=<placement_string>] [--unmanaged] /usr/bin/ceph: stderr Traceback (most recent call last): File "/usr/sbin/cephadm", line 9653, in <module> main() File "/usr/sbin/cephadm", line 9641, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2205, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 5774, in command_bootstrap prepare_ssh(ctx, cli, wait_for_mgr_restart) File "/usr/sbin/cephadm", line 5275, in prepare_ssh cli(['orch', 'apply', t]) File "/usr/sbin/cephadm", line 5708, in cli return CephContainer( File "/usr/sbin/cephadm", line 4144, in run out, _, _ = call_throws(self.ctx, self.run_cmd(), File "/usr/sbin/cephadm", line 1853, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v17 -e NODE_NAME=node1-ceph -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4ce3a92a-8ddd-11ee-9b23-6341187f70c1:/var/log/ceph:z -v /tmp/ceph-tmp6yz3vt5s:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfhd01qwu:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v17 orch apply ceph-exporter -- *Francisco Arencibia Quesada.* *DevOps Engineer*

3 6

Re: Where is a simple getting started guide for a very basic cluster?
by Leo28C 30 Nov '23

30 Nov '23

The system has been running for years and it would take me weeks to reconfigure all of its services properly. I'm proceeding with just the Debian nodes for now; worst case scenario I'll just have to rent a new server to act as a third node. I was really hoping to use the CentOS one since it's already there. Somebody else mentioned DRBD earlier so I've looked into that too, but in that case I ran into problems with the Debian systems instead because apparently only DRBD 9 and up supports more than two nodes, and the only Debian kernel module packages I've found are for DRDB 8. Ironically, CentOS does have a working kmod-drbd90 package available. On Wed, Nov 29, 2023 at 5:53 PM Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Update it to 8 or Rocky 9? > > > On Nov 29, 2023, at 14:05, Leo28C <leo28c(a)gmail.com> wrote: > > > >> Why are you talking about version 14 now anyhow? > > > > One of my nodes is running CentOS 7 and the latest version I found for it > > is 14, unless there's a way to get 15 working on it that I don't know > about? > > > > On Wed, Nov 29, 2023 at 1:48 AM Robert Sander < > r.sander(a)heinlein-support.de> > > wrote: > > > >> On 11/28/23 17:50, Leo28C wrote: > >>> Problem is I don't have the cephadm command on every node. Do I need it > >>> on all nodes or just one of them? I tried installing it via curl but my > >>> ceph version is 14.2.22 which is not on the archive anymore so the curl > >>> command returns a 404 error html file. How do I get cephadm for 14.2? > >> > >> There is no cephadm for Ceph 14 as the orchestrator was first introduced > >> in version 15. > >> > >> Why are you talking about version 14 now anyhow? > >> As long as your nodes fulfill the requirements for cephadm you can > >> install the latest version of Ceph. > >> > >> PS: Please reply to the list. > >> > >> Regards > >> -- > >> Robert Sander > >> Heinlein Consulting GmbH > >> Schwedter Str. 8/9b, 10119 Berlin > >> > >> https://www.heinlein-support.de > >> > >> Tel: 030 / 405051-43 > >> Fax: 030 / 405051-19 > >> > >> Amtsgericht Berlin-Charlottenburg - HRB 220009 B > >> Geschäftsführer: Peer Heinlein - Sitz: Berlin > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Uploading file from admin to other users bucket in multi tenant mode
by Rok Jaklič 29 Nov '23

29 Nov '23

Hi, I have set following permission to admin user: radosgw-admin caps add --uid=admin --tenant=admin --caps="users=*;buckets=*" Now I would like to upload some object with admin user to some other user/tenant (tester1$tester1) to his bucket test1. Other user has uid tester1 and tenant tester1 and bucket test1 created. I've tried with python: headers = {'x-amz-meta-tenancy': 'tester1'} client.upload_file(file_path, bucket_name, object_name, ExtraArgs={'Metadata': headers}) But I get response: An error occurred (NoSuchBucket) when calling the PutObject operation: None Any ideas why I get this error even though bucket test1 for tester1$tester1 exists? Kind regards, Rok

1 0

Re: Where is a simple getting started guide for a very basic cluster?
by Robert Sander 29 Nov '23

29 Nov '23

On 11/28/23 17:50, Leo28C wrote: > Problem is I don't have the cephadm command on every node. Do I need it > on all nodes or just one of them? I tried installing it via curl but my > ceph version is 14.2.22 which is not on the archive anymore so the curl > command returns a 404 error html file. How do I get cephadm for 14.2? There is no cephadm for Ceph 14 as the orchestrator was first introduced in version 15. Why are you talking about version 14 now anyhow? As long as your nodes fulfill the requirements for cephadm you can install the latest version of Ceph. PS: Please reply to the list. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 1

Bucket/object create/update/delete notification
by Rok Jaklič 29 Nov '23

29 Nov '23

Hi, I would like to get info if the bucket or object got updated. I can get this info with a changed etag of an object, but not I cannot get etag from bucket, so I am looking at https://docs.ceph.com/en/latest/radosgw/notifications/ How do I create a topic and where do I send request with parameters? Anyone have an example (cli, python, ...) with topic creation where I can get notification to a http endpoint when an object got created in a bucket? Kind regards, Rok

2 2