- ceph-users - lists.ceph.io

Cephadm Drive upgrade process
by brentk＠cfl.rr.com 13 Nov '24

13 Nov '24

We are contemplating an upgrade of 4TB HDD drives to 20TB HDD drives (cluster info below, size 3 ), but as part of that discussion, we were trying to see if there was a more efficient way to do so. Our current process is as follows for failed drives: 1. Pulled failed drive ( after troubleshooting of course ) 2. Cephadm gui - find OSD, purge osd 3. Wait for rebalance 4. Insert new drive ( let cluster rebalance after it automatically adds the drive as an OSD ) ( yes, we have auto-add on in the clusters ) I imagine with an existing good drive, we would use delete instead of purge, but the process would be the similar, except the drive swap would happen after the data was moved. Would the replace flag ( or keep OSD option in gui ) allow us to avoid the initial rebalance by backfilling the new drive with the old drives content? I would be nice if we could just copy the content to the new drive and go from there. We would like to avoid lots of read/write cluster recovery activity if possible sine we could be replacing 40+ drives in each cluster. Open to trying a few methods of course, but we thought the group here might have some insight into the most optimal way of doing these swaps. For a history, last time we did this, we just drained an entire host, upgraded the drives and then added it back. This caused temporary imbalance that we would like to avoid this time around. Regards, -Brent Existing Clusters: Test: Reef 18.2.0 ( all virtual on nvme ) US Production(HDD): Reef 18.2.4 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2 mds UK Production(HDD): Reef 18.2.4 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw, 2mds US Production(SSD): Reef 18.2.4 Cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds UK Production(SSD): Reef 18.2.4 cephadm with 6 osd servers, 5 mons, 4 rgw, 2 mds

4 3

Strange container restarts?
by Jan Marek 13 Nov '24

13 Nov '24

Hello, we have ceph cluster which consists of 12 host, on every host we have 12 NVMe "disks". On most of these host (9 of 12) we have in logs errors, see attached file. We tried to check this problem, and we have these points: 1) On every host there is only one OSD. Thus it's not problem in version 18.2.2 generally, because there will be on another OSD, not only one of host? 2) Sometimes one of this OSD crashed :-( It seems, that crashed OSD are from set of OSDs, which have this problem. 3) ceph cluster goes OK and it "doesn't know" about any problem with these OSD. It seem's, that this new instance of ceph-osd daemon tried to start either podman or conmon itself. We've tried to control PID files for conman, but they're seems to be OK? 4) We tried to check 'ceph orch' command, but it does not try to start these containers, because it know, that they exists and run ('ceph orch ps' list these containers as running). 5) I've tried to pause ochestrator, but I've still found in syslog these entries... :-( Please, is there any possibility to find out, where is problem and stop this? We have all of the ceph host prepared by ansible, thus there is the same environment. On every machine we have podman version 4.3.1+ds1-8+deb12u1 and conmon version 2.1.6+ds1-1. OS is Debian bookworm. Attached logs was prepared by: grep exec_died /var/log/syslog Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

3 3

9 out of 11 missing shards of shadow object in ERC 8:3 pool.
by Robert Kihlberg 12 Nov '24

12 Nov '24

After an upgrade from Nautilus to Pacific the scrub has found an inconsistent object and reports that 9 out of 11 shards are missing. (However, we're not sure this has to do with the upgrade). We have been able to trace it to a S3 bucket, but not to a specific S3 object. # radosgw-admin object stat --bucket=$BUCKET --object=$OBJECT ERROR: failed to stat object, returned error: (2) No such file or directory By design, we have a complete mirror of the bucket in another Ceph cluster and the amount of objects in the buckets match between the clusters. We are therefore somewhat confident that we are not missing any objects. Could this be a failed garbage collection where perhaps the primary OSD failed during gc? The garbage collector does not show anything that seems relevant though... radosgw-admin gc list --include-all | grep "eaa6801e-3967-4541-9b8ca98aa5c2.791015596" Any suggestions on how we can trace and/or fix this inconsistent object? # rados list-inconsistent-obj 11.3ff | jq { "epoch": 177981, "inconsistents": [ { "object": { "name": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "nspace": "", "locator": "", "snap": "head", "version": 109853 }, "errors": [], "union_shard_errors": [ "missing" ], "selected_object_info": { "oid": { "oid": "eaa6801e-3967-4541-9b8ca98aa5c2.791015596.129__shadow_.3XHvgPjrJa3erG4rPlW3brboBWagE95_5", "key": "", "snapid": -2, "hash": 4294967295, "max": 0, "pool": 11, "namespace": "" }, "version": "17636'109853", "prior_version": "0'0", "last_reqid": "client.791015590.0:449317175", "user_version": 109853, "size": 8388608, "mtime": "2022-01-24T03:33:42.457722+0000", "local_mtime": "2022-01-24T03:33:42.471042+0000", "lost": 0, "flags": [ "dirty", "data_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xe588978d", "omap_digest": "0xffffffff", "expected_object_size": 0, "expected_write_size": 0, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 14, "primary": true, "shard": 0, "errors": [], "size": 1048576 }, { "osd": 67, "primary": false, "shard": 1, "errors": [ "missing" ] }, { "osd": 77, "primary": false, "shard": 4, "errors": [], "size": 1048576 }, { "osd": 225, "primary": false, "shard": 9, "errors": [ "missing" ] }, { "osd": 253, "primary": false, "shard": 8, "errors": [ "missing" ] }, { "osd": 327, "primary": false, "shard": 6, "errors": [ "missing" ] }, { "osd": 568, "primary": false, "shard": 2, "errors": [ "missing" ] }, { "osd": 610, "primary": false, "shard": 7, "errors": [ "missing" ] }, { "osd": 700, "primary": false, "shard": 3, "errors": [ "missing" ] }, { "osd": 736, "primary": false, "shard": 10, "errors": [ "missing" ] }, { "osd": 764, "primary": false, "shard": 5, "errors": [ "missing" ] } ] } ] }

3 4

Error ENOENT: Module not found - ceph orch commands stoppd working
by Torkil Svensgaard 12 Nov '24

12 Nov '24

Hi 18.2.4. After failing over the active manager ceph orch commands seems to have stopped working. There's this in the mgr log: " 2024-11-12T08:16:30.136+0000 7f1b2d887640 0 log_channel(audit) log [DBG] : from='client.2088861125 -' entity='client.admin' cmd=[{"prefix": "orch osd rm status", "target": ["mon-mgr", ""]}]: dispatch 2024-11-12T08:16:30.136+0000 7f1b23cf4640 -1 no module 'cephadm' 2024-11-12T08:16:30.136+0000 7f1b23cf4640 -1 no module 'cephadm' 2024-11-12T08:16:30.136+0000 7f1b23cf4640 -1 mgr.server reply reply (2) No such file or directory Module not found " The module is still enabled: " [root@ceph-flash1 ~]# ceph mgr module ls MODULE balancer on (always on) crash on (always on) devicehealth on (always on) orchestrator on (always on) pg_autoscaler on (always on) progress on (always on) rbd_support on (always on) status on (always on) telemetry on (always on) volumes on (always on) alerts on cephadm on dashboard on insights on iostat on nfs on prometheus on stats on diskprediction_local - influx - k8sevents - localpool - mds_autoscaler - mirroring - osd_perf_query - osd_support - restful - rgw - rook - selftest - snap_schedule - telegraf - test_orchestrator - zabbix - " Cluster is working: " [root@ceph-flash1 ~]# ceph -s cluster: id: 8ee2d228-ed21-4580-8bbf-0649f229e21d health: HEALTH_WARN noout flag(s) set 5 nearfull osd(s) Degraded data redundancy: 22742466/3557557778 objects degraded (0.639%), 559 pgs degraded, 559 pgs undersized 4 pool(s) nearfull services: mon: 5 daemons, quorum ceph-flash1,ceph-flash2,ceph-flash3,grouchy,klutzy (age 8d) mgr: ceph-flash2.utlhuz(active, since 10m), standbys: ceph-flash3.ciudre, ceph-flash1.erhakb mds: 1/1 daemons up, 2 standby osd: 567 osds: 555 up (since 14h), 555 in (since 4d); 2689 remapped pgs flags noout data: volumes: 1/1 healthy pools: 17 pools, 15521 pgs objects: 619.72M objects, 1.3 PiB usage: 2.3 PiB used, 2.0 PiB / 4.3 PiB avail pgs: 22742466/3557557778 objects degraded (0.639%) 135987731/3557557778 objects misplaced (3.823%) 12832 active+clean 2111 active+remapped+backfill_wait 479 active+undersized+degraded+remapped+backfill_wait 80 active+undersized+degraded+remapped+backfilling 19 active+remapped+backfilling io: client: 73 MiB/s rd, 5.4 MiB/s wr, 574 op/s rd, 169 op/s wr recovery: 2.9 GiB/s, 1.01k objects/s " Suggestions? I tried failing over the manager again which didn't help. Mvh. Torkil

2 4

Ceph pacific error when add new host
by nguyenvandiep＠baoviet.com.vn 12 Nov '24

12 Nov '24

We have a Ceph cluster run with podman2.0.5 - ceph 16.2.4 Our public network is 10.29.1.* and private network is 172.28.1.* We need deploy new host But when run ceph orch host add newhost ip ceph return added successfully, BUT: No daemon were deployed on newhost podman on newhost pulled the right Ceph image when we try ceph orch daemon add crash newhost on newhost, crash container started and running BUT on master node, crash.newhost is STARTING Could anyone help us. Thank you all

2 3

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

quincy v17.2.8 QE Validation status
by Yuri Weinstein 11 Nov '24

11 Nov '24

Details of this release are summarized here: https://tracker.ceph.com/issues/68643#note-2 Release Notes - TBD Seeking approvals/reviews for: rados - Laura, Radek, Travis, Ernesto, Adam King rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade/octopus-x (quincy) - Josh pls approve upgrade/pacific-x (quincy) - Josh pls approve upgrade/quincy-p2p - Known issue IIRC, Laura, Casey pls confirm/approve client-upgrade-quincy-reef - in progress powercycle - Brad pls approve ceph-volume - Guillaume pls take a look Please reply to this email with approval and/or trackers of known issues/PRs to address them. gibba and LRC upgrades -- N/A for quincy now after reef release. Thx YuriW

9 19

Ceph Steering Committee 2024-11-11
by Gregory Farnum 11 Nov '24

11 Nov '24

We had a short meeting today, with no pre-set topics. The main topic that came up is a bug detected on upgrades to squid when using the balancer in large clusters: https://tracker.ceph.com/issues/68657. The RADOS team would like to do a squid point release once our quincy release is out the door (imminent). Guarav from the Ceph ambassadors program is doing planning for FOSDEM. He's looking for anybody in Ceph technical leadership who is interested and available for co-presenting a Ceph talk. Also, the election closes for our executive council next week. We'll see the results then. -Greg

1 0

NFS and Service Dependencies
by Alex Buie 10 Nov '24

10 Nov '24

Hello all, Facing an issue with my first ceph with cephfs deployment. I have 2 admin (mgr) hosts, 3 mon/mds hosts, and 3 OSD hosts with 3 OSDs each. I also have been using the OSDs to run the nfs service that my hypervisor connects to. I have these service definitions I've applied: service_type: nfs service_id: nfs service_name: nfs.nfs placement: count: 1 label: nfs spec: port: 2049 virtual_ip: 172.19.19.165 service_type: ingress service_id: nfs.nfs service_name: ingress.nfs.nfs placement: count: 1 label: nfs spec: backend_service: nfs.nfs monitor_port: 9049 virtual_ip: 172.19.19.165 keepalive_only: true and I have tagged the 3 OSD nodes with the `nfs` label. Generally, both the ingress and the nfs service start on the same node (osd01, for example). But sometimes, errors occur with the nfs service and it gets rescheduled and ends up on a different node (osd03, for example) - and because the ingress service isn't running on this node, it breaks my NFS mount and the hypervisor gets very unhappy. Is there a way I can specify some kind of scheduling dependency between the nfs.nfs and ingress.nfs.nfs services so that they will get scheduled to run on the same node so that the virtual IP is present for nfs to bind to? Or, am I doing something wrong here? Thanks a bunch! *Alex*

3 2

OSD META Capacity issue of rgw ceph cluster
by Jaemin Joo 08 Nov '24

08 Nov '24

Hi All, I have RADOS gw(rgw) ceph cluster (v18.2.1) applied Erasure Coding and 10009 shards to save a lot of objects in a bucket root:/home/ceph# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.00490 1.00000 5.0 GiB 318 MiB 58 MiB 1 KiB 260 MiB 4.7 GiB 6.21 0.94 143 up 1 hdd 0.00490 1.00000 5.0 GiB 330 MiB 56 MiB 2 KiB 274 MiB 4.7 GiB 6.45 0.98 155 up 2 hdd 0.00490 1.00000 5.0 GiB 372 MiB 60 MiB 1 KiB 313 MiB 4.6 GiB 7.28 1.10 148 up (The osds are just data pool, it's not index pool for rgw..I think a lot of shards are not impact data pool META size) the OSD META occupied same as data size. I know that OSD META is rocksdb. I think oversized even though rocksdb META is needed. I can use osd compact to decrease META size. but I want to know why OSD META size is big than I expect, Is there way to decrease the META size by changing parameter instead of osd compact? I guess that some of "bluestore_rocksdb_options" can impact the META size. And let me know if you need the other parameter of ceph cluster to figure out the symptom.

2 3