- ceph-users - lists.ceph.io

MDs stuck in rejoin with '[ERR] : loaded dup inode'
by Noe P. 30 May '24

30 May '24

Hi, I'm still unable to get our filesystem back. I now have this: fs_cluster - 0 clients ========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 rejoin cephmd4b 90.0k 89.4k 14.7k 0 1 rejoin cephmd6b 105k 105k 21.3k 0 2 failed POOL TYPE USED AVAIL fs_cluster_meta metadata 288G 55.2T fs_cluster_data data 421T 55.2T Still cannot get rid of the 3rd failed rank. But the other two currently stay in state rejoin forever. After all clients were stopped, the log complains about a 'dup inode': 2024-05-30T07:59:46.252+0200 7f2fe9146700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x1001710ea1d [12bc6a,head] v1432525092 at /homes/YYY/ZZZ/.bash_history-21032.tmp, but inode 0x1001710ea1d.head v1432525109 already exists at /homes/YYY/ZZZ/.bash_history Questions: - Is there a way to scan/repair the metadata without any MD in 'active' state ? - Is there a way to remove (or otherwise fix) the inode in question given the above inode number ? - Is the state 'rejoin' due to the inode error or because of that 3rd rank ? Regard, N.

1 0

Ceph data got missed
by prabu.jawahar＠gmail.com 30 May '24

30 May '24

We are using Ceph Octopus version with a total disk size of 136 TB, configured with two replicas. Currently, our usage is 57 TB, and the available size is 5.3 TB. An incident occurred yesterday where around 3 TB of data was deleted automatically. Upon analysis, we couldn't find the reason for the deletion. All OSDs are functioning properly and actively running. We have 3 MDS , we try to restarted all MDS services. Is there any solution to recover those data. Can anyone please help us find the issue? cluster: id: 0d605d58-5caf-4f76-b6bd-e12402a22296 health: HEALTH_WARN insufficient standby MDS daemons available 5 nearfull osd(s) 3 pool(s) nearfull 1 pool(s) have non-power-of-two pg_num services: mon: 4 daemons, quorum download-mon3,download-mon4,download-mon1,download-mon2 (age 14h) mgr: download-mon2(active, since 14h), standbys: download-mon1, download-mon3 mds: integdownload:2 {0=download-mds3=up:active,1=download-mds1=up:active} osd: 39 osds: 39 up (since 16h), 39 in (since 4d) data: pools: 3 pools, 1087 pgs objects: 71.76M objects, 51 TiB usage: 105 TiB used, 31 TiB / 136 TiB avail pgs: 1087 active+clean io: client: 414 MiB/s rd, 219 MiB/s wr, 513 op/s rd, 1.22k op/s wr ================================================================ ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 download-osd1 2995G 581G 14 4785k 6 6626k exists,up 1 download-osd2 2578G 998G 84 3644k 18 10.1M exists,up 2 download-osd3 3093G 483G 17 5114k 5 4152k exists,nearfull,up 3 download-osd4 2757G 819G 12 996k 2 4107k exists,up 4 download-osd5 2889G 687G 28 3355k 20 8660k exists,up 5 download-osd6 2448G 1128G 183 3312k 10 9435k exists,up 6 download-osd7 2814G 762G 7 1667k 4 6354k exists,up 7 download-osd8 2872G 703G 14 1672k 15 10.5M exists,up 8 download-osd9 2577G 999G 10 6615k 3 6960k exists,up 9 download-osd10 2651G 924G 16 4736k 3 7378k exists,up 10 download-osd11 2889G 687G 15 4810k 6 8980k exists,up 11 download-osd12 2912G 664G 11 2516k 2 4106k exists,up 12 download-osd13 2785G 791G 74 4643k 11 3717k exists,up 13 download-osd14 3150G 426G 214 6133k 4 7389k exists,nearfull,up 14 download-osd15 2728G 848G 11 4959k 4 6603k exists,up 15 download-osd16 2682G 894G 13 3170k 3 2503k exists,up 16 download-osd17 2555G 1021G 53 2183k 7 5058k exists,up 17 download-osd18 3013G 563G 18 3497k 3 4427k exists,up 18 download-osd19 2924G 651G 24 3534k 12 10.4M exists,up 19 download-osd20 3003G 573G 19 5149k 3 2531k exists,up 20 download-osd21 2757G 819G 16 3707k 9 9816k exists,up 21 download-osd22 2576G 999G 15 2526k 8 7739k exists,up 22 download-osd23 2758G 818G 13 4412k 16 7125k exists,up 23 download-osd24 2862G 714G 18 4424k 6 5787k exists,up 24 download-osd25 2792G 783G 16 1972k 9 9749k exists,up 25 download-osd26 2397G 1179G 14 4296k 9 12.0M exists,up 26 download-osd27 2308G 1267G 8 3149k 22 6280k exists,up 27 download-osd29 2732G 844G 12 3357k 3 7372k exists,up 28 download-osd28 2814G 761G 11 476k 5 3316k exists,up 29 download-osd30 3069G 507G 15 9043k 17 5628k exists,nearfull,up 30 download-osd31 2660G 916G 15 841k 14 7798k exists,up 31 download-osd32 2037G 1539G 10 1153k 15 3719k exists,up 32 download-osd33 3116G 460G 20 7704k 12 9041k exists,nearfull,up 33 download-osd34 2847G 728G 19 5788k 4 9014k exists,up 34 download-osd35 3088G 488G 17 7178k 7 5730k exists,nearfull,up 35 download-osd36 2414G 1161G 27 2017k 14 7612k exists,up 36 download-osd37 2760G 815G 17 4292k 5 10.6M exists,up 37 download-osd38 2679G 897G 12 2610k 5 10.0M exists,up 38 download-osd39 3013G 563G 18 1804k 7 9235k exists,up

1 0

Error EINVAL: check-host failed - Failed to add host
by Suryanarayana Raju 30 May '24

30 May '24

Hello, Please help: CEPH cluster using docker. using below command for the bootstrap with provided key and pub cephadm -v bootstrap --mon-ip <ip> --allow-overwrite --ssh-private-key id_ed25519 --ssh-public-key id_ed25519.pub able to ssh directly with id_ed25519 key. RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=<hostname> -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp1sw6a5s0:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpethxlwxr:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add <hostname> <ip>: Error EINVAL: check-host failed: Unable to write <host>:/var/lib/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed also validated below and it was successful: > ceph cephadm get-ssh-config > ssh_config > ceph config-key get mgr/cephadm/ssh_identity_key > key > ssh -F ssh_config -i key root@<hostname> --- root@<hostname>:~/.ssh# ceph orch host add <hostname> <ip> Error EINVAL: check-host failed: Unable to write <hostname>:/var/lib/ceph/fe8ecd30-1da2-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed root@<hostname>:~/.ssh# Thanks, Surya

2 1

Unable to Install librados2 18.2.0 on RHEL 7 from Ceph Repository
by abdel.douichi＠gmail.com 30 May '24

30 May '24

The Ceph repository at https://download.ceph.com/ does not seem to have the librados2 package version 18.2.0 for RHEL 7. The directory <b>https://download.ceph.com/rpm-18.2.0/el7/</b> is empty, and the specific package URL https://download.ceph.com/rpm-18.2.0/el7/x86_64/librados2-18.2.0-0.el7.x86_… returns a 404 error.

2 1

MDS crashing
by Johan 29 May '24

29 May '24

Hi, I have a small cluster with 11 osds and 4 filesystems. Each server (Debian 11, ceph 17.2.7) usually run several services. After troubles with a host with OSD:s I removed the OSD:s and let the cluster repair it self (x3 replica). After a while it returned to a healthy state and everything was well. This might not be important for what followed, but I mention it just in case. A couple of days later a mds-services gave a health-warning. First one was (2024-05-28 10:02) mds.cloudfs.stugan6.ywuomz(mds.0): 1 slow requests are blocked > 30 secs followed by filesystem being degraded (2024-05-28 10:22) fs cloudfs is degraded the other filesystems have been marked degraded from time to time but later cleared. at 2024-05-28 10:28 daemon mds.mediafs.stugan7.zzxavs on stugan7 is in error state daemon mds.cloudfs.stugan7.cmjbun on stugan7 is in error state at 2024-05-28 10:33 daemon mds.cloudfs.stugan4.qxwzox on stugan4 is in error state daemon mds.cloudfs.stugan5.hbkkad on stugan5 is in error state daemon mds.oxylfs.stugan7.iazekf on stugan7 is in error state MDS-services went on crashing ... I put the osd:s on pause and nodown,noout,nobackfill,norebalance,norecover flags, but at present only the flags as I have tried to get the system up and running again. While the osd:s were paused, I could 'clear up' the mess and remove all services in error state. The monitors and managers seems to function well. I could also start getting the mds-services running again. BUT, when I removed the pause from the osd:s the mds-services once again started to go inte error state. Now I have removed the mds-label from all ceph servers and therefore it has calmed down. But if I let the services be recreated the crashes will start over again. If I check the filesystem (I have marked them down for now) status the cloudfs is strange... oxylfs - 0 clients ====== POOL TYPE USED AVAIL oxylfs_metadata metadata 154M 20.8T oxylfs_data data 1827G 20.8T cloudfs - 0 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) backupfs.stugan6.bgcltx 0 0 0 0 POOL TYPE USED AVAIL cloudfs_metadata metadata 337M 20.8T cloudfs_data data 356G 20.8T mediafs - 0 clients ======= POOL TYPE USED AVAIL mediafs_metadata metadata 66.0M 20.8T mediafs_data data 2465G 20.8T backupfs - 0 clients ======== POOL TYPE USED AVAIL backupfsnew_metadata metadata 221M 20.8T backupfsnew_data data 8740G 20.8T MDS version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Why is the mds (in error state) for the backupfs-filesystem shown with the cloudfs-filesystem? Now... Is there a way back to normal? /Johan

1 0

Re: Lousy recovery for mclock and reef
by Kai Stian Olstad 29 May '24

29 May '24

On 24.05.2024 21:07, Mazzystr wrote: > I did the obnoxious task of updating ceph.conf and restarting all my > osds. > > ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get > osd_op_queue > { > "osd_op_queue": "wpq" > } > > I have some spare memory on my target host/osd and increased the target > memory of that OSD to 10 Gb and restarted. No effect observed. In > fact > mem usage on the host is stable so I don't think the change took effect > even with updating ceph.conf, restart and a direct asok config set. > target > memory value is confirmed to be set via asok config get > > Nothing has helped. I still cannot break the 21 MiB/s barrier. > > Does anyone have any more ideas? For recovery you can adjust the following. osd_max_backfills default is 1, in my system I get the best performance with 3 and wpq. The following I have not adjusted myself, but you can try. osd_recovery_max_active is default to 3. osd_recovery_op_priority is default to 3, a lower number increases the priority for recovery. All of them can be runtime adjusted. -- Kai Stian Olstad

8 11

Bug Found in Reef Releases - Action Required for pg-upmap-primary Interface Users
by Laura Flores 29 May '24

29 May '24

Dear Ceph Users, We have discovered a bug with the pg-upmap-primary interface (related to the offline read balancer [1]) that affects all Reef releases. In all Reef versions, users are required to set `require-min-compat-client=reef` in order to use the pg-upmap-primary interface to prevent pre-reef clients from connecting and not understanding the new interface. We found this setting is simply not enforced [2], which leads to miscommunication between older and newer peers or, depending on version, to an assert in the mons and/or osds [3]. However, the fundamental precondition is making use of the new `pg-upmap-primary` feature. If you have not yet upgraded to v18.2.2, we recommend that you refrain from upgrading to v18.2.2 until a later version is out with a fix. We also recommend removing any existing pg-upmap-primary mappings to prevent hitting the assert [3], as well as to prevent any miscommunication between older and newer peers about pg primaries [2]. Remove mappings by: $ `ceph osd dump` For each pg_upmap_primary entry in the above output: $ `ceph osd rm-pg-upmap-primary <pgid>` If you have already upgraded to v18.2.2, your cluster is more likely to hit the osd/mon assert [3] when you set a `pg-upmap-primary` mapping (this would involve explicitly setting a mapping via the osdmaptool or the CLI command). As long as you refrain from setting any pg-upmap-primary mappings, your cluster will NOT be affected by [3]. Follow the trackers below for further updates. 1. pg-upmap-primary documentation: https://docs.ceph.com/en/reef/rados/operations/read-balancer/ 2. mon, osd, *: require-min-compat-client is not really honored - https://tracker.ceph.com/issues/66260 3. Failed assert "pg_upmap_primaries.empty()" in the read balancer - https://tracker.ceph.com/issues/61948 Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

3 2

Rocky 8 to Rocky 9 upgrade and ceph without data loss
by Christopher Durham 28 May '24

28 May '24

I have both a small test cluster and a larger production cluster. They are (were, for the test cluster) running Rocky Linux 8.9. They are both updated originally from Pacific, currently at reef 18.2.2.These are all rpm installs. It has come time to consider upgrades to Rocky 9.3. As there is no officially supported upgrade procedure from Rocky 8 to Rocky 9, I wanted to document my procedure for the test cluster(it worked!) before I move on to my production cluster, and verify that what I have done makes sense and ask a few questions. As such, this involves a reinstall of the OS. For my test cluster, my mon nodes also have all the other services: ceph-mon, ceph-mds, ceph-radosgw and ceph-mgr. ALL of my OSDs are LVMs, and they each have wal devices, also as LVMs. My aim was to preserve these across the reinstall of the OS, and I did. To be clear,all the OSDs are not on the same physical disk as the OS and the mon/mgr/mds/radosgw, and most are grouped on their own node. First, I set noout, norecover, nobackfill # ceph osd set noout# ceph osd set norecover# ceph osd set nobackfill On one of the mon nodes, I did the following, to save off the monmap, osdmap, and crushmap in case something went wildly wrong # systemctl stop ceph-mon.target# ceph-mon -i <short servername> --extract-monmap /tmp/monmap.bin# systemcl start ceph-mon.target# ceph osd getcrushmap -o /tmp/crush.bin# ceph osd getmap -o /tmp/osd.bin Then on EVERY node, both mons and osd nodes, I tarred up /etc/ceph and /var/lib/ceph # tar cvf /tmp/varlibceph.tar /etc/ceph /var/lib/ceph I then saved off each tar file on each node to a location not being reinstalled, as such I had a tarfile per osd node and also one for each mon node. I also saved off the monmap, crushmap and osdmaps created above tothe same location. I then went sequentially through the mon/mds/radosgw/mgr servers, one at a time. As I am using kickstart/cobbler, I told kickstartto ignore all disks EXCEPT the one that the original 8.9 OS was installed on. For this I had to use the following in the kickstart file. ignoredisk --only-use=disk/by-path/pci-id-of-root-disk I did this because, at least on my hardware, Rocky 9 reorders drive letters sda, sdb, sdc, etc based on recognition order on *every* boot, which mightoverwrite an OSD LVM if I wasn't careful enough. Note that while I group some commands that follow below, I only did the mon nodes sequentially, whille I grouped together osd nodes and did those groups allat the same time based on my failure domain for my crushmap. After the Rocky 9 reinstall and a reboot: I then installed the appropriate el9 18.2.2 ceph packages $ dnf install ceph-mon (mon nodes)# dnf install ceph-mds (mds nodes) # dnf install ceph-radosgw (radosgw nodes)# dnf install ceph-mgr (mgr nodes)# dnf install ceph-osd (osd nodes) For the various nodes, once the reinstall was done to Rocky 9, I re-enabled the firewall # firewall-cmd --add-service ceph-mon --permanent ( for mon nodes)# firewall-cmd --add-service ceph --permanent (for osd nodes)# firewall-cmd --add-service https --permanent (for radosgw servers I am running them on port 443, with certs)# systemctl restart firewalld and then of course restarted firewalld I then restored /etc/ceph and /var/lib/ceph for each node from the individual tar file backups per node. For the non-OSD nodes, I re-enabled services: # systemctl enable ceph-mon(a)short-name-of-server.service (mon nodes)# systemctl enable ceph-mds(a)short-name-of-server.service (mds nodes)# systemctl enable ceph-radosgw(a)rgw.short-name-of-server.service (radosgw servers)# systemctl enable ceph-mgr(a)short-name-of-server.service (mgr nodes) For the OSD nodes, "ceph-volume lvm list" shows this for each OSD, even after reinstallingto Rocky 9 (which is good evidence that I installed on the proper disk and did not overwritean OSD: # ceph-volume lvm list block device /dev/ceph-c5b97619-4184-4637-9b82-2575004dba41/osd-block-abb210f3-52cf-4d4b-8745-126dc57287da block uuid lmxbj0-zgcB-5MwI-wpsf-9yut-3VlG-p9DoFL cephx lockbox secret cluster fsid 0f3b6c81-3814-4dc5-a19e-f7307543e56c cluster name ceph crush device class encrypted 0 osd fsid abb210f3-52cf-4d4b-8745-126dc57287da osd id 7 osdspec affinity type block vdo 0 devices /dev/sdj I had to enable units of the form: # systemctl enable ceph-lvm-$osd-id-$osd-fsid.service For the above osd lvm, this would be: # systemctl enable ceph-volume(a)lvm-7-abb210f3-52cf-4d4b-8745-126dc57287da.service So If I had 10 OSD LVMs on a given node, I have to enable 10 services of the form above. After reboot of either a mon or osd node (depending on which server I am doing), all comes up just fine. My questions: 1. Did I do anything wrong?2. Did I miss something?3. is there an official way to do this that is different from my way?4. Will anything I did (or did not) do bite me when I later do an update say to 18.2.6 or 19? 4. Is there an 'official' way to back up things like /etc/ceph and /var/lib/ceph that would be '*better**? Of course, after I update all the OSDs, I take off noout, norecover, nobackfill # ceph osd unset nobackfill# ceph osd unset norecover# ceph osd unset noout Any thoughts would be appreciated. Again I am updating from 18.2.2 on Rocky 8 to the same release (but different rpms), 18.2.2 on Rocky 9. -Chris

1 0

OSD processes crashes on repair 'unexpected clone'
by Thomas Björklund 28 May '24

28 May '24

Hi, We have an old cluster with 3 nodes running ceph version 15.2.17. We have a PG in state active+clean+inconsistent which we are unable to repair. It's an RBD pool in use by kubernetes. The earliest indication of the issue comes from ceph-osd.4.log on one of the nodes: *2024-05-23T14:34:38.003+0200 7f00b907c700 0 log_channel(cluster) log [DBG] : 20.4 repair starts2024-05-23T14:34:46.095+0200 7f00b907c700 -1 log_channel(cluster) log [ERR] : 20.4 shard 4 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : missing2024-05-23T14:34:46.095+0200 7f00b907c700 -1 log_channel(cluster) log [ERR] : 20.4 shard 16 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : missing2024-05-23T14:34:46.095+0200 7f00b907c700 -1 log_channel(cluster) log [ERR] : repair 20.4 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : is an unexpected clone2024-05-23T14:34:46.095+0200 7f00b907c700 -1 osd.4 pg_epoch: 30768 pg[20.4( v 30768'359661769 (30767'359657067,30768'359661769] local-lis/les=30660/30665 n=10628 ec=28562/20971 lis/c=30660/30660 les/c/f=30665/30665/0 sis=30660) [4,10,16] r=0 lpr=30660 luod=30768'359661760 crt=30768'359661769 lcod 30768'359661759 mlcod 30768'359661759 active+clean+scrubbing+deep+inconsistent+repair REQ_SCRUB] _scan_snaps no head for 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e (have 20:20a42186:stc::rbd_data.44feb146358046.00000000000425b6:head)2024-05-23T14:35:27.712+0200 7f00b5074700 -1 log_channel(cluster) log [ERR] : 20.4 repair 1 missing, 0 inconsistent objects2024-05-23T14:35:27.712+0200 7f00b5074700 -1 log_channel(cluster) log [ERR] : 20.4 repair 3 errors, 2 fixed2024-05-23T14:35:27.720+0200 7f00b5074700 -1 ./src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::recover_missing(const hobject_t&, eversion_t, int, PGBackend::RecoveryHandle*)' thread 7f00b5074700 time 2024-05-23T14:35:27.718485+0200./src/osd/PrimaryLogPG.cc: 11540: FAILED ceph_assert(head_obc) ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x131) [0x55a740013abb] 2: (()+0x9d5c46) [0x55a740013c46] 3: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0x6cf) [0x55a7401fabbf] 4: (PrimaryLogPG::recover_primary(unsigned long, ThreadPool::TPHandle&)+0xf7e) [0x55a74023b16e] 5: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x5ee) [0x55a740246b4e] 6: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x295) [0x55a7400c4c85] 7: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0x55a74031c229] 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xf13) [0x55a7400e13c3] 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x41a) [0x55a74072ae6a] 10: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a74072d410] 11: (()+0x7ea7) [0x7f00d59e2ea7] 12: (clone()+0x3f) [0x7f00d556ba6f]2024-05-23T14:35:27.724+0200 7f00b5074700 -1 *** Caught signal (Aborted) ** in thread 7f00b5074700 thread_name:tp_osd_tp ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable) 1: (()+0x13140) [0x7f00d59ee140] 2: (gsignal()+0x141) [0x7f00d54a8ce1] 3: (abort()+0x123) [0x7f00d5492537] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x55a740013b05] 5: (()+0x9d5c46) [0x55a740013c46] 6: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0x6cf) [0x55a7401fabbf] 7: (PrimaryLogPG::recover_primary(unsigned long, ThreadPool::TPHandle&)+0xf7e) [0x55a74023b16e] 8: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x5ee) [0x55a740246b4e] 9: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x295) [0x55a7400c4c85] 10: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0x55a74031c229] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xf13) [0x55a7400e13c3] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x41a) [0x55a74072ae6a] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a74072d410] 14: (()+0x7ea7) [0x7f00d59e2ea7] 15: (clone()+0x3f) [0x7f00d556ba6f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.* Every time we try to start a repair on 20.4 all three OSD crashes (4, 10 & 16) *2024-05-28T11:14:30.573+0200 7f4d3405a700 0 log_channel(cluster) log [DBG] : 20.4 repair starts2024-05-28T11:14:39.481+0200 7f4d30052700 -1 log_channel(cluster) log [ERR] : 20.4 shard 10 soid 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : candidate had a read error2024-05-28T11:14:39.481+0200 7f4d30052700 -1 log_channel(cluster) log [ERR] : 20.4 soid 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : repairing object info data_digest2024-05-28T11:14:39.481+0200 7f4d30052700 -1 log_channel(cluster) log [ERR] : 20.4 soid 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : data_digest 0x43d61c5d != data_digest 0x867fac72 from auth oi 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e(29034'359526115 osd.4.0:147412411 dirty|data_digest|omap_digest s 4194304 uv 287854203 dd 867fac72 od ffffffff alloc_hint [0 0 0])2024-05-28T11:14:39.481+0200 7f4d30052700 -1 log_channel(cluster) log [ERR] : 20.4 shard 16 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : missing2024-05-28T11:14:39.481+0200 7f4d30052700 -1 log_channel(cluster) log [ERR] : repair 20.4 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : is an unexpected clone2024-05-28T11:14:39.489+0200 7f4d30052700 -1 ./src/osd/PG.cc: In function 'void PG::update_snap_map(const std::vector<pg_log_entry_t>&, ObjectStore::Transaction&)' thread 7f4d30052700 time 2024-05-28T11:14:39.487768+0200./src/osd/PG.cc: 1225: FAILED ceph_assert(r == 0) ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x131) [0x55c8140ebabb] 2: (()+0x9d5c46) [0x55c8140ebc46] 3: (PG::update_snap_map(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, ceph::os::Transaction&)+0x6f8) [0x55c814241d28] 4: (PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)+0x230) [0x55c814358510] 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t>&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x6cd) [0x55c8144df8ad] 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x53d) [0x55c8142b250d] 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >)+0x9b) [0x55c8142b411b] 8: (PrimaryLogPG::scrub_snapshot_metadata(ScrubMap&, std::map<hobject_t, std::pair<std::optional<unsigned int>, std::optional<unsigned int> >, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::pair<std::optional<unsigned int>, std::optional<unsigned int> > > > > const&)+0x24b1) [0x55c8142dc0e1] 9: (PG::scrub_compare_maps()+0x18db) [0x55c81424beeb] 10: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x14a) [0x55c81424eb1a] 11: (ceph::osd::scheduler::PGScrub::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1a) [0x55c8143f41fa] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xf13) [0x55c8141b93c3] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x41a) [0x55c814802e6a] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55c814805410] 15: (()+0x7ea7) [0x7f4d509c0ea7] 16: (clone()+0x3f) [0x7f4d50549a6f]2024-05-28T11:14:39.497+0200 7f4d30052700 -1 *** Caught signal (Aborted) ** in thread 7f4d30052700 thread_name:tp_osd_tp ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable) 1: (()+0x13140) [0x7f4d509cc140] 2: (gsignal()+0x141) [0x7f4d50486ce1] 3: (abort()+0x123) [0x7f4d50470537] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x55c8140ebb05] 5: (()+0x9d5c46) [0x55c8140ebc46] 6: (PG::update_snap_map(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, ceph::os::Transaction&)+0x6f8) [0x55c814241d28] 7: (PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)+0x230) [0x55c814358510] 8: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t>&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x6cd) [0x55c8144df8ad] 9: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x53d) [0x55c8142b250d] 10: (PrimaryLogPG::simple_opc_submit(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >)+0x9b) [0x55c8142b411b] 11: (PrimaryLogPG::scrub_snapshot_metadata(ScrubMap&, std::map<hobject_t, std::pair<std::optional<unsigned int>, std::optional<unsigned int> >, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::pair<std::optional<unsigned int>, std::optional<unsigned int> > > > > const&)+0x24b1) [0x55c8142dc0e1] 12: (PG::scrub_compare_maps()+0x18db) [0x55c81424beeb] 13: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x14a) [0x55c81424eb1a] 14: (ceph::osd::scheduler::PGScrub::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1a) [0x55c8143f41fa] 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xf13) [0x55c8141b93c3] 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x41a) [0x55c814802e6a] 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55c814805410] 18: (()+0x7ea7) [0x7f4d509c0ea7] 19: (clone()+0x3f) [0x7f4d50549a6f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.* We are unsure on how to rectify the issue and get the cluster healthy again. Best regards, Thomas Björklund

1 1

Call Us - 8264348440 , Call Girls in Sarai Kale Khan - We brings offer model independent genuine call girls/Women Seeking Men Justdial Delhi escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you.
by bedraj poudeel 28 May '24

28 May '24

1 0