January 2023 - ceph-users

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards

by Christian Rohmann

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

2 months

3
4
0 0

cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting

by Zakhar Kirpichenko

Hi, Attempted to upgrade 16.2.10 to 16.2.11, 2 OSDs out of many started crashing in a loop on the very 1st host: Jan 25 23:07:53 ceph01 bash[2553123]: Uptime(secs): 0.0 total, 0.0 interval Jan 25 23:07:53 ceph01 bash[2553123]: Flush(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Total Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(L0 Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Keys): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Jan 25 23:07:53 ceph01 bash[2553123]: ** File Read Latency Histogram By Level [P] ** Jan 25 23:07:53 ceph01 bash[2553123]: debug -10> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986439) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -9> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986493) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -8> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986500) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -7> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986505) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -6> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1676] [O-2] [JOB 9] Compacting 4@0 + 2@1 files to L1, score 1.0 0 Jan 25 23:07:53 ceph01 bash[2553123]: debug -5> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1680] [O-2] Compaction start summary: Base version 33 Base level 0, inputs: [649058(959KB) 649046(1510KB) 649024(1323KB) 649002(1396KB)], [648981(66MB) 648982(52MB)] Jan 25 23:07:53 ceph01 bash[2553123]: debug -4> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1674688072986547, "job": 9, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [649058, 649046, 649024, 649002], "files_L1": [648981, 648982], "score": 1, "input_data_size": 129161327} Jan 25 23:07:53 ceph01 bash[2553123]: debug -3> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 bdev(0x5619bf0ce400 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-1 ((1) Operation not permitted) Jan 25 23:07:53 ceph01 bash[2553123]: debug -2> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7c8b66e700 time 2023-01-25T23:07:52.993976+0000 Jan 25 23:07:53 ceph01 bash[2553123]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please check your dmesg!") Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x5619b2fc1adc] Jan 25 23:07:53 ceph01 bash[2553123]: 2: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] Jan 25 23:07:53 ceph01 bash[2553123]: 5: clone() Jan 25 23:07:53 ceph01 bash[2553123]: debug -1> 2023-01-25T23:07:52.994+0000 7f7c8b66e700 -1 *** Caught signal (Aborted) ** Jan 25 23:07:53 ceph01 bash[2553123]: in thread 7f7c8b66e700 thread_name:bstore_aio Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7c9788fcf0] Jan 25 23:07:53 ceph01 bash[2553123]: 2: gsignal() Jan 25 23:07:53 ceph01 bash[2553123]: 3: abort() Jan 25 23:07:53 ceph01 bash[2553123]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x5619b2fc1bad] Jan 25 23:07:53 ceph01 bash[2553123]: 5: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] OSD kept crashing until the host reboot, an OSD restart wouldn't help. This hasn't happened during any previous upgrades, so was a rather unexpected development. Unclear what caused this, but a host reboot seems to have fixed it. It happened to 1 other OSD on another host, exactly the same symptoms, also solved by a reboot. Best regards, Zakhar

2 months, 1 week

4
4
0 0

Small RGW objects and RADOS 64KB minimun size

by Loïc Dachary

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

3 months, 3 weeks

4
6
0 0

Re: MDS crashes to damaged metadata

by Patrick Donnelly

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

4 months

2
2
0 0

MDS crash on FAILED ceph_assert(cur->is_auth())

by Peter van Heusden

I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers have started failing with the error message In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f0ca9908700 time 2021-06-28T09:21:11.484768+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el7/BUILD/ceph-15.2.13/src/mds/Server.cc: 4149: FAILED ceph_assert(cur->is_auth()) Complete log is: https://gist.github.com/pvanheus/4da555a6de6b5fa5e46cbf74f5500fbd ceph status output is: # ceph status cluster: id: ed7b2c16-b053-45e2-a1fe-bf3474f90508 health: HEALTH_WARN 30 OSD(s) experiencing BlueFS spillover insufficient standby MDS daemons available 1 MDSs report slow requests 2 mgr modules have failed dependencies 4347046/326505282 objects misplaced (1.331%) 6 nearfull osd(s) 23 pgs not deep-scrubbed in time 23 pgs not scrubbed in time 8 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 22m) mgr: ceph-mon1(active, since 11w), standbys: ceph-mon2, ceph-mon3 mds: SANBI_FS:2 {0=ceph-mon1=up:active(laggy or crashed),1=ceph-mon2=up:stopping} osd: 54 osds: 54 up (since 2w), 54 in (since 11w); 50 remapped pgs data: pools: 8 pools, 833 pgs objects: 42.37M objects, 89 TiB usage: 159 TiB used, 105 TiB / 264 TiB avail pgs: 4347046/326505282 objects misplaced (1.331%) 782 active+clean 49 active+clean+remapped 1 active+clean+scrubbing+deep 1 active+clean+remapped+scrubbing io: client: 29 KiB/s rd, 427 KiB/s wr, 37 op/s rd, 48 op/s wr When restarting a MDS it goes through states replace, reconnect, resolve and finally sets itself to active before this crash happens. Any advice on what to do? Thanks, Peter P.S. apologies if you received this email more than once - I have had some trouble figuring out the correct mailing list to use.

4 months, 3 weeks

3
5
0 0

deploying Ceph using FQDN for MON / MDS Services

by Lokendra Rathour

Hi Team, We have a ceph cluster with 3 storage nodes: 1. storagenode1 - abcd:abcd:abcd::21 2. storagenode2 - abcd:abcd:abcd::22 3. storagenode3 - abcd:abcd:abcd::23 The requirement is to mount ceph using the domain name of MON node: Note: we resolved the domain name via DNS server. For this we are using the command: ``` mount -t ceph [storagenode.storage.com]:6789:/ /backup -o name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== ``` We are getting the following logs in /var/log/messages: ``` Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' (ret=-3): failed Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip ' storagenode.storage.com:6789' ``` We also tried mounting ceph storage using IP of MON which is working fine. Query: Could you please help us out with how we can mount ceph using FQDN. My /etc/ceph/ceph.conf is as follows: [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] public network = abcd:abcd:abcd::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 4294967296 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 rgw thread pool size = 512 -- ~ Lokendra skype: lokendrarathour

5 months

7
20
0 0

ceph/daemon stable tag

by Jonas Nemeikšis

Hello, What's the status with the *-stable-* tags? https://quay.io/repository/ceph/daemon?tab=tags No longer build/support? What should we use until we'll migrate from ceph-ansible to cephadm? Thanks. -- Jonas

5 months, 1 week

2
6
0 0

cephadm cluster move /var/lib/docker to separate device fails

by Karsten Nielsen

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

6 months

2
1
0 0

Stuck OSD service specification - can't remove

by David Orman

Has anybody run into a 'stuck' OSD service specification? I've tried to delete it, but it's stuck in 'deleting' state, and has been for quite some time (even prior to upgrade, on 15.2.x). This is on 16.2.3: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd_spec 504/525 <deleting> 12m label:osd root@ceph01:/# ceph orch rm osd.osd_spec Removed service osd.osd_spec From active monitor: debug 2021-05-06T23:14:48.909+0000 7f17d310b700 0 log_channel(cephadm) log [INF] : Remove service osd.osd_spec Yet in ls, it's still there, same as above. --export on it: root@ceph01:/# ceph orch ls osd.osd_spec --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: {} unmanaged: true spec: filter_logic: AND objectstore: bluestore We've tried --force, as well, with no luck. To be clear, the --export even prior to delete looks nothing like the actual service specification we're using, even after I re-apply it, so something seems 'bugged'. Here's the OSD specification we're applying: service_type: osd service_id: osd_spec placement: label: "osd" data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 I would appreciate any insight into how to clear this up (without removing the actual OSDs, we're just wanting to apply the updated service specification - we used to use host placement rules and are switching to label-based). Thanks, David

6 months, 1 week

3
4
0 0

OSD upgrade problem nautilus->octopus - snap_mapper upgrade stuck

by Jan Pekař - Imatic

Hi all, I have problem upgrading nautilus to octopus on my OSD. Upgrade mon and mgr was OK and first OSD stuck on 2023-01-12T09:25:54.122+0100 7f49ff3eae00 1 osd.0 126556 init upgrade snap_mapper (first start as octopus) and there were no activity after that for more than 48 hours. No disk activity. I restarted OSD many times and nothing changed. It is old, filestore OSD based on XFS filesystem. Is upgrade to snap mapper 2 reliable? What is OSD waiting for? Can I start OSD without upgrade and get cluster healthy with old snap structure? Or should I skip octopus upgrade and go to pacific directly (some bug backport is missing?). Thank you for help, I'm sending some logs below.. Log shows 2023-01-09T19:12:49.471+0100 7f41f60f1e00 0 ceph version 15.2.17 (694d03a6f6c6e9f814446223549caf9a9f60dba0) octopus (stable), process ceph-osd, pid 2566563 2023-01-09T19:12:49.471+0100 7f41f60f1e00 0 pidfile_write: ignore empty --pid-file 2023-01-09T19:12:49.499+0100 7f41f60f1e00 -1 missing 'type' file, inferring filestore from current/ dir 2023-01-09T19:12:49.531+0100 7f41f60f1e00 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal 2023-01-09T19:12:49.531+0100 7f41f60f1e00 -1 Falling back to public interface 2023-01-09T19:12:49.871+0100 7f41f60f1e00 0 load: jerasure load: lrc load: isa 2023-01-09T19:12:49.875+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342) 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: splice() is disabled via 'filestore splice' config option 2023-01-09T19:12:49.983+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2023-01-09T19:12:49.983+0100 7f41f60f1e00 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf 2023-01-09T19:12:50.015+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) start omap initiation 2023-01-09T19:12:50.079+0100 7f41f60f1e00 1 leveldb: Recovering log #165531 2023-01-09T19:12:50.083+0100 7f41f60f1e00 1 leveldb: Level-0 table #165533: started 2023-01-09T19:12:50.235+0100 7f41f60f1e00 1 leveldb: Level-0 table #165533: 1598 bytes OK 2023-01-09T19:12:50.583+0100 7f41f60f1e00 1 leveldb: Delete type=0 #165531 2023-01-09T19:12:50.615+0100 7f41f60f1e00 1 leveldb: Delete type=3 #165529 2023-01-09T19:12:51.339+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) mount(1861): enabling WRITEAHEAD journal mode: checkpoint is not enabled 2023-01-09T19:12:51.379+0100 7f41f60f1e00 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 35: 2998927360 bytes, block size 4096 bytes, directio = 1, aio = 1 2023-01-09T19:12:51.931+0100 7f41f60f1e00 -1 journal do_read_entry(243675136): bad header magic 2023-01-09T19:12:51.939+0100 7f41f60f1e00 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 35: 2998927360 bytes, block size 4096 bytes, directio = 1, aio = 1 2023-01-09T19:12:51.943+0100 7f41f60f1e00 1 filestore(/var/lib/ceph/osd/ceph-0) upgrade(1466) 2023-01-09T19:12:52.015+0100 7f41f60f1e00 1 osd.0 126556 init upgrade snap_mapper (first start as octopus) lsof shows COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ceph-osd 225860 ceph cwd DIR 9,127 4096 2 / ceph-osd 225860 ceph rtd DIR 9,127 4096 2 / ceph-osd 225860 ceph txt REG 9,127 31762544 5021 /usr/bin/ceph-osd ceph-osd 225860 ceph mem REG 8,70 2147237 68104224 /var/lib/ceph/osd/ceph-0/current/omap/165546.ldb ceph-osd 225860 ceph mem REG 8,70 2147792 68104190 /var/lib/ceph/osd/ceph-0/current/omap/165545.ldb ceph-osd 225860 ceph mem REG 8,70 2147689 68104240 /var/lib/ceph/osd/ceph-0/current/omap/165466.ldb ceph-osd 225860 ceph mem REG 8,70 2142721 68102679 /var/lib/ceph/osd/ceph-0/current/omap/165544.ldb ceph-osd 225860 ceph mem REG 8,70 2142677 68104239 /var/lib/ceph/osd/ceph-0/current/omap/165465.ldb ceph-osd 225860 ceph mem REG 8,70 2144979 68078254 /var/lib/ceph/osd/ceph-0/current/omap/165543.ldb ceph-osd 225860 ceph mem REG 8,70 2143705 68163491 /var/lib/ceph/osd/ceph-0/current/omap/165526.ldb ceph-osd 225860 ceph mem REG 8,70 2141468 68163492 /var/lib/ceph/osd/ceph-0/current/omap/165527.ldb ceph-osd 225860 ceph mem REG 8,70 145986 68018644 /var/lib/ceph/osd/ceph-0/current/omap/165541.ldb ceph-osd 225860 ceph mem REG 8,70 2143434 68163490 /var/lib/ceph/osd/ceph-0/current/omap/165525.ldb ceph-osd 225860 ceph mem REG 8,70 2136002 68122351 /var/lib/ceph/osd/ceph-0/current/omap/165472.ldb ceph-osd 225860 ceph mem REG 8,70 1965262 68119647 /var/lib/ceph/osd/ceph-0/current/omap/165467.ldb ceph-osd 225860 ceph mem REG 8,70 2145206 68104229 /var/lib/ceph/osd/ceph-0/current/omap/165464.ldb ceph-osd 225860 ceph mem REG 8,70 61600 68002130 /var/lib/ceph/osd/ceph-0/current/omap/165536.ldb ceph-osd 225860 ceph mem REG 8,70 352689 67945734 /var/lib/ceph/osd/ceph-0/current/omap/165530.ldb ceph-osd 225860 ceph mem REG 8,70 82500 67480519 /var/lib/ceph/osd/ceph-0/current/omap/165539.ldb ceph-osd 225860 ceph mem REG 9,127 189200 2878 /usr/lib/ceph/erasure-code/libec_isa.so ceph-osd 225860 ceph mem REG 9,127 2021136 3033 /usr/lib/ceph/erasure-code/libec_lrc.so ceph-osd 225860 ceph mem REG 9,127 360544 2942 /usr/lib/ceph/erasure-code/libec_jerasure.so ceph-osd 225860 ceph mem REG 9,127 55792 15227 /lib/x86_64-linux-gnu/libnss_files-2.28.so ceph-osd 225860 ceph mem REG 9,127 22368 133997 /usr/lib/x86_64-linux-gnu/liburcu-common.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 42976 133996 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 51128 149821 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0.0.0 ceph-osd 225860 ceph mem REG 8,70 33977 68116674 /var/lib/ceph/osd/ceph-0/current/omap/165547.ldb ceph-osd 225860 ceph DEL REG 0,19 64261192 /[aio] ceph-osd 225860 ceph DEL REG 0,19 64261191 /[aio] ceph-osd 225860 ceph mem REG 9,127 158400 2523 /lib/x86_64-linux-gnu/liblzma.so.5.2.4 ceph-osd 225860 ceph mem REG 9,127 135064 12635 /lib/x86_64-linux-gnu/libnl-3.so.200.26.0 ceph-osd 225860 ceph mem REG 9,127 488272 133248 /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200.26.0 ceph-osd 225860 ceph mem REG 9,127 35808 15263 /lib/x86_64-linux-gnu/librt-2.28.so ceph-osd 225860 ceph mem REG 9,127 55512 133042 /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1 ceph-osd 225860 ceph mem REG 9,127 30776 16463 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 ceph-osd 225860 ceph mem REG 9,127 1820400 15070 /lib/x86_64-linux-gnu/libc-2.28.so ceph-osd 225860 ceph mem REG 9,127 100712 7687 /lib/x86_64-linux-gnu/libgcc_s.so.1 ceph-osd 225860 ceph mem REG 9,127 1579448 15202 /lib/x86_64-linux-gnu/libm-2.28.so ceph-osd 225860 ceph mem REG 9,127 1570256 133576 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 ceph-osd 225860 ceph mem REG 9,127 112800 133289 /usr/lib/x86_64-linux-gnu/librdmacm.so.1.1.22.1 ceph-osd 225860 ceph mem REG 9,127 104496 133287 /usr/lib/x86_64-linux-gnu/libibverbs.so.1.5.22.1 ceph-osd 225860 ceph mem REG 9,127 149704 5527 /lib/x86_64-linux-gnu/libudev.so.1.6.13 ceph-osd 225860 ceph mem REG 9,127 146968 15251 /lib/x86_64-linux-gnu/libpthread-2.28.so ceph-osd 225860 ceph mem REG 9,127 3044224 135371 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 ceph-osd 225860 ceph mem REG 9,127 14592 15128 /lib/x86_64-linux-gnu/libdl-2.28.so ceph-osd 225860 ceph mem REG 9,127 93000 15255 /lib/x86_64-linux-gnu/libresolv-2.28.so ceph-osd 225860 ceph mem REG 9,127 301288 134118 /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3 ceph-osd 225860 ceph mem REG 9,127 14184 134682 /usr/lib/x86_64-linux-gnu/libaio.so.1.0.1 ceph-osd 225860 ceph mem REG 9,127 117184 2738 /lib/x86_64-linux-gnu/libz.so.1.2.11 ceph-osd 225860 ceph mem REG 9,127 121184 136027 /usr/lib/x86_64-linux-gnu/liblz4.so.1.8.3 ceph-osd 225860 ceph mem REG 9,127 31104 133334 /usr/lib/x86_64-linux-gnu/libsnappy.so.1.1.7 ceph-osd 225860 ceph mem REG 9,127 378160 133086 /usr/lib/x86_64-linux-gnu/libleveldb.so.1d.20 ceph-osd 225860 ceph mem REG 9,127 256120 7650 /lib/x86_64-linux-gnu/libfuse.so.2.9.9 ceph-osd 225860 ceph mem REG 9,127 343008 16890 /lib/x86_64-linux-gnu/libblkid.so.1.1.0 ceph-osd 225860 ceph mem REG 8,70 1546 68022625 /var/lib/ceph/osd/ceph-0/current/omap/165542.ldb ceph-osd 225860 ceph mem REG 8,70 1598 67109142 /var/lib/ceph/osd/ceph-0/current/omap/165533.ldb ceph-osd 225860 ceph mem REG 9,127 35112 133884 /usr/lib/x86_64-linux-gnu/liburcu-bp.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 165632 15000 /lib/x86_64-linux-gnu/ld-2.28.so ceph-osd 225860 ceph 0r CHR 1,3 0t0 6 /dev/null ceph-osd 225860 ceph 1u unix 0x00000000ab7b4943 0t0 64259884 type=STREAM ceph-osd 225860 ceph 2u unix 0x00000000ab7b4943 0t0 64259884 type=STREAM ceph-osd 225860 ceph 3u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 4r FIFO 0,13 0t0 64259943 pipe ceph-osd 225860 ceph 5w FIFO 0,13 0t0 64259943 pipe ceph-osd 225860 ceph 6u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 7r FIFO 0,13 0t0 64259944 pipe ceph-osd 225860 ceph 8w FIFO 0,13 0t0 64259944 pipe ceph-osd 225860 ceph 9u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 10r FIFO 0,13 0t0 64259945 pipe ceph-osd 225860 ceph 11w FIFO 0,13 0t0 64259945 pipe ceph-osd 225860 ceph 12w REG 9,127 8765 3314 /var/log/ceph/ceph-osd.0.log ceph-osd 225860 ceph 13r FIFO 0,13 0t0 64259950 pipe ceph-osd 225860 ceph 14w FIFO 0,13 0t0 64259950 pipe ceph-osd 225860 ceph 15u unix 0x0000000097fdc46a 0t0 64259951 /var/run/ceph/ceph-osd.0.asok type=STREAM ceph-osd 225860 ceph 24r FIFO 0,13 0t0 64261188 pipe ceph-osd 225860 ceph 25w FIFO 0,13 0t0 64261188 pipe ceph-osd 225860 ceph 26r FIFO 0,13 0t0 64261189 pipe ceph-osd 225860 ceph 27w FIFO 0,13 0t0 64261189 pipe ceph-osd 225860 ceph 28uW REG 8,70 37 21 /var/lib/ceph/osd/ceph-0/fsid ceph-osd 225860 ceph 29r DIR 8,70 262 16 /var/lib/ceph/osd/ceph-0 ceph-osd 225860 ceph 30r DIR 8,70 24576 27 /var/lib/ceph/osd/ceph-0/current ceph-osd 225860 ceph 31u REG 8,70 10 28 /var/lib/ceph/osd/ceph-0/current/commit_op_seq ceph-osd 225860 ceph 32uW REG 8,70 0 67108882 /var/lib/ceph/osd/ceph-0/current/omap/LOCK ceph-osd 225860 ceph 33w REG 8,70 813568 68035819 /var/lib/ceph/osd/ceph-0/current/omap/165540.log ceph-osd 225860 ceph 34w REG 8,70 32068 68072533 /var/lib/ceph/osd/ceph-0/current/omap/MANIFEST-165538 ceph-osd 225860 ceph 36u BLK 8,65 0t0 422 /dev/sde1 ceph-osd 225860 ceph 37u REG 8,70 538 134217745 /var/lib/ceph/osd/ceph-0/current/meta/DIR_E/DIR_D/DIR_C/osd\\usuperblock__0_23C2FCDE__none ceph-osd 225860 ceph 38u REG 8,70 22453 1075330317 /var/lib/ceph/osd/ceph-0/current/meta/DIR_4/DIR_B/DIR_4/osdmap.126556__0_DF0524B4__none strace shows strace: Process 225860 attached futex(0x555f3e1c05c8, FUTEX_WAIT_PRIVATE, 0, NULL -- ============ Ing. Jan Pekař jan.pekar(a)imatic.cz ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 https://www.imatic.cz | +420326555326 ============ --

6 months, 3 weeks

3
8
0 0

2023

2022

2021

2020

2019

ceph-users January 2023