- ceph-users - lists.ceph.io

Radosgw-admin bucket list has duplicate objects
by mahnoosh shahidi 17 Apr '23

17 Apr '23

Hi, I observed duplicate object names in the result of the admin list bucket on 15.2.12 cluster. I used the following command and some of the object names in the result list appeared more than once. There is no versioning config for the bucket. radosgw-admin bucket list --allow-unordered --max-entries=200000000 > --bucket=my-bucket > my-bucket.list Does anyone have any opinion on this? Should I make a bug report for this? Best Regards, Mahnoosh

3 3

OSDs remain not in after update to v17
by Alexandre Becholey 17 Apr '23

17 Apr '23

Dear Ceph Users, I have a small ceph cluster for VMs on my local machine. It used to be installed with the system packages and I migrated it to docker following the documentation. It worked OK until I migrated from v16 to v17 a few months ago. Now the OSDs remain "not in" as shown in the status: # ceph -s cluster: id: abef2e91-cd07-4359-b457-f0f8dc753dfa health: HEALTH_WARN 6 stray daemon(s) not managed by cephadm 1 stray host(s) with 6 daemon(s) not managed by cephadm 2 devices (4 osds) down 4 osds down 1 host (4 osds) down 1 root (4 osds) down Reduced data availability: 129 pgs inactive services: mon: 1 daemons, quorum bjorn (age 8m) mgr: bjorn(active, since 8m) osd: 4 osds: 0 up (since 4w), 4 in (since 4w) data: pools: 2 pools, 129 pgs objects: 0 objects, 0 B usage: 1.8 TiB used, 1.8 TiB / 3.6 TiB avail pgs: 100.000% pgs unknown 129 unknown I can see some network communication between the OSDs and the monitor and the OSDs are running: # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f8fbe8177a63 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-2 6768ec871404 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-1 ff82f84504d5 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-0 4c89e50ce974 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-3 fe0b6089edda quay.io/ceph/ceph:v17 "/usr/bin/ceph-mon -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-mon-bjorn f76ac9dcdd6d quay.io/ceph/ceph:v17 "/usr/bin/ceph-mgr -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-mgr-bjorn However when I try to use any `ceph orch` commands, they hang. I can also see some blacklist on the OSDs: # ceph osd blocklist ls 10.99.0.13:6833/3770763474 2023-04-13T08:17:38.885128+0000 10.99.0.13:6832/3770763474 2023-04-13T08:17:38.885128+0000 10.99.0.13:0/2634718754 2023-04-13T08:17:38.885128+0000 10.99.0.13:0/1103315748 2023-04-13T08:17:38.885128+0000 listed 4 entries The first two entries correspond to the manager process. `ceph osd blocked-by` does not show anything. I think I might have forgotten to set the `ceph osd require-osd-release ...` because 14 is written in `/var/lib/ceph/<ID>/osd.?/require_osd_release`. If I try to do it now, the monitor hits an abort: debug 0> 2023-04-12T08:43:27.788+0000 7f0fcf2aa700 -1 *** Caught signal (Aborted) ** in thread 7f0fcf2aa700 thread_name:ms_dispatch ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f0fd94bbcf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0fdb5124e3] 5: /usr/lib64/ceph/libceph-common.so.2(+0x26a64f) [0x7f0fdb51264f] 6: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basi 7: (OSDMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x38d) [0x562719cb127d] 8: (OSDMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x17b) [0x562719cb18cb] 9: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x2ce) [0x562719c20ade] 10: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x1ebb) [0x562719ab9f6b] 11: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x9f2) [0x562719abe152] 12: (Monitor::_ms_dispatch(Message*)+0x406) [0x562719abf066] 13: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x562719aef13d] 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7f0fdb78e0e8] 15: (DispatchQueue::entry()+0x50f) [0x7f0fdb78b52f] 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0fdb8543b1] 17: /lib64/libpthread.so.0(+0x81ca) [0x7f0fd94b11ca] 18: clone() Any ideas on what is going on? Many thanks, Alexandre

3 4

CEPH Mirrors are lacking packages
by Burkhard Linke 17 Apr '23

17 Apr '23

Hi, at least eu.ceph.com and de.ceph.com are lacking packages for the pacific release. All package not start with "c" (e.g. librbd, librados, radosgw) are missing. Best regards, Burkhard Linke

3 2

Cannot add disks back after their OSDs were drained and removed from a cluster
by stillsmil＠gmail.com 17 Apr '23

17 Apr '23

I find that I cannot re-add a disk to a Ceph cluster after the OSD on the disk is removed. Ceph seems to know about the existence of these disks, but not about their "host:dev" information: ``` # ceph device ls DEVICE HOST:DEV DAEMONS WEAR LIFE EXPECTANCY SAMSUNG_MZ7L37T6_S6KHNE0T100049 0% <-- should be host01:sda, was osd.0 SAMSUNG_MZ7L37T6_S6KHNE0T100050 host01:sdb osd.1 0% SAMSUNG_MZ7L37T6_S6KHNE0T100052 0% <-- should be host02:sda SAMSUNG_MZ7L37T6_S6KHNE0T100053 host01:sde osd.9 0% SAMSUNG_MZ7L37T6_S6KHNE0T100061 host01:sdf osd.11 0% SAMSUNG_MZ7L37T6_S6KHNE0T100062 0% SAMSUNG_MZ7L37T6_S6KHNE0T100063 host01:sdc osd.5 0% SAMSUNG_MZ7L37T6_S6KHNE0T100064 host01:sdg osd.13 0% SAMSUNG_MZ7L37T6_S6KHNE0T100065 0% SAMSUNG_MZ7L37T6_S6KHNE0T100066 host01:sdd osd.7 0% SAMSUNG_MZ7L37T6_S6KHNE0T100067 0% SAMSUNG_MZ7L37T6_S6KHNE0T100068 0% <-- should be host02:sdb SAMSUNG_MZ7L37T6_S6KHNE0T100069 0% SAMSUNG_MZ7L37T6_S6KHNE0T100070 0% SAMSUNG_MZ7L37T6_S6KHNE0T100071 0% SAMSUNG_MZ7L37T6_S6KHNE0T100072 host01:sdh osd.15 0% SAMSUNG_MZQL27T6HBLA-00B7C_S6CVNG0T321600 host03:nvme4n1 osd.20 0% ... obmitted ... SAMSUNG_MZQL27T6HBLA-00B7C_S6CVNG0T321608 host03:nvme8n1 osd.22 0% ``` For disk "SAMSUNG_MZ7L37T6_S6KHNE0T100049", the "HOST:DEV" field is empty, while I believe it should be "host01:sda", as I have confirmed by running `smartctl -i /dev/sda" on host01. I guess the missing information is the reason that OSDs cannot be created, either manually or automatically on these devices. I have tried: 1. `ceph orch daemon add osd host01:/dev/sda` Prints "Created no osd(s) on host host01; already created?" 2. `ceph orch apply osd --all-available-devices` adds nothing I arrived at such situation while testing if draining a host works: I drained host02, removed it and added it back via: ``` ceph orch host drain host02 ceph orch host rm host02 ceph orch host add host02 <internal_ip> --labels _admin ``` I am running Ceph 17.2.6 on ubuntu. Ceph was deployed via cephadm. FYI the relavent orchestrator spec for osd: ``` # ceph orch ls osd --export service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore --- service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore ``` Any thoughts on what maybe wrong here? Is there a way that I can tell Ceph "you are wrong about the whereabouts of these disks, forget what you know and fetch disk information afresh"? Any help much appreciated!

1 0

Ceph mon status stuck at "probing"
by York Huang 16 Apr '23

16 Apr '23

Hi Ceph community, I had a small lab cluster of version Nautilus (ceph-ansible / containerized) which functions quite well. As part of the upgrade experiment, I replaced one of the mon with an Octopus one (containerized as well, the OS is purged before mon deployment), the daemon seems working but its status is always "probing", and "ceph -s" showed that mon is down and out of quorum. I checked and compared the output of "ceph daemon XXX mon_status" of all monitors, ensuring there is no network issue, the new mon seemed to have the correct monmap but always stucked in status "probing". I doubt if Ceph mon of different versions do co-exist. Any further steps suggest? Thanks in advanced.

1 0

Dead node (watcher) won't timeout on RBD
by Max Boone 16 Apr '23

16 Apr '23

After a critical node failure on my lab cluster, which won't come back up and is still down, the RBD objects are still being watched / mounted according to ceph. I can't shell to the node to rbd unbind them as the node is down. I am absolutely certain that nothing is using these images and they don't have snapshots either (and this IP is not even remotely close to the those of the monitors in the cluster). I blocked the IP usingceph osd blocklist add but after 30 minutes, they are still being watched. Them being watched (they are RWO ceph-csi volumes) prevents me from re-using them in the cluster. As far as I'm aware, ceph should remove the watchers after 30 minutes and they've been blocklisted for hours now.root@node0:~# rbd status kubernetes/csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff Watchers: watcher=10.0.0.103:0/992994811 client.1634081 cookie=139772597209280 root@node0:~# rbd snap list kubernetes/csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff root@node0:~# rbd info kubernetes/csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff rbd image 'csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff': size 10 GiB in 2560 objects order 22 (4 MiB objects) snapshot_count: 0 id: 4ff5353b865e1 block_name_prefix: rbd_data.4ff5353b865e1 format: 2 features: layering op_features: flags: create_timestamp: Fri Mar 31 14:46:51 2023 access_timestamp: Fri Mar 31 14:46:51 2023 modify_timestamp: Fri Mar 31 14:46:51 2023 root@node0:~# rados -p kubernetes listwatchers rbd_header.4ff5353b865e1 watcher=10.0.0.103:0/992994811 client.1634081 cookie=139772597209280 root@node0:~# ceph osd blocklist ls 10.0.0.103:0/0 2023-04-16T13:58:34.854232+0200 listed 1 entries root@node0:~# ceph daemon osd.0 config get osd_client_watch_timeout { "osd_client_watch_timeout": "30" } Is it possible to kick a watcher out manually, or is there not much I can do here besides shutting down the entire cluster (or OSDs) and getting them back up? If it is a bug, I'm happy to help figuring out it's root cause and see if I can help writing a fix. Cheers, Max.

2 1

Re: Nothing provides libthrift-0.14.0.so()(64bit)
by Will Nilges 15 Apr '23

15 Apr '23

It's funny, I had the EPEL repos added yesterday, and no joy. Left it alone for a day, and voila, packages exist. Must have forgotten to update or clear my cache or something? Thanks for the help!

1 0

Nothing provides libthrift-0.14.0.so()(64bit)
by Will Nilges 15 Apr '23

15 Apr '23

Hello! I'm trying to Install the ceph-common package on a Rocky Linux 9 box so that I can connect to our ceph cluster and mount user directories. I've added the ceph repo to yum.repos.d, but when I run `dnf install ceph-common`, I get the following error ``` [root@jet yum.repos.d]# dnf install ceph-common Last metadata expiration check: 0:33:06 ago on Fri 14 Apr 2023 11:29:26 AM EDT. Error: Problem: conflicting requests - nothing provides libthrift-0.14.0.so()(64bit) needed by ceph-common-2:17.2.6-0.el9.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) ``` I've looked far and wide for this libthrift package, but I can't figure out what repo to add to get it. Has anyone had success installing ceph on Rocky 9 and would be willing to share some guidance?

2 1

ceph/daemon stable tag
by Jonas Nemeikšis 15 Apr '23

15 Apr '23

Hello, What's the status with the *-stable-* tags? https://quay.io/repository/ceph/daemon?tab=tags No longer build/support? What should we use until we'll migrate from ceph-ansible to cephadm? Thanks. -- Jonas

2 6

Mysteriously dead OSD process
by J-P Methot 15 Apr '23

15 Apr '23

Hi, We currently use Ceph Pacific 16.2.10 deployed with Cephadm on this storage cluster. Last night, one of our OSD died. However, since its storage is a SSD, we ran hardware checks and found no issue with the SSD itself. However, if we try starting the service again, the container just crashes 1 second after booting up. If I look at the logs, there's no error. You can see the OSD starting up normally and then the last line before the crash is : debug 2023-04-05T18:32:57.433+0000 7f8078e0c700 1 osd.87 pg_epoch: 207175 pg[2.99s3( v 207174'218628609 (207134'218623666,207174'218628609] local-lis/les=207140/207141 n=38969 ec=41966/315 lis/c=207140/207049 les/c/f=207141/207050/0 sis=207175 pruub=11.464111328s) [5,228,217,NONE,17,25,167,114,158,178,159]/[5,228,217,87,17,25,167,114,158,178,159]p5(0) r=3 lpr=207175 pi=[207049,207175)/1 crt=207174'218628605 mlcod 0'0 remapped NOTIFY pruub 12054.601562500s@ mbc={}] state<Start>: transitioning to Stray I don't really see how this line could cause the OSD to crash. Systemd just writes : Stopping Ceph osd.83 for (uuid) What could cause this OSD to boot up and then suddenly die? Outside the ceph daemon logs and the systemd logs, is there another way I could gain more information? -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.

2 1