- ceph-users - lists.ceph.io

About scrub and deep-scrub
by Phong Tran Thanh 09 Oct '24

09 Oct '24

Hi ceph users! How about the disable scrub and deep-scrub, i want to disable it because of its effect on many I/O of my cluster. If I disable scrub how will it affect my cluster? If enabled, scrubbing is not complete and takes a long time. Thank Skype: tranphong079

4 9

The ceph monitor crashes every few days
by 李明 09 Oct '24

09 Oct '24

Hello, ceph version is 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) and the rbd info command is also slow, some times it needs 6 seconds. rbd snap create command takes 17 seconds. There is another cluster with the same configuration it only takes less than 1 second. crash log -2> 2024-10-09 09:52:50.782 7f30ddcb4700 5 prioritycache tune_memory target: 2147483648 mapped: 1286012928 unmapped: 1695842304 heap: 2981855232 old mem: 1020054731 new mem: 1020054731 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/mon/OSDMonitor.cc: 4112: ceph_abort_msg("abort() called") ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f30ea6c2a66] 2: (OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned long)+0x89f) [0x563b0175382f] 3: (OSDMonitor::send_incremental(unsigned int, MonSession*, bool, boost::intrusive_ptr<MonOpRequest>)+0x11d) [0x563b01753e6d] 4: (OSDMonitor::check_osdmap_sub(Subscription*)+0xca) [0x563b0175a53a] 5: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xbbe) [0x563b0162c9ee] 6: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x525) [0x563b0164b0a5] 7: (Monitor::_ms_dispatch(Message*)+0xcdb) [0x563b0164cd9b] 8: (Monitor::ms_dispatch(Message*)+0x26) [0x563b0167ad96] 9: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x563b01677776] 10: (DispatchQueue::entry()+0x1699) [0x7f30ea8e7699] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f30ea99564d] 12: (()+0x7ea5) [0x7f30e74acea5] 13: (clone()+0x6d) [0x7f30e636fb0d] 0> 2024-10-09 09:52:55.260 7f30db4af700 -1 *** Caught signal (Aborted) ** in thread 7f30db4af700 thread_name:ms_dispatch ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (()+0xf630) [0x7f30e74b4630] 2: (gsignal()+0x37) [0x7f30e62a7387] 3: (abort()+0x148) [0x7f30e62a8a78] 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f30ea6c2b2e] 5: (OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned long)+0x89f) [0x563b0175382f] 6: (OSDMonitor::send_incremental(unsigned int, MonSession*, bool, boost::intrusive_ptr<MonOpRequest>)+0x11d) [0x563b01753e6d] 7: (OSDMonitor::check_osdmap_sub(Subscription*)+0xca) [0x563b0175a53a] 8: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xbbe) [0x563b0162c9ee] 9: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x525) [0x563b0164b0a5] 10: (Monitor::_ms_dispatch(Message*)+0xcdb) [0x563b0164cd9b] 11: (Monitor::ms_dispatch(Message*)+0x26) [0x563b0167ad96] 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x563b01677776] 13: (DispatchQueue::entry()+0x1699) [0x7f30ea8e7699] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f30ea99564d] 15: (()+0x7ea5) [0x7f30e74acea5] 16: (clone()+0x6d) [0x7f30e636fb0d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this ceph -s cluster: id: 6e36443f-326b-4344-b013-c03a4b7bb8d7 health: HEALTH_WARN noout flag(s) set 314 pgs not deep-scrubbed in time 660 pgs not scrubbed in time 23 daemons have recently crashed services: mon: 3 daemons, quorum rbd02,rbd01,rbd03 (age 81m) mgr: rbd03(active, since 9d), standbys: rbd01, rbd02 osd: 360 osds: 360 up (since 9d), 360 in (since 12M) flags noout data: pools: 1 pools, 8192 pgs objects: 109.45M objects, 274 TiB usage: 488 TiB used, 3.4 PiB / 3.8 PiB avail pgs: 8116 active+clean 76 active+clean+scrubbing+deep io: client: 61 MiB/s rd, 34 MiB/s wr, 74.51k op/s rd, 2.06k op/s wr

1 0

Administrative test, please ignore
by Anthony D'Atri 08 Oct '24

08 Oct '24

Testing

1 0

Ceph.io down?!
by ceph＠elchaka.de 08 Oct '24

08 Oct '24

Hi, Not sure of this is the right path.. but the site Ceph.io Is since at least 1,5h down for me... Intendet? Mehmet

1 0

Question about bucket / object policy
by Andrea Martra 08 Oct '24

08 Oct '24

Hello ceph-users, I need to prevent the use of public objects in some Ceph S3 clusters where clients access directly via haproxy as balancer and backend on radosgw. I would like to know if anyone has found a solution to prevent entire Buckets or even single objects from being made accessible without authentication and that they can be exposed with a specific URL. I tried to set the policies in 'deny' for "s3:PutBucketPolicy", "s3:PutObjectAcl" and the objects uploaded to the bucket after the policy is applied cannot be made public. However, I would need to have a way to set the Deny on all buckets when they are created by clients because otherwise I have to iterate on all existing buckets and apply the policy, but in this way if the client creates a bucket, uploads some objects and makes them public before the Deny is applied I no longer have control. I should then also execute a policy on all objects but it becomes a very long operation. If anyone has any ideas on this it would be greatly appreciated. Thank you very much. Andrea

1 0

Ceph Steering Committee Team Meeting 2024-10-07
by Ernesto Puerta 07 Oct '24

07 Oct '24

Hi Cephers, The big topic today has been on the *Ceph project Governance*, especifically, *Ceph Steering Committee (CSC) changes *in the context of the ongoing* Ceph Executive Council (CEC) renewal*. Please check this email thread <https://www.spinics.net/lists/ceph-users/msg84293.html> and this etherpad <https://pad.ceph.com/p/css-vote-2024q4> for more details. Additionally: - We reviewed the *new candidates* proposed to the CSC (removals have been deferred), - Discussed about the *CSC vs. CLT naming* (decision to continue with CSC), - Proposed an *alternating CSC/CLT meeting* (+12h) to facilitate everyone around the Globe to join (e.g: APAC). - Proposed to *limit the max number of CSC members* (deferred). The first ballot <https://vote.heliosvoting.org/helios/elections/f276a15a-84c5-11ef-a0e4-b69e…> is now open and will run until next Monday. --- Also: @Radoslaw Zarzynski <rzarzyns(a)redhat.com> brought up the following topics: - *Facilities for testing intended FTBFSes*: driving factor: https://github.com/ceph/ceph/pull/60159/files - *Namespaces in ceph-dencoder*: let's discuss next week, CC @Ilya Dryomov <idryomov(a)redhat.com> @Nizamudeen A <nia(a)redhat.com> communicated *he's stepping down from the Ceph Dashboard team* leadership, and that @Afreen Misbah <afrahman(a)redhat.com>* will take over*. Welcome and good luck, Afreen! @John Zachary Dover <zac.dover(a)gmail.com> announced that *Ceph Quarterly <https://ceph.io/en/community/cq/> #6 will be published on the 15th of October*, and asked for Squid highlights. @Casey Bodley <cbodley(a)redhat.com> asked about the *status of the latest Quincy* release, and @Yuri Weinstein <yweinste(a)redhat.com> confirmed that it is still *in progress*, with ~8 PRs pending. --- We wanted to remind you that the CSC/CLT weekly meeting (Mondays 11am ET) is open to anyone actively involved in the project's both short and long-term challenges (feel free to review and suggest topics in https://pad.ceph.com/p/clt-weekly-minutes) Kind Regards, Ernesto

1 0

Ceph Meetup in Berlin, Germany and Frankfurt/Main, Germany - interested people welcome !
by Matthias Muench 07 Oct '24

07 Oct '24

Hi people in Germany, we run two meetup groups in Berlin and Frankfurt with meetings nearly quarterly. For each location, we provide also a quick introduction to Ceph before the main event. Currently, we have a new set of appointments. You can join us by RSVP to the events: - Ceph Frankfurt: 09.10.2024 in Frankfurt/Main, Germany Introduction: https://www.meetup.com/de-DE/ceph-frankfurt/events/303545788/ Main meetup: https://www.meetup.com/de-DE/ceph-frankfurt/events/303545742/ - Ceph Berlin: 14.10.2024 in Berlin, Germany Introduction: https://www.meetup.com/de-DE/ceph-berlin/events/303546142/ Main meetup: https://www.meetup.com/de-DE/ceph-berlin/events/303546178/ To get the invites also for any future meetings, one could also join the meetup groups: - Ceph Frankfurt: https://www.meetup.com/de-DE/ceph-frankfurt/ - Ceph Berlin: https://www.meetup.com/de-DE/ceph-berlin/ Happy to see you soon, -matt -- —————————————————— Matthias Muench Principal Specialist Solution Architect, OpenShift EMEA Storage Specialist matthias.muench(a)redhat.com Phone: +49-160-92654111 Red Hat GmbH Technopark II Werner-von-Siemens-Ring 12 85630 Grasbrunn Germany _______________________________________________________________________ Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross

1 0

cephadm crush_device_class not applied
by Eugen Block 05 Oct '24

05 Oct '24

Hi, I'm struggling to create OSDs with a dedicated crush_device_class. It worked sometimes when creating a new osd via command line (ceph orch daemon add osd host:data_devices=/dev/vdg,crush_device_class=test-hdd), but most of the time it doesn't work. I tried it with a spec file as well, it seems to be correctly parsed and everything, but the new OSDs are created with hdd class, not hdd-ec. I have this spec: cat osd-class.yaml service_type: osd service_id: hdd-ec service_name: hdd-ec crush_device_class: hdd-ec placement: label: osd spec: data_devices: rotational: 1 size: 10G objectstore: bluestore I see that cephadm has stored it correctly: ceph config-key get mgr/cephadm/spec.osd.hdd-ec {"created": "2024-10-03T08:35:41.364216Z", "needs_configuration": true, "spec": {"placement": {"label": "osd"}, "service_id": "hdd-ec", "service_name": "osd.hdd-ec", "service_type": "osd", "spec": {"crush_device_class": "hdd-ec", "data_devices": {"rotational": 1, "size": "10G"}, "filter_logic": "AND", "objectstore": "bluestore"}}} And it has the OSDSPEC_AFFINITY set: cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=hdd-ec', '--image', 'registry.domain/ceph@sha256:ca901f9ff84d77f8734afad20556775f0ebaea6c62af8cca733161f5338d3f6c', '--timeout', '895', 'ceph-volume', '--fsid', '7d60533e-7e9e-11ef-b140-fa163e2ad8c5', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/vdb', '/dev/vdc', '/dev/vdd', '/dev/vdf', '/dev/vdg', '/dev/vdh', '--yes', '--no-systemd'] But the OSDs still are created with hdd device class: ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.05878 root default -3 0.05878 host soc9-ceph 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000 I have tried it with two different indentations: spec: crush_device_class: hdd-ec and as seen above: crush_device_class: hdd-ec placement: label: osd spec: According to the docs [0], it's not supposed to be indented, so my current spec seems valid. But I see in the mgr log with debug_mgr 10 that apparently it is parsed with indentation: 2024-10-03T09:59:23.029+0000 7efef1cc6700 0 [cephadm DEBUG cephadm.services.osd] Translating DriveGroup <DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: hdd-ec service_name: osd.hdd-ec placement: label: osd spec: crush_device_class: hdd-ec data_devices: rotational: 1 size: 10G filter_logic: AND objectstore: bluestore '''))> to ceph-volume command Now I'm wondering how it's actually supposed to work. Yesterday we saw the same behaviour on a customer cluster as well with Quincy 17.2.7. This is Reef 18.2.2. Trying to create it manually also doesn't work as expected: soc9-ceph:~ # ceph orch daemon add osd soc9-ceph:data_devices=/dev/vdf,crush_device_class=hdd-ec Created osd(s) 3 on host 'soc9-ceph' soc9-ceph:~ # ceph osd tree | grep osd.3 3 hdd 0.00980 osd.3 up 1.00000 1.00000 This is in the mgr debug output from the manual creation: 2024-10-03T10:06:00.329+0000 7efeeecc0700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.create_osds(*(DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')),), **{}) 2024-10-03T10:06:00.333+0000 7efeeecc0700 0 [cephadm DEBUG cephadm.services.osd] Processing DriveGroup DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_name: osd placement: host_pattern: soc9-ceph spec: crush_device_class: hdd-ec data_devices: paths: - /dev/vdf filter_logic: AND objectstore: bluestore ''')) So parsing the manual command also results in the indented crush_device_class. Am I doing something wrong here? Thanks! Eugen [0] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-…

4 8

Compatibility check before updating from Centros stream 8 ceph 16.2.15 to 17.2.7
by xadhoom76＠gmail.com 04 Oct '24

04 Oct '24

Anyone knowns if there is any issue to upgrade from 16.2.16 to 17.2.7 (containerized ceph) on centos stream 8 ? Best regards

1 0

v19 & IPv6: unable to convert chosen address to string
by Sascha Frey 04 Oct '24

04 Oct '24

Hi, after upgrading our Ceph cluster from 18.2.2 to 19.2.0, I get the following error messages: 'ceph status' shows HEALTH_ERR 504 osds(s) are not reachable but, luckily, everything works fine. I see these related messages in the monitor logs (these messages show up for each OSD): 2024-10-01T16:26:36.185+0200 7ffbd5600640 -1 log_channel(cluster) log [ERR] : osd.101's public address is not in '2001:638:504:2011::/64' subnet ... 2024-10-01T16:34:01.011+0200 7ffbdba00640 -1 unable to convert chosen address to string: 2001:638:504:2011:9:4:1:1 ... IPv6 is enabled in ceph.conf: ms bind ipv4 = false ms bind ipv6 = true public network = 2001:638:504:2011::/64 Is this a bug? Thanks, Sascha

3 3