- ceph-users - lists.ceph.io

Reduced data availability: 3 pgs inactive, 3 pgs down
by Shain Miley 13 Oct '24

13 Oct '24

Hello, I am seeing the following information after reviewing ‘ceph health detail’: [WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive, 3 pgs down pg 0.1a is down, acting [234,35] pg 0.20 is down, acting [226,267] pg 0.2f is down, acting [227,161] When I query each of those pgs I see the following message on each of them: "peering_blocked_by": [ { "osd": 233, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } Osd.233 crashed a while ago and when I try to start it the log shows some sort of issue with the filesystem: ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable) 1: (()+0x12980) [0x7f2779617980] 2: (gsignal()+0xc7) [0x7f27782c9fb7] 3: (abort()+0x141) [0x7f27782cb921] 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x556ebe773ddf] 5: (FileStore::_do_transaction(ceph::os::Transaction&, unsigned long, int, ThreadPool::TPHandle*, char const*)+0x62b3) [0x556ebebe2753] 6: (FileStore::_do_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, unsigned long, ThreadPool::TPHandle*, char const*)+0x48) [0x556ebebe3f38] 7: (JournalingObjectStore::journal_replay(unsigned long)+0x105a) [0x556ebebfc56a] 8: (FileStore::mount()+0x438a) [0x556ebebda82a] 9: (OSD::init()+0x4d1) [0x556ebe80fdc1] 10: (main()+0x3f8c) [0x556ebe77ad2c] 11: (__libc_start_main()+0xe7) [0x7f27782acbf7] 12: (_start()+0x2a) [0x556ebe78fc4a] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. At this point I am thinking about either running an xfs repair on osd.233 and trying to see if I can get it back up (once the pgs are healthy again I would likey zap/readd or replace the drive). Another option it sounds like is to mark the osd as lost. I am just looking for advice on what exactly I should do next to try to minimize the chances of any data loss. Here is the query output for each of those pgs: https://pastebin.com/YbfnpZGC Thank you, Shain

2 4

GUI Block Images --> Restart mgr
by przemek.kuczynski＠gmail.com 12 Oct '24

12 Oct '24

When i want open block images in GUI mgr beeing restarted on 18.2.4 . After image conteiner is restarted i cannot access crash log Oct 11 13:46:38 hostname bash[27409]: debug -41> 2024-10-11T11:46:38.389+0000 7f4e2f823640 5 librbd::DiffIterate: fast diff enabled Oct 11 13:46:38 hostname bash[27409]: debug -40> 2024-10-11T11:46:38.389+0000 7f4e2f823640 5 librbd::DiffIterate: diff_iterate from 0 to 12 size from 0 to 2361393152 Oct 11 13:46:38 hostname bash[27409]: debug -39> 2024-10-11T11:46:38.389+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff121f3ef0 register_dispatch: dispatch_layer=3 Oct 11 13:46:38 hostname bash[27409]: debug -38> 2024-10-11T11:46:38.389+0000 7f518e504640 10 monclient: get_auth_request con 0x55ff103b6000 auth_method 0 Oct 11 13:46:38 hostname bash[27409]: debug -37> 2024-10-11T11:46:38.393+0000 7f4e2f823640 5 librbd::DiffIterate: fast diff enabled Oct 11 13:46:38 hostname bash[27409]: debug -36> 2024-10-11T11:46:38.393+0000 7f4e2f823640 5 librbd::DiffIterate: diff_iterate from 12 to 18446744073709551614 size from 2361393152 to 2361393152 Oct 11 13:46:38 hostname bash[27409]: debug -35> 2024-10-11T11:46:38.397+0000 7f4e2f823640 5 librbd::Watcher: 0x55ff132fac00 block_notifies: blocked_count=1 Oct 11 13:46:38 hostname bash[27409]: debug -34> 2024-10-11T11:46:38.397+0000 7f4e1a47a640 5 librbd::io::Dispatcher: 0x55ff121f3ef0 shut_down_dispatch: dispatch_layer=3 Oct 11 13:46:38 hostname bash[27409]: debug -33> 2024-10-11T11:46:38.397+0000 7f4e1a47a640 5 librbd::io::Dispatcher: 0x55ff121f3ef0 shut_down: Oct 11 13:46:38 hostname bash[27409]: debug -31> 2024-10-11T11:46:38.397+0000 7f4e1a47a640 5 librbd::io::ObjectDispatch: 0x55ff107bde10 shut_down: Oct 11 13:46:38 hostname bash[27409]: debug -29> 2024-10-11T11:46:38.397+0000 7f4e1a47a640 5 librbd::cache::WriteAroundObjectDispatch: 0x55ff1253a900 shut_down: Oct 11 13:46:38 hostname bash[27409]: debug -28> 2024-10-11T11:46:38.397+0000 7f4e1a47a640 5 asok(0x55ff051a8000) unregister_commands rbd cache flush Images/0408a651-e894-4d01-9f49-65b2418d1505 Oct 11 13:46:38 hostname bash[27409]: debug -26> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff13076090 register_dispatch: dispatch_layer=10 Oct 11 13:46:38 hostname bash[27409]: debug -25> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::QueueImageDispatch: 0x55ff1370c0e0 QueueImageDispatch: ictx=0x55ff11696000 Oct 11 13:46:38 hostname bash[27409]: debug -24> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff13076090 register_dispatch: dispatch_layer=1 Oct 11 13:46:38 hostname bash[27409]: debug -23> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::QosImageDispatch: 0x55ff13050100 QosImageDispatch: ictx=0x55ff11696000 Oct 11 13:46:38 hostname bash[27409]: debug -22> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff13076090 register_dispatch: dispatch_layer=2 Oct 11 13:46:38 hostname bash[27409]: debug -21> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::RefreshImageDispatch: 0x55ff107bdd60 RefreshImageDispatch: ictx=0x55ff11696000 Oct 11 13:46:38 hostname bash[27409]: debug -20> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff13076090 register_dispatch: dispatch_layer=4 Oct 11 13:46:38 hostname bash[27409]: debug -19> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::WriteBlockImageDispatch: 0x55ff0e6540a0 WriteBlockImageDispatch: ictx=0x55ff11696000 Oct 11 13:46:38 hostname bash[27409]: debug -18> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff13076090 register_dispatch: dispatch_layer=7 Oct 11 13:46:38 hostname bash[27409]: debug -17> 2024-10-11T11:46:38.401+0000 7f4e2f823640 5 librbd::io::Dispatcher: 0x55ff0ea65000 register_dispatch: dispatch_layer=6 Oct 11 13:46:38 hostname bash[27409]: debug -16> 2024-10-11T11:46:38.401+0000 7f4e1a47a640 5 asok(0x55ff051a8000) register_command rbd cache flush Images/10f3af9e-1766-4dbf-9cdb-416436027b23 hook 0x55ff13050f00 Oct 11 13:46:38 hostname bash[27409]: debug -15> 2024-10-11T11:46:38.401+0000 7f4e1a47a640 5 asok(0x55ff051a8000) register_command rbd cache invalidate Images/10f3af9e-1766-4dbf-9cdb-416436027b23 hook 0x55ff13050f00 Oct 11 13:46:38 hostname bash[27409]: debug -14> 2024-10-11T11:46:38.401+0000 7f4e1a47a640 5 librbd::ImageCtx: 0x55ff11696000: disabling zero-copy writes Oct 11 13:46:38 hostname bash[27409]: debug -12> 2024-10-11T11:46:38.401+0000 7f4e1a47a640 5 librbd::cache::WriteAroundObjectDispatch: 0x55ff1253a900 init: Oct 11 13:46:38 hostname bash[27409]: debug -11> 2024-10-11T11:46:38.401+0000 7f4e1a47a640 5 librbd::io::Dispatcher: 0x55ff0ea65000 register_dispatch: dispatch_layer=1 Oct 11 13:46:38 hostname bash[27409]: debug -10> 2024-10-11T11:46:38.405+0000 7f4e1ac7b640 5 librbd::io::SimpleSchedulerObjectDispatch: 0x55ff1304c6c0 SimpleSchedulerObjectDispatch: ictx=0x55ff11696000 Oct 11 13:46:38 hostname bash[27409]: debug -9> 2024-10-11T11:46:38.405+0000 7f4e1ac7b640 5 librbd::io::SimpleSchedulerObjectDispatch: 0x55ff1304c6c0 init: Oct 11 13:46:38 hostname bash[27409]: debug -8> 2024-10-11T11:46:38.405+0000 7f4e1ac7b640 5 librbd::io::Dispatcher: 0x55ff0ea65000 register_dispatch: dispatch_layer=5 Oct 11 13:46:38 hostname bash[27409]: debug -6> 2024-10-11T11:46:38.405+0000 7f4e1ac7b640 5 librbd::io::Dispatcher: 0x55ff13076090 shut_down_dispatch: dispatch_layer=3 Oct 11 13:46:38 hostname bash[27409]: debug -5> 2024-10-11T11:46:38.405+0000 7f4e1a47a640 5 librbd::io::WriteBlockImageDispatch: 0x55ff0e6540a0 unblock_writes: 0x55ff11696000, num=0 Oct 11 13:46:38 hostname bash[27409]: debug -3> 2024-10-11T11:46:38.409+0000 7f4e1a47a640 5 librbd::io::WriteBlockImageDispatch: 0x55ff0e6540a0 unblock_writes: 0x55ff11696000, num=0 Oct 11 13:46:38 hostname bash[27409]: debug -2> 2024-10-11T11:46:38.409+0000 7f4e2f823640 5 librbd::DiffIterate: fast diff enabled Oct 11 13:46:38 hostname bash[27409]: debug -1> 2024-10-11T11:46:38.409+0000 7f4e2f823640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/librbd/api/DiffIterate.cc: In function 'int librbd::api::DiffIterate<ImageCtxT>::execute() [with ImageCtxT = librbd::ImageCtx]' thread 7f4e2f823640 time 2024-10-11T11:46:38.414077+0000 Oct 11 13:46:38 hostname bash[27409]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/librbd/api/DiffIterate.cc: 341: FAILED ceph_assert(object_diff_state.size() == end_object_no - start_object_no) Oct 11 13:46:38 hostname bash[27409]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Oct 11 13:46:38 hostname bash[27409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f51910e504d] Oct 11 13:46:38 hostname bash[27409]: 4: /lib64/librbd.so.1(+0x51ada7) [0x7f5181bf1da7] Oct 11 13:46:38 hostname bash[27409]: 6: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x630bc) [0x7f5181e7c0bc] Oct 11 13:46:38 hostname bash[27409]: 8: PyVectorcall_Call() Oct 11 13:46:38 hostname bash[27409]: 9: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x44d50) [0x7f5181e5dd50] Oct 11 13:46:38 hostname bash[27409]: 10: _PyObject_MakeTpCall() Oct 11 13:46:38 hostname bash[27409]: 11: /lib64/libpython3.9.so.1.0(+0x125133) [0x7f5191c0a133] Oct 11 13:46:38 hostname bash[27409]: 12: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 14: _PyFunction_Vectorcall() Oct 11 13:46:38 hostname bash[27409]: 17: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7f5191c01b73] Oct 11 13:46:38 hostname bash[27409]: 18: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 19: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 20: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7f5191c01b73] Oct 11 13:46:38 hostname bash[27409]: 21: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 22: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 23: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7f5191bf3c35] Oct 11 13:46:38 hostname bash[27409]: 25: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 26: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 27: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7f5191bf3c35] Oct 11 13:46:38 hostname bash[27409]: 29: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 30: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 31: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7f5191bf3c35] Oct 11 13:46:38 hostname bash[27409]: debug 0> 2024-10-11T11:46:38.413+0000 7f4e2f823640 -1 *** Caught signal (Aborted) ** Oct 11 13:46:38 hostname bash[27409]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5190a8e6f0] Oct 11 13:46:38 hostname bash[27409]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5190adb94c] Oct 11 13:46:38 hostname bash[27409]: 3: raise() Oct 11 13:46:38 hostname bash[27409]: 4: abort() Oct 11 13:46:38 hostname bash[27409]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f51910e50a7] Oct 11 13:46:38 hostname bash[27409]: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7f51910e520b] Oct 11 13:46:38 hostname bash[27409]: 7: /lib64/librbd.so.1(+0x193403) [0x7f518186a403] Oct 11 13:46:38 hostname bash[27409]: 9: rbd_diff_iterate2() Oct 11 13:46:38 hostname bash[27409]: 11: /lib64/libpython3.9.so.1.0(+0x11d7a1) [0x7f5191c027a1] Oct 11 13:46:38 hostname bash[27409]: 13: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x44d50) [0x7f5181e5dd50] Oct 11 13:46:38 hostname bash[27409]: 14: _PyObject_MakeTpCall() Oct 11 13:46:38 hostname bash[27409]: 16: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 17: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7f5191bf3c35] Oct 11 13:46:38 hostname bash[27409]: 18: _PyFunction_Vectorcall() Oct 11 13:46:38 hostname bash[27409]: 19: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 20: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 21: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7f5191c01b73] Oct 11 13:46:38 hostname bash[27409]: 22: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 24: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7f5191c01b73] Oct 11 13:46:38 hostname bash[27409]: 26: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 27: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7f5191bf3c35] Oct 11 13:46:38 hostname bash[27409]: 29: /lib64/libpython3.9.so.1.0(+0x125031) [0x7f5191c0a031] Oct 11 13:46:38 hostname bash[27409]: 30: _PyEval_EvalFrameDefault() Oct 11 13:46:38 hostname bash[27409]: 0/ 5 none Oct 11 13:46:38 hostname bash[27409]: 0/ 1 context Oct 11 13:46:38 hostname bash[27409]: 1/ 5 mds_balancer Oct 11 13:46:38 hostname bash[27409]: 1/ 5 mds_log Oct 11 13:46:38 hostname bash[27409]: 1/ 5 mds_log_expire Oct 11 13:46:38 hostname bash[27409]: 1/ 5 mds_migrator Oct 11 13:46:38 hostname bash[27409]: 0/ 1 buffer Oct 11 13:46:38 hostname bash[27409]: 0/ 1 timer Oct 11 13:46:38 hostname bash[27409]: 0/ 1 objecter Oct 11 13:46:38 hostname bash[27409]: 0/ 5 rados Oct 11 13:46:38 hostname bash[27409]: 0/ 5 rbd_mirror Oct 11 13:46:38 hostname bash[27409]: 0/ 5 rbd_replay Oct 11 13:46:38 hostname bash[27409]: 0/ 5 rbd_pwl Oct 11 13:46:38 hostname bash[27409]: 0/ 5 journaler Oct 11 13:46:38 hostname bash[27409]: 0/ 5 immutable_obj_cache Oct 11 13:46:38 hostname bash[27409]: 1/ 5 osd Oct 11 13:46:38 hostname bash[27409]: 0/ 5 objclass Oct 11 13:46:38 hostname bash[27409]: 0/ 0 ms Oct 11 13:46:38 hostname bash[27409]: 0/10 monc Oct 11 13:46:38 hostname bash[27409]: 1/ 5 paxos Oct 11 13:46:38 hostname bash[27409]: 0/ 5 tp Oct 11 13:46:38 hostname bash[27409]: 1/ 5 crypto Oct 11 13:46:38 hostname bash[27409]: 1/ 5 heartbeatmap Oct 11 13:46:38 hostname bash[27409]: 1/ 5 rgw_sync Oct 11 13:46:38 hostname bash[27409]: 1/ 5 rgw_datacache Oct 11 13:46:38 hostname bash[27409]: 1/ 5 rgw_flight Oct 11 13:46:38 hostname bash[27409]: 1/ 5 asok Oct 11 13:46:38 hostname bash[27409]: 1/ 1 throttle Oct 11 13:46:38 hostname bash[27409]: 0/ 0 refs Oct 11 13:46:38 hostname bash[27409]: 1/ 5 compressor Oct 11 13:46:38 hostname bash[27409]: 1/ 5 bluestore Oct 11 13:46:38 hostname bash[27409]: 1/ 5 kstore Oct 11 13:46:38 hostname bash[27409]: 4/ 5 rocksdb Oct 11 13:46:38 hostname bash[27409]: 1/ 5 fuse Oct 11 13:46:38 hostname bash[27409]: 2/ 5 mgr Oct 11 13:46:38 hostname bash[27409]: 0/ 5 test Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_onode Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_odata Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_t Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_cleaner Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_epm Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_lba Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_fixedkv_tree Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_cache Oct 11 13:46:38 hostname bash[27409]: 0/ 5 seastore_device Oct 11 13:46:38 hostname bash[27409]: 0/ 5 cyanstore Oct 11 13:46:38 hostname bash[27409]: 1/ 5 ceph_exporter Oct 11 13:46:38 hostname bash[27409]: -2/-2 (syslog threshold) Oct 11 13:46:38 hostname bash[27409]: 99/99 (stderr threshold) Oct 11 13:46:38 hostname bash[27409]: 7f4e1a47a640 / io_context_pool Oct 11 13:46:38 hostname bash[27409]: 7f4e1b47c640 / safe_timer Oct 11 13:46:38 hostname bash[27409]: 7f4e1e000640 / ms_dispatch Oct 11 13:46:38 hostname bash[27409]: 7f4e1f803640 / io_context_pool Oct 11 13:46:38 hostname bash[27409]: 7f4e2b01a640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e2c01c640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e2c81d640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e2e020640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e2e821640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e2f022640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e2f823640 / dashboard Oct 11 13:46:38 hostname bash[27409]: 7f4e31827640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e32829640 / prometheus Oct 11 13:46:38 hostname bash[27409]: 7f4e3402c640 / prometheus Oct 11 13:46:38 hostname bash[27409]: 7f4e36030640 / prometheus Oct 11 13:46:38 hostname bash[27409]: 7f4e36831640 / prometheus Oct 11 13:46:38 hostname bash[27409]: 7f4e38034640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e3a9b8640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e3e1bf640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e42b08640 / safe_timer Oct 11 13:46:38 hostname bash[27409]: 7f4e43b0a640 / ms_dispatch Oct 11 13:46:38 hostname bash[27409]: 7f4e453cd640 / io_context_pool Oct 11 13:46:38 hostname bash[27409]: 7f4e45c0e640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e47c12640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e4a417640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e4ac18640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e4c41b640 / safe_timer Oct 11 13:46:38 hostname bash[27409]: 7f4e4d41d640 / ms_dispatch Oct 11 13:46:38 hostname bash[27409]: 7f4e4ec20640 / io_context_pool Oct 11 13:46:38 hostname bash[27409]: 7f4e51465640 / prometheus Oct 11 13:46:38 hostname bash[27409]: 7f4e5552d640 / pg_autoscaler Oct 11 13:46:38 hostname bash[27409]: 7f4e5652f640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e58d34640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e5a537640 / Oct 11 13:46:38 hostname bash[27409]: 7f4e5bd7a640 / devicehealth Oct 11 13:46:38 hostname bash[27409]: 7f4e5fd82640 / crash Oct 11 13:46:38 hostname bash[27409]: 7f4e60d84640 / cephadm Oct 11 13:46:38 hostname bash[27409]: 7f4e62587640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e64e0c640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e65e0e640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e66e10640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e67611640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e68613640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e68e14640 / mgr-fin Oct 11 13:46:38 hostname bash[27409]: 7f4e69e96640 / balancer Oct 11 13:46:38 hostname bash[27409]: 7f4e6dede640 / cmdfin Oct 11 13:46:38 hostname bash[27409]: 7f4e6f6e1640 / ms_dispatch Oct 11 13:46:38 hostname bash[27409]: 7f51864f4640 / safe_timer Oct 11 13:46:38 hostname bash[27409]: 7f518acfd640 / ms_dispatch Oct 11 13:46:38 hostname bash[27409]: 7f518e504640 / msgr-worker-1 Oct 11 13:46:38 hostname bash[27409]: 7f518ed05640 / msgr-worker-0 Oct 11 13:46:38 hostname bash[27409]: max_recent 10000 Oct 11 13:46:38 hostname bash[27409]: max_new 1000 Oct 11 13:46:38 hostname bash[27409]: log_file /var/lib/ceph/crash/2024-10-11T11:46:38.415833Z_b3978f24-6697-44f5-80dc-4915b5ec144d/log Oct 11 13:46:38 hostname bash[27409]: --- end dump of recent events --- ^C

2 1

Questions about inactive pgs. thank you
by 苏察哈尔灿 12 Oct '24

12 Oct '24

I am deleting two OSD nodes on a node, and ceph health detail indicates that there are 15 pgs inactive, as follows: [WRN] PG_AVAILABILITY: Reduced data availability: 15 pgs inactive     pg 37.4 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [NONE,40,38,88,0,22,62,32,NONE,21]     pg 37.a is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [38,103,NONE,57,20,NONE,63,51,8,18]     pg 37.c is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [58,7,18,38,52,101,80,NONE,62,NONE]     pg 37.62 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [67,101,NONE,57,62,NONE,7,26,14,20]     pg 37.97 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [35,19,NONE,13,58,17,20,6,25,NONE]     pg 37.9d is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [103,22,NONE,63,NONE,34,45,56,90,14]     pg 37.b4 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [61,NONE,NONE,46,86,60,9,66,21,35]     pg 37.136 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [10,NONE,24,78,48,89,NONE,56,8,51]     pg 37.13a is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [71,33,NONE,14,32,27,NONE,5,25,57]     pg 37.149 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [6,69,65,84,27,25,NONE,0,14,NONE]     pg 37.15c is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [103,66,6,23,NONE,NONE,61,45,8,42]     pg 37.16d is stuck inactive for 113m, current state recovering+undersized+remapped+peered, last acting [79,68,26,66,58,NONE,52,8,NONE,60]     pg 37.17b is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [80,47,27,14,24,NONE,67,35,NONE,5]     pg 37.1d0 is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [0,76,68,57,NONE,10,48,85,101,NONE]     pg 37.1ea is stuck inactive for 113m, current state recovery_wait+undersized+degraded+remapped+peered, last acting [28,24,52,20,10,43,56,NONE,NONE,80] My pool is used using 8+2 erasure-code configuration: ceph osd erasure-code-profile get my-ec-profile02 crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=4 m=2 plugin=jerasure technique=reed_sol_van w=8 In theory, it should be possible to access the corresponding pool, but in fact, the client cannot access normally, or it will be stuck in the middle of the access. Why? Then, when using the command "rados -p cephfs-backup.data-ec ls" to display the pool object, it also stuck 10000000e74.00000cd8 1000000080f.00004436 10000000e3d.000012e3 10000000e50.000018a2 1000000081b.00005b22 100000003d3.0000777a 100000003d7.00003c64 Cursor motionless thank you!

2 1

The ceph monitor crashes every few days
by 李明 10 Oct '24

10 Oct '24

Hello, ceph version is 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) and the rbd info command is also slow, some times it needs 6 seconds. rbd snap create command takes 17 seconds. There is another cluster with the same configuration it only takes less than 1 second. crash log -2> 2024-10-09 09:52:50.782 7f30ddcb4700 5 prioritycache tune_memory target: 2147483648 mapped: 1286012928 unmapped: 1695842304 heap: 2981855232 old mem: 1020054731 new mem: 1020054731 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/mon/OSDMonitor.cc: 4112: ceph_abort_msg("abort() called") ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f30ea6c2a66] 2: (OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned long)+0x89f) [0x563b0175382f] 3: (OSDMonitor::send_incremental(unsigned int, MonSession*, bool, boost::intrusive_ptr<MonOpRequest>)+0x11d) [0x563b01753e6d] 4: (OSDMonitor::check_osdmap_sub(Subscription*)+0xca) [0x563b0175a53a] 5: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xbbe) [0x563b0162c9ee] 6: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x525) [0x563b0164b0a5] 7: (Monitor::_ms_dispatch(Message*)+0xcdb) [0x563b0164cd9b] 8: (Monitor::ms_dispatch(Message*)+0x26) [0x563b0167ad96] 9: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x563b01677776] 10: (DispatchQueue::entry()+0x1699) [0x7f30ea8e7699] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f30ea99564d] 12: (()+0x7ea5) [0x7f30e74acea5] 13: (clone()+0x6d) [0x7f30e636fb0d] 0> 2024-10-09 09:52:55.260 7f30db4af700 -1 *** Caught signal (Aborted) ** in thread 7f30db4af700 thread_name:ms_dispatch ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (()+0xf630) [0x7f30e74b4630] 2: (gsignal()+0x37) [0x7f30e62a7387] 3: (abort()+0x148) [0x7f30e62a8a78] 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x7f30ea6c2b2e] 5: (OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned long)+0x89f) [0x563b0175382f] 6: (OSDMonitor::send_incremental(unsigned int, MonSession*, bool, boost::intrusive_ptr<MonOpRequest>)+0x11d) [0x563b01753e6d] 7: (OSDMonitor::check_osdmap_sub(Subscription*)+0xca) [0x563b0175a53a] 8: (Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xbbe) [0x563b0162c9ee] 9: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x525) [0x563b0164b0a5] 10: (Monitor::_ms_dispatch(Message*)+0xcdb) [0x563b0164cd9b] 11: (Monitor::ms_dispatch(Message*)+0x26) [0x563b0167ad96] 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x563b01677776] 13: (DispatchQueue::entry()+0x1699) [0x7f30ea8e7699] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f30ea99564d] 15: (()+0x7ea5) [0x7f30e74acea5] 16: (clone()+0x6d) [0x7f30e636fb0d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this ceph -s cluster: id: 6e36443f-326b-4344-b013-c03a4b7bb8d7 health: HEALTH_WARN noout flag(s) set 314 pgs not deep-scrubbed in time 660 pgs not scrubbed in time 23 daemons have recently crashed services: mon: 3 daemons, quorum rbd02,rbd01,rbd03 (age 81m) mgr: rbd03(active, since 9d), standbys: rbd01, rbd02 osd: 360 osds: 360 up (since 9d), 360 in (since 12M) flags noout data: pools: 1 pools, 8192 pgs objects: 109.45M objects, 274 TiB usage: 488 TiB used, 3.4 PiB / 3.8 PiB avail pgs: 8116 active+clean 76 active+clean+scrubbing+deep io: client: 61 MiB/s rd, 34 MiB/s wr, 74.51k op/s rd, 2.06k op/s wr

3 2

Erasure coding scheme 2+4 = good idea?
by Andre Tann 10 Oct '24

10 Oct '24

Hi all, I'd like your thoughts and comments on this idea: Setup: - two fault domains = DCs - connected with 100 GBit, < 1 ms - 80 NVME SSDs on each side Goal: One fault domain can be lost, and then there's still have some redundancy. Option 1: Replicated pool with size = 4. This gives me two copies on each side, thus meets the goal. But the efficiency is only 25%. Option 2: Erasure coded pool with 2 + 4 scheme. This gives me 3 chunks on each side. If I lose one side, I still have 3 chunks left, where I only need 2. Thus the goal is also met. Efficiency is 33%. Even though I did a lot of googling, I couldn't find anything about a similar setup. In all profiles, there is k <= m. What do you think about 2+4, is it a good idea or a bad one, or do I miss something and it doesn't work at all? In particular: is it possible to recover two data chunks out of 2 coding chunks? As I read the documentation, this should be no problem, just want to confirm. -- Andre Tann

4 5

About 100g network card for ceph
by Phong Tran Thanh 10 Oct '24

10 Oct '24

Hi ceph users I have a 100G network card with dual ports for a Ceph node with NVMe disks. Should I bond them or not? Should I bond 200G for both the public and cluster networks, or separate it: one port for the public network and one for the cluster? Thank ceph users -- Email: tranphong079(a)gmail.com Skype: tranphong079

3 2

Packets Drops in bond interface
by Alfredo Rezinovsky 10 Oct '24

10 Oct '24

I'm having lots of those warning in all nodes: CephNodeNetworkPacketDrops Node ceph-06 experiences packet drop > 0.5% or > 10 packets/s on interface bond0. bonds are 802.3ad with Mikrotik switches. Physical interfaces are OK, without drops. They are Intel e1000 dual PCIe NICs Running with kernel 6.8.0-45-generic ubuntu 22.04 -- Alfrenovsky

1 0

Forced upgrade OSD from Luminous to Pacific
by Alex Rydzewski 10 Oct '24

10 Oct '24

Hello, dear community! I kindly ask for your help in resolving my issue. I have a server with a single-node CEPH setup with 5 OSDs. This server has been powered off for about two years, and when I needed the data from it, I found that the SSD where the system was installed had died. I tried to recover the cluster. First, assuming the old CEPH is there I installed Debian 10 with CEPH 12.2.11, mounted the OSDs to /var/lib/ceph/osd/ceph-xx and assembled the monitor, as described here https://forum.proxmox.com/threads/recover-ceph-from-osds-only.113699/. However, the monitor wouldn't start, giving an error I don't remember. Then I made a series of mistakes, upgrading the system and CEPH first to Nautilus and then to Pacific. Eventually, I managed to start the monitor, but a compatibility issue with the OSDs remains. When the OSDs start, I see the message: /check_osdmap_features require_osd_release unknown -> luminous /At the same time, the monitor log shows: /disallowing boot of octopus+ OSD osd.xx. /After starting, the OSD remains in the state: /tick checking mon for new map/ Then I enabledmsgrv2 protocol and tried enabling RocksDB sharding for the OSD, as described here https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#b…, but it didn’t help. Attempts to start the OSD with lower versions of CEPH event with Octopus end with the error: /2024-10-08 10:45:38.402975 7fba61b34ec0 -1 bluefs _replay 0x0: stop: unrecognized op 12 2024-10-08 10:45:38.402992 7fba61b34ec0 -1 bluefs mount failed to replay log: (5) Input/output error/ So, currently, I have CEPH 16.2.15, and the OSD is in the following state: /"/var/lib/ceph/osd/ceph-1/block": { "osd_uuid": "2bb56721-28c7-45cc-9344-6cc5c699a642", "size": 4000681103360, "btime": "2018-06-02 13:16:57.042205", "description": "main", "bfm_blocks": "61045632", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "65536", "bfm_size": "4000681099264", "bluefs": "1", "ceph_fsid": "96b6ff1d-25bf-403f-be3d-78c2fb0ff747", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "ready": "ready", "require_osd_release": "12", "whoami": "1" }/ with modified RocksDB to enable sharding. Suggest me, please, Is there a way to upgrade such OSDs so they can run with this version of Ceph? If you need more information here, let me know and I will provide whatever is needed. -- Alexander Rydzewski

3 10

rgw connection resets
by Nathan Gleason 10 Oct '24

10 Oct '24

Hello, I’ve noticed what I think may be strange behavior from some rgw transfers. It only seems to be with larger multipart mp4 files. Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”. The client then reconnects and the transfer seems to be ok. This doesn’t appear to happen just downloading a regular file. This is happening to both my production and test cluster. I’m not seeing errors anywhere else, the network looks good and the clusters are healthy. I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring. We run haproxy in front of the rgw. For troubleshooting purposes we have a basic haproxy config to just one rgw. We have tested with multiple browsers. Ceph 17.2.5 - cluster built and managed with cephadm Ubuntu 20.04.6 LTS Log: 2024-01-03T17:40:23.916+0000 7f646033d700 1 ====== starting new request req=0x7f640ca95730 ===== 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n athan:tshoot/testfile.mp4 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s getting op 0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch _data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser </ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.c om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init op 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p refetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n ame=, subuser=, perm_mask=15, is_admin=0) mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask =15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj executing 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode: 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1 2024-01-03T17:40:24.068+0000 7f642a2d1700 4 write_data failed: Connection reset by peer 2024-01-03T17:40:24.068+0000 7f642a2d1700 0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj completing 2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data =0 2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry) 2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj http status=200 2024-01-03T17:40:24.156+0000 7f642eada700 1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ====== 2024-01-03T17:40:24.156+0000 7f642eada700 1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s 2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/* 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0- 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https 2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000 2024-01-03T17:40:24.160+0000 7f6503483700 1 ====== starting new request req=0x7f640ca95730 ===== Thanks, Nathan

2 1

Can't pull container images from quay-quay-quay.apps.os.sepia.ceph.com
by John Robert Mendoza 09 Oct '24

09 Oct '24

Hello, I am trying to set up a 3-node Ceph cluster using cephadm and bootstrapping fails because podman can't pull ceph container images from quay-quay-quay.apps.os.sepia.ceph.com. Excerpt of the error message I encountered: ... /usr/bin/podman: stderr time="2024-10-09T21:57:22+08:00" level=warning msg="Failed, retrying in 1s ... (3/3). Error: parsing image configuration: Get \"https://quay-quay-quay.apps.os.sepia.ceph.com/_storage_proxy/ZXlKaGJHY2lPaU…": dial tcp 172.21.0.91:443: i/o timeout" ... There must be a mistake with quay-quay-quay.apps.os.sepia.ceph.com DNS record because it's resolving to a RFC1918 address (172.16/12 prefix). I tried various online DNS tools (including Google's Admin Toolbox) to help isolate the issue and they all return the same IP address (172.21.0.91). Anyone else having this problem?

2 1