- ceph-users - lists.ceph.io

Re: Bucket sync policy
by Yixin Jin 24 Apr '23

24 Apr '23

Re-post my rely to make sure it goes through the mailing list. ----- Forwarded Message ----- From: Yixin Jin <yjin77(a)yahoo.ca>To: Soumya Koduri <skoduri(a)redhat.com>Sent: Monday, April 24, 2023 at 02:37:46 p.m. EDTSubject: Re: [ceph-users] Re: Bucket sync policy An update: After creating and enabling the bucket sync policy, I ran "bucket sync markers" and saw that each shard had the status of "init". The run "bucket sync run" in the end marked the status to be "incremental-sync", which seems to go through full-sync stage. However, the lone object in the source zone wasn't synced over to the destination zone. I actually used gdb to walk through radosgw-admin to run "bucket sync run". It seems not to do anything for full-sync and it printed a log saying "finished iterating over all available prefixes:...", which actually broke off the do-while loop after the call to prefix_handler.revalidate_marker(&list_marker). This call returned false because it couldn't find rules from the sync pipe. I haven't drilled deeper to see why it didn't get rules, whatever it means. Nevertheless, the workaround with "bucket sync run" doesn't seem to work, at least not with Quincy. Regards,Yixin On Monday, April 24, 2023 at 12:37:24 p.m. EDT, Soumya Koduri <skoduri(a)redhat.com> wrote: On 4/24/23 21:52, Yixin Jin wrote: > Hello ceph gurus, > > We are trying bucket-specific sync policy feature with Quincy release and we encounter something strange. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-specific sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1. > > > > Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. Why wasn’t there a full sync and how can one force a full sync? yes this is known_issue yet to be addressed with bucket level sync policy ( - https://tracker.ceph.com/issues/57489 ). The interim workaround to sync existing objects is to either * create new objects (or) * execute "bucket sync run" after creating/enabling the bucket policy. Please note that this issue is specific to only bucket policy but doesn't exist for sync-policy set at zonegroup level. Thanks, Soumya _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

v16.2.12 Pacific (hot-fix) released
by Yuri Weinstein 24 Apr '23

24 Apr '23

We're happy to announce the 12th hot-fix release in the Pacific series. https://ceph.io/en/news/blog/2023/v16-2-12-pacific-released/ Notable Changes --------------- This is a hotfix release that resolves several performance flaws in ceph-volume, particularly during osd activation (https://tracker.ceph.com/issues/57627) Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.12.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 5a2d516ce4b134bfafc80c4274532ac0d56fc1e2

4 4

Bucket sync policy
by yjin77＠yahoo.ca 24 Apr '23

24 Apr '23

Hello, ceph gurus, We are trying multisite sync policy feature with Quincy release and we encounter something strange, which we cannot solve even after combing through the internet for clues. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-level sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1. Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. So, why wasn’t there a full sync and how can one force a full sync? I also tried “sync error list” and they are all empty. I also tried to apply the fix in https://tracker.ceph.com/issues/57853, although I am not sure if it is relevant. The fix didn’t change the behavior that we observed. Thanks, Yixin

1 0

Disks are filling up
by Omar Siam 24 Apr '23

24 Apr '23

Hi list, we created a cluster for using cephfs with a kubernetes cluster. Since a few weeks now the cluster keeps filling up at an alarming rate (100 GB per day). This is while the most relevant pg is deep scrubbing and was interupted a few times. We use about 150G (du using the mounted filesystem) on the cephfs filesystem and try not to use snapshots (.snap directories "exist" but are empty). We do not understand why the pgs get bigger and bigger while cephfs stays about the same size (overwrites on files certainly happen). I suspect some snapshots mechanism. Any ideas how to debug this to stop it? Maybe we should try to speed up the deep scrubbing somehow? ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) cluster: id: ece0290c-cd32-11ec-a0e2-005056a9dd02 health: HEALTH_WARN 1 MDSs report slow metadata IOs 3 nearfull osd(s) 13 pool(s) nearfull services: mon: 3 daemons, quorum acdh-gluster-hdd3,acdh-gluster-hdd1,acdh-gluster-hdd2 (age 3d) mgr: acdh-gluster-hdd3.kzsplh(active, since 5d), standbys: acdh-gluster-hdd2.kiotbg, acdh-gluster-hdd1.ywgyfx mds: 1/1 daemons up, 2 standby osd: 3 osds: 3 up (since 4d), 3 in (since 7w) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 13 pools, 292 pgs objects: 167.25M objects, 1.2 TiB usage: 3.3 TiB used, 1.2 TiB / 4.5 TiB avail pgs: 290 active+clean 1 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 58 MiB/s rd, 3.6 MiB/s wr, 51 op/s rd, 148 op/s wr rancher-ceph-fs - 227 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-mds.acdh-gluster-hdd1.pqydya Reqs: 68 /s 793k 792k 102k 210k POOL TYPE USED AVAIL rancherFsPoolMetadata metadata 160G 329G rancherFsPoolDefaultData data 2268k 329G rancherFsPoolMainData data 2584G 658G STANDBY MDS ceph-mds.acdh-gluster-hdd2.zfleqe ceph-mds.acdh-gluster-hdd3.etaobl MDS version: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) (rancherFsPoolMainData is a 2+1 erasure encoded pool) --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 4.5 TiB 1.2 TiB 3.3 TiB 3.3 TiB 73.46 TOTAL 4.5 TiB 1.2 TiB 3.3 TiB 3.3 TiB 73.46 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 331 GiB rancher-rbd-erasure 2 32 8.4 GiB 2.16k 13 GiB 1.25 661 GiB rancher-rbd-meta 3 32 55 B 11 36 KiB 0 331 GiB rancherFsPoolMetadata 4 32 53 GiB 5.18M 160 GiB 13.88 331 GiB rancherFsPoolDefaultData 5 1 29 KiB 80.00M 2.2 MiB 0 331 GiB rancherFsPoolMainData 6 1 1.7 TiB 82.08M 2.5 TiB 72.23 661 GiB .rgw.root 7 32 1.3 KiB 4 48 KiB 0 331 GiB default.rgw.log 8 32 3.6 KiB 209 408 KiB 0 331 GiB default.rgw.control 9 32 0 B 8 0 B 0 331 GiB default.rgw.meta 10 32 3.8 KiB 11 124 KiB 0 331 GiB default.rgw.buckets.index 11 32 2.4 MiB 33 7.2 MiB 0 331 GiB default.rgw.buckets.non-ec 12 32 0 B 0 0 B 0 331 GiB default.rgw.buckets.data 14 1 55 GiB 16.57k 83 GiB 7.70 661 GiB HEALTH_WARN 1 MDSs report slow metadata IOs; 3 nearfull osd(s); 13 pool(s) nearfull [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.ceph-mds.acdh-gluster-hdd1.pqydya(mds.0): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 306 secs [WRN] OSD_NEARFULL: 3 nearfull osd(s) osd.0 is near full osd.2 is near full osd.3 is near full [WRN] POOL_NEARFULL: 13 pool(s) nearfull pool 'device_health_metrics' is nearfull pool 'rancher-rbd-erasure' is nearfull pool 'rancher-rbd-meta' is nearfull pool 'rancherFsPoolMetadata' is nearfull pool 'rancherFsPoolDefaultData' is nearfull pool 'rancherFsPoolMainData' is nearfull pool '.rgw.root' is nearfull pool 'default.rgw.log' is nearfull pool 'default.rgw.control' is nearfull pool 'default.rgw.meta' is nearfull pool 'default.rgw.buckets.index' is nearfull pool 'default.rgw.buckets.non-ec' is nearfull pool 'default.rgw.buckets.data' is nearfull (near full is set to 0.66) -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

1 0

Troubleshooting cephadm OSDs aborting start
by André Gemünd 24 Apr '23

24 Apr '23

Dear Ceph-users, we have trouble with a Ceph cluster after a full shutdown. A couple of OSDs don't start anymore, exiting with SIGABRT very quickly. With debug logs and lots of work (I find cephadm clusters hard to debug btw) we received the following stack trace: debug -16> 2023-04-14T11:52:17.617+0000 7f10ab4d2700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/osd/PGLog.h: In function 'void PGLog::IndexedLog::add(const pg_log_entry_t&, bool)' thread 7f10ab4d2700 time 2023-04-14T11:52:17.614095+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/osd/PGLog.h: 607: FAILED ceph_assert(head.version == 0 || e.version.version > head.version) ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x55b2dafc7b7e] 2: /usr/bin/ceph-osd(+0x56ad98) [0x55b2dafc7d98] 3: (bool PGLog::append_log_entries_update_missing<pg_missing_set<true> >(hobject_t const&, std::__cxx11::list<pg_log_entry_t, mempool::pool_allocator<(mempool::pool_index_t)22, pg_log_entry_t> > const&, bool, PGLog::IndexedLog*, pg_missing_set<true>&, PGLog::LogEntryHandler*, DoutPrefixProvider const*)+0xc19) [0x55b2db1bb6b9] 4: (PGLog::merge_log(pg_info_t&, pg_log_t&&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0xee2) [0x55b2db1adf22] 5: (PeeringState::merge_log(ceph::os::Transaction&, pg_info_t&, pg_log_t&&, pg_shard_t)+0x75) [0x55b2db33c165] 6: (PeeringState::Stray::react(MLogRec const&)+0xcc) [0x55b2db37adec] 7: (boost::statechart::simple_state<PeeringState::Stray, PeeringState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd5) [0x55b2db3a6e65] 8: (boost::statechart::state_machine<PeeringState::PeeringMachine, PeeringState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x5b) [0x55b2db18ef6b] 9: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PeeringCtx&)+0x2d1) [0x55b2db1839e1] 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x29c) [0x55b2db0fde5c] 11: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x56) [0x55b2db32d0e6] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x55b2db0efd48] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x55b2db7615b4] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b2db764254] 15: /lib64/libpthread.so.0(+0x817f) [0x7f10cef1117f] 16: clone() debug -15> 2023-04-14T11:52:17.618+0000 7f10b64e8700 3 osd.70 72507 handle_osd_map epochs [72507,72507], i have 72507, src has [68212,72507] debug -14> 2023-04-14T11:52:17.619+0000 7f10b64e8700 3 osd.70 72507 handle_osd_map epochs [72507,72507], i have 72507, src has [68212,72507] debug -13> 2023-04-14T11:52:17.619+0000 7f10ac4d4700 5 osd.70 pg_epoch: 72507 pg[18.7( v 64162'106 (0'0,64162'106] local-lis/les=72506/72507 n=14 ec=17104/17104 lis/c=72506/72480 les/c/f=72507/72481/0 sis=72506 pruub=9.160680771s) [70,86,41] r=0 lpr=72506 pi=[72480,72506)/1 crt=64162'106 lcod 0'0 mlcod 0'0 active+wait pruub 12.822580338s@ mbc={}] exit Started/Primary/Active/Activating 0.011269 7 0.000114 # ceph versions { "mon": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "osd": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 92 }, "mds": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "rgw": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "overall": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 103 } } Another thing is that things like `ceph -s`, `ceph osd tree`, `rbd ls`, etc. work, but `ceph orch ps` (or generally any orch commands) simply hang forever, seemingly in a futex waiting on a socket to the mons. If anyone has any ideas how we could get those OSDs back online, I'd be very grateful for any hints. I'm also on slack. Greetings -- André Gemünd, Leiter IT / Head of IT Fraunhofer-Institute for Algorithms and Scientific Computing andre.gemuend(a)scai.fraunhofer.de Tel: +49 2241 14-4199 /C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend

2 2

How to replace an HDD in a OSD with shared SSD for DB/WAL
by Tao LIU 22 Apr '23

22 Apr '23

HI, I build a Ceph Cluster with cephadm. Every cehp node has 4 OSDs. These 4 OSD were build with 4 HDD (block) and 1 SDD (DB). At present , one HDD is broken, and I am trying to replace the HDD,and build the OSD with the new HDD and the free space of the SDD. I did the follows: #ceph osd stop osd.23 #ceph osd out osd.23 #ceph osd crush remove osd.23 #ceph osd rm osd.23 #ceph orch daemon rm osd.23 --force #lvremove /dev/ceph-ae21e618-601e-4273-9185-99180edb8453/osd-block-96eda371-1a3f-4139-9123-24ec1ba362c4 #wipefs -af /dev/sda #lvremove /dev/ceph-e50203a6-8b8e-480f-965c-790e21515395/osd-db-70f7a032-cf2c-4964-b979-2b90f43f2216 #ceph orch daemon add osd compute11:data_devices=/dev/sda,db_devices=/dev/sdc,osds_per_device=1 The OSD can be built, but is always down. Is there anyting that I missed during the building? Thank you very much! Regards, LIUTao

3 2

radosgw-admin bucket rm fails
by James Turner 22 Apr '23

22 Apr '23

I've got buckets that are visible to radosgw-admin bucket list but not to an S3 client and the delete fails.. e.g.: radosgw-admin bucket rm --bucket=appsshared1 2023-04-22T19:43:55.705-0400 7f9a87616680 0 ERROR: cls_bucket_list_unordered error in rgw_rados_operate (bucket list op), r=-2 2023-04-22T19:43:55.705-0400 7f9a87616680 0 ERROR: int RGWRados::Bucket::List::list_objects_unordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield) cls_bucket_list_unordered returned -2 for :appsshared1[07c644e6-08ba-4c2a-aa43-46c47fcaa3d2.2815988.1]) Similarly, trying to delete the owning user fails: radosgw-admin user rm --uid=apps_shared_user --purge-data could not remove user: unable to remove user, unable to delete user data 2023-04-22T19:54:34.266-0400 7f84c598c680 0 ERROR: cls_bucket_list_unordered error in rgw_rados_operate (bucket list op), r=-2 2023-04-22T19:54:34.266-0400 7f84c598c680 0 ERROR: int RGWRados::Bucket::List::list_objects_unordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield) cls_bucket_list_unordered returned -2 for :appsshared1[07c644e6-08ba-4c2a-aa43-46c47fcaa3d2.2815988.1]) As does trying to fix the bucket: radosgw-admin bucket check --bucket appsshared1 --fix 2023-04-22T19:56:17.510-0400 7f981eddd680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for :appsshared1[07c644e6-08ba-4c2a-aa43-46c47fcaa3d2.2815988.1]) failed The release is quincy - Any ideas on how to approach the problem? -- Jim Turner

1 0

advise on adding RGW and NFS/iSCSI on proxmox
by MartijnF 22 Apr '23

22 Apr '23

Hello, i'm maintaining a 3 small infrastructures in NL. 7 primary, 4 secondary and 3 testing proxmox nodes (hyperconverged). Till now we only use the embedded RBD facilities of proxmox using ceph 17.2.5 To facilitate S3/swift and persistent storage to our internal k8s cluster i am looking around for best-practices like: - should i install it on-top of proxmox-hosts or create a few ha-VM's with public and private lan-segment to avoid upgrade errors. - can i use the new ceph-dashboard to create and install mentioned services needed? Should i virtualize a proxmox or other OS for this? I can imagine it is not their primary focus. A lot of information can be found when using google-foo but it is hard to find answers reflecting my setup. Thanks in advance for any pointers! Martijn

1 0

[sync policy] multisite bucket full sync
by yjin77＠yahoo.ca 21 Apr '23

21 Apr '23

Hi folks, We are trying multisite sync policy feature with Quincy release and we encounter something strange. Maybe our understanding of sync policy is incorrect. I hope the community could help us uncover the mystery. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm "world", a single zonegroup "zg" and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with status “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy status is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-level sync policy with status “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind for data sync, which is expected since that only object exists in z0 but not in z1. Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. My questions are: 1. With bucket-specific sync policy, why wasn’t there a full sync and how can one force a full sync? 2. Does the sync policy only work datalogs, without which no action it may take? I wonder if it would explain why it didn't sync even though it knows it is behind? BTW, I also tried “sync error list” and they are all empty. I also tried to apply the fix in https://tracker.ceph.com/issues/57853, although I am not sure if it is relevant. The fix didn’t change the behavior that we observed. Thanks in advance, Yixin

1 0

Re: pg_autoscaler using uncompressed bytes as pool current total_bytes triggering false POOL_TARGET_SIZE_BYTES_OVERCOMMITTED warnings?
by Christian Rohmann 21 Apr '23

21 Apr '23

Hey ceph-users, may I ask (nag) again about this issue? I am wondering if anybody can confirm my observations? I raised a bug https://tracker.ceph.com/issues/54136, but apart from the assignment to a dev a while ago here was not response yet. Maybe I am just holding it wrong, please someone enlighten me. Thank you and with kind regards Christian On 02/02/2022 20:10, Christian Rohmann wrote: > > Hey ceph-users, > > > I am debugging a mgr pg_autoscaler WARN which states a > target_size_bytes on a pool would overcommit the available storage. > There is only one pool with value for target_size_bytes (=5T) defined > and that apparently would consume more than the available storage: > > --- cut --- > # ceph health detail > HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes > [WRN] POOL_TARGET_SIZE_BYTES_OVERCOMMITTED: 1 subtrees have > overcommitted pool target_size_bytes > Pools ['backups', 'images', 'device_health_metrics', '.rgw.root', > 'redacted.rgw.control', 'redacted.rgw.meta', 'redacted.rgw.log', > 'redacted.rgw.otp', 'redacted.rgw.buckets.index', > 'redacted.rgw.buckets.data', 'redacted.rgw.buckets.non-ec'] overcommit > available storage by 1.011x due to target_size_bytes 15.0T on pools > ['redacted.rgw.buckets.data']. > --- cut --- > > > But then looking at the actual usage it seems strange that 15T (5T * 3 > replicas) should not fit onto the remaining 122 TiB AVAIL: > > > --- cut --- > # ceph df detail > --- RAW STORAGE --- > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 293 TiB 122 TiB 171 TiB 171 TiB 58.44 > TOTAL 293 TiB 122 TiB 171 TiB 171 TiB 58.44 > > --- POOLS --- > POOL ID PGS STORED (DATA) (OMAP) > OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS > QUOTA BYTES DIRTY USED COMPR UNDER COMPR > backups 1 1024 92 TiB 92 TiB 3.8 MiB > 28.11M 156 TiB 156 TiB 11 MiB 64.77 28 TiB N/A > N/A N/A 39 TiB 123 TiB > images 2 64 1.7 TiB 1.7 TiB 249 KiB > 471.72k 5.2 TiB 5.2 TiB 748 KiB 5.81 28 TiB N/A > N/A N/A 0 B 0 B > device_health_metrics 19 1 82 MiB 0 B 82 > MiB 43 245 MiB 0 B 245 MiB 0 28 TiB > N/A N/A N/A 0 B 0 B > .rgw.root 21 32 23 KiB 23 KiB 0 B > 25 4.1 MiB 4.1 MiB 0 B 0 28 TiB N/A > N/A N/A 0 B 0 B > redacted.rgw.control 22 32 0 B 0 B 0 B > 8 0 B 0 B 0 B 0 28 TiB N/A > N/A N/A 0 B 0 B > redacted.rgw.meta 23 32 1.7 MiB 394 KiB 1.3 > MiB 1.38k 237 MiB 233 MiB 3.9 MiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.log 24 32 53 MiB 500 KiB 53 > MiB 7.60k 204 MiB 47 MiB 158 MiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.otp 25 32 5.2 KiB 0 B 5.2 > KiB 0 16 KiB 0 B 16 KiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.buckets.index 26 32 1.2 GiB 0 B 1.2 > GiB 7.46k 3.5 GiB 0 B 3.5 GiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.buckets.data 27 128 3.1 TiB 3.1 TiB 0 B > 3.53M 9.5 TiB 9.5 TiB 0 B 10.11 28 TiB N/A > N/A N/A 0 B 0 B > redacted.rgw.buckets.non-ec 28 32 0 B 0 B 0 B > 0 0 B 0 B 0 B 0 28 TiB N/A > N/A N/A 0 B 0 B > --- cut --- > > > I then looked at how those values are determined at > https://github.com/ceph/ceph/blob/9f723519257eca039126a20aa6a2a7d2dbfb5dba/…. > Apparently "total_bytes" are compared with the capacity of the > root_map. I added a debug line and found that the total in my cluster > was already at: > > total=325511007759696 > > so in excess of 300 TiB - Looking at "ceph df" again this usage seems > strange. > > > > Looking at how this total is calculated at > https://github.com/ceph/ceph/blob/9f723519257eca039126a20aa6a2a7d2dbfb5dba/…, > you see that the larger value (max) of "actual_raw_used" vs. > "target_bytes*raw_used_rate" is determined and then summed up. > > > I dumped the values for all pools my cluster with yet another line of > debug code: > > ---cut --- > pool_id 1 - actual_raw_used=303160109187420.0, target_bytes=0 > raw_used_rate=3.0 > pool_id 2 - actual_raw_used=5714098884702.0, target_bytes=0 > raw_used_rate=3.0 > pool_id 19 - actual_raw_used=256550760.0, target_bytes=0 raw_used_rate=3.0 > pool_id 21 - actual_raw_used=71433.0, target_bytes=0 raw_used_rate=3.0 > pool_id 22 - actual_raw_used=0.0, target_bytes=0 raw_used_rate=3.0 > pool_id 23 - actual_raw_used=5262798.0, target_bytes=0 raw_used_rate=3.0 > pool_id 24 - actual_raw_used=162299940.0, target_bytes=0 raw_used_rate=3.0 > pool_id 25 - actual_raw_used=16083.0, target_bytes=0 raw_used_rate=3.0 > pool_id 26 - actual_raw_used=3728679936.0, target_bytes=0 > raw_used_rate=3.0 > pool_id 27 - actual_raw_used=10035209699328.0, > target_bytes=5497558138880 raw_used_rate=3.0 > pool_id 28 - actual_raw_used=0.0, target_bytes=0 raw_used_rate=3.0 > --- cut --- > > > All values but those of pool_id 1 (backups) make sense. For backups > it's just reporting a MUCH larger actual_raw_used value than what is > shown via ceph df. > The only difference of that pool compared to the others is the enabled > compression: > > > --- cut --- > # ceph osd pool get backups compression_mode > compression_mode: aggressive > --- cut --- > > > Apparently there already was a similar issue > (https://tracker.ceph.com/issues/41567) with a resulting commit > (https://github.com/ceph/ceph/commit/dd6e752826bc762095be4d276e3c1b8d31293eb0) > > changing which from "bytes_used" to the "stored" field for > "pool_logical_used". > > But how does that take compressed (away) data into account? Does > "bytes_used" count all the "stored" bytes, summing up all uncompressed > bytes for pools with compression? > This surely must be a bug then, as those bytes are not really > "actual_raw_used". > > > > I was about to raise a bug, but I wanted to ask here on the ML first > if I misunderstood the mechanisms at play here. > Thanks and with kind regards, > > > Christian >

1 0