Hi The issue is open: https://tracker.ceph.com/issues/64244 If you could take a look or let me know what are the next steps I would be super grateful. In the meantime I will try to increase the read throughput. Thanks Marc On 1/26/24 15:23, Matt Benjamin wrote: > Hi Marc, > > 1. if you can, yes, create a tracker issue on tracker.ceph.com? > 2. you might be able to get more throughput with (some number) of > additional threads; the first thing I would try is prioritization (nice) > > regards, > > Matt > > > On Fri, Jan 26, 2024 at 6:08 AM Marc Singer&lt;marc(a)singer.services&gt; wrote: > >> Hi Matt >> >> Thanks for your answer. >> >> Should I open a bug report then? >> >> How would I be able to read more from it? Have multiple threads access >> it and read from it simultaneously? >> >> Marc >> >> On 1/25/24 20:25, Matt Benjamin wrote: >>> Hi Marc, >>> >>> No, the only thing you need to do with the Unix socket is to keep >>> reading from it. So it probably is getting backlogged. And while you >>> could arrange things to make that less likely, you likely can't make >>> it impossible, so there's a bug here. >>> >>> Matt >>> >>> On Thu, Jan 25, 2024 at 10:52 AM Marc Singer&lt;marc(a)singer.services&gt; >> wrote: >>> Hi >>> >>> I am using a unix socket client to connect with it and read the data >>> from it. >>> Do I need to do anything like signal the socket that this data has >>> been >>> read? Or am I not reading fast enough and data is backing up? >>> >>> What I am also noticing that at some point (probably after something >>> with the ops socket happens), the log level seems to increase for >>> some >>> reason? I did not find anything in the logs yet why this would be >>> the case. >>> >>> *Normal:* >>> >>> 2024-01-25T15:47:58.444+0000 7fe98a5c0b00 1 ====== starting new >>> request >>> req=0x7fe98712c720 ===== >>> 2024-01-25T15:47:58.548+0000 7fe98b700b00 1 ====== req done >>> req=0x7fe98712c720 op status=0 http_status=200 >>> latency=0.104001537s ====== >>> 2024-01-25T15:47:58.548+0000 7fe98b700b00 1 beast: 0x7fe98712c720: >>> redacted - redacted [25/Jan/2024:15:47:58.444 +0000] "PUT >>> /redacted/redacted/chunks/27/27242/27242514_10_4194304 HTTP/1.1" 200 >>> 4194304 - "redacted" - latency=0.104001537s >>> >>> *Close before crashing: >>> * >>> >>> -509> 2024-01-25T14:54:31.588+0000 7f5186648b00 1 ====== starting >>> new request req=0x7f517ffca720 ===== >>> -508> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000s initializing for trans_id = >>> tx0000023a42eb7515dcdc0-0065b27627-823feaa-central >>> -507> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000s getting op 1 >>> -506> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj verifying requester >>> -505> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj normalizing buckets >>> and tenants >>> -504> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj init permissions >>> -503> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj recalculating target >>> -502> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj reading permissions >>> -501> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj init op >>> -500> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj verifying op mask >>> -499> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj verifying op permissions >>> -498> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req >>> 2568229052387020224 0.000000000ss3:put_obj Searching permissions for >>> identity=rgw::auth::SysReqApplier -> >>> rgw::auth::LocalApplier(acct_user=redacted, acct_name=redacted, >>> subuser=, perm_mask=15, is_admin=0) mask=50 >>> -497> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req >>> 2568229052387020224 0.000000000ss3:put_obj Searching permissions for >>> uid=redacted >>> -496> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req >>> 2568229052387020224 0.000000000ss3:put_obj Found permission: 15 >>> -495> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req >>> 2568229052387020224 0.000000000ss3:put_obj Searching permissions for >>> group=1 mask=50 >>> -494> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req >>> 2568229052387020224 0.000000000ss3:put_obj Permissions for group >>> not found >>> -493> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req >>> 2568229052387020224 0.000000000ss3:put_obj Searching permissions for >>> group=2 mask=50 >>> -492> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req >>> 2568229052387020224 0.000000000ss3:put_obj Permissions for group >>> not found >>> -491> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req >>> 2568229052387020224 0.000000000ss3:put_obj -- Getting permissions >>> done >>> for identity=rgw::auth::SysReqApplier -> >>> rgw::auth::LocalApplier(acct_user=redacted, acct_name=redacted, >>> subuser=, perm_mask=15, is_admin=0), owner=redacted, perm=2 >>> -490> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj verifying op params >>> -489> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj pre-executing >>> -488> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj check rate limiting >>> -487> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req >>> 2568229052387020224 0.000000000ss3:put_obj executing >>> -486> 2024-01-25T14:54:31.624+0000 7f5183898b00 5 req >>> 2568229052387020224 0.036000550ss3:put_obj NOTICE: call to >>> do_aws4_auth_completion >>> -485> 2024-01-25T14:54:31.624+0000 7f5183898b00 5 req >>> 2568229052387020224 0.036000550ss3:put_obj NOTICE: call to >>> do_aws4_auth_completion >>> -484> 2024-01-25T14:54:31.680+0000 7f5185bc8b00 2 req >>> 2568229052387020224 0.092001401ss3:put_obj completing >>> -483> 2024-01-25T14:54:31.680+0000 7f5185bc8b00 2 req >>> 2568229052387020224 0.092001401ss3:put_obj op status=0 >>> -482> 2024-01-25T14:54:31.680+0000 7f5185bc8b00 2 req >>> 2568229052387020224 0.092001401ss3:put_obj http status=200 >>> >>> -481> 2024-01-25T14:54:31.680+0000 7f5185bc8b00 1 ====== req done >>> req=0x7f517ffca720 op status=0 http_status=200 >>> latency=0.092001401s ====== >>> >>> Thanks for your help. >>> >>> Marc Singer >>> >>> On 1/25/24 16:22, Matt Benjamin wrote: >>> > Hi Marc, >>> > >>> > The ops log code is designed to discard data if the socket is >>> > flow-controlled, iirc. Maybe we just need to handle the signal. >>> > >>> > Of course, you should have something consuming data on the >>> socket, but it's >>> > still a problem if radosgw exits unexpectedly. >>> > >>> > Matt >>> > >>> > On Thu, Jan 25, 2024 at 10:08 AM Marc >>> Singer&lt;marc(a)singer.services&gt; wrote: >>> > >>> >> Hi Ceph Users >>> >> >>> >> I am encountering a problem with the RGW Admin Ops Socket. >>> >> >>> >> I am setting up the socket as follows: >>> >> >>> >> rgw_enable_ops_log = true >>> >> rgw_ops_log_socket_path = /tmp/ops/rgw-ops.socket >>> >> rgw_ops_log_data_backlog = 16Mi >>> >> >>> >> Seems like the socket fills up over time and it doesn't seem to >> get >>> >> flushed, at some point the process runs out of file space. >>> >> >>> >> Do I need to configure something or send something for the >>> socket to flush? >>> >> >>> >> See the log here: >>> >> >>> >> 0> 2024-01-25T13:10:13.908+0000 7f247b00eb00 -1 *** Caught >>> signal (File >>> >> size limit exceeded) ** >>> >> in thread 7f247b00eb00 thread_name:ops_log_file >>> >> >>> >> ceph version 18.2.0 >>> (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef >>> >> (stable) >>> >> NOTE: a copy of the executable, or `objdump -rdS >>> <executable>` is >>> >> needed to interpret this. >>> >> >>> >> --- logging levels --- >>> >> 0/ 5 none >>> >> 0/ 1 lockdep >>> >> 0/ 1 context >>> >> 1/ 1 crush >>> >> 1/ 5 mds >>> >> 1/ 5 mds_balancer >>> >> 1/ 5 mds_locker >>> >> 1/ 5 mds_log >>> >> 1/ 5 mds_log_expire >>> >> 1/ 5 mds_migrator >>> >> 0/ 1 buffer >>> >> 0/ 1 timer >>> >> 0/ 1 filer >>> >> 0/ 1 striper >>> >> 0/ 1 objecter >>> >> 0/ 5 rados >>> >> 0/ 5 rbd >>> >> 0/ 5 rbd_mirror >>> >> 0/ 5 rbd_replay >>> >> 0/ 5 rbd_pwl >>> >> 0/ 5 journaler >>> >> 0/ 5 objectcacher >>> >> 0/ 5 immutable_obj_cache >>> >> 0/ 5 client >>> >> 1/ 5 osd >>> >> 0/ 5 optracker >>> >> 0/ 5 objclass >>> >> 1/ 3 filestore >>> >> 1/ 3 journal >>> >> 0/ 0 ms >>> >> 1/ 5 mon >>> >> 0/10 monc >>> >> 1/ 5 paxos >>> >> 0/ 5 tp >>> >> 1/ 5 auth >>> >> 1/ 5 crypto >>> >> 1/ 1 finisher >>> >> 1/ 1 reserver >>> >> 1/ 5 heartbeatmap >>> >> 1/ 5 perfcounter >>> >> 1/ 5 rgw >>> >> 1/ 5 rgw_sync >>> >> 1/ 5 rgw_datacache >>> >> 1/ 5 rgw_access >>> >> 1/ 5 rgw_dbstore >>> >> 1/ 5 rgw_flight >>> >> 1/ 5 javaclient >>> >> 1/ 5 asok >>> >> 1/ 1 throttle >>> >> 0/ 0 refs >>> >> 1/ 5 compressor >>> >> 1/ 5 bluestore >>> >> 1/ 5 bluefs >>> >> 1/ 3 bdev >>> >> 1/ 5 kstore >>> >> 4/ 5 rocksdb >>> >> 4/ 5 leveldb >>> >> 1/ 5 fuse >>> >> 2/ 5 mgr >>> >> 1/ 5 mgrc >>> >> 1/ 5 dpdk >>> >> 1/ 5 eventtrace >>> >> 1/ 5 prioritycache >>> >> 0/ 5 test >>> >> 0/ 5 cephfs_mirror >>> >> 0/ 5 cephsqlite >>> >> 0/ 5 seastore >>> >> 0/ 5 seastore_onode >>> >> 0/ 5 seastore_odata >>> >> 0/ 5 seastore_omap >>> >> 0/ 5 seastore_tm >>> >> 0/ 5 seastore_t >>> >> 0/ 5 seastore_cleaner >>> >> 0/ 5 seastore_epm >>> >> 0/ 5 seastore_lba >>> >> 0/ 5 seastore_fixedkv_tree >>> >> 0/ 5 seastore_cache >>> >> 0/ 5 seastore_journal >>> >> 0/ 5 seastore_device >>> >> 0/ 5 seastore_backref >>> >> 0/ 5 alienstore >>> >> 1/ 5 mclock >>> >> 0/ 5 cyanstore >>> >> 1/ 5 ceph_exporter >>> >> 1/ 5 memstore >>> >> -2/-2 (syslog threshold) >>> >> 99/99 (stderr threshold) >>> >> --- pthread ID / name mapping for recent threads --- >>> >> 7f2472a89b00 / safe_timer >>> >> 7f2472cadb00 / radosgw >>> >> ... >>> >> log_file >>> >> >>> >> >>> >> /var/lib/ceph/crash/2024-01-25T13:10:13.909546Z_01ee6e6a-e946-4006-9d32-e17ef2f9df74/log >>> >> --- end dump of recent events --- >>> >> reraise_fatal: default handler for signal 25 didn't terminate >>> the process? >>> >> >>> >> Thank you for your help. >>> >> >>> >> Marc >>> >> _______________________________________________ >>> >> ceph-users mailing list --ceph-users(a)ceph.io >>> >> To unsubscribe send an emailtoceph-users-leave(a)ceph.io >>> >> >>> > >>> _______________________________________________ >>> ceph-users mailing list --ceph-users(a)ceph.io >>> To unsubscribe send an email toceph-users-leave(a)ceph.io >>> >>> >>> >>> -- >>> >>> Matt Benjamin >>> Red Hat, Inc. >>> 315 West Huron Street, Suite 140A >>> Ann Arbor, Michigan 48103 >>> >>> http://www.redhat.com/en/technologies/storage >>> >>> tel. 734-821-5101 >>> fax. 734-769-8938 >>> cel. 734-216-5309 >> _______________________________________________ >> ceph-users mailing list --ceph-users(a)ceph.io >> To unsubscribe send an email toceph-users-leave(a)ceph.io >> >