- ceph-users - lists.ceph.io

Behavior of EC pool when a host goes offline
by majia xiao 03 Dec '19

03 Dec '19

Hi all, We have a Ceph（version 12.2.4）cluster that adopts EC pools, and it consists of 10 hosts for OSDs. The corresponding commands to create the EC pool are listed as follows: ceph osd erasure-code-profile set profile_jerasure_4_3_reed_sol_van \ plugin=jerasure \ k=4 \ m=3 \ technique=reed_sol_van \ packetsize=2048 \ crush-device-class=hdd \ crush-failure-domain=host ceph osd pool create pool_jerasure_4_3_reed_sol_van 2048 2048 erasure profile_jerasure_4_3_reed_sol_van Since that the EC pool's crush-failure-domain is configured to be "host", we just disable the network interfaces of some hosts (using "ifdown" command) to verify the functionality of the EC pool. And here are the phenomena we have observed: First of all, the IO rate (of "rados bench", which we used for benchmark) drops immediately to 0 when one host goes offline. Secondly, it takes a lot of time (around 100 seconds) for Ceph to detect corresponding OSDs on that host are down. Finally, once the Ceph has detected all offline OSDs, the EC pool seems to act normally and it is ready for IO operations again. So, here are my questions: 1. Is this normal that the IO rate drops to 0 immediately even though there is only one host goes offline? 2. How to make Ceph reduce the time needed to detect failed OSDs? Thanks for any help. Best regards, Majia Xiao

2 1

how to speed up mount a ceph fs when a node unusual down in ceph cluster
by hfx＠portsip.cn 03 Dec '19

03 Dec '19

I created one ceph cluster. node-1: mon, mgr, osd.0, mds node-2: mon, mgr, osd.1, mds node-3: mon, mgr, osd.2, mds When the cluster is working normally, using command "mount -t ceph <node-*-ip:6789>:/ /mnt -o name=admin,secret=<admin client secret>" to mount is ok. But when a node unusual down(like poweroff), and using same command to mount, it will be hanging long time(maybe more then 1 minutes). I trying to configure "mds reconnect timeout = 0" in ceph.conf, mount time has been shortened. My Question: What are the configurations that affect the ceph file system mount in this scenario? BRs. hfx(a)portsip.cn

1 0

Possible data corruption with 14.2.3 and 14.2.4
by Sage Weil 03 Dec '19

03 Dec '19

Hi everyone, We've identified a data corruption bug[1], first introduced[2] (by yours truly) in 14.2.3 and affecting both 14.2.3 and 14.2.4. The corruption appears as a rocksdb checksum error or assertion that looks like os/bluestore/fastbmap_allocator_impl.h: 750: FAILED ceph_assert(available >= allocated) or in some cases a rocksdb checksum error. It only affects BlueStore OSDs that have a separate 'db' or 'wal' device. We have a fix[3] that is working its way through testing, and will expedite the next Nautilus point release (14.2.5) once it is ready. If you are running 14.2.2 or 14.2.1 and use BlueStore OSDs with separate 'db' volumes, you should consider waiting to upgrade until 14.2.5 is released. A big thank you to Igor Fedotov and several *extremely* helpful users who managed to reproduce and track down this problem! sage [1] https://tracker.ceph.com/issues/42223 [2] https://github.com/ceph/ceph/commit/096033b9d931312c0688c2eea7e14626bfde0ad… [3] https://github.com/ceph/ceph/pull/31621

5 8

Can min_read_recency_for_promote be -1
by Robert LeBlanc 03 Dec '19

03 Dec '19

I'd like to configure a cache tier to act as a write buffer, so that if writes come in, it promotes objects, but reads never promote an object. We have a lot of cold data so we would like to tier down to an EC pool (CephFS) after a period of about 30 days to save space. The storage tier and the 'cache' tier would be on the same spindles, so the only performance improvement would be from the faster writes with replication. So we don't want to really move data between tiers. The idea would be to not promote on read since EC read performance is good enough and have writes go to the cache tier where the data may be 'hot' for a week or so, then get cold. It seems that we would only need one hit_set and if -1 can't be set for min_read_recency_for_promote, I could probably use 2 which would never hit because there is only one set, but that may error too. The follow up is how big a set should be as it only really tells if an object "may" be in cache and does not determine when things are flushed, so it really only matters how out-of-date we are okay with the bloom filter being out of date, right? So we could have it be a day long if we are okay with that stale rate? Is there any advantage to having a longer period for a bloom filter? Now, I'm starting to wonder if I even need a bloom filter for this use case, can I get tiering to work without it and only use cache_min_flush_age/cach_min_evict_age since I don't care about promoting when there are X hits in Y time? Thanks ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 1

ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)
by Marc Roos 02 Dec '19

02 Dec '19

I have been asking before[1]. Since Nautilus upgrade I am having these, with a total node failure as a result(?). Was not expecting this in my 'low load' setup. Maybe now someone can help resolving this? I am also waiting quite some time to get access at https://tracker.ceph.com/issues. Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9287 ffff911a9a26bd00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9283 ffff911d34e69d00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9276 ffff911d34e69c00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c926c ffff912068b92c00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9268 ffff912068b93000 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c926d ffff912068b92900 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c928a ffff912118e5be00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9272 ffff9119950d9500 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9269 ffff911940f3d000 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9270 ffff911748427c00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c926b ffff91169b000600 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9281 ffff91169b000500 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9288 ffff9115844d2500 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c927d ffff9115844d2e00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9280 ffff91186401b000 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9267 ffff9121535ecc00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c927c ffff9121cecb1e00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9271 ffff9121cecb0400 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9279 ffff911d26646300 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c927f ffff911d26646900 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9275 ffff9121cecb1700 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9259 ffff91170c9f6600 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9257 ffff9118ef2a8000 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c924e ffff911a1e091800 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9262 ffff911a1e090c00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9266 ffff9115e3859500 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c924f ffff9118aefd1300 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c925f ffff91170c9f6100 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9252 ffff9115e3859800 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9256 ffff912045dc5300 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9254 ffff91170c9f6900 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9261 ffff91170c9f7100 fail -12 Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020d4ec4 ffff9118aefd0000 fail -12 [1] https://www.mail-archive.com/ceph-users@ceph.io/msg01088.html https://www.mail-archive.com/ceph-users@ceph.io/msg00969.html

2 2

Re: ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)
by Ilya Dryomov 02 Dec '19

02 Dec '19

On Mon, Dec 2, 2019 at 12:48 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > Hi Ilya, > > > > > > >ISTR there were some anti-spam measures put in place. Is your account > >waiting for manual approval? If so, David should be able to help. > > Yes if I remember correctly I get waiting approval when I try to log in. > > >> > >> > >> > >> Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9287 > >> ffff911a9a26bd00 fail -12 > >> Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9283 > > > > > >It is failing to allocate memory. "low load" isn't very specific, > >can you describe the setup and the workload in more detail? > > 4 nodes (osd, mon combined), the 4th node has local cephfs mount, which > is rsync'ing some files from vm's. 'low load' I have sort of test setup, > going to production. Mostly the nodes are below a load of 1 (except when > the concurrent rsync starts) > > >How many snapshots do you have? > > Don't know how to count them. I have script running on a 2000 dirs. If > one of these dirs is not empty it creates a snapshot. So in theory I > could have 2000 x 7 days = 14000 snapshots. > (btw the cephfs snapshots are in a different tree than the rsync is > using) Is there a reason you are snapshotting each directory individually instead of just snapshotting a common parent? If you have thousands of snapshots, you may eventually hit a different bug: https://tracker.ceph.com/issues/21420 https://docs.ceph.com/docs/master/cephfs/experimental-features/#snapshots Be aware that each set of 512 snapshots amplify your writes by 4K in terms of network consumption. With 14000 snapshots, a 4K write would need to transfer ~109K worth of snapshot metadata to carry itself out. Thanks, Ilya

2 2

Re: ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)
by Ilya Dryomov 02 Dec '19

02 Dec '19

On Mon, Dec 2, 2019 at 1:23 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > I guess this is related? kworker 100% > > > [Mon Dec 2 13:05:27 2019] SysRq : Show backtrace of all active CPUs > [Mon Dec 2 13:05:27 2019] sending NMI to all CPUs: > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 0 skipped: idling at pc > 0xffffffffb0581e94 > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 1 skipped: idling at pc > 0xffffffffb0581e94 > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 2 skipped: idling at pc > 0xffffffffb0581e94 > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 3 skipped: idling at pc > 0xffffffffb0581e94 > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 4 > [Mon Dec 2 13:05:27 2019] CPU: 4 PID: 426200 Comm: kworker/4:2 Not > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > [Mon Dec 2 13:05:27 2019] Hardware name: Supermicro > X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.0b 05/27/2014 > [Mon Dec 2 13:05:27 2019] Workqueue: ceph-msgr ceph_con_workfn > [libceph] > [Mon Dec 2 13:05:27 2019] task: ffffa0c8e1240000 ti: ffffa0ccb6364000 > task.ti: ffffa0ccb6364000 > [Mon Dec 2 13:05:27 2019] RIP: 0010:[<ffffffffc08d7db9>] > [<ffffffffc08d7db9>] cmpu64_rev+0x19/0x20 [ceph] > [Mon Dec 2 13:05:27 2019] RSP: 0018:ffffa0ccb6367a20 EFLAGS: 00000202 > [Mon Dec 2 13:05:27 2019] RAX: 0000000000000001 RBX: 0000000000000038 > RCX: 0000000000000008 > [Mon Dec 2 13:05:27 2019] RDX: 0000000000025c33 RSI: ffffa0cbbe380050 > RDI: ffffa0cbbe380030 > [Mon Dec 2 13:05:27 2019] RBP: ffffa0ccb6367a20 R08: 0000000000000018 > R09: 00000000000013ed > [Mon Dec 2 13:05:27 2019] R10: 0000000000000002 R11: ffffe94994f8e000 > R12: ffffa0cbbe380030 > [Mon Dec 2 13:05:27 2019] R13: ffffffffc08d7da0 R14: ffffa0cbbe380018 > R15: ffffa0cbbe380050 > [Mon Dec 2 13:05:27 2019] FS: 0000000000000000(0000) > GS:ffffa0d2cfb00000(0000) knlGS:0000000000000000 > [Mon Dec 2 13:05:27 2019] CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > [Mon Dec 2 13:05:27 2019] CR2: 000055a7c413fcb9 CR3: 0000001813010000 > CR4: 00000000000607e0 > [Mon Dec 2 13:05:27 2019] Call Trace: > [Mon Dec 2 13:05:27 2019] [<ffffffffb019303f>] sort+0x1af/0x260 > [Mon Dec 2 13:05:27 2019] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > [Mon Dec 2 13:05:27 2019] [<ffffffffc08d807b>] > build_snap_context+0x12b/0x290 [ceph] > [Mon Dec 2 13:05:27 2019] [<ffffffffc08d820c>] > rebuild_snap_realms+0x2c/0x90 [ceph] > [Mon Dec 2 13:05:27 2019] [<ffffffffc08d822b>] > rebuild_snap_realms+0x4b/0x90 [ceph] > [Mon Dec 2 13:05:27 2019] [<ffffffffc08d91fc>] > ceph_update_snap_trace+0x3ec/0x530 [ceph] > [Mon Dec 2 13:05:27 2019] [<ffffffffc08e2239>] > handle_reply+0x359/0xc60 [ceph] > [Mon Dec 2 13:05:27 2019] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > [ceph] > [Mon Dec 2 13:05:27 2019] [<ffffffffb042e56a>] ? > kernel_recvmsg+0x3a/0x50 > [Mon Dec 2 13:05:27 2019] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > [libceph] > [Mon Dec 2 13:05:27 2019] [<ffffffffafee13ce>] ? > account_entity_dequeue+0xae/0xd0 > [Mon Dec 2 13:05:27 2019] [<ffffffffafee4d5c>] ? > dequeue_entity+0x11c/0x5e0 > [Mon Dec 2 13:05:27 2019] [<ffffffffb042e417>] ? > kernel_sendmsg+0x37/0x50 > [Mon Dec 2 13:05:27 2019] [<ffffffffc05fdfb4>] > ceph_con_workfn+0xe4/0x1530 [libceph] > [Mon Dec 2 13:05:27 2019] [<ffffffffb057f568>] ? > __schedule+0x448/0x9c0 > [Mon Dec 2 13:05:27 2019] [<ffffffffafebe21f>] > process_one_work+0x17f/0x440 > [Mon Dec 2 13:05:27 2019] [<ffffffffafebf336>] > worker_thread+0x126/0x3c0 > [Mon Dec 2 13:05:27 2019] [<ffffffffafebf210>] ? > manage_workers.isra.26+0x2a0/0x2a0 > [Mon Dec 2 13:05:27 2019] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > [Mon Dec 2 13:05:27 2019] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Mon Dec 2 13:05:27 2019] [<ffffffffb058cd37>] > ret_from_fork_nospec_begin+0x21/0x21 > [Mon Dec 2 13:05:27 2019] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Mon Dec 2 13:05:27 2019] Code: 87 c8 fc ff ff 5d 0f 94 c0 0f b6 c0 c3 > 0f 1f 44 00 00 66 66 66 66 90 48 8b 16 48 39 17 b8 01 00 00 00 55 48 89 > e5 72 08 0f 97 c0 <0f> b6 c0 f7 d8 5d c3 66 66 66 66 90 55 f6 05 ed 92 > 02 00 04 48 > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 5 Yes, seems related. I'm not sure how it relates to an upgrade to nautilus, but as I mentioned in a different message, with thousands of snapshots you are in a dangerous territory anyway. Thanks, Ilya

1 0

Re: ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)
by Ilya Dryomov 02 Dec '19

02 Dec '19

On Mon, Dec 2, 2019 at 10:27 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I have been asking before[1]. Since Nautilus upgrade I am having these, > with a total node failure as a result(?). Was not expecting this in my > 'low load' setup. Maybe now someone can help resolving this? I am also > waiting quite some time to get access at > https://tracker.ceph.com/issues. Hi Marc, ISTR there were some anti-spam measures put in place. Is your account waiting for manual approval? If so, David should be able to help. > > > > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9287 > ffff911a9a26bd00 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9283 > ffff911d34e69d00 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9276 > ffff911d34e69c00 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c926c > ffff912068b92c00 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9268 > ffff912068b93000 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c926d > ffff912068b92900 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c928a > ffff912118e5be00 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9272 > ffff9119950d9500 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9269 > ffff911940f3d000 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9270 > ffff911748427c00 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c926b > ffff91169b000600 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9281 > ffff91169b000500 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9288 > ffff9115844d2500 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c927d > ffff9115844d2e00 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9280 > ffff91186401b000 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9267 > ffff9121535ecc00 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c927c > ffff9121cecb1e00 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9271 > ffff9121cecb0400 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9279 > ffff911d26646300 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c927f > ffff911d26646900 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9275 > ffff9121cecb1700 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9259 > ffff91170c9f6600 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9257 > ffff9118ef2a8000 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c924e > ffff911a1e091800 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9262 > ffff911a1e090c00 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9266 > ffff9115e3859500 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c924f > ffff9118aefd1300 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c925f > ffff91170c9f6100 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9252 > ffff9115e3859800 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9256 > ffff912045dc5300 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9254 > ffff91170c9f6900 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9261 > ffff91170c9f7100 fail -12 > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020d4ec4 > ffff9118aefd0000 fail -12 It is failing to allocate memory. "low load" isn't very specific, can you describe the setup and the workload in more detail? How many snapshots do you have? Do you keep track of memory consumption on the node? Finally, you say "crash" in the subject. Does the kernel actually crash or perhaps it locks up? If it actually crashes, do you have the panic message? Thanks, Ilya

2 2

Dual network board setup info
by Rodrigo Severo - Fábrica 02 Dec '19

02 Dec '19

Hi, I have a CephFS instance and I am also planning on also deploying an Object Storage interface. My servers have 2 network boards each. I would like to use the current local one to talk to Cephs clients (both CephFS and Object Storage) and use the second one to all Cephs processes to talk one to the other. I'm quite sure that Ceph has support for this kind of setup but I can't find how to do such a thing. I already have a "public network" setting that, AFAIU, sets the interface used by Ceph to talk to clients but I don't know if it defines CephFS MDS, RGW, or all of them. And I can't see how to make the different Ceph processes to talk to each other through the other interface. I found "mon_host" setting but should I change it to get what I want? And what else? Is there some example of such a setup that I could learn from? Or maybe someone would be kind enough to scratch a basic config to achieve this setup? Or maybe there is some documentation that deals with such a scenario that I haven't found. Well, I'm in search for more info, any kind. Thanks in advance for your help and attention, Rodrigo Severo

4 7

Disable pgmap messages? Still having this Bug #39646
by Marc Roos 02 Dec '19

02 Dec '19

I am getting these every 2 seconds, does it make sense to log this? log_channel(cluster) log [DBG] : pgmap v32653: 384 pgs: 2 active+clean+scrubbing+deep, 382 active+clean; log_channel(cluster) log [DBG] : pgmap v32653: 384 pgs: 2 active+clean+scrubbing+deep, 382 active+clean; log_channel(cluster) log [DBG] : pgmap v32653: 384 pgs: 2 active+clean+scrubbing+deep, 382 active+clean; log_channel(cluster) log [DBG] : pgmap v32653: 384 pgs: 2 active+clean+scrubbing+deep, 382 active+clean; log_channel(cluster) log [DBG] : pgmap v32653: 384 pgs: 2 active+clean+scrubbing+deep, 382 active+clean; The mentioned solution in the comments of the bug[1] do not work [@c01 ~]# ceph daemon mon.a config show | grep cluster_log "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", "mon_cluster_log_file_level": "info", "mon_cluster_log_to_file": "true", "mon_cluster_log_to_graylog": "false", "mon_cluster_log_to_graylog_host": "127.0.0.1", "mon_cluster_log_to_graylog_port": "12201", "mon_cluster_log_to_stderr": "false", "mon_cluster_log_to_syslog": "default=false", "mon_cluster_log_to_syslog_facility": "daemon", "mon_cluster_log_to_syslog_level": "info", PS. These are logs on the remote syslog server. [1] https://tracker.ceph.com/issues/39646

1 0