Ceph 16.2.x excessive logging, how to reduce?

List overview All Threads
Download

newer

older

Received signal: Hangup from...

Manual resharding with multisite

Zakhar Kirpichenko

19 Sep 2023 19 Sep '23

9:01 a.m.

Hi, Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very detailed messages, Ceph logs alone on hosts with monitors and several OSDs has already eaten through 50% of the endurance of the flash system drives over a couple of years. Cluster logging settings are default, and it seems that all daemons are writing lots and lots of debug information to the logs, such as for example: https://pastebin.com/ebZq8KZk (it's just a snippet, but there's lots and lots of various information). Is there a way to reduce the amount of logging and, for example, limit the logging to warnings or important messages so that it doesn't include every successful authentication attempt, compaction etc, etc, when the cluster is healthy and operating normally? I would very much appreciate your advice on this. Best regards, Zakhar

Show replies by date

Zakhar Kirpichenko

19 Sep 19 Sep

10:54 p.m.

Any input from anyone, please? On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko <zakhar(a)gmail.com> wrote:

...

Zakhar Kirpichenko

4 Oct 4 Oct

8:51 p.m.

Any input from anyone, please? On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko <zakhar(a)gmail.com> wrote:

...

Igor Fedotov

9:23 p.m.

Hi Zakhar, do reduce rocksdb logging verbosity you might want to set debug_rocksdb to 3 (or 0). I presume it produces a significant part of the logging traffic. Thanks, Igor On 04/10/2023 20:51, Zakhar Kirpichenko wrote: > Any input from anyone, please? > > On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko <zakhar(a)gmail.com> wrote: > >> Hi, >> >> Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very >> detailed messages, Ceph logs alone on hosts with monitors and several OSDs >> has already eaten through 50% of the endurance of the flash system drives >> over a couple of years. >> >> Cluster logging settings are default, and it seems that all daemons are >> writing lots and lots of debug information to the logs, such as for >> example: https://pastebin.com/ebZq8KZk (it's just a snippet, but there's >> lots and lots of various information). >> >> Is there a way to reduce the amount of logging and, for example, limit the >> logging to warnings or important messages so that it doesn't include every >> successful authentication attempt, compaction etc, etc, when the cluster is >> healthy and operating normally? >> >> I would very much appreciate your advice on this. >> >> Best regards, >> Zakhar >> >> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Zakhar Kirpichenko

5 Oct 5 Oct

6:47 a.m.

Thank you for your response, Igor. Currently debug_rocksdb is set to 4/5: # ceph config get osd debug_rocksdb 4/5 This setting seems to be default. Is my understanding correct that you're suggesting setting it to 3/5 or even 0/5? Would setting it to 0/5 have any negative effects on the cluster? /Z On Wed, 4 Oct 2023 at 21:23, Igor Fedotov <igor.fedotov(a)croit.io> wrote:

...

Any input from anyone, please? On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko <zakhar(a)gmail.com>

wrote:

> Hi, > > Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very > detailed messages, Ceph logs alone on hosts with monitors and several

OSDs

> has already eaten through 50% of the endurance of the flash system

drives

> over a couple of years. > > Cluster logging settings are default, and it seems that all daemons are > writing lots and lots of debug information to the logs, such as for > example: https://pastebin.com/ebZq8KZk (it's just a snippet, but

there's

> lots and lots of various information). > > Is there a way to reduce the amount of logging and, for example, limit

the

> logging to warnings or important messages so that it doesn't include

every

> successful authentication attempt, compaction etc, etc, when the

cluster is

healthy and operating normally? I would very much appreciate your advice on this. Best regards, Zakhar

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Zakhar Kirpichenko

7 Oct 7 Oct

10 p.m.

...

Any input from anyone, please? On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko <zakhar(a)gmail.com>

wrote:

> Hi, > > Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very > detailed messages, Ceph logs alone on hosts with monitors and several

OSDs

> has already eaten through 50% of the endurance of the flash system

drives

there's

> lots and lots of various information). > > Is there a way to reduce the amount of logging and, for example, limit

the

> logging to warnings or important messages so that it doesn't include

every

> successful authentication attempt, compaction etc, etc, when the

cluster is

healthy and operating normally? I would very much appreciate your advice on this. Best regards, Zakhar

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Zakhar Kirpichenko

9 Oct 9 Oct

9:35 a.m.

Any input from anyone, please? This part of Ceph is very poorly documented. Perhaps there's a better place to ask this question? Please let me know. /Z On Sat, 7 Oct 2023 at 22:00, Zakhar Kirpichenko <zakhar(a)gmail.com> wrote:

...

Hi! I am still fighting excessive logging. I've reduced unnecessary logging from most components except for mon audit: https://pastebin.com/jjWvUEcQ How can I stop logging this particular type of messages? I would appreciate your help and advice. /Z On Thu, 5 Oct 2023 at 06:47, Zakhar Kirpichenko <zakhar(a)gmail.com> wrote: > Thank you for your response, Igor. > > Currently debug_rocksdb is set to 4/5: > > # ceph config get osd debug_rocksdb > 4/5 > > This setting seems to be default. Is my understanding correct that you're > suggesting setting it to 3/5 or even 0/5? Would setting it to 0/5 have any > negative effects on the cluster? > > /Z > > On Wed, 4 Oct 2023 at 21:23, Igor Fedotov <igor.fedotov(a)croit.io> wrote: > >> Hi Zakhar, >> >> do reduce rocksdb logging verbosity you might want to set debug_rocksdb >> to 3 (or 0). >> >> I presume it produces a significant part of the logging traffic. >> >> >> Thanks, >> >> Igor >> >> On 04/10/2023 20:51, Zakhar Kirpichenko wrote: >> > Any input from anyone, please? >> > >> > On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko <zakhar(a)gmail.com> >> wrote: >> > >> >> Hi, >> >> >> >> Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very >> >> detailed messages, Ceph logs alone on hosts with monitors and several >> OSDs >> >> has already eaten through 50% of the endurance of the flash system >> drives >> >> over a couple of years. >> >> >> >> Cluster logging settings are default, and it seems that all daemons >> are >> >> writing lots and lots of debug information to the logs, such as for >> >> example: https://pastebin.com/ebZq8KZk (it's just a snippet, but >> there's >> >> lots and lots of various information). >> >> >> >> Is there a way to reduce the amount of logging and, for example, >> limit the >> >> logging to warnings or important messages so that it doesn't include >> every >> >> successful authentication attempt, compaction etc, etc, when the >> cluster is >> >> healthy and operating normally? >> >> >> >> I would very much appreciate your advice on this. >> >> >> >> Best regards, >> >> Zakhar >> >> >> >> >> >> >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

Marc

10:03 a.m.

Did you do something like this Getting keys with ceph daemon mon.a config show | grep debug_ | grep mgr ceph tell mon.* injectargs --$monk=0/0

...

you're

> suggesting setting it to 3/5 or even 0/5? Would setting it to 0/5 have

any

> negative effects on the cluster? > > /Z > > On Wed, 4 Oct 2023 at 21:23, Igor Fedotov <igor.fedotov(a)croit.io> wrote: > >> Hi Zakhar, >> >> do reduce rocksdb logging verbosity you might want to set debug_rocksdb >> to 3 (or 0). >> >> I presume it produces a significant part of the logging traffic. >> >> >> Thanks, >> >> Igor >> >> On 04/10/2023 20:51, Zakhar Kirpichenko wrote: >> > Any input from anyone, please? >> > >> > On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko <zakhar(a)gmail.com> >> wrote: >> > >> >> Hi, >> >> >> >> Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very >> >> detailed messages, Ceph logs alone on hosts with monitors and

several

>> OSDs >> >> has already eaten through 50% of the endurance of the flash system >> drives >> >> over a couple of years. >> >> >> >> Cluster logging settings are default, and it seems that all daemons >> are >> >> writing lots and lots of debug information to the logs, such as for >> >> example: https://pastebin.com/ebZq8KZk (it's just a snippet, but >> there's >> >> lots and lots of various information). >> >> >> >> Is there a way to reduce the amount of logging and, for example, >> limit the >> >> logging to warnings or important messages so that it doesn't include >> every >> >> successful authentication attempt, compaction etc, etc, when the >> cluster is >> >> healthy and operating normally? >> >> >> >> I would very much appreciate your advice on this. >> >> >> >> Best regards, >> >> Zakhar >> >> >> >> >> >> >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Zakhar Kirpichenko

11:05 a.m.

Thanks for your reply, Marc! I did try to play with various debug settings. The issue is that mons produce logs of all commands issued by clients, not just mgr. For example, an Openstack Cinder node asking for space it can use: Oct 9 07:59:01 ceph03 bash[4019]: debug 2023-10-09T07:59:01.303+0000 7f489da8f700 0 log_channel(audit) log [DBG] : from='client.? 10.208.1.11:0/3286277243' entity='client.cinder' cmd=[{"prefix":"osd pool get-quota", "pool": "volumes-ssd", "format":"json"}]: dispatch It is unclear which particular mon debug option out of many controls this particular type of debug. I tried searching for documentation of mon debug options to no avail. /Z On Mon, 9 Oct 2023 at 10:03, Marc <Marc(a)f1-outsourcing.eu> wrote:

...

Did you do something like this Getting keys with ceph daemon mon.a config show | grep debug_ | grep mgr ceph tell mon.* injectargs --$monk=0/0

Any input from anyone, please? This part of Ceph is very poorly documented. Perhaps there's a better

place

to ask this question? Please let me know. /Z On Sat, 7 Oct 2023 at 22:00, Zakhar Kirpichenko <zakhar(a)gmail.com>

wrote:

> Hi! > > I am still fighting excessive logging. I've reduced unnecessary logging > from most components except for mon audit:

https://pastebin.com/jjWvUEcQ

> > How can I stop logging this particular type of messages? > > I would appreciate your help and advice. > > /Z > > On Thu, 5 Oct 2023 at 06:47, Zakhar Kirpichenko <zakhar(a)gmail.com>

wrote:

> Thank you for your response, Igor. > > Currently debug_rocksdb is set to 4/5: > > # ceph config get osd debug_rocksdb > 4/5 > > This setting seems to be default. Is my understanding correct that

you're

> suggesting setting it to 3/5 or even 0/5? Would setting it to 0/5 have

any >> negative effects on the cluster? >> >> /Z >> >> On Wed, 4 Oct 2023 at 21:23, Igor Fedotov <igor.fedotov(a)croit.io>

wrote:

>> >>> Hi Zakhar, >>> >>> do reduce rocksdb logging verbosity you might want to set

debug_rocksdb

>>> to 3 (or 0). >>> >>> I presume it produces a significant part of the logging traffic. >>> >>> >>> Thanks, >>> >>> Igor >>> >>> On 04/10/2023 20:51, Zakhar Kirpichenko wrote: >>> > Any input from anyone, please? >>> > >>> > On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko <

zakhar(a)gmail.com>

>>> wrote: >>> > >>> >> Hi, >>> >> >>> >> Our Ceph 16.2.x cluster managed by cephadm is logging a lot of

very

>> >> detailed messages, Ceph logs alone on hosts with monitors and

several >>> OSDs >>> >> has already eaten through 50% of the endurance of the flash system >>> drives >>> >> over a couple of years. >>> >> >>> >> Cluster logging settings are default, and it seems that all

daemons

>>> are >>> >> writing lots and lots of debug information to the logs, such as

for

>>> >> example: https://pastebin.com/ebZq8KZk (it's just a snippet, but >>> there's >>> >> lots and lots of various information). >>> >> >>> >> Is there a way to reduce the amount of logging and, for example, >>> limit the >>> >> logging to warnings or important messages so that it doesn't

include

>> every >> >> successful authentication attempt, compaction etc, etc, when the >> cluster is >> >> healthy and operating normally? >> >> >> >> I would very much appreciate your advice on this. >> >> >> >> Best regards, >> >> Zakhar >> >> >> >> >> >> >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Marc

11:24 a.m.

Hi Zakhar,

...

I did try to play with various debug settings. The issue is that mons produce logs of all commands issued by clients, not just mgr. For example, an Openstack Cinder node asking for space it can use: Oct 9 07:59:01 ceph03 bash[4019]: debug 2023-10-09T07:59:01.303+0000 7f489da8f700 0 log_channel(audit) log [DBG] : from='client.? 10.208.1.11:0/3286277243 <http://10.208.1.11:0/3286277243> ' entity='client.cinder' cmd=[{"prefix":"osd pool get-quota", "pool": "volumes-ssd", "format":"json"}]: dispatch

I am on a older version of ceph still so I am not sure if I even have these. There is also an option ceph.conf to do client logging [client] #debug client = 5

...

It is unclear which particular mon debug option out of many controls this particular type of debug. I tried searching for documentation of mon debug options to no avail.

Maybe there is something equal to this for logging? ceph daemon mon.a perf schema|less ceph daemon osd.0 perf schema|less

...

Did you do something like this Getting keys with ceph daemon mon.a config show | grep debug_ | grep mgr ceph tell mon.* injectargs --$monk=0/0

Any input from anyone, please? This part of Ceph is very poorly documented. Perhaps there's a

better place

to ask this question? Please let me know. /Z On Sat, 7 Oct 2023 at 22:00, Zakhar Kirpichenko <zakhar(a)gmail.com

<mailto:zakhar@gmail.com> > wrote:

> Hi! > > I am still fighting excessive logging. I've reduced unnecessary

logging

> from most components except for mon audit:

https://pastebin.com/jjWvUEcQ

> > How can I stop logging this particular type of messages? > > I would appreciate your help and advice. > > /Z > > >> Thank you for your response, Igor. >> >> Currently debug_rocksdb is set to 4/5: >> >> # ceph config get osd debug_rocksdb >> 4/5 >> >> This setting seems to be default. Is my understanding correct

that

you're >> suggesting setting it to 3/5 or even 0/5? Would setting it to

0/5 have

any >> negative effects on the cluster? >> >> /Z >> >> On Wed, 4 Oct 2023 at 21:23, Igor Fedotov <igor.fedotov(a)croit.io

<mailto:igor.fedotov@croit.io> > wrote:

>> >>> Hi Zakhar, >>> >>> do reduce rocksdb logging verbosity you might want to set

debug_rocksdb

>>> to 3 (or 0). >>> >>> I presume it produces a significant part of the logging

traffic.

>>> >>> >>> Thanks, >>> >>> Igor >>> >>> On 04/10/2023 20:51, Zakhar Kirpichenko wrote: >>> > Any input from anyone, please? >>> > >>> > On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko

<zakhar(a)gmail.com <mailto:zakhar@gmail.com> >

>>> wrote: >>> > >>> >> Hi, >>> >> >>> >> Our Ceph 16.2.x cluster managed by cephadm is logging a lot

of very

>>> >> detailed messages, Ceph logs alone on hosts with monitors

and

several >>> OSDs >>> >> has already eaten through 50% of the endurance of the flash

system

>>> drives >>> >> over a couple of years. >>> >> >>> >> Cluster logging settings are default, and it seems that all

daemons

>>> are >>> >> writing lots and lots of debug information to the logs, such

as for

>>> >> example: https://pastebin.com/ebZq8KZk (it's just a snippet,

but

>>> there's >>> >> lots and lots of various information). >>> >> >>> >> Is there a way to reduce the amount of logging and, for

example,

>>> limit the >>> >> logging to warnings or important messages so that it doesn't

include

>>> every >>> >> successful authentication attempt, compaction etc, etc, when

the

>>> cluster is >>> >> healthy and operating normally? >>> >> >>> >> I would very much appreciate your advice on this. >>> >> >>> >> Best regards, >>> >> Zakhar >>> >> >>> >> >>> >> >>> > _______________________________________________ >>> > ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-

users(a)ceph.io>

>>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io

<mailto:ceph-users-leave@ceph.io>

>> >

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-

users(a)ceph.io>

To unsubscribe send an email to ceph-users-leave(a)ceph.io

<mailto:ceph-users-leave@ceph.io>

Zakhar Kirpichenko

12:41 p.m.

I have `mon debug_client`, but it's already at 0/5 by default. /Z On Mon, 9 Oct 2023 at 11:24, Marc <Marc(a)f1-outsourcing.eu> wrote:

...

Hi Zakhar,

I did try to play with various debug settings. The issue is that mons produce logs of all commands issued by clients, not just mgr. For

example,

an Openstack Cinder node asking for space it can use: Oct 9 07:59:01 ceph03 bash[4019]: debug 2023-10-09T07:59:01.303+0000 7f489da8f700 0 log_channel(audit) log [DBG] : from='client.? 10.208.1.11:0/3286277243 <http://10.208.1.11:0/3286277243> ' entity='client.cinder' cmd=[{"prefix":"osd pool get-quota", "pool": "volumes-ssd", "format":"json"}]: dispatch

I am on a older version of ceph still so I am not sure if I even have these. There is also an option ceph.conf to do client logging [client] #debug client = 5

It is unclear which particular mon debug option out of many controls this particular type of debug. I tried searching for documentation of mon

debug

options to no avail.

Maybe there is something equal to this for logging? ceph daemon mon.a perf schema|less ceph daemon osd.0 perf schema|less

Did you do something like this Getting keys with ceph daemon mon.a config show | grep debug_ | grep mgr ceph tell mon.* injectargs --$monk=0/0

Any input from anyone, please? This part of Ceph is very poorly documented. Perhaps there's a

better place > to ask this question? Please let me know. > > /Z > > On Sat, 7 Oct 2023 at 22:00, Zakhar Kirpichenko <

zakhar(a)gmail.com

<mailto:zakhar@gmail.com> > wrote:

> Hi! > > I am still fighting excessive logging. I've reduced unnecessary

logging

> from most components except for mon audit:

https://pastebin.com/jjWvUEcQ

that

you're >> suggesting setting it to 3/5 or even 0/5? Would setting it to

0/5 have > any > >> negative effects on the cluster? > >> > >> /Z > >> > >> On Wed, 4 Oct 2023 at 21:23, Igor Fedotov <

igor.fedotov(a)croit.io

<mailto:igor.fedotov@croit.io> > wrote:

>> >>> Hi Zakhar, >>> >>> do reduce rocksdb logging verbosity you might want to set

debug_rocksdb

>>> to 3 (or 0). >>> >>> I presume it produces a significant part of the logging

traffic.

>>> >>> >>> Thanks, >>> >>> Igor >>> >>> On 04/10/2023 20:51, Zakhar Kirpichenko wrote: >>> > Any input from anyone, please? >>> > >>> > On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko

<zakhar(a)gmail.com <mailto:zakhar@gmail.com> > > >>> wrote: > >>> > > >>> >> Hi, > >>> >> > >>> >> Our Ceph 16.2.x cluster managed by cephadm is logging a

lot

of very

>>> >> detailed messages, Ceph logs alone on hosts with monitors

and > several > >>> OSDs > >>> >> has already eaten through 50% of the endurance of the

flash

system > >>> drives > >>> >> over a couple of years. > >>> >> > >>> >> Cluster logging settings are default, and it seems that

all

daemons > >>> are > >>> >> writing lots and lots of debug information to the logs,

such

as for > >>> >> example: https://pastebin.com/ebZq8KZk (it's just a

snippet,

but

>>> there's >>> >> lots and lots of various information). >>> >> >>> >> Is there a way to reduce the amount of logging and, for

example, > >>> limit the > >>> >> logging to warnings or important messages so that it

doesn't

include > >>> every > >>> >> successful authentication attempt, compaction etc, etc,

when

the > >>> cluster is > >>> >> healthy and operating normally? > >>> >> > >>> >> I would very much appreciate your advice on this. > >>> >> > >>> >> Best regards, > >>> >> Zakhar > >>> >> > >>> >> > >>> >> > >>> > _______________________________________________ > >>> > ceph-users mailing list -- ceph-users(a)ceph.io <mailto:

ceph-

users(a)ceph.io>

>>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io

<mailto:ceph-users-leave@ceph.io>

>> >

_______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-

users(a)ceph.io>

To unsubscribe send an email to ceph-users-leave(a)ceph.io

<mailto:ceph-users-leave@ceph.io>

Kai Stian Olstad

2:24 p.m.

On 09.10.2023 10:05, Zakhar Kirpichenko wrote:

...

This log say that it's bash with PID 4019 that is creating the log entry. Maybe start there, check what what other thing you are running on the server that creates this messages. -- Kai Stian Olstad

Zakhar Kirpichenko

2:27 p.m.

Thanks for the suggestion. That pid belongs to the mon process. I.e. the monitor is logging all client connections and commands. /Z On Mon, 9 Oct 2023 at 14:24, Kai Stian Olstad <ceph+list(a)olstad.com> wrote:

...

On 09.10.2023 10:05, Zakhar Kirpichenko wrote:

This log say that it's bash with PID 4019 that is creating the log entry. Maybe start there, check what what other thing you are running on the server that creates this messages. -- Kai Stian Olstad

223

days inactive

243

days old

ceph-users@ceph.io

Manage subscription

12 comments

4 participants

tags (0)

participants (4)

Igor Fedotov
Kai Stian Olstad
Marc
Zakhar Kirpichenko