bug? cant turn off rbd cache?
Oops.. sent this to the "wrong" list previously. (lists.ceph.com) Lets try the proper one this time :-/ not sure if this an actual bug or I'm doing something else wrong. but in Octopus, I have on the master node # ceph --version ceph version 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) octopus (stable) # tail -3 /etc/ceph/ceph.conf [client] rbd cache = false rbd cache writethrough until flush = false but I have restarted ALL nodes.. and yet rbd cache is still on. # ceph --admin-daemon `find /var/run/ceph -name 'ceph-mon*'` config show |grep rbd_cache "rbd_cache": "true", What else am I supposed to do??? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| www.medata.com
Hi,
[client] rbd cache = false rbd cache writethrough until flush = false
this is the rbd client's config, not the global MON config you're reading here:
# ceph --admin-daemon `find /var/run/ceph -name 'ceph-mon*'` config show |grep rbd_cache "rbd_cache": "true",
If you want to change the global "rbd_cache" setting you have to change that in the [global] (or maybe [mon]) section. But I'm not sure which effect this will have. Regards, Eugen Zitat von Philip Brown <pbrown@medata.com>:
Oops.. sent this to the "wrong" list previously. (lists.ceph.com) Lets try the proper one this time :-/
not sure if this an actual bug or I'm doing something else wrong. but in Octopus, I have on the master node
# ceph --version ceph version 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) octopus (stable) # tail -3 /etc/ceph/ceph.conf [client] rbd cache = false rbd cache writethrough until flush = false
but I have restarted ALL nodes.. and yet rbd cache is still on.
# ceph --admin-daemon `find /var/run/ceph -name 'ceph-mon*'` config show |grep rbd_cache "rbd_cache": "true",
What else am I supposed to do???
-- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| www.medata.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, Dec 17, 2020 at 7:22 AM Eugen Block <eblock@nde.ag> wrote:
Hi,
[client] rbd cache = false rbd cache writethrough until flush = false
this is the rbd client's config, not the global MON config you're reading here:
# ceph --admin-daemon `find /var/run/ceph -name 'ceph-mon*'` config show |grep rbd_cache "rbd_cache": "true",
If you want to change the global "rbd_cache" setting you have to change that in the [global] (or maybe [mon]) section. But I'm not sure which effect this will have.
If you want to change global (librbd) settings, please use "rbd config global set ..." (which is just a wrapper around "ceph config set global ..."). The conf files only apply to the single host where they exist.
Regards, Eugen
Zitat von Philip Brown <pbrown@medata.com>:
Oops.. sent this to the "wrong" list previously. (lists.ceph.com) Lets try the proper one this time :-/
not sure if this an actual bug or I'm doing something else wrong. but in Octopus, I have on the master node
# ceph --version ceph version 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) octopus (stable) # tail -3 /etc/ceph/ceph.conf [client] rbd cache = false rbd cache writethrough until flush = false
but I have restarted ALL nodes.. and yet rbd cache is still on.
# ceph --admin-daemon `find /var/run/ceph -name 'ceph-mon*'` config show |grep rbd_cache "rbd_cache": "true",
What else am I supposed to do???
-- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| www.medata.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
Huhhh... Its unfortunate that every google search i did for turning off rbd cache, specified "put it in the [client] section". Doh. Maybe this would make a good candidate to update the ceph rbd docs? Speaking of which.. what is the *exact* syntax for that command please? None of the below work: rbd config global set rbd_cache false rbd config global set rbd cache false rbd config global set conf_rbd_cache false rbd config global set rbd cache false rbd config global set global cache false and the error messages are not helpful. for example, # rbd config global set cache false rbd: invalid config entity: cache (must be global, client or client.<id>) ??? I already specified global?? but then # rbd config global set global cache false rbd: not rbd option: cache Very frustrating. ----- Original Message ----- From: "Jason Dillaman" <jdillama@redhat.com> To: "Eugen Block" <eblock@nde.ag> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, December 17, 2020 5:09:17 AM Subject: [ceph-users] Re: bug? cant turn off rbd cache? If you want to change global (librbd) settings, please use "rbd config global set ..." (which is just a wrapper around "ceph config set global ..."). The conf files only apply to the single host where they exist.
On Thu, Dec 17, 2020 at 10:41 AM Philip Brown <pbrown@medata.com> wrote:
Huhhh...
Its unfortunate that every google search i did for turning off rbd cache, specified "put it in the [client] section". Doh.
Maybe this would make a good candidate to update the ceph rbd docs?
As an open source project, I highly encourage you to give back in any way you can [1].
Speaking of which.. what is the *exact* syntax for that command please? None of the below work:
rbd config global set rbd_cache false rbd config global set rbd cache false rbd config global set conf_rbd_cache false rbd config global set rbd cache false rbd config global set global cache false
and the error messages are not helpful. for example,
# rbd config global set cache false rbd: invalid config entity: cache (must be global, client or client.<id>)
??? I already specified global??
... because a global configuration override can apply to all entities, to all clients, or to specific clients (hence the examples for the entity parameter).
but then
# rbd config global set global cache false rbd: not rbd option: cache
... the configuration option is "rbd_cache" as documented here [2].
Very frustrating.
----- Original Message ----- From: "Jason Dillaman" <jdillama@redhat.com> To: "Eugen Block" <eblock@nde.ag> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, December 17, 2020 5:09:17 AM Subject: [ceph-users] Re: bug? cant turn off rbd cache?
If you want to change global (librbd) settings, please use "rbd config global set ..." (which is just a wrapper around "ceph config set global ..."). The conf files only apply to the single host where they exist. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] https://docs.ceph.com/en/latest/start/documenting-ceph/ [2] https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#cache-settings -- Jason
I guess I left out in my examples, where I tried rbd_cache as well, and failed # rbd config global set rbd_cache false rbd: invalid config entity: rbd_cache (must be global, client or client.<id>) So, while i am happy to file a documentation pull request.. I still need to find the specific command line that actually *works*, for the "rbd config" variant, etc. ----- Original Message ----- From: "Jason Dillaman" <jdillama@redhat.com> To: "Philip Brown" <pbrown@medata.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, December 17, 2020 7:48:22 AM Subject: Re: [ceph-users] Re: bug? cant turn off rbd cache? On Thu, Dec 17, 2020 at 10:41 AM Philip Brown <pbrown@medata.com> wrote:
Huhhh...
Its unfortunate that every google search i did for turning off rbd cache, specified "put it in the [client] section". Doh.
Maybe this would make a good candidate to update the ceph rbd docs?
As an open source project, I highly encourage you to give back in any way you can [1].
Speaking of which.. what is the *exact* syntax for that command please? None of the below work:
rbd config global set rbd_cache false rbd config global set rbd cache false rbd config global set conf_rbd_cache false rbd config global set rbd cache false rbd config global set global cache false
and the error messages are not helpful. for example,
# rbd config global set cache false rbd: invalid config entity: cache (must be global, client or client.<id>)
??? I already specified global??
... because a global configuration override can apply to all entities, to all clients, or to specific clients (hence the examples for the entity parameter).
but then
# rbd config global set global cache false rbd: not rbd option: cache
... the configuration option is "rbd_cache" as documented here [2].
Very frustrating.
On Thu, Dec 17, 2020 at 11:21 AM Philip Brown <pbrown@medata.com> wrote:
I guess I left out in my examples, where I tried rbd_cache as well, and failed
# rbd config global set rbd_cache false rbd: invalid config entity: rbd_cache (must be global, client or client.<id>)
But that's not a valid command -- you attempted to provide the configuration key name for the entity (as the error message is telling you). $ rbd config global set global rbd_cache false ... or ... $ rbd config global set client rbd_cache false ... or ... $ rbd config global set client.host_a rbd_cache false
So, while i am happy to file a documentation pull request.. I still need to find the specific command line that actually *works*, for the "rbd config" variant, etc.
----- Original Message ----- From: "Jason Dillaman" <jdillama@redhat.com> To: "Philip Brown" <pbrown@medata.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, December 17, 2020 7:48:22 AM Subject: Re: [ceph-users] Re: bug? cant turn off rbd cache?
On Thu, Dec 17, 2020 at 10:41 AM Philip Brown <pbrown@medata.com> wrote:
Huhhh...
Its unfortunate that every google search i did for turning off rbd cache, specified "put it in the [client] section". Doh.
Maybe this would make a good candidate to update the ceph rbd docs?
As an open source project, I highly encourage you to give back in any way you can [1].
Speaking of which.. what is the *exact* syntax for that command please? None of the below work:
rbd config global set rbd_cache false rbd config global set rbd cache false rbd config global set conf_rbd_cache false rbd config global set rbd cache false rbd config global set global cache false
and the error messages are not helpful. for example,
# rbd config global set cache false rbd: invalid config entity: cache (must be global, client or client.<id>)
??? I already specified global??
... because a global configuration override can apply to all entities, to all clients, or to specific clients (hence the examples for the entity parameter).
but then
# rbd config global set global cache false rbd: not rbd option: cache
... the configuration option is "rbd_cache" as documented here [2].
Very frustrating.
-- Jason
Huhhh.. It seems worthwhile to point out two inconsistencies, then. 1. the "old way", of ceph config set global rbd_cache false doesnt require this odd redundant "global set global" syntax. It is confusing to users to have to specify "global" twice. may I suggest that the syntax for rbd config be adjusted, so that people can simply use rbd config global ... or rbd config client ... not rbd config global set (global/client) 2. The fact that "ceph config set global rbd_cache false" works, seems to imply that putting [global] rbd cache = false in /etc/ceph/ceph.conf should work also. Except it doesnt. Even after fully shutting down every node in the ceph cluster and doing a cold startup. is that a bug? ----- Original Message ----- From: "Jason Dillaman" <jdillama@redhat.com> To: "Philip Brown" <pbrown@medata.com> Cc: "dillaman" <dillaman@redhat.com>, "ceph-users" <ceph-users@ceph.io> Sent: Thursday, December 17, 2020 8:24:59 AM Subject: Re: [ceph-users] Re: bug? cant turn off rbd cache? On Thu, Dec 17, 2020 at 11:21 AM Philip Brown <pbrown@medata.com> wrote:
I guess I left out in my examples, where I tried rbd_cache as well, and failed
# rbd config global set rbd_cache false rbd: invalid config entity: rbd_cache (must be global, client or client.<id>)
But that's not a valid command -- you attempted to provide the configuration key name for the entity (as the error message is telling you). $ rbd config global set global rbd_cache false ... or ... $ rbd config global set client rbd_cache false ... or ... $ rbd config global set client.host_a rbd_cache false
On Thu, Dec 17, 2020 at 12:09 PM Philip Brown <pbrown@medata.com> wrote:
Huhhh.. It seems worthwhile to point out two inconsistencies, then.
1. the "old way", of ceph config set global rbd_cache false
doesnt require this odd redundant "global set global" syntax. It is confusing to users to have to specify "global" twice. may I suggest that the syntax for rbd config be adjusted, so that people can simply use rbd config global ... or rbd config client ...
not rbd config global set (global/client)
Everyone has different opinions. "rbd help config global set" clearly defines all the expected parameters. We can't be all things to all people.
2. The fact that "ceph config set global rbd_cache false" works, seems to imply that putting
[global] rbd cache = false
in /etc/ceph/ceph.conf should work also.
Except it doesnt. Even after fully shutting down every node in the ceph cluster and doing a cold startup.
is that a bug?
Nope [1]. How would changing a random configuration file on a random node affect the configuration on another node?
----- Original Message ----- From: "Jason Dillaman" <jdillama@redhat.com> To: "Philip Brown" <pbrown@medata.com> Cc: "dillaman" <dillaman@redhat.com>, "ceph-users" <ceph-users@ceph.io> Sent: Thursday, December 17, 2020 8:24:59 AM Subject: Re: [ceph-users] Re: bug? cant turn off rbd cache?
On Thu, Dec 17, 2020 at 11:21 AM Philip Brown <pbrown@medata.com> wrote:
I guess I left out in my examples, where I tried rbd_cache as well, and failed
# rbd config global set rbd_cache false rbd: invalid config entity: rbd_cache (must be global, client or client.<id>)
But that's not a valid command -- you attempted to provide the configuration key name for the entity (as the error message is telling you).
$ rbd config global set global rbd_cache false ... or ... $ rbd config global set client rbd_cache false ... or ... $ rbd config global set client.host_a rbd_cache false
[1] https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#config-source... -- Jason
participants (3)
-
Eugen Block
-
Jason Dillaman
-
Philip Brown