Ceph Configuration Diff Tool - Request for comments
Hi all, We are planning to introduce a new tool/script which would help with the problem of visibility of the changes done to Ceph configuration options. Currently there are two main limitations: * Changes to the configuration are missed in our release notes * No easy way to know what configuration have modified/added/deleted between two releases Some context on ceph configuration management system: The default values for all the configuration options for a ceph cluster when starting up are in the yaml files present in the `src/common/options` directory. For example: `osd.yaml.in` file tells us all the configuration options available for `ceph-osd` daemon. The config options from yaml files are directly injected to the `CephContext` and these files are the sole source of truth to know about the configurations present for all the systems of ceph. It is important to note that, the values in the yaml files only represent the default values with which the cluster starts with, any changes made to these config options are not reflected in the yaml files. The detailed information about the configuration management is present here: https://docs.ceph.com/en/latest/dev/config/ Proposal to solve this problem is two fold: 1. Tool to compare configuration options between releases or commit hashes * A script which takes commit hashes or release names and diff the yaml files present in the `src/common/options` directory and output sections like: `Deleted`, `Added` and `Updated` * Developers can run this script whenever they wish to know the configuration diff between two releases * For example: `ceph-config-diff --mode diff-branch --ref-repo <repo-url> --ref-branch main --cmp-branch squid`, a command like this will compare the current main branch with the squid branch. 2. A Github action runs on every PR (it only executes when the yaml files are updated). This will be added as an optional check. * This action will check if any changes have occurred to yaml config files and write a comment to the PR notifying the PR author to also update the relevant release docs. * This action will reuse the script mentioned above to find the diff, if any present I believe, these two options should help solve the current visibility problems in ceph configuration system. Please let us know your thoughts about this feature. All inputs are welcome. Thanks, Naveen
Hi Naveen, This is a great idea! +1 on both parts of the proposal. This exact problem has definitely caught people off guard in the past when performing upgrades, so having a quick and easy way to verify changes will help reduce some friction for sure :) For the Github action, would you consider making the check non-optional to enforce that the release notes have been updated? I'm trying to imagine a scenario where we would want to change the defaults/add new options and not have that documented and I'm drawing a blank. Thank you, Nathan From: naveen.naidu@ibm.com At: 03/13/25 10:09:34 UTC-4:00To: dev@ceph.io Subject: Ceph Configuration Diff Tool - Request for comments Hi all, We are planning to introduce a new tool/script which would help with the problem of visibility of the changes done to Ceph configuration options. Currently there are two main limitations: *Changes to the configuration are missed in our release notes *No easy way to know what configuration have modified/added/deleted between two releases Some context on ceph configuration management system: The default values for all the configuration options for a ceph cluster when starting up are in the yaml files present in the `src/common/options` directory. For example: `osd.yaml.in` file tells us all the configuration options available for `ceph-osd` daemon. The config options from yaml files are directly injected to the `CephContext` and these files are the sole source of truth to know about the configurations present for all the systems of ceph. It is important to note that, the values in the yaml files only represent the default values with which the cluster starts with, any changes made to these config options are not reflected in the yaml files. The detailed information about the configuration management is present here: https://docs.ceph.com/en/latest/dev/config/ Proposal to solve this problem is two fold: 1. Tool to compare configuration options between releases or commit hashes *A script which takes commit hashes or release names and diff the yaml files present in the `src/common/options` directory and output sections like: `Deleted`, `Added` and `Updated` *Developers can run this script whenever they wish to know the configuration diff between two releases *For example: `ceph-config-diff --mode diff-branch --ref-repo <repo-url> --ref-branch main --cmp-branch squid`, a command like this will compare the current main branch with the squid branch. 2. A Github action runs on every PR (it only executes when the yaml files are updated). This will be added as an optional check. *This action will check if any changes have occurred to yaml config files and write a comment to the PR notifying the PR author to also update the relevant release docs. * This action will reuse the script mentioned above to find the diff, if any present I believe, these two options should help solve the current visibility problems in ceph configuration system. Please let us know your thoughts about this feature. All inputs are welcome. Thanks, Naveen _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Thanks for taking the time to looking at the proposal and leaving a feedback Nathan ^^! Regarding the GitHub action, the reason we wanted it to be optional is for two reasons: * Update happens to a field of already existing configuration that does not warrant an entry in release notes, for example: the "description" field. * There may be configuration options that have been introduced only for experimental purpose, that we do not want to expose to the users as of yet. (this maybe a very rare scenario yet imho a valid one) I totally understand the concern that making it non-optional would mean increased chances of missing the documentation of a configuration option, but I think we can reduce the probability of this happening by adding the label (say, "release-note-update-maybe" - I'll come up with better name!) to the PR in addition to the PR comments to make the author and reviewers aware of the extra check of ensuring release notes are updated whenever necessary. Please let me know if you have any other suggestion ^^ Thanks, Naveen ________________________________ From: Nathan Hoad (BLOOMBERG/ 120 PARK) <nhoad@bloomberg.net> Sent: Thursday, March 13, 2025 8:53 PM To: Naveen Naidu <naveen.naidu@ibm.com> Cc: dev@ceph.io <dev@ceph.io> Subject: [EXTERNAL] Re:Ceph Configuration Diff Tool - Request for comments This Message Is From an External Sender This message came from outside your organization. Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/AdhS1Rd-!9_FbsriyHM57WzV5juTOvz0ReJnWOZ9AU22-9qnMX7AmgtK5nh-stQiEBZSu7160I1zjzH_m0KkziOi3sq72Y5pa4w7vExBR$> Hi Naveen, This is a great idea! +1 on both parts of the proposal. This exact problem has definitely caught people off guard in the past when performing upgrades, so having a quick and easy way to verify changes will help reduce some friction for sure :) For the Github action, would you consider making the check non-optional to enforce that the release notes have been updated? I'm trying to imagine a scenario where we would want to change the defaults/add new options and not have that documented and I'm drawing a blank. Thank you, Nathan From: naveen.naidu@ibm.com At: 03/13/25 10:09:34 UTC-4:00 To: dev@ceph.io<mailto:dev@ceph.io> Subject: Ceph Configuration Diff Tool - Request for comments Hi all, We are planning to introduce a new tool/script which would help with the problem of visibility of the changes done to Ceph configuration options. Currently there are two main limitations: * Changes to the configuration are missed in our release notes * No easy way to know what configuration have modified/added/deleted between two releases Some context on ceph configuration management system: The default values for all the configuration options for a ceph cluster when starting up are in the yaml files present in the `src/common/options` directory. For example: `osd.yaml.in` file tells us all the configuration options available for `ceph-osd` daemon. The config options from yaml files are directly injected to the `CephContext` and these files are the sole source of truth to know about the configurations present for all the systems of ceph. It is important to note that, the values in the yaml files only represent the default values with which the cluster starts with, any changes made to these config options are not reflected in the yaml files. The detailed information about the configuration management is present here: https://docs.ceph.com/en/latest/dev/config/<https://docs.ceph.com/en/latest/dev/config/ > Proposal to solve this problem is two fold: 1. Tool to compare configuration options between releases or commit hashes * A script which takes commit hashes or release names and diff the yaml files present in the `src/common/options` directory and output sections like: `Deleted`, `Added` and `Updated` * Developers can run this script whenever they wish to know the configuration diff between two releases * For example: `ceph-config-diff --mode diff-branch --ref-repo <repo-url> --ref-branch main --cmp-branch squid`, a command like this will compare the current main branch with the squid branch. 2. A Github action runs on every PR (it only executes when the yaml files are updated). This will be added as an optional check. * This action will check if any changes have occurred to yaml config files and write a comment to the PR notifying the PR author to also update the relevant release docs. * This action will reuse the script mentioned above to find the diff, if any present I believe, these two options should help solve the current visibility problems in ceph configuration system. Please let us know your thoughts about this feature. All inputs are welcome. Thanks, Naveen _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Thanks for doings this! For the concerns you called out 1. Description field - can we just ignore changes that are only in the Description field of the yaml, programmatically? 2. Experimental fields - should this just be a field/tag in the yaml file itself? Ie mark the field experimental. This will also be useful for docs generation in general? -Joseph From: naveen.naidu@ibm.com At: 03/14/25 04:28:15 UTC-4:00To: Nathan Hoad (BLOOMBERG/ 120 PARK ) Cc: dev@ceph.io Subject: RE:Ceph Configuration Diff Tool - Request for comments Thanks for taking the time to looking at the proposal and leaving a feedback Nathan ^^! Regarding the GitHub action, the reason we wanted it to be optional is for two reasons: *Update happens to a field of already existing configuration that does not warrant an entry in release notes, for example: the "description" field. *There may be configuration options that have been introduced only for experimental purpose, that we do not want to expose to the users as of yet. (this maybe a very rare scenario yet imho a valid one) I totally understand the concern that making it non-optional would mean increased chances of missing the documentation of a configuration option, but I think we can reduce the probability of this happening by adding the label (say, "release-note-update-maybe" - I'll come up with better name!) to the PR in addition to the PR comments to make the author and reviewers aware of the extra check of ensuring release notes are updated whenever necessary. Please let me know if you have any other suggestion ^^ Thanks, Naveen From: Nathan Hoad (BLOOMBERG/ 120 PARK) <nhoad@bloomberg.net> Sent: Thursday, March 13, 2025 8:53 PM To: Naveen Naidu <naveen.naidu@ibm.com> Cc: dev@ceph.io <dev@ceph.io> Subject: [EXTERNAL] Re:Ceph Configuration Diff Tool - Request for comments This Message Is From an External Sender This message came from outside your organization. Report Suspicious Hi Naveen, This is a great idea! +1 on both parts of the proposal. This exact problem has definitely caught people off guard in the past when performing upgrades, so having a quick and easy way to verify changes will help reduce some friction for sure :) For the Github action, would you consider making the check non-optional to enforce that the release notes have been updated? I'm trying to imagine a scenario where we would want to change the defaults/add new options and not have that documented and I'm drawing a blank. Thank you, Nathan From: naveen.naidu@ibm.com At: 03/13/25 10:09:34 UTC-4:00 To: dev@ceph.io Subject: Ceph Configuration Diff Tool - Request for comments Hi all, We are planning to introduce a new tool/script which would help with the problem of visibility of the changes done to Ceph configuration options. Currently there are two main limitations: *Changes to the configuration are missed in our release notes *No easy way to know what configuration have modified/added/deleted between two releases Some context on ceph configuration management system: The default values for all the configuration options for a ceph cluster when starting up are in the yaml files present in the `src/common/options` directory. For example: `osd.yaml.in` file tells us all the configuration options available for `ceph-osd` daemon. The config options from yaml files are directly injected to the `CephContext` and these files are the sole source of truth to know about the configurations present for all the systems of ceph. It is important to note that, the values in the yaml files only represent the default values with which the cluster starts with, any changes made to these config options are not reflected in the yaml files. The detailed information about the configuration management is present here: https://docs.ceph.com/en/latest/dev/config/ Proposal to solve this problem is two fold: 1. Tool to compare configuration options between releases or commit hashes *A script which takes commit hashes or release names and diff the yaml files present in the `src/common/options` directory and output sections like: `Deleted`, `Added` and `Updated` *Developers can run this script whenever they wish to know the configuration diff between two releases *For example: `ceph-config-diff --mode diff-branch --ref-repo <repo-url> --ref-branch main --cmp-branch squid`, a command like this will compare the current main branch with the squid branch. 2. A Github action runs on every PR (it only executes when the yaml files are updated). This will be added as an optional check. *This action will check if any changes have occurred to yaml config files and write a comment to the PR notifying the PR author to also update the relevant release docs. * This action will reuse the script mentioned above to find the diff, if any present I believe, these two options should help solve the current visibility problems in ceph configuration system. Please let us know your thoughts about this feature. All inputs are welcome. Thanks, Naveen _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
That's a great initiative, Naveen! Regarding #2, if I may: - I'd suggest that we add some label (e.g.: "config-change") to those PRs. And we also run this against the last N thousand PRs, so that we also get a retrospective analysis of config changes. - BTW, in the Dashboard team we also have a CI check that adds a label ("api-changes") if a change is made to the OpenAPI spec file. I hope that's useful as a reference. - If the comment added to the PR follows some "structure", that might allow the Pull Request search for fine-grained look-ups. For example, based on a recent change, searching for "osd_scrub_load_threshold = 10" should bring me to this PR <https://github.com/ceph/ceph/pull/61351> where that setting changed from 0.5 to 10 (in this case, the commit/PR message was nicely written and a simple PR search already worked <https://github.com/search?q=repo%3Aceph%2Fceph+osd_scrub_load_threshold+%3D+0.5&type=pullrequests>, but that's not always the case). BTW, maybe not for the first iteration, but it'd be interesting to consider that there are Ceph settings lying outside the `src/common/options/` dir, like the `MODULE_OPTIONS` from the mgr-modules. Kind Regards, Ernesto On Fri, Mar 14, 2025 at 1:36 PM Joseph Mundackal (BLOOMBERG/ 120 PARK) < jmundackal@bloomberg.net> wrote:
Thanks for doings this!
For the concerns you called out 1. Description field - can we just ignore changes that are only in the Description field of the yaml, programmatically? 2. Experimental fields - should this just be a field/tag in the yaml file itself? Ie mark the field experimental. This will also be useful for docs generation in general?
-Joseph
From: naveen.naidu@ibm.com At: 03/14/25 04:28:15 UTC-4:00 To: Nathan Hoad (BLOOMBERG/ 120 PARK ) <nhoad@bloomberg.net> Cc: dev@ceph.io Subject: RE:Ceph Configuration Diff Tool - Request for comments
Thanks for taking the time to looking at the proposal and leaving a feedback Nathan ^^!
Regarding the GitHub action, the reason we wanted it to be optional is for two reasons:
- Update happens to a field of already existing configuration that does not warrant an entry in release notes, for example: the "description" field. - There may be configuration options that have been introduced only for experimental purpose, that we do not want to expose to the users as of yet. (this maybe a very rare scenario yet imho a valid one)
I totally understand the concern that making it non-optional would mean increased chances of missing the documentation of a configuration option, but I think we can reduce the probability of this happening by adding the label (say, "release-note-update-maybe" - I'll come up with better name!) to the PR in addition to the PR comments to make the author and reviewers aware of the extra check of ensuring release notes are updated whenever necessary. Please let me know if you have any other suggestion ^^
Thanks, Naveen
------------------------------ *From:* Nathan Hoad (BLOOMBERG/ 120 PARK) <nhoad@bloomberg.net> *Sent:* Thursday, March 13, 2025 8:53 PM *To:* Naveen Naidu <naveen.naidu@ibm.com> *Cc:* dev@ceph.io <dev@ceph.io> *Subject:* [EXTERNAL] Re:Ceph Configuration Diff Tool - Request for comments
This Message Is From an External Sender This message came from outside your organization. Report Suspicious <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/AdhS1Rd-!9_FbsriyHM57WzV5juTOvz0ReJnWOZ9AU22-9qnMX7AmgtK5nh-stQiEBZSu7160I1zjzH_m0KkziOi3sq72Y5pa4w7vExBR$>
Hi Naveen,
This is a great idea! +1 on both parts of the proposal. This exact problem has definitely caught people off guard in the past when performing upgrades, so having a quick and easy way to verify changes will help reduce some friction for sure :)
For the Github action, would you consider making the check non-optional to enforce that the release notes have been updated? I'm trying to imagine a scenario where we would want to change the defaults/add new options and not have that documented and I'm drawing a blank.
Thank you,
Nathan
From: naveen.naidu@ibm.com At: 03/13/25 10:09:34 UTC-4:00 To: dev@ceph.io Subject: Ceph Configuration Diff Tool - Request for comments
Hi all,
We are planning to introduce a new tool/script which would help with the problem of visibility of the changes done to Ceph configuration options. Currently there are two main limitations:
- Changes to the configuration are missed in our release notes - No easy way to know what configuration have modified/added/deleted between two releases
Some context on ceph configuration management system:
The default values for all the configuration options for a ceph cluster when starting up are in the yaml files present in the `src/common/options` directory.
For example: `osd.yaml.in` file tells us all the configuration options available for `ceph-osd` daemon.
The config options from yaml files are directly injected to the `CephContext` and these files are the sole source of truth to know about the configurations present for all the systems of ceph. It is important to note that, the values in the yaml files only represent the default values with which the cluster starts with, any changes made to these config options are not reflected in the yaml files.
The detailed information about the configuration management is present here: https://docs.ceph.com/en/latest/dev/config/
Proposal to solve this problem is two fold:
1. Tool to compare configuration options between releases or commit hashes
- A script which takes commit hashes or release names and diff the yaml files present in the `src/common/options` directory and output sections like: `Deleted`, `Added` and `Updated` - Developers can run this script whenever they wish to know the configuration diff between two releases - For example: `ceph-config-diff --mode diff-branch --ref-repo <repo-url> --ref-branch main --cmp-branch squid`, a command like this will compare the current main branch with the squid branch.
2. A Github action runs on every PR (it only executes when the yaml files are updated). This will be added as an optional check.
- This action will check if any changes have occurred to yaml config files and write a comment to the PR notifying the PR author to also update the relevant release docs. - This action will reuse the script mentioned above to find the diff, if any present
I believe, these two options should help solve the current visibility problems in ceph configuration system.
Please let us know your thoughts about this feature. All inputs are welcome.
Thanks, Naveen
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Thanks for comment Ernesto!, The idea about using "config-change" label to do a retrospective analysis of config changes did not cross my mind. This is an amazing use case. The reference to the "api-change" label is very helpful. Regarding the format of the comment. A sample comment format looks like below, it's a simple JSON format so it should be very easy to search for the keys that are suspected to have their configuration changed: { "new": { "mgr.yaml.in": [ "mon_warn_on_pool_no_app_grace", "mgr_max_pg_creating" ], ... }, "deleted": { "mon.yaml.in": [ "mon_osd_max_creating_pgs" ], ... }, "modified": { "osd_scrub_load_threshold": { "default": { "before": 0.5, "after": 10.0 }, ... }, } } Thanks, Naveen ________________________________ From: Ernesto Puerta <epuertat@redhat.com> Sent: Tuesday, March 18, 2025 9:44 PM To: Naveen Naidu <naveen.naidu@ibm.com> Cc: dev@ceph.io <dev@ceph.io> Subject: [EXTERNAL] Re: Ceph Configuration Diff Tool - Request for comments That's a great initiative, Naveen! Regarding #2, if I may: I'd suggest that we add some label (e. g. : "config-change") to those PRs. And we also run this against the last N thousand PRs, so that we also get a retrospective analysis That's a great initiative, Naveen! Regarding #2, if I may: * I'd suggest that we add some label (e.g.: "config-change") to those PRs. And we also run this against the last N thousand PRs, so that we also get a retrospective analysis of config changes. * BTW, in the Dashboard team we also have a CI check that adds a label ("api-changes") if a change is made to the OpenAPI spec file. I hope that's useful as a reference. * If the comment added to the PR follows some "structure", that might allow the Pull Request search for fine-grained look-ups. For example, based on a recent change, searching for "osd_scrub_load_threshold = 10" should bring me to this PR<https://github.com/ceph/ceph/pull/61351 > where that setting changed from 0.5 to 10 (in this case, the commit/PR message was nicely written and a simple PR search already worked<https://github.com/search?q=repo%3Aceph%2Fceph+osd_scrub_load_threshold+%3D+0.5&type=pullrequests >, but that's not always the case). BTW, maybe not for the first iteration, but it'd be interesting to consider that there are Ceph settings lying outside the `src/common/options/` dir, like the `MODULE_OPTIONS` from the mgr-modules. Kind Regards, Ernesto On Fri, Mar 14, 2025 at 1:36 PM Joseph Mundackal (BLOOMBERG/ 120 PARK) <jmundackal@bloomberg.net<mailto:jmundackal@bloomberg.net>> wrote: Thanks for doings this! For the concerns you called out 1. Description field - can we just ignore changes that are only in the Description field of the yaml, programmatically? 2. Experimental fields - should this just be a field/tag in the yaml file itself? Ie mark the field experimental. This will also be useful for docs generation in general? -Joseph From: naveen.naidu@ibm.com<mailto:naveen.naidu@ibm.com> At: 03/14/25 04:28:15 UTC-4:00 To: Nathan Hoad (BLOOMBERG/ 120 PARK ) <mailto:nhoad@bloomberg.net> Cc: dev@ceph.io<mailto:dev@ceph.io> Subject: RE:Ceph Configuration Diff Tool - Request for comments Thanks for taking the time to looking at the proposal and leaving a feedback Nathan ^^! Regarding the GitHub action, the reason we wanted it to be optional is for two reasons: * Update happens to a field of already existing configuration that does not warrant an entry in release notes, for example: the "description" field. * There may be configuration options that have been introduced only for experimental purpose, that we do not want to expose to the users as of yet. (this maybe a very rare scenario yet imho a valid one) I totally understand the concern that making it non-optional would mean increased chances of missing the documentation of a configuration option, but I think we can reduce the probability of this happening by adding the label (say, "release-note-update-maybe" - I'll come up with better name!) to the PR in addition to the PR comments to make the author and reviewers aware of the extra check of ensuring release notes are updated whenever necessary. Please let me know if you have any other suggestion ^^ Thanks, Naveen ________________________________ From: Nathan Hoad (BLOOMBERG/ 120 PARK) <nhoad@bloomberg.net<mailto:nhoad@bloomberg.net>> Sent: Thursday, March 13, 2025 8:53 PM To: Naveen Naidu <naveen.naidu@ibm.com<mailto:naveen.naidu@ibm.com>> Cc: dev@ceph.io<mailto:dev@ceph.io> <dev@ceph.io<mailto:dev@ceph.io>> Subject: [EXTERNAL] Re:Ceph Configuration Diff Tool - Request for comments This Message Is From an External Sender This message came from outside your organization. Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/AdhS1Rd-!9_FbsriyHM57WzV5juTOvz0ReJnWOZ9AU22-9qnMX7AmgtK5nh-stQiEBZSu7160I1zjzH_m0KkziOi3sq72Y5pa4w7vExBR$> Hi Naveen, This is a great idea! +1 on both parts of the proposal. This exact problem has definitely caught people off guard in the past when performing upgrades, so having a quick and easy way to verify changes will help reduce some friction for sure :) For the Github action, would you consider making the check non-optional to enforce that the release notes have been updated? I'm trying to imagine a scenario where we would want to change the defaults/add new options and not have that documented and I'm drawing a blank. Thank you, Nathan From: naveen.naidu@ibm.com<mailto:naveen.naidu@ibm.com> At: 03/13/25 10:09:34 UTC-4:00 To: dev@ceph.io<mailto:dev@ceph.io> Subject: Ceph Configuration Diff Tool - Request for comments Hi all, We are planning to introduce a new tool/script which would help with the problem of visibility of the changes done to Ceph configuration options. Currently there are two main limitations: * Changes to the configuration are missed in our release notes * No easy way to know what configuration have modified/added/deleted between two releases Some context on ceph configuration management system: The default values for all the configuration options for a ceph cluster when starting up are in the yaml files present in the `src/common/options` directory. For example: `osd.yaml.in<http://osd.yaml.in >` file tells us all the configuration options available for `ceph-osd` daemon. The config options from yaml files are directly injected to the `CephContext` and these files are the sole source of truth to know about the configurations present for all the systems of ceph. It is important to note that, the values in the yaml files only represent the default values with which the cluster starts with, any changes made to these config options are not reflected in the yaml files. The detailed information about the configuration management is present here: https://docs.ceph.com/en/latest/dev/config/<https://docs.ceph.com/en/latest/dev/config/ > Proposal to solve this problem is two fold: 1. Tool to compare configuration options between releases or commit hashes * A script which takes commit hashes or release names and diff the yaml files present in the `src/common/options` directory and output sections like: `Deleted`, `Added` and `Updated` * Developers can run this script whenever they wish to know the configuration diff between two releases * For example: `ceph-config-diff --mode diff-branch --ref-repo <repo-url> --ref-branch main --cmp-branch squid`, a command like this will compare the current main branch with the squid branch. 2. A Github action runs on every PR (it only executes when the yaml files are updated). This will be added as an optional check. * This action will check if any changes have occurred to yaml config files and write a comment to the PR notifying the PR author to also update the relevant release docs. * This action will reuse the script mentioned above to find the diff, if any present I believe, these two options should help solve the current visibility problems in ceph configuration system. Please let us know your thoughts about this feature. All inputs are welcome. Thanks, Naveen _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io> _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io> _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io>
Those are great suggestions Joseph! After thinking about it, the plan would be to eventually make the GitHub Action as non-optional. For the first cut, it would still be an optional check since it requires greater effort to implement these suggestions. Currently there is no easy way to mark an option as experimental fields. Rest assured though, as soon as the first cut is out - I will be working on improving it and also make the GH action as required. Thanks, Naveen ________________________________ From: Joseph Mundackal (BLOOMBERG/ 120 PARK) <jmundackal@bloomberg.net> Sent: Friday, March 14, 2025 6:05 PM To: Naveen Naidu <naveen.naidu@ibm.com> Cc: dev@ceph.io <dev@ceph.io> Subject: [EXTERNAL] RE:Ceph Configuration Diff Tool - Request for comments This Message Is From an External Sender This message came from outside your organization. Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/AdhS1Rd-!9_Fbsriy0k8625V5juSBkclWGUBMySn_uVeumUCQw5aoNn2K9UNkGp-4-12AGbUiC64x8t7hSo2FlwvDZV2J6N5EAgM9I6Wu$> Thanks for doings this! For the concerns you called out 1. Description field - can we just ignore changes that are only in the Description field of the yaml, programmatically? 2. Experimental fields - should this just be a field/tag in the yaml file itself? Ie mark the field experimental. This will also be useful for docs generation in general? -Joseph From: naveen.naidu@ibm.com At: 03/14/25 04:28:15 UTC-4:00 To: Nathan Hoad (BLOOMBERG/ 120 PARK ) <mailto:nhoad@bloomberg.net> Cc: dev@ceph.io<mailto:dev@ceph.io> Subject: RE:Ceph Configuration Diff Tool - Request for comments Thanks for taking the time to looking at the proposal and leaving a feedback Nathan ^^! Regarding the GitHub action, the reason we wanted it to be optional is for two reasons: * Update happens to a field of already existing configuration that does not warrant an entry in release notes, for example: the "description" field. * There may be configuration options that have been introduced only for experimental purpose, that we do not want to expose to the users as of yet. (this maybe a very rare scenario yet imho a valid one) I totally understand the concern that making it non-optional would mean increased chances of missing the documentation of a configuration option, but I think we can reduce the probability of this happening by adding the label (say, "release-note-update-maybe" - I'll come up with better name!) to the PR in addition to the PR comments to make the author and reviewers aware of the extra check of ensuring release notes are updated whenever necessary. Please let me know if you have any other suggestion ^^ Thanks, Naveen ________________________________ From: Nathan Hoad (BLOOMBERG/ 120 PARK) <nhoad@bloomberg.net> Sent: Thursday, March 13, 2025 8:53 PM To: Naveen Naidu <naveen.naidu@ibm.com> Cc: dev@ceph.io <dev@ceph.io> Subject: [EXTERNAL] Re:Ceph Configuration Diff Tool - Request for comments This Message Is From an External Sender This message came from outside your organization. Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/AdhS1Rd-!9_FbsriyHM57WzV5juTOvz0ReJnWOZ9AU22-9qnMX7AmgtK5nh-stQiEBZSu7160I1zjzH_m0KkziOi3sq72Y5pa4w7vExBR$> Hi Naveen, This is a great idea! +1 on both parts of the proposal. This exact problem has definitely caught people off guard in the past when performing upgrades, so having a quick and easy way to verify changes will help reduce some friction for sure :) For the Github action, would you consider making the check non-optional to enforce that the release notes have been updated? I'm trying to imagine a scenario where we would want to change the defaults/add new options and not have that documented and I'm drawing a blank. Thank you, Nathan From: naveen.naidu@ibm.com At: 03/13/25 10:09:34 UTC-4:00 To: dev@ceph.io<mailto:dev@ceph.io> Subject: Ceph Configuration Diff Tool - Request for comments Hi all, We are planning to introduce a new tool/script which would help with the problem of visibility of the changes done to Ceph configuration options. Currently there are two main limitations: * Changes to the configuration are missed in our release notes * No easy way to know what configuration have modified/added/deleted between two releases Some context on ceph configuration management system: The default values for all the configuration options for a ceph cluster when starting up are in the yaml files present in the `src/common/options` directory. For example: `osd.yaml.in` file tells us all the configuration options available for `ceph-osd` daemon. The config options from yaml files are directly injected to the `CephContext` and these files are the sole source of truth to know about the configurations present for all the systems of ceph. It is important to note that, the values in the yaml files only represent the default values with which the cluster starts with, any changes made to these config options are not reflected in the yaml files. The detailed information about the configuration management is present here: https://docs.ceph.com/en/latest/dev/config/<https://docs.ceph.com/en/latest/dev/config/ > Proposal to solve this problem is two fold: 1. Tool to compare configuration options between releases or commit hashes * A script which takes commit hashes or release names and diff the yaml files present in the `src/common/options` directory and output sections like: `Deleted`, `Added` and `Updated` * Developers can run this script whenever they wish to know the configuration diff between two releases * For example: `ceph-config-diff --mode diff-branch --ref-repo <repo-url> --ref-branch main --cmp-branch squid`, a command like this will compare the current main branch with the squid branch. 2. A Github action runs on every PR (it only executes when the yaml files are updated). This will be added as an optional check. * This action will check if any changes have occurred to yaml config files and write a comment to the PR notifying the PR author to also update the relevant release docs. * This action will reuse the script mentioned above to find the diff, if any present I believe, these two options should help solve the current visibility problems in ceph configuration system. Please let us know your thoughts about this feature. All inputs are welcome. Thanks, Naveen _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Naveen, This is an excellent idea and the config-diff tool would definitely be very useful to many operators! For the ceph-config-diff tool, it would be cool if it accepts also git version tags in addition to branches and release names. From personal experience, it would be of great help to know if defaults have changed between the version currently running on my cluster and the one I target for update. Also, the fact that defaults are now nicely scoped in to YAMLs for each daemon may further help to draw the attention where needed (e.g., --daemon osd). My 2 cents ;) Cheers, Enrico On 3/13/25 15:09, Naveen Naidu wrote:
Hi all,
We are planning to introduce a new tool/script which would help with the problem of visibility of the changes done to Ceph configuration options. Currently there are two main limitations:
* Changes to the configuration are missed in our release notes * No easy way to know what configuration have modified/added/deleted between two releases
Some context on ceph configuration management system: The default values for all the configuration options for a ceph cluster when starting up are in the yaml files present in the `src/common/options` directory.
For example: `osd.yaml.in` file tells us all the configuration options available for `ceph-osd` daemon.
The config options from yaml files are directly injected to the `CephContext` and these files are the sole source of truth to know about the configurations present for all the systems of ceph. It is important to note that, the values in the yaml files only represent the default values with which the cluster starts with, any changes made to these config options are not reflected in the yaml files.
The detailed information about the configuration management is present here: https://docs.ceph.com/en/latest/dev/config/ <https://docs.ceph.com/en/latest/dev/config/>
Proposal to solve this problem is two fold:
1. Tool to compare configuration options between releases or commit hashes
* A script which takes commit hashes or release names and diff the yaml files present in the `src/common/options` directory and output sections like: `Deleted`, `Added` and `Updated` * Developers can run this script whenever they wish to know the configuration diff between two releases * For example: `ceph-config-diff --mode diff-branch --ref-repo <repo-url> --ref-branch main --cmp-branch squid`, a command like this will compare the current main branch with the squid branch.
2. A Github action runs on every PR (it only executes when the yaml files are updated). This will be added as an optional check.
* This action will check if any changes have occurred to yaml config files and write a comment to the PR notifying the PR author to also update the relevant release docs. * This action will reuse the script mentioned above to find the diff, if any present
I believe, these two options should help solve the current visibility problems in ceph configuration system.
Please let us know your thoughts about this feature. All inputs are welcome.
Thanks, Naveen
_______________________________________________ Dev mailing list --dev@ceph.io To unsubscribe send an email todev-leave@ceph.io
-- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland
Hi Enrico, Thanks for the amazing suggestions, really appreciate it - they should be easily doable ^^. I'll add the following two suggestions to the scope of the tool: * diff between git version tags * ability to filter the config changes to a certain daemon via `—daemon` flag Thanks, Naveen ________________________________ From: Enrico Bocchi <enrico.bocchi@cern.ch> Sent: Thursday, March 13, 2025 9:18 PM To: Naveen Naidu <naveen.naidu@ibm.com>; dev@ceph.io <dev@ceph.io> Subject: [EXTERNAL] Re: Ceph Configuration Diff Tool - Request for comments This Message Is From an External Sender This message came from outside your organization. Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/AdhS1Rd-!9DFaWJwSMy_R37bZrgSAMn-U4_taH1z-idDg85Hki6Fm3VjZLDPh_9k5fWVoY7cQHZ-Q06HHT_iB_ASrDNz59Br25jFTCg$> Hi Naveen, This is an excellent idea and the config-diff tool would definitely be very useful to many operators! For the ceph-config-diff tool, it would be cool if it accepts also git version tags in addition to branches and release names. From personal experience, it would be of great help to know if defaults have changed between the version currently running on my cluster and the one I target for update. Also, the fact that defaults are now nicely scoped in to YAMLs for each daemon may further help to draw the attention where needed (e.g., --daemon osd). My 2 cents ;) Cheers, Enrico On 3/13/25 15:09, Naveen Naidu wrote: Hi all, We are planning to introduce a new tool/script which would help with the problem of visibility of the changes done to Ceph configuration options. Currently there are two main limitations: * Changes to the configuration are missed in our release notes * No easy way to know what configuration have modified/added/deleted between two releases Some context on ceph configuration management system: The default values for all the configuration options for a ceph cluster when starting up are in the yaml files present in the `src/common/options` directory. For example: `osd.yaml.in` file tells us all the configuration options available for `ceph-osd` daemon. The config options from yaml files are directly injected to the `CephContext` and these files are the sole source of truth to know about the configurations present for all the systems of ceph. It is important to note that, the values in the yaml files only represent the default values with which the cluster starts with, any changes made to these config options are not reflected in the yaml files. The detailed information about the configuration management is present here: https://docs.ceph.com/en/latest/dev/config/<https://docs.ceph.com/en/latest/dev/config/ > Proposal to solve this problem is two fold: 1. Tool to compare configuration options between releases or commit hashes * A script which takes commit hashes or release names and diff the yaml files present in the `src/common/options` directory and output sections like: `Deleted`, `Added` and `Updated` * Developers can run this script whenever they wish to know the configuration diff between two releases * For example: `ceph-config-diff --mode diff-branch --ref-repo <repo-url> --ref-branch main --cmp-branch squid`, a command like this will compare the current main branch with the squid branch. 2. A Github action runs on every PR (it only executes when the yaml files are updated). This will be added as an optional check. * This action will check if any changes have occurred to yaml config files and write a comment to the PR notifying the PR author to also update the relevant release docs. * This action will reuse the script mentioned above to find the diff, if any present I believe, these two options should help solve the current visibility problems in ceph configuration system. Please let us know your thoughts about this feature. All inputs are welcome. Thanks, Naveen _______________________________________________ Dev mailing list -- dev@ceph.io<mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io<mailto:dev-leave@ceph.io> -- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage & Data Management - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland
participants (5)
-
Enrico Bocchi
-
Ernesto Puerta
-
Joseph Mundackal (BLOOMBERG/ 120 PARK)
-
Nathan Hoad (BLOOMBERG/ 120 PARK)
-
Naveen Naidu