smartctl failed with error -22
Hi, On a new cluster with version 19.2.3 the device health metrics only show a smartctl error: { "20250821-000313": { "dev": "/dev/sdc", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 1", "nvme_smart_health_information_add_log_error_code": -22, "nvme_vendor": "ata", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\n" } } The device in question (like all the other in the cluster) is a Samsung MZ7L37T6 SATA SSD. What is happening here? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi Robert, I'm not an expert on the low-level details and "modern" Ceph, so I hope I don't lead you on any wild goose chases, but I might at least give some leads. It seems odd that the metrics mention NVM/e - I'm guessing that it's just a cross-product test and tries all tools on all devices. SMART test failure is more of an issue. It's a pity the error message is so nondescript. Some things I can think of from simplest to most complicated are: * Are smartmontools installed on the drive host? * Does the monitoring UID have sudo access? * Does a manual "sudo smartctl -a /dev/sdc" give the same or similar result? * Is the drive managed by a hardware RAID controller or concentrator (Like Dell PERC or a USB adapter or something) * (This is a stretch) Is there an OSD for the drive that's given the "NVME" class? Hope that gives you something. M0les. On Thu, 21 Aug 2025, at 17:15, Robert Sander wrote:
Hi,
On a new cluster with version 19.2.3 the device health metrics only show a smartctl error:
{ "20250821-000313": { "dev": "/dev/sdc", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 1", "nvme_smart_health_information_add_log_error_code": -22, "nvme_vendor": "ata", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\n" } }
The device in question (like all the other in the cluster) is a Samsung MZ7L37T6 SATA SSD.
What is happening here?
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Am 8/21/25 um 10:07 schrieb Miles Goodhew:
It seems odd that the metrics mention NVM/e - I'm guessing that it's just a cross-product test and tries all tools on all devices. SMART test failure is more of an issue. It's a pity the error message is so nondescript. Some things I can think of from simplest to most complicated are: * Are smartmontools installed on the drive host? * Does the monitoring UID have sudo access?
This is not under our control as this is a containerized installation. AFAICS smartmontools are available inside the container.
* Does a manual "sudo smartctl -a /dev/sdc" give the same or similar result?
This works outside the container on the host.
* Is the drive managed by a hardware RAID controller or concentrator (Like Dell PERC or a USB adapter or something)
Yes, but as smartctl gives us values the controller seems not to be the issue. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
On Aug 21, 2025, at 4:07 AM, Miles Goodhew <ceph@m0les.com> wrote:
Hi Robert, I'm not an expert on the low-level details and "modern" Ceph, so I hope I don't lead you on any wild goose chases, but I might at least give some leads. It seems odd that the metrics mention NVM/e - I'm guessing that it's just a cross-product test and tries all tools on all devices.
Recent releases of smartctl pass through stats for NVMe devices via the name-cli command "nvme". Whether it invokes that for all devices, ordering, etc I don't know.
SMART test failure is more of an issue. It's a pity the error message is so nondescript. Some things I can think of from simplest to most complicated are: * Are smartmontools installed on the drive host?
Does it happen with other drives on the same host? If you have availability through your chassis vendor, look for a firmware update.
* Does the monitoring UID have sudo access? * Does a manual "sudo smartctl -a /dev/sdc" give the same or similar result? * Is the drive managed by a hardware RAID controller or concentrator (Like Dell PERC or a USB adapter or something) * (This is a stretch) Is there an OSD for the drive that's given the "NVME" class?
Hope that gives you something.
M0les.
On Thu, 21 Aug 2025, at 17:15, Robert Sander wrote:
Hi,
On a new cluster with version 19.2.3 the device health metrics only show a smartctl error:
{ "20250821-000313": { "dev": "/dev/sdc", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 1", "nvme_smart_health_information_add_log_error_code": -22, "nvme_vendor": "ata", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\n" } }
The device in question (like all the other in the cluster) is a Samsung MZ7L37T6 SATA SSD.
What is happening here?
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I rely on SMART for 2 things: 1. Repeatedly sending me messages about nonfatal bad sectors that no one seems to know how to correct for. 2. Not saying anything before a device crashes. Yeah. But I run it anyway because you never know. The reported error is too far abstracted from the actual failure and I cannot find anything about -22 as a SMART result code. *n*x errno 22 is EINVAL, which seems unlikely, but it is possible that smartd got misconfigured. Run smartctl -l /dev/sdc to launch an out-of-band long test. When it is done, use smartctl to report the results and see if anything is flagged. On 8/21/25 09:10, Anthony D'Atri wrote:
On Aug 21, 2025, at 4:07 AM, Miles Goodhew <ceph@m0les.com> wrote:
Hi Robert, I'm not an expert on the low-level details and "modern" Ceph, so I hope I don't lead you on any wild goose chases, but I might at least give some leads. It seems odd that the metrics mention NVM/e - I'm guessing that it's just a cross-product test and tries all tools on all devices. Recent releases of smartctl pass through stats for NVMe devices via the name-cli command "nvme". Whether it invokes that for all devices, ordering, etc I don't know.
SMART test failure is more of an issue. It's a pity the error message is so nondescript. Some things I can think of from simplest to most complicated are: * Are smartmontools installed on the drive host? Does it happen with other drives on the same host?
If you have availability through your chassis vendor, look for a firmware update.
* Does the monitoring UID have sudo access? * Does a manual "sudo smartctl -a /dev/sdc" give the same or similar result? * Is the drive managed by a hardware RAID controller or concentrator (Like Dell PERC or a USB adapter or something) * (This is a stretch) Is there an OSD for the drive that's given the "NVME" class?
Hope that gives you something.
M0les.
On Thu, 21 Aug 2025, at 17:15, Robert Sander wrote:
Hi,
On a new cluster with version 19.2.3 the device health metrics only show a smartctl error:
{ "20250821-000313": { "dev": "/dev/sdc", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 1", "nvme_smart_health_information_add_log_error_code": -22, "nvme_vendor": "ata", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\n" } }
The device in question (like all the other in the cluster) is a Samsung MZ7L37T6 SATA SSD.
What is happening here?
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I rely on SMART for 2 things:
Bit of nomenclature clarification: SMART is a mechanism for interacting with storage devices, mostly but not exclusively reading status and metrics smartctl is a CLI utility smartd is a daemon smartmontools is a package that includes smartctl and smartd
1. Repeatedly sending me messages about nonfatal bad sectors that no one seems to know how to correct for.
That sounds like you have smartd running and configured to send email? I personally haven't found value in smartd; ymmv. Drives sometimes encounter failed writes that result in a grown defect and mapping of the subject LBA. When that LBA is written again, it generally succeeds. Since Nautilus the OSD will retry a certain number of writes, and as a result we see a lot fewer inconsistent PGs than we used to. When a drive reports grown errors, those are worth tracking. Every drive has some number of factory bad blocks that are remapped out of the box. A portion of individual HDDs will develop additional grown defects over its lifetime. A few of these are not cause for alarm, a lot of them IMHO is cause to replace the drive. The threshold for "a lot" is not clear, perhaps somewhere in the 5-10 range? SSDs will report reallocated blocks and in some cases the numbers of spares used/remaining. It is worth tracking these and alerting if a drive is running short on spares or experiences a high rate of new reallocated blocks.
2. Not saying anything before a device crashes.
Yeah. But I run it anyway because you never know.
The overall health status? Yeah that usually has limited utility. I've seen plenty of drives that are clearly throwing errors yet report healthy. Arguably a firmware bug.
The reported error is too far abstracted from the actual failure and I cannot find anything about -22 as a SMART result code.
I'm pretty sure that's an errno number for launching the subprocess, nothing to do with SMART itself. I'd check dmesg and /var/log/{messages, syslog} to see if anything was reported for that drive. If the drive is SAS/SATA and hung off an LSI HBA, also try `storcli64 /c0 show termlog >/var/tmp/termlog.txt`
*n*x errno 22 is EINVAL, which seems unlikely, but it is possible that smartd got misconfigured.
Best configuration IMHO is to stop, disable, and mask the service.
Run smartctl -l /dev/sdc to launch an out-of-band long test. When it is done, use smartctl to report the results and see if anything is flagged.
On 8/21/25 09:10, Anthony D'Atri wrote:
On Aug 21, 2025, at 4:07 AM, Miles Goodhew <ceph@m0les.com> wrote:
Hi Robert, I'm not an expert on the low-level details and "modern" Ceph, so I hope I don't lead you on any wild goose chases, but I might at least give some leads. It seems odd that the metrics mention NVM/e - I'm guessing that it's just a cross-product test and tries all tools on all devices. Recent releases of smartctl pass through stats for NVMe devices via the name-cli command "nvme". Whether it invokes that for all devices, ordering, etc I don't know.
SMART test failure is more of an issue. It's a pity the error message is so nondescript. Some things I can think of from simplest to most complicated are: * Are smartmontools installed on the drive host? Does it happen with other drives on the same host?
If you have availability through your chassis vendor, look for a firmware update.
* Does the monitoring UID have sudo access? * Does a manual "sudo smartctl -a /dev/sdc" give the same or similar result? * Is the drive managed by a hardware RAID controller or concentrator (Like Dell PERC or a USB adapter or something) * (This is a stretch) Is there an OSD for the drive that's given the "NVME" class?
Hope that gives you something.
M0les.
On Thu, 21 Aug 2025, at 17:15, Robert Sander wrote:
Hi,
On a new cluster with version 19.2.3 the device health metrics only show a smartctl error:
{ "20250821-000313": { "dev": "/dev/sdc", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 1", "nvme_smart_health_information_add_log_error_code": -22, "nvme_vendor": "ata", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\n" } }
The device in question (like all the other in the cluster) is a Samsung MZ7L37T6 SATA SSD.
What is happening here?
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
As I said, I rely on SMART for 2 things. Alas. My servers all run SMART (smartd) on all storage devices and report to Nagios, which then spreads the joy. I generally run smartctl when I first install a drive just to make sure there are no surprises. Some of these drives have been moved around to accommodate local conditions so they're not always factory-fresh. Traditionally, a hard drive would remap factory-defective sectors before sale. That would make them effectively invisible for practical computing use. Within the Linux OS environment you can run badblocks to map post-sale defective sectors. As far as I know, the hardware sector remap has no standard utility available to add failing sectors and the badblock list is something that would have to be used by the specific filesystem driver in question. By all appearances, SMART continues to test all sectors, regardless of OS usage, since it runs internal to the drive itself and thus knows nothing of the OS. So it would flag sectors that have been walled off from actual use. So far, the only SMART errors I've been annoyed by have been in OS partitions and never on an OSD, so I don't know just how closely SMART and Ceph interact. My bad sector messages (usually no more than 3 for a given drive) typically run for up to 3 years without further degradation and since I have both robust data storage (e.g., Ceph) and robust backups, I'm happy to run drives into the ground. It's not like I'm being paid a handsome sum to keep everything flawless. Tim On Thu, 2025-08-21 at 16:04 -0400, Anthony D'Atri wrote:
I rely on SMART for 2 things:
Bit of nomenclature clarification:
SMART is a mechanism for interacting with storage devices, mostly but not exclusively reading status and metrics smartctl is a CLI utility smartd is a daemon smartmontools is a package that includes smartctl and smartd
1. Repeatedly sending me messages about nonfatal bad sectors that no one seems to know how to correct for.
That sounds like you have smartd running and configured to send email? I personally haven't found value in smartd; ymmv.
Drives sometimes encounter failed writes that result in a grown defect and mapping of the subject LBA. When that LBA is written again, it generally succeeds. Since Nautilus the OSD will retry a certain number of writes, and as a result we see a lot fewer inconsistent PGs than we used to.
When a drive reports grown errors, those are worth tracking. Every drive has some number of factory bad blocks that are remapped out of the box. A portion of individual HDDs will develop additional grown defects over its lifetime. A few of these are not cause for alarm, a lot of them IMHO is cause to replace the drive. The threshold for "a lot" is not clear, perhaps somewhere in the 5-10 range?
SSDs will report reallocated blocks and in some cases the numbers of spares used/remaining. It is worth tracking these and alerting if a drive is running short on spares or experiences a high rate of new reallocated blocks.
2. Not saying anything before a device crashes.
Yeah. But I run it anyway because you never know.
The overall health status? Yeah that usually has limited utility. I've seen plenty of drives that are clearly throwing errors yet report healthy. Arguably a firmware bug.
The reported error is too far abstracted from the actual failure and I cannot find anything about -22 as a SMART result code.
I'm pretty sure that's an errno number for launching the subprocess, nothing to do with SMART itself. I'd check dmesg and /var/log/{messages, syslog} to see if anything was reported for that drive. If the drive is SAS/SATA and hung off an LSI HBA, also try `storcli64 /c0 show termlog >/var/tmp/termlog.txt`
*n*x errno 22 is EINVAL, which seems unlikely, but it is possible that smartd got misconfigured.
Best configuration IMHO is to stop, disable, and mask the service.
Run smartctl -l /dev/sdc to launch an out-of-band long test. When it is done, use smartctl to report the results and see if anything is flagged.
On 8/21/25 09:10, Anthony D'Atri wrote:
On Aug 21, 2025, at 4:07 AM, Miles Goodhew <ceph@m0les.com> wrote:
Hi Robert, I'm not an expert on the low-level details and "modern" Ceph, so I hope I don't lead you on any wild goose chases, but I might at least give some leads. It seems odd that the metrics mention NVM/e - I'm guessing that it's just a cross-product test and tries all tools on all devices. Recent releases of smartctl pass through stats for NVMe devices via the name-cli command "nvme". Whether it invokes that for all devices, ordering, etc I don't know.
SMART test failure is more of an issue. It's a pity the error message is so nondescript. Some things I can think of from simplest to most complicated are: * Are smartmontools installed on the drive host? Does it happen with other drives on the same host?
If you have availability through your chassis vendor, look for a firmware update.
* Does the monitoring UID have sudo access? * Does a manual "sudo smartctl -a /dev/sdc" give the same or similar result? * Is the drive managed by a hardware RAID controller or concentrator (Like Dell PERC or a USB adapter or something) * (This is a stretch) Is there an OSD for the drive that's given the "NVME" class?
Hope that gives you something.
M0les.
On Thu, 21 Aug 2025, at 17:15, Robert Sander wrote:
Hi,
On a new cluster with version 19.2.3 the device health metrics only show a smartctl error:
{ "20250821-000313": { "dev": "/dev/sdc", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 1", "nvme_smart_health_information_add_log_error_code": - 22, "nvme_vendor": "ata", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\n" } }
The device in question (like all the other in the cluster) is a Samsung MZ7L37T6 SATA SSD.
What is happening here?
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
As far as I know, the hardware sector remap has no standard utility available to add failing sectors and the badblock list is something that would have to be used by the specific filesystem driver in question.
Modern drives generally do this for us, unlike the C/H/S days of e.g. SMD where one would manually slip a new offender.
By all appearances, SMART continues to test all sectors, regardless of OS usage, since it runs internal to the drive itself and thus knows nothing of the OS. So it would flag sectors that have been walled off from actual use.
Same with an HBA patrol read.
So far, the only SMART errors I've been annoyed by have been in OS partitions and never on an OSD, so I don't know just how closely SMART and Ceph interact.
My bad sector messages (usually no more than 3 for a given drive) typically run for up to 3 years without further degradation and since I have both robust data storage (e.g., Ceph) and robust backups, I'm happy to run drives into the ground. It's not like I'm being paid a handsome sum to keep everything flawless.
I agree that 3 grown defects is nothing to get excited about. Maybe an action threshold would be 1 per TB?
Tim
On Thu, 2025-08-21 at 16:04 -0400, Anthony D'Atri wrote:
I rely on SMART for 2 things:
Bit of nomenclature clarification:
SMART is a mechanism for interacting with storage devices, mostly but not exclusively reading status and metrics smartctl is a CLI utility smartd is a daemon smartmontools is a package that includes smartctl and smartd
1. Repeatedly sending me messages about nonfatal bad sectors that no one seems to know how to correct for.
That sounds like you have smartd running and configured to send email? I personally haven't found value in smartd; ymmv.
Drives sometimes encounter failed writes that result in a grown defect and mapping of the subject LBA. When that LBA is written again, it generally succeeds. Since Nautilus the OSD will retry a certain number of writes, and as a result we see a lot fewer inconsistent PGs than we used to.
When a drive reports grown errors, those are worth tracking. Every drive has some number of factory bad blocks that are remapped out of the box. A portion of individual HDDs will develop additional grown defects over its lifetime. A few of these are not cause for alarm, a lot of them IMHO is cause to replace the drive. The threshold for "a lot" is not clear, perhaps somewhere in the 5-10 range?
SSDs will report reallocated blocks and in some cases the numbers of spares used/remaining. It is worth tracking these and alerting if a drive is running short on spares or experiences a high rate of new reallocated blocks.
2. Not saying anything before a device crashes.
Yeah. But I run it anyway because you never know.
The overall health status? Yeah that usually has limited utility. I've seen plenty of drives that are clearly throwing errors yet report healthy. Arguably a firmware bug.
The reported error is too far abstracted from the actual failure and I cannot find anything about -22 as a SMART result code.
I'm pretty sure that's an errno number for launching the subprocess, nothing to do with SMART itself. I'd check dmesg and /var/log/{messages, syslog} to see if anything was reported for that drive. If the drive is SAS/SATA and hung off an LSI HBA, also try `storcli64 /c0 show termlog >/var/tmp/termlog.txt`
*n*x errno 22 is EINVAL, which seems unlikely, but it is possible that smartd got misconfigured.
Best configuration IMHO is to stop, disable, and mask the service.
Run smartctl -l /dev/sdc to launch an out-of-band long test. When it is done, use smartctl to report the results and see if anything is flagged.
On 8/21/25 09:10, Anthony D'Atri wrote:
On Aug 21, 2025, at 4:07 AM, Miles Goodhew <ceph@m0les.com> wrote:
Hi Robert, I'm not an expert on the low-level details and "modern" Ceph, so I hope I don't lead you on any wild goose chases, but I might at least give some leads. It seems odd that the metrics mention NVM/e - I'm guessing that it's just a cross-product test and tries all tools on all devices. Recent releases of smartctl pass through stats for NVMe devices via the name-cli command "nvme". Whether it invokes that for all devices, ordering, etc I don't know.
SMART test failure is more of an issue. It's a pity the error message is so nondescript. Some things I can think of from simplest to most complicated are: * Are smartmontools installed on the drive host? Does it happen with other drives on the same host?
If you have availability through your chassis vendor, look for a firmware update.
* Does the monitoring UID have sudo access? * Does a manual "sudo smartctl -a /dev/sdc" give the same or similar result? * Is the drive managed by a hardware RAID controller or concentrator (Like Dell PERC or a USB adapter or something) * (This is a stretch) Is there an OSD for the drive that's given the "NVME" class?
Hope that gives you something.
M0les.
On Thu, 21 Aug 2025, at 17:15, Robert Sander wrote:
Hi,
On a new cluster with version 19.2.3 the device health metrics only show a smartctl error:
{ "20250821-000313": { "dev": "/dev/sdc", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 1", "nvme_smart_health_information_add_log_error_code": - 22, "nvme_vendor": "ata", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\n" } }
The device in question (like all the other in the cluster) is a Samsung MZ7L37T6 SATA SSD.
What is happening here?
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 8/21/25 um 15:10 schrieb Anthony D'Atri:
Does it happen with other drives on the same host?
It happens with all drives on all hosts (6 hosts with 10 SSDs each.). Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
participants (5)
-
Anthony D'Atri
-
Anthony D'Atri
-
Miles Goodhew
-
Robert Sander
-
Tim Holloway