Device missing from "ceph device ls"
Hi 19.2.1 " [root@franky ~]# ceph device ls | grep franky ATA_HGST_HDN726060ALE614_K1GV9P4B franky:sda osd.579 now ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P franky:sdn osd.577 now ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ franky:sdg osd.580 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA franky:sde osd.533 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA franky:sdf osd.527 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA franky:sdc osd.538 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A franky:sdx osd.583 ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN franky:sdp osd.532 5% now ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN franky:sdt osd.531 7% now ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH franky:sdh osd.576 now ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH franky:sdq osd.535 now ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH franky:sdv osd.574 now ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH franky:sdm osd.581 now ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH franky:sdk osd.582 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 franky:sdu osd.536 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 franky:sdr osd.530 ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 franky:sdl osd.528 ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 franky:sdaa osd.529 ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ franky:sdd osd.537 HGST_HUS724040ALS640_PBG458BX franky:sdb osd.568 HGST_HUS724040ALS640_PCJ2VXRX franky:sdj osd.578 HGST_HUS724040ALS640_PCJN3LWX franky:sdi osd.539 HGST_HUS724040ALS640_PCKX3UWX franky:sdo osd.564 HGST_HUS724040ALS640_PCKX4KSX franky:sds osd.575 SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 franky:nvme0n1 osd.527 osd.536 osd.539 osd.576 osd.577 osd.581 osd.582 osd.583 0% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 franky:nvme1n1 osd.530 osd.535 osd.538 osd.574 osd.575 4% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 franky:nvme3n1 osd.528 osd.533 osd.578 osd.579 osd.580 4% [root@franky ~]# ceph orch device ls | grep franky franky /dev/md127 ssd 221G No 12m ago Has a FileSystem franky /dev/nvme0n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme1n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme2n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R800346 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme3n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/sda hdd ATA_HGST_HDN726060ALE614_K1GV9P4B 5589G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdaa hdd ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdb hdd HGST_HUS724040ALS640_PBG458BX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdc hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdd hdd ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sde hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdf hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdg hdd ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdh hdd ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdi hdd HGST_HUS724040ALS640_PCJN3LWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdj hdd HGST_HUS724040ALS640_PCJ2VXRX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdk hdd ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdl hdd ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdm hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdn hdd ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdo hdd HGST_HUS724040ALS640_PCKX3UWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdp ssd ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN 186G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdq hdd ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdr hdd ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sds hdd HGST_HUS724040ALS640_PCKX4KSX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdt ssd ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN 447G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdu hdd ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdv hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdw hdd ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2273861 2794G Yes 12m ago franky /dev/sdx hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected " Any suggestions as to why a device (nvme2n1) might be missing from "ceph device ls"? It is being used as db device for 4 OSDs. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
On 27/03/2025 10:10, Torkil Svensgaard wrote:
Hi
19.2.1
" [root@franky ~]# ceph device ls | grep franky ATA_HGST_HDN726060ALE614_K1GV9P4B franky:sda osd.579 now ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P franky:sdn osd.577 now ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ franky:sdg osd.580
ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA franky:sde osd.533 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA franky:sdf osd.527
ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA franky:sdc osd.538 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A franky:sdx osd.583
ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN franky:sdp osd.532 5% now ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN franky:sdt osd.531 7% now ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH franky:sdh osd.576 now ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH franky:sdq osd.535 now ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH franky:sdv osd.574 now ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH franky:sdm osd.581 now ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH franky:sdk osd.582 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 franky:sdu osd.536 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 franky:sdr osd.530
ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 franky:sdl osd.528
ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 franky:sdaa osd.529
ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ franky:sdd osd.537
HGST_HUS724040ALS640_PBG458BX franky:sdb osd.568
HGST_HUS724040ALS640_PCJ2VXRX franky:sdj osd.578
HGST_HUS724040ALS640_PCJN3LWX franky:sdi osd.539
HGST_HUS724040ALS640_PCKX3UWX franky:sdo osd.564
HGST_HUS724040ALS640_PCKX4KSX franky:sds osd.575
SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 franky:nvme0n1 osd.527 osd.536 osd.539 osd.576 osd.577 osd.581 osd.582 osd.583 0% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 franky:nvme1n1 osd.530 osd.535 osd.538 osd.574 osd.575 4% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 franky:nvme3n1 osd.528 osd.533 osd.578 osd.579 osd.580 4% [root@franky ~]# ceph orch device ls | grep franky franky /dev/md127 ssd 221G No 12m ago Has a FileSystem franky /dev/nvme0n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme1n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme2n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R800346 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme3n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/sda hdd ATA_HGST_HDN726060ALE614_K1GV9P4B 5589G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdaa hdd ATA_WDC_WD30EFRX-68AX9N0_WD- WMC1T2483048 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdb hdd HGST_HUS724040ALS640_PBG458BX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdc hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdd hdd ATA_WDC_WD30EFRX-68EUZN0_WD- WCC4N6TYT6KZ 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sde hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdf hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdg hdd ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdh hdd ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdi hdd HGST_HUS724040ALS640_PCJN3LWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdj hdd HGST_HUS724040ALS640_PCJ2VXRX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdk hdd ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdl hdd ATA_WDC_WD2002FYPS-02W3B0_WD- WCAVY6793009 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdm hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdn hdd ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdo hdd HGST_HUS724040ALS640_PCKX3UWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdp ssd ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN 186G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdq hdd ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdr hdd ATA_WDC_WD2002FYPS-01U1B1_WD- WCAVY2258167 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sds hdd HGST_HUS724040ALS640_PCKX4KSX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdt ssd ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN 447G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdu hdd ATA_WDC_WD2002FYPS-01U1B1_WD- WCAVY2249800 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdv hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdw hdd ATA_WDC_WD30EFRX-68AX9N0_WD- WMC1T2273861 2794G Yes 12m ago
franky /dev/sdx hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected "
Any suggestions as to why a device (nvme2n1) might be missing from "ceph device ls"?
Verbose output shows the device but with no device name: " submit {"prefix": "device ls-by-host", "host": "franky", "target": ["mon-mgr", ""]} to mon-mgr DEVICE DEV DAEMONS EXPECTED FAILURE ATA_HGST_HDN726060ALE614_K1GV9P4B sda osd.579 now ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P sdn osd.577 now ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ sdg osd.580 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA sde osd.533 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA sdf osd.527 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA sdc osd.538 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A sdx osd.583 ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN sdp osd.532 now ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN sdt osd.531 now ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH sdh osd.576 now ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH sdq osd.535 now ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH sdv osd.574 now ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH sdm osd.581 now ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH sdk osd.582 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 sdu osd.536 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 sdr osd.530 ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 sdl osd.528 ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 sdaa osd.529 ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ sdd osd.537 HGST_HUS724040ALS640_PBG458BX sdb osd.568 HGST_HUS724040ALS640_PCJ2VXRX sdj osd.578 HGST_HUS724040ALS640_PCJN3LWX sdi osd.539 HGST_HUS724040ALS640_PCKX3UWX sdo osd.564 HGST_HUS724040ALS640_PCKX4KSX sds osd.575 SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 nvme0n1 osd.527 osd.536 osd.539 osd.576 osd.577 osd.581 osd.582 osd.583 SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R800346 osd.529 osd.537 osd.564 osd.568 SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 nvme1n1 osd.530 osd.535 osd.538 osd.574 osd.575 SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 nvme3n1 osd.528 osd.533 osd.578 osd.579 osd.580 " Mvh. Torkil
It is being used as db device for 4 OSDs.
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
I don’t really have an explanation, only that I’ve seen that for years. On our own cluster I see it as well, still on latest Pacific. I also noticed that removed devices are not cleaned up from the inventory. I still see some drives in there which we removed months ago. I thought maybe it would go away when we upgrade to Reef (waiting for 18.2.5), but it doesn’t seem like that will happen in newer versions. Zitat von Torkil Svensgaard <torkil@drcmr.dk>:
On 27/03/2025 10:10, Torkil Svensgaard wrote:
Hi
19.2.1
" [root@franky ~]# ceph device ls | grep franky ATA_HGST_HDN726060ALE614_K1GV9P4B franky:sda osd.579 now ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P franky:sdn osd.577 now ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ franky:sdg osd.580
ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA franky:sde osd.533 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA franky:sdf osd.527
ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA franky:sdc osd.538 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A franky:sdx osd.583
ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN franky:sdp osd.532 5% now ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN franky:sdt osd.531 7% now ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH franky:sdh osd.576 now ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH franky:sdq osd.535 now ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH franky:sdv osd.574 now ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH franky:sdm osd.581 now ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH franky:sdk osd.582 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 franky:sdu osd.536 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 franky:sdr osd.530
ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 franky:sdl osd.528
ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 franky:sdaa osd.529
ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ franky:sdd osd.537
HGST_HUS724040ALS640_PBG458BX franky:sdb osd.568
HGST_HUS724040ALS640_PCJ2VXRX franky:sdj osd.578
HGST_HUS724040ALS640_PCJN3LWX franky:sdi osd.539
HGST_HUS724040ALS640_PCKX3UWX franky:sdo osd.564
HGST_HUS724040ALS640_PCKX4KSX franky:sds osd.575
SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 franky:nvme0n1 osd.527 osd.536 osd.539 osd.576 osd.577 osd.581 osd.582 osd.583 0% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 franky:nvme1n1 osd.530 osd.535 osd.538 osd.574 osd.575 4% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 franky:nvme3n1 osd.528 osd.533 osd.578 osd.579 osd.580 4% [root@franky ~]# ceph orch device ls | grep franky franky /dev/md127 ssd 221G No 12m ago Has a FileSystem franky /dev/nvme0n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme1n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme2n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R800346 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme3n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/sda hdd ATA_HGST_HDN726060ALE614_K1GV9P4B 5589G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdaa hdd ATA_WDC_WD30EFRX-68AX9N0_WD- WMC1T2483048 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdb hdd HGST_HUS724040ALS640_PBG458BX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdc hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdd hdd ATA_WDC_WD30EFRX-68EUZN0_WD- WCC4N6TYT6KZ 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sde hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdf hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdg hdd ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdh hdd ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdi hdd HGST_HUS724040ALS640_PCJN3LWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdj hdd HGST_HUS724040ALS640_PCJ2VXRX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdk hdd ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdl hdd ATA_WDC_WD2002FYPS-02W3B0_WD- WCAVY6793009 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdm hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdn hdd ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdo hdd HGST_HUS724040ALS640_PCKX3UWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdp ssd ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN 186G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdq hdd ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdr hdd ATA_WDC_WD2002FYPS-01U1B1_WD- WCAVY2258167 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sds hdd HGST_HUS724040ALS640_PCKX4KSX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdt ssd ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN 447G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdu hdd ATA_WDC_WD2002FYPS-01U1B1_WD- WCAVY2249800 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdv hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdw hdd ATA_WDC_WD30EFRX-68AX9N0_WD- WMC1T2273861 2794G Yes 12m ago
franky /dev/sdx hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected "
Any suggestions as to why a device (nvme2n1) might be missing from "ceph device ls"?
Verbose output shows the device but with no device name:
" submit {"prefix": "device ls-by-host", "host": "franky", "target": ["mon-mgr", ""]} to mon-mgr DEVICE DEV DAEMONS EXPECTED FAILURE ATA_HGST_HDN726060ALE614_K1GV9P4B sda osd.579 now ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P sdn osd.577 now ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ sdg osd.580 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA sde osd.533 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA sdf osd.527 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA sdc osd.538 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A sdx osd.583 ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN sdp osd.532 now ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN sdt osd.531 now ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH sdh osd.576 now ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH sdq osd.535 now ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH sdv osd.574 now ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH sdm osd.581 now ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH sdk osd.582 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 sdu osd.536 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 sdr osd.530 ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 sdl osd.528 ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 sdaa osd.529 ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ sdd osd.537 HGST_HUS724040ALS640_PBG458BX sdb osd.568 HGST_HUS724040ALS640_PCJ2VXRX sdj osd.578 HGST_HUS724040ALS640_PCJN3LWX sdi osd.539 HGST_HUS724040ALS640_PCKX3UWX sdo osd.564 HGST_HUS724040ALS640_PCKX4KSX sds osd.575 SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 nvme0n1 osd.527 osd.536 osd.539 osd.576 osd.577 osd.581 osd.582 osd.583 SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R800346 osd.529 osd.537 osd.564 osd.568 SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 nvme1n1 osd.530 osd.535 osd.538 osd.574 osd.575 SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 nvme3n1 osd.528 osd.533 osd.578 osd.579 osd.580 "
Mvh.
Torkil
It is being used as db device for 4 OSDs.
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen Thanks, we'll not worry about it then. Another reboot of the host have fixed it for now but unfortunately it didn't help anything with the main issue we are having: https://www.mail-archive.com/ceph-users@ceph.io/msg29255.html Mvh. Torkil On 29-03-2025 01:21, Eugen Block wrote:
I don’t really have an explanation, only that I’ve seen that for years. On our own cluster I see it as well, still on latest Pacific. I also noticed that removed devices are not cleaned up from the inventory. I still see some drives in there which we removed months ago. I thought maybe it would go away when we upgrade to Reef (waiting for 18.2.5), but it doesn’t seem like that will happen in newer versions.
Zitat von Torkil Svensgaard <torkil@drcmr.dk>:
On 27/03/2025 10:10, Torkil Svensgaard wrote:
Hi
19.2.1
" [root@franky ~]# ceph device ls | grep franky ATA_HGST_HDN726060ALE614_K1GV9P4B franky:sda osd.579 now ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P franky:sdn osd.577 now ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ franky:sdg osd.580
ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA franky:sde osd.533 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA franky:sdf osd.527
ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA franky:sdc osd.538 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A franky:sdx osd.583
ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN franky:sdp osd.532 5% now ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN franky:sdt osd.531 7% now ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH franky:sdh osd.576 now ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH franky:sdq osd.535 now ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH franky:sdv osd.574 now ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH franky:sdm osd.581 now ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH franky:sdk osd.582 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 franky:sdu osd.536 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 franky:sdr osd.530
ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 franky:sdl osd.528
ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 franky:sdaa osd.529
ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ franky:sdd osd.537
HGST_HUS724040ALS640_PBG458BX franky:sdb osd.568
HGST_HUS724040ALS640_PCJ2VXRX franky:sdj osd.578
HGST_HUS724040ALS640_PCJN3LWX franky:sdi osd.539
HGST_HUS724040ALS640_PCKX3UWX franky:sdo osd.564
HGST_HUS724040ALS640_PCKX4KSX franky:sds osd.575
SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 franky:nvme0n1 osd.527 osd.536 osd.539 osd.576 osd.577 osd.581 osd.582 osd.583 0% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 franky:nvme1n1 osd.530 osd.535 osd.538 osd.574 osd.575 4% SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 franky:nvme3n1 osd.528 osd.533 osd.578 osd.579 osd.580 4% [root@franky ~]# ceph orch device ls | grep franky franky /dev/md127 ssd 221G No 12m ago Has a FileSystem franky /dev/nvme0n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme1n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme2n1 ssd SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R800346 2980G No 12m ago Has a FileSystem, LVM detected franky /dev/nvme3n1 ssd SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 1490G No 12m ago Has a FileSystem, LVM detected franky /dev/sda hdd ATA_HGST_HDN726060ALE614_K1GV9P4B 5589G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdaa hdd ATA_WDC_WD30EFRX-68AX9N0_WD- WMC1T2483048 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdb hdd HGST_HUS724040ALS640_PBG458BX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdc hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdd hdd ATA_WDC_WD30EFRX-68EUZN0_WD- WCC4N6TYT6KZ 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sde hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdf hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdg hdd ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdh hdd ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdi hdd HGST_HUS724040ALS640_PCJN3LWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdj hdd HGST_HUS724040ALS640_PCJ2VXRX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdk hdd ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdl hdd ATA_WDC_WD2002FYPS-02W3B0_WD- WCAVY6793009 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdm hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdn hdd ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdo hdd HGST_HUS724040ALS640_PCKX3UWX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdp ssd ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN 186G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdq hdd ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdr hdd ATA_WDC_WD2002FYPS-01U1B1_WD- WCAVY2258167 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sds hdd HGST_HUS724040ALS640_PCKX4KSX 3726G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdt ssd ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN 447G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdu hdd ATA_WDC_WD2002FYPS-01U1B1_WD- WCAVY2249800 1863G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdv hdd ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH 16.3T No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected franky /dev/sdw hdd ATA_WDC_WD30EFRX-68AX9N0_WD- WMC1T2273861 2794G Yes 12m ago
franky /dev/sdx hdd ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A 2794G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected "
Any suggestions as to why a device (nvme2n1) might be missing from "ceph device ls"?
Verbose output shows the device but with no device name:
" submit {"prefix": "device ls-by-host", "host": "franky", "target": ["mon-mgr", ""]} to mon-mgr DEVICE DEV DAEMONS EXPECTED FAILURE ATA_HGST_HDN726060ALE614_K1GV9P4B sda osd.579 now ATA_HGST_HDS724040ALE640_PK1334PBH7PZ5P sdn osd.577 now ATA_HGST_HDS724040ALE640_PK2381P1GNV2UZ sdg osd.580 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DDUA sde osd.533 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG6DGZA sdf osd.527 ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R0ZA sdc osd.538 now ATA_Hitachi_HDS5C3030ALA630_MJ1311YNG7R45A sdx osd.583 ATA_INTEL_SSDSC2BA200G4_BTHV6032026G200MGN sdp osd.532 now ATA_INTEL_SSDSC2KG480G7_PHYM743202DW480BGN sdt osd.531 now ATA_TOSHIBA_MG09ACA18TA_81P0A00PFJEH sdh osd.576 now ATA_TOSHIBA_MG09ACA18TA_81P0A01KFJEH sdq osd.535 now ATA_TOSHIBA_MG09ACA18TA_81P0A02RFJEH sdv osd.574 now ATA_TOSHIBA_MG09ACA18TA_81P0A02WFJEH sdm osd.581 now ATA_TOSHIBA_MG09ACA18TA_81Q0A00FFJEH sdk osd.582 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2249800 sdu osd.536 now ATA_WDC_WD2002FYPS-01U1B1_WD-WCAVY2258167 sdr osd.530 ATA_WDC_WD2002FYPS-02W3B0_WD-WCAVY6793009 sdl osd.528 ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T2483048 sdaa osd.529 ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N6TYT6KZ sdd osd.537 HGST_HUS724040ALS640_PBG458BX sdb osd.568 HGST_HUS724040ALS640_PCJ2VXRX sdj osd.578 HGST_HUS724040ALS640_PCJN3LWX sdi osd.539 HGST_HUS724040ALS640_PCKX3UWX sdo osd.564 HGST_HUS724040ALS640_PCKX4KSX sds osd.575 SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R600134 nvme0n1 osd.527 osd.536 osd.539 osd.576 osd.577 osd.581 osd.582 osd.583 SAMSUNG_MZPLJ3T2HBJR-00007_S55HNC0R800346 osd.529 osd.537 osd.564 osd.568 SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601201 nvme1n1 osd.530 osd.535 osd.538 osd.574 osd.575 SAMSUNG_MZPLL1T6HAJQ-00005_S4CBNA0N601254 nvme3n1 osd.528 osd.533 osd.578 osd.579 osd.580 "
Mvh.
Torkil
It is being used as db device for 4 OSDs.
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
participants (2)
-
Eugen Block
-
Torkil Svensgaard