- ceph-users - lists.ceph.io

e5 failed to get devid for : udev_device_new_from_subsystem_sysname failed on ''
by Matthias Leopold 03 Mar '20

03 Mar '20

Hi, I installed a basic three host cluster with ceph-deploy for development today and I'm seeing messages like mon.somehost@2(electing) e5 failed to get devid for : udev_device_new_from_subsystem_sysname failed on '' in monitor logs when monitor starts up. What is this? I found this: https://tracker.ceph.com/issues/42053, but I wasn't using ceph-ansible. Cluster has 3 nodes, 3 monitors (all have that message), 2 managers and is healthy Ceph version is 14.2.8 on CentOS 7 thx Matthias

1 0

leftover: spilled over 128 KiB metadata after adding db device
by Stefan Priebe - Profihost AG 03 Mar '20

03 Mar '20

Hello, i added a db device to my osds running nautilus. The DB data migratet over some days from the hdd to ssd (db device). But now it seems all are stuck at: # ceph health detail HEALTH_WARN BlueFS spillover detected on 8 OSD(s) BLUEFS_SPILLOVER BlueFS spillover detected on 8 OSD(s) osd.0 spilled over 128 KiB metadata from 'db' device (12 GiB used of 185 GiB) to slow device osd.1 spilled over 3.4 MiB metadata from 'db' device (12 GiB used of 185 GiB) to slow device osd.2 spilled over 128 KiB metadata from 'db' device (12 GiB used of 185 GiB) to slow device osd.3 spilled over 128 KiB metadata from 'db' device (12 GiB used of 185 GiB) to slow device osd.4 spilled over 128 KiB metadata from 'db' device (12 GiB used of 185 GiB) to slow device osd.5 spilled over 128 KiB metadata from 'db' device (12 GiB used of 185 GiB) to slow device osd.6 spilled over 128 KiB metadata from 'db' device (12 GiB used of 185 GiB) to slow device osd.7 spilled over 128 KiB metadata from 'db' device (12 GiB used of 185 GiB) to slow device any idea how to force ceph to move the last 128kb to the db device? Greets, Stefan

3 5

Nautilus 14.2.8
by Fyodor Ustinov 03 Mar '20

03 Mar '20

Hi! Again. New version in repository without announce. :( I wonder who needs to write a letter and complain that there would always be an announcement, and then a new version in the repository? WBR, Fyodor.

3 3

Restrict client access to a certain rbd pool with seperate metadata and data pool
by Rainer Krienke 03 Mar '20

03 Mar '20

Hello, I do not know how to restrict a client.user to a certain rbd pool where this pool has a replicated metadata pool pool.rbd and an erasure coded data pool named pool.ec . I am running ceph nautilus. I tried this for a client.user: # ceph auth caps client.user mon 'profile rbd' osd 'profile rbd pool=pool.rbd' # ceph auth get client.user > ./client.user # rbd -n client.user -k ./client.user create pool.rbd/test --size=1G --data-pool=pool.ec 2020-03-03 15:54:43.813 7f2817fff700 -1 librbd::image::ValidatePoolRequest: handle_read_rbd_info: failed to read RBD info: (1) Operation not permitted 2020-03-03 15:54:43.813 7f2817fff700 -1 librbd::image::CreateRequest: 0x563421cf4730 handle_validate_data_pool: failed to validate pool: (1) Operation not permitted rbd: create error: (1) Operation not permitted If I remove the "... pool=pool.rbd" -section in "ceph auth caps ..." call from above everything works. Any idea how I can get this setup to work? Thanks Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html

3 2

building ceph Nautilus for Debian Stretch
by Stefan Priebe - Profihost AG 03 Mar '20

03 Mar '20

Hello list, does anybody have a guide to build ceph Nautilus for Debian stretch? I wasn't able to find a backported gcc-8 for stretch. Otherwise i would start one. Greets, Stefan

2 2

How can I fix "object unfound" error?
by Simone Lazzaris 03 Mar '20

03 Mar '20

Hi there; I've got a ceph cluster with 4 nodes, each with 9 4TB drives. Last night a disk failed, and unfortunately this lead to a kernel panic on the hosting server (supermicro: never again). One reboot later, the cluster rebalances. This morning, I'm in this situation: root@s3:~# ceph status cluster: id: 9ec27b0f-acfd-40a3-b35d-db301ac5ce8c health: HEALTH_ERR 1/13122293 objects unfound (0.000%) Possible data damage: 1 pg backfill_unfound Degraded data redundancy: 1 pg undersized 27 slow ops, oldest one blocked for 68 sec, osd.5 has slow ops services: mon: 3 daemons, quorum s1,s2,s3 (age 11h) mgr: s1(active, since 6w), standbys: s2, s3 osd: 36 osds: 35 up (since 11h), 35 in (since 11h); 21 remapped pgs rgw: 3 daemons active (s1, s2, s3) data: pools: 10 pools, 1200 pgs objects: 13.12M objects, 41 TiB usage: 63 TiB used, 65 TiB / 127 TiB avail pgs: 186357/39366879 objects misplaced (0.473%) 1/13122293 objects unfound (0.000%) 1179 active+clean 11 active+remapped+backfilling 9 active+remapped+backfill_wait 1 active+backfill_unfound+undersized+remapped io: client: 42 KiB/s rd, 5.2 MiB/s wr, 43 op/s rd, 11 op/s wr recovery: 163 MiB/s, 48 objects/s One PG is in "backfill_unfound" status. The PG is the 6.36a, which is on server 1; the failed disk is the OSD.5, on server 3 (which was rebooted after the panic) so I don't understand the relation. This is the unfound object: root@s3:~# ceph pg 6.36a list_unfound { "num_missing": 1, "num_unfound": 1, "objects": [ { "oid": { "oid": "8a257939-05c9-4ba8-9fd3-fb8504226607.4332.4__shadow_.H5AtB0LjzRSbUWy- hnVSLf4fs884okG_1", "key": "", "snapid": -2, "hash": 961006442, "max": 0, "pool": 6, "namespace": "" }, "need": "263'18213", "have": "0'0", "flags": "none", "locations": [] } ], "more": false } How can I handle this error? The docs are not much comforting, as far as I can see the only thing to do is to mark the missing object as lost and try to cope with that. I'd prefer not. Any ideas? *Simone Lazzaris* *Qcom S.p.A.* simone.lazzaris(a)qcom.it[1] | www.qcom.it[2] * LinkedIn[3]* | *Facebook*[4] -------- [1] mailto:simone.lazzaris@qcom.it [2] https://www.qcom.it [3] https://www.linkedin.com/company/qcom-spa [4] http://www.facebook.com/qcomspa

2 2

Octopus release announcement
by Alex Chalkias 03 Mar '20

03 Mar '20

Hello, I was looking for an official announcement for Octopus release, as the latest update (back in Q3/2019) on the subject said it was scheduled for March 1st. Any updates on that? BR, -- Alex Chalkias *Product Manager* alex.chalkias(a)canonical.com +33 766599367 *Canonical | **Ubuntu*

3 3

Expected Mgr Memory Usage
by m＠silvenga.com 02 Mar '20

02 Mar '20

Hello all, I'm maintaining a small Nautilus 12 OSD cluster (36TB raw). My mon nodes have the mgr/mds collocated/stacked with the mon. Each are allocated 10gb of RAM. During a recent single disk failure and corresponding recovery, I noticed my mgr/mon's were starting to get OOM killed/restarted every 5ish hours - the mgr using around 6.5GB on all my nodes. My monitoring shows an interesting sawtooth pattern with network usage (100MB/s at max), disk storage usage, and disk IO (up to 300MB/s against SSD's at max) usage increasing in parallel with memory usage. I know the docs for hardware recommendations say: > Monitor and manager daemon memory usage generally scales with the size of the cluster. For small clusters, 1-2 GB is generally sufficient. For large clusters, you should provide more (5-10 GB). Now, I would like to think my cluster is on the small size of things, so I was hoping 10gb is enough for the mgr and mon (my OSD nodes are only allocated 32GB of ram), but that assumption appears to be false. So I was wondering how mgr's (and to a lesser extent mon's) are expected to scale in terms of memory. Is it the osd count, or the osd's size, number of pg's, etc.? And if there's a way to limit the amount of RAM used by these mgr's (it seems the mon_osd_cache_size and rocksdb_cache_size settings are for mons if I'm not mistaken). Regards, Mark

1 0

I have different bluefs formatted labels
by Marc Roos 02 Mar '20

02 Mar '20

I have some osd's having a bluefs label formatted like this: { "/dev/sdc2": { "osd_uuid": "cfb2eaa3-1811-4108-b9bb-aad49555246c", "size": 4000681099264, "btime": "2017-07-14 14:58:09.627614", "description": "main", "require_osd_release": "14" } } And I have some osd's having a bluefs label formatted like this: { "/dev/sdd2": { "osd_uuid": "d8912a1b-696c-4668-9337-c740ec47e0d0", "size": 8001457295360, "btime": "2018-06-01 18:27:47.760695", "description": "main", "bluefs": "1", "ceph_fsid": "0f1701f5-453a-4a3b-928d-f652a2bbbcb0", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "ready": "ready", "require_osd_release": "14", "whoami": "18" } } I assume this is a version difference? What are the implications of having this? How to upgrade?

1 0

HEALTH_WARN 1 pools have many more objects per pg than average
by Marcel Ceph 02 Mar '20

02 Mar '20

Hi We have a new mimic (13.2.6, will upgrade to nautilus next month) cluster where the rados gateway pool has currently many more objects per PG then the other pools. This leads to a warning with ceph status 1 pools have many more objects per pg than average I tries to get rid of this warning by setting on the commandline ceph config set mgr mon_pg_warn_max_object_skew 0 I recall that it leaded to HEALTH_OK but some time later it was at HEALTH_WARN again with the same message. I tried setting the value to -1 as well, restarting mon and mgr daemons. Unfortunately the message stays. Setting in ceph.conf does not help either I know that in the future the number of objects per PG increase for the other pools but in the meantime monitoring for another incoming health warn issue is hard to automate so we aim to always have a HEALTH_OK state Anyone any luck in getting rid of the warning by setting mon_pg_warn_max_object_skew? Thx Marcel

1 0