March 2021 - ceph-users - lists.ceph.io

How to create single OSD with SSD db device with cephadm

by 胡玮文

Hi all, I’m new to ceph. I recently deployed a ceph cluster with cephadm. Now I want to add a single new OSD daemon with a db device on SSD. But I can’t find any documentation about this. I have tried: 1. Using web dashboard. This requires at least one filter to proceed (type, vendor, model or size). But I just want to select the block device manually. 2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based. 3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot specify my SSD db device in this way. 4. On the target host, run ‘cephadm shell’ then ceph-volume prepare and activate. But ceph-volume seems can’t create systemd service outside the container like ‘ceph orch’ does. 5. On the target host, run ‘cephadm ceph-volume’, but it requires a json config file, I can’t figure out what is that. Any help is appreciated. Thanks.

1 day, 22 hours

2
5
1 0

Upgrading Ceph luminous to mimic on debian-buster

by Jean-Marc FONTANA

Hello everyone, We just installed a Ceph cluster version luminous (12.2.11) on servers working with Debian buster (10.8) using ceph-deploy and we are trying to upgrade it to mimic but can't find a way to do it. We tried ceph-deploy install --release mimic mon1 mon2 mon3 (after having modified /etc/apt/sources.list.d/ceph.list) but this does nothing because the packets are said to be up to date. Could someone help us, please ? Best regards

1 week, 1 day

3
2
0 0

cephfs forward scrubbing docs

by Dan van der Ster

Hi, Today while debugging something we had a few questions that might lead to improving the cephfs forward scrub docs: https://docs.ceph.com/en/latest/cephfs/scrub/ tldr: 1. Should we document which sorts of issues that the forward scrub is able to fix? 2. Can we make it more visible (in docs) that scrubbing is not supported with multi-mds? 3. Isn't the new `ceph -s` scrub task status misleading with multi-mds? Details here: 1) We found a CephFS directory with a number of zero sized files: # ls -l ... -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58 upload_fc501199e3e7abe6b574101cf34aeefb.png -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23 upload_fce4f55348185fefa0abdd8d11095ba8.gif -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54 upload_fd95b8358851f0dac22fb775046a6163.png ... The user claims that those files were non-zero sized last week. The sequence of zero sized files includes *all* files written between Nov 2 and 9. The user claims that his client was running out of memory, but this is now fixed. So I suspect that his ceph client (kernel 3.10.0-1127.19.1.el7.x86_64) was not behaving well. Anyway, I noticed that even though the dentries list 0 bytes, the underlying rados objects have data, and the data looks good. E.g. # rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx 200212e68b5.00000000 # file 200212e68b5.00000000 200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA, non-interlaced So I managed to recover the files doing something like this (using an input file mapping inode to filename) [see PS 0]. But I'm wondering if a forward scrub is able to fix this sort of problem directly? Should we document which sorts of issues that the forward scrub is able to fix? I anyway tried to scrub it, which led to: # ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx recursive repair Scrub is not currently supported for multiple active MDS. Please reduce max_mds to 1 and then scrub. So ... 2) Shouldn't we update the doc to mention loud and clear that scrub is not currently supported for multiple active MDS? 3) I was somehow surprised by this, because I had thought that the new `ceph -s` multi-mds scrub status implied that multi-mds scrubbing was now working: task status: scrub status: mds.x: idle mds.y: idle mds.z: idle Is it worth reporting this task status for cephfs if we can't even scrub them? Thanks!! Dan [0] mkdir -p recovered while read -r a b; do for i in {0..9} do echo "rados stat --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i $(printf "%x" $a).0000000$i done echo cat $(printf "%x" $a).* ">" $(printf "%x" $a) echo mv $(printf "%x" $a) recovered/$b done < inones_fnames.txt

1 month

2
2
0 0

does ceph rgw has any option to limit bandwidth

by Zhenshi Zhou

Hi, Is there any option of rados gateway that limit bandwidth?

1 month, 3 weeks

5
6
0 0

Recommendations on problem with PG

by Gabriel Medve

Hi, We have a problem with a PG that was inconsistent, currently the PG in our cluster have 3 copies. It was not possible for us to repair this pg with "ceph pg repair" (This PG is in osd 14,1,2) so we deleted some of the copies of osd 14 with the following command. ceph-objectstore-tool --data-path /var/lib/ceph/osd.14/ --pgid 22.f --op remove --force This caused an automatic attempt to create the missing copy entering the backfilling state, but when doing this it crashed osd 1 and 2 and threw the IOPS to 0, freezing the cluster. Is there any way to remove this entire pg or try to recreate the missing copy or ignore it completely? It causes instability in the cluster. Thank you, I await comments -- Untitled Document ------------------------------------------------------------------------ Gabriel I. Medve

2 months, 1 week

2
2
0 0

ceph-ansible in Pacific and beyond?

by Matthew Vernon

Hi, I caught up with Sage's talk on what to expect in Pacific ( https://www.youtube.com/watch?v=PVtn53MbxTc ) and there was no mention of ceph-ansible at all. Is it going to continue to be supported? We use it (and uncontainerised packages) for all our clusters, so I'd be a bit alarmed if it was going to go away... Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

2 months, 2 weeks

17
25
0 0

RGW: Multiple Site does not sync olds data

by 特木勒

Hi all: ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) I have a strange question, I just create a multiple site for Ceph cluster. But I notice the old data of source cluster is not synced. Only new data will be synced into second zone cluster. Is there anything I need to do to enable full sync for bucket or this is a bug? Thanks

2 months, 2 weeks

3
17
0 0

Zabbix module Octopus 15.2.3

by Gert Wieberdink

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

2 months, 3 weeks

5
4
0 0

OSD id 241 != my id 248: conversion from "ceph-disk" to "ceph-volume simple" destroys OSDs

by Frank Schilder

Hi all, this is a follow-up on "reboot breaks OSDs converted from ceph-disk to ceph-volume simple". I converted a number of ceph-disk OSDs to ceph-volume using "simple scan" and "simple activate". Somewhere along the way, the OSDs meta-data gets rigged and the prominent symptom is that the symlink block is changes from a part-uuid target to an unstable device name target like: before conversion: block -> /dev/disk/by-partuuid/9123be91-7620-495a-a9b7-cc85b1de24b7 after conversion: block -> /dev/sdj2 This is a huge problem as the "after conversion" device names are unstable. I have now a cluster that I cannot reboot servers on due to this problem. OSDs randomly re-assigned devices will refuse to start with: 2021-03-02 15:56:21.709 7fb7c2549b80 -1 OSD id 241 != my id 248 Please help me with getting out of this mess. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 months

2
7
0 0

OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag

by Jonas Jelten

Hi! After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag". The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs". Tracker issue with logs: https://tracker.ceph.com/issues/50017 Anyone else encountered this error? I've "suspended" the upgrade for now :) -- Jonas

3 months, 1 week

5
12
0 0

2021

2020

2019

ceph-users March 2021