April 2021 - ceph-users - lists.ceph.io

by Matthew Vernon

Hi, I caught up with Sage's talk on what to expect in Pacific ( https://www.youtube.com/watch?v=PVtn53MbxTc ) and there was no mention of ceph-ansible at all. Is it going to continue to be supported? We use it (and uncontainerised packages) for all our clusters, so I'd be a bit alarmed if it was going to go away... Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

2 years, 11 months

17
25
0 0

libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping

by Markus Kienast

I am seeing these messages when booting from RBD and booting hangs there. libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping However, Ceph Health is OK, so I have no idea what is going on. I reboot my 3 node cluster and it works again for about two weeks. How can I find out more about this issue, how can I dig deeper? Also there has been at least one report about this issue before on this mailing list - "[ceph-users] Strange Data Issue - Unexpected client hang on OSD I/O Error" - but no solution has been presented. This report was from 2018, so no idea if this is still an issue for Dyweni the original reporter. If you read this, I would be happy to hear how you solved the problem. Cheers Markus

2 years, 11 months

4
14
0 0

RGW: Multiple Site does not sync olds data

by 特木勒

Hi all: ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) I have a strange question, I just create a multiple site for Ceph cluster. But I notice the old data of source cluster is not synced. Only new data will be synced into second zone cluster. Is there anything I need to do to enable full sync for bucket or this is a bug? Thanks

2 years, 11 months

3
17
0 0

Zabbix module Octopus 15.2.3

by Gert Wieberdink

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

2 years, 11 months

5
4
0 0

x-amz-request-id logging with beast + rgw (ceph 15.2.10/containerized)?

by David Orman

Hi, Is there any way to log the x-amz-request-id along with the request in the rgw logs? We're using beast and don't see an option in the configuration documentation to add headers to the request lines. We use centralized logging and would like to be able to search all layers of the request path (edge, lbs, ceph, etc) with a x-amz-request-id. Right now, all we see is this: debug 2021-04-01T15:55:31.105+0000 7f54e599b700 1 beast: 0x7f5604c806b0: x.x.x.x - - [2021-04-01T15:55:31.105455+0000] "PUT /path/object HTTP/1.1" 200 556 - "aws-sdk-go/1.36.15 (go1.15.3; linux; amd64)" - We've also tried this: ceph config set global rgw_enable_ops_log true ceph config set global rgw_ops_log_socket_path /tmp/testlog After doing this, inside the rgw container, we can socat - UNIX-CONNECT:/tmp/testlog and see the log entries being recorded that we want, but there has to be a better way to do this, where the logs are emitted like the request logs above by beast, so that we can handle it using journald. If there's an alternative that would accomplish the same thing, we're very open to suggestions. Thank you, David

2 years, 11 months

3
6
0 0

Upgrade and lost osds Operation not permitted

by Behzad Khoshbakhti

Hi all, As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages: root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 --path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2 Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory* root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd(a)2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE) Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2# root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part └─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4 b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~# root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2# any help is much appreciated -- Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)

2 years, 11 months

6
14
0 0

Re: How to set bluestore_rocksdb_options_annex

by ceph＠elchaka.de

Hello Anthony, it was introduced in octopus 15.2.10 See: https://docs.ceph.com/en/latest/releases/octopus/ Do you know how you would set it in pacific? :) Guess, there shouldnt be much difference... Thank you Mehmet Am 28. April 2021 19:21:19 MESZ schrieb Anthony D'Atri <anthony.datri(a)gmail.com>: >I think that’s new with Pacific. > >> On Apr 28, 2021, at 1:26 AM, ceph(a)elchaka.de wrote: >> >> >> >> Hello, >> >> I have an octopus cluster and want to change some values - but i >cannot find any documentation on how to set values(multiple) with >> >> bluestore_rocksdb_options_annex >> >> Could someone give me some examples. >> I would like to do this like ceph config set ... >> >> Thanks in advice >> Mehmet >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 years, 11 months

2
5
0 0

Failed cephadm Upgrade - ValueError

by Ashley Merrick

Hello All,I was running 15.2.8 via cephadm on docker Ubuntu 20.04I just attempted to upgrade to 16.2.1 via the automated method, it successfully upgraded the mon/mgr/mds and some OSD's, however it then failed on an OSD and hasn't been able to pass even after stopping and restarting the upgrade.It reported the following ""message": "Error: UPGRADEREDEPLOYDAEMON: Upgrading daemon osd.35 on host sn-s01 failed.""If I run 'ceph health detail' I get lot's of the following error : "ValueError: not enough values to unpack (expected 2, got 1)" throughout the detail reportUpon googling, it looks like I am hitting something along the lines of https://158.69.68.89/issues/48924 & https://tracker.ceph.com/issues/49522What do I need to do to either get around this bug, or a way I can manually upgrade the remaining ceph OSD's to 16.2.1, currently my cluster is working but the last OSD it failed to upgrade is currently offline (I guess as no image attached to it now as it failed to pull it), and I have a cluster with OSD's from not 15.2.8 and 16.2.1Thanks Sent via MXlogin

2 years, 11 months

2
4
0 0

OSD id 241 != my id 248: conversion from "ceph-disk" to "ceph-volume simple" destroys OSDs

by Frank Schilder

Hi all, this is a follow-up on "reboot breaks OSDs converted from ceph-disk to ceph-volume simple". I converted a number of ceph-disk OSDs to ceph-volume using "simple scan" and "simple activate". Somewhere along the way, the OSDs meta-data gets rigged and the prominent symptom is that the symlink block is changes from a part-uuid target to an unstable device name target like: before conversion: block -> /dev/disk/by-partuuid/9123be91-7620-495a-a9b7-cc85b1de24b7 after conversion: block -> /dev/sdj2 This is a huge problem as the "after conversion" device names are unstable. I have now a cluster that I cannot reboot servers on due to this problem. OSDs randomly re-assigned devices will refuse to start with: 2021-03-02 15:56:21.709 7fb7c2549b80 -1 OSD id 241 != my id 248 Please help me with getting out of this mess. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 years, 11 months

2
7
0 0

Cannot create issue in bugtracker

by Tobias Urdin

Hello, Is it only me that's getting Internal error when trying to create issues in the bugtracker for some day or two? https://tracker.ceph.com/issues/new Best regards

2 years, 11 months

2
2
0 0

2024

2023

2022

2021

2020

2019

ceph-users April 2021