April 2021 - ceph-users - lists.ceph.io

RGW: Multiple Site does not sync olds data

by 特木勒

Hi all: ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) I have a strange question, I just create a multiple site for Ceph cluster. But I notice the old data of source cluster is not synced. Only new data will be synced into second zone cluster. Is there anything I need to do to enable full sync for bucket or this is a bug? Thanks

2 years, 4 months

3
17
0 0

Zabbix module Octopus 15.2.3

by Gert Wieberdink

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

2 years, 4 months

5
4
0 0

x-amz-request-id logging with beast + rgw (ceph 15.2.10/containerized)?

by David Orman

Hi, Is there any way to log the x-amz-request-id along with the request in the rgw logs? We're using beast and don't see an option in the configuration documentation to add headers to the request lines. We use centralized logging and would like to be able to search all layers of the request path (edge, lbs, ceph, etc) with a x-amz-request-id. Right now, all we see is this: debug 2021-04-01T15:55:31.105+0000 7f54e599b700 1 beast: 0x7f5604c806b0: x.x.x.x - - [2021-04-01T15:55:31.105455+0000] "PUT /path/object HTTP/1.1" 200 556 - "aws-sdk-go/1.36.15 (go1.15.3; linux; amd64)" - We've also tried this: ceph config set global rgw_enable_ops_log true ceph config set global rgw_ops_log_socket_path /tmp/testlog After doing this, inside the rgw container, we can socat - UNIX-CONNECT:/tmp/testlog and see the log entries being recorded that we want, but there has to be a better way to do this, where the logs are emitted like the request logs above by beast, so that we can handle it using journald. If there's an alternative that would accomplish the same thing, we're very open to suggestions. Thank you, David

2 years, 4 months

3
6
0 0

Upgrade and lost osds Operation not permitted

by Behzad Khoshbakhti

Hi all, As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages: root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 --path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2 Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory* root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd(a)2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE) Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2# root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part └─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4 b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~# root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2# any help is much appreciated -- Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)

2 years, 4 months

6
14
0 0

Re: How to set bluestore_rocksdb_options_annex

by ceph＠elchaka.de

Hello Anthony, it was introduced in octopus 15.2.10 See: https://docs.ceph.com/en/latest/releases/octopus/ Do you know how you would set it in pacific? :) Guess, there shouldnt be much difference... Thank you Mehmet Am 28. April 2021 19:21:19 MESZ schrieb Anthony D'Atri <anthony.datri(a)gmail.com>: >I think that’s new with Pacific. > >> On Apr 28, 2021, at 1:26 AM, ceph(a)elchaka.de wrote: >> >> >> >> Hello, >> >> I have an octopus cluster and want to change some values - but i >cannot find any documentation on how to set values(multiple) with >> >> bluestore_rocksdb_options_annex >> >> Could someone give me some examples. >> I would like to do this like ceph config set ... >> >> Thanks in advice >> Mehmet >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 years, 5 months

2
5
0 0

Failed cephadm Upgrade - ValueError

by Ashley Merrick

Hello All,I was running 15.2.8 via cephadm on docker Ubuntu 20.04I just attempted to upgrade to 16.2.1 via the automated method, it successfully upgraded the mon/mgr/mds and some OSD's, however it then failed on an OSD and hasn't been able to pass even after stopping and restarting the upgrade.It reported the following ""message": "Error: UPGRADEREDEPLOYDAEMON: Upgrading daemon osd.35 on host sn-s01 failed.""If I run 'ceph health detail' I get lot's of the following error : "ValueError: not enough values to unpack (expected 2, got 1)" throughout the detail reportUpon googling, it looks like I am hitting something along the lines of https://158.69.68.89/issues/48924 & https://tracker.ceph.com/issues/49522What do I need to do to either get around this bug, or a way I can manually upgrade the remaining ceph OSD's to 16.2.1, currently my cluster is working but the last OSD it failed to upgrade is currently offline (I guess as no image attached to it now as it failed to pull it), and I have a cluster with OSD's from not 15.2.8 and 16.2.1Thanks Sent via MXlogin

2 years, 5 months

2
4
0 0

OSD id 241 != my id 248: conversion from "ceph-disk" to "ceph-volume simple" destroys OSDs

by Frank Schilder

Hi all, this is a follow-up on "reboot breaks OSDs converted from ceph-disk to ceph-volume simple". I converted a number of ceph-disk OSDs to ceph-volume using "simple scan" and "simple activate". Somewhere along the way, the OSDs meta-data gets rigged and the prominent symptom is that the symlink block is changes from a part-uuid target to an unstable device name target like: before conversion: block -> /dev/disk/by-partuuid/9123be91-7620-495a-a9b7-cc85b1de24b7 after conversion: block -> /dev/sdj2 This is a huge problem as the "after conversion" device names are unstable. I have now a cluster that I cannot reboot servers on due to this problem. OSDs randomly re-assigned devices will refuse to start with: 2021-03-02 15:56:21.709 7fb7c2549b80 -1 OSD id 241 != my id 248 Please help me with getting out of this mess. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 years, 5 months

2
7
0 0

Cannot create issue in bugtracker

by Tobias Urdin

Hello, Is it only me that's getting Internal error when trying to create issues in the bugtracker for some day or two? https://tracker.ceph.com/issues/new Best regards

2 years, 5 months

2
2
0 0

How radosgw works ?

by Fabrice Bacchella

I'm trying to understand what and where radosgw listen ? There is a lot of contradictory or redundant informations about that. First about the contradictory informations for the socket. At https://docs.ceph.com/en/pacific/radosgw/config-ref/ <https://docs.ceph.com/en/pacific/radosgw/config-ref/>, it says rgw_socket_path, but at https://docs.ceph.com/en/pacific/man/8/radosgw/ <https://docs.ceph.com/en/pacific/man/8/radosgw/> is says 'rgw socket path' That problem is quite common in the ceph documentation. Are both value accepted ? Next about some naming, or binding IP. Where it's defined, and how ? You have: rgw_frontends = "beast ssl_endpoint=0.0.0.0:443 port=443 ..." rgw_host = rgw_port = rgw_dns_name = That's a lot of redundancy, or contradictory informations. What is the purpose of each one ? What is the difference between rgw_frontends = ".. port = ..." and rgw_port = ? Or rgw_host and rgw_dns_name. What is the difference ? The documentation provides no help at all: rgw_dns_name Description: The DNS name of the served domain. See also the hostnames setting within regions. The description says nothing new, it just repeat the field name. Is one of them used by the manager for communication ? I already had the problem for the entry in the certificate used by the frontend, it used an IP coming from nowhere. If a fcgi is used, how the manager find the endpoint ?

2 years, 5 months

2
2
0 0

Best distro to run ceph.

by Peter Childs

I'm trying to set up a new ceph cluster, and I've hit a bit of a blank. I started off with centos7 and cephadm. Worked fine to a point, except I had to upgrade podman but it mostly worked with octopus. Since this is a fresh cluster and hence no data at risk, I decided to jump straight into Pacific when it came out and upgrade. Which is where my trouble began. Mostly because Pacific needs a version on lvm later than what's in centos7. I can't upgrade to centos8 as my boot drives are not supported by centos8 due to the way redhst disabled lots of disk drivers. I think I'm looking at Ubuntu or debian. Given cephadm has a very limited set of depends it would be good to have a supported matrix, it would also be good to have a check in cephadm on upgrade, that says no I won't upgrade if the version of lvm2 is too low on any host and let's the admin fix the issue and try again. I was thinking to upgrade to centos8 for this project anyway until I relised that centos8 can't support my hardware I've inherited. But currently I've got a broken cluster unless I can workout some way to upgrade lvm in centos7. Peter.

2 years, 5 months

3
3
0 0

2023

2022

2021

2020

2019

ceph-users April 2021