- ceph-users - lists.ceph.io

Moving OSD from one node to another
by huxiaoyu＠horebdata.cn 09 Sep '20

09 Sep '20

Dear ceph folks, I encoutered an interesting situation as follows: an old FC SAN is connected two ceph OSD nodes, and its LUNs are used as virtual OSDs. When one node fails, its LUN can be taken over by anther node. My question is, how to start up the OSD on the new node without reconstructing its data? In other words, is there a simple to move OSD from one node to another? many thanks, samuel huxiaoyu(a)horebdata.cn

1 0

Storage class usage stats
by Tobias Urdin 09 Sep '20

09 Sep '20

Hello, I've been trying to understand if there is any way to get usage information based on storage classes for buckets. Since there is no information available from the "radosgw-admin bucket stats" commands nor any other endpoint I tried to browse the source code but couldn't find any references where the storage class would be exposed in such a way. It also seems that RadosGW today is not saving any counters on amount of objects stored in storage classes when it's collecting usage stats, which means there is no such metadata saved for a bucket. I was hoping it was atleast saved but not exposed because then it would have been a easier fix than adding support to count number of objects in storage classes based on operations which would involve a lot of places and mean writing to the bucket metadata on each op :( Is my assumptions correct that there is no way to retrieve such information, meaning there is no way to measure such usage? If the answer is yes, I assume the only way to get something that could be measured would be to instead have multiple placement targets since that is exposed from in bucket info. The bad things would be though that you lose a lot of functionality related to lifecycle and moving a single object to another storage class. Best regards Tobias

2 2

OSDs and tmpfs
by Shain Miley 09 Sep '20

09 Sep '20

Hi, I recently added 3 new servers to Ceph cluster. These servers use the H740p mini raid card and I had to install the HWE kernel in Ubuntu 16.04 in order to get the drives recognized. We have a 23 node cluster and normally when we add OSDs they end up mounting like this: /dev/sde1 3.7T 2.0T 1.8T 54% /var/lib/ceph/osd/ceph-15 /dev/sdj1 3.7T 2.0T 1.7T 55% /var/lib/ceph/osd/ceph-20 /dev/sdd1 3.7T 2.1T 1.6T 58% /var/lib/ceph/osd/ceph-14 /dev/sdc1 3.7T 1.8T 1.9T 49% /var/lib/ceph/osd/ceph-13 However I noticed this morning that the 3 new servers have the OSDs mounted like this: tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-246 tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-240 tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-248 tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-237 Is this normal for deployments going forward…or did something go wrong? These are 12TB drives but they are showing up as 47G here instead. We are using ceph version 12.2.13 and I installed this using ceph-deply version 2.0.1. Thanks in advance, Shain Shain Miley | Director of Platform and Infrastructure | Digital Media | smiley(a)npr.org

1 0

Cleanup orphan osd process in octopus
by levindecaro＠gmail.com 09 Sep '20

09 Sep '20

Hi there, I got a handicap problem on ceph orch which cannot found anywhere from doc to solve this problem, I was removed an osd using ceph purge osd.27 (old style) and forgotten there are new way in Octopus. Now i got two osd process via ceph orch ps, is it have a way to clean it up properly? osd.27 sds-0 running (7h) 49s ago 7h 15.2.4 docker.io/ceph/ceph:v15 852b28cb10de acab3229c945 osd.27 sds-2 stopped 19s ago 4d <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown>

1 0

How to delete OSD benchmark data
by Jayesh Labade 09 Sep '20

09 Sep '20

Dear Ceph Users, I am testing my 3 node Proxmox + Ceph cluster. I have performed osd benchmark with the below command. # ceph tell osd.0 bench Do I need to perform any cleanup to delete benchmark data from osd ? I have googled for same but nowhere mentioned post steps after osd benchmark commands Thanks Jayesh

2 2

RGW bucket sync
by Eugen Block 09 Sep '20

09 Sep '20

Hi *, I'm wondering about what actually happens in the ceph cluster if I copy/sync the content of one bucket into a different bucket. I'll just describe what I saw and maybe someone could clarify what is happening. I have a RGW in a small test cluster (15.2.2) and created a bucket (bucket1) with s3cmd, then put a large file into bucket1. This takes some time, of course, I see the network utilization on the client and also the OSD load on the cluster during the upload (expected). Then I create bucket2 and run 's3cmd cp s3://bucket1/file s3://bucket2' which takes only a couple of seconds. I only see some OSD load for a short period of time during the copy process, but that's it. The copied file is available almost immediately. How does this work? It seems as if there's (almost) no client traffic (except for the cp command, of course) to recreate the file in the second bucket, as if the OSDs are directly instructed to create copies of the objects. I would highly appreciate it if anyone could clarify this for me. Thanks! Eugen

3 4

How to working with ceph octopus multisite-sync-policy
by system.engineer.mon＠gmail.com 09 Sep '20

09 Sep '20

Dear All I build 2 ceph poc cluster My ceph version is octopus 15.2.4 with centos 8 I configure ceph multisite is working Cluster_1 realm 20f3a458-b0d4-4699-b8f7-75411366c635 (global) zonegroup 94b70bdb-1492-49b7-8cda-210d3552ecd2 (data) zone bb4106bc-6a01-44e7-be53-ce8c5dbbce97 (hy-1) metadata sync no sync (zone is master) data sync source: 36373df5-bcd8-4eea-954b-3b6158d57c80 (hy-2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source Cluster_2 realm 20f3a458-b0d4-4699-b8f7-75411366c635 (global) zonegroup 94b70bdb-1492-49b7-8cda-210d3552ecd2 (data) zone 36373df5-bcd8-4eea-954b-3b6158d57c80 (hy-2) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: bb4106bc-6a01-44e7-be53-ce8c5dbbce97 (hy-1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source 2 Cluster is Active/Active Create bucket in cluster_1 => It automatic sync to cluster_2 Create bucket in cluster_2 => It automatic sync to cluster_1 Put data in bucket cluster_1 => It automatic sync to bucket in cluster_2 Put data in bucket cluster_2 => It automatic sync to bucket in cluster_1 I try configure example_3 and example_4 but not working https://docs.ceph.com/docs/master/radosgw/multisite-sync-policy/ Anyone, Can help or recommend me ? Best Regrads, Engineer_Mon

1 0

Problem with /etc/ceph/iscsi-gateway.cfg checksum
by Matthias Leopold 09 Sep '20

09 Sep '20

Hi, I'm preparing three Ceph Nautilus nodes as iSCSI gateways following this documentation: https://docs.ceph.com/docs/nautilus//rbd/iscsi-overview/. For some reason rbd-target-api for one node doesn't agree on /etc/ceph/iscsi-gateway.cfg although "sha256sum /etc/ceph/iscsi-gateway.cfg" is the same: ~$ ssh -l root tentakel4 "sha256sum /etc/ceph/iscsi-gateway.cfg" e65444a15bbe9a115fd27040233883b681226e922367f7c61f7378d596ed12fd /etc/ceph/iscsi-gateway.cfg ~$ ssh -l root tentakel5 "sha256sum /etc/ceph/iscsi-gateway.cfg" e65444a15bbe9a115fd27040233883b681226e922367f7c61f7378d596ed12fd /etc/ceph/iscsi-gateway.cfg ~$ ssh -l root tentakel6 "sha256sum /etc/ceph/iscsi-gateway.cfg" e65444a15bbe9a115fd27040233883b681226e922367f7c61f7378d596ed12fd /etc/ceph/iscsi-gateway.cfg ~$ curl --user foo:bar -X GET http://tentakel4:5000/api/sysinfo/checkconf { "data": "110a32477ca026225bceab337535e0cad522d8ea10e69922e1ef9412331b8d94" } ~$ curl --user foo:bar -X GET http://tentakel5:5000/api/sysinfo/checkconf { "data": "faee0b958ea8f1bcbfaae4a40ca247af2271d439b54fa345be5a3fa2283dcc84" } ~$ curl --user foo:bar -X GET http://tentakel6:5000/api/sysinfo/checkconf { "data": "110a32477ca026225bceab337535e0cad522d8ea10e69922e1ef9412331b8d94" } On tentakel5 I'm constantly seeing "INFO [rbd-target-api:2821:run()] - Change detected - internal 0 / xattr 21 refreshing" in /var/log/rbd-target-api/rbd-target-api.log Can somebody give me a hint how to fix this? Software: - CentOS 7.8 - ceph-iscsi-3.4-1.el7.noarch - tcmu-runner-1.5.2-1.el7.x86_64 - sestatus: permissive thx Matthias

1 0

Error in OS causing Epson Error Code 0x97 pop up? Get to assistance.
by mary smith 09 Sep '20

09 Sep '20

The OS can have some compatibility issues with the printer and that might result in Epson Error Code 0x97. Therefore, to deal with the issue you can use the help steps that are mentioned in the tech consultancy sites or you can also get the issue resolved by using the assistance that is available in the tech videos. https://www.epsonprintersupportpro.net/epson-error-code-0x97/

1 0

cephadm didn't create journals
by Darrin Hodges 08 Sep '20

08 Sep '20

HI all, I have created a new cephs cluster using the cephadm command and it appears to work very well. I tried to specify using an ssd for the journals but it doesn't appear to have worked. My yaml file is: service_type: osd service_id: default_drive_group placement: host_pattern: 'ceph-osd0[1-4]' journal_devices: rotational: 0 data_devices: rotational: 1 encrypted: true all of the osd's were created but it appears the journals are on each disk rather than the ssd. Can I add journals after the fact with cephadm ? Are they even still a thing? thanks Darrin -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

3 2