- ceph-users - lists.ceph.io

17.2.6 dashboard: unable to get RGW dashboard working
by Michel Jouvin 21 Apr '23

21 Apr '23

Hi, I just upgraded in 17.2.6 but in fact I had the same problem in 16.2.10. I'm trying to configure the Ceph dashboard to monitor the RGWs (object gateways used as S3 gw). Our cluster has 2 RGW realms (eros, fink) with 1 zonegroup per realm (p2io-eros and p2io-fink respectively) and 1 zone per zonegroup (eros-z1 and fink-z1 respectively). RGW have been configured without SSL due to the bug encountered in 16.2.10 (not sure if it still applies, use of IP addresses that don't match the certificate). There is a dashboard user in each realm and its access/secret keys are matching what is in mgr/dashboard/RGW_API_ACCESS_KEY and mgr/dashboard/RGW_API_SECRET_KEY (a dict with one entry per realm). Every attempt to access the RGW daemons section of the dashboard results in : ----- The Object Gateway Service is not configured Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"InvalidAccessKeyId","RequestId":"tx00000b4c2212597168bf0-00644116e3' b'-edf7c3-eros-z1","HostId":"edf7c3-eros-z1-ijclab"}') ---- In the mgr logs, I have not been able to find more than this error. In the RGW logs, for every dashboard attempt, I find: ------ Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: ====== starting new request req=0x7f2dcdb0b710 ===== Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: req 13025009517641239536 0.000000000s op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: ====== req done req=0x7f2dcdb0b710 op status=0 http_status=403 latency=0.000000000s ====== Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: beast: 0x7f2dcdb0b710: 134.158.194.10 - - [20/Apr/2023:10:41:39.843 +0000] "GET /admin/metadata/user?myself HTTP/1.1" 403 126 - "python-requests/2.20.0" - latency=0.000000000s ------ In the logs above, I find a little bit strange the "HostID" value ending with "eros-z1-ijclab". I'd expected it to be the zone name "eros-z1". The zone has been renamed a couple of times and it may be an old name, despite the zone information in radosgw-admin shows nothing with this name (in particular the API name has been renamed to match the zone name). Anway the RGW works perfectly and the dashboard request seems to reach the RGW so I don't have the feeling it is important... Any suggestion on how to troubleshoot what happens would be appreciated. Thanks in advance. Best regards, Michel

1 1

deploying Ceph using FQDN for MON / MDS Services
by Lokendra Rathour 21 Apr '23

21 Apr '23

Hi Team, We have a ceph cluster with 3 storage nodes: 1. storagenode1 - abcd:abcd:abcd::21 2. storagenode2 - abcd:abcd:abcd::22 3. storagenode3 - abcd:abcd:abcd::23 The requirement is to mount ceph using the domain name of MON node: Note: we resolved the domain name via DNS server. For this we are using the command: ``` mount -t ceph [storagenode.storage.com]:6789:/ /backup -o name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== ``` We are getting the following logs in /var/log/messages: ``` Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' (ret=-3): failed Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip ' storagenode.storage.com:6789' ``` We also tried mounting ceph storage using IP of MON which is working fine. Query: Could you please help us out with how we can mount ceph using FQDN. My /etc/ceph/ceph.conf is as follows: [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] public network = abcd:abcd:abcd::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 4294967296 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 rgw thread pool size = 512 -- ~ Lokendra skype: lokendrarathour

7 20

How to replace an HDD in a OSD with shared SSD for DB/WAL
by enochlew＠gmail.com 20 Apr '23

20 Apr '23

HI, I build a Ceph Cluster with cephadm. Every cehp node has 4 OSDs. These 4 OSD were build with 4 HDD (block) and 1 SDD (DB). At present , one HDD is broken, and I am trying to replace the HDD,and build the OSD with the new HDD and the free space of the SDD. I did the follows: #ceph osd stop osd.23 #ceph osd out osd.23 #ceph osd crush remove osd.23 #ceph osd rm osd.23 #ceph orch daemon rm osd.23 --force #lvremove /dev/ceph-ae21e618-601e-4273-9185-99180edb8453/osd-block-96eda371-1a3f-4139-9123-24ec1ba362c4 #wipefs -af /dev/sda #lvremove /dev/ceph-e50203a6-8b8e-480f-965c-790e21515395/osd-db-70f7a032-cf2c-4964-b979-2b90f43f2216 #ceph orch daemon add osd compute11:data_devices=/dev/sda,db_devices=/dev/sdc,osds_per_device=1 The OSD can be built, but is always down. Is there anyting that I missed during the building? Thank you very much! Regards, LIUTao

1 0

I am unable to execute 'rbd map xxx' as it returns the error 'rbd: map failed: (5) Input/output error'.
by siriusa51＠outlook.com 20 Apr '23

20 Apr '23

$ rbd map xxx rbd: sysfs write failed 2023-04-21 11:29:13.786418 7fca1bfff700 -1 librbd::image::OpenRequest: failed to retrieve image id: (5) Input/output error 2023-04-21 11:29:13.786456 7fca1b7fe700 -1 librbd::ImageState: 0x55a60108a040 failed to open image: (5) Input/output error rbd: error opening image xxx: (5) Input/output error In some cases useful info is found in syslog - try "dmesg | tail" or so. rbd: map failed: (5) Input/output error the command 'dmesg | tail' did not show any useful results. and other command: $ rbd info xxx 2023-04-21 11:33:10.223701 7f3547fff700 -1 librbd::image::OpenRequest: failed to retrieve image id: (5) Input/output error 2023-04-21 11:33:10.223739 7f35477fe700 -1 librbd::ImageState: 0x5647d5cfeeb0 failed to open image: (5) Input/output error rbd: error opening image xxx: (5) Input/output error i know header id is c2c061579478fe ,so i get omapvals: $ rados -p rbd listomapvals rbd_header.c2c061579478fe features value (8 bytes) : 00000000 01 00 00 00 00 00 00 00 |........| 00000008 object_prefix value (27 bytes) : 00000000 17 00 00 00 72 62 64 5f 64 61 74 61 2e 63 32 63 |....rbd_data.c2c| 00000010 30 36 31 35 37 39 34 37 38 66 65 |061579478fe| 0000001b order value (1 bytes) : 00000000 16 |.| 00000001 size value (8 bytes) : 00000000 00 00 00 00 71 02 00 00 |....q...| 00000008 snap_seq value (8 bytes) : 00000000 00 00 00 00 00 00 00 00 |........| 00000008 I scanned all existing data blocks using the command 'rados -p rbd list | grep c2c061579478fe', and found many 'rbd_data' blocks. I suspect that the loss of important blocks caused this situation. Currently, my plan is to obtain all 'rbd' blocks, concatenate them into an XFS file, and then perform a repair. However, as the data size is huge, I am still experimenting with this method. Are there any other feasible methods to repair this data? rbd_data: rbd_data.c2c061579478fe.0000000000000000 rbd_data.c2c061579478fe.0000000000000001 rbd_data.c2c061579478fe.0000000000000002 rbd_data.c2c061579478fe.0000000000000003 rbd_data.c2c061579478fe.0000000000000004 rbd_data.c2c061579478fe.0000000000000005 rbd_data.c2c061579478fe.0000000000000006 rbd_data.c2c061579478fe.0000000000000007 rbd_data.c2c061579478fe.0000000000000008 rbd_data.c2c061579478fe.0000000000000009 rbd_data.c2c061579478fe.000000000000000a rbd_data.c2c061579478fe.000000000000000b rbd_data.c2c061579478fe.000000000000000c rbd_data.c2c061579478fe.000000000000000d rbd_data.c2c061579478fe.000000000000000e rbd_data.c2c061579478fe.000000000000000f rbd_data.c2c061579478fe.0000000000000010 rbd_data.c2c061579478fe.0000000000000011 ... rbd_data.c2c061579478fe.000000000009c3f6 rbd_data.c2c061579478fe.000000000009c3f7 rbd_data.c2c061579478fe.000000000009c3fa rbd_data.c2c061579478fe.000000000009c3fb rbd_data.c2c061579478fe.000000000009c3fc rbd_data.c2c061579478fe.000000000009c3fd rbd_data.c2c061579478fe.000000000009c3fe rbd_data.c2c061579478fe.000000000009c3ff rbd_header.c2c061579478fe thanks

1 0

Can I delete rgw log entries?
by Richard Bade 20 Apr '23

20 Apr '23

Hi Everyone, I've been having trouble finding an answer to this question. Basically I'm wanting to know if stuff in the .log pool is actively used for anything or if it's just logs that can be deleted. In particular I was wondering about sync logs. In my particular situation I have had some tests of zone sync setup, but now I've removed the secondary zone and pools. My primary zone is filled with thousands of logs like this: data_log.71 data.full-sync.index.e2cf2c3e-7870-4fc4-8ab9-d78a17263b4f.47 meta.full-sync.index.7 datalog.sync-status.shard.e2cf2c3e-7870-4fc4-8ab9-d78a17263b4f.13 bucket.sync-status.f3113d30-ecd3-4873-8537-aa006e54b884:{bucketname}:default.623958784.455 I assume that because I'm not doing any sync anymore I can delete all the sync related logs? Is anyone able to confirm this? What about if the sync is running? Are these being written and read from and therefore must be left alone? It seems like these are more of a status than just a log and that deleting them might confuse the sync process. If so, does that mean that the log pool is not just output that can be removed as needed? Are there perhaps other things in there that need to stay? Regards, Richard

3 3

Be careful with primary-temp to balance primaries ...
by Stefan Kooman 20 Apr '23

20 Apr '23

Hi, A word of caution for Ceph operators out there. Be careful with "ceph osd primary-temp" command. TL;DR: with primary_temp active, a CRUSH change might CRASH your OSDs ... and they won't come back online after a restart (in almost all cases). The bug is described in this tracker [1], and fixed with this PR [2] (thanks Igor!). The longer story is that we were inspired by the work done on the read-balancer [3] and wondered if we could leverage this on older clusters. It turned out this is indeed possible by using the "primary-temp" command, instead of "pg-primary-temp" that will be available from Reef onward. We compiled a main version of the "osdmaptool", fed it OSD maps from pacific clusters and have it calculate the optimal primary PG distributions. Then we replaced the pg-primary-temp command with "primary-temp" and applied the commands. That worked as expected. However, we hit a bug [1] in the Ceph code that could not handle a situation when there were changes in the CRUSH map when primary_temp where active. We added a new storage node to the cluster that triggered this condition as soon as we put it in the proper failure domain. It tried to make an OSD primary that was not in the active set anymore, and hence crashed (with a Segmentation Fault most often, or aborted). This resulted in multiple (many) OSD crashes across the failure domains and basically took down the whole cluster. If the Reef / main read-balancer code can suffer from the same bug is as of yet unknown (at least to us). We will try to build a Reef test cluster and find out. For those of you who want to know how we handled this incident can read the RFO ([4] in dutch, [5] in English). Gr. Stefan [1]: https://tracker.ceph.com/issues/59491?next_issue_id=59490 [2]: https://github.com/ceph/ceph/pull/51160 [3]: https://github.com/ljflores/ceph_read_balancer_2023 [4]: https://www.bit.nl/uploads/images/PDF-Files/RFO-20230314-185335.pdf [5]: https://www.bit.nl/uploads/images/PDF-Files/2023.04.20%20RFO_Ceph Cluster_185335_EN.pdf

2 1

User + Dev Monthly Meetup cancelled
by Laura Flores 20 Apr '23

20 Apr '23

Hi users, The User + Dev monthly meetup is cancelled today since some folks are still at or returning from Cephalocon. See you next month! - Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

quincy user metadata constantly changing versions on multisite slave with radosgw roles
by Christopher Durham 20 Apr '23

20 Apr '23

Hi, I am using 17.2.6 on rocky linux for both the master and the slave site I noticed that: radosgw-admin sync status often shows that the metadata sync is behind a minute or two on the slave. This didn't make sense, as the metadata isn't changing as far as I know. radosgw-admin mdlog list (on slave) showed me that there were user changes in metadata very often. After doing a little research, here is a scenario I was able to develop: 1. user continually writes to a bucket he owns, pointing his aws cli (for this test) to the master side endpoint as specified in ~/.aws/config while this is running, do the following on the slave: radosgw-admin metadata get user:<username> This always shows the same result for the user, no changes. The data gets to the slave side bucket. 2. Restart the continual copy, but this time use a role that the user is a member of via profile in ~/.aws/credentials and .~/aws/config, again writing to the master endpoint as specified in ~/.aws/config aws --profile <profile> s3 cp <file> s3://<bucket>/file where profile is set up to use a role definiiton. The data gets to the bucket on both sides. I do not have access if I do not use the role profille (to confirm I set it up right) However, while doing this second test, if I continually do: radosgw-admin metadata get user:<username> on the slave, I see a definite increase in versions. Here is a section of the json output: "key": "user:<username>", "ver": { "tag": "somestring", "ver:" 12145} the 12145 value increases over and over again, and the mtime value in the json output increases too based on the current date. (not shown here). The same value, when queried on the master side, remains 1, and the mtime value is the date the user was created or last changed by an admin. If I write a file only once, the vers value increases by 1 too, but not sure if the increase in vers is ncessarily 1:1 with the number of writes. This seems to be the source of my continual metadata lag. Am I missing something? I suspect that this has been happening for awhile and not specific to 17.2.6 as I just upgraded and the ver value is over 12000 for the user that I discovered. (I used a python script to sync roles between master and slave prior to 17.2.6. Now roles are replicated in 17.2.6). -Chris

2 1

Some hint for a DELL PowerEdge T440/PERC H750 Controller...
by Marco Gaiarin 20 Apr '23

20 Apr '23

We are testing an experimental Ceph cluster with server and controller at subject. The controller have not an HBA mode, but only a 'NonRAID' mode, come sort of 'auto RAID0' configuration. We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well, and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad (particulary, very low IOPS). There's some hint for disk/controller configuration/optimization? Thanks. -- Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso... mi va bene fino a quando non riguarda me :) (Emanuele Pucciarelli)

7 18

[ceph 17.2.6] unable to create rbd snapshots for images with erasure code data-pool
by Reto Gysi 20 Apr '23

20 Apr '23

Dear Ceph Users, After upgrading from version 17.2.5 to 17.2.6 I no longer seem to be able to create snapshots of images that have an erasure coded datapool. root@zephir:~# rbd snap create ceph-dev@backup_20230417 Creating snap: 10% complete...failed. rbd: failed to create snapshot: (95) Operation not supported root@zephir:~# rbd info ceph-dev rbd image 'ceph-dev': size 10 GiB in 2560 objects order 22 (4 MiB objects) snapshot_count: 11 id: d2f3d287f13c7b data_pool: ecpool_hdd block_name_prefix: rbd_data.7.d2f3d287f13c7b format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Wed Nov 23 17:01:03 2022 access_timestamp: Sun Apr 16 17:20:58 2023 modify_timestamp: Wed Nov 23 17:01:03 2022 root@zephir:~# Before the upgrade I was able to create snapshots of this pool: SNAPID NAME SIZE PROTECTED TIMESTAMP 1538 ceph-dev_2023-03-05T02:00:09.030+01:00 10 GiB Sun Mar 5 02:00:14 2023 1545 ceph-dev_2023-03-06T02:00:03.832+01:00 10 GiB Mon Mar 6 02:00:05 2023 1903 ceph-dev_2023-04-05T03:22:01.315+02:00 10 GiB Wed Apr 5 03:22:02 2023 1909 ceph-dev_2023-04-05T03:35:56.748+02:00 10 GiB Wed Apr 5 03:35:57 2023 1915 ceph-dev_2023-04-05T03:37:23.778+02:00 10 GiB Wed Apr 5 03:37:24 2023 1930 ceph-dev_2023-04-06T02:00:06.159+02:00 10 GiB Thu Apr 6 02:00:07 2023 1940 ceph-dev_2023-04-07T02:00:05.913+02:00 10 GiB Fri Apr 7 02:00:06 2023 1952 ceph-dev_2023-04-08T02:00:06.534+02:00 10 GiB Sat Apr 8 02:00:07 2023 1964 ceph-dev_2023-04-09T02:00:06.430+02:00 10 GiB Sun Apr 9 02:00:07 2023 2003 ceph-dev_2023-04-11T02:00:09.750+02:00 10 GiB Tue Apr 11 02:00:10 2023 2014 ceph-dev_2023-04-12T02:00:09.528+02:00 10 GiB Wed Apr 12 02:00:10 2023 root@zephir:~# I have looked through the release notes of 17.2.6 but couldn't find anything obvious regarding rbd and ec pools. Does anyone else have this problem? Do I need to change some config setting, or was this feature disabled or is it a bug? Ceph version info: root@zephir:~# ceph orch upgrade check --ceph_version 17.2.6 { "needs_update": {}, "non_ceph_image_daemons": [ "promtail.debian", "node-exporter.debian", "promtail.zephir", "grafana.zephir", "node-exporter.zephir", "prometheus.zephir", "loki.zephir", "alertmanager.zephir" ], "target_digest": " quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635", "target_id": "9cea3956c04b2d889b91b58f957577fcb4eacd3852df073e3e2567f159fcdbf8", "target_name": "quay.io/ceph/ceph:v17.2.6", "target_version": "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)", "up_to_date": [ "iscsi.rbd.debian.ijztzu", "mds.jellyfin.debian.dcsocv", "mon.debian", "osd.13", "osd.6", "mds.backups.debian.runngh", "mds.rgysi.debian.uhgqen", "crash.debian", "mgr.debian.sookxi", "iscsi.rbd.zephir.viqahd", "osd.1", "mds.jellyfin.zephir.iqywsn", "osd.12", "osd.7", "osd.2", "crash.zephir", "rgw.default.zephir.jqmick", "mds.backups.zephir.ygigch", "osd.0", "osd.4", "mon.zephir", "mgr.zephir.enywvy", "mds.rgysi.zephir.diylss", "osd.3", "osd.10", "osd.5", "osd.8", "osd.11" ] } root@zephir:~# root@zephir:~# rbd --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) root@zephir:~# Cheers Reto Gysi

5 21