- ceph-users - lists.ceph.io

Re: Unable to mount with 18.2.2
by David C. 18 Jul '24

18 Jul '24

Albert, The network is ok. However, strangely, the osd and mds did not activate msgr v2 (msgr v2 was activated on mon). It is possible to bypass by adding the "ms_mode=legacy" option but you need to find out why msgr v2 is not activated Le mar. 16 juil. 2024 à 15:18, Albert Shih <Albert.Shih(a)obspm.fr> a écrit : > Le 16/07/2024 à 15:04:05+0200, David C. a écrit > Hi, > > > > > I think it's related to your network change. > > I though about it but in that case why the old (and before upgrade) server > works ? > > > Can you send me the return of "ceph report" ? > > Nothing related to the old subnet. (see attach file) > > Regards. > > JAS > -- > Albert SHIH 🦫 🐸 > Observatoire de Paris > France > Heure locale/Local time: > mar. 16 juil. 2024 15:14:21 CEST >

4 8

Large amount of empty objects in unused cephfs data pool
by Petr Bena 18 Jul '24

18 Jul '24

I created a cephfs using mgr dashboard, which created two pools: cephfs.fs.meta and cephfs.fs.data We are using custom provisioning for user defined volumes (users provide yaml manifests with definition of what they want) which creates dedicated data pools for them, so cephfs.fs.data is never used for anything, it's literally empty despite dashboard reporting about 120kb in there (probably metadata files from new subvolume API that we are using). I wanted to decrease number of PGs in that unused cephfs.fs.data pool and to my surprise I received warning about uneved object distribution (too many objects per PG). So I dived deeper and figured out that there are in fact many objects in seemingly empty cephfs.fs.data: PROD [root@ceph-drc-mgmt ~]# rados -p cephfs.fs.data ls | head 100002a3b4a.00000000 100001a1876.00000000 100001265b5.00000000 100002af216.00000000 100004e07ec.00000000 1000053a31f.00000000 10000455214.00000000 100003e4c36.00000000 1000049e91a.00000000 100005d0bc7.00000000 When I tried dumping any of those objects - they are empty, 0 bytes each of them. But there is over 7 millions of them: PROD [root@ceph-drc-mgmt ~]# rados -p cephfs.fs.data ls | wc -l 7260394 Why is that unused pool containing 7 million empty objects? Is that some kind of bug in MDS? It's 18.2.2 Thanks

2 1

Mount cephfs
by filip Mutterer 18 Jul '24

18 Jul '24

In the official guide I found this example to mount Cephfs: mount -t ceph name@.fs_name=/ /mnt/mycephfs -o mon_addr=1.2.3.4 How to adapt to part "name@.fs_name=/" to my setup. Which command do I have to run in cephadm shell to pick up the details?

2 1

Failing update check when using quay.io mirror
by Marianne Spiller 17 Jul '24

17 Jul '24

Hi, I'm about setting up a Ceph stage cluster using `cephadm` and `podman`. As I'm not allowed to connect to the outside, our Artifactory keeps the needed quay.io packages mirrored. What I did: * put a config `/etc/containers/registries.conf.d` for `prefix=quay.io` on the first node * created an Artifactory user with permissions for the quay.io packages * bootstrapped the cluster using `registry-url`, `registry-username` and `registry-password` An `/etc/ceph/podman-auth.json` got created; the bootstrapped cluster is up & running (reef, 18.2.2). But the "New version" check does not work, and it floods the logs with messages. **Lots** of messages. **Of course**, a direct connection to quay.io will lead to a `connection refused` - that's why it should use the mirror. So: why does the update check ignore my mirror settings? And what can I do to make it work? Thanks a lot for any kind of an idea, Marianne ``` 7/17/24 7:45:47 PM [ERR] Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) Traceback (most recent call last): File "/lib/python3.6/site-packages/urllib3/connection.py", line 162, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request self._validate_conn(conn) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in _validate_conn conn.connect() File "/lib/python3.6/site-packages/urllib3/connection.py", line 315, in connect conn = self._new_conn() File "/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/registry.py", line 43, in get_tags r = requests.get(url, headers=headers) File "/lib/python3.6/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/lib/python3.6/site-packages/requests/sessions.py", line 535, in request resp = self.send(prep, **send_kwargs) File "/lib/python3.6/site-packages/requests/sessions.py", line 648, in send r = adapter.send(request, **kwargs) File "/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 283, in upgrade_ls ls = reg.get_tags(bare_image) File "/usr/share/ceph/mgr/cephadm/registry.py", line 46, in get_tags raise ValueError(msg) from e ValueError: Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 137, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3251, in upgrade_ls return self.upgrade.upgrade_ls(image, tags, show_all_versions) File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 285, in upgrade_ls raise OrchestratorError(f'{e}') orchestrator._interface.OrchestratorError: Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) ```

2 1

Multisite stuck data shard recovery after bucket deletion
by Adam Prycki 17 Jul '24

17 Jul '24

Hello, I'm testing multisite sync on reef 18.2.2, cephadm and ubuntu 22.04. Right now I'm testing symmetrical sync policy making backup to read-only zone. My sync policy allows for replication and I enable replication via put-bucket-replication. My multisite setup fails at seemingly basic operation. My test looks like this: 1. create bucket 2. upload some data to bucket 3. wait for replication to copy some of the data 4. run `rclone purge` on the bucket in master zone while replication is in progress. All data and bucket itself are deleted. I've tested this on normal secondary zone and archive zone. It seems that bucket is deleted so quickly that replication gets stuck. Buckets are gone from both zones but data sync shard still tries to replicate them Example of a recovering shard. { "shard_id": 100, "marker": { "status": "full-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" }, "pending_buckets": [ "bucket6:58642236-4f66-46f5-b863-1d6a8667c4c3.61059.5:9", "bucket6:58642236-4f66-46f5-b863-1d6a8667c4c3.61059.7:9" ], "recovering_buckets": [ "bucket6:58642236-4f66-46f5-b863-1d6a8667c4c3.61059.7:9[0]" ], "current_time": "2024-07-17T13:23:11Z" } In this case there are 2 pending buckets because I've reused the bucket name. The only semi-automatic solution I've found is to recreate bucket with the same name and wait for recovering shards to disappear. Is there any way to make ceph clean up these stuck shards automatically? Best regards Adam Prycki

1 0

Separated multisite sync and user traffic, doable?
by Szabo, Istvan (Agoda) 17 Jul '24

17 Jul '24

Hi, Could that cause any issue if the endpoints defined in the zonegroups are not in the endpoint list behind haproxy? The question is mainly about the role of the endpoint servers in the zonegroup list. Their role is the sync only or something else also? This would be the scenario, could it work? * * I have 3 mon/mgr server and 15 OSD * RGWs on the mon/mgr would be in the zonegroup definition like this "zones": [ { "id": "61c9sdf40-fdsd-4sdd-9rty9-ed56jda41817", "name": "dc", "endpoints": [ "http://mon1:8080", "http://mon2:8080", "http://mon3:8080" ], * However for user traffic I'd use an haproxy endpoint with the 15 OSD node rgws (each osd node 1x). Ty ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

RGW Multisite with a Self-Signed CA
by Alex Hussein-Kershaw (HE/HIM) 17 Jul '24

17 Jul '24

Hi ceph-users! I'm going through the process of migrating to use cephadm for my clusters. Previously I used ceph-ansible. My question is essentially "How can I configure RGW multisite with self-signed certificates with cephadm?". I have prototyped the migration and redeployed RGWs. Everything on my adopted site is running the latest version of Reef (18.2.2). The remote site is using Octopus as historically deployed with ceph-ansible. My RGWs on both sites are up and I can make requests to them, but they are failing to sync, and "radosgw-admin sync status" shows a generic input/output error. Taking some network capture I can see that the TLS handshake is failing with "Unknown CA", so it looks like the RGWs don't trust my self-signed certificate, I suppose that's not a surprise. However, I can't work out how to establish the trust. I've tried mounting in the /etc/pki directory from the machine it's running on into the RGW containers, which does contain the self-signed CA, but I still see errors in my multi-site sync. I did notice that I can curl from within the containers successfully to the remote HTTPS RGWs after this though, so it did do something. Where do the RGWs infer which CAs to trust from? I should also mention that when stepping all the RGWs down to HTTP the sync works with no issues (I was nervous about the Reef to Octopus pairing, but it seems fine). Kind regards, Alex

2 4

Scheduled cluster maintenance tasks
by Szabo, Istvan (Agoda) 17 Jul '24

17 Jul '24

Hi, Trying to figure out what is the trigger time for scrubbing and garbage collection but in the config related to these operations are not straightforward. I'm still looking for my daily laggy pg with slow ops around the same time and what I've found that some cleaning triggered around that time, however due to scrubbing running 0-24 based on config, GC running in theory hourly, lifecycle I'm not sure the schedule either, don't know what is the exact task. Haven't really found proper metrics about these jobs either. What I've tried is Frank S scrub report but that one no info about schedules, it has information about intervals how many scrubbed etc ... So if there is some proper documentation about it, please share with me. Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Node Exporter keep failing while upgrading cluster in Air-gapped ( isolated environment ).
by Saif Mohammad 16 Jul '24

16 Jul '24

Hello, We are facing an issue with node-exporter entering an error state while upgrading our cluster in an air-gapped environment. Specifically, we are upgrading from quincyv17.2.0 to reefv18.2.2. To facilitate this upgrade, we have set up a custom repository on a separate machine within the same network and pushed required images to this private repository. Here are the images that we have pushed: root@custom-registry:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 192.168.1.10:5000/ceph/ceph v18.2.2 3c937764e6f5 7 weeks ago 1.25GB 192.168.1.10:5000/ceph/ceph-grafana 9.4.7 954c08fa6188 7 months ago 633MB 192.168.1.10:5000/prometheus/prometheus v2.43.0 a07b618ecd1d 16 months ago 234MB 192.168.1.10:5000/prometheus/alertmanager v0.25.0 c8568f914cd2 19 months ago 65.1MB 192.168.1.10:5000/prometheus/node-exporter v1.5.0 0da6a335fe13 19 months ago 22.5MB Since we have configured insecure private registry, we added following lines in "/etc/containers/registries.conf" file on each node of the cluster. [[registry]] location = "192.168.1.10:5000" insecure = true Ceph image upgradation done by the command (ceph orch upgrade start --image 192.168.1.10:5000/ceph/ceph:v18.2.2) but we encountered issues from the node-exporter image, which fails to start and remains in an error state. For the remaining images ( except ceph ) we used "ceph-config" command. We are here now getting the new images values by executing the following command that seems to be default ones. root@node-01:~# ceph config get mgr mgr/cephadm/container_image_node_exporter quay.io/prometheus/node-exporter:v1.5.0 Does an internet connection is required to pull these images (Monitoring stack components images)? I am unsure because whenever we attempt to run the following command, node-exporter enters an error state and not coming into the running state even after redeploy. root@node-01:~# ceph config set mgr mgr/cephadm/container_image_node_exporter quay.io/prometheus/node-exporter:v1.5.0 Please advise the best approach to upgrade the cluster including all images in Air-gapped environment? Any guidance on resolving this issue would be appreciated. Regards, Mohammad Saif

2 3

Schödinger's OSD
by Tim Holloway 16 Jul '24

16 Jul '24

Incidentally, I just noticed that my phantom host isn't completely gone. It's not in the host list, either command-line or dashboard, but it does list (with no assets) as a host under "ceph osd tree". --- More seriously, I've been having problems with OSDs that report as being both up and down at the same time. This is on 2 new hosts. One host saw this when I made it the _admin host. The other caught it because it's running in a VM with the OSD mapped out as an imported disk and the host os managed to flip which drive was sda and which was sbd, resulting in having to delete and re- define the OSD in the VM. But now the OSD on this VM reports as "UP/IN" on the dashboard while it's "error" on "ceph orch ps" and on the actual vbox, the osd container fails on startup. viz: ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) De> Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount object store Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 ** ERROR: osd init failed: (16) Device or resource busy Note that truncated message above reads bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) Device or resource busy Rebooting doesn't help. Nor does freeing us resources and stopping/starting processes manually. The problem eventually cleared up spontaneously on the admin box, but I have no idea why. --- Also noted that now the OSD on the admin box shows in ceph orch ps as "stopped", though again, the dashboard lists it as "UP/IN". Here's what systemctl thinks about it: systemctl status ceph-osd(a)5.service ● ceph-osd(a)5.service - Ceph object storage daemon osd.5 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; preset: disabled) Active: active (running) since Fri 2024-07-12 16:45:51 EDT; 1min 40s ago Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh - -cluster ${CLUSTER} --id 5 (code=exited, status=0/SUCCESS) Main PID: 8517 (ceph-osd) Tasks: 70 Memory: 478.6M CPU: 3.405s CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd(a)5.service └─8517 /usr/bin/ceph-osd -f --cluster ceph --id 5 -- setuser ceph --setgroup ceph Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Starting Ceph object storage daemon osd.5... Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Started Ceph object storage daemon osd.5. Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to public interface Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 log_to_monitors {default=true} Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory The actual container is not running. Ceph version, incidentally, is 16.2.15. Except for that one node that apparently didn't move up from Octupus (I'll be nuking that one shortly).

3 11