January 2023 - ceph-users

Re: iscsi target lun error
by Frédéric Nass 13 Jan '23

13 Jan '23

Hi Xiubo, Randy, This is due to '<host_ip_address> host.containers.internal' being added to the container's /etc/hosts since Podman 4.1+. The workaround consists of either downgrading Podman package to v4.0 (on RHEL8, dnf downgrade podman-4.0.2-6.module+el8.6.0+14877+f643d2d6) or adding the --no-hosts option to 'podman run' command in /var/lib/ceph/$(ceph fsid)/iscsi.iscsi.test-iscsi1.xxxxxx/unit.run and restart the iscsi container service. [1] and [2] could well have the same cause. RHCS Block Device Guide [3] quotes RHEL 8.4 as a prerequisites. I don't know what was the version of Podman in RHEL 8.4 at the time, but with RHEL 8.7 and Podman 4.2, it's broken. I'll open a RHCS case today to have it fixed and have other containers like grafana, prometheus, etc. being checked against this new podman behavior. Regards, Frédéric. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1979449 [2] https://tracker.ceph.com/issues/57018 [3] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html-s… ----- Le 21 Nov 22, à 6:45, Xiubo Li xiubli(a)redhat.com a écrit : > On 15/11/2022 23:44, Randy Morgan wrote: >> You are correct I am using the cephadm to create the iscsi portals. >> The cluster had been one I was learning a lot with and I wondered if >> it was because of the number of creations and deletions of things, so >> I rebuilt the cluster, now I am getting this response even when >> creating my first iscsi target. Here is the output of the gwcli ls: >> >> sh-4.4# gwcli ls >> o- / >> ........................................................................................................................ >> [...] >> o- cluster >> ........................................................................................................ >> [Clusters: 1] >> | o- ceph >> ......................................................................................................... >> [HEALTH_WARN] >> | o- pools >> ......................................................................................................... >> [Pools: 8] >> | | o- .rgw.root >> ............................................................ [(x3), >> Commit: 0.00Y/71588776M (0%), Used: 1323b] >> | | o- cephfs_data >> .......................................................... [(x3), >> Commit: 0.00Y/71588776M (0%), Used: 1639b] >> | | o- cephfs_metadata >> ...................................................... [(x3), Commit: >> 0.00Y/71588776M (0%), Used: 3434b] >> | | o- default.rgw.control >> .................................................. [(x3), Commit: >> 0.00Y/71588776M (0%), Used: 0.00Y] >> | | o- default.rgw.log >> ...................................................... [(x3), Commit: >> 0.00Y/71588776M (0%), Used: 3702b] >> | | o- default.rgw.meta >> ...................................................... [(x3), Commit: >> 0.00Y/71588776M (0%), Used: 382b] >> | | o- device_health_metrics >> ................................................ [(x3), Commit: >> 0.00Y/71588776M (0%), Used: 0.00Y] >> | | o- rhv-ceph-ssd >> ..................................................... [(x3), Commit: >> 0.00Y/7868560896K (0%), Used: 511746b] >> | o- topology >> .............................................................................................. >> [OSDs: 36,MONs: 3] >> o- disks >> ...................................................................................................... >> [0.00Y, Disks: 0] >> o- iscsi-targets >> .............................................................................. >> [DiscoveryAuth: None, Targets: 1] >> o- iqn.2001-07.com.ceph:1668466555428 >> ............................................................... [Auth: >> None, Gateways: 1] >> o- disks >> ......................................................................................................... >> [Disks: 0] >> o- gateways >> ........................................................................................... >> [Up: 1/1, Portals: 1] >> | o- host.containers.internal >> ........................................................................ >> [192.168.105.145 (UP)] > > Please manually remove this gateway before doing further steps. > > It should be a bug in cephadm and you can raise one tracker for this. > > Thanks > > >> o- host-groups >> ................................................................................................. >> [Groups : 0] >> o- hosts >> ...................................................................................... >> [Auth: ACL_ENABLED, Hosts: 0] >> sh-4.4# >> >> Randy >> >> On 11/9/2022 6:36 PM, Xiubo Li wrote: >>> >>> On 10/11/2022 02:21, Randy Morgan wrote: >>>> I am trying to create a second iscsi target and I keep getting an >>>> error when I create the second target: >>>> >>>> >>>> Failed to update target 'iqn.2001-07.com.ceph:1667946365517' >>>> >>>> disk create/update failed on host.containers.internal. LUN >>>> allocation failure >>>> >>> I think you were using the cephadm to add the iscsi targets, not the >>> gwcli or Rest APIs directly. >>> >>> Before we hit other issues were login failures, that because there >>> were two gateways using the same IP address. Please share your `gwcli >>> ls` output to see what the 'host.containers.internal' gateway's config. >>> >>> Thanks! >>> >>> >>>> I am running ceph Pacific: *Version* >>>> 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) >>>> pacific (stable) >>>> >>>> All of the information I can find on this problem is from 3 years >>>> ago and doesn't seem to apply any more. Does anyone know how to >>>> correct this problem? >>>> >>>> Randy >>>> >>> >> > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Newer linux kernel cephfs clients is more trouble?
by Manuel Holtgrewe 13 Jan '23

13 Jan '23

Dear Xiubo, could you explain how to enable kernel debug logs (I assume this is on the client)? Thanks, Manuel On Fri, May 13, 2022 at 9:39 AM Xiubo Li <xiubli(a)redhat.com> wrote: > > On 5/12/22 12:06 AM, Stefan Kooman wrote: > > Hi List, > > > > We have quite a few linux kernel clients for CephFS. One of our > > customers has been running mainline kernels (CentOS 7 elrepo) for the > > past two years. They started out with 3.x kernels (default CentOS 7), > > but upgraded to mainline when those kernels would frequently generate > > MDS warnings like "failing to respond to capability release". That > > worked fine until 5.14 kernel. 5.14 and up would use a lot of CPU and > > *way* more bandwidth on CephFS than older kernels (order of > > magnitude). After the MDS was upgraded from Nautilus to Octopus that > > behavior is gone (comparable CPU / bandwidth usage as older kernels). > > However, the newer kernels are now the ones that give "failing to > > respond to capability release", and worse, clients get evicted > > (unresponsive as far as the MDS is concerned). Even the latest 5.17 > > kernels have that. No difference is observed between using messenger > > v1 or v2. MDS version is 15.2.16. > > Surprisingly the latest stable kernels from CentOS 7 work flawlessly > > now. Although that is good news, newer operating systems come with > > newer kernels. > > > > Does anyone else observe the same behavior with newish kernel clients? > > There have some known bugs, which have been fixed or under fixing > recently, even in the mainline and, not sure whether are they related. > Such as [1][2][3][4]. More detail please see ceph-client repo testing > branch [5]. > > I have never see the "failing to respond to capability release" issue > yet, if you have the MDS logs(debug_mds = 25 and debug_ms = 1) and > kernel debug logs will be better to help debug it further, or provide > the steps to reproduce it. > > [1] https://tracker.ceph.com/issues/55332 > [2] https://tracker.ceph.com/issues/55421 > [3] https://bugzilla.redhat.com/show_bug.cgi?id=2063929 > [4] https://tracker.ceph.com/issues/55377 > [5] https://github.com/ceph/ceph-client/commits/testing > > Thanks > > -- Xiubo > > > > > Gr. Stefan > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

BlueFS spillover warning gone after upgrade to Quincy
by Peter van Heusden 13 Jan '23

13 Jan '23

Hello everyone I have a Ceph installation where some of the OSDs were misconfigured to use 1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS *spillover* detected"). I recently upgraded to quincy using cephadm (17.2.5) the spillover warning vanished. This is despite bluestore_warn_on_bluefs_spillover still being set to true. Is there a way to investigate the current state of the DB to see if spillover is, indeed, still happening? Thank you, Peter

5 5

heavy rotation in store.db folder alongside with traces and exceptions in the .log
by Jürgen Stawska 13 Jan '23

13 Jan '23

Hi everyone, I'm facing a weird issue with one of my pacific clusters. Brief into: - 5 Nodes Ubuntu 20.04. on 16.2.7 ( ceph01…05 ) - bootstrapped with cephadm recent image from quay.io (around 1 year ago) - approx. 200TB capacity 5% used - 5 OSD (2 HDD / 2 SSD / 1 NVMe) on each node - each node has a MON, yeah 5 MONs in charge - 3 RGW - 2 MGR - 3 MDS (2 active and 1 stby) The cluster is serving S3 files and cephFS for k8s PVCs and is doing very well. But: During a regular maintenance I found a heavy rotating store.db on EVERY node. Taking a further look, I found weird stuff going on in the #####.log The log is growing with a rate of approx. 400k/s and is rotating when reaching a certain size. store.db -rw-r--r-- 1 ceph ceph 11445745 Jan 13 09:53 1546576.log -rw-r--r-- 1 ceph ceph 67352998 Jan 13 09:53 1546578.sst -rw-r--r-- 1 ceph ceph 67349926 Jan 13 09:53 1546579.sst -rw-r--r-- 1 ceph ceph 67363989 Jan 13 09:53 1546580.sst -rw-r--r-- 1 ceph ceph 41063487 Jan 13 09:53 1546581.sst executing refresh((['ceph01', 'ceph02', 'ceph03', 'ceph04', 'ceph05'],)) failed. Traceback (most recent call last): File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 48, in bootstrap_exec s = io.read(1) File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 402, in read raise EOFError("expected %d bytes, got %d" % (numbytes, len(buf))) EOFError: expected 1 bytes, got 0 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1357, in _remote_connection conn, connr = self.mgr._get_connection(addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1340, in _get_connection sudo=True if self.ssh_user != 'root' else False) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 35, in __init__ self.gateway = self._make_gateway(hostname) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 46, in _make_gateway self._make_connection_string(hostname) File "/lib/python3.6/site-packages/execnet/multi.py", line 134, in makegateway gw = gateway_bootstrap.bootstrap(io, spec) File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 102, in bootstrap bootstrap_exec(io, spec) File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 53, in bootstrap_exec raise HostNotFound(io.remoteaddress) execnet.gateway_bootstrap.HostNotFound: -F /tmp/cephadm-conf-6p_ae5op -i /tmp/cephadm-identity-hc1rt28x ubuntuadmin@<< IP_OF_CEPH-01 REPLACED >> The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/serve.py", line 312, in refresh with self._remote_connection(host) as tpl: File "/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1391, in _remote_connection raise OrchestratorError(msg) from e orchestrator._interface.OrchestratorError: Failed to connect to ceph01 << IP_OF_CEPH-01 REPLACED >>). Please make sure that the host is reachable and accepts connections using the cephadm SSH key ... ... [some binary stuff here] … ... ceph01.sjtrntß$Skd???>ö#?c????Z+Removing orphan daemon mds.cephfs.ceph02…cephadm ceph01.sjtrntß$Skd???>ö#?cXx??Z-Removing daemon mds.cephfs.ceph02 from ceph01cephadm ceph01.sjtrntß$Skd???>_#?cԕ?0?Z"Removing key for mds.cephfs.ceph02cephadm ceph01.sjtrntß$Skd???>_#?cUƾ0?Z=Reconfiguring mds.cephfs.ceph02 (unknown last config time)...cephadm ceph01.sjtrntß$Skd???>_#?cE?"2?Z0Reconfiguring daemon mds.cephfs.ceph02 on ceph01cephadm ceph01.sjtrntß$Skd???>`#?c??&?Zcephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format ää.State.Status¨¨ ceph-<<cluster-ID REPLACED>>-mds-cephfs-ceph02 /usr/bin/docker: stdout /usr/bin/docker: stderr Error: No such container: ceph-<<cluster-ID REPLACED>>-mds-cephfs-ceph02 Non-zero exit code 1 from /usr/bin/docker container inspect --format ää.State.Status¨¨ ceph-<<cluster-ID REPLACED>>-mds.cephfs.ceph02 /usr/bin/docker: stdout /usr/bin/docker: stderr Error: No such container: ceph-<<cluster-ID REPLACED>>-mds.cephfs.ceph02 Reconfig daemon mds.cephfs.ceph02 ... ERROR: cannot reconfig, data path /var/lib/ceph/<<cluster-ID REPLACED>>/mds.cephfs.ceph02 does not exist Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1363, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1256, in _run_cephadm code, 'ön'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format ää.State.Status¨¨ ceph-<<cluster-ID REPLACED>>-mds-cephfs-ceph02 /usr/bin/docker: stdout /usr/bin/docker: stderr Error: No such container: ceph-<<cluster-ID REPLACED>>-mds-cephfs-ceph02 Non-zero exit code 1 from /usr/bin/docker container inspect --format ää.State.Status¨¨ ceph-<<cluster-ID REPLACED>>-mds.cephfs.ceph02 /usr/bin/docker: stdout /usr/bin/docker: stderr Error: No such container: ceph-<<cluster-ID REPLACED>>-mds.cephfs.ceph02 Reconfig daemon mds.cephfs.ceph02 ... ERROR: cannot reconfig, data path /var/lib/ceph/<<cluster-ID REPLACED>>/mds.cephfs.ceph02 does not existcephadm Unable to add a Daemon without Service. ?t Please use `ceph orch apply ...` to create a Service. Note, you might want to create the service with "unmanaged=true" Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2440, in add_daemon ret.extend(self._add_daemon(d_type, spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2378, in _add_daemon raise OrchestratorError('Unable to add a Daemon without Service.\n' orchestrator._interface.OrchestratorError: Unable to add a Daemon without Service. Please use `ceph orch apply ...` to create a Service. I’m confused about the attempt of cephadm to do „things“ to a ceph02 daemon which is obviously not residing on node ceph01. Almost the same log lines are appearing on each MON host in its store.db. All in all it looks fare from healthy and I’m really concerned about that. Any help is highly appreciated! Thanks a lot. Cheers, Jürgen

1 0

Re: RGW error Coundn't init storage provider (RADOS)
by Alexander Y. Fomichev 13 Jan '23

13 Jan '23

Hi I facing similar error a couple of days ago: radosgw-admin --cluster=cl00 realm create --rgw-realm=data00 --default ... (0 rgw main: rgw_init_ioctx ERROR: librados::Rados::pool_create returned (34) Numerical result out of range (this can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd exceeded) ... obviously radosgw-admin unable to create pool .rgw.root (at the same time "ceph pool create" works as expected) Crowling on a mon logs with debug=20 leads to record: "... prepare_new_pool got -34 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1" As for me pg_num=1 looks strange because default value of osd_pool_default_pg_num=32. On the other side default osd_pool_default_pgp_num=0 so I tried to set osd_pool_default_pgp_num=1 and it worked: pool .rgw.root was built. What really looks strange, after first success I can't reproduce it any more. After that "radosgw-admin ... realm create" successfully builds .rgw.root even with osd_pool_default_pgp_num=0.Nevertheless I suspect a record "pgp_num must be greater than 0 and lower or equal than 'pg_num', which in this case is 1" points to existing bug. It looks like default values of osd_pool_default_pg[p]_num somway ignored/omitted. On Tue, Jul 19, 2022 at 9:11 AM Robert Reihs <robert.reihs(a)gmail.com> wrote: > Yes, I checked pg_num, pgp_num and mon_max_pg_per_osd. I also setup a > single node cluster with the same ansible script we have. Using cephadm for > setting um and managing the cluster. I had the same problem on the new > single node cluster without setup of any other services. When I created the > pools manually the service started and also the dashboard connection > directly worked. > > On Mon, Jul 18, 2022 at 10:20 AM Janne Johansson <icepic.dz(a)gmail.com> > wrote: > > > No, rgw should have the ability to create its own pools. Check the caps > on > > tve keys used by the rgw daemon. > > > > Den mån 18 juli 2022 09:59Robert Reihs <robert.reihs(a)gmail.com> skrev: > > > >> Hi, > >> I had to manually create the pools, than the service automatically > started > >> and is now available. > >> pools: > >> .rgw.root > >> default.rgw.log > >> default.rgw.control > >> default.rgw.meta > >> default.rgw.buckets.index > >> default.rgw.buckets.data > >> default.rgw.buckets.non-ec > >> > >> Is this normal behavior? Should then the error message be changed? Or is > >> this a bug? > >> Best > >> Robert Reihs > >> > >> > >> On Fri, Jul 15, 2022 at 3:47 PM Robert Reihs <robert.reihs(a)gmail.com> > >> wrote: > >> > >> > Hi, > >> > When I have no luck yet solving the issue, but I can add some > >> > more information. The system pools ".rgw.root" and "default.rgw.log" > are > >> > not created. I have created them manually, Now there is more log > >> activity, > >> > but still getting the same error message in the log: > >> > rgw main: rgw_init_ioctx ERROR: librados::Rados::pool_create returned > >> (34) > >> > Numerical result out of range (this can be due to a pool or placement > >> group > >> > misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd > exceeded) > >> > I can't find the correct pool to create manually. > >> > Thanks for any help > >> > Best > >> > Robert > >> > > >> > On Tue, Jul 12, 2022 at 5:22 PM Robert Reihs <robert.reihs(a)gmail.com> > >> > wrote: > >> > > >> >> Hi, > >> >> > >> >> We have a problem with deloing radosgw vi cephadm. We have a Ceph > >> cluster > >> >> with 3 nodes deployed via cephadm. Pool creation, cephfs and block > >> storage > >> >> are working. > >> >> > >> >> ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy > >> >> (stable) > >> >> > >> >> The service specs is like this for the rgw: > >> >> > >> >> --- > >> >> > >> >> service_type: rgw > >> >> > >> >> service_id: rgw > >> >> > >> >> placement: > >> >> > >> >> count: 3 > >> >> > >> >> label: "rgw" > >> >> > >> >> --- > >> >> > >> >> service_type: ingress > >> >> > >> >> service_id: rgw.rgw > >> >> > >> >> placement: > >> >> > >> >> count: 3 > >> >> > >> >> label: "ingress" > >> >> > >> >> spec: > >> >> > >> >> backend_service: rgw.rgw > >> >> > >> >> virtual_ip: [IPV6] > >> >> > >> >> virtual_interface_networks: [IPV6 CIDR] > >> >> > >> >> frontend_port: 8080 > >> >> > >> >> monitor_port: 1967 > >> >> > >> >> The error I get in the logfiles: > >> >> > >> >> 0 deferred set uid:gid to 167:167 (ceph:ceph) > >> >> > >> >> 0 ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) > quincy > >> >> (stable), process radosgw, pid 2 > >> >> > >> >> 0 framework: beast > >> >> > >> >> 0 framework conf key: port, val: 80 > >> >> > >> >> 1 radosgw_Main not setting numa affinity > >> >> > >> >> 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 > >> >> > >> >> 1 D3N datacache enabled: 0 > >> >> > >> >> 0 rgw main: rgw_init_ioctx ERROR: librados::Rados::pool_create > returned > >> >> (34) Numerical result out of range (this can be due to a pool or > >> placement > >> >> group misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd > >> >> exceeded) > >> >> > >> >> 0 rgw main: failed reading realm info: ret -34 (34) Numerical result > >> out > >> >> of range > >> >> > >> >> 0 rgw main: ERROR: failed to start notify service ((34) Numerical > >> result > >> >> out of range > >> >> > >> >> 0 rgw main: ERROR: failed to init services (ret=(34) Numerical result > >> out > >> >> of range) > >> >> > >> >> -1 Couldn't init storage provider (RADOS) > >> >> > >> >> I have for testing set the pg_num and pgp_num to 16 and the > >> >> mon_max_pg_per_osd to 1000 and still getting the same error. I have > >> also > >> >> tried creating the rgw with ceph command, same error. Pool creation > is > >> >> working, I created multiple other pools and there was no problem. > >> >> > >> >> Thanks for any help. > >> >> > >> >> Best > >> >> > >> >> Robert > >> >> > >> >> The 5 fails services are 3 from the rgw and 2 haproxy for the rgw, > >> there > >> >> is only one running: > >> >> > >> >> ceph -s > >> >> > >> >> cluster: > >> >> > >> >> id: 40ddf > >> >> > >> >> health: HEALTH_WARN > >> >> > >> >> 5 failed cephadm daemon(s) > >> >> > >> >> > >> >> > >> >> services: > >> >> > >> >> mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 (age 4d) > >> >> > >> >> mgr: ceph-01.hbvyqi(active, since 4d), standbys: ceph-02.pqtxbv > >> >> > >> >> mds: 1/1 daemons up, 3 standby > >> >> > >> >> osd: 6 osds: 6 up (since 4d), 6 in (since 4d) > >> >> > >> >> > >> >> > >> >> data: > >> >> > >> >> volumes: 1/1 healthy > >> >> > >> >> pools: 5 pools, 65 pgs > >> >> > >> >> objects: 87 objects, 170 MiB > >> >> > >> >> usage: 1.4 GiB used, 19 TiB / 19 TiB avail > >> >> > >> >> pgs: 65 active+clean > >> >> > >> >> > >> > > >> > -- > >> > Robert Reihs > >> > Jakobsweg 22 > >> > 8046 Stattegg > >> > AUSTRIA > >> > > >> > mobile: +43 (664) 51 035 90 > >> > robert.reihs(a)gmail.com > >> > > >> > >> > >> -- > >> Robert Reihs > >> Jakobsweg 22 > >> 8046 Stattegg > >> AUSTRIA > >> > >> mobile: +43 (664) 51 035 90 > >> robert.reihs(a)gmail.com > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > > -- > Robert Reihs > Jakobsweg 22 > 8046 Stattegg > AUSTRIA > > mobile: +43 (664) 51 035 90 > robert.reihs(a)gmail.com > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Best regards. Alexander Y. Fomichev <git.user(a)gmail.com>

1 0

OSDs failed to start after host reboot | Cephadm
by Ben Meinhart 12 Jan '23

12 Jan '23

Hello all! Linked stackoverflow post: https://stackoverflow.com/questions/75101087/cephadm-ceph-osd-fails-to-star… <https://stackoverflow.com/questions/75101087/cephadm-ceph-osd-fails-to-star…> A couple of weeks ago I deployed a new Ceph cluster using Cephadm. It is a three node cluster (node1, node2, & node3) with 6 OSD’s each; 6x18TB Seagate hard drives with a 2TB NVMe drive set as a DB device. Everything has been running smoothly until today when I went to perform maintenance on one of the nodes. I first moved all of the services off the host and put it into maintenance mode. I then made some changes to once of the NIC’s and ran updates. After the updates were done, I rebooted the machine. This is when the issue occurred. When the node (node1) finished rebooting, it was still showing as offline in the Ceph Dashboard so from one of the host I ran `ceph orch host rescan node1` and it came back online in the Ceph dashboard. I’ve seen this before when I’ve had to reboot host so NBD so far. However, after a couple of minutes passed the OSD’s on that host still haven’t come online. I then checked the status of the services `systemctl | grep ceph` and saw that all of the OSD’s had failed. # systemctl status ceph-0a7ec2ae-816d-11ed-9791-97c1d8fb9dc6(a)osd.0.service × ceph-0a7ec2ae-816d-11ed-9791-97c1d8fb9dc6(a)osd.0.service - Ceph osd.0 for 0a7ec2ae-816d-11ed-9791-97c1d8fb9dc6 Loaded: loaded (/etc/systemd/system/ceph-0a7ec2ae-816d-11ed-9791-97c1d8fb9dc6@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2023-01-12 18:14:27 UTC; 1h 42min ago Main PID: 385982 (code=exited, status=1/FAILURE) CPU: 292ms Jan 12 19:48:30 node1 systemd[1]: /etc/systemd/system/ceph-0a7ec2ae-816d-11ed-9791-97c1d8fb9dc6@.service:24: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer Kill It was at the reset counter max so I had to run `systemctl reset-failed` and I tried restarting the OSD’s by running `systemctl restart ceph.target`. I watched the service try to load but it kept failing. This was the output of /var/log/ceph/<fsid>/ceph-osd.0.log: 2023-01-12T18:12:06.501+0000 7fb5d3b1e3c0 0 set uid:gid to 167:167 (ceph:ceph) 2023-01-12T18:12:06.501+0000 7fb5d3b1e3c0 0 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process ceph-osd, pid 7 2023-01-12T18:12:06.501+0000 7fb5d3b1e3c0 0 pidfile_write: ignore empty --pid-file 2023-01-12T18:12:06.505+0000 7fb5d3b1e3c0 1 bdev(0x5591e1f87400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2023-01-12T18:12:06.505+0000 7fb5d3b1e3c0 1 bdev(0x5591e1f87400 /var/lib/ceph/osd/ceph-0/block) open size 20000584761344 (0x1230bfc00000, 18 TiB) block_size 4096 (4 KiB) rotational discard not supported 2023-01-12T18:12:06.505+0000 7fb5d3b1e3c0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-01-12T18:12:06.505+0000 7fb5d3b1e3c0 1 bdev(0x5591e1f86c00 /var/lib/ceph/osd/ceph-0/block.db) open path /var/lib/ceph/osd/ceph-0/block.db 2023-01-12T18:12:06.505+0000 7fb5d3b1e3c0 1 bdev(0x5591e1f86c00 /var/lib/ceph/osd/ceph-0/block.db) open size 333396836352 (0x4da0000000, 310 GiB) block_size 4096 (4 KiB) non-rotational discard supported 2023-01-12T18:12:06.505+0000 7fb5d3b1e3c0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block.db size 310 GiB 2023-01-12T18:12:06.513+0000 7fb5d3b1e3c0 1 bdev(0x5591e1f86800 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2023-01-12T18:12:06.513+0000 7fb5d3b1e3c0 1 bdev(0x5591e1f86800 /var/lib/ceph/osd/ceph-0/block) open size 20000584761344 (0x1230bfc00000, 18 TiB) block_size 4096 (4 KiB) rotational discard not supported 2023-01-12T18:12:06.513+0000 7fb5d3b1e3c0 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-0/block size 18 TiB 2023-01-12T18:12:06.513+0000 7fb5d3b1e3c0 1 bdev(0x5591e1f86c00 /var/lib/ceph/osd/ceph-0/block.db) close 2023-01-12T18:12:06.817+0000 7fb5d3b1e3c0 1 bdev(0x5591e1f86800 /var/lib/ceph/osd/ceph-0/block) close 2023-01-12T18:12:07.085+0000 7fb5d3b1e3c0 1 bdev(0x5591e1f87400 /var/lib/ceph/osd/ceph-0/block) close 2023-01-12T18:12:07.305+0000 7fb5d3b1e3c0 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 0 load: jerasure load: lrc 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 -1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 -1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 mClockScheduler: set_max_osd_capacity #op shards: 5 max osd capacity(iops) per shard: 863.20 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0250000 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000052 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 0 osd.0:0.OSDShard using op scheduler mClockScheduler 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 -1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 mClockScheduler: set_max_osd_capacity #op shards: 5 max osd capacity(iops) per shard: 863.20 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0250000 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000052 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 0 osd.0:1.OSDShard using op scheduler mClockScheduler 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2023-01-12T18:12:07.321+0000 7fb5d3b1e3c0 -1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_max_osd_capacity #op shards: 5 max osd capacity(iops) per shard: 863.20 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0250000 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000052 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 0 osd.0:2.OSDShard using op scheduler mClockScheduler 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 -1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_max_osd_capacity #op shards: 5 max osd capacity(iops) per shard: 863.20 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0250000 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000052 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 0 osd.0:3.OSDShard using op scheduler mClockScheduler 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 -1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_max_osd_capacity #op shards: 5 max osd capacity(iops) per shard: 863.20 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_osd_mclock_cost_per_io osd_mclock_cost_per_io: 0.0250000 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_osd_mclock_cost_per_byte osd_mclock_cost_per_byte: 0.0000052 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 mClockScheduler: set_mclock_profile mclock profile: high_client_ops 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 0 osd.0:4.OSDShard using op scheduler mClockScheduler 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 -1 bdev(0x5591e2d8e000 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 -1 osd.0 0 OSD:init: unable to mount object store 2023-01-12T18:12:07.325+0000 7fb5d3b1e3c0 -1 [0;31m ** ERROR: osd init failed: (13) Permission denied[0m Judging by the final error, it looked like some sort of permissions issue with mounting the volume to the container. I did notice on the other two host, node2 & node3, that I have not yet reboot since deploying Ceph with cephadm that it was more docker overlays mounted when I ran the `mount` command. My theory is that the LVM volume stored on the OSD’s is not being mounted at boot. Otherwise it might also be the case that the user that Ceph is passing to the containers is not allowed to mount the volumes for some reason. I’ve looked through most of the docs and forums I could find and haven’t found any solutions. I would like to say I’m fairly experienced with Linux 5+ years, but I am new to Ceph (~6 months) and I haven’t emailed this list before. Sorry in advance if I’ve mistakenly broken any roles and thanks for the help! - Ben M

1 0

Laggy PGs on a fairly high performance cluster
by Matthew Stroud 12 Jan '23

12 Jan '23

We have a 14 osd node all ssd cluster and for some reason we are continually getting laggy PGs and those seem to correlate to slow requests on Quincy (doesn't seem to happen on our Pacific clusters). These laggy pgs seem to shift between osds. The network seems solid, as in I'm not seeing errors or slowness. OSD hosts are heavily underutilized, normally sub 1 load and the cpus are 98% idle. I have been looking through the logs and nothing is really standing out in the OSD or ceph logs. Some things we have tried: 1. Updating our cluster to 17.2.5 2. Manually setting our mClock profile to high_client_ops. 3. Increasing our total number of PGs (this something that should've happened anyways.) 4. Verified that jumbo frames, lacp, and throughput were functioning as intended. 5. Took some of our newer nodes out to see if that was an issue. Also rebooted the cluster just to be sure. I'm curious if someone in the community has experience with this kind of issue and maybe could point to something I have overlooked. Some example logs: 2023-01-10T22:50:23.245823+0000 mgr.openstack-mon01.b.pc.ostk.com.flbudm (mgr.120371640) 231175 : cluster [DBG] pgmap v235204: 2625 pgs: 1 active+clean+laggy, 2624 active+clean; 6.0 TiB data, 18 TiB used, 84 TiB / 102 TiB avail; 19 MiB/s rd, 67 MiB/s wr, 4.76k op/s 2023-01-10T22:50:23.762562+0000 osd.83 (osd.83) 906 : cluster [WRN] 6 slow requests (by type [ 'delayed' : 5 'waiting for sub ops' : 1 ] most affected pool [ 'vms' : 6 ]) 2023-01-10T22:50:24.771260+0000 osd.83 (osd.83) 907 : cluster [WRN] 6 slow requests (by type [ 'delayed' : 5 'waiting for sub ops' : 1 ] most affected pool [ 'vms' : 6 ]) ________________________________ CONFIDENTIALITY NOTICE: This message is intended only for the use and review of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message solely to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately by telephone or return email. Thank you.

1 0

CephFS: Questions regarding Namespaces, Subvolumes and Mirroring
by Jonas Schwab 12 Jan '23

12 Jan '23

Dear everyone, I have several questions regarding CephFS connected to Namespaces, Subvolumes and snapshot Mirroring: *1. How to display/create namespaces used for isolating subvolumes?* I have created multiple subvolumes with the option --namespace-isolated, so I was expecting to see the namespaces returned from ceph fs subvolume info <volume_name> <subvolume_name> also returned by rbd namespace ls <cephfs_data_pool> --format=json But the latter command just returns an empty list. Are the namespaces used for rdb and CephFS different ones? *2. Can CephFS Snapshot mirroring also be applied to subvolumes?* I tried this, but without success. Is there something to take into account rather than just mirroring the directory, or is it just not possible right now? *3. Can xattr for namespaces and pools also be mirrored?* Or more specifically, is there a way to preserve the namespace and pool layout of mirrored directories? Thank you for your help! Best regrads, Jonas PS: You could receive this mail twice, sine this email address somehow got removed from the ceph-users list.

1 0

CephFS: Questions regarding Namespaces, Subvolumes and Mirroring
by Jonas Schwab 12 Jan '23

12 Jan '23

Dear everyone, I have several questions regarding CephFS connected to Namespaces, Subvolumes and snapshot Mirroring: *1. How to display/create namespaces used for isolating subvolumes?* I have created multiple subvolumes with the option --namespace-isolated, so I was expecting to see the namespaces returned from ceph fs subvolume info <volume_name> <subvolume_name> also returned by rbd namespace ls <cephfs_data_pool> --format=json But the latter command just returns an empty list. Are the namespaces used for rdb and CephFS different ones? *2. Can CephFS Snapshot mirroring also be applied to subvolumes?* I tried this, but without success. Is there something to take into account rather than just mirroring the directory, or is it just not possible right now? *3. Can xattr for namespaces and pools also be mirrored?* Or more specifically, is there a way to preserve the namespace and pool layout of mirrored directories? Thank you for your help! Best regrads, Jonas

2 1

Creating nfs RGW export makes nfs-gnaesha server in crash loop
by Ruidong Gao 12 Jan '23

12 Jan '23

Hi, This is running Quincy 17.2.5 deployed by rook on k8s. RGW nfs export will crash Ganesha server pod. CephFS export works just fine. Here are steps of it: 1, create export: bash-4.4$ ceph nfs export create rgw --cluster-id nfs4rgw --pseudo-path /bucketexport --bucket testbk { "bind": "/bucketexport", "path": "testbk", "cluster": "nfs4rgw", "mode": "RW", "squash": "none" } 2, check pods status afterwards: rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running 0 4h3m rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 1/2 Error 2 4h6m 3, check failing pod’s logs: 11/01/2023 08:11:53 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 11/01/2023 08:11:54 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 11/01/2023 08:11:54 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 11/01/2023 08:11:57 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE 11/01/2023 08:11:57 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] export_defaults_commit :CONFIG :INFO :Export Defaults now (options=03303002/00080000 , , , , , , , , expire= 0) 2023-01-11T08:11:57.853+0000 7f59dac7c200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory 2023-01-11T08:11:57.853+0000 7f59dac7c200 -1 AuthRegistry(0x56476817a480) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx 2023-01-11T08:11:57.855+0000 7f59dac7c200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory 2023-01-11T08:11:57.855+0000 7f59dac7c200 -1 AuthRegistry(0x7ffe4d092c90) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx 2023-01-11T08:11:57.856+0000 7f5987537700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:11:57.856+0000 7f5986535700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:12:00.861+0000 7f5986d36700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:12:00.861+0000 7f59dac7c200 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication failed to fetch mon config (--no-mon-config to skip) 4, delete the export: ceph nfs export delete nfs4rgw /bucketexport Ganesha servers go back normal: rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running 0 4h30m rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 2/2 Running 10 4h33m Any ideas to make it work? Thanks Ben

2 2

Mysterious Disk-Space Eater
by duluxoz 12 Jan '23

12 Jan '23

Hi All, Got a funny one, which I'm hoping someone can help us with. We've got three identical(?) Ceph Quincy Nodes running on Rocky Linux 8.7. Each Node has 4 OSDs, plus Monitor, Manager, and iSCSI G/W services running on them (we're only a small shop). Each Node has a separate 16 GiB partition mounted as /var. Everything is running well and the Ceph Cluster is handling things very well). However, one of the Nodes (not the one currently acting as the Active Manager) is running out of space on /var. Normally, all of the Nodes have around 10% space used (via a df -H command), but the problem Node only takes 1 to 3 days to run out of space, hence taking it out of Quorum. Its currently at 85% and growing. At first we thought this was caused by an overly large log file, but investigations showed that all the logs on all 3 Nodes were of comparable size. Also, searching for the 20 largest files on the problem Node's /var didn't produce any significant results. Coincidentally, unrelated to this issue, the problem Node (but not the other 2 Nodes) was re-booted a couple of days ago and, when the Cluster had re-balanced itself and everything was back online and reporting as Healthy, the problem Node's /var was back down to around 10%, the same as the other two Nodes. This lead us to suspect that there was some sort of "run-away" process or journaling/logging/temporary file(s) or whatever that the re-boot has "cleaned up". So we've been keeping an eye on things but we can't see anything causing the issue and now, as I said above, the problem Node's /var is back up to 85% and growing. I've been looking at the log files, tying to determine the issue, but as I don't really know what I'm looking for I don't even know if I'm looking in the *correct* log files... Obviously rebooting the problem Node every couple of days is not a viable option, and increasing the size of the /var partition is only going to postpone the issue, not resolve it. So if anyone has any ideas we'd love to hear about it - thanks Cheers Dulux-Oz

4 3

Ceph Octopus rbd images stuck in trash
by Jeff Welling 12 Jan '23

12 Jan '23

Hello there, I'm running Ceph 15.2.17 (Octopus) on Debian Buster and I'm starting an upgrade but I'm seeing a problem and I wanted to ask how best to proceed in case I make things worse by mucking with it without asking experts. I've moved an rbd image to the trash without clearing the snapshots first, and then tried to 'trash purge'. This resulted in an error because the image still has snapshots, but I'm unable to remove the image from the pool to clear the snapshots either. At least one of these images is from a clone of a snapshot from another trashed image, which I'm already kicking myself for. The contents of my trash: # rbd trash ls 07afadac0ed69c nfsroot_pi08 240ae5a5eb3214 bigdisk 7fd5138848231e nfsroot_pi01 f33e1f5bad0952 bigdisk2 fcdeb1f96a6124 raspios-64bit-lite-manuallysetup-p1 fcdebd2237697a raspios-64bit-lite-manuallysetup-p2 fd51418d5c43da nfsroot_pi02 fd514a6b4d3441 nfsroot_pi03 fd515061816c70 nfsroot_pi04 fd51566859250b nfsroot_pi05 fd5162c5885d9c nfsroot_pi07 fd5171c27c36c2 nfsroot_pi09 fd51743cb8813c nfsroot_pi10 fd517ad3bc3c9d nfsroot_pi11 fd5183bfb1e588 nfsroot_pi12 This is the error I get trying to purge the trash: # rbd trash purge Removing images: 0% complete...failed. rbd: some expired images could not be removed Ensure that they are closed/unmapped, do not have snapshots (including trashed snapshots with linked clones), are not in a group and were moved to the trash successfully. This is the error when I try and restore one of the trashed images: # rbd trash restore nfsroot_pi08 rbd: error: image does not exist in trash 2023-01-11T12:28:52.982-0800 7f4b69a7c3c0 -1 librbd::api::Trash: restore: error getting image id nfsroot_pi08 info from trash: (2) No such file or directory Trying to restore other images gives the same error. These trash images are now taking up a significant portion of the cluster space. One thought was to upgrade and see if that resolves the problem, but I've shot myself in the foot doing that in the past without confirming it would solve the problem, so I'm looking for a second opinion on how best to clear these? These are all Debian Buster systems, the kernel version of the host I'm running these commands on is: Linux zim 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27) x86_64 GNU/Linux I'm going to be upgrading that too but one step at a time. The exact ceph version is: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) This was installed from the ceph repos, not the debian repos, using cephadm. If there's any additional details I can share please let me know, any and all thoughts welcome! I've been googling and have found folks with similar issues but nothing similar enough to feel helpful. Thanks in advance, and thank you to any and everyone who contributes to Ceph, it's awesome!

2 1

rbd-mirror | ceph quincy Not able to find rbd_mirror_journal_max_fetch_bytes config in rbd mirror
by ankit raikwar 12 Jan '23

12 Jan '23

Hello All, In the ceph quincy Not able to find rbd_mirror_journal_max_fetch_bytes config in rbd mirror i configured the ceph cluster almost 400 tb and enable the rbd-mirror in the starting stage i'm able to achive the almost 9 GB speed , but after the rebalane completed of the all the images . rbd-mirror speed got automaticily reduce to between 4 to 5 mbps. in my primary cluster we are continuelsy writing the 50 to 400 mbps data but replication speed only we get the 4 to 5 mbps. also we have the 10 Gbps replication network bandwidth. Note::- I also try to find the option rbd_mirror_journal_max_fetch_bytes but i'm not able to find the this option in the configuration. also when i try to set from the command line it's showing error like command: ceph config set client.rbd rbd_mirror_journal_max_fetch_bytes 33554432 error: Error EINVAL: unrecognized config option 'rbd_mirror_journal_max_fetch_bytes' Cluster version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Please suggest any alternative way to configurre this option or how i can improve the replication n/w speed. 0 0 Reply

1 0

nfs RGW export makes nfs-gnaesha server in crash loop
by Ben Gao 12 Jan '23

12 Jan '23

Hi, This is running Quincy 17.2.5 deployed by rook on k8s. RGW nfs export will crash Ganesha server pod. CephFS export works just fine. Here are steps of it: 1, create export: bash-4.4$ ceph nfs export create rgw --cluster-id nfs4rgw --pseudo-path /bucketexport --bucket testbk { "bind": "/bucketexport", "path": "testbk", "cluster": "nfs4rgw", "mode": "RW", "squash": "none" } 2, check pods status afterwards: rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running 0 4h3m rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 1/2 Error 2 4h6m 3, check failing pod’s logs: 11/01/2023 08:11:53 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 11/01/2023 08:11:54 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 11/01/2023 08:11:54 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 11/01/2023 08:11:57 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE 11/01/2023 08:11:57 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] export_defaults_commit :CONFIG :INFO :Export Defaults now (options=03303002/00080000 , , , , , , , , expire= 0) 2023-01-11T08:11:57.853+0000 7f59dac7c200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory 2023-01-11T08:11:57.853+0000 7f59dac7c200 -1 AuthRegistry(0x56476817a480) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx 2023-01-11T08:11:57.855+0000 7f59dac7c200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory 2023-01-11T08:11:57.855+0000 7f59dac7c200 -1 AuthRegistry(0x7ffe4d092c90) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx 2023-01-11T08:11:57.856+0000 7f5987537700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:11:57.856+0000 7f5986535700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:12:00.861+0000 7f5986d36700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:12:00.861+0000 7f59dac7c200 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication failed to fetch mon config (--no-mon-config to skip) 4, delete the export: ceph nfs export delete nfs4rgw /bucketexport Ganesha servers go back normal: rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running 0 4h30m rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 2/2 Running 10 4h33m Any ideas to make it work? Thanks Ben

1 0

Removing OSDs - draining but never completes.
by Wyll Ingersoll 12 Jan '23

12 Jan '23

Running ceph-pacific 16.2.9 using ceph orchestrator. We made a mistake adding a disk to the cluster and immediately issued a command to remove it using "ceph orch osd rm ### --replace --force". This OSD had no data on it at the time and was removed after just a few minutes. "ceph orch osd rm status" shows that it is still "draining". ceph osd df shows that the osd being removed has -1 PGs. So - why is the simple act of removal taking so long and can we abort it and manually remove that osd somehow? Note: the cluster is also doing a rebalance while this is going on, but the osd being removed never had any data and should not be affected by the rebalance. thanks!

2 1

[rgw] Upload object with bad performance after the cluster running few months
by can zhu 12 Jan '23

12 Jan '23

ceph version is: 16.2.10 Use “rclone” tools to upload the big object: rclone copy ./zh-cn_windows_10_business_editions_version_21h1_updated_jul_2021_x64_dvd_f49026f5.iso smd:test --progres Transferred: 670 MiB / 5.293 GiB, 12%, 15.195 MiB/s, ETA 5m12s Transferred: 0 / 1, 0% Elapsed time: 46.0s Transferring: * zh-cn_windows_10_busin…1_x64_dvd_f49026f5.iso: 12% /5.293Gi, 15.195Mi/s, 5m12s The speed is: 15MB/s But at the beginning the speed can reach 100MB/s

1 0

Move bucket between realms
by mahnoosh shahidi 11 Jan '23

11 Jan '23

Hi all, Is there any way in rgw to move a bucket from one realm to another one in the same cluster? Best regards, Mahnoosh

1 0

adding OSD to orchestrated system, ignoring osd service spec.
by Wyll Ingersoll 11 Jan '23

11 Jan '23

When adding a new OSD to a ceph orchestrated system (16.2.9) on a storage node that has a specification profile that dictates which devices to use as the db_devices (SSDs), the newly added OSDs seem to be ignoring the db_devices (there are several available) and putting the data and db/wal on the same device. We installed the new disk (HDD) and then ran "ceph orch device zap /dev/xyz --force" to initialize the addition process. The OSDs that were added originally on that node were layed out correctly, but the new ones seem to be ignoring the OSD service spec. How can we make sure the new devices added are layed out correctly? thanks, Wyllys

2 3

Re: Serious cluster issue - Incomplete PGs
by Deep Dish 11 Jan '23

11 Jan '23

Eugen, I never insinuated my circumstance is resultant from buggy software, and acknowledged operational missteps. Let's please leave that there. Ceph remains a technology I like and will continue to use. Our operational understanding has evolved greatly as a result of current circumstances. Removed OSDs are gone and not recoverable. (ie. lockbox keys gone, VG groups removed).. My objective of this post is to validate understanding of an alternate recovery (of available, not complete) data scenario: 1. Cluster has blocked IO due to Incomplete pages. Therefore any online operations on affected pools / images / filesystems are blocked. # ceph -s cluster: id: health: HEALTH_WARN 1 hosts fail cephadm check cephadm background work is paused Reduced data availability: 28 pgs inactive, 28 pgs incomplete 5 pgs not deep-scrubbed in time 3 slow ops, oldest one blocked for 347227 sec, daemons [osd.25,osd.50,osd.51] have slow ops. services: mon: 5 daemons, quorum (age 8h) mgr: (active, since 27m) mds: 2/2 daemons up, 3 standby osd: 70 osds: 70 up (since 3d), 45 in (since 3d); 24 remapped pgs data: volumes: 2/2 healthy pools: 9 pools, 1056 pgs objects: 10.64M objects, 40 TiB usage: 61 TiB used, 266 TiB / 327 TiB avail pgs: 2.652% pgs not active 1027 active+clean 24 remapped+incomplete 4 incomplete 1 active+clean+scrubbing+deep 2. Since pages are incomplete and supporting data lost, I found a documented process that will mark pages are complete and unblock IO for the cluster. I fully understand that pgs that have 0 objects will have no impact on data integrity, however those pgs containing objects will result in complete data loss for only those affected pgs. Link: https://medium.com/opsops/recovering-ceph-from-reduced-data-availability-3-… Based on above referenced link, commands to this effect would mark incomplete PGs as complete (examples): ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op mark-complete --pgid 2.50 ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op mark-complete --pgid 2.57 3. My cluster, at present, has a total 28 incomplete pgs. Of these, 7 reference approximately 644 GB of now lost / irrecoverable data, the rest reference 0 objects and 0 bytes (empty). The cluster holds a total of 61.3T of data, leaving ~60.8T available for recovery. 4. If I were to mark ALL incomplete pgs as complete, the cluster would be operable - meaning I can interact with pool images and surviving files on cephfs pools. 5. Although data loss may affect the contents RBD images, these images would be able to be mapped ( rbd map ) and available for alternate recovery methods (e.g. DD contents to a seperate volume for use at a recovery facility, or attempt to read via available recovery tools that interact with the filesystem on those block devices (XFS in this case). Lost data would be equivalent of blocks of 0's in the overall image data stream where data was lost. 6. The above could be successful in extracting available / recoverable. 7. Upon marking the 2 incomplete pages affecting CephFS volume as complete, CephFS would be accessible minus affected files. How would these files be represented? (Corrupted or simply 0 bytes)? Thank you. Date: Tue, 10 Jan 2023 08:15:31 +0000 From: Eugen Block <eblock(a)nde.ag> Subject: [ceph-users] Re: Serious cluster issue - Incomplete PGs To: ceph-users(a)ceph.io Message-ID: <20230110081531.Horde.NfeIXEvXkBYy6JFyMgYbpX2(a)webmail.nde.ag> Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Hi, > Backups will be challenging. I honestly didn't anticipate this kind of > failure with ceph to be possible, we've been using it for several years now > and were encouraged by orchestrator and performance improvements in the 17 > code branch. that's exactly what a backup is for, to be prepared for the unexpected. Besides the fact that ceph didn't actually fail (you removed too many/too early OSDs) you can't expect a bug free software, no matter how long it has been running successfully. > - Identifying the pools / images / files that are affected by incomplete > pages; The PGs start with a number which reflects the pools in your cluster, check the output of 'ceph osd pool ls detail'. There's no easy way to tell which images or files are affected, you can query each OSD and list the PG's objects, but that doesn't work for missing OSDs/PGs, of course. I'm not sure how promising it is, but maybe try a for loop over all rbd images and just execute a 'rbd info <pool>/<image>' for each image, maybe it will tell you which image is incomplete. > - Extracting and reconstructing data for RBD images (these images are XFS > formatted filesystems); > - Extracting and reconstructing data for CephFS Files not affected by > incomplete PGs. If you kept the disks you removed too early (and didn't wipe them) there may be a chance to export the PG chunks with ceph-objectstore-tool [2]. I haven't used that myself in a production cluster so be careful and get familiar with the commands in a test environment first. If you already wiped the temporary OSDs I don't see a chance to recover from this. Regards, Eugen [2] https://docs.ceph.com/en/pacific/man/8/ceph-objectstore-tool/ Zitat von Deep Dish <deeepdish(a)gmail.com>: > Thanks for the insight Eugen. > > Here's what basically happened: > > - Upgrade from Nautilus to Quincy via migration to new cluster on temp > hardware; > - Data from Nautilus migrated successfully to older / lab-type equipment > running Quincy; > - Nautilus Hardware rebuilt for Quincy, data migrated back; > - As data was migrating we set the older notes to maintenance mode and > started to drain them; > - After several days many OSDs were showing as spinning in "deleting" > status on portal and we were marked OUT; > - This point we made the incorrect assumption those OSDs were no longer > required and proceeded to remove those nodes / OSDs. > > I understand Incomplete pages are basically lost. And it's likely a > lengthy task to attempt to salvage data. > > Backups will be challenging. I honestly didn't anticipate this kind of > failure with ceph to be possible, we've been using it for several years now > and were encouraged by orchestrator and performance improvements in the 17 > code branch. > > The fact is of the Incomplete pages that have object counts > 0, there's > about 644 GB of data that's tied up in this mess. There are other > incomplete PGs with object = 0 which I understand can be manually marked as > complete. The cluster has a data usage of 61 TiB. Of this I can > categorize about 14TB of critical data, 40 TB of data that is of medium / > high importance. > > There's 14TB in RBD images that would be critical on an EC pool there are > other images, however of lower importance at this point; > > There's also about a 20TB CephFS file system of lower data importance as > well. > > Question - Can you kindly point me to procedures for: > > - Identifying the pools / images / files that are affected by incomplete > pages; > - Extracting and reconstructing data for RBD images (these images are XFS > formatted filesystems); > - Extracting and reconstructing data for CephFS Files not affected by > incomplete PGs. > > Much appreciated. > > > ------------------------------ > > Date: Mon, 09 Jan 2023 10:12:49 +0000 > From: Eugen Block <eblock(a)nde.ag> > Subject: [ceph-users] Re: Serious cluster issue - Incomplete PGs > To: ceph-users(a)ceph.io > Message-ID: > <20230109101249.Horde.hAHCWQijFMYLNdX8a2YQDVV(a)webmail.nde.ag> > Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes > > Hi, > > can you clarify what exactly you did to get into this situation? What > about the undersized PGs, any chance to bring those OSDs back online? > Regarding the incomplete PGs I'm not sure there's much you can do if > the OSDs are lost. To me it reads like you may have > destroyed/recreated more OSDs than you should have, just recreating > OSDs with the same IDs is not sufficient if you destroyed too many > chunks. Each OSD only contains a chunk of the PG due to the erasure > coding. I'm afraid those objects are lost and you would have to > restore from backup. To get the cluster into a healthy state again > there a couple of threads, e. g. [1], but recovering the lost chunks > from ceph will probably not work. > > Regards, > Eugen > > [1] https://www.mail-archive.com/ceph-users@ceph.io/msg14757.html > > Zitat von Deep Dish <deeepdish(a)gmail.com>: > >> Hello. I really screwed up my ceph cluster. Hoping to get data off it >> so I can rebuild it. >> >> In summary, too many changes too quickly caused the cluster to develop >> incomplete pgs. Some PGS were reporting that OSDs were to be probes. >> I've created those OSD IDs (empty), however this wouldn't clear >> incompletes. Incompletes are part of EC pools. Running 17.2.5. >> >> This is the overall state: >> >> cluster: >> >> id: 49057622-69fc-11ed-b46e-d5acdedaae33 >> >> health: HEALTH_WARN >> >> Failed to apply 1 service(s): > osd.dashboard-admin-1669078094056 >> >> 1 hosts fail cephadm check >> >> cephadm background work is paused >> >> Reduced data availability: 28 pgs inactive, 28 pgs incomplete >> >> Degraded data redundancy: 55 pgs undersized >> >> 2 slow ops, oldest one blocked for 4449 sec, daemons >> [osd.25,osd.50,osd.51] have slow ops. >> >> >> >> These are PGs that are incomplete that HAVE DATA (Objects > 0) [ via ceph >> pg ls incomplete ]: >> >> 2.35 23199 0 0 0 95980273664 0 >> 0 2477 incomplete 10s 2104'46277 28260:686871 >> [44,4,37,3,40,32]p44 [44,4,37,3,40,32]p44 >> 2023-01-03T03:54:47.821280+0000 2022-12-29T18:53:09.287203+0000 >> 14 queued for deep scrub >> 2.53 22821 0 0 0 94401175552 0 >> 0 2745 remapped+incomplete 10s 2104'45845 28260:565267 >> [60,48,52,65,67,7]p60 [60]p60 >> 2023-01-03T10:18:13.388383+0000 2023-01-03T10:18:13.388383+0000 >> 408 queued for scrub >> 2.9f 22858 0 0 0 94555983872 0 >> 0 2736 remapped+incomplete 10s 2104'45636 28260:759872 >> [56,59,3,57,5,32]p56 [56]p56 >> 2023-01-03T10:55:49.848693+0000 2023-01-03T10:55:49.848693+0000 >> 376 queued for scrub >> 2.be 22870 0 0 0 94429110272 0 >> 0 2661 remapped+incomplete 10s 2104'45561 28260:813759 >> [41,31,37,9,7,69]p41 [41]p41 >> 2023-01-03T14:02:15.790077+0000 2023-01-03T14:02:15.790077+0000 >> 360 queued for scrub >> 2.e4 22953 0 0 0 94912278528 0 >> 0 2648 remapped+incomplete 20m 2104'46048 28259:732896 >> [37,46,33,4,48,49]p37 [37]p37 >> 2023-01-02T18:38:46.268723+0000 2022-12-29T18:05:47.431468+0000 >> 18 queued for deep scrub >> 17.78 20169 0 0 0 84517834400 0 >> 0 2198 remapped+incomplete 10s 3735'53405 28260:1243673 >> [4,37,2,36,66,0]p4 [41]p41 >> 2023-01-03T14:21:41.563424+0000 2023-01-03T14:21:41.563424+0000 >> 348 queued for scrub >> 17.d8 20328 0 0 0 85196053130 0 >> 0 1852 remapped+incomplete 10s 3735'54458 28260:1309564 >> [38,65,61,37,58,39]p38 [53]p53 >> 2023-01-02T18:32:35.371071+0000 2022-12-28T19:08:29.492244+0000 >> 21 queued for deep scrub >> >> At present I'm unable to reliably access my data due to incomplete pages >> above. I'll post whatever outputs requested (won't post now as it can be >> rather verbose). Is there hope?

2 1

Move bucket between realms
by mahnoosh shahidi 11 Jan '23

11 Jan '23

Hi all, Is there any way in rgw to move a bucket from one realm to another one in the same cluster? Best regards, Mahnoosh

1 0

Move bucket between realms
by mahnoosh shahidi 11 Jan '23

11 Jan '23

Hi all, Is there any way in rgw to move a bucket from one realm to another one in the same cluster? Best regards, Mahnoosh

1 0

OSD crash with "FAILED ceph_assert(v.length() == p->shard_info->bytes)"
by Yu Changyuan 11 Jan '23

11 Jan '23

One of OSD(other OSDs are fine) was crashed, and try "ceph-bluestore-tool fsck" also crashed with same error. Besides destroy this OSD and re-create, are there any other steps I can do to restore the OSD? Below is part of message: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc: 3228: FAILED ceph_assert(v.length() == p->shard_info->bytes) ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x562c19f7f73c] 2: /usr/bin/ceph-osd(+0x57f956) [0x562c19f7f956] 3: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x7cf) [0x562c1a56d1ef] 4: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x5dd) [0x562c1a5c5ebd] 5: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0xd1) [0x562c1a5c70e1] 6: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x2077) [0x562c1a5cb237] 7: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x316) [0x562c1a5e66d6] 8: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x58) [0x562c1a22a878] 9: (ReplicatedBackend::do_repop(boost::intrusive_ptr<OpRequest>)+0xeb0) [0x562c1a41cff0] 10: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x267) [0x562c1a42d357] 11: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x52) [0x562c1a25dd52] 12: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5de) [0x562c1a20168e] 13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309) [0x562c1a088fc9] 14: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0x562c1a2e7e78] 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x562c1a0a64c8] 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x562c1a7232a4] 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562c1a726184] 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2d2a4081ca] 19: clone() -10> 2023-01-10T09:28:02.143+0000 7f2cff3de700 -1 *** Caught signal (Aborted) ** And this is "meta" file of crash log: { "crash_id": "2023-01-10T09:28:02.137396Z_a504670d-32c3-46ee-8398-84389c9c2d95", "timestamp": "2023-01-10T09:28:02.137396Z", "process_name": "ceph-osd", "entity_name": "osd.3", "ceph_version": "16.2.10", "utsname_hostname": "dskm1-r0", "utsname_sysname": "Linux", "utsname_release": "5.18.19", "utsname_version": "#1-NixOS SMP PREEMPT_DYNAMIC Sun Aug 21 13:18:56 UTC 2022", "utsname_machine": "x86_64", "os_name": "CentOS Stream", "os_id": "centos", "os_version_id": "8", "os_version": "8", "assert_condition": "v.length() == p->shard_info->bytes", "assert_func": "void BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc", "assert_line": 3228, "assert_thread_name": "tp_osd_tp", "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)' thread 7f2cff3de700 time 2023-01-10T09:28:02.016735+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/os/bluestore/BlueStore.cc: 3228: FAILED ceph_assert(v.length() == p->shard_info->bytes)\n", "backtrace": [ "/lib64/libpthread.so.0(+0x12cf0) [0x7f2d2a412cf0]", "gsignal()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x562c19f7f78d]", "/usr/bin/ceph-osd(+0x57f956) [0x562c19f7f956]", "(BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x7cf) [0x562c1a56d1ef]", "(BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x5dd) [0x562c1a5c5ebd]", "(BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0xd1) [0x562c1a5c70e1]", "(BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x2077) [0x562c1a5cb237]", "(BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x316) [0x562c1a5e66d6]", "(non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x58) [0x562c1a22a878]", "(ReplicatedBackend::do_repop(boost::intrusive_ptr<OpRequest>)+0xeb0) [0x562c1a41cff0]", "(ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x267) [0x562c1a42d357]", "(PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x52) [0x562c1a25dd52]", "(PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5de) [0x562c1a20168e]", "(OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309) [0x562c1a088fc9]", "(ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0x562c1a2e7e78]", "(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x562c1a0a64c8]", "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x562c1a7232a4]", "(ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562c1a726184]", "/lib64/libpthread.so.0(+0x81ca) [0x7f2d2a4081ca]", "clone()" ] } I also update full crash log to github gist: https://gist.github.com/yuchangyuan/2016f259175940f64e2eed528d633794 -- Best wishes ~

1 0

Snap trimming best practice
by Szabo, Istvan (Agoda) 11 Jan '23

11 Jan '23

Hi, Wonder have you ever faced issue with snaptrimming if you follow ceph pg allocation recommendation (100pg/osd)? We have a nautilus cluster and we scare to increase the pg-s of the pools because seems like even if we have 4osd/nvme, if the pg number is higher = the snaptrimming is slower. Eg.: We have these pools: Db1 pool size 64,504G with 512 PGs Db2 pool size 92,242G with 256 PGs Db2 snapshot remove faster than Db1. Our osds are very underutilized regarding pg point of view due to this reason, each osd is holding maximum 25 gigantic pgs which makes all the maintenance very difficult due to backfilling full, osd full issues. Any recommendation if you use this feature? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Re: OSD crash on Onode::put
by Igor Fedotov 11 Jan '23

11 Jan '23

Hi Dongdong, thanks a lot for your post, it's really helpful. Thanks, Igor On 1/5/2023 6:12 AM, Dongdong Tao wrote: > > I see many users recently reporting that they have been struggling > with this Onode::put race condition issue[1] on both the latest > Octopus and pacific. > Igor opened a PR [2] to address this issue, I've reviewed it > carefully, and looks good to me. I'm hoping this could get some > priority from the community. > > For those who had been hitting this issue, I would like to share a > workaround that could unblock you: > > During the investigation of this issue, I found this race condition > always happens after the bluestore onode cache size becomes 0. > Setting debug_bluestore = 1/30 will allow you to see the cache size > after the crash: > --- > 2022-10-25T00:47:26.562+0000 7f424f78e700 30 > bluestore.MempoolThread(0x564a9dae2a68) _resize_shards > max_shard_onodes: 0 max_shard_buffer: 8388608 > --- > > This is apparently wrong as this means the bluestore metadata cache is > basically disabled, > but it makes much sense to explain why we are hitting the race > condition so easily -- An onode will be trimmed right away after it's > unpinned. > > Keep going with the investigation, it turns out the culprit for the > 0-sized cache is the leak that happened in bluestore_cache_other mempool > Please refer to the bug tracker [3] which has the detail of the leak > issue, it was already fixed by [4], and the next Pacific point > release will have it. > But it was never backported to Octopus. > So if you are hitting the same: > For those who are on Octopus, you can manually backport this patch to > fix the leak and prevent the race condition from happening. > For those who are on Pacific, you can wait for the next Pacific point > release. > > By the way, I'm backporting the fix to ubuntu Octopus and Pacific > through this SRU [5], so it will be landed in ubuntu's package soon. > > [1] https://tracker.ceph.com/issues/56382 > [2] https://github.com/ceph/ceph/pull/47702 > [3] https://tracker.ceph.com/issues/56424 > [4] https://github.com/ceph/ceph/pull/46911 > [5] https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1996010 > > Cheers, > Dongdong > > -- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunicatio…>! <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunicatio…>

5 10

[ERR] OSD_SCRUB_ERRORS: 2 scrub errors
by Szabo, Istvan (Agoda) 11 Jan '23

11 Jan '23

Hi, I’m continuously getting scrub errors in my index pool and log pool that I need to repair always. HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 2 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 20.19 is active+clean+inconsistent, acting [39,41,37] Why is this? I have no cue at all, no log entry no anything ☹ ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

6 9

Intel Cache Solution with HA Cluster on the iSCSI Gateway node
by Kamran Zafar Syed 11 Jan '23

11 Jan '23

HI There, Is there someone, who had some experience of implementing Intel Cache Accelerator Solution on top of iSCSI Gateway. Thanks and Regards, Koki

1 0

nfs RGW export makes nfs-gnaesha server in crash loop
by Ben 11 Jan '23

11 Jan '23

Hi, This is running Quincy 17.2.5 deployed by rook on k8s. RGW nfs export will crash Ganesha server pod. CephFS export works just fine. Here are steps of it: 1, create export: bash-4.4$ ceph nfs export create rgw --cluster-id nfs4rgw --pseudo-path /bucketexport --bucket testbk { "bind": "/bucketexport", "path": "testbk", "cluster": "nfs4rgw", "mode": "RW", "squash": "none" } 2, check pods status afterwards: rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running 0 4h3m rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 1/2 Error 2 4h6m 3, check failing pod’s logs: 11/01/2023 08:11:53 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 11/01/2023 08:11:54 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 11/01/2023 08:11:54 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 11/01/2023 08:11:57 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE 11/01/2023 08:11:57 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] export_defaults_commit :CONFIG :INFO :Export Defaults now (options=03303002/00080000 , , , , , , , , expire= 0) 2023-01-11T08:11:57.853+0000 7f59dac7c200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory 2023-01-11T08:11:57.853+0000 7f59dac7c200 -1 AuthRegistry(0x56476817a480) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx 2023-01-11T08:11:57.855+0000 7f59dac7c200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory 2023-01-11T08:11:57.855+0000 7f59dac7c200 -1 AuthRegistry(0x7ffe4d092c90) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx 2023-01-11T08:11:57.856+0000 7f5987537700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:11:57.856+0000 7f5986535700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:12:00.861+0000 7f5986d36700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:12:00.861+0000 7f59dac7c200 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication failed to fetch mon config (--no-mon-config to skip) 4, delete the export: ceph nfs export delete nfs4rgw /bucketexport Ganesha servers go back normal: rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running 0 4h30m rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 2/2 Running 10 4h33m Any ideas to make it work? Thanks Ben

1 0

Issues with cephadm adopt cluster with name
by armsby 11 Jan '23

11 Jan '23

I am trying to adopt a cluster with cephadm, and everything was ok when it came to mon and mgr servers, But when I try to run "cephadm adopt --name osd.340 --style legacy —cluster prod" It runs everything, but when the container starts, it says that it can not open /etc/ceph/prod.conf as it binds config in as /etc/ceph/ceph.conf If I change the unit.run file so it mounts it in as prod.conf it starts but has issues connecting to mon servers. Has someone else experienced this, and are there solutions for this issue? Armsby

1 0

2 pgs backfill_toofull but plenty of space
by Torkil Svensgaard 10 Jan '23

10 Jan '23

Hi Ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable) Looking at this: " Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull " " [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull pg 3.11f is active+remapped+backfill_wait+backfill_toofull, acting [98,51,39,100] pg 3.74c is active+remapped+backfill_wait+backfill_toofull, acting [96,120,58,48] " But the disks are noway near being full as far as I can determine, so why backfill_toofull? The PGs in question are in the rbd_data pool. " # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.4 PiB 730 TiB 686 TiB 686 TiB 48.46 ssd 1.3 TiB 1.2 TiB 162 GiB 162 GiB 12.11 TOTAL 1.4 PiB 731 TiB 686 TiB 686 TiB 48.42 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 1.1 GiB 273 545 MiB 0.05 549 GiB rbd_data 3 4096 294 TiB 78.56M 450 TiB 45.72 267 TiB rbd 4 32 4.1 MiB 26 3.5 MiB 0 549 GiB rbd_internal 5 32 54 KiB 16 172 KiB 0 549 GiB cephfs_data 6 2048 127 TiB 148.64M 229 TiB 29.99 267 TiB cephfs_metadata 7 128 71 GiB 2.84M 142 GiB 11.46 549 GiB libvirt 8 32 37 MiB 221 74 MiB 0 549 GiB nfs-ganesha 9 32 2.7 KiB 7 52 KiB 0 366 GiB .nfs 10 32 53 KiB 47 306 KiB 0 366 GiB " The top utilized disk is at 57% and the PGs in that pool are ~50GB. " TOP BOTTOM USE WEIGHT PGS ID |USE WEIGHT PGS ID --------------------------------+-------------------------------- 57.71% 1.00000 54 osd.68 |46.60% 1.00000 286 osd.17 57.08% 1.00000 53 osd.80 |46.55% 1.00000 286 osd.99 54.95% 1.00000 70 osd.86 |46.48% 1.00000 284 osd.106 54.86% 1.00000 52 osd.63 |45.88% 1.00000 187 osd.27 54.06% 1.00000 68 osd.88 |45.81% 1.00000 279 osd.5 53.89% 1.00000 51 osd.79 |44.95% 1.00000 272 osd.13 53.65% 1.00000 51 osd.67 |43.63% 1.00000 269 osd.16 53.59% 1.00000 52 osd.65 |43.30% 1.00000 261 osd.12 53.58% 1.00000 51 osd.82 |32.17% 1.00000 172 osd.4 53.52% 1.00000 50 osd.72 |0% 0 0 osd.49 --------------------------------+-------------------------------- " Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

3 2

Re: What's happening with ceph-users?
by Neha Ojha 10 Jan '23

10 Jan '23

Re-adding the dev list and adding the user list because others might benefit from this information. Thanks, Neha On Tue, Jan 10, 2023 at 10:21 AM Wyll Ingersoll < wyllys.ingersoll(a)keepertech.com> wrote: > Also, it was only my ceph-users account that was lost, dev account was > still active. > ------------------------------ > *From:* Wyll Ingersoll <wyllys.ingersoll(a)keepertech.com> > *Sent:* Tuesday, January 10, 2023 1:20 PM > *To:* Neha Ojha <nojha(a)redhat.com>; Adam Kraitman <akraitma(a)redhat.com>; > Dan Mick <dan.mick(a)redhat.com> > *Subject:* Re: What's happening with ceph-users? > > I ended up re-subscribing this morning. But it might be worth > investigating if others are having similar issues. > ------------------------------ > *From:* Neha Ojha <nojha(a)redhat.com> > *Sent:* Tuesday, January 10, 2023 1:14 PM > *To:* Wyll Ingersoll <wyllys.ingersoll(a)keepertech.com>; Adam Kraitman < > akraitma(a)redhat.com>; Dan Mick <dan.mick(a)redhat.com> > *Subject:* Re: What's happening with ceph-users? > > +Adam Kraitman <akraitma(a)redhat.com> +Dan Mick <dan.mick(a)redhat.com> Is > this expected? > > On Tue, Jan 10, 2023 at 6:15 AM Wyll Ingersoll < > wyllys.ingersoll(a)keepertech.com> wrote: > > > All of my subscriptions to the ceph.io lists (users and developers) seem > to have been deleted. Do we need to re-subscribe or is this something > that is being fixed? > ------------------------------ > *From:* Neha Ojha <nojha(a)redhat.com> > *Sent:* Monday, January 9, 2023 2:40 PM > *To:* Dan van der Ster <dvanders(a)gmail.com> > *Cc:* Ceph Developers <dev(a)ceph.io>; Josh Durgin <jdurgin(a)redhat.com>; > Mike Perez <miperez(a)redhat.com>; Adam Kraitman <akraitma(a)redhat.com> > *Subject:* Re: What's happening with ceph-users? > > Our mailing lists were down due to the recent lab issues. They should be > back up now. Please let us know if you see any issues. > > Thanks, > Neha > > On Sun, Jan 8, 2023 at 9:53 AM Dan van der Ster <dvanders(a)gmail.com> > wrote: > > Hi, > > Has ceph-users been down a few days? And now it seems to have been > reverted to an old backup? (I'm referring mail on an address I unsubbed > many months ago) > > Thanks, Dan > >

1 0

Re: docs.ceph.com -- Do you use the header navigation bar? (RESPONSES REQUESTED)
by Frank Schilder 10 Jan '23

10 Jan '23

Hi John, firstly, image attachments are filtered out by the list. How about you upload the image somewhere like https://imgur.com/ and post a link instead? In my browser, the sticky header contains only "home" and "edit on github", which are both entirely useless for a user. What exactly is "header navigation" expected to do if it contains nothing else? Unless I'm looking at the wrong thing (I can't see the attached image), this header can be removed. The "edit on github" link can be added to the end of a page. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: John Zachary Dover <zac.dover(a)gmail.com> Sent: 04 January 2023 16:35:56 To: ceph-users Subject: [ceph-users] docs.ceph.com -- Do you use the header navigation bar? (RESPONSES REQUESTED) Do you use the header navigation bar on docs.ceph.com? See the attached file (sticky_header.png) if you are unsure of what "header navigation bar" means. In the attached file, the header navigation bar is indicated by means of two large, ugly, red-and-green arrows. *Cards on the Table* The navigation bar is the kind of thing that is sometimes referred to as a "sticky header", and it can get in the way of linked-to sections. I would like to remove this header bar. If there is community support for the header bar, though, I won't remove it. *What is Zac Complaining About?* Follow this procedure to see the behavior that has provoked my complaint: 1. Go to https://docs.ceph.com/en/quincy/glossary/ 2. Scroll down to the "Ceph Cluster Map" entry. 3. Click the "Cluster Map" link in the line that reads "See Cluster Map". 4. Notice that the header navigation bar obscures the headword "Cluster Map". If you have any opinion at all on this matter, voice it. Please. Zac Dover Docs Upstream Ceph _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Re: rbd-mirror stops replaying journal on primary cluster
by Josef Johansson 10 Jan '23

10 Jan '23

Hi, Actually, the test case was even more simple than that. A misaligned discard (discard_granularity_bytes=4096, offset=0, length=4096+512) made the journal stop replaying entries. This is now well covered in tests and example e2e-tests. The workaround is quite easy, set `rbd_discard_granularity_bytes = 0` in the client ceph conf and all discards will be applied to the rbd image. The fix should hopefully be backported to stable releases. Thanks for review by the ceph-team on this. If anyone could confirm that this indeed solves the problem, please let me know anyhow. Regards Josef On Thu, Dec 8, 2022 at 11:15 AM Josef Johansson <josef86(a)gmail.com> wrote: > > Hi, > > Running a simple > `echo 1>a;sync;rm a;sync;fstrim --all` > Triggers the problem. No need to have the mount point mounted with discard. > > On Thu, Dec 8, 2022 at 12:33 AM Josef Johansson <josef86(a)gmail.com> wrote: > > > > Hi, > > > > I've updated https://tracker.ceph.com/issues/57396 with some more > > info, it seems that disabling discard within a guest solves the > > problem (or switching from virtio-scsi-single to virtio-blk in older > > kernels). I'm testing two different VMs on the same hypervisor with > > identical configs, one works the other doesn't. > > > > Not sure what to make of it, seems that the kernel around 4.18+ are > > sending a weird discard? > > > > On Tue, Aug 30, 2022 at 8:43 AM Josef Johansson <josef86(a)gmail.com> wrote: > > > > > > Hi, > > > > > > There's nothing special in the cluster when it stops replaying. It > > > seems that a journal entry that the local replayer doesn't handle and > > > just stops. Since it's the local replayer that stops there's no logs > > > in rbd-mirror. The odd part is that rbd-mirror handles this totally > > > fine and is the one syncing correctly. > > > > > > What's worse is that this is reported as HEALTHY in status > > > information, even though when restarting that VM it will stall until > > > replaying is complete. The replay function inside rbd client seems to > > > be fine handling the journal, but only on start of the vm. I will try > > > to get a ticket open on tracker.ceph.com as soon as my account is > > > approved. > > > > > > I have tried to see what component is responsible for local replay but > > > I have not been successful yet. > > > > > > Thanks for answering :) > > > > > > On Mon, Aug 22, 2022 at 11:05 AM Eugen Block <eblock(a)nde.ag> wrote: > > > > > > > > Hi, > > > > > > > > IIRC the rbd mirror journals will grow if the sync stops to work, > > > > which seems to be the case here. Does the primary cluster experience > > > > any high load when the replay stops? How is the connection between the > > > > two sites and is the link saturated? Does the rbd-mirror log reveal > > > > anything useful (maybe also in debug mode)? > > > > > > > > Regards, > > > > Eugen > > > > > > > > Zitat von Josef Johansson <josef(a)oderland.se>: > > > > > > > > > Hi, > > > > > > > > > > I'm running ceph octopus 15.2.16 and I'm trying out two way mirroring. > > > > > > > > > > Everything seems to running fine except sometimes when the replay > > > > > stops at the primary clusters. > > > > > > > > > > This means that VMs will not start properly until all journal > > > > > entries are replayed, but also that the journal grows by time. > > > > > > > > > > I am trying to find out why this occurs, and where to look for more > > > > > information. > > > > > > > > > > I am currently using rbd --pool <pool> --image <image> journal > > > > > status to see if the clients are in sync or not. > > > > > > > > > > Example output when things went sideways > > > > > > > > > > minimum_set: 0 > > > > > active_set: 2 > > > > > registered clients: > > > > > [id=, commit_position=[positions=[[object_number=0, tag_tid=1, > > > > > entry_tid=4592], [object_number=3, tag_tid=1, entry_tid=4591], > > > > > [object_number=2, tag_tid=1, entry_tid=4590], [object_number=1, > > > > > tag_tid=1, entry_tid=4589]]], state=connected] > > > > > [id=bdde9b90-df26-4e3d-84b3-66605dc45608, > > > > > commit_position=[positions=[[object_number=5, tag_tid=1, > > > > > entry_tid=19913], [object_number=4, tag_tid=1, entry_tid=19912], > > > > > [object_number=7, tag_tid=1, entry_tid=19911], [object_number=6, > > > > > tag_tid=1, entry_tid=19910]]], state=disconnected] > > > > > > > > > > Right now I'm trying to catch it red handed in the primary osd logs. > > > > > But I'm not even sure if that's the process that is replaying the > > > > > journal.. > > > > > > > > > > Regards > > > > > Josef > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Octopus RGW large omaps in usage
by Boris Behrens 10 Jan '23

10 Jan '23

Hi, I am currently trying to figure out how to resolve the "large objects found in pool 'rgw.usage'" error. In the past I trimmed the usage log, but now I am at the point that I need to trim it down to two weeks. I checked and amount of omapkeys and the distribution is quite off: # for OBJECT in `rados -p rgw.usage ls`; do rados -p eu-central-1.rgw.usage listomapkeys ${OBJECT} | wc -l done 86968 144388 6188 87854 46652 194788 46234 9622 45768 28376 104348 10018 11112 34374 44744 40638 93664 35476 107794 18020 7172 17836 37344 73496 15572 31570 149352 740 113566 35292 5318 442176 Maybe it would be an option to increase this value?rgw usage max user shards -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: Serious cluster issue - Incomplete PGs
by Deep Dish 10 Jan '23

10 Jan '23

Thanks for the insight Eugen. Here's what basically happened: - Upgrade from Nautilus to Quincy via migration to new cluster on temp hardware; - Data from Nautilus migrated successfully to older / lab-type equipment running Quincy; - Nautilus Hardware rebuilt for Quincy, data migrated back; - As data was migrating we set the older notes to maintenance mode and started to drain them; - After several days many OSDs were showing as spinning in "deleting" status on portal and we were marked OUT; - This point we made the incorrect assumption those OSDs were no longer required and proceeded to remove those nodes / OSDs. I understand Incomplete pages are basically lost. And it's likely a lengthy task to attempt to salvage data. Backups will be challenging. I honestly didn't anticipate this kind of failure with ceph to be possible, we've been using it for several years now and were encouraged by orchestrator and performance improvements in the 17 code branch. The fact is of the Incomplete pages that have object counts > 0, there's about 644 GB of data that's tied up in this mess. There are other incomplete PGs with object = 0 which I understand can be manually marked as complete. The cluster has a data usage of 61 TiB. Of this I can categorize about 14TB of critical data, 40 TB of data that is of medium / high importance. There's 14TB in RBD images that would be critical on an EC pool there are other images, however of lower importance at this point; There's also about a 20TB CephFS file system of lower data importance as well. Question - Can you kindly point me to procedures for: - Identifying the pools / images / files that are affected by incomplete pages; - Extracting and reconstructing data for RBD images (these images are XFS formatted filesystems); - Extracting and reconstructing data for CephFS Files not affected by incomplete PGs. Much appreciated. ------------------------------ Date: Mon, 09 Jan 2023 10:12:49 +0000 From: Eugen Block <eblock(a)nde.ag> Subject: [ceph-users] Re: Serious cluster issue - Incomplete PGs To: ceph-users(a)ceph.io Message-ID: <20230109101249.Horde.hAHCWQijFMYLNdX8a2YQDVV(a)webmail.nde.ag> Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Hi, can you clarify what exactly you did to get into this situation? What about the undersized PGs, any chance to bring those OSDs back online? Regarding the incomplete PGs I'm not sure there's much you can do if the OSDs are lost. To me it reads like you may have destroyed/recreated more OSDs than you should have, just recreating OSDs with the same IDs is not sufficient if you destroyed too many chunks. Each OSD only contains a chunk of the PG due to the erasure coding. I'm afraid those objects are lost and you would have to restore from backup. To get the cluster into a healthy state again there a couple of threads, e. g. [1], but recovering the lost chunks from ceph will probably not work. Regards, Eugen [1] https://www.mail-archive.com/ceph-users@ceph.io/msg14757.html Zitat von Deep Dish <deeepdish(a)gmail.com>: > Hello. I really screwed up my ceph cluster. Hoping to get data off it > so I can rebuild it. > > In summary, too many changes too quickly caused the cluster to develop > incomplete pgs. Some PGS were reporting that OSDs were to be probes. > I've created those OSD IDs (empty), however this wouldn't clear > incompletes. Incompletes are part of EC pools. Running 17.2.5. > > This is the overall state: > > cluster: > > id: 49057622-69fc-11ed-b46e-d5acdedaae33 > > health: HEALTH_WARN > > Failed to apply 1 service(s): osd.dashboard-admin-1669078094056 > > 1 hosts fail cephadm check > > cephadm background work is paused > > Reduced data availability: 28 pgs inactive, 28 pgs incomplete > > Degraded data redundancy: 55 pgs undersized > > 2 slow ops, oldest one blocked for 4449 sec, daemons > [osd.25,osd.50,osd.51] have slow ops. > > > > These are PGs that are incomplete that HAVE DATA (Objects > 0) [ via ceph > pg ls incomplete ]: > > 2.35 23199 0 0 0 95980273664 0 > 0 2477 incomplete 10s 2104'46277 28260:686871 > [44,4,37,3,40,32]p44 [44,4,37,3,40,32]p44 > 2023-01-03T03:54:47.821280+0000 2022-12-29T18:53:09.287203+0000 > 14 queued for deep scrub > 2.53 22821 0 0 0 94401175552 0 > 0 2745 remapped+incomplete 10s 2104'45845 28260:565267 > [60,48,52,65,67,7]p60 [60]p60 > 2023-01-03T10:18:13.388383+0000 2023-01-03T10:18:13.388383+0000 > 408 queued for scrub > 2.9f 22858 0 0 0 94555983872 0 > 0 2736 remapped+incomplete 10s 2104'45636 28260:759872 > [56,59,3,57,5,32]p56 [56]p56 > 2023-01-03T10:55:49.848693+0000 2023-01-03T10:55:49.848693+0000 > 376 queued for scrub > 2.be 22870 0 0 0 94429110272 0 > 0 2661 remapped+incomplete 10s 2104'45561 28260:813759 > [41,31,37,9,7,69]p41 [41]p41 > 2023-01-03T14:02:15.790077+0000 2023-01-03T14:02:15.790077+0000 > 360 queued for scrub > 2.e4 22953 0 0 0 94912278528 0 > 0 2648 remapped+incomplete 20m 2104'46048 28259:732896 > [37,46,33,4,48,49]p37 [37]p37 > 2023-01-02T18:38:46.268723+0000 2022-12-29T18:05:47.431468+0000 > 18 queued for deep scrub > 17.78 20169 0 0 0 84517834400 0 > 0 2198 remapped+incomplete 10s 3735'53405 28260:1243673 > [4,37,2,36,66,0]p4 [41]p41 > 2023-01-03T14:21:41.563424+0000 2023-01-03T14:21:41.563424+0000 > 348 queued for scrub > 17.d8 20328 0 0 0 85196053130 0 > 0 1852 remapped+incomplete 10s 3735'54458 28260:1309564 > [38,65,61,37,58,39]p38 [53]p53 > 2023-01-02T18:32:35.371071+0000 2022-12-28T19:08:29.492244+0000 > 21 queued for deep scrub > > At present I'm unable to reliably access my data due to incomplete pages > above. I'll post whatever outputs requested (won't post now as it can be > rather verbose). Is there hope?

2 1

Re: 1 pg recovery_unfound after multiple crash of an OSD
by Kai Stian Olstad 10 Jan '23

10 Jan '23

Hi Just a follow up, the issue was solved by running command ceph pg 404.1ff mark_unfound_lost delete - Kai Stian Olstad On 04.01.2023 13:00, Kai Stian Olstad wrote: > Hi > > We are running Ceph 16.2.6 deployed with Cephadm. > > Around Christmas OSD 245 and 327 had about 20 read error so I set them > to out. > > Around new year another OSD 313 more or less died since is become so > slow that it triggered Linux default I/O-timeout of 30 seconds. > In this period the OSD crashed 8 times and was restartet by Systemd > and we ended up with > > [WRN] OBJECT_UNFOUND: 1/416287126 objects unfound (0.000%) > pg 404.1ff has 1 unfound objects > [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound > pg 404.1ff is active+recovery_unfound+degraded+remapped, acting > [208,220,269,175,313,329], 1 unfound > [WRN] PG_DEGRADED: Degraded data redundancy: 5/2364745884 objects > degraded (0.000%), 1 pg degraded > pg 404.1ff is active+recovery_unfound+degraded+remapped, acting > [208,220,269,175,313,329], 1 unfound > > The pool 404 is "default.rgw.buckets.data" and pool 404 is erasure > encoding 4+2. > > I have search for a solution but with no luck, what I have tried is > > - Restarted all 6 OSD for the PG one by one > - Running repair of 404.1ff > > Output of following command > - ceph -s > - ceph health detail > - ceph pg ls | grep -e PG -e ^404.1ff > - ceph osd pool ls detail | grep 404 > - ceph osd tree out > - ceph crash ls | grep -e ID -e osd.313 > - ceph pg 404.1ff list_unfound > - ceph pg 404.1ff > > Is appended below, can also be read here > https://gitlab.com/-/snippets/2479624 > or cloned with "git clone https://gitlab.com/-/snippets/2479624" > > Does anyone have any idea on how to resolv the problem? > Any help is much appreciated. > > - > Kai Stian Olstad > > > > :::::::::::::: > ceph-s.txt > :::::::::::::: > ceph -s > ------- > cluster: > id: d13c6b81-51ee-4d22-84e9-456f9307296c > health: HEALTH_ERR > 1/416287125 objects unfound (0.000%) > Possible data damage: 1 pg recovery_unfound > Degraded data redundancy: 5/2364745860 objects degraded > (0.000%), 1 pg degraded > > services: > mon: 3 daemons, quorum ceph-mon-1,ceph-mon-2,ceph-mon-3 (age 2M) > mgr: ceph-mon-2.mfdanx(active, since 3w), standbys: > ceph-mon-1.ptrsea > mds: 1/1 daemons up, 1 standby > osd: 355 osds: 355 up (since 20h), 352 in (since 2d); 1 remapped > pgs > rgw: 4 daemons active (4 hosts, 1 zones) > > data: > volumes: 1/1 healthy > pools: 14 pools, 2505 pgs > objects: 416.29M objects, 540 TiB > usage: 939 TiB used, 2.1 PiB / 3.0 PiB avail > pgs: 5/2364745860 objects degraded (0.000%) > 137931/2364745860 objects misplaced (0.006%) > 1/416287125 objects unfound (0.000%) > 2489 active+clean > 14 active+clean+scrubbing+deep > 1 active+recovery_unfound+degraded+remapped > 1 active+clean+scrubbing > > io: > client: 38 MiB/s rd, 23 MiB/s wr, 2.58k op/s rd, 326 op/s wr > > progress: > Global Recovery Event (6d) > [===========================.] (remaining: 3m) > > > :::::::::::::: > ceph_health_detail.txt > :::::::::::::: > ceph health detail > ------------------ > HEALTH_ERR 1/416287126 objects unfound (0.000%); Possible data damage: > 1 pg recovery_unfound; Degraded data redundancy: 5/2364745884 objects > degraded (0.000%), 1 pg degraded > [WRN] OBJECT_UNFOUND: 1/416287126 objects unfound (0.000%) > pg 404.1ff has 1 unfound objects > [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound > pg 404.1ff is active+recovery_unfound+degraded+remapped, acting > [208,220,269,175,313,329], 1 unfound > [WRN] PG_DEGRADED: Degraded data redundancy: 5/2364745884 objects > degraded (0.000%), 1 pg degraded > pg 404.1ff is active+recovery_unfound+degraded+remapped, acting > [208,220,269,175,313,329], 1 unfound > > > :::::::::::::: > ceph_pg_ls.txt > :::::::::::::: > ceph pg ls | grep -e PG -e ^404.1ff > ----------------------------------- > PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES > OMAP_BYTES* OMAP_KEYS* LOG STATE > SINCE VERSION REPORTED UP > ACTING SCRUB_STAMP > DEEP_SCRUB_STAMP > 404.1ff 137912 5 137908 1 282417561722 > 0 0 5528 active+recovery_unfound+degraded+remapped > 19h 141748'724163 141748:3558203 [208,220,269,175,343,329]p208 > [208,220,269,175,313,329]p208 2022-12-31T19:27:10.993286+0000 > 2022-12-31T19:27:10.993286+0000 > > > :::::::::::::: > ceph_osd_pool_ls_detail.txt > :::::::::::::: > ceph osd pool ls detail | grep 404 > ---------------------------------- > pool 404 'default.rgw.buckets.data' erasure profile > ec42-jerasure-blaum_roth-hdd size 6 min_size 5 crush_rule 2 > object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode on > last_change 124077 lfor 0/52091/108555 flags hashpspool stripe_width > 229376 target_size_bytes 1099511627776000 application rgw > > > :::::::::::::: > ceph_osd_tree_out.txt > :::::::::::::: > ceph osd tree out > ----------------- > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 3112.43481 root default > -58 192.35847 host ceph-hd-010 > 245 hdd 12.82390 osd.245 up 0 1.00000 > -79 192.35847 host ceph-hd-014 > 313 hdd 12.82390 osd.313 up 0 1.00000 > 327 hdd 12.82390 osd.327 up 0 1.00000 > > > :::::::::::::: > ceph_crash_ls.txt > :::::::::::::: > ceph crash ls | grep -e ID -e osd.313 > ------------------------------------- > ID > ENTITY NEW > 2022-12-31T01:19:06.859805Z_8e94201c-8817-4c4e-b4dd-67c5b1d3a73a > osd.313 > 2022-12-31T01:19:47.265925Z_551b2de6-d1b0-4cbc-9e62-27ede7a46803 > osd.313 > 2022-12-31T01:20:18.648279Z_64f232ec-2cb5-4dcf-8686-b12987ed7e45 > osd.313 > 2022-12-31T20:28:43.093203Z_9c9a590f-7490-4f74-affe-5bda27b87618 > osd.313 > 2023-01-02T01:34:25.087309Z_b7a4b341-5f79-467b-a4e2-3df5456a2ff7 > osd.313 > 2023-01-02T03:46:32.295053Z_d71b5b62-5974-4b3e-b1c5-cc878d376f06 > osd.313 > 2023-01-02T10:58:58.350473Z_ca87b412-bae5-4873-9997-092587d7c699 > osd.313 > 2023-01-02T11:42:13.433011Z_c0e9ab85-a162-4331-b690-68847c641bd0 > osd.313 > > > :::::::::::::: > ceph_pg_404.1ff_list_unfound.json > :::::::::::::: > { > "num_missing": 1, > "num_unfound": 1, > "objects": [ > { > "oid": { > "oid": > "1f244892-a2e7-406b-aa62-1b13511333a2.902205.1__multipart_ingest/date=2022-12-30/2022-12-30.json.gz.2~pORMz8G-kyQx8weY8GhNt-pgAmQDufG.684", > "key": "", > "snapid": -2, > "hash": 1069593087, > "max": 0, > "pool": 404, > "namespace": "" > }, > "need": "140483'722076", > "have": "0'0", > "flags": "none", > "clean_regions": "clean_offsets: [], clean_omap: 0, > new_object: 1", > "locations": [ > "329(5)" > ] > } > ], > "state": "NotRecovering", > "available_might_have_unfound": true, > "might_have_unfound": [], > "more": false > } > > > :::::::::::::: > ceph_pg_404.1ff.json > :::::::::::::: > { > "snap_trimq": "[]", > "snap_trimq_len": 0, > "state": "active+recovery_unfound+degraded+remapped", > "epoch": 141748, > "up": [ > 208, > 220, > 269, > 175, > 343, > 329 > ], > "acting": [ > 208, > 220, > 269, > 175, > 313, > 329 > ], > "backfill_targets": [ > "343(4)" > ], > "acting_recovery_backfill": [ > "175(3)", > "208(0)", > "220(1)", > "269(2)", > "313(4)", > "329(5)", > "343(4)" > ], > "info": { > "pgid": "404.1ffs0", > "last_update": "141747'724161", > "last_complete": "140483'722075", > "log_tail": "138455'718635", > "last_user_version": 724161, > "last_backfill": "MAX", > "purged_snaps": [], > "history": { > "epoch_created": 36041, > "epoch_pool_created": 27862, > "last_epoch_started": 141660, > "last_interval_started": 141659, > "last_epoch_clean": 140452, > "last_interval_clean": 140451, > "last_epoch_split": 108548, > "last_epoch_marked_full": 0, > "same_up_since": 141659, > "same_interval_since": 141659, > "same_primary_since": 141615, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "prior_readable_until_ub": 0 > }, > "stats": { > "version": "141747'724161", > "reported_seq": 3558199, > "reported_epoch": 141748, > "state": "active+recovery_unfound+degraded+remapped", > "last_fresh": "2023-01-04T07:48:25.201886+0000", > "last_change": "2023-01-03T12:15:50.155062+0000", > "last_active": "2023-01-04T07:48:25.201886+0000", > "last_peered": "2023-01-04T07:48:25.201886+0000", > "last_clean": "2022-12-31T19:25:42.758665+0000", > "last_became_active": "2023-01-03T12:15:49.875672+0000", > "last_became_peered": "2023-01-03T12:15:49.875672+0000", > "last_unstale": "2023-01-04T07:48:25.201886+0000", > "last_undegraded": "2023-01-03T12:15:49.758975+0000", > "last_fullsized": "2023-01-04T07:48:25.201886+0000", > "mapping_epoch": 141659, > "log_start": "138455'718635", > "ondisk_log_start": "138455'718635", > "created": 36041, > "last_epoch_clean": 140452, > "parent": "0.0", > "parent_split_bits": 10, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "log_size": 5526, > "ondisk_log_size": 5526, > "stats_invalid": false, > "dirty_stats_invalid": false, > "omap_stats_invalid": false, > "hitset_stats_invalid": false, > "hitset_bytes_stats_invalid": false, > "pin_stats_invalid": false, > "manifest_stats_invalid": false, > "snaptrimq_len": 0, > "stat_sum": { > "num_bytes": 282409304186, > "num_objects": 137910, > "num_object_clones": 0, > "num_object_copies": 827460, > "num_objects_missing_on_primary": 1, > "num_objects_missing": 1, > "num_objects_degraded": 5, > "num_objects_misplaced": 137906, > "num_objects_unfound": 1, > "num_objects_dirty": 137910, > "num_whiteouts": 0, > "num_read": 732600, > "num_read_kb": 181255155, > "num_write": 471445, > "num_write_kb": 102310260, > "num_scrub_errors": 0, > "num_shallow_scrub_errors": 0, > "num_deep_scrub_errors": 0, > "num_objects_recovered": 406583, > "num_bytes_recovered": 891273764570, > "num_keys_recovered": 0, > "num_objects_omap": 0, > "num_objects_hit_set_archive": 0, > "num_bytes_hit_set_archive": 0, > "num_flush": 0, > "num_flush_kb": 0, > "num_evict": 0, > "num_evict_kb": 0, > "num_promote": 0, > "num_flush_mode_high": 0, > "num_flush_mode_low": 0, > "num_evict_mode_some": 0, > "num_evict_mode_full": 0, > "num_objects_pinned": 0, > "num_legacy_snapsets": 0, > "num_large_omap_objects": 0, > "num_objects_manifest": 0, > "num_omap_bytes": 0, > "num_omap_keys": 0, > "num_objects_repaired": 3 > }, > "up": [ > 208, > 220, > 269, > 175, > 343, > 329 > ], > "acting": [ > 208, > 220, > 269, > 175, > 313, > 329 > ], > "avail_no_missing": [ > "329(5)" > ], > "object_location_counts": [ > { > "shards": > "175(3),208(0),220(1),269(2),313(4),329(5)", > "objects": 137909 > }, > { > "shards": "329(5)", > "objects": 1 > } > ], > "blocked_by": [], > "up_primary": 208, > "acting_primary": 208, > "purged_snaps": [] > }, > "empty": 0, > "dne": 0, > "incomplete": 0, > "last_epoch_started": 141660, > "hit_set_history": { > "current_last_update": "0'0", > "history": [] > } > }, > "peer_info": [ > { > "peer": "175(3)", > "pgid": "404.1ffs3", > "last_update": "141747'724161", > "last_complete": "141668'723872", > "log_tail": "138378'718235", > "last_user_version": 723769, > "last_backfill": "MAX", > "purged_snaps": [], > "history": { > "epoch_created": 36041, > "epoch_pool_created": 27862, > "last_epoch_started": 141660, > "last_interval_started": 141659, > "last_epoch_clean": 140452, > "last_interval_clean": 140451, > "last_epoch_split": 108548, > "last_epoch_marked_full": 0, > "same_up_since": 141659, > "same_interval_since": 141659, > "same_primary_since": 141615, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "prior_readable_until_ub": 0 > }, > "stats": { > "version": "141624'723769", > "reported_seq": 3550912, > "reported_epoch": 141624, > "state": "active+recovery_unfound+degraded+remapped", > "last_fresh": "2023-01-03T11:57:15.480994+0000", > "last_change": "2023-01-03T11:31:07.335605+0000", > "last_active": "2023-01-03T11:57:15.480994+0000", > "last_peered": "2023-01-03T11:57:15.480994+0000", > "last_clean": "2022-12-31T19:25:42.758665+0000", > "last_became_active": > "2023-01-03T11:31:07.116154+0000", > "last_became_peered": > "2023-01-03T11:31:07.116154+0000", > "last_unstale": "2023-01-03T11:57:15.480994+0000", > "last_undegraded": "2023-01-03T11:31:07.099558+0000", > "last_fullsized": "2023-01-03T11:57:15.480994+0000", > "mapping_epoch": 141659, > "log_start": "138378'718235", > "ondisk_log_start": "138378'718235", > "created": 36041, > "last_epoch_clean": 140452, > "parent": "0.0", > "parent_split_bits": 10, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "log_size": 5534, > "ondisk_log_size": 5534, > "stats_invalid": false, > "dirty_stats_invalid": false, > "omap_stats_invalid": false, > "hitset_stats_invalid": false, > "hitset_bytes_stats_invalid": false, > "pin_stats_invalid": false, > "manifest_stats_invalid": false, > "snaptrimq_len": 0, > "stat_sum": { > "num_bytes": 281850993538, > "num_objects": 137711, > "num_object_clones": 0, > "num_object_copies": 826266, > "num_objects_missing_on_primary": 1, > "num_objects_missing": 1, > "num_objects_degraded": 5, > "num_objects_misplaced": 137707, > "num_objects_unfound": 1, > "num_objects_dirty": 137711, > "num_whiteouts": 0, > "num_read": 720741, > "num_read_kb": 180044133, > "num_write": 469883, > "num_write_kb": 101652239, > "num_scrub_errors": 0, > "num_shallow_scrub_errors": 0, > "num_deep_scrub_errors": 0, > "num_objects_recovered": 406583, > "num_bytes_recovered": 891273764570, > "num_keys_recovered": 0, > "num_objects_omap": 0, > "num_objects_hit_set_archive": 0, > "num_bytes_hit_set_archive": 0, > "num_flush": 0, > "num_flush_kb": 0, > "num_evict": 0, > "num_evict_kb": 0, > "num_promote": 0, > "num_flush_mode_high": 0, > "num_flush_mode_low": 0, > "num_evict_mode_some": 0, > "num_evict_mode_full": 0, > "num_objects_pinned": 0, > "num_legacy_snapsets": 0, > "num_large_omap_objects": 0, > "num_objects_manifest": 0, > "num_omap_bytes": 0, > "num_omap_keys": 0, > "num_objects_repaired": 3 > }, > "up": [ > 208, > 220, > 269, > 175, > 343, > 329 > ], > "acting": [ > 208, > 220, > 269, > 175, > 313, > 329 > ], > "avail_no_missing": [ > "329(5)" > ], > "object_location_counts": [ > { > "shards": > "175(3),208(0),220(1),269(2),313(4),329(5)", > "objects": 137710 > }, > { > "shards": "329(5)", > "objects": 1 > } > ], > "blocked_by": [], > "up_primary": 208, > "acting_primary": 208, > "purged_snaps": [] > }, > "empty": 0, > "dne": 0, > "incomplete": 0, > "last_epoch_started": 141660, > "hit_set_history": { > "current_last_update": "0'0", > "history": [] > } > }, > { > "peer": "220(1)", > "pgid": "404.1ffs1", > "last_update": "141747'724161", > "last_complete": "141668'723872", > "log_tail": "138378'718235", > "last_user_version": 723769, > "last_backfill": "MAX", > "purged_snaps": [], > "history": { > "epoch_created": 36041, > "epoch_pool_created": 27862, > "last_epoch_started": 141660, > "last_interval_started": 141659, > "last_epoch_clean": 140452, > "last_interval_clean": 140451, > "last_epoch_split": 108548, > "last_epoch_marked_full": 0, > "same_up_since": 141659, > "same_interval_since": 141659, > "same_primary_since": 141615, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "prior_readable_until_ub": 0 > }, > "stats": { > "version": "141624'723769", > "reported_seq": 3550912, > "reported_epoch": 141624, > "state": "active+recovery_unfound+degraded+remapped", > "last_fresh": "2023-01-03T11:57:15.480994+0000", > "last_change": "2023-01-03T11:31:07.335605+0000", > "last_active": "2023-01-03T11:57:15.480994+0000", > "last_peered": "2023-01-03T11:57:15.480994+0000", > "last_clean": "2022-12-31T19:25:42.758665+0000", > "last_became_active": > "2023-01-03T11:31:07.116154+0000", > "last_became_peered": > "2023-01-03T11:31:07.116154+0000", > "last_unstale": "2023-01-03T11:57:15.480994+0000", > "last_undegraded": "2023-01-03T11:31:07.099558+0000", > "last_fullsized": "2023-01-03T11:57:15.480994+0000", > "mapping_epoch": 141659, > "log_start": "138378'718235", > "ondisk_log_start": "138378'718235", > "created": 36041, > "last_epoch_clean": 140452, > "parent": "0.0", > "parent_split_bits": 10, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "log_size": 5534, > "ondisk_log_size": 5534, > "stats_invalid": false, > "dirty_stats_invalid": false, > "omap_stats_invalid": false, > "hitset_stats_invalid": false, > "hitset_bytes_stats_invalid": false, > "pin_stats_invalid": false, > "manifest_stats_invalid": false, > "snaptrimq_len": 0, > "stat_sum": { > "num_bytes": 281850993538, > "num_objects": 137711, > "num_object_clones": 0, > "num_object_copies": 826266, > "num_objects_missing_on_primary": 1, > "num_objects_missing": 1, > "num_objects_degraded": 5, > "num_objects_misplaced": 137707, > "num_objects_unfound": 1, > "num_objects_dirty": 137711, > "num_whiteouts": 0, > "num_read": 720741, > "num_read_kb": 180044133, > "num_write": 469883, > "num_write_kb": 101652239, > "num_scrub_errors": 0, > "num_shallow_scrub_errors": 0, > "num_deep_scrub_errors": 0, > "num_objects_recovered": 406583, > "num_bytes_recovered": 891273764570, > "num_keys_recovered": 0, > "num_objects_omap": 0, > "num_objects_hit_set_archive": 0, > "num_bytes_hit_set_archive": 0, > "num_flush": 0, > "num_flush_kb": 0, > "num_evict": 0, > "num_evict_kb": 0, > "num_promote": 0, > "num_flush_mode_high": 0, > "num_flush_mode_low": 0, > "num_evict_mode_some": 0, > "num_evict_mode_full": 0, > "num_objects_pinned": 0, > "num_legacy_snapsets": 0, > "num_large_omap_objects": 0, > "num_objects_manifest": 0, > "num_omap_bytes": 0, > "num_omap_keys": 0, > "num_objects_repaired": 3 > }, > "up": [ > 208, > 220, > 269, > 175, > 343, > 329 > ], > "acting": [ > 208, > 220, > 269, > 175, > 313, > 329 > ], > "avail_no_missing": [ > "329(5)" > ], > "object_location_counts": [ > { > "shards": > "175(3),208(0),220(1),269(2),313(4),329(5)", > "objects": 137710 > }, > { > "shards": "329(5)", > "objects": 1 > } > ], > "blocked_by": [], > "up_primary": 208, > "acting_primary": 208, > "purged_snaps": [] > }, > "empty": 0, > "dne": 0, > "incomplete": 0, > "last_epoch_started": 141660, > "hit_set_history": { > "current_last_update": "0'0", > "history": [] > } > }, > { > "peer": "269(2)", > "pgid": "404.1ffs2", > "last_update": "141747'724161", > "last_complete": "141668'723872", > "log_tail": "138378'718235", > "last_user_version": 723769, > "last_backfill": "MAX", > "purged_snaps": [], > "history": { > "epoch_created": 36041, > "epoch_pool_created": 27862, > "last_epoch_started": 141660, > "last_interval_started": 141659, > "last_epoch_clean": 140452, > "last_interval_clean": 140451, > "last_epoch_split": 108548, > "last_epoch_marked_full": 0, > "same_up_since": 141659, > "same_interval_since": 141659, > "same_primary_since": 141615, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "prior_readable_until_ub": 0 > }, > "stats": { > "version": "141624'723769", > "reported_seq": 3550912, > "reported_epoch": 141624, > "state": "active+recovery_unfound+degraded+remapped", > "last_fresh": "2023-01-03T11:57:15.480994+0000", > "last_change": "2023-01-03T11:31:07.335605+0000", > "last_active": "2023-01-03T11:57:15.480994+0000", > "last_peered": "2023-01-03T11:57:15.480994+0000", > "last_clean": "2022-12-31T19:25:42.758665+0000", > "last_became_active": > "2023-01-03T11:31:07.116154+0000", > "last_became_peered": > "2023-01-03T11:31:07.116154+0000", > "last_unstale": "2023-01-03T11:57:15.480994+0000", > "last_undegraded": "2023-01-03T11:31:07.099558+0000", > "last_fullsized": "2023-01-03T11:57:15.480994+0000", > "mapping_epoch": 141659, > "log_start": "138378'718235", > "ondisk_log_start": "138378'718235", > "created": 36041, > "last_epoch_clean": 140452, > "parent": "0.0", > "parent_split_bits": 10, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "log_size": 5534, > "ondisk_log_size": 5534, > "stats_invalid": false, > "dirty_stats_invalid": false, > "omap_stats_invalid": false, > "hitset_stats_invalid": false, > "hitset_bytes_stats_invalid": false, > "pin_stats_invalid": false, > "manifest_stats_invalid": false, > "snaptrimq_len": 0, > "stat_sum": { > "num_bytes": 281850993538, > "num_objects": 137711, > "num_object_clones": 0, > "num_object_copies": 826266, > "num_objects_missing_on_primary": 1, > "num_objects_missing": 1, > "num_objects_degraded": 5, > "num_objects_misplaced": 137707, > "num_objects_unfound": 1, > "num_objects_dirty": 137711, > "num_whiteouts": 0, > "num_read": 720741, > "num_read_kb": 180044133, > "num_write": 469883, > "num_write_kb": 101652239, > "num_scrub_errors": 0, > "num_shallow_scrub_errors": 0, > "num_deep_scrub_errors": 0, > "num_objects_recovered": 406583, > "num_bytes_recovered": 891273764570, > "num_keys_recovered": 0, > "num_objects_omap": 0, > "num_objects_hit_set_archive": 0, > "num_bytes_hit_set_archive": 0, > "num_flush": 0, > "num_flush_kb": 0, > "num_evict": 0, > "num_evict_kb": 0, > "num_promote": 0, > "num_flush_mode_high": 0, > "num_flush_mode_low": 0, > "num_evict_mode_some": 0, > "num_evict_mode_full": 0, > "num_objects_pinned": 0, > "num_legacy_snapsets": 0, > "num_large_omap_objects": 0, > "num_objects_manifest": 0, > "num_omap_bytes": 0, > "num_omap_keys": 0, > "num_objects_repaired": 3 > }, > "up": [ > 208, > 220, > 269, > 175, > 343, > 329 > ], > "acting": [ > 208, > 220, > 269, > 175, > 313, > 329 > ], > "avail_no_missing": [ > "329(5)" > ], > "object_location_counts": [ > { > "shards": > "175(3),208(0),220(1),269(2),313(4),329(5)", > "objects": 137710 > }, > { > "shards": "329(5)", > "objects": 1 > } > ], > "blocked_by": [], > "up_primary": 208, > "acting_primary": 208, > "purged_snaps": [] > }, > "empty": 0, > "dne": 0, > "incomplete": 0, > "last_epoch_started": 141660, > "hit_set_history": { > "current_last_update": "0'0", > "history": [] > } > }, > { > "peer": "313(4)", > "pgid": "404.1ffs4", > "last_update": "141747'724161", > "last_complete": "141668'723872", > "log_tail": "138228'717435", > "last_user_version": 723769, > "last_backfill": "MAX", > "purged_snaps": [], > "history": { > "epoch_created": 36041, > "epoch_pool_created": 27862, > "last_epoch_started": 141660, > "last_interval_started": 141659, > "last_epoch_clean": 140452, > "last_interval_clean": 140451, > "last_epoch_split": 108548, > "last_epoch_marked_full": 0, > "same_up_since": 141659, > "same_interval_since": 141659, > "same_primary_since": 141615, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "prior_readable_until_ub": 0 > }, > "stats": { > "version": "141624'723769", > "reported_seq": 3551314, > "reported_epoch": 141659, > "state": "active+recovery_unfound+degraded+remapped", > "last_fresh": "2023-01-03T11:57:15.480994+0000", > "last_change": "2023-01-03T11:31:07.335605+0000", > "last_active": "2023-01-03T11:57:15.480994+0000", > "last_peered": "2023-01-03T11:57:15.480994+0000", > "last_clean": "2022-12-31T19:25:42.758665+0000", > "last_became_active": > "2023-01-03T11:31:07.116154+0000", > "last_became_peered": > "2023-01-03T11:31:07.116154+0000", > "last_unstale": "2023-01-03T11:57:15.480994+0000", > "last_undegraded": "2023-01-03T11:31:07.099558+0000", > "last_fullsized": "2023-01-03T11:57:15.480994+0000", > "mapping_epoch": 141659, > "log_start": "138378'718235", > "ondisk_log_start": "138378'718235", > "created": 36041, > "last_epoch_clean": 140452, > "parent": "0.0", > "parent_split_bits": 10, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "log_size": 5534, > "ondisk_log_size": 5534, > "stats_invalid": false, > "dirty_stats_invalid": false, > "omap_stats_invalid": false, > "hitset_stats_invalid": false, > "hitset_bytes_stats_invalid": false, > "pin_stats_invalid": false, > "manifest_stats_invalid": false, > "snaptrimq_len": 0, > "stat_sum": { > "num_bytes": 281850993538, > "num_objects": 137711, > "num_object_clones": 0, > "num_object_copies": 826266, > "num_objects_missing_on_primary": 1, > "num_objects_missing": 1, > "num_objects_degraded": 5, > "num_objects_misplaced": 137707, > "num_objects_unfound": 1, > "num_objects_dirty": 137711, > "num_whiteouts": 0, > "num_read": 720741, > "num_read_kb": 180044133, > "num_write": 469883, > "num_write_kb": 101652239, > "num_scrub_errors": 0, > "num_shallow_scrub_errors": 0, > "num_deep_scrub_errors": 0, > "num_objects_recovered": 406583, > "num_bytes_recovered": 891273764570, > "num_keys_recovered": 0, > "num_objects_omap": 0, > "num_objects_hit_set_archive": 0, > "num_bytes_hit_set_archive": 0, > "num_flush": 0, > "num_flush_kb": 0, > "num_evict": 0, > "num_evict_kb": 0, > "num_promote": 0, > "num_flush_mode_high": 0, > "num_flush_mode_low": 0, > "num_evict_mode_some": 0, > "num_evict_mode_full": 0, > "num_objects_pinned": 0, > "num_legacy_snapsets": 0, > "num_large_omap_objects": 0, > "num_objects_manifest": 0, > "num_omap_bytes": 0, > "num_omap_keys": 0, > "num_objects_repaired": 3 > }, > "up": [ > 208, > 220, > 269, > 175, > 343, > 329 > ], > "acting": [ > 208, > 220, > 269, > 175, > 313, > 329 > ], > "avail_no_missing": [ > "329(5)" > ], > "object_location_counts": [ > { > "shards": > "175(3),208(0),220(1),269(2),313(4),329(5)", > "objects": 137710 > }, > { > "shards": "329(5)", > "objects": 1 > } > ], > "blocked_by": [], > "up_primary": 208, > "acting_primary": 208, > "purged_snaps": [] > }, > "empty": 0, > "dne": 0, > "incomplete": 0, > "last_epoch_started": 141660, > "hit_set_history": { > "current_last_update": "0'0", > "history": [] > } > }, > { > "peer": "329(5)", > "pgid": "404.1ffs5", > "last_update": "141747'724161", > "last_complete": "141668'723872", > "log_tail": "138378'718235", > "last_user_version": 723769, > "last_backfill": "MAX", > "purged_snaps": [], > "history": { > "epoch_created": 36041, > "epoch_pool_created": 27862, > "last_epoch_started": 141660, > "last_interval_started": 141659, > "last_epoch_clean": 140452, > "last_interval_clean": 140451, > "last_epoch_split": 108548, > "last_epoch_marked_full": 0, > "same_up_since": 141659, > "same_interval_since": 141659, > "same_primary_since": 141615, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "prior_readable_until_ub": 0 > }, > "stats": { > "version": "141624'723769", > "reported_seq": 3550912, > "reported_epoch": 141624, > "state": "active+recovery_unfound+degraded+remapped", > "last_fresh": "2023-01-03T11:57:15.480994+0000", > "last_change": "2023-01-03T11:31:07.335605+0000", > "last_active": "2023-01-03T11:57:15.480994+0000", > "last_peered": "2023-01-03T11:57:15.480994+0000", > "last_clean": "2022-12-31T19:25:42.758665+0000", > "last_became_active": > "2023-01-03T11:31:07.116154+0000", > "last_became_peered": > "2023-01-03T11:31:07.116154+0000", > "last_unstale": "2023-01-03T11:57:15.480994+0000", > "last_undegraded": "2023-01-03T11:31:07.099558+0000", > "last_fullsized": "2023-01-03T11:57:15.480994+0000", > "mapping_epoch": 141659, > "log_start": "138378'718235", > "ondisk_log_start": "138378'718235", > "created": 36041, > "last_epoch_clean": 140452, > "parent": "0.0", > "parent_split_bits": 10, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "log_size": 5534, > "ondisk_log_size": 5534, > "stats_invalid": false, > "dirty_stats_invalid": false, > "omap_stats_invalid": false, > "hitset_stats_invalid": false, > "hitset_bytes_stats_invalid": false, > "pin_stats_invalid": false, > "manifest_stats_invalid": false, > "snaptrimq_len": 0, > "stat_sum": { > "num_bytes": 281850993538, > "num_objects": 137711, > "num_object_clones": 0, > "num_object_copies": 826266, > "num_objects_missing_on_primary": 1, > "num_objects_missing": 0, > "num_objects_degraded": 5, > "num_objects_misplaced": 137707, > "num_objects_unfound": 1, > "num_objects_dirty": 137711, > "num_whiteouts": 0, > "num_read": 720741, > "num_read_kb": 180044133, > "num_write": 469883, > "num_write_kb": 101652239, > "num_scrub_errors": 0, > "num_shallow_scrub_errors": 0, > "num_deep_scrub_errors": 0, > "num_objects_recovered": 406583, > "num_bytes_recovered": 891273764570, > "num_keys_recovered": 0, > "num_objects_omap": 0, > "num_objects_hit_set_archive": 0, > "num_bytes_hit_set_archive": 0, > "num_flush": 0, > "num_flush_kb": 0, > "num_evict": 0, > "num_evict_kb": 0, > "num_promote": 0, > "num_flush_mode_high": 0, > "num_flush_mode_low": 0, > "num_evict_mode_some": 0, > "num_evict_mode_full": 0, > "num_objects_pinned": 0, > "num_legacy_snapsets": 0, > "num_large_omap_objects": 0, > "num_objects_manifest": 0, > "num_omap_bytes": 0, > "num_omap_keys": 0, > "num_objects_repaired": 3 > }, > "up": [ > 208, > 220, > 269, > 175, > 343, > 329 > ], > "acting": [ > 208, > 220, > 269, > 175, > 313, > 329 > ], > "avail_no_missing": [ > "329(5)" > ], > "object_location_counts": [ > { > "shards": > "175(3),208(0),220(1),269(2),313(4),329(5)", > "objects": 137710 > }, > { > "shards": "329(5)", > "objects": 1 > } > ], > "blocked_by": [], > "up_primary": 208, > "acting_primary": 208, > "purged_snaps": [] > }, > "empty": 0, > "dne": 0, > "incomplete": 0, > "last_epoch_started": 141660, > "hit_set_history": { > "current_last_update": "0'0", > "history": [] > } > }, > { > "peer": "343(4)", > "pgid": "404.1ffs4", > "last_update": "141747'724161", > "last_complete": "141668'723872", > "log_tail": "138378'718235", > "last_user_version": 723769, > "last_backfill": > "404:ff80001b:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-09T12%3a41%3a05,478440857+00%3a00.2~XwOh1_IhVzghlM-zi7DfwoUIZl5UxVt.1:head", > "purged_snaps": [], > "history": { > "epoch_created": 36041, > "epoch_pool_created": 27862, > "last_epoch_started": 141660, > "last_interval_started": 141659, > "last_epoch_clean": 140452, > "last_interval_clean": 140451, > "last_epoch_split": 108548, > "last_epoch_marked_full": 0, > "same_up_since": 141659, > "same_interval_since": 141659, > "same_primary_since": 141615, > "last_scrub": "140525'722168", > "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", > "last_deep_scrub": "140525'722168", > "last_deep_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "last_clean_scrub_stamp": > "2022-12-31T19:27:10.993286+0000", > "prior_readable_until_ub": 0 > }, > "stats": { > "version": "0'0", > "reported_seq": 0, > "reported_epoch": 0, > "state": "unknown", > "last_fresh": "0.000000", > "last_change": "0.000000", > "last_active": "0.000000", > "last_peered": "0.000000", > "last_clean": "0.000000", > "last_became_active": "0.000000", > "last_became_peered": "0.000000", > "last_unstale": "0.000000", > "last_undegraded": "0.000000", > "last_fullsized": "0.000000", > "mapping_epoch": 141659, > "log_start": "0'0", > "ondisk_log_start": "0'0", > "created": 0, > "last_epoch_clean": 0, > "parent": "0.0", > "parent_split_bits": 0, > "last_scrub": "0'0", > "last_scrub_stamp": "0.000000", > "last_deep_scrub": "0'0", > "last_deep_scrub_stamp": "0.000000", > "last_clean_scrub_stamp": "0.000000", > "log_size": 0, > "ondisk_log_size": 0, > "stats_invalid": false, > "dirty_stats_invalid": false, > "omap_stats_invalid": false, > "hitset_stats_invalid": false, > "hitset_bytes_stats_invalid": false, > "pin_stats_invalid": false, > "manifest_stats_invalid": false, > "snaptrimq_len": 0, > "stat_sum": { > "num_bytes": 9404416, > "num_objects": 3, > "num_object_clones": 0, > "num_object_copies": 0, > "num_objects_missing_on_primary": 0, > "num_objects_missing": 137907, > "num_objects_degraded": 0, > "num_objects_misplaced": 0, > "num_objects_unfound": 0, > "num_objects_dirty": 3, > "num_whiteouts": 0, > "num_read": 0, > "num_read_kb": 0, > "num_write": 0, > "num_write_kb": 0, > "num_scrub_errors": 0, > "num_shallow_scrub_errors": 0, > "num_deep_scrub_errors": 0, > "num_objects_recovered": 0, > "num_bytes_recovered": 0, > "num_keys_recovered": 0, > "num_objects_omap": 0, > "num_objects_hit_set_archive": 0, > "num_bytes_hit_set_archive": 0, > "num_flush": 0, > "num_flush_kb": 0, > "num_evict": 0, > "num_evict_kb": 0, > "num_promote": 0, > "num_flush_mode_high": 0, > "num_flush_mode_low": 0, > "num_evict_mode_some": 0, > "num_evict_mode_full": 0, > "num_objects_pinned": 0, > "num_legacy_snapsets": 0, > "num_large_omap_objects": 0, > "num_objects_manifest": 0, > "num_omap_bytes": 0, > "num_omap_keys": 0, > "num_objects_repaired": 0 > }, > "up": [ > 208, > 220, > 269, > 175, > 343, > 329 > ], > "acting": [ > 208, > 220, > 269, > 175, > 313, > 329 > ], > "avail_no_missing": [], > "object_location_counts": [], > "blocked_by": [], > "up_primary": 208, > "acting_primary": 208, > "purged_snaps": [] > }, > "empty": 0, > "dne": 0, > "incomplete": 1, > "last_epoch_started": 141660, > "hit_set_history": { > "current_last_update": "0'0", > "history": [] > } > } > ], > "recovery_state": [ > { > "name": "Started/Primary/Active", > "enter_time": "2023-01-03T12:15:49.758685+0000", > "might_have_unfound": [ > { > "osd": "175(3)", > "status": "already probed" > }, > { > "osd": "220(1)", > "status": "already probed" > }, > { > "osd": "269(2)", > "status": "already probed" > }, > { > "osd": "313(4)", > "status": "already probed" > }, > { > "osd": "329(5)", > "status": "already probed" > }, > { > "osd": "343(4)", > "status": "already probed" > } > ], > "recovery_progress": { > "backfill_targets": [ > "343(4)" > ], > "waiting_on_backfill": [], > "last_backfill_started": > "404:ff80001b:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-09T12%3a41%3a05,478440857+00%3a00.2~XwOh1_IhVzghlM-zi7DfwoUIZl5UxVt.1:head", > "backfill_info": { > "begin": "MIN", > "end": "MIN", > "objects": [] > }, > "peer_backfill_info": [], > "backfills_in_flight": [], > "recovering": [], > "pg_backend": { > "recovery_ops": [], > "read_ops": [] > } > } > }, > { > "name": "Started", > "enter_time": "2023-01-03T12:15:48.766366+0000" > } > ], > "scrubber": { > "epoch_start": "0", > "active": false > }, > "agent_state": {} > } > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

User migration between clusters
by Szabo, Istvan (Agoda) 10 Jan '23

10 Jan '23

Hi, Normally I use rclone to migrate buckets across clusters. However this time the user has close to 1000 buckets so I wonder what would be the best approach to do this rather buckets by buckets, any idea? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

ceph orch osd rm - draining forever, shows -1 pgs
by Wyll Ingersoll 09 Jan '23

09 Jan '23

Running ceph-pacific 16.2.9 using ceph orchestrator. We made a mistake adding a disk to the cluster and immediately issued a command to remove it using "ceph orch osd rm ### --replace --force". This OSD no data on it at the time and was removed after just a few minutes. "ceph orch osd rm status" shows that it is still "draining". ceph osd df shows that the osd being removed has -1 PGs. So - why is the simple act of removal taking so long and can we abort it and manually remove that osd somehow? Note: the cluster is also doing a rebalance while this is going on, but the osd being removed never had any data and should not be affected by the rebalance. thanks!

1 0

VolumeGroup must have a non-empty name / 17.2.5
by Peter Eisch 09 Jan '23

09 Jan '23

Hi, I updated from pacific 16.2.10 to 17.2.5 and the orchestration update went perfectly. Very impressive. I have one host which then started throwing a cephadm warning after the upgrade. 2023-01-07 11:17:50,080 7f0b26c8ab80 INFO Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:0560b16bec6e84345f29fb6693cd2430884e6efff16a95d5bdd0bb06d7661c45 -e NODE_NAME=kelli.domain.name -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/404b94ab-b4d6-4218-9a4e-ecb8899108ca:/var/run/ceph:z -v /var/log/ceph/404b94ab-b4d6-4218-9a4e-ecb8899108ca:/var/log/ceph:z -v /var/lib/ceph/404b94ab-b4d6-4218-9a4e-ecb8899108ca/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/404b94ab-b4d6-4218-9a4e-ecb8899108ca/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /tmp/ceph-tmpltrnmxf8:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:0560b16bec6e84345f29fb6693cd2430884e6efff16a95d5bdd0bb06d7661c45 inventory --format=json-pretty --filter-for-batch 2023-01-07 11:17:50,081 7f0b26c8ab80 INFO /usr/bin/podman: stderr Traceback (most recent call last): 2023-01-07 11:17:50,081 7f0b26c8ab80 INFO /usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in <module> 2023-01-07 11:17:50,081 7f0b26c8ab80 INFO /usr/bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2023-01-07 11:17:50,081 7f0b26c8ab80 INFO /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2023-01-07 11:17:50,081 7f0b26c8ab80 INFO /usr/bin/podman: stderr self.main(self.argv) 2023-01-07 11:17:50,082 7f0b26c8ab80 INFO /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2023-01-07 11:17:50,082 7f0b26c8ab80 INFO /usr/bin/podman: stderr return f(*a, **kw) 2023-01-07 11:17:50,082 7f0b26c8ab80 INFO /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2023-01-07 11:17:50,082 7f0b26c8ab80 INFO /usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2023-01-07 11:17:50,082 7f0b26c8ab80 INFO /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2023-01-07 11:17:50,082 7f0b26c8ab80 INFO /usr/bin/podman: stderr instance.main() 2023-01-07 11:17:50,082 7f0b26c8ab80 INFO /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/inventory/main.py", line 53, in main 2023-01-07 11:17:50,082 7f0b26c8ab80 INFO /usr/bin/podman: stderr with_lsm=self.args.with_lsm)) 2023-01-07 11:17:50,082 7f0b26c8ab80 INFO /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 39, in __init__ 2023-01-07 11:17:50,082 7f0b26c8ab80 INFO /usr/bin/podman: stderr all_devices_vgs = lvm.get_all_devices_vgs() 2023-01-07 11:17:50,082 7f0b26c8ab80 INFO /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 797, in get_all_devices_vgs 2023-01-07 11:17:50,083 7f0b26c8ab80 INFO /usr/bin/podman: stderr return [VolumeGroup(**vg) for vg in vgs] 2023-01-07 11:17:50,083 7f0b26c8ab80 INFO /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 797, in <listcomp> 2023-01-07 11:17:50,083 7f0b26c8ab80 INFO /usr/bin/podman: stderr return [VolumeGroup(**vg) for vg in vgs] 2023-01-07 11:17:50,083 7f0b26c8ab80 INFO /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 517, in __init__ 2023-01-07 11:17:50,083 7f0b26c8ab80 INFO /usr/bin/podman: stderr raise ValueError('VolumeGroup must have a non-empty name') 2023-01-07 11:17:50,083 7f0b26c8ab80 INFO /usr/bin/podman: stderr ValueError: VolumeGroup must have a non-empty name This host is the only one which has 14 drives which aren't being used. I'm guessing this is why its getting this error. The drives may have been used previous in a cluster (maybe not the same cluster) or something. I don't know. Any suggestions for what to try to get past this issue? peter Peter Eisch DevOps Manager peter.eisch(a)virginpulse.com T1.612.445.5135 Confidentiality Notice: This email was sent securely using Transport Layer Security (TLS) Encryption. Please ensure your email systems support TLS before replying with any confidential information. The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v3.02

2 1

Mixing SSD and HDD disks for data in ceph cluster deployment
by Michel Niyoyita 09 Jan '23

09 Jan '23

Hello team I have an issue on ceph-deployment using ceph-ansible . we have two categories of disk , HDD and SSD , while deploying ceph only HDD are appearing no SSD appearing . the cluster is running on ubuntu OS 20.04 , unfortunately no errors appearing , did I miss something in configuration? hdd: 7,2936 T ssd: 7 T Kindly advise and help below is how the cluster is behavior, actually we have 20 disks per each host (16 hdd and 4 ssd), from /dev/sda to /dev/sdt but if you look after deployment we have 48 osds instead of 60 osds which are missing are ssd according to ceph osd crush class ls command . root@ceph-mon1:~# ceph osd crush class ls [ "hdd" ] root@ceph-mon1:~# ceph -s cluster: id: 02786875-6dca-46e6-8590-dba92c27e6f8 health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 36m) mgr: ceph-mon1(active, since 5m), standbys: ceph-mon3, ceph-mon2 osd: 48 osds: 48 up (since 30m), 48 in (since 31m) rgw: 3 daemons active (3 hosts, 1 zones) data: pools: 5 pools, 105 pgs objects: 195 objects, 7.7 KiB usage: 21 TiB used, 349 TiB / 370 TiB avail pgs: 105 active+clean and below is the output of lsblk command which shows that there is LVM created on ssd but its size doesn't appear . from /dev/sde to /dev/sdh are ssd . sda 8:0 0 7.3T 0 disk └─ceph--d856437a--0af5--48e1--a99d--b9f8f5b74165-osd--block--2856935f--a293--47b2--a917--7671e91e207d 253:0 0 7.3T 0 lvm sdb 8:16 0 7.3T 0 disk └─ceph--60119ee2--6537--4129--88b2--82bc0994766f-osd--block--c87a88ca--2c34--4a71--bffa--55789475e950 253:2 0 7.3T 0 lvm sdc 8:32 0 7.3T 0 disk └─ceph--1667e64f--8327--4a13--886e--23bbd8d4fb73-osd--block--1b9da829--4210--4956--b3c3--32042c719879 253:5 0 7.3T 0 lvm sdd 8:48 0 7.3T 0 disk └─ceph--4b9a897f--253e--4078--85fc--0133baad23c6-osd--block--fc9d3bb0--0d63--4201--a49a--6204a928bbd4 253:7 0 7.3T 0 lvm sde 8:64 0 7T 0 disk ├─ceph--54bc00ad--d575--4269--b637--a15e4aed88d2-osd--db--21ccd262--0488--49b4--9994--9cf8a61b7bbd 253:26 0 447.1G 0 lvm ├─ceph--54bc00ad--d575--4269--b637--a15e4aed88d2-osd--db--404cefe2--c24c--4955--89c9--28e8d7c44db9 253:28 0 447.1G 0 lvm ├─ceph--54bc00ad--d575--4269--b637--a15e4aed88d2-osd--db--6bea87e8--d883--46a8--b9de--e4b7b311ce20 253:30 0 447.1G 0 lvm └─ceph--54bc00ad--d575--4269--b637--a15e4aed88d2-osd--db--6767b370--a63f--402e--864c--0085bc589baa 253:32 0 447.1G 0 lvm sdf 8:80 0 7T 0 disk ├─ceph--7461bded--a2c2--4874--a08f--fca940f3a511-osd--db--b62845ed--ce46--4afd--8c48--79afd89409ca 253:18 0 447.1G 0 lvm ├─ceph--7461bded--a2c2--4874--a08f--fca940f3a511-osd--db--a40a1938--ea8b--4c9b--bf10--01ca097719ca 253:20 0 447.1G 0 lvm ├─ceph--7461bded--a2c2--4874--a08f--fca940f3a511-osd--db--abfa052f--07e4--4504--9cf8--eee3b527489c 253:22 0 447.1G 0 lvm └─ceph--7461bded--a2c2--4874--a08f--fca940f3a511-osd--db--5eef2536--829f--4893--9b94--51ae55bfd742 253:24 0 447.1G 0 lvm sdg 8:96 0 7T 0 disk ├─ceph--fb912c50--2c94--403b--b690--73ef6f4eda69-osd--db--bd3fff8a--40d2--4ed9--b15e--52812da3a15d 253:10 0 447.1G 0 lvm ├─ceph--fb912c50--2c94--403b--b690--73ef6f4eda69-osd--db--1bd422bf--5d7e--468f--a7ca--3c70d38dcd5b 253:12 0 447.1G 0 lvm ├─ceph--fb912c50--2c94--403b--b690--73ef6f4eda69-osd--db--cecccf90--6b03--47a1--9ce1--8b6d403816e1 253:14 0 447.1G 0 lvm └─ceph--fb912c50--2c94--403b--b690--73ef6f4eda69-osd--db--b8181faf--595b--4118--a35a--e51a527d6e57 253:16 0 447.1G 0 lvm sdh 8:112 0 7T 0 disk ├─ceph--bc347c07--8636--462e--82fd--d81ed149ed2c-osd--db--9d14b346--ff65--4045--95db--4c5cb43e7829 253:1 0 447.1G 0 lvm ├─ceph--bc347c07--8636--462e--82fd--d81ed149ed2c-osd--db--ec81234e--f12f--414d--8210--bf9f6ccd5d7e 253:3 0 447.1G 0 lvm ├─ceph--bc347c07--8636--462e--82fd--d81ed149ed2c-osd--db--c8135b3f--42c5--44f9--885c--6413a16b86b4 253:6 0 447.1G 0 lvm └─ceph--bc347c07--8636--462e--82fd--d81ed149ed2c-osd--db--14ba1fee--a896--4cdf--8ded--ba2e08a4c0d5 253:8 0 447.1G 0 lvm sdi 8:128 0 7.3T 0 disk └─ceph--5737bd78--e275--44f0--af42--20c70f6ecce0-osd--block--3aa9e13f--fee6--4e6f--8696--4dfa855def35 253:9 0 7.3T 0 lvm sdj 8:144 0 7.3T 0 disk └─ceph--5420bf20--6fa4--4d70--8020--3df67e6a664a-osd--block--74157cb4--74f9--43dc--af00--9f180cda1dc9 253:11 0 7.3T 0 lvm sdk 8:160 0 7.3T 0 disk └─ceph--726e1a55--a71b--4fe7--baa2--c48e80a7cabd-osd--block--461ca84d--1bdc--4df8--8235--7b8d7cfd9c86 253:13 0 7.3T 0 lvm sdl 8:176 0 7.3T 0 disk └─ceph--f4dfe3a2--6494--4ccf--9062--991524995bfb-osd--block--f8bd6329--9c62--4b75--8bb9--c34271b4a09b 253:15 0 7.3T 0 lvm sdm 8:192 0 7.3T 0 disk └─ceph--53e9c97b--714b--415c--90bd--7f5faf7d7389-osd--block--8917684c--129a--4754--a0d7--3e347e07de21 253:17 0 7.3T 0 lvm sdn 8:208 0 7.3T 0 disk └─ceph--9f14efa0--208a--4b6c--9063--04b73151eb02-osd--block--a9d289d8--ed05--4e41--8d52--598d3cbaa61d 253:19 0 7.3T 0 lvm sdo 8:224 0 7.3T 0 disk └─ceph--c396bc2b--bf57--4096--816a--58f383dda801-osd--block--cba8794d--8ab4--4765--9df1--9510927d02eb 253:21 0 7.3T 0 lvm sdp 8:240 0 7.3T 0 disk └─ceph--97b6b983--a0e7--4521--b2b4--c8ab8d251ef7-osd--block--616e2606--aa7b--4d36--9996--3ac188b1fb39 253:23 0 7.3T 0 lvm sdq 65:0 0 7.3T 0 disk └─ceph--e36dad3b--b6d2--4949--9e18--920f65807a6a-osd--block--58d102ba--bad8--47eb--8c3f--4ece53ab5d6a 253:25 0 7.3T 0 lvm sdr 65:16 0 7.3T 0 disk └─ceph--9a24f150--d275--456c--8d85--f5db51800557-osd--block--5e751e55--00ed--497c--b6e7--b802108bbb1e 253:27 0 7.3T 0 lvm sds 65:32 0 7.3T 0 disk └─ceph--2daaf0ec--1483--4ffa--b831--846bc2ed3fe5-osd--block--e2a23118--41f4--4285--8b1c--de25419ff56e 253:29 0 7.3T 0 lvm sdt 65:48 0 7.3T 0 disk └─ceph--660fb368--1dad--4275--8868--522f4e60354a-osd--block--e8731791--4560--4b22--aa4f--a650893b100d 253:31 0 7.3T 0 lvm Best Regards Michel

3 3

Erasing Disk to the initial state
by Michel Niyoyita 09 Jan '23

09 Jan '23

Hello team, I have deployed ceph cluster in production , the cluster composed by two types of disks HDD and SSD , and the cluster was deployed using ceph-ansible , unfortunately after deployment the HDD disks appear only without SSD , would like to restart deployment from scratch , but I miss the way on how to erase disk to the initial state . try to format disks but LVM comeback with disks. sda 8:0 0 7.3T 0 disk └─ceph--da4a5d58--73ef--473b--9960--371f837cb5ed-osd--block--6e800937--c4d2--4fc9--84ca--083c39d057a8 253:1 0 7.3T 0 lvm sdb 8:16 0 7.3T 0 disk └─ceph--773f50a1--79ed--4908--8f81--74f85efeb473-osd--block--9737a046--ba8b--4494--91f7--b80dd894df0b 253:7 0 7.3T 0 lvm sdc 8:32 0 7.3T 0 disk └─ceph--02000cec--fdbc--4def--967e--a7c32c851964-osd--block--c54d8182--b5e7--4c73--8d7b--7d24c7a3ce15 253:6 0 7.3T 0 lvm Kindly help me to sort this out. Best regards Michel

2 1

NoSuchBucket when bucket exists ..
by Shashi Dahal 09 Jan '23

09 Jan '23

Hi, In a working All-in-one test setup ( where making the bucket public works from the browser) radosgw-admin bucket list [ "711138fc95764303b83002c567ce0972/demo" ] I have another cluster where openstack and ceph are separate. I have set same config options in ceph.conf .. rgw_enable_apis = swift rgw_keystone_accepted_roles = member, _member_, admin, swiftoperator rgw_keystone_admin_domain = default rgw_keystone_admin_password = **** rgw_keystone_admin_project = service rgw_keystone_admin_user = **** rgw_keystone_api_version = 3 rgw_keystone_implicit_tenants = true rgw_keystone_url = https://<keystone-url>:5000 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true but the output is different radosgw-admin bucket list [ "demo", ] ## this is created without the project-uuid. What is happening is when I make the bucket public, it gives https://cloud.domain.com:8080/swift/v1/AUTH_b9a4b517525a483a9e111044713bfa1… -> NoSuchBucket Please let me know what setting could I be missing so that when the bucket is created, it is created with the project_id as well and the link works when the bucket is public. Thanks

1 1

Serious cluster issue - Incomplete PGs
by Deep Dish 09 Jan '23

09 Jan '23

Hello. I really screwed up my ceph cluster. Hoping to get data off it so I can rebuild it. In summary, too many changes too quickly caused the cluster to develop incomplete pgs. Some PGS were reporting that OSDs were to be probes. I've created those OSD IDs (empty), however this wouldn't clear incompletes. Incompletes are part of EC pools. Running 17.2.5. This is the overall state: cluster: id: 49057622-69fc-11ed-b46e-d5acdedaae33 health: HEALTH_WARN Failed to apply 1 service(s): osd.dashboard-admin-1669078094056 1 hosts fail cephadm check cephadm background work is paused Reduced data availability: 28 pgs inactive, 28 pgs incomplete Degraded data redundancy: 55 pgs undersized 2 slow ops, oldest one blocked for 4449 sec, daemons [osd.25,osd.50,osd.51] have slow ops. These are PGs that are incomplete that HAVE DATA (Objects > 0) [ via ceph pg ls incomplete ]: 2.35 23199 0 0 0 95980273664 0 0 2477 incomplete 10s 2104'46277 28260:686871 [44,4,37,3,40,32]p44 [44,4,37,3,40,32]p44 2023-01-03T03:54:47.821280+0000 2022-12-29T18:53:09.287203+0000 14 queued for deep scrub 2.53 22821 0 0 0 94401175552 0 0 2745 remapped+incomplete 10s 2104'45845 28260:565267 [60,48,52,65,67,7]p60 [60]p60 2023-01-03T10:18:13.388383+0000 2023-01-03T10:18:13.388383+0000 408 queued for scrub 2.9f 22858 0 0 0 94555983872 0 0 2736 remapped+incomplete 10s 2104'45636 28260:759872 [56,59,3,57,5,32]p56 [56]p56 2023-01-03T10:55:49.848693+0000 2023-01-03T10:55:49.848693+0000 376 queued for scrub 2.be 22870 0 0 0 94429110272 0 0 2661 remapped+incomplete 10s 2104'45561 28260:813759 [41,31,37,9,7,69]p41 [41]p41 2023-01-03T14:02:15.790077+0000 2023-01-03T14:02:15.790077+0000 360 queued for scrub 2.e4 22953 0 0 0 94912278528 0 0 2648 remapped+incomplete 20m 2104'46048 28259:732896 [37,46,33,4,48,49]p37 [37]p37 2023-01-02T18:38:46.268723+0000 2022-12-29T18:05:47.431468+0000 18 queued for deep scrub 17.78 20169 0 0 0 84517834400 0 0 2198 remapped+incomplete 10s 3735'53405 28260:1243673 [4,37,2,36,66,0]p4 [41]p41 2023-01-03T14:21:41.563424+0000 2023-01-03T14:21:41.563424+0000 348 queued for scrub 17.d8 20328 0 0 0 85196053130 0 0 1852 remapped+incomplete 10s 3735'54458 28260:1309564 [38,65,61,37,58,39]p38 [53]p53 2023-01-02T18:32:35.371071+0000 2022-12-28T19:08:29.492244+0000 21 queued for deep scrub At present I'm unable to reliably access my data due to incomplete pages above. I'll post whatever outputs requested (won't post now as it can be rather verbose). Is there hope?

2 1

Re: mon scrub error (scrub mismatch)
by Frank Schilder 09 Jan '23

09 Jan '23

Hi Dan, it went unnoticed and is in all log files + rotated. I also wondered about the difference in #auth keys and looked at it. However, we have only 23 auth keys (its a small test cluster). No idea what the 77/78 mean. Maybe including some history? I went ahead and rebuilt the mon store before I got your e-mail, so no more debugging possible unless some log info might be useful. I'm more wondering why this is not flagged as a health issue. Is it harmless? What if things degrade even more over time? In older versions (well, luminous) it seems that it was flagged as an error. It would also be nice to have a command like "ceph mon repair" or "ceph mon resync" instead of having to do a complete manual daemon rebuild. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 03 January 2023 19:41 To: Frank Schilder Cc: Eugen Block; ceph-users(a)ceph.io Subject: Re: [ceph-users] Re: mon scrub error (scrub mismatch) Hi Frank, Can you work backwards in the logs to when this first appeared? The scrub error is showing that mon.0 has 78 auth keys and the other two have 77. So you'd have query the auth keys of each mon to see if you get a different response each time (e.g. ceph auth list), and compare with what you expect. Cheers, Dan On Tue, Jan 3, 2023 at 9:29 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi Eugen, > > thanks for your answer. All our mons use rocksdb. > > I found some old threads, but they never really explained anything. What irritates me is that this is a silent corruption. If you don't read the logs every day, you will not see it, ceph status reports health ok. That's also why I'm wondering if this is a real issue or not. > > It would be great if someone could shed light on (1) how serious this is, (2) why it doesn't trigger a health warning/error and (3) why the affected mon doesn't sync back from the majority right away. > > Thanks and best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Eugen Block <eblock(a)nde.ag> > Sent: 03 January 2023 15:04:34 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: mon scrub error (scrub mismatch) > > Hi Frank, > > I had this a few years back and ended up recreating the MON with the > scrub mismatch, so in your case it probably would be mon.0. To test if > the problem still exists you can trigger a mon scrub manually: > > ceph mon scrub > > Are all MONs on rocksdb back end in this cluster? I didn't check back > then if this was the case in our cluster, so I'm just wondering if > that could be an explanation. > > Regards, > Eugen > > Zitat von Frank Schilder <frans(a)dtu.dk>: > > > Hi all, > > > > we have these messages in our logs daily: > > > > 1/3/23 12:20:00 PM[INF]overall HEALTH_OK > > 1/3/23 12:19:46 PM[ERR] mon.2 ScrubResult(keys > > {auth=77,config=2,health=11,logm=10} crc > > {auth=688385498,config=4279003239,health=3522308637,logm=132403602}) > > 1/3/23 12:19:46 PM[ERR] mon.0 ScrubResult(keys > > {auth=78,config=2,health=11,logm=9} crc > > {auth=325876668,config=4279003239,health=3522308637,logm=1083913445}) > > 1/3/23 12:19:46 PM[ERR]scrub mismatch > > 1/3/23 12:19:46 PM[ERR] mon.1 ScrubResult(keys > > {auth=77,config=2,health=11,logm=10} crc > > {auth=688385498,config=4279003239,health=3522308637,logm=132403602}) > > 1/3/23 12:19:46 PM[ERR] mon.0 ScrubResult(keys > > {auth=78,config=2,health=11,logm=9} crc > > {auth=325876668,config=4279003239,health=3522308637,logm=1083913445}) > > 1/3/23 12:19:46 PM[ERR]scrub mismatch > > 1/3/23 12:17:04 PM[INF]Cluster is now healthy > > 1/3/23 12:17:04 PM[INF]Health check cleared: MON_CLOCK_SKEW (was: > > clock skew detected on mon.tceph-02) > > > > Cluster is health OK: > > > > # ceph status > > cluster: > > id: bf1f51f5-b381-4cf7-b3db-88d044c1960c > > health: HEALTH_OK > > > > services: > > mon: 3 daemons, quorum tceph-01,tceph-02,tceph-03 (age 3M) > > mgr: tceph-01(active, since 8w), standbys: tceph-03, tceph-02 > > mds: fs:1 {0=tceph-02=up:active} 2 up:standby > > osd: 9 osds: 9 up (since 3M), 9 in > > > > task status: > > > > data: > > pools: 4 pools, 321 pgs > > objects: 9.94M objects, 336 GiB > > usage: 1.6 TiB used, 830 GiB / 2.4 TiB avail > > pgs: 321 active+clean > > > > Unfortunately, google wasn't of too much help. Is this scrub error > > something to worry about? > > > > Thanks and best regards, > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: MDS crashes to damaged metadata
by Venky Shankar 09 Jan '23

09 Jan '23

Hi Felix, On Thu, Dec 15, 2022 at 8:03 PM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) This crash is the same as detailed in https://tracker.ceph.com/issues/49132. Fix is under backport to p/q releases. > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. > > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. > > Regards > Felix > --------------------------------------------------------------------------------------------- > --------------------------------------------------------------------------------------------- > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Volker Rieke > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior > --------------------------------------------------------------------------------------------- > --------------------------------------------------------------------------------------------- > > Am 02.12.2022 um 20:08 schrieb Patrick Donnelly <pdonnell(a)redhat.com>: > > On Thu, Dec 1, 2022 at 5:08 PM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Script is running for ~2 hours and according to the line count in the memo file we are at 40% (cephfs is still online). > > We had to modify the script putting a try/catch arround the for loop in line 78 to 87. For some reasons there are some objects (186 at this moment) which throw an UnicodeDecodeError exception during the iteration: > > <rados.OmapIterator object at 0x7f9606f8bcf8> Traceback (most recent call last): File "first-damage.py", line 138, in <module> traverse(f, ioctx) File "first-damage.py", line 79, in traverse for (dnk, val) in it: File "rados.pyx", line 1382, in rados.OmapIterator.__next__ File "rados.pyx", line 311, in rados.decode_cstr UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 10-11: invalid continuation byte > > Don’t know if this is because of the filesystem still running. We saved the object names in a separate file and i will investigate further tomorrow. We should be able to modify the script to only check for the objects which threw the exception instead of searching through the whole pool again. > > That shouldn't be caused by teh fs running. It may be you have some > file names which have invalid unicode characters? > > Regarding the mds logfiles with debug 20: > We cannot run this debug level for longer than one hour since the logfile size increase is to high for the local storage on the mds servers where logs are stored (don’t have a central logging yet). > > Okay. > > But if you are just interested in the time frame arround the crash, i could set the debug level to 20, trigger the crash on the weekend and sent you the logs. > > The crash is unlikely to point to what causes the corruption. I was > hoping we could locate an instance of damage while the MDS is running. > > Regards Felix > > > --------------------------------------------------------------------------------------------- > --------------------------------------------------------------------------------------------- > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Volker Rieke > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior > --------------------------------------------------------------------------------------------- > --------------------------------------------------------------------------------------------- > > Am 01.12.2022 um 20:51 schrieb Patrick Donnelly <pdonnell(a)redhat.com>: > > On Thu, Dec 1, 2022 at 3:55 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > > I set debug_mds=20 in ceph.conf and inserted it on the running daemon via "ceph daemon mds.mon-e2-1 config set debug_mds 20“. I have to check with my superiors, if i am allowed to provide yout the logs though. > > > Suggest using `ceph config set` instead of ceph.conf. It's much easier. > > Regarding the tool: > <pool> is refering to the cephfs_metadata pool? (just want to be sure) > > > Yes. > > How long will the runs gonna take? We have 15M Objects in our metadata pool and 330M in data pools > > > Not sure. You can monitor the number of lines generated on the memo > file to get an idea of objects/s. > > You can speed test the tool without bringing the file system by > **not** using `--remove`. > > Regarding the root cause: > As far as i can tell, all damaged inodes have been only accessed via two samba servers running with ctdb. We are also running nfs gateways on different systems, but there hasn’t been a damaged inode (yet). > > Samba Servers running Ubuntu 18.04 with kernel 5.4.0-132 and samba version 4.7.6. > Cephfs is accessed via kernel mount and > > ceph version is 16.2.10 across all nodes > we have one filesystem and two data pools and using cehpfs snapshots > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Principal Software Engineer > Red Hat, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > > > > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Principal Software Engineer > Red Hat, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Cheers, Venky

1 0

OSD crash on Onode::put
by Dongdong Tao 09 Jan '23

09 Jan '23

-- Resending this mail, it seems ceph-users(a)ceph.io was down for the last few days. I see many users recently reporting that they have been struggling with this Onode::put race condition issue[1] on both the latest Octopus and pacific. Igor opened a PR [2] to address this issue, I've been reviewing it for a while and looks good to me. I'm hoping this could get some priority from the community. For those who had been hitting this issue, I would like to share a workaround that could very likely unblock you: During the investigation of this issue, I found this race condition always happens after the bluestore onode cache size becomes 0. Setting debug_bluestore = 1/30 will allow you to see the cache size after the crash: --- 2022-10-25T00:47:26.562+0000 7f424f78e700 30 bluestore.MempoolThread(0x564a9dae2a68) _resize_shards max_shard_onodes: 0 max_shard_buffer: 8388608 --- This is apparently wrong as this means the bluestore metadata cache is basically disabled, but it makes much sense to explain why we are hitting the race condition so easily -- An onode will be trimmed right away after it's unpinned. Keep going with the investigation, it turns out the culprit for the 0-sized cache is the leak that happened in bluestore_cache_other mempool Please refer to the bug tracker [3] which has the detail of the leak issue, it was already fixed by [4], and the next Pacific point release will have it. But it was never backported to Octopus, so if you are hitting the same: For Octopus, you can manually backport this patch to fix the leak and prevent the race condition from happening. For Pacific, you can wait for 16.2.11 (or manually backport the fix as well if can't wait). By the way, I'm backporting the fix to ubuntu Octopus and Pacific through this SRU [5], so it will be landed in ubuntu's package soon. [1] https://tracker.ceph.com/issues/56382 [2] https://github.com/ceph/ceph/pull/47702 [3] https://tracker.ceph.com/issues/56424 [4] https://github.com/ceph/ceph/pull/46911 [5] https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1996010 Cheers, Dongdong

1 0

ceph-users list archive missing almost all mail
by Matthias Ferdinand 08 Jan '23

08 Jan '23

Hi, I found some mailing list archive links from my notes to throw "Page not found" errors, e.g. https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/J4U24YRJEJ… Looking around in the archive web interface, it appears only some of the most recent threads are found, everything else says "no email threads could be found for this month". Could somebody please look into this? Regards Matthias Ferdinand

1 0

Serious cluster issue - data inaccessible
by Deep Dish 08 Jan '23

08 Jan '23

Hello. I really screwed up my ceph cluster. Hoping to get data off it so I can rebuild it. In summary, too many changes too quickly caused the cluster to develop incomplete pgs. Some PGS were reporting that OSDs were to be probes. I've created those OSD IDs (empty), however this wouldn't clear incompletes. Incompletes are part of EC pools. Running 17.2.5. This is the overall state: cluster: id: 49057622-69fc-11ed-b46e-d5acdedaae33 health: HEALTH_WARN Failed to apply 1 service(s): osd.dashboard-admin-1669078094056 1 hosts fail cephadm check cephadm background work is paused Reduced data availability: 28 pgs inactive, 28 pgs incomplete Degraded data redundancy: 55 pgs undersized 2 slow ops, oldest one blocked for 4449 sec, daemons [osd.25,osd.50,osd.51] have slow ops. These are PGs that are incomplete that HAVE DATA (Objects > 0) [ via ceph pg ls incomplete ]: 2.35 23199 0 0 0 95980273664 0 0 2477 incomplete 10s 2104'46277 28260:686871 [44,4,37,3,40,32]p44 [44,4,37,3,40,32]p44 2023-01-03T03:54:47.821280+0000 2022-12-29T18:53:09.287203+0000 14 queued for deep scrub 2.53 22821 0 0 0 94401175552 0 0 2745 remapped+incomplete 10s 2104'45845 28260:565267 [60,48,52,65,67,7]p60 [60]p60 2023-01-03T10:18:13.388383+0000 2023-01-03T10:18:13.388383+0000 408 queued for scrub 2.9f 22858 0 0 0 94555983872 0 0 2736 remapped+incomplete 10s 2104'45636 28260:759872 [56,59,3,57,5,32]p56 [56]p56 2023-01-03T10:55:49.848693+0000 2023-01-03T10:55:49.848693+0000 376 queued for scrub 2.be 22870 0 0 0 94429110272 0 0 2661 remapped+incomplete 10s 2104'45561 28260:813759 [41,31,37,9,7,69]p41 [41]p41 2023-01-03T14:02:15.790077+0000 2023-01-03T14:02:15.790077+0000 360 queued for scrub 2.e4 22953 0 0 0 94912278528 0 0 2648 remapped+incomplete 20m 2104'46048 28259:732896 [37,46,33,4,48,49]p37 [37]p37 2023-01-02T18:38:46.268723+0000 2022-12-29T18:05:47.431468+0000 18 queued for deep scrub 17.78 20169 0 0 0 84517834400 0 0 2198 remapped+incomplete 10s 3735'53405 28260:1243673 [4,37,2,36,66,0]p4 [41]p41 2023-01-03T14:21:41.563424+0000 2023-01-03T14:21:41.563424+0000 348 queued for scrub 17.d8 20328 0 0 0 85196053130 0 0 1852 remapped+incomplete 10s 3735'54458 28260:1309564 [38,65,61,37,58,39]p38 [53]p53 2023-01-02T18:32:35.371071+0000 2022-12-28T19:08:29.492244+0000 21 queued for deep scrub At present I'm unable to reliably access my data due to incomplete pages above. I'll post whatever outputs requested (won't post now as it can be rather verbose). Is there hope?

1 0

Setting Prometheus retention_time
by Robert Sander 08 Jan '23

08 Jan '23

Hi, The Quincy documentation shows that we could set the Prometheus retention_time within a service specification: https://docs.ceph.com/en/quincy/cephadm/services/monitoring/#setting-up-pro… When trying this "ceph orch apply" only shows: Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'retention_time' It looks like release 17.2.5 does not contain this code yet. Why is the content of the documentation already online when https://github.com/ceph/ceph/pull/47943 has not been released yet? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 1

test message
by Joe Comeau 08 Jan '23

08 Jan '23

Hi Just testing as I have not received a message from the list in a couple days Thanks Joe

1 0

Re: Missing SSDs disk on ceph deployment
by Michel Niyoyita 08 Jan '23

08 Jan '23

Dear team, Kindly help on this , I am completely blocked. Best Regards Michel On Thu, Jan 5, 2023 at 2:45 PM Michel Niyoyita <micou12(a)gmail.com> wrote: > Dear team, > > I have deployed the ceph cluster in production using ceph-ansible on > ubuntu OS 20.04 it consists of 3 monitors and 3 osds (each osd has 20 disks > , 16 hdd and 4 ssd) after deployment the ceph cluster healthy is OK , but > instead of getting 60 disk it appears 48 disks which are hdd only > (according to the output of ceph osd df tree) below are output of ceph osd > df tree and lsblk command. > > hdd: 7.71349 T > ssd: 7 T > > ceph osd df tree > root@ceph-mon3:~# ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS TYPE NAME > -1 370.24731 - 370 TiB 21 TiB 249 MiB 0 B 1.2 GiB > 349 TiB 5.66 1.00 - root default > -5 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 403 MiB > 116 TiB 5.66 1.00 - host ceph-osd1 > 0 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 6 up osd.0 > 3 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 8 up osd.3 > 6 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 35 MiB > 7.3 TiB 5.66 1.00 12 up osd.6 > 9 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 4 up osd.9 > 12 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 6 up osd.12 > 16 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 3 up osd.16 > 18 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 5 up osd.18 > 21 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 6 up osd.21 > 24 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 35 MiB > 7.3 TiB 5.66 1.00 7 up osd.24 > 27 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 7 up osd.27 > 30 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 7 up osd.30 > 33 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 7 up osd.33 > 36 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 5 up osd.36 > 39 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 9 up osd.39 > 42 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 6 up osd.42 > 45 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 7 up osd.45 > -3 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 397 MiB > 116 TiB 5.66 1.00 - host ceph-osd2 > 1 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 7 up osd.1 > 5 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 7 up osd.5 > 8 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 6 up osd.8 > 11 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 5 up osd.11 > 14 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 7 up osd.14 > 15 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 12 up osd.15 > 19 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 5 up osd.19 > 22 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 4 up osd.22 > 25 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 2 up osd.25 > 28 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 3 up osd.28 > 31 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 6 up osd.31 > 34 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 7 up osd.34 > 37 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.37 > 40 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.40 > 43 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 6 up osd.43 > 46 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 12 up osd.46 > -7 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 409 MiB > 116 TiB 5.66 1.00 - host ceph-osd3 > 2 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 4 up osd.2 > 4 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 7 up osd.4 > 7 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 5 up osd.7 > 10 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 4 up osd.10 > 13 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 12 up osd.13 > 17 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 6 up osd.17 > 20 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 7 up osd.20 > 23 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 8 up osd.23 > 26 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.26 > 29 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 6 up osd.29 > 32 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 6 up osd.32 > 35 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.35 > 38 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 9 up osd.38 > 41 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 3 up osd.41 > 44 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 4 up osd.44 > 47 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.47 > TOTAL 370 TiB 21 TiB 249 MiB 0 B 1.2 GiB > 349 TiB 5.66 > > lsblk out put > ========== > root@ceph-osd1:~# lsblk > NAME > MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > loop0 > 7:0 0 55.4M 1 loop /snap/core18/1944 > loop1 > 7:1 0 69.9M 1 loop /snap/lxd/19188 > loop2 > 7:2 0 31.1M 1 loop /snap/snapd/10707 > sda > 8:0 0 7.3T 0 disk > └─ceph--f011a10f--3f22--4785--8fe8--1eb4e64d380f-osd--block--1c3eaee6--60a7--4b2d--aff1--dff0a909d48e > 253:1 0 7.3T 0 lvm > sdb > 8:16 0 7.3T 0 disk > └─ceph--ba3fef60--84a2--4cd7--9aeb--4af05a17733d-osd--block--a4926772--4913--4ad1--8c01--348e8cb30159 > 253:3 0 7.3T 0 lvm > sdc > 8:32 0 7.3T 0 disk > └─ceph--14676d53--0a82--47b2--acc2--10ebae5e2cb1-osd--block--b1032109--80e0--4065--923a--80541669c316 > 253:5 0 7.3T 0 lvm > sdd > 8:48 0 7.3T 0 disk > └─ceph--2e7875da--a136--4b39--b354--21cfa8643951-osd--block--5a1ad840--225c--4d79--ab23--8589f13b9f3d > 253:7 0 7.3T 0 lvm > sde > 8:64 0 7T 0 disk > ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--be9d5f0f--ac60--42e1--850d--bf2baf913b3b > 253:26 0 447.1G 0 lvm > ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--5e7310c4--2b18--4938--ab6e--dc37e860d9d5 > 253:28 0 447.1G 0 lvm > ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--d2d78859--2e60--4ccc--b417--d33ff55837b6 > 253:30 0 447.1G 0 lvm > └─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--31ff57f6--e9d3--4bf1--b08d--95939083b4da > 253:32 0 447.1G 0 lvm > sdf > 8:80 0 7T 0 disk > ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--7a8d393e--3af3--4c89--a99f--d3efcbb78066 > 253:18 0 447.1G 0 lvm > ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--5abd44ca--aed8--48af--8f12--a64a6ae7938c > 253:20 0 447.1G 0 lvm > ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--cd5768aa--b9ce--4c3b--9bc8--38c01b5bdc49 > 253:22 0 447.1G 0 lvm > └─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--364e42ff--6e66--4a76--9df2--b2bb04080cd0 > 253:24 0 447.1G 0 lvm > sdg > 8:96 0 7T 0 disk > ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--3c469d07--0f1d--4544--861a--dd84283332fe > 253:10 0 447.1G 0 lvm > ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--61d08fe1--8d5a--4ef8--ba2d--ba1fc84636db > 253:12 0 447.1G 0 lvm > ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--2b2a8af7--33f5--4ca5--b09b--2f24ebf583b3 > 253:14 0 447.1G 0 lvm > └─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--fd2e105a--4fdf--4f15--a48b--1303717b8961 > 253:16 0 447.1G 0 lvm > sdh > 8:112 0 7T 0 disk > ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--6e289081--5404--4294--a75a--f4a1a8fabe52 > 253:2 0 447.1G 0 lvm > ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--e89285a3--4efa--4239--bbb7--a95064ced7f2 > 253:4 0 447.1G 0 lvm > ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--0b7f1fd0--bc32--47b7--b8e4--c6f0f9f21047 > 253:6 0 447.1G 0 lvm > └─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--614e085d--ce04--4f5d--a485--ddca51bae65b > 253:8 0 447.1G 0 lvm > sdi > 8:128 0 7.3T 0 disk > └─ceph--8343ded8--765b--4038--9855--5e6684b55906-osd--block--054a0d83--4aea--4d4f--b623--77db554e207d > 253:9 0 7.3T 0 lvm > sdj > 8:144 0 7.3T 0 disk > └─ceph--88f64697--ab62--4fed--a1b4--88ef3563eaea-osd--block--97189221--a7f6--4d9d--b98a--a474fec95fc7 > 253:11 0 7.3T 0 lvm > sdk > 8:160 0 7.3T 0 disk > └─ceph--592f15db--face--49d8--b028--15fee31d77cd-osd--block--776b773d--347f--4a5c--8de7--66d4a9d56feb > 253:13 0 7.3T 0 lvm > sdl > 8:176 0 7.3T 0 disk > └─ceph--3ef3da28--0aa0--4bfe--80fa--7871a0ebbb2a-osd--block--9b91fccb--3562--4543--916a--6fd0baa00efd > 253:15 0 7.3T 0 lvm > sdm > 8:192 0 7.3T 0 disk > └─ceph--604f22a2--bf89--40a8--bd37--bb5c317f0525-osd--block--ff387140--c528--40e2--ac75--3add53fd5bbb > 253:17 0 7.3T 0 lvm > sdn > 8:208 0 7.3T 0 disk > └─ceph--b702f897--06ad--43fe--b04b--3e03477ffbc3-osd--block--07cc426f--b4c1--4459--a6bc--81ee7a034060 > 253:19 0 7.3T 0 lvm > sdo > 8:224 0 7.3T 0 disk > └─ceph--5e1b4c03--f01b--45d7--971f--a02fe109ef13-osd--block--653b2370--dcb0--46bc--a71b--f62af6777feb > 253:21 0 7.3T 0 lvm > sdp > 8:240 0 7.3T 0 disk > └─ceph--e3ed1f19--15f6--4bd2--81f0--93f9720d7fad-osd--block--4121fcc9--1c00--4da5--b01e--a041280d330a > 253:23 0 7.3T 0 lvm > sdq > 65:0 0 7.3T 0 disk > └─ceph--417cb977--615a--4b86--a4a9--d6af7587b670-osd--block--e77b7eab--7dd8--4300--bd98--27d010c2e17e > 253:25 0 7.3T 0 lvm > sdr > 65:16 0 7.3T 0 disk > └─ceph--ad66d2d6--581d--4985--b945--154586b39903-osd--block--6b1b3de2--8e13--4361--bcd0--16ad609177b0 > 253:27 0 7.3T 0 lvm > sds > 65:32 0 7.3T 0 disk > └─ceph--66aebaac--de6b--42ff--94f1--a489a755e51a-osd--block--a581f3e1--7d86--4f8f--9d0a--de87ef713436 > 253:29 0 7.3T 0 lvm > sdt > 65:48 0 7.3T 0 disk > └─ceph--1558f2d7--741a--4feb--9e68--a900b823eac7-osd--block--52d32f21--9f3a--491e--a26c--41304d63a0ee > 253:31 0 7.3T 0 lvm > sdu > 65:64 0 223.5G 0 disk > ├─sdu1 > 65:65 0 512M 0 part /boot/efi > ├─sdu2 > 65:66 0 1G 0 part /boot > └─sdu3 > 65:67 0 222G 0 part > └─ubuntu--vg-ubuntu--lv > 253:0 0 111G 0 lvm / > > Kindly help me to configure in order to get all disk up and work. > > Best Regards > > Michel > > >

1 0

Re: docs.ceph.com -- Do you use the header navigation bar? (RESPONSES REQUESTED)
by John Mulligan 08 Jan '23

08 Jan '23

On Wednesday, January 4, 2023 10:35:56 AM EST John Zachary Dover wrote: > Do you use the header navigation bar on docs.ceph.com? See the attached > file (sticky_header.png) if you are unsure of what "header navigation bar" > means. In the attached file, the header navigation bar is indicated by > means of two large, ugly, red-and-green arrows. > > *Cards on the Table* > The navigation bar is the kind of thing that is sometimes referred to as a > "sticky header", and it can get in the way of linked-to sections. I would > like to remove this header bar. If there is community support for the > header bar, though, I won't remove it. > > *What is Zac Complaining About?* > Follow this procedure to see the behavior that has provoked my complaint: > > 1. Go to https://docs.ceph.com/en/quincy/glossary/ > 2. Scroll down to the "Ceph Cluster Map" entry. > 3. Click the "Cluster Map" link in the line that reads "See Cluster Map". > 4. Notice that the header navigation bar obscures the headword "Cluster > Map". > > If you have any opinion at all on this matter, voice it. Please. > FWIW I am not able to reproduce the problem you are describing. In all cases the thin blue-green bar appeared above the term with the selected anchor link. I tried Firefox (108, Linux), Chromium (107, Linux) and for giggles Firefox on Android. In all cases things looked fine to me and the selected term was not hidden by that nav bar. I share because I was surprised by the result given that others on the list seem to see the problem. But I also don't see what I would describe as "two large, ugly, red-and-green arrows." Perhaps the page is rendering differently for some people and we don't hit the issue in that case? PS. I also didn't see the png file in question. Perhaps this list strips attachments?

1 0