- ceph-users - lists.ceph.io

ceph orch ps shows version, container and image id as unknown
by Adiga, Anantha 27 Mar '23

27 Mar '23

Hi, Has anybody noticed this? ceph orch ps shows version, container and image id as unknown only for mon, mgr and osds. Ceph health is OK and all daemons are running fine. cephadm ls shows values for version, container and image id. root@cr21meg16ba0101:~# cephadm shell ceph orch ps Inferring fsid a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID crash.cr21meg16ba0101 cr21meg16ba0101 running (4d) 2m ago 5w 7107k - 16.2.5 6e73176320aa 1001776a7c02 crash.cr21meg16ba0102 cr21meg16ba0102 running (5w) 2m ago 5w 63.8M - 16.2.5 6e73176320aa ecfd19d15dbb crash.cr21meg16ba0103 cr21meg16ba0103 running (5w) 2m ago 5w 8131k - 16.2.5 6e73176320aa c508ad3979b0 grafana.cr21meg16ba0101 cr21meg16ba0101 *:3000 running (4d) 2m ago 4d 58.0M - 6.7.4 be4c69a1aae8 ce2741a091c7 grafana.cr21meg16ba0102 cr21meg16ba0102 *:3000 running (4d) 2m ago 7d 53.0M - 6.7.4 be4c69a1aae8 c09f53b31999 grafana.cr21meg16ba0103 cr21meg16ba0103 *:3000 running (4d) 2m ago 7d 54.5M - 6.7.4 be4c69a1aae8 e58f6d9f44a2 haproxy.nfs.nfs-1.cr21meg16ba0101.cwsweq cr21meg16ba0101 *:2049,9049 running (4d) 2m ago 5w 66.7M - 2.3.21-3ce4ee0 7ecd3fda00f4 c5f4d94b5354 haproxy.nfs.nfs-1.cr21meg16ba0102.yodyxa cr21meg16ba0102 *:2049,9049 running (5w) 2m ago 5w 75.9M - 2.3.21-3ce4ee0 7ecd3fda00f4 0a6629e27463 haproxy.rgw.default.default.cr21meg16ba0101.ecpnxq cr21meg16ba0101 *:80,9050 running (4d) 2m ago 5w 102M - 2.3.21-3ce4ee0 7ecd3fda00f4 3c61d34b8b7d haproxy.rgw.default.default.cr21meg16ba0102.nffdzb cr21meg16ba0102 *:80,9050 running (5w) 2m ago 5w 114M - 2.3.21-3ce4ee0 7ecd3fda00f4 406ee603a311 haproxy.rgw.default.default.cr21meg16ba0103.lvypmb cr21meg16ba0103 *:80,9050 running (5w) 2m ago 5w 108M - 2.3.21-3ce4ee0 7ecd3fda00f4 a514c26c0a8e keepalived.nfs.nfs-1.cr21meg16ba0101.qpvesr cr21meg16ba0101 running (4d) 2m ago 5w 26.9M - 2.0.5 073e0c3cd1b9 c4003cb45da6 keepalived.nfs.nfs-1.cr21meg16ba0102.hedpuo cr21meg16ba0102 running (5w) 2m ago 5w 38.0M - 2.0.5 073e0c3cd1b9 b654e661493b keepalived.rgw.default.default.cr21meg16ba0101.biaqvq cr21meg16ba0101 running (4d) 2m ago 5w 39.2M - 2.0.5 073e0c3cd1b9 020c7cb700c4 keepalived.rgw.default.default.cr21meg16ba0102.dufodx cr21meg16ba0102 running (5w) 2m ago 5w 46.0M - 2.0.5 073e0c3cd1b9 fe218ecaf398 keepalived.rgw.default.default.cr21meg16ba0103.utplxz cr21meg16ba0103 running (5w) 2m ago 5w 45.1M - 2.0.5 073e0c3cd1b9 18a99c36ef29 mds.cephfs.cr21meg16ba0101.tmfknc cr21meg16ba0101 running (4d) 2m ago 5w 29.5M - 16.2.5 6e73176320aa e753a2498ccf mds.cephfs.cr21meg16ba0102.vdrcvi cr21meg16ba0102 running (5w) 2m ago 5w 212M - 16.2.5 6e73176320aa 925f151da4de mds.cephfs.cr21meg16ba0103.yacxeu cr21meg16ba0103 running (5w) 2m ago 5w 38.2M - 16.2.5 6e73176320aa 79599f7ca3c8 mgr.cr21meg16ba0101 cr21meg16ba0101 running 2m ago 5w - - <unknown> <unknown> <unknown> mgr.cr21meg16ba0102 cr21meg16ba0102 running 2m ago 5w - - <unknown> <unknown> <unknown> mgr.cr21meg16ba0103 cr21meg16ba0103 running 2m ago 5w - - <unknown> <unknown> <unknown> mon.cr21meg16ba0101 cr21meg16ba0101 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> mon.cr21meg16ba0102 cr21meg16ba0102 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> mon.cr21meg16ba0103 cr21meg16ba0103 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> nfs.nfs-1.0.63.cr21meg16ba0102.kkxpfh cr21meg16ba0102 *:12049 running (5w) 2m ago 5w 132M - 3.5 6e73176320aa fd617b99d70a nfs.nfs-1.1.0.cr21meg16ba0101.qhdmrq cr21meg16ba0101 *:12049 running (4d) 2m ago 5w 117M - 3.5 6e73176320aa 15e21064041d nfs.nfs-1.2.0.cr21meg16ba0103.wvmquo cr21meg16ba0103 *:12049 running (5w) 2m ago 5w 136M - 3.5 6e73176320aa edb9df63489d node-exporter.cr21meg16ba0101 cr21meg16ba0101 *:9100 running (4d) 2m ago 5w 79.3M - 0.17.0 b3e7f67a1480 e94d14814946 node-exporter.cr21meg16ba0102 cr21meg16ba0102 *:9100 running (5w) 2m ago 5w 102M - 0.17.0 b3e7f67a1480 86165fa7b916 node-exporter.cr21meg16ba0103 cr21meg16ba0103 *:9100 running (5w) 2m ago 5w 88.1M - 0.17.0 b3e7f67a1480 898008baaa66 osd.0 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.1 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.10 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.11 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.12 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.13 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.14 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.15 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.16 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.17 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.18 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.19 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.2 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.20 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.21 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.22 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.23 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.24 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.25 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.26 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.27 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.28 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.29 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.3 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.30 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.31 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.32 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.33 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.34 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.35 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.36 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.37 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.38 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.39 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.4 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.40 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.41 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.42 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.43 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.44 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.45 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.46 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.47 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.48 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.49 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.5 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.50 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.51 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.52 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.53 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.54 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.55 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.56 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.57 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.58 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.59 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.6 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.60 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.61 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.62 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.63 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.64 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.65 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.66 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.67 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.68 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.69 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.7 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.70 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.71 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.8 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.9 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> prometheus.cr21meg16ba0101 cr21meg16ba0101 *:9095 running (4d) 2m ago 4d 358M - 2.7.2 2a68492b2466 2aeb3c975175 prometheus.cr21meg16ba0102 cr21meg16ba0102 *:9095 running (5d) 2m ago 5d 362M - 2.7.2 2a68492b2466 b5669b2b4756 rgw.default.default.cr21meg16ba0101.ppytbp cr21meg16ba0101 *:8080 running (4d) 2m ago 5w 188M - 16.2.5 6e73176320aa c13967bccdfc rgw.default.default.cr21meg16ba0102.fnmtmo cr21meg16ba0102 *:8080 running (5w) 2m ago 5w 168M - 16.2.5 6e73176320aa ddaa73b49c08 rgw.default.default.cr21meg16ba0103.xhxprp cr21meg16ba0103 *:8080 running (5w) 2m ago 5w 298M - 16.2.5 6e73176320aa cff4d77b7283 root@cr21meg16ba0101:~# cephadm shell ceph -s Inferring fsid a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 cluster: id: a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe health: HEALTH_OK services: mon: 3 daemons, quorum cr21meg16ba0101,cr21meg16ba0102,cr21meg16ba0103 (age 4d) mgr: cr21meg16ba0101(active, since 42h), standbys: cr21meg16ba0103, cr21meg16ba0102 mds: 1/1 daemons up, 2 standby osd: 72 osds: 72 up (since 4d), 72 in (since 5w) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 16 pools, 577 pgs objects: 3.86M objects, 12 TiB usage: 18 TiB used, 108 TiB / 126 TiB avail pgs: 577 active+clean io: client: 255 B/s rd, 115 KiB/s wr, 0 op/s rd, 3 op/s wr root@cr21meg16ba0101:~# cephadm shell ceph health Inferring fsid a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 HEALTH_OK root@cr21meg16ba0101:~#

1 0

ceph orch ps shows unknown in version, container and image id columns
by anantha.adiga＠intel.com 27 Mar '23

27 Mar '23

Hi Has anybody noticed this issue? For all mgr, mon and osd daemons orch ps shows version, container and image ids as unknown. Ceph health is ok and all daemons are running fine. cephadm ls shows correct details of version, container and image ids . What could be the issue? and how to resolve it? ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) cephadm shell ceph -s Inferring fsid a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 cluster: id: a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe health: HEALTH_OK services: mon: 3 daemons, quorum cr21meg16ba0101,cr21meg16ba0102,cr21meg16ba0103 (age 4d) mgr: cr21meg16ba0101(active, since 42h), standbys: cr21meg16ba0103, cr21meg16ba0102 mds: 1/1 daemons up, 2 standby osd: 72 osds: 72 up (since 4d), 72 in (since 5w) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 16 pools, 577 pgs objects: 3.86M objects, 12 TiB usage: 18 TiB used, 108 TiB / 126 TiB avail pgs: 577 active+clean io: client: 255 B/s rd, 115 KiB/s wr, 0 op/s rd, 3 op/s wr mds.cephfs.cr21meg16ba0103.yacxeu cr21meg16ba0103 running (5w) 2m ago 5w 38.2M - 16.2.5 6e73176320aa 79599f7ca3c8 mgr.cr21meg16ba0101 cr21meg16ba0101 running 2m ago 5w - - <unknown> <unknown> <unknown> mgr.cr21meg16ba0102 cr21meg16ba0102 running 2m ago 5w - - <unknown> <unknown> <unknown> mgr.cr21meg16ba0103 cr21meg16ba0103 running 2m ago 5w - - <unknown> <unknown> <unknown> mon.cr21meg16ba0101 cr21meg16ba0101 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> mon.cr21meg16ba0102 cr21meg16ba0102 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> mon.cr21meg16ba0103 cr21meg16ba0103 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> nfs.nfs-1.0.63.cr21meg16ba0102.kkxpfh cr21meg16ba0102 *:12049 running (5w) 2m ago 5w 132M - 3.5 6e73176320aa fd617b99d70a nfs.nfs-1.1.0.cr21meg16ba0101.qhdmrq cr21meg16ba0101 *:12049 running (4d) 2m ago 5w 117M - 3.5 6e73176320aa 15e21064041d nfs.nfs-1.2.0.cr21meg16ba0103.wvmquo cr21meg16ba0103 *:12049 running (5w) 2m ago 5w 136M - 3.5 6e73176320aa edb9df63489d node-exporter.cr21meg16ba0101 cr21meg16ba0101 *:9100 running (4d) 2m ago 5w 79.3M - 0.17.0 b3e7f67a1480 e94d14814946 node-exporter.cr21meg16ba0102 cr21meg16ba0102 *:9100 running (5w) 2m ago 5w 102M - 0.17.0 b3e7f67a1480 86165fa7b916 node-exporter.cr21meg16ba0103 cr21meg16ba0103 *:9100 running (5w) 2m ago 5w 88.1M - 0.17.0 b3e7f67a1480 898008baaa66 osd.0 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.1 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown>

1 0

Ceph cluster out of balance after adding OSDs
by Pat Vaughan 27 Mar '23

27 Mar '23

We setup a small Ceph cluster about 6 months ago with just 6x 200GB OSDs with one EC 4x2 pool. When we created that pool, we enabled pg_autoscale. The OSDs stayed pretty well balanced. After our developers released a new "feature" that caused the storage to balloon up to over 80%, we added another 6x 200GB OSDs. When we did that, we looked at the number of PGs for that pool, and found that there was only 1 for the rgw.data and rgw.log pools, and "osd pool autoscale-status" doesn't return anything, so it looks like that hasn't been working. The rebalance operation was extremely slow, and wasn't balancing out osd.0, so we bumped up the PGs for the rgw.data pool to 16. All the OSDs except osd.0 balanced out quickly, but that one OSDs utilization keeps climbing, and the number of misplaced objects is increasing, rather than decreasing. We set noscrub and nodeep-scrub so scrubbing wouldn't slow down the process. At this point, I don't want to do any more tuning to this cluster until we can get it back to a healthy state, but it's not fixing itself. I'm open to any ideas. Here's the output of ceph -s: cluster: id: 159d23e4-2a36-11ed-8b6e-fd27d573fa65 health: HEALTH_WARN 1 pools have many more objects per pg than average noscrub,nodeep-scrub flag(s) set 1 backfillfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 12 pgs backfill_toofull 7 pool(s) backfillfull services: mon: 3 daemons, quorum ceph3,ceph5,ceph6 (age 6h) mgr: ceph5.ksxevx(active, since 23h), standbys: ceph4.frkyyl, ceph6.slvpzl osd: 12 osds: 12 up (since 11h), 12 in (since 11h); 12 remapped pgs flags noscrub,nodeep-scrub rgw: 3 daemons active (3 hosts, 1 zones) data: pools: 7 pools, 161 pgs objects: 28.61M objects, 211 GiB usage: 1.5 TiB used, 834 GiB / 2.3 TiB avail pgs: 91779228/171665865 objects misplaced (53.464%) 149 active+clean 12 active+remapped+backfill_toofull io: client: 11 KiB/s rd, 61 KiB/s wr, 11 op/s rd, 27 op/s wr progress: Global Recovery Event (23h) [=========================...] (remaining: 115m) ceph df: --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 2.3 TiB 834 GiB 1.5 TiB 1.5 TiB 65.24 TOTAL 2.3 TiB 834 GiB 1.5 TiB 1.5 TiB 65.24 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 897 KiB 2 2.6 MiB 0.18 479 MiB .rgw.root 2 32 7.1 KiB 18 204 KiB 0.01 479 MiB charlotte.rgw.log 3 32 27 KiB 347 2.0 MiB 0.14 479 MiB charlotte.rgw.control 4 32 0 B 9 0 B 0 479 MiB charlotte.rgw.meta 5 32 9.7 KiB 16 167 KiB 0.01 479 MiB charlotte.rgw.buckets.data 6 16 734 GiB 28.61M 1.1 TiB 99.87 958 MiB charlotte.rgw.buckets.index 7 16 16 GiB 691 47 GiB 97.12 479 MiB ceph osd tree: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2.34357 root default -3 0.39059 host ceph1 0 ssd 0.19530 osd.0 up 0.89999 1.00000 1 ssd 0.19530 osd.1 up 1.00000 1.00000 -5 0.39059 host ceph2 6 ssd 0.19530 osd.6 up 1.00000 1.00000 7 ssd 0.19530 osd.7 up 1.00000 1.00000 -7 0.39059 host ceph3 2 ssd 0.19530 osd.2 up 1.00000 1.00000 8 ssd 0.19530 osd.8 up 1.00000 1.00000 -9 0.39059 host ceph4 3 ssd 0.19530 osd.3 up 1.00000 1.00000 9 ssd 0.19530 osd.9 up 1.00000 1.00000 -11 0.39059 host ceph5 4 ssd 0.19530 osd.4 up 1.00000 1.00000 10 ssd 0.19530 osd.10 up 1.00000 1.00000 -13 0.39059 host ceph6 5 ssd 0.19530 osd.5 up 1.00000 1.00000 11 ssd 0.19530 osd.11 up 1.00000 1.00000 ceph osd df: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 ssd 0.19530 0.89999 200 GiB 190 GiB 130 GiB 12 GiB 48 GiB 10 GiB 94.94 1.46 52 up 1 ssd 0.19530 1.00000 200 GiB 7.3 GiB 9.8 MiB 6.4 GiB 858 MiB 193 GiB 3.64 0.06 42 up 6 ssd 0.19530 1.00000 200 GiB 148 GiB 97 GiB 14 GiB 38 GiB 52 GiB 74.06 1.14 51 up 7 ssd 0.19530 1.00000 200 GiB 133 GiB 97 GiB 2 KiB 35 GiB 67 GiB 66.47 1.02 43 up 2 ssd 0.19530 1.00000 200 GiB 134 GiB 97 GiB 12 KiB 37 GiB 66 GiB 66.94 1.03 40 up 8 ssd 0.19530 1.00000 200 GiB 136 GiB 97 GiB 2.2 GiB 36 GiB 64 GiB 67.85 1.04 40 up 3 ssd 0.19530 1.00000 200 GiB 134 GiB 97 GiB 4 KiB 37 GiB 66 GiB 66.95 1.03 41 up 9 ssd 0.19530 1.00000 200 GiB 138 GiB 97 GiB 5.2 GiB 36 GiB 62 GiB 69.19 1.06 49 up 4 ssd 0.19530 1.00000 200 GiB 137 GiB 97 GiB 4.3 GiB 36 GiB 63 GiB 68.62 1.05 42 up 10 ssd 0.19530 1.00000 200 GiB 139 GiB 97 GiB 5.5 GiB 36 GiB 61 GiB 69.31 1.06 48 up 5 ssd 0.19530 1.00000 200 GiB 134 GiB 97 GiB 7 KiB 38 GiB 66 GiB 67.13 1.03 34 up 11 ssd 0.19530 1.00000 200 GiB 136 GiB 97 GiB 2.2 GiB 36 GiB 64 GiB 67.80 1.04 49 up TOTAL 2.3 TiB 1.5 TiB 1.1 TiB 52 GiB 414 GiB 834 GiB 65.24 MIN/MAX VAR: 0.06/1.46 STDDEV: 19.95 Thanks in advance if anyone has any suggestions.

2 1

EC profiles where m>k (EC 8+12)
by Fabien Sirjean 27 Mar '23

27 Mar '23

Hi Ceph users! I've been proposed an interesting EC setup I hadn't thought about before. Scenario is : we have two server rooms and want to store ~4PiB with the ability to loose 1 server room without loss of data or RW availability. For the context, performance is not needed (cold storage mostly, used as a big filesystem). The idea is to use EC 8+12 over 24 servers (12 on each server room), so if we loose 1 room we still have half of the EC parts (10/20) and are able to loose 2 more servers before reaching the point where we loose data. I find this pretty elegant when working on a two-sites context, as efficiency is 40% (better than 33% three times replication) and the redundancy is good. What do you think of this setup ? Did you ever used EC profiles with M > K ? Thanks for sharing your thoughts! Cheers, Fabien

5 5

Workload performance varying between 2 executions
by Nguetchouang Ngongang Kevin 27 Mar '23

27 Mar '23

Good morning. I just setup a ceph environment, 9 storage nodes and i mount it on a cephfs on a 10th independent node. I execute a fio workload once i got a 3Mb/s throughput. When i reexecute the same workload after a certain time i got this time a 9Mb/s throughput. Do you know why this is happening ? Cordially, -- Nguetchouang Ngongang Kevin ENS de Lyon https://perso.ens-lyon.fr/kevin.nguetchouang/

1 0

cephadm cluster move /var/lib/docker to separate device fails
by Karsten Nielsen 25 Mar '23

25 Mar '23

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

2 1

Unbalanced OSDs when pg_autoscale enabled
by 郑亮 24 Mar '23

24 Mar '23

Hi all, I have a 9 node cluster running *Pacific 16.2.10*. OSDs live on 9 of the nodes with each one having 4 x 1.8T ssd and 8 x 10.9T hdd for a total of 108 OSDs. We create three crush roots as belows. 1. The hdds (8x9=72) of all nodes form a large crush root, which is used as a data pool, and object storage and cephfs share this crush root. 2. Take 3 ssds from the 4 ssds of each node as rbd block storage. 3. An ssd on each remaining node is used as an index pool for cephfs and object storage. [root@node01 smd]# ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -92 15.71910 root root-1c31624a-ad18-445e-8e42-86b71c1fd76f -112 1.74657 host node01-fa2cdf3e-7212-4b5f-b62a-3ab1e803547f 13 ssd 1.74657 osd.13 up 1.00000 1.00000 -103 1.74657 host node02-4e232f27-fe4b-4d0e-bd2a-67d5006a0cdd 34 ssd 1.74657 osd.34 up 1.00000 1.00000 -109 1.74657 host node03-3ae63d7a-9f65-4bea-b2ba-ff3fe342753d 28 ssd 1.74657 osd.28 up 1.00000 1.00000 -118 1.74657 host node04-37a3f92a-f6d8-41f9-a774-3069fc2f50b8 54 ssd 1.74657 osd.54 up 1.00000 1.00000 -106 1.74657 host node05-f667fa27-cc13-4b93-ad56-5dc4c31ffd77 53 ssd 1.74657 osd.53 up 1.00000 1.00000 -91 1.74657 host node06-3808c8f6-8e10-47c7-8456-62c1e0e800ed 61 ssd 1.74657 osd.61 up 1.00000 1.00000 -97 1.74657 host node07-78216b0d-0999-44e8-905d-8737a5f6f51f 50 ssd 1.74657 osd.50 up 1.00000 1.00000 -115 1.74657 host node08-947bd556-fb06-497d-8f2c-c4a679d2b06f 86 ssd 1.74657 osd.86 up 1.00000 1.00000 -100 1.74657 host node09-d9ae9046-0716-454f-ba0c-b03cf9986ba8 85 ssd 1.74657 osd.85 up 1.00000 1.00000 -38 785.80701 root root-6041a4dc-7c9a-44ed-999c-a847cca81012 -85 87.31189 host node01-e5646053-2cf8-4ba5-90d5-bb1a63b1234c 1 hdd 10.91399 osd.1 up 1.00000 1.00000 22 hdd 10.91399 osd.22 up 0.90002 1.00000 31 hdd 10.91399 osd.31 up 1.00000 1.00000 51 hdd 10.91399 osd.51 up 1.00000 1.00000 60 hdd 10.91399 osd.60 up 1.00000 1.00000 70 hdd 10.91399 osd.70 up 1.00000 1.00000 78 hdd 10.91399 osd.78 up 1.00000 1.00000 96 hdd 10.91399 osd.96 up 1.00000 1.00000 -37 87.31189 host node02-be9925fd-60de-4147-81eb-720d7145715f 9 hdd 10.91399 osd.9 up 1.00000 1.00000 19 hdd 10.91399 osd.19 up 1.00000 1.00000 29 hdd 10.91399 osd.29 up 1.00000 1.00000 47 hdd 10.91399 osd.47 up 1.00000 1.00000 56 hdd 10.91399 osd.56 up 1.00000 1.00000 65 hdd 10.91399 osd.65 up 1.00000 1.00000 88 hdd 10.91399 osd.88 up 1.00000 1.00000 98 hdd 10.91399 osd.98 up 1.00000 1.00000 -52 87.31189 host node03-7828653d-6033-4e88-92b0-d8709b0ab218 2 hdd 10.91399 osd.2 up 1.00000 1.00000 30 hdd 10.91399 osd.30 up 1.00000 1.00000 40 hdd 10.91399 osd.40 up 1.00000 1.00000 48 hdd 10.91399 osd.48 up 1.00000 1.00000 58 hdd 10.91399 osd.58 up 1.00000 1.00000 74 hdd 10.91399 osd.74 up 1.00000 1.00000 83 hdd 10.91399 osd.83 up 1.00000 1.00000 92 hdd 10.91399 osd.92 up 1.00000 1.00000 -46 87.31189 host node04-e986c3fc-a21b-44ff-9b02-b60b82ee63d7 12 hdd 10.91399 osd.12 up 1.00000 1.00000 23 hdd 10.91399 osd.23 up 1.00000 1.00000 32 hdd 10.91399 osd.32 up 1.00000 1.00000 43 hdd 10.91399 osd.43 up 1.00000 1.00000 52 hdd 10.91399 osd.52 up 1.00000 1.00000 71 hdd 10.91399 osd.71 up 1.00000 1.00000 95 hdd 10.91399 osd.95 up 1.00000 1.00000 104 hdd 10.91399 osd.104 up 1.00000 1.00000 -88 87.31189 host node05-fe31d85f-b3b9-4393-b24b-030dbcdfacea 3 hdd 10.91399 osd.3 up 1.00000 1.00000 24 hdd 10.91399 osd.24 up 1.00000 1.00000 33 hdd 10.91399 osd.33 up 1.00000 1.00000 45 hdd 10.91399 osd.45 up 1.00000 1.00000 69 hdd 10.91399 osd.69 up 1.00000 1.00000 79 hdd 10.91399 osd.79 up 1.00000 1.00000 89 hdd 10.91399 osd.89 up 1.00000 1.00000 99 hdd 10.91399 osd.99 up 1.00000 1.00000 -55 87.31189 host node06-6f16ba4b-0082-472a-b243-b1a058070918 5 hdd 10.91399 osd.5 up 1.00000 1.00000 15 hdd 10.91399 osd.15 up 1.00000 1.00000 25 hdd 10.91399 osd.25 up 1.00000 1.00000 44 hdd 10.91399 osd.44 up 1.00000 1.00000 63 hdd 10.91399 osd.63 up 1.00000 1.00000 72 hdd 10.91399 osd.72 up 1.00000 1.00000 81 hdd 10.91399 osd.81 up 1.00000 1.00000 91 hdd 10.91399 osd.91 up 1.00000 1.00000 -43 87.31189 host node07-5dee846a-2814-4e04-bcfd-ff689d49795c 7 hdd 10.91399 osd.7 up 1.00000 1.00000 37 hdd 10.91399 osd.37 up 1.00000 1.00000 46 hdd 10.91399 osd.46 up 1.00000 1.00000 55 hdd 10.91399 osd.55 up 1.00000 1.00000 64 hdd 10.91399 osd.64 up 1.00000 1.00000 84 hdd 10.91399 osd.84 up 1.00000 1.00000 94 hdd 10.91399 osd.94 up 1.00000 1.00000 102 hdd 10.91399 osd.102 up 1.00000 1.00000 -58 87.31189 host node08-2d6b7ab3-2067-4e94-b77c-24d6e626e396 6 hdd 10.91399 osd.6 up 1.00000 1.00000 18 hdd 10.91399 osd.18 up 1.00000 1.00000 41 hdd 10.91399 osd.41 up 1.00000 1.00000 49 hdd 10.91399 osd.49 up 1.00000 1.00000 59 hdd 10.91399 osd.59 up 1.00000 1.00000 68 hdd 10.91399 osd.68 up 1.00000 1.00000 97 hdd 10.91399 osd.97 up 1.00000 1.00000 105 hdd 10.91399 osd.105 up 1.00000 1.00000 -49 87.31189 host node09-e6bf0642-e3b4-48f0-9d0e-3d87ceacced8 8 hdd 10.91399 osd.8 up 1.00000 1.00000 17 hdd 10.91399 osd.17 up 1.00000 1.00000 27 hdd 10.91399 osd.27 up 1.00000 1.00000 38 hdd 10.91399 osd.38 up 1.00000 1.00000 57 hdd 10.91399 osd.57 up 1.00000 1.00000 67 hdd 10.91399 osd.67 up 1.00000 1.00000 93 hdd 10.91399 osd.93 up 1.00000 1.00000 103 hdd 10.91399 osd.103 up 1.00000 1.00000 -8 47.16115 root root-ea7a1878-722e-49d5-8a91-c618a6aefe29 -13 5.24013 host node01-4c465825-6bd1-42a5-b087-51a82cb2865c 0 ssd 1.74657 osd.0 up 0.95001 1.00000 11 ssd 1.74699 osd.11 up 1.00000 1.00000 16 ssd 1.74657 osd.16 up 1.00000 1.00000 -22 5.24013 host node02-0e3e418d-9129-44ee-8453-53680171270e 4 ssd 1.74657 osd.4 up 1.00000 1.00000 14 ssd 1.74657 osd.14 up 0.95001 1.00000 82 ssd 1.74699 osd.82 up 1.00000 1.00000 -28 5.24013 host node03-274397b4-8559-4a17-a9eb-1c16920ad432 10 ssd 1.74657 osd.10 up 1.00000 1.00000 76 ssd 1.74657 osd.76 up 1.00000 1.00000 100 ssd 1.74699 osd.100 up 0.95001 1.00000 -31 5.24013 host node04-df05c3cc-08ef-426e-85b8-088cb8c1b4e2 20 ssd 1.74657 osd.20 up 1.00000 1.00000 39 ssd 1.74657 osd.39 up 0.95001 1.00000 90 ssd 1.74699 osd.90 up 0.95001 1.00000 -25 5.24013 host node05-d07e7e92-8290-49ba-b09c-ad34bddb1eae 26 ssd 1.74657 osd.26 up 0.95001 1.00000 62 ssd 1.74699 osd.62 up 1.00000 1.00000 73 ssd 1.74657 osd.73 up 1.00000 1.00000 -16 5.24013 host node06-a7a40371-a128-4ab1-90a2-62c99e040036 42 ssd 1.74657 osd.42 up 0.95001 1.00000 75 ssd 1.74657 osd.75 up 0.95001 1.00000 107 ssd 1.74699 osd.107 up 1.00000 1.00000 -7 5.24013 host node07-971dee6e-dec4-4c0a-86d1-54d0b23832bd 21 ssd 1.74699 osd.21 up 1.00000 1.00000 66 ssd 1.74657 osd.66 up 0.95001 1.00000 77 ssd 1.74657 osd.77 up 1.00000 1.00000 -19 5.24013 host node08-9a47282b-9530-4cc7-9e29-7f6c0b2f5184 35 ssd 1.74699 osd.35 up 1.00000 1.00000 80 ssd 1.74657 osd.80 up 1.00000 1.00000 101 ssd 1.74657 osd.101 up 1.00000 1.00000 -34 5.24013 host node09-f09a8013-2426-4ced-b9c7-02c06ca9d6fc 36 ssd 1.74699 osd.36 up 0.95001 1.00000 87 ssd 1.74657 osd.87 up 1.00000 1.00000 106 ssd 1.74657 osd.106 up 1.00000 1.00000 After ceph has been running normally for a period of time, I found that the distribution of osd data is seriously uneven, and autoscale PGs were turned on in my envs. My hdds are identical, but utilization ranges from 6.14% to 74.35% root@node09:/# ceph osd df | grep hdd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1 hdd 10.91399 1.00000 11 TiB 5.3 TiB 5.2 TiB 15 KiB 53 GiB 5.7 TiB 48.15 1.38 174 up 22 hdd 10.91399 0.90002 11 TiB 8.1 TiB 8.0 TiB 12 KiB 71 GiB 2.8 TiB 74.35 2.13 160 up 31 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 13 KiB 27 GiB 8.0 TiB 27.14 0.78 179 up 51 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 9 KiB 36 GiB 7.2 TiB 34.12 0.98 173 up 60 hdd 10.91399 1.00000 11 TiB 973 GiB 962 GiB 14 KiB 11 GiB 10 TiB 8.71 0.25 167 up 70 hdd 10.91399 1.00000 11 TiB 2.0 TiB 2.0 TiB 11 KiB 3.9 GiB 8.9 TiB 18.42 0.53 173 up 78 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 12 KiB 28 GiB 7.9 TiB 27.18 0.78 173 up 96 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 10 KiB 36 GiB 7.2 TiB 34.15 0.98 176 up 9 hdd 10.91399 1.00000 11 TiB 2.8 TiB 2.8 TiB 16 KiB 13 GiB 8.1 TiB 25.38 0.73 178 up 19 hdd 10.91399 1.00000 11 TiB 3.8 TiB 3.8 TiB 13 KiB 43 GiB 7.1 TiB 35.04 1.00 177 up 29 hdd 10.91399 1.00000 11 TiB 5.1 TiB 5.0 TiB 14 KiB 38 GiB 5.9 TiB 46.38 1.33 182 up 47 hdd 10.91399 1.00000 11 TiB 686 GiB 684 GiB 9 KiB 1.4 GiB 10 TiB 6.14 0.18 172 up 56 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 12 KiB 36 GiB 7.2 TiB 34.16 0.98 174 up 65 hdd 10.91399 1.00000 11 TiB 5.9 TiB 5.9 TiB 7 KiB 54 GiB 5.0 TiB 54.27 1.55 183 up 88 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 12 KiB 44 GiB 6.4 TiB 41.17 1.18 173 up 98 hdd 10.91399 1.00000 11 TiB 7.3 TiB 7.2 TiB 13 KiB 56 GiB 3.7 TiB 66.49 1.90 179 up 2 hdd 10.91399 1.00000 11 TiB 1.5 TiB 1.5 TiB 12 KiB 18 GiB 9.4 TiB 14.02 0.40 177 up 30 hdd 10.91399 1.00000 11 TiB 4.4 TiB 4.4 TiB 10 KiB 37 GiB 6.5 TiB 40.29 1.15 180 up 40 hdd 10.91399 1.00000 11 TiB 1.4 TiB 1.4 TiB 10 KiB 11 GiB 9.5 TiB 13.14 0.38 172 up 48 hdd 10.91399 1.00000 11 TiB 1.4 TiB 1.4 TiB 12 KiB 11 GiB 9.5 TiB 13.17 0.38 168 up 58 hdd 10.91399 1.00000 11 TiB 7.5 TiB 7.4 TiB 14 KiB 70 GiB 3.5 TiB 68.31 1.96 182 up 74 hdd 10.91399 1.00000 11 TiB 6.0 TiB 6.0 TiB 10 KiB 61 GiB 4.9 TiB 55.15 1.58 181 up 83 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 22 KiB 19 GiB 8.7 TiB 20.13 0.58 174 up 92 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 11 KiB 27 GiB 8.0 TiB 27.15 0.78 171 up 12 hdd 10.91399 1.00000 11 TiB 5.1 TiB 5.0 TiB 9 KiB 38 GiB 5.9 TiB 46.37 1.33 176 up 23 hdd 10.91399 1.00000 11 TiB 4.4 TiB 4.4 TiB 22 KiB 37 GiB 6.5 TiB 40.24 1.15 173 up 32 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 10 KiB 19 GiB 8.7 TiB 20.13 0.58 176 up 43 hdd 10.91399 1.00000 11 TiB 2.0 TiB 2.0 TiB 10 KiB 4.4 GiB 8.9 TiB 18.41 0.53 176 up 52 hdd 10.91399 1.00000 11 TiB 3.6 TiB 3.6 TiB 12 KiB 29 GiB 7.3 TiB 33.27 0.95 171 up 71 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 15 KiB 27 GiB 8.0 TiB 27.13 0.78 170 up 95 hdd 10.91399 1.00000 11 TiB 5.4 TiB 5.3 TiB 10 KiB 60 GiB 5.6 TiB 49.04 1.40 173 up104 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 8 KiB 36 GiB 7.2 TiB 34.13 0.98 171 up 3 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 9 KiB 19 GiB 8.7 TiB 20.14 0.58 173 up 24 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 9 KiB 19 GiB 8.7 TiB 20.12 0.58 170 up 33 hdd 10.91399 1.00000 11 TiB 6.8 TiB 6.7 TiB 14 KiB 69 GiB 4.1 TiB 62.15 1.78 181 up 45 hdd 10.91399 1.00000 11 TiB 2.9 TiB 2.8 TiB 10 KiB 20 GiB 8.0 TiB 26.27 0.75 178 up 69 hdd 10.91399 1.00000 11 TiB 2.9 TiB 2.8 TiB 15 KiB 20 GiB 8.0 TiB 26.25 0.75 173 up 79 hdd 10.91399 1.00000 11 TiB 5.9 TiB 5.9 TiB 28 KiB 54 GiB 5.0 TiB 54.25 1.55 178 up 89 hdd 10.91399 1.00000 11 TiB 2.1 TiB 2.1 TiB 16 KiB 11 GiB 8.8 TiB 19.25 0.55 173 up 99 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 13 KiB 44 GiB 6.4 TiB 41.14 1.18 174 up 5 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 14 KiB 19 GiB 8.7 TiB 20.12 0.58 172 up 15 hdd 10.91399 1.00000 11 TiB 7.3 TiB 7.2 TiB 7 KiB 56 GiB 3.7 TiB 66.48 1.90 177 up 25 hdd 10.91399 1.00000 11 TiB 1.4 TiB 1.4 TiB 11 KiB 10 GiB 9.5 TiB 13.13 0.38 176 up 44 hdd 10.91399 1.00000 11 TiB 3.1 TiB 3.0 TiB 9 KiB 34 GiB 7.9 TiB 28.03 0.80 179 up 63 hdd 10.91399 1.00000 11 TiB 2.9 TiB 2.8 TiB 10 KiB 20 GiB 8.0 TiB 26.26 0.75 178 up 72 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 12 KiB 36 GiB 7.2 TiB 34.13 0.98 180 up 81 hdd 10.91399 1.00000 11 TiB 6.7 TiB 6.6 TiB 11 KiB 62 GiB 4.2 TiB 61.30 1.76 181 up 91 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 16 KiB 19 GiB 8.7 TiB 20.11 0.58 176 up 7 hdd 10.91399 1.00000 11 TiB 4.6 TiB 4.5 TiB 13 KiB 52 GiB 6.3 TiB 42.03 1.20 180 up 37 hdd 10.91399 1.00000 11 TiB 5.3 TiB 5.2 TiB 14 KiB 52 GiB 5.7 TiB 48.16 1.38 182 up 46 hdd 10.91399 1.00000 11 TiB 784 GiB 775 GiB 14 KiB 8.8 GiB 10 TiB 7.01 0.20 173 up 55 hdd 10.91399 1.00000 11 TiB 5.9 TiB 5.9 TiB 20 KiB 54 GiB 5.0 TiB 54.25 1.55 177 up 64 hdd 10.91399 1.00000 11 TiB 3.6 TiB 3.6 TiB 18 KiB 28 GiB 7.3 TiB 33.23 0.95 176 up 84 hdd 10.91399 1.00000 11 TiB 784 GiB 775 GiB 10 KiB 9.0 GiB 10 TiB 7.01 0.20 176 up 94 hdd 10.91399 1.00000 11 TiB 2.1 TiB 2.1 TiB 8 KiB 12 GiB 8.8 TiB 19.28 0.55 169 up102 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 10 KiB 28 GiB 8.0 TiB 27.13 0.78 170 up 6 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 23 KiB 44 GiB 6.4 TiB 41.14 1.18 181 up 18 hdd 10.91399 1.00000 11 TiB 6.0 TiB 6.0 TiB 9 KiB 61 GiB 4.9 TiB 55.15 1.58 183 up 41 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 15 KiB 27 GiB 8.0 TiB 27.12 0.78 179 up 49 hdd 10.91399 1.00000 11 TiB 785 GiB 775 GiB 19 KiB 9.5 GiB 10 TiB 7.02 0.20 176 up 59 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 8 KiB 19 GiB 8.7 TiB 20.12 0.58 178 up 68 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 13 KiB 36 GiB 7.2 TiB 34.15 0.98 172 up 97 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 15 KiB 27 GiB 8.0 TiB 27.13 0.78 178 up105 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 13 KiB 36 GiB 7.2 TiB 34.14 0.98 179 up 8 hdd 10.91399 1.00000 11 TiB 6.6 TiB 6.5 TiB 13 KiB 55 GiB 4.3 TiB 60.40 1.73 181 up 17 hdd 10.91399 1.00000 11 TiB 4.6 TiB 4.5 TiB 14 KiB 52 GiB 6.3 TiB 42.03 1.20 177 up 27 hdd 10.91399 1.00000 11 TiB 783 GiB 774 GiB 8 KiB 9.4 GiB 10 TiB 7.01 0.20 176 up 38 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 15 KiB 19 GiB 8.7 TiB 20.11 0.58 174 up 57 hdd 10.91399 1.00000 11 TiB 2.3 TiB 2.3 TiB 12 KiB 26 GiB 8.6 TiB 21.01 0.60 177 up 67 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 9 KiB 44 GiB 6.4 TiB 41.14 1.18 176 up 93 hdd 10.91399 1.00000 11 TiB 3.1 TiB 3.0 TiB 7 KiB 35 GiB 7.9 TiB 28.01 0.80 171 up103 hdd 10.91399 1.00000 11 TiB 4.2 TiB 4.2 TiB 10 KiB 22 GiB 6.7 TiB 38.52 1.10 176 up three pools (deeproute-replica-hdd-pool, os-dsglczutvqsgowpz.rgw.buckets.data, and cephfs-replicated-pool ) are set to use the same device class hdd. I noticed that the *effective ratio* of the three pools is very different, not sure if it will affect the data rebalancing. [root@node01 smd]# ceph osd pool autoscale-statusPOOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK device_health_metrics 278.5M 3.0 48289G 0.0000 1.0 1 on False deeproute-replica-hdd-pool 12231M 3.0 785.8T 0.9901 50.0000 0.9901 1.0 4096 on False deeproute-replica-ssd-pool 12333G 3.0 48289G 0.9901 50.0000 0.9901 1.0 1024 on False .rgw.root 5831 3.0 48289G 0.0099 0.5000 0.0099 1.0 8 on False default.rgw.log 182 3.0 48289G 0.0000 1.0 32 on False default.rgw.control 0 3.0 48289G 0.0000 1.0 32 on False default.rgw.meta 0 3.0 48289G 0.0000 4.0 8 on False os-dsglczutvqsgowpz.rgw.control 0 3.0 16096G 0.1667 0.5000 0.1667 1.0 64 on False os-dsglczutvqsgowpz.rgw.meta 99.81k 3.0 16096G 0.1667 0.5000 0.1667 1.0 64 on False os-dsglczutvqsgowpz.rgw.buckets.index 59437M 3.0 16096G 0.1667 0.5000 0.1667 1.0 32 on False os-dsglczutvqsgowpz.rgw.buckets.non-ec 498.7M 3.0 16096G 0.1667 0.5000 0.1667 1.0 32 on False os-dsglczutvqsgowpz.rgw.log 1331M 3.0 16096G 0.1667 0.5000 0.1667 1.0 32 on False os-dsglczutvqsgowpz.rgw.buckets.data 143.1T 1.3333333730697632 785.8T 0.2429 0.5000 0.0099 1.0 32 on False cephfs-metadata 3161M 3.0 16096G 0.0006 4.0 32 on False cephfs-replicated-pool 21862G 3.0 785.8T 0.0815 1.0 32 on False .nfs 84599 3.0 48289G 0.0000 1.0 32 on False os-dsglczutvqsgowpz.rgw.otp 0 3.0 16096G 0.1667 0.5000 [root@node01 smd]# ceph df--- RAW STORAGE ---CLASS SIZE AVAIL USED RAW USED %RAW USEDhdd 786 TiB 545 TiB 240 TiB 240 TiB 30.60ssd 63 TiB 25 TiB 38 TiB 38 TiB 59.95TOTAL 849 TiB 571 TiB 278 TiB 278 TiB 32.77 --- POOLS ---POOL ID PGS STORED OBJECTS USED %USED MAX AVAILdevice_health_metrics 1 1 280 MiB 135 841 MiB 0.02 1.8 TiBdeeproute-replica-hdd-pool 11 4096 12 GiB 9.44k 36 GiB 0.02 69 TiBdeeproute-replica-ssd-pool 12 1024 12 TiB 3.43M 37 TiB 87.62 1.8 TiB.rgw.root 25 8 5.7 KiB 20 228 KiB 0 1.8 TiBdefault.rgw.log 26 32 182 B 2 24 KiB 0 1.8 TiBdefault.rgw.control 27 32 0 B 8 0 B 0 1.8 TiBdefault.rgw.meta 28 8 0 B 0 0 B 0 1.8 TiBos-dsglczutvqsgowpz.rgw.control 29 64 0 B 8 0 B 0 4.9 TiBos-dsglczutvqsgowpz.rgw.meta 30 64 100 KiB 456 4.3 MiB 0 4.9 TiBos-dsglczutvqsgowpz.rgw.buckets.index 31 32 58 GiB 10.40k 174 GiB 1.15 4.9 TiBos-dsglczutvqsgowpz.rgw.buckets.non-ec 32 32 500 MiB 247.78k 4.3 GiB 0.03 4.9 TiBos-dsglczutvqsgowpz.rgw.log 33 32 689 MiB 432 2.0 GiB 0.01 4.9 TiBos-dsglczutvqsgowpz.rgw.buckets.data 34 32 130 TiB 134.96M 174 TiB 45.69 155 TiBcephfs-metadata 35 32 3.1 GiB 1.67M 9.3 GiB 0.06 4.9 TiBcephfs-replicated-pool 36 32 21 TiB 8.25M 64 TiB 23.79 69 TiB.nfs 37 32 89 KiB 9 341 KiB 0 1.8 TiBos-dsglczutvqsgowpz.rgw.otp 38 8 0 B 0 0 B 0 4.9 TiB [root@node01 smd]# ceph -s cluster: id: f7f1c8ba-f793-436b-bb73-0964108a30c1 health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 7w) mgr: a(active, since 2w), standbys: b mds: 1/1 daemons up, 1 hot standby osd: 108 osds: 108 up (since 5h), 108 in (since 4M); 1 remapped pgs rgw: 9 daemons active (9 hosts, 1 zones) data: volumes: 1/1 healthy pools: 17 pools, 5561 pgs objects: 148.37M objects, 162 TiB usage: 276 TiB used, 572 TiB / 849 TiB avail pgs: 2786556/1119434807 objects misplaced (0.249%) 5559 active+clean 1 active+remapped+backfilling 1 active+clean+scrubbing+deep io: client: 212 MiB/s rd, 27 MiB/s wr, 4.04k op/s rd, 2.01k op/s wr recovery: 4.0 MiB/s, 4 objects/s progress: Global Recovery Event (2d) [===========================.] (remaining: 39m) So my question is, how can I adjust to make the osd data more evenly distributed? Thanks! Best Regards, Liang Zheng

1 1

Generated signurl is accessible from restricted IPs in bucket policy
by Aggelos Toumasis 24 Mar '23

24 Mar '23

Hi there, We noticed after creating a signurl that the bucket resources were accessible from IPs that were originally restricted from accessing them (using a bucket policy). Using the s3cmd utility we confirmed that the Policy is correctly applied and you can access it only for the allowed IPs. Is this an expected behavior or do we miss something? Thanks!

3 2

With Ceph Quincy, the "ceph" package does not include ceph-volume anymore
by Geert Kloosterman 24 Mar '23

24 Mar '23

Hi all, Until Ceph Pacific, installing just the "ceph" package was enough to get everything needed to deploy Ceph. However, with Quincy, ceph-volume was split off into its own package, and it is not automatically installed anymore. Here we can see it is not listed as a dependency: $ rpm -q --requires ceph binutils ceph-mds = 2:17.2.5-0.el8 ceph-mgr = 2:17.2.5-0.el8 ceph-mon = 2:17.2.5-0.el8 ceph-osd = 2:17.2.5-0.el8 rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(PayloadIsXz) <= 5.2-1 systemd Should I file a bug for this? Best regards, Geert Kloosterman

4 4

RGW can't create bucket
by Kamil Madac 24 Mar '23

24 Mar '23

Hi, One of my customers had a correctly working RGW cluster with two zones in one zonegroup and since a few days ago users are not able to create buckets and are always getting Access denied. Working with existing buckets works (like listing/putting objects into existing bucket). The only operation which is not working is bucket creation. We also tried to create a new user, but the behavior is the same, and he is not able to create the bucket. We tried s3cmd, python script with boto library and also Dashboard as admin user. We are always getting Access Denied. Zones are in-sync. Has anyone experienced such behavior? Thanks in advance, here are some outputs: $ s3cmd -c .s3cfg_python_client mb s3://test ERROR: Access to bucket 'test' was denied ERROR: S3 error: 403 (AccessDenied) Zones are in-sync: Primary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) metadata sync no sync (zone is master) data sync source: e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source Secondary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source -- Kamil Madac <https://kmadac.github.io/>

1 0