Hi to all!
We are in a situation where we have 3 PG in
"active+remapped+backfill_toofull". It happened when we executed a
"gentle-reweight" to zero of one OSD (osd.77) to swap it with a new one
(the current one registered some read errors and it's to be replaced
just-in-case).
> # ceph health detail:
> [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull
> pg 10.46c is active+remapped+backfill_toofull, acting [77,607,96]
> pg 10.8ad is active+remapped+backfill_toofull, acting [577,152,77]
> pg 10.b15 is active+remapped+backfill_toofull, acting [483,348,77]
Our cluster is a little unbalanced and we have 7 OSD nearfull (I think
it's because we have 4 nodes with 6 TB disks and the other 19 have 10
TB, but should be unrelated, why is the cluster unbalanced I mean, to
the backfill_toofull ) not by too much (less then 88%). I'm not too much
worried about it, we will add new storage this month (if the servers
will arrive) and we will get rid of the old 6 TB servers.
If I dump the PGs I see, if I'm not mistaken, that the osd.77 will be
"replaced" by the osd.60, which is one of the nearfull ones (the top one
with 87.53% used).
> # ceph pg dump:
>
> 10.b15 37236 0 0 37236 0 155249620992 0 0 5265 5265 active+remapped+backfill_toofull 2023-01-16T14:45:46.155801+0100 305742'144106 305742:901513 [483,348,60] 483 [483,348,77] 483 305211'144056 2023-01-11T10:20:56.600135+0100 305211'144056 2023-01-11T10:20:56.600135+0100 0
> 10.8ad 37518 0 0 37518 0 156345024512 0 0 5517 5517 active+remapped+backfill_toofull 2023-01-16T14:45:38.510038+0100 305213'142117 305742:937228 [577,60,152] 577 [577,152,77] 577 303828'142043 2023-01-06T17:52:02.523104+0100 303334'141645 2022-12-20T17:39:22.668083+0100 0
> 10.46c 36710 0 0 36710 0 153023443456 0 0 8172 8172 active+remapped+backfill_toofull 2023-01-16T14:45:29.284223+0100 305298'141437 305741:877331 [60,607,96] 60 [77,607,96] 77 304802'141358 2023-01-08T21:39:23.469198+0100 304363'141349 2023-01-01T18:13:45.645494+0100 0
> # ceph osd df:
> 60 hdd 5.45999 1.00000 5.5 TiB 4.8 TiB 697 GiB 128 MiB 0 B 697 GiB 87.53 1.29 37 up
In this situation was the correct way to address the problem?
reweight-by-utilization the osd.60 to free up space (the OSD is a 6 TB
disk, and other OSD on the same host are nearfull)? There is other way
to manually map a PG to a different OSD?
Thank you for your attention
Iztok Gregori
I am really hoping you can help. THANKS in advance. I have inherited a Docker swarm running CEPH but I know very little about it.
Current I have an unhealthy ceph environment that will not mount my data drive.
Its a cluster of 4 vm servers. docker01,docker02, docker03, docker-cloud
CL has the /data that is on a separate drive, currently failing to mount.
how can I recover this without loosing the data?
on server docker-cloud, mount /data returns:
mount error 113 = No route to host
docker ps is healthy on all nodes.
bc81d14dde92 ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 34 minutes ceph-mds
d4fecec5e0e8 ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 34 minutes ceph-osd
482ba41803af ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 34 minutes ceph-mgr
d6a5c44179c7 ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 32 minutes ceph-mon
ceph -s:
cluster:
id: 7a5b2243-8e92-4e03-aee7-aa64cea666ec
health: HEALTH_ERR
1 filesystem is degraded
1 filesystem is offline
1 mds daemon damaged
noout,noscrub,nodeep-scrub flag(s) set
clock skew detected on mon.docker02, mon.docker03, mon.docker-cloud
mons docker-cloud,docker01,docker02,docker03 are low on available space
services:
mon: 4 daemons, quorum docker01,docker02,docker03,docker-cloud
mgr: docker01(active), standbys: docker02, docker03, docker-cloud
mds: cephfs-0/1/1 up , 4 up:standby, 1 damaged
osd: 4 osds: 4 up, 4 in
flags noout,noscrub,nodeep-scrub
data:
pools: 2 pools, 256 pgs
objects: 194.2 k objects, 241 GiB
usage: 499 GiB used, 1.5 TiB / 2.0 TiB avail
pgs: 256 active+clean
Dear Ceph users,
after a host failure in my cluster (quincy 17.2.3 managed by cephadm) it
seems that ceph orch got somehow stuck and it cannot operate. For
example, it seems that it cannot refresh the status of several services
since about 20 hours:
# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE
PLACEMENT
alertmanager ?:9093,9094 1/1 3m ago 3M
count:1
crash 9/10 20h ago 3M *
grafana ?:3000 1/1 3m ago 3M
count:1
mds.wizard_fs 0/3 <deleting> 13h
bofur;balin;aka;count:3
mds.wizardfs 2/3 20h ago 70m
bofur;balin;aka;count:3
mgr 2/2 20h ago 15m
bofur;balin;count:2
mon 4/5 20h ago 93m
bofur;balin;aka;romolo;dwalin;count:5
node-exporter ?:9100 9/10 20h ago 3M *
osd 24 3m ago -
<unmanaged>
osd.all-available-devices 72 20h ago 4w *
prometheus ?:9095 1/1 3m ago 3M count:1
The failed machine (named bifur) is offline but still in the cluster
since I'm planning to restore it:
# ceph orch host ls
HOST ADDR LABELS STATUS
aka 172.16.253.7 _admin
balin 172.16.253.3
bifur 172.16.253.5 _admin Offline
bofur 172.16.253.2 _admin
dwalin 172.16.253.10
ogion 172.16.253.6 _no_autotune_memory
prestno 172.16.253.9
remolo 172.16.253.1
rokanan 172.16.253.8
romolo 172.16.253.4
10 hosts in cluster
Since this machine hosted a mon I tried to redeploy it with:
# ceph orch apply mon --placement="5 bofur balin aka romolo dwalin"
but even if ceph orch ls shows that the mons should currently be on the
machines specified buy --placement (see above) it seems that somehow the
mon on bifur is somehow still present in ceph orch status, e.g.
# ceph orch restart mon
Scheduled to restart mon.aka on host 'aka'
Scheduled to restart mon.balin on host 'balin'
Scheduled to restart mon.bifur on host 'bifur'
Scheduled to restart mon.bofur on host 'bofur'
Scheduled to restart mon.romolo on host 'romolo'
I manually restarted all the mon and mgr daemons on online hosts to no
avail. At this point I am clueless, so any help is greatly appreciated.
Nicola
Dear cephers,
I have a strange problem. An OSD went down and recovery finished. For some reason, I have a slow ops warning for the failed OSD stuck in the system:
health: HEALTH_WARN
430 slow ops, oldest one blocked for 36 sec, osd.580 has slow ops
The OSD is auto-out:
| 580 | ceph-22 | 0 | 0 | 0 | 0 | 0 | 0 | autoout,exists |
It is probably a warning dating back to just before the fail. How can I clear it?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
I am experiencing the same problem with 'Large omap object' in Ceph, so I
wrote a script to find the large objects in the pool, count the number of
"omap" in each object, and compare it with the value set for
large_omap_object_key_threshold in the Ceph configuration.
https://gist.github.com/RaminNietzsche/0297e9163834c050234686f5b4acb1a4
On Sun, Oct 30, 2022 at 5:12 AM Sarah Coxon <sazzle2611(a)gmail.com> wrote:
> Hi Anthony,
>
> Thank you for getting back to me, the post you sent was helpful although
> the log in that post was usage log and there aren't the same options for
> the sync error log.
>
> I understand that I can get rid of the warning by increasing the
> threshold but I would really like to get rid of the old data if possible.
>
> The sync error log is full of errors from buckets that have since been
> deleted and the sync errors are over 2 years old. Trimming the sync error
> log does nothing.
>
> From everything I've read this is an issue with having a multisite
> configuration and removing buckets doesn't clear previous data properly
> https://www.spinics.net/lists/ceph-devel/msg45359.html
>
> I already knew this as in I have been manually deleting metadata and index
> data for buckets a while after clearing them of objects and deleting the
> bucket itself (but making note of the bucket id to use in the later
> commands)
>
> I have found this command to get object ids in the sync error log related
> to the deleted buckets
>
> radosgw-admin sync error list | jq '.[0].entries[] |
> select(.name|test("^company-report-images.")) | .id'
>
> but I don't know how to get rid of them and I really don't want to screw up
> the whole setup by deleting the wrong thing.
>
> On Thu, Oct 27, 2022 at 9:52 AM Anthony D'Atri <anthony.datri(a)gmail.com>
> wrote:
>
> > This prior post
> >
> >
> >
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2QNKWK642LW…
> >
> > may help. You can bump up the warning threshold to make the warning go
> > away - a few releases ago it was reduced to 1/10 of the prior value.
> >
> > There’s also information about trimming usage logs and for removing
> > specific usage log objects.
> >
> > > On Oct 27, 2022, at 4:05 AM, Sarah Coxon <sazzle2611(a)gmail.com> wrote:
> > >
> > > Hey, I would really appreciate any help I can get on this as googling
> has
> > > led me to a dead end.
> > >
> > > We have 2 data centers each with 4 servers running ceph on kubernetes
> in
> > > multisite config, everything is working great but recently the master
> > > cluster changed status to HEALTH_WARN and the issues are large omap
> > objects
> > > in the .rgw.log pool. Second cluster is still HEALTH_OK
> > >
> > > Viewing the sync error log from master shows a lot of very ancient logs
> > > related to a bucket that has since been deleted.
> > >
> > > Is there any way to clear this log?
> > >
> > > bash-4.4$ radosgw-admin sync error list | wc -l
> > > 352162
> > >
> > > I believe, although I'm not sure that this is a massive part of the
> data
> > > stored in the .rgw.log pool, I haven't been able to find any info on
> this
> > > except for several other posts about clearing the error log but none of
> > > them had a resolution.
> > >
> > > I am tempted to increase the PG's for this pool from 16 to 32 to see if
> > it
> > > helps but holding off because that is not an ideal solution just to get
> > rid
> > > of this warning when all I want is to get rid of the errors related to
> a
> > > bucket that no longer exists.
> > >
> > > Thanks to anyone that can offer advice!
> > >
> > > Sarah
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
Hi,
For a given crush rule and pool that uses it, how can I verify hat the pgs in that pool folllow the rule? I have a requirement to 'prove' that the pgs are mapping correctly.
I see: https://pypi.org/project/crush/
This allows me to read in a crushmap file that I could then use to verify a pg with some scripting, but this pypi is very old and seems not to be maintained or updatedsince 2017.
I am sure there is a way, using osdmaptool or something else, but it is not obvious. Before i spend alot of time searching, I thought I would ask here.
Basically, having a list of pgs like this:
[[1,2,3,4,5],[2,3,4,5,6],...]
Given a read-in crushmap and a specific rule therein, I want to verify that all pgs in my list are consistent with the rule specified.
Let me know if there is a proper way to do this, and thanks.
-Chris
Hi all,
for a ceph cluster with RAM size of 256 GB per node, I would increase
the osd_memory_target from default 4GB up to 12GB. Through the ceph
dashboard, different values are given to set the new value (global, mon,
..., osd). Is there any difference between them? From my point of view,
I should simply set the osd value to 12 GB, am I right?
Best,
Mevludin
Hi,
Is it possible/supported to build Ceph containers on Debian? The build
instructions[0] talk about building packages (incl. .debs), but now
building containers.
Cephadm only supports containerised deployments, but our local policy is
that we should only deploy containers we've built ourselves. Is it still
the case that only building centos-based images is supported?
Perhaps with a follow-up question of why container building isn't
supported on the range of platforms that package building is supported
on if containerised deployments are where people are meant to be going...
Thanks,
Matthew
[0] https://docs.ceph.com/en/quincy/install/build-ceph/
Ceph 17.2.5:
Hi,
I'm looking for a reasonable and useful MDS configuration for a – in
future, no experiences until now – heavily used CephFS (~100TB).
For example, does it make a difference to increase the
mds_cache_memory_limit or the number of MDS instances?
The hardware does not set any limits, I just want to know where the default
values can be optimized usefully before problem occur.
Thanks,
