- ceph-users - lists.ceph.io

ceph orch cannot refresh
by Nicola Mori 17 Jan '23

17 Jan '23

Dear Ceph users, after a host failure in my cluster (quincy 17.2.3 managed by cephadm) it seems that ceph orch got somehow stuck and it cannot operate. For example, it seems that it cannot refresh the status of several services since about 20 hours: # ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 3m ago 3M count:1 crash 9/10 20h ago 3M * grafana ?:3000 1/1 3m ago 3M count:1 mds.wizard_fs 0/3 <deleting> 13h bofur;balin;aka;count:3 mds.wizardfs 2/3 20h ago 70m bofur;balin;aka;count:3 mgr 2/2 20h ago 15m bofur;balin;count:2 mon 4/5 20h ago 93m bofur;balin;aka;romolo;dwalin;count:5 node-exporter ?:9100 9/10 20h ago 3M * osd 24 3m ago - <unmanaged> osd.all-available-devices 72 20h ago 4w * prometheus ?:9095 1/1 3m ago 3M count:1 The failed machine (named bifur) is offline but still in the cluster since I'm planning to restore it: # ceph orch host ls HOST ADDR LABELS STATUS aka 172.16.253.7 _admin balin 172.16.253.3 bifur 172.16.253.5 _admin Offline bofur 172.16.253.2 _admin dwalin 172.16.253.10 ogion 172.16.253.6 _no_autotune_memory prestno 172.16.253.9 remolo 172.16.253.1 rokanan 172.16.253.8 romolo 172.16.253.4 10 hosts in cluster Since this machine hosted a mon I tried to redeploy it with: # ceph orch apply mon --placement="5 bofur balin aka romolo dwalin" but even if ceph orch ls shows that the mons should currently be on the machines specified buy --placement (see above) it seems that somehow the mon on bifur is somehow still present in ceph orch status, e.g. # ceph orch restart mon Scheduled to restart mon.aka on host 'aka' Scheduled to restart mon.balin on host 'balin' Scheduled to restart mon.bifur on host 'bifur' Scheduled to restart mon.bofur on host 'bofur' Scheduled to restart mon.romolo on host 'romolo' I manually restarted all the mon and mgr daemons on online hosts to no avail. At this point I am clueless, so any help is greatly appreciated. Nicola

2 1

OSD slow ops warning not clearing after OSD down
by Frank Schilder 17 Jan '23

17 Jan '23

Dear cephers, I have a strange problem. An OSD went down and recovery finished. For some reason, I have a slow ops warning for the failed OSD stuck in the system: health: HEALTH_WARN 430 slow ops, oldest one blocked for 36 sec, osd.580 has slow ops The OSD is auto-out: | 580 | ceph-22 | 0 | 0 | 0 | 0 | 0 | 0 | autoout,exists | It is probably a warning dating back to just before the fail. How can I clear it? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 7

Unable to subscribe
by Abhinav Singh 16 Jan '23

16 Jan '23

I m unable to subscribe even after sending mail to ceph-leave-user email id. Is there any other way to unsbuscribe? Thanks & Regards Abhinav Singh

1 0

Re: large omap objects in the .rgw.log pool
by Ramin Najjarbashi 16 Jan '23

16 Jan '23

I am experiencing the same problem with 'Large omap object' in Ceph, so I wrote a script to find the large objects in the pool, count the number of "omap" in each object, and compare it with the value set for large_omap_object_key_threshold in the Ceph configuration. https://gist.github.com/RaminNietzsche/0297e9163834c050234686f5b4acb1a4 On Sun, Oct 30, 2022 at 5:12 AM Sarah Coxon <sazzle2611(a)gmail.com> wrote: > Hi Anthony, > > Thank you for getting back to me, the post you sent was helpful although > the log in that post was usage log and there aren't the same options for > the sync error log. > > I understand that I can get rid of the warning by increasing the > threshold but I would really like to get rid of the old data if possible. > > The sync error log is full of errors from buckets that have since been > deleted and the sync errors are over 2 years old. Trimming the sync error > log does nothing. > > From everything I've read this is an issue with having a multisite > configuration and removing buckets doesn't clear previous data properly > https://www.spinics.net/lists/ceph-devel/msg45359.html > > I already knew this as in I have been manually deleting metadata and index > data for buckets a while after clearing them of objects and deleting the > bucket itself (but making note of the bucket id to use in the later > commands) > > I have found this command to get object ids in the sync error log related > to the deleted buckets > > radosgw-admin sync error list | jq '.[0].entries[] | > select(.name|test("^company-report-images.")) | .id' > > but I don't know how to get rid of them and I really don't want to screw up > the whole setup by deleting the wrong thing. > > On Thu, Oct 27, 2022 at 9:52 AM Anthony D'Atri <anthony.datri(a)gmail.com> > wrote: > > > This prior post > > > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2QNKWK642LW… > > > > may help. You can bump up the warning threshold to make the warning go > > away - a few releases ago it was reduced to 1/10 of the prior value. > > > > There’s also information about trimming usage logs and for removing > > specific usage log objects. > > > > > On Oct 27, 2022, at 4:05 AM, Sarah Coxon <sazzle2611(a)gmail.com> wrote: > > > > > > Hey, I would really appreciate any help I can get on this as googling > has > > > led me to a dead end. > > > > > > We have 2 data centers each with 4 servers running ceph on kubernetes > in > > > multisite config, everything is working great but recently the master > > > cluster changed status to HEALTH_WARN and the issues are large omap > > objects > > > in the .rgw.log pool. Second cluster is still HEALTH_OK > > > > > > Viewing the sync error log from master shows a lot of very ancient logs > > > related to a bucket that has since been deleted. > > > > > > Is there any way to clear this log? > > > > > > bash-4.4$ radosgw-admin sync error list | wc -l > > > 352162 > > > > > > I believe, although I'm not sure that this is a massive part of the > data > > > stored in the .rgw.log pool, I haven't been able to find any info on > this > > > except for several other posts about clearing the error log but none of > > > them had a resolution. > > > > > > I am tempted to increase the PG's for this pool from 16 to 32 to see if > > it > > > helps but holding off because that is not an ideal solution just to get > > rid > > > of this warning when all I want is to get rid of the errors related to > a > > > bucket that no longer exists. > > > > > > Thanks to anyone that can offer advice! > > > > > > Sarah > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

pg mapping verification
by Christopher Durham 16 Jan '23

16 Jan '23

Hi, For a given crush rule and pool that uses it, how can I verify hat the pgs in that pool folllow the rule? I have a requirement to 'prove' that the pgs are mapping correctly. I see: https://pypi.org/project/crush/ This allows me to read in a crushmap file that I could then use to verify a pg with some scripting, but this pypi is very old and seems not to be maintained or updatedsince 2017. I am sure there is a way, using osdmaptool or something else, but it is not obvious. Before i spend alot of time searching, I thought I would ask here. Basically, having a list of pgs like this: [[1,2,3,4,5],[2,3,4,5,6],...] Given a read-in crushmap and a specific rule therein, I want to verify that all pgs in my list are consistent with the rule specified. Let me know if there is a proper way to do this, and thanks. -Chris

3 4

osd_memory_target values
by Mevludin Blazevic 16 Jan '23

16 Jan '23

Hi all, for a ceph cluster with RAM size of 256 GB per node, I would increase the osd_memory_target from default 4GB up to 12GB. Through the ceph dashboard, different values are given to set the new value (global, mon, ..., osd). Is there any difference between them? From my point of view, I should simply set the osd value to 12 GB, am I right? Best, Mevludin

2 1

Building Ceph containers
by Matthew Vernon 16 Jan '23

16 Jan '23

Hi, Is it possible/supported to build Ceph containers on Debian? The build instructions[0] talk about building packages (incl. .debs), but now building containers. Cephadm only supports containerised deployments, but our local policy is that we should only deploy containers we've built ourselves. Is it still the case that only building centos-based images is supported? Perhaps with a follow-up question of why container building isn't supported on the range of platforms that package building is supported on if containerised deployments are where people are meant to be going... Thanks, Matthew [0] https://docs.ceph.com/en/quincy/install/build-ceph/

1 0

Useful MDS configuration for heavily used Cephfs
by E Taka 16 Jan '23

16 Jan '23

Ceph 17.2.5: Hi, I'm looking for a reasonable and useful MDS configuration for a – in future, no experiences until now – heavily used CephFS (~100TB). For example, does it make a difference to increase the mds_cache_memory_limit or the number of MDS instances? The hardware does not set any limits, I just want to know where the default values can be optimized usefully before problem occur. Thanks,

4 5

RGW - large omaps even when buckets are sharded
by Boris Behrens 16 Jan '23

16 Jan '23

Hi, since last week the scrubbing results in large omap warning. After some digging I've got these results: # searching for indexes with large omaps: $ for i in `rados -p eu-central-1.rgw.buckets.index ls`; do rados -p eu-central-1.rgw.buckets.index listomapkeys $i | wc -l | tr -d '\n' >> omapkeys echo " - ${i}" >> omapkeys done $ sort -n omapkeys | tail -n 15 212010 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2342226177.1.0 212460 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2342226177.1.3 212466 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2342226177.1.10 213165 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2342226177.1.4 354692 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.7 354760 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.5 354799 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.1 355040 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.10 355874 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.2 355930 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.3 356499 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.6 356583 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.8 356925 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.4 356935 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.9 358986 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.0 So I have a bucket (ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1) with 11 shards where each shard got around 350k omapkeys. When checking what bucket it is is get a total different number: $ radosgw-admin bucket stats --bucket-id=ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1 { "bucket": "bucket", "num_shards": 11, "tenant": "", "zonegroup": "da651dc1-2663-4e1b-af2e-ac4454f24c9d", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1", "marker": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2296333939.13", "index_type": "Normal", "owner": "user", "ver": "0#45265,1#44764,2#44631,3#44777,4#44859,5#44637,6#44814,7#44506,8#44853,9#44764,10#44813", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", "mtime": "2022-11-16T08:34:17.298979Z", "creation_time": "2021-11-16T09:13:34.480637Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", "usage": { "rgw.main": { "size": 66897607205, "size_actual": 68261179392, "size_utilized": 66897607205, "size_kb": 65329695, "size_kb_actual": 66661308, "size_kb_utilized": 65329695, "num_objects": 663369 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 0 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } It got 11 shards, with a total of 663k files. radosgw-admin bucket limit check gives 60k objects per shard. After getting a list with all omapkeys (total of 3917043) I see that entries look like this (only in less, not in cat) - the ^@ char: $ grep -aF object1 2421332952.1_omapkeys object1 object1^@v910^@i3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK object1^@v913^@iPVPdb60UlfOu4Mwzr.oqojwWzRdgheZ <80>1000_object1^@i3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK <80>1000_object1^(a)iPVPdb60UlfOu4Mwzr.oqojwWzRdgheZ <80>1001_object1 I also pulled the whole bucket index of said bucket via radosgw-admin bi list --bucket bucket > bucket_index_list and searched via jq for the object1: $ jq '.[] | select(.entry.name == "object1")' bucket_index_list { "type": "plain", "idx": "object1", "entry": { "name": "object1", "instance": "", "ver": { "pool": -1, "epoch": 0 }, "locator": "", "exists": "false", "meta": { "category": 0, "size": 0, "mtime": "0.000000", "etag": "", "storage_class": "", "owner": "", "owner_display_name": "", "content_type": "", "accounted_size": 0, "user_data": "", "appendable": "false" }, "tag": "", "flags": 8, "pending_map": [], "versioned_epoch": 0 } } { "type": "plain", "idx": "object1\u0000v910\u0000i3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK", "entry": { "name": "object1", "instance": "3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK", "ver": { "pool": -1, "epoch": 0 }, "locator": "", "exists": "false", "meta": { "category": 0, "size": 0, "mtime": "2022-12-16T00:00:28.651053Z", "etag": "", "storage_class": "", "owner": "user", "owner_display_name": "user", "content_type": "", "accounted_size": 0, "user_data": "", "appendable": "false" }, "tag": "delete-marker", "flags": 7, "pending_map": [], "versioned_epoch": 5 } } { "type": "plain", "idx": "object1\u0000v913\u0000iPVPdb60UlfOu4Mwzr.oqojwWzRdgheZ", "entry": { "name": "object1", "instance": "PVPdb60UlfOu4Mwzr.oqojwWzRdgheZ", "ver": { "pool": 11, "epoch": 2375707 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 10858, "mtime": "2021-12-15T12:05:30.351753Z", "etag": "8cc8ba9599322c17af56996bc0a85af0", "storage_class": "", "owner": "user", "owner_display_name": "user", "content_type": "image/jpeg", "accounted_size": 10858, "user_data": "", "appendable": "false" }, "tag": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297644265.64076989", "flags": 1, "pending_map": [], "versioned_epoch": 2 } } { "type": "instance", "idx": "�1000_object1\u0000i3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK", "entry": { "name": "object1", "instance": "3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK", "ver": { "pool": -1, "epoch": 0 }, "locator": "", "exists": "false", "meta": { "category": 0, "size": 0, "mtime": "2022-12-16T00:00:28.651053Z", "etag": "", "storage_class": "", "owner": "user", "owner_display_name": "user", "content_type": "", "accounted_size": 0, "user_data": "", "appendable": "false" }, "tag": "delete-marker", "flags": 7, "pending_map": [], "versioned_epoch": 5 } } { "type": "instance", "idx": "�1000_object1\u0000iPVPdb60UlfOu4Mwzr.oqojwWzRdgheZ", "entry": { "name": "object1", "instance": "PVPdb60UlfOu4Mwzr.oqojwWzRdgheZ", "ver": { "pool": 11, "epoch": 2375707 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 10858, "mtime": "2021-12-15T12:05:30.351753Z", "etag": "8cc8ba9599322c17af56996bc0a85af0", "storage_class": "", "owner": "user", "owner_display_name": "user", "content_type": "image/jpeg", "accounted_size": 10858, "user_data": "", "appendable": "false" }, "tag": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297644265.64076989", "flags": 1, "pending_map": [], "versioned_epoch": 2 } } Does anyone know what is happening here? And what should I do about the large omap objects? Reshard again?

1 0

User access
by Rhys Powell 13 Jan '23

13 Jan '23

rhys.g.powell(a)gmail.com

1 0