January 2023 - ceph-users

Re: Telemetry service is temporarily down

by Yaarit Hatuka

Hi everyone, Our telemetry service is up and running again. Thanks Adam Kraitman and Dan Mick for restoring the service. We thank you for your patience and appreciate your contribution to the project! Thanks, Yaarit On Tue, Jan 3, 2023 at 3:14 PM Yaarit Hatuka <yhatuka(a)redhat.com> wrote: > Hi everyone, > > We are having some infrastructure issues with our telemetry backend, and > we are working on fixing it. > Thanks Jan Horacek for opening this issue > <https://tracker.ceph.com/issues/58371> [1]. We will update once the > service is back up. > We are sorry for any inconvenience you may be experiencing, and appreciate > your patience. > > Thanks, > Yaarit > > [1] https://tracker.ceph.com/issues/58371 >

9 months, 3 weeks

1
2
0 0

January Ceph Science Virtual User Group

by Kevin Hrpcek

Hey all, We will be having a Ceph science/research/big cluster call on Tuesday January 31st. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20230131 Ceph calendar event details: January 31, 2023 15:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc <http://bjn.vc> 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS/TROPICS Space Science & Engineering Center University of Wisconsin-Madison

9 months, 4 weeks

4
3
0 0

How to get RBD client log?

by Jinhao Hu

Hi, How can I collect the logs of the RBD client?

10 months

3
3
0 0

Re: Debian update to 16.2.11-1~bpo11+1 failing

by Matthias Aebi

I can only confirm that. The file https://download.ceph.com/debian-pacific/pool/main/c/ceph/python3-rados_16.… is clearly missing on the ceph download server which makes it impossible to install the upgrade on Debian. And as the previous 16.2.10 Package definition has been replaced with the latest one, this currently makes it impossible to install Pacific on a Debian system.

10 months

3
2
0 0

Write amplification for CephFS?

by Manuel Holtgrewe

Dear all, is there information available anywhere about the write amplification for CephFS? I found quite some material on write amplification of VMs using journaled file system on top of RBD but nothing as it relates to CephFS? From my understanding I would expect the following: - for X-rep, data needs to be written to X pgs (factor X) - for k-m EC, data is written to k+m pgs (factor (k+m)/k) Am I correct? How much is added by meta data, WAL? Do I miss anything? Best wishes, Manuel

10 months

2
5
0 0

rbd online sparsify image

by Jiatong Shen

Hello community experts, I would like to know the status of rbd image sparsify. From the website, it should be added at Nautilus ( https://docs.ceph.com/en/latest/releases/nautilus/ from pr (26226 <https://github.com/ceph/ceph/pull/26226>,) ) but on mimic it is related again https://docs.ceph.com/en/latest/releases/octopus/) Is it still the same commit? or is there any update/bug fixes that is not backported to 14.x? I am also curious why it is named re-sparsify in the pr title? Thank you very much for the help. -- Best Regards, Jiatong Shen

10 months

2
3
0 0

OSDs will not start

by Geoffrey Rhodes

Good day all, I've an issue with a few OSDs (in two different nodes) that attempt to start but fail / crash quite quickly. They are all LVM disks. I've tried upgrading software, health checks on the hardware (nodes and disks) and there doesn't seem to be any issues there. Recently I've had a few "other" disks physically fail in the cluster and now have one PG down which is blocking some IO on CephFS. I've added the output of the osd journalctl and the osd log below in case it's helpful to identify anything obvious. I also set debug bluefs = 20 , saw this in another post. I recently manually upgraded this node to (17.2.0) before the problem began, later to (17.2.5). - The other osds in this node start / run fine. The other node (15.2.17) also has a few osds that will not start and some that run without issue. Could anyone point me in the right direction to investigate and solve my osd issues. https://pastebin.com/3PkCabdf https://pastebin.com/BT9bnhSb Production system mainly used for CephFS OS: Ubuntu 20.04.5 LTS Ceph versions: 15.2.17 - Octopus (one OSD node manually upgraded to 17.2.5 - Quincy) Erasure data pool (K=4, M=2) - The journal's for each osd are co-located on each drive Kind regards Geoffrey Rhodes

10 months

1
1
0 0

rbd-mirror ceph quincy Not able to find rbd_mirror_journal_max_fetch_bytes config in rbd mirror

by ankit raikwar

Hello All, In the ceph quincy Not able to find rbd_mirror_journal_max_fetch_bytes config in rbd mirror i configured the ceph cluster almost 400 tb and enable the rbd-mirror in the starting stage i'm able to achive the almost 9 GB speed , but after the rebalane completed of the all the images . rbd-mirror speed got automaticily reduce to between 4 to 5 mbps. in my primary cluster we are continuelsy writing the 50 to 400 mbps data but replication speed only we get the 4 to 5 mbps. also we have the 10 Gbps replication network bandwidth. Note::- I also try to find the option rbd_mirror_journal_max_fetch_bytes but i'm not able to find the this option in the configuration. also when i try to set from the command line it's showing error like command: ceph config set client.rbd rbd_mirror_journal_max_fetch_bytes 33554432 error: Error EINVAL: unrecognized config option 'rbd_mirror_journal_max_fetch_bytes' cluster version ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Please suggest any alternative way to configurre this option or how i can improve the replication n/w speed.

10 months

2
5
0 0

Real memory usage of the osd(s)

by Szabo, Istvan (Agoda)

Hello, If buffered_io is enabled, is there a way to know what is the exactly used physical memory from each osd? What I've found is the dump_mempools which last entries are the following, but this bytes would be the real physical memory usage? "total": { "items": 60005205, "bytes": 995781359 Also which metric is this value? I haven't found any. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

10 months

1
0
0 0

All pgs unknown

by Daniel Brunner

Hi, my ceph cluster started to show HEALTH_WARN, there are no healthy pgs left, all are unknown, but it seems my cephfs is still readable, how to investigate this any further? $ sudo ceph -s cluster: id: ddb7ebd8-65b5-11ed-84d7-22aca0408523 health: HEALTH_WARN failed to probe daemons or devices noout flag(s) set Reduced data availability: 339 pgs inactive services: mon: 1 daemons, quorum flucky-server (age 3m) mgr: flucky-server.cupbak(active, since 3m) mds: 1/1 daemons up osd: 18 osds: 18 up (since 26h), 18 in (since 7w) flags noout rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 11 pools, 339 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 339 unknown $ sudo ceph fs status cephfs - 2 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.flucky-server.ldzavv Reqs: 0 /s 61.9k 61.9k 17.1k 54.5k POOL TYPE USED AVAIL cephfs_metadata metadata 0 0 cephfs_data data 0 0 MDS version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) $ docker logs ceph-ddb7ebd8-65b5-11ed-84d7-22aca0408523-mon-flucky-server cluster 2023-01-27T12:15:30.437140+0000 mgr.flucky-server.cupbak (mgr.144098) 200 : cluster [DBG] pgmap v189: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2023-01-27T12:15:31.995+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:15:32.437854+0000 mgr.flucky-server.cupbak (mgr.144098) 201 : cluster [DBG] pgmap v190: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:32.373735+0000 osd.9 (osd.9) 123948 : cluster [DBG] 9.a deep-scrub starts cluster 2023-01-27T12:15:33.013990+0000 osd.2 (osd.2) 41797 : cluster [DBG] 5.6 scrub starts cluster 2023-01-27T12:15:33.402881+0000 osd.9 (osd.9) 123949 : cluster [DBG] 9.13 scrub starts cluster 2023-01-27T12:15:34.438591+0000 mgr.flucky-server.cupbak (mgr.144098) 202 : cluster [DBG] pgmap v191: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:35.461575+0000 osd.9 (osd.9) 123950 : cluster [DBG] 7.16 deep-scrub starts debug 2023-01-27T12:15:37.005+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:15:36.439416+0000 mgr.flucky-server.cupbak (mgr.144098) 203 : cluster [DBG] pgmap v192: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:36.925368+0000 osd.2 (osd.2) 41798 : cluster [DBG] 7.15 deep-scrub starts cluster 2023-01-27T12:15:37.960907+0000 osd.2 (osd.2) 41799 : cluster [DBG] 6.6 scrub starts cluster 2023-01-27T12:15:38.440099+0000 mgr.flucky-server.cupbak (mgr.144098) 204 : cluster [DBG] pgmap v193: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:38.482333+0000 osd.9 (osd.9) 123951 : cluster [DBG] 2.2 scrub starts cluster 2023-01-27T12:15:38.959557+0000 osd.2 (osd.2) 41800 : cluster [DBG] 9.47 scrub starts cluster 2023-01-27T12:15:39.519980+0000 osd.9 (osd.9) 123952 : cluster [DBG] 4.b scrub starts cluster 2023-01-27T12:15:40.440711+0000 mgr.flucky-server.cupbak (mgr.144098) 205 : cluster [DBG] pgmap v194: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2023-01-27T12:15:42.012+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:15:41.536421+0000 osd.9 (osd.9) 123953 : cluster [DBG] 2.7 scrub starts cluster 2023-01-27T12:15:42.441314+0000 mgr.flucky-server.cupbak (mgr.144098) 206 : cluster [DBG] pgmap v195: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:43.954128+0000 osd.2 (osd.2) 41801 : cluster [DBG] 9.4f scrub starts cluster 2023-01-27T12:15:44.441897+0000 mgr.flucky-server.cupbak (mgr.144098) 207 : cluster [DBG] pgmap v196: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:45.944038+0000 osd.2 (osd.2) 41802 : cluster [DBG] 1.1f deep-scrub starts debug 2023-01-27T12:15:47.019+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:15:46.442532+0000 mgr.flucky-server.cupbak (mgr.144098) 208 : cluster [DBG] pgmap v197: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:47.543275+0000 osd.9 (osd.9) 123954 : cluster [DBG] 2.3 scrub starts cluster 2023-01-27T12:15:48.443081+0000 mgr.flucky-server.cupbak (mgr.144098) 209 : cluster [DBG] pgmap v198: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:48.515994+0000 osd.9 (osd.9) 123955 : cluster [DBG] 1.19 scrub starts cluster 2023-01-27T12:15:49.957501+0000 osd.2 (osd.2) 41803 : cluster [DBG] 7.11 scrub starts cluster 2023-01-27T12:15:50.443740+0000 mgr.flucky-server.cupbak (mgr.144098) 210 : cluster [DBG] pgmap v199: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:50.473278+0000 osd.9 (osd.9) 123956 : cluster [DBG] 5.10 scrub starts debug 2023-01-27T12:15:52.026+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:15:51.506790+0000 osd.9 (osd.9) 123957 : cluster [DBG] 5.1b deep-scrub starts cluster 2023-01-27T12:15:51.957026+0000 osd.2 (osd.2) 41804 : cluster [DBG] 4.16 scrub starts cluster 2023-01-27T12:15:52.444197+0000 mgr.flucky-server.cupbak (mgr.144098) 211 : cluster [DBG] pgmap v200: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:52.939466+0000 osd.2 (osd.2) 41805 : cluster [DBG] 5.1c scrub starts cluster 2023-01-27T12:15:53.470511+0000 osd.9 (osd.9) 123958 : cluster [DBG] 8.8 scrub starts cluster 2023-01-27T12:15:53.916653+0000 osd.2 (osd.2) 41806 : cluster [DBG] 5.6 deep-scrub starts cluster 2023-01-27T12:15:54.422547+0000 osd.9 (osd.9) 123959 : cluster [DBG] 9.3b deep-scrub starts cluster 2023-01-27T12:15:54.444675+0000 mgr.flucky-server.cupbak (mgr.144098) 212 : cluster [DBG] pgmap v201: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:55.409322+0000 osd.9 (osd.9) 123960 : cluster [DBG] 9.34 deep-scrub starts cluster 2023-01-27T12:15:55.921989+0000 osd.2 (osd.2) 41807 : cluster [DBG] 7.15 deep-scrub starts debug 2023-01-27T12:15:57.029+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 audit 2023-01-27T12:15:56.339185+0000 mgr.flucky-server.cupbak (mgr.144098) 213 : audit [DBG] from='client.144120 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch cluster 2023-01-27T12:15:56.445186+0000 mgr.flucky-server.cupbak (mgr.144098) 214 : cluster [DBG] pgmap v202: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:57.883819+0000 osd.2 (osd.2) 41808 : cluster [DBG] 6.6 deep-scrub starts cluster 2023-01-27T12:15:58.445697+0000 mgr.flucky-server.cupbak (mgr.144098) 215 : cluster [DBG] pgmap v203: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:59.415908+0000 osd.9 (osd.9) 123961 : cluster [DBG] 9.25 scrub starts cluster 2023-01-27T12:16:00.446210+0000 mgr.flucky-server.cupbak (mgr.144098) 216 : cluster [DBG] pgmap v204: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2023-01-27T12:16:02.033+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:16:02.446670+0000 mgr.flucky-server.cupbak (mgr.144098) 217 : cluster [DBG] pgmap v205: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2023-01-27T12:16:04.953+0000 7fa908bf2700 0 mon.flucky-server@0(leader) e1 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/mirror_snapshot_schedule"} v 0) v1 debug 2023-01-27T12:16:04.953+0000 7fa908bf2700 0 log_channel(audit) log [INF] : from='mgr.144098 172.18.0.1:0/3192812764' entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/mirror_snapshot_schedule"}]: dispatch debug 2023-01-27T12:16:04.969+0000 7fa908bf2700 0 mon.flucky-server@0(leader) e1 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/trash_purge_schedule"} v 0) v1 debug 2023-01-27T12:16:04.969+0000 7fa908bf2700 0 log_channel(audit) log [INF] : from='mgr.144098 172.18.0.1:0/3192812764' entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/trash_purge_schedule"}]: dispatch cluster 2023-01-27T12:16:04.447207+0000 mgr.flucky-server.cupbak (mgr.144098) 218 : cluster [DBG] pgmap v206: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:16:04.537785+0000 osd.9 (osd.9) 123962 : cluster [DBG] 9.27 scrub starts cluster 2023-01-27T12:16:04.795757+0000 osd.2 (osd.2) 41809 : cluster [DBG] 9.47 scrub starts audit 2023-01-27T12:16:04.956941+0000 mon.flucky-server (mon.0) 304 : audit [INF] from='mgr.144098 172.18.0.1:0/3192812764' entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/mirror_snapshot_schedule"}]: dispatch audit 2023-01-27T12:16:04.973875+0000 mon.flucky-server (mon.0) 305 : audit [INF] from='mgr.144098 172.18.0.1:0/3192812764' entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/trash_purge_schedule"}]: dispatch debug 2023-01-27T12:16:07.039+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:16:06.447964+0000 mgr.flucky-server.cupbak (mgr.144098) 219 : cluster [DBG] pgmap v207: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:16:07.606921+0000 osd.9 (osd.9) 123963 : cluster [DBG] 9.1c scrub starts cluster 2023-01-27T12:16:08.448450+0000 mgr.flucky-server.cupbak (mgr.144098) 220 : cluster [DBG] pgmap v208: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:16:08.629529+0000 osd.9 (osd.9) 123964 : cluster [DBG] 9.2c scrub starts

10 months

2
1
0 0

2023

2022

2021

2020

2019

ceph-users January 2023