- ceph-users - lists.ceph.io

ERROR: read_key_entry() idx= 1000_ ret=-2
by by morphin 14 Apr '21

14 Apr '21

Hello everyone! I'm running nautilus 14.2.16 and I'm using RGW with Beast frontend. I see this eror log in every SSD osd which is using for rgw index. Can you please tell me what is the problem? OSD LOG: cls_rgw.cc:1102: ERROR: read_key_entry() idx=�1000_matches/xdir/05/21/27260.jpg ret=-2 cls_rgw.cc:1102: ERROR: read_key_entry() idx=�1000_matches/xdir/05/21/27253.jpg ret=-2 RADOSGW LOG: 2021-04-15 01:53:54.385 7f2e0f8e7700 1 beast: 0x55a4439f8710: 10.151.101.15 - - [2021-04-15 01:53:54.0.385327s] "HEAD /xdir/04/13/704745.jpg HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1062.12.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.251-b08 java/1.8.0_251 groovy/2.4.3 vendor/Oracle_Corporation" - 2021-04-15 01:53:54.385 7f2d8b7df700 1 ====== starting new request req=0x55a4439f8710 ===== 2021-04-15 01:53:54.405 7f2e008c9700 1 ====== req done req=0x55a43dbc6710 op status=0 http_status=204 latency=0.300003s ====== 2021-04-15 01:53:54.405 7f2e008c9700 1 beast: 0x55a43dbc6710: 10.151.101.15 - - [2021-04-15 01:53:54.0.405327s] "DELETE /xdir/05/21/21586.gz HTTP/1.1" 204 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1062.12.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.251-b08 java/1.8.0_251 groovy/2.4.3 vendor/Oracle_Corporation" - 2021-04-15 01:53:54.405 7f2d92fee700 1 ====== starting new request req=0x55a43dbc6710 ===== 2021-04-15 01:53:54.405 7f2d92fee700 0 WARNING: couldn't find acl header for object, generating default 2021-04-15 01:53:54.405 7f2d92fee700 1 ====== req done req=0x55a43dbc6710 op status=0 http_status=200 latency=0s ====== 2021-04-15 01:53:54.405 7f2d92fee700 1 beast: 0x55a43dbc6710: 10.151.101.15 - - [2021-04-15 01:53:54.0.405327s] "HEAD /xdir/2013/11/20/2a67508e-d7dd-4e0f-b959-d7575d5f65b1 HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281 groovy/2.5.6 vendor/Oracle_Corporation" - CEPH OSD DF ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 19 ssd 0.87320 1.00000 894 GiB 436 GiB 101 GiB 332 GiB 2.5 GiB 458 GiB 48.75 1.84 115 up 208 ssd 0.87329 1.00000 894 GiB 161 GiB 87 GiB 73 GiB 978 MiB 733 GiB 18.00 0.68 113 up 199 ssd 0.87320 1.00000 894 GiB 272 GiB 106 GiB 163 GiB 2.4 GiB 623 GiB 30.37 1.14 123 up 202 ssd 0.87329 1.00000 894 GiB 239 GiB 73 GiB 165 GiB 1.4 GiB 655 GiB 26.77 1.01 106 up 39 ssd 0.87320 1.00000 894 GiB 450 GiB 87 GiB 361 GiB 2.3 GiB 444 GiB 50.36 1.90 113 up 207 ssd 0.87329 1.00000 894 GiB 204 GiB 100 GiB 98 GiB 6.0 GiB 691 GiB 22.76 0.86 118 up 59 ssd 0.87320 1.00000 894 GiB 372 GiB 107 GiB 263 GiB 3.0 GiB 522 GiB 41.64 1.57 122 up 203 ssd 0.87329 1.00000 894 GiB 206 GiB 79 GiB 124 GiB 2.4 GiB 689 GiB 23.00 0.87 117 up 79 ssd 0.87320 1.00000 894 GiB 447 GiB 103 GiB 342 GiB 1.8 GiB 447 GiB 49.97 1.88 120 up 206 ssd 0.87329 1.00000 894 GiB 200 GiB 81 GiB 119 GiB 1.0 GiB 694 GiB 22.38 0.84 94 up 99 ssd 0.87320 1.00000 894 GiB 333 GiB 87 GiB 244 GiB 2.0 GiB 562 GiB 37.19 1.40 106 up 205 ssd 0.87329 1.00000 894 GiB 316 GiB 83 GiB 232 GiB 1.1 GiB 579 GiB 35.29 1.33 117 up 114 ssd 0.87329 1.00000 894 GiB 256 GiB 100 GiB 154 GiB 1.7 GiB 638 GiB 28.61 1.08 113 up 200 ssd 0.87329 1.00000 894 GiB 266 GiB 100 GiB 165 GiB 1.1 GiB 628 GiB 29.76 1.12 128 up 139 ssd 0.87320 1.00000 894 GiB 234 GiB 79 GiB 153 GiB 1.7 GiB 660 GiB 26.14 0.98 104 up 204 ssd 0.87329 1.00000 894 GiB 173 GiB 113 GiB 59 GiB 1.2 GiB 721 GiB 19.37 0.73 124 up 119 ssd 0.87329 1.00000 894 GiB 248 GiB 108 GiB 139 GiB 1.9 GiB 646 GiB 27.76 1.05 130 up 159 ssd 0.87329 1.00000 894 GiB 196 GiB 94 GiB 99 GiB 2.6 GiB 699 GiB 21.87 0.82 109 up 179 ssd 0.87329 1.00000 894 GiB 427 GiB 81 GiB 341 GiB 4.7 GiB 467 GiB 47.73 1.80 114 up 201 ssd 0.87329 1.00000 894 GiB 346 GiB 102 GiB 242 GiB 1.8 GiB 548 GiB 38.71 1.46 128 up CEPH IOSTAT +---------------+---------------+---------------+---------------+---------------+---------------+ | Read | Write | Total | Read IOPS | Write IOPS | Total IOPS | +---------------+---------------+---------------+---------------+---------------+---------------+ | 329 MiB/s | 39 MiB/s | 368 MiB/s | 109027 | 1646 | 110673 | | 329 MiB/s | 39 MiB/s | 368 MiB/s | 109027 | 1646 | 110673 | | 331 MiB/s | 39 MiB/s | 371 MiB/s | 114915 | 1631 | 116547 | | 331 MiB/s | 39 MiB/s | 371 MiB/s | 114915 | 1631 | 116547 | | 308 MiB/s | 42 MiB/s | 350 MiB/s | 108469 | 1635 | 110104 | | 308 MiB/s | 42 MiB/s | 350 MiB/s | 108469 | 1635 | 110104 | | 291 MiB/s | 44 MiB/s | 335 MiB/s | 105828 | 1687 | 107516 |

1 1

How to disable ceph-grafana during cephadm bootstrap
by mabi 14 Apr '21

14 Apr '21

Hello, When bootstrapping a new ceph Octopus cluster with "cephadm bootstrap", how can I tell the cephadm bootstrap NOT to install the ceph-grafana container? Thank you very much in advance for your answer. Best regards, Mabi

2 2

Ceph Month June 2021 Event
by Mike Perez 14 Apr '21

14 Apr '21

Hi everyone, In June 2021, we're hosting a month of Ceph presentations, lightning talks, and unconference sessions such as BOFs. There is no registration or cost to attend this event. The CFP is now open until May 12th. https://ceph.io/events/ceph-month-june-2021/cfp Speakers will receive confirmation that their presentation is accepted and further instructions for scheduling by May 16th. The schedule will be available on May 19th. Join the Ceph community as we discuss how Ceph, the massively scalable, open-source, software-defined storage system, can radically improve the economics and management of data storage for your enterprise. -- Mike Perez

1 0

Monitor dissapears/stopped after testing monitor-host loss and recovery
by Kai Börnert 14 Apr '21

14 Apr '21

Hi, I'm currently testing some disaster scenarios. When removing one osd/monitor host, I see that a new quorum is build without the missing host. The missing host is listed in the dashboard under Not In Quorum, so probably everything as expected. After restarting the host, I see that the osd's come back online and everything appears to be working, however the quorum is still only with two monitors. Looking at the services, I can see that it is somehow stopped. Is this expected and I must start it manually somehow, or should it work? The whole cluster is deployed using cephadm (the node was the initial bootstrap one, if that is important) Greetings, Kai

1 0

Exporting CephFS using Samba preferred method
by Martin Palma 14 Apr '21

14 Apr '21

Hello, what is the currently preferred method, in terms of stability and performance, for exporting a CephFS directory with Samba? - locally mount the CephFS directory and export it via Samba? - using the "vfs_ceph" module of Samba? Best, Martin

4 3

As the cluster is filling up, write performance decreases
by zp_8483 13 Apr '21

13 Apr '21

Backend: XFS for the filestore back-end. In our testing, we found the performance decreases when cluster usage exceed default nearfull ratio(85%), is it by design?

3 2

ceph rgw why are reads faster for larger than 64kb object size
by Ronnie Puthukkeril 13 Apr '21

13 Apr '21

Environment: Ceph Nautilus 14.2.8 Object Storage Data nodes: 12 * HDD OSDs drives each with a 12TB capacity + 2 * SSD OSDs drives for rgw bucket index pool & rgw meta pool. Custom configs (since we dealing with a majority smaller sized objects) bluestore_min_alloc_size_ssd 4096 bluestore_min_alloc_size_hdd 4096 Observations from cosbench performance tests Stage Op-Type Op-Count Byte-Count Avg-ResTime s7-read1KB 48W read 2004202 2004202000 43.11 s13-read2KB 48W read 2013906 4027812000 42.9 s19-read4KB 48W read 2014701 8058804000 42.88 s25-read8KB 48W read 2002337 16018696000 43.15 s31-read16KB 48W read 1987785 31804560000 43.46 s37-read32KB 48W read 1976190 63238080000 43.7 s43-read64KB 48W read 1929183 123467712000 44.78 s49-read128KB 48W read 9965032 1275524096000 8.67 s55-read256KB 48W read 6505554 1665421824000 13.28 The response time improves drastically when the object size is greater than 64KB. What could be the reason? Thanks, Ronnie [https://d1dejaj6dcqv24.cloudfront.net/asset/image/email-banner-384-2x.png]<https://www.qualys.com/email-banner> This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. NOTE that all incoming emails sent to Qualys email accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional emails ("spam"). If you have any concerns about this process, please contact us.

1 3

Announcing go-ceph v0.9.0
by John Mulligan 13 Apr '21

13 Apr '21

I'm happy to announce another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.9.0 Changes in the release are detailed in the link above. The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. Specific questions, comments, bugs etc are best directed at our github issues tracker or github discussions forum. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

1 0

BADAUTHORIZER in Nautilus, unknown PGs, slow peering, very slow client I/O
by Nico Schottelius 13 Apr '21

13 Apr '21

Good morning, I've look somewhat intensively through the list and it seems we are rather hard hit by this. Originally yesterday started on a mixed 14.2.9 and 14.2.16 cluster (osds, mons were all 14.2.16). We started phasing in 7 new osds, 6 of them throttled by reweighting to 0.1. Symptoms are many unknown PGs, very long stuck in peering (hours), slow activating and the infamous BADAUTHORIZER message. Client I/O is almost 0, not only of the pool with new OSDs, but also of other pools. We tried restarting all OSDs one by one, which seemed to clear the unknown PGs, however after around 1h they came back to unknown state. Tuning the --osd-max-backfills=.. and --osd-recovery-max-active=... to 1 does not improve the situation, so we are currently running at =7 for both of them. An excerpt from one of the osds which heavily logs this issue is below. Any pointer on how this problem was solved on nautilus before is much appreciated, as the issue started late last night. Best regards, Nico 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: (Original Log Time 2021/04/13-07:38:37.275255) EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "compaction_finished", "compaction_time_micros": 5288403, "compaction_time_cpu_micros": 2940370, "output_level": 2, "num_output_files": 6, "total_output_size": 370673432, "num_input_records": 983055, "num_output_records": 641964, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 35, 0, 0, 0, 0]} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 421257} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401289} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401288} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401278} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401277} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401258} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401257} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401256} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 6] Compacting 1@1 + 5@2 files to L2, score 1.17 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 5 Base level 1, inputs: [421255(65MB)], [401318(65MB) 401319(65MB) 401320(65MB) 401321(65MB) 401322(65MB)] 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 6, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [421255], "files_L2": [401318, 401319, 401320, 401321, 401322], "score": 1.17224, "input_data_size": 413630461} 2021-04-13 07:38:37.307 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6862/8751 conn(0x55d520fbcd80 0x55d520ffa000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:37.315 7f15a372c700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6835/13651 conn(0x55d521b1bf80 0x55d521b19800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:37.707 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6862/8751 conn(0x55d520fbcd80 0x55d520ffa000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:37.715 7f15a372c700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6835/13651 conn(0x55d521b1bf80 0x55d521b19800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.119 7f159a469700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 6] Generated table #421271: 130853 keys, 68923910 bytes 2021-04-13 07:38:38.119 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292318119320, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 421271, "file_size": 68923910, "table_properties": {"data_size": 67111841, "index_size": 1483972, "filter_size": 327237, "raw_key_size": 10909111, "raw_average_key_size": 83, "raw_value_size": 62457370, "raw_average_value_size": 477, "num_data_blocks": 16807, "num_entries": 130853, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-04-13 07:38:38.331 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febb:68f0]:6884/29245 conn(0x55d521b1a000 0x55d521b16800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.331 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febb:68f0]:6819/17174 conn(0x55d521b1c400 0x55d521b38000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.507 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6862/8751 conn(0x55d520fbcd80 0x55d520ffa000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.515 7f15a372c700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6835/13651 conn(0x55d521b1bf80 0x55d521b19800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.731 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febb:68f0]:6884/29245 conn(0x55d521b1a000 0x55d521b16800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.731 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febb:68f0]:6819/17174 conn(0x55d521b1c400 0x55d521b38000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER -- Sustainable and modern Infrastructures by ungleich.ch

1 2

Enable Dashboard Active Alerts
by E Taka 13 Apr '21

13 Apr '21

Hi, this is documented with many links to other documents, which unfortunately only confused me. In our 6-Node-Ceph-Cluster (Pacific) the Dashboard tells me that I should "provide the URL to the API of Prometheus' Alertmanager". We only use Grafana and Prometheus which are deployed by cephadm. We did not configure anything unusual, with own containers or so. We use just the standard cephadm installation. What the documentation writes about how it "should look like" (https://docs.ceph.com/en/pacific/mgr/dashboard/#enabling-prometheus-alerting) seems to exist in the Docker-Container "prom/alertmanager:v0.20.0" in file /etc/alertmanager/alertmanager.yml […] - name: 'ceph-dashboard' webhook_configs: - url: 'https://ceph01:8443/api/prometheus_receiver' - url: 'https://10.149.12.22:8443/api/prometheus_receiver' […] (10.149.12.22 is the IP address for ceph01) Nevertheless I get the message above from the Dashboard. My questions: What do I have to write in which file or which commands, so that I can access the alerts via the dashboard? Of course this should survive reboots and updates. Thanks. Erich

1 0