- ceph-users - lists.ceph.io

Unable to mount with 18.2.2
by Albert Shih 16 Jul '24

16 Jul '24

Hi everyone My cluster ceph run currently 18.2.2 and ceph -s say everything are OK root@cthulhu1:/var/lib/ceph/crash# ceph -s cluster: id: 9c5bb196-c212-11ee-84f3-c3f2beae892d health: HEALTH_OK services: mon: 5 daemons, quorum cthulhu1,cthulhu5,cthulhu3,cthulhu4,cthulhu2 (age 4d) mgr: cthulhu1.yhgean(active, since 4d), standbys: cthulhu3.ylmosn, cthulhu5.hqiarz, cthulhu4.odtqjw, cthulhu2.ynvnob mds: 1/1 daemons up, 4 standby osd: 370 osds: 370 up (since 4d), 370 in (since 3M) data: volumes: 1/1 healthy pools: 4 pools, 259 pgs objects: 333.68M objects, 279 TiB usage: 423 TiB used, 5.3 PiB / 5.8 PiB avail pgs: 226 active+clean 19 active+clean+scrubbing+deep 14 active+clean+scrubbing I got 3 clients cephfs : 2 with debian 12 + 18.2.2 1 with Debian 11 + 17.2.7 The Debian 11 client work fine, I try to umount the cephfs and remount it and it's working The first Debian 12 + 18.2.2 are a upgrade from Debian 11 + 17.2.7 and before the update the mount was working, after the upgrade I'm unable to mount the cephfs The second Debian 12 is a fresh install and I'm also unable to mount the cephfs. I check the network and don't see any firewall problem. On the client when I try mount they take few minutes to answer me mount error: no mds server is up or the cluster is laggy on the client I can see : Jul 16 14:10:43 Debian12-1 kernel: [ 860.636012] ceph: corrupt mdsmap Jul 16 14:23:37 Debian12-2 kernel: [11497.406652] ceph: corrupt mdsmap I try to google this error but all I can find are the situation when ceph -s say it's in big trouble. On my case not only ceph -s say everything are ok but the Debian 11 are able to umount/mount/umount/mount/write Any clue or debugging method ? Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: mar. 16 juil. 2024 14:26:47 CEST

3 3

Large omap in index pool even if properly sharded and not "OVER"
by Szabo, Istvan (Agoda) 16 Jul '24

16 Jul '24

Hi, I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects. Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK" This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1" This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920 2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458 Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing? Thank you in advance for your help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 14

How to detect condition for offline compaction of RocksDB?
by Александр Руденко 16 Jul '24

16 Jul '24

Hi, We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case. The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers. For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days… And, I think the problem is in the rocksdb database which grows by levels and slows down requests. And I have few question about compaction: 1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state. We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol. We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs: Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-0] ** ** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 and this log is not usable as ‘metric’. Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options.

1 0

Apt update failing?
by Daniel Brown 15 Jul '24

15 Jul '24

1 0

when calling the CreateTopic operation: Unknown
by farhad kh 15 Jul '24

15 Jul '24

hi, i want to use ceph bucket notification . i try to created topic with below command but get error when used kafka with user/password how can i solved this problem ? my syntax have any problem? https://www.ibm.com/docs/en/storage-ceph/7?topic=management-creating-bucket… https://docs.ceph.com/en/latest/radosgw/notifications/ # aws sns create-topic --name storage --profile=test-notifications --endpoint-url=http://192.168.115.59 --attributes=file://topic.json --debug ##### topic.json: {"push-endpoint": "kafka://usr-test:p@ssw#0rd@kafka.test:9092","verify-ssl": "False", "kafka-ack-level": "broker", "persistent":"true"} ##### error is :---------------- content-type:application/x-www-form-urlencoded; charset=utf-8 host:192.168.115.59 x-amz-date:20240712T120700Z content-type;host;x-amz-date 6f328bc5b0736f23ae2cdf68ccffe1a45c705dd1636f61b999350ae18f8d5ad1 2024-07-12 12:07:00,439 - MainThread - botocore.auth - DEBUG - StringToSign: AWS4-HMAC-SHA256 20240712T120700Z 20240712/podspace/sns/aws4_request 38b7d8721abdd98c214ea763d9dcc324fcbc5982990353140f6b734455e9c01e 2024-07-12 12:07:00,439 - MainThread - botocore.auth - DEBUG - Signature: 99a0f7d40bc011c09dfe7c6aed4bae7f739df52278e23bd4f16fdb1888c001b7 2024-07-12 12:07:00,440 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url= http://192.168.115.59/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.18.156 Python/3.6.8 Linux/5.4.17-2136.326.6.el8uek.x86_64 botocore/1.18.15', 'X-Amz-Date': b'20240712T120700Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=2MRG0TXJFISQAIVE47F1/20240712/podspace/sns/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=99a0f7d40bc011c09dfe7c6aed4bae7f739df52278e23bd4f16fdb1888c001b7', 'Content-Length': '371'}> 2024-07-12 12:07:00,441 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 192.168.115.59:80 2024-07-12 12:07:00,446 - MainThread - urllib3.connectionpool - DEBUG - http://192.168.115.59:80 "POST / HTTP/1.1" 400 206 2024-07-12 12:07:00,447 - MainThread - botocore.parsers - DEBUG - Response headers: {'content-length': '206', 'x-amz-request-id': 'tx00000c7a58038e67798c0-0066911c64-19910-site1', 'accept-ranges': 'bytes', 'content-type': 'application/xml', 'date': 'Fri, 12 Jul 2024 12:07:00 GMT'} 2024-07-12 12:07:00,447 - MainThread - botocore.parsers - DEBUG - Response body: b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><Message></Message><RequestId>tx00000c7a58038e67798c0-0066911c64-19910-site1</RequestId><HostId>19910-site1-podspace</HostId></Error>' 2024-07-12 12:07:00,448 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sns.CreateTopic: calling handler <botocore.retryhandler.RetryHandler object at 0x7f35e0a7f860> 2024-07-12 12:07:00,448 - MainThread - botocore.retryhandler - DEBUG - No retry needed. 2024-07-12 12:07:00,449 - MainThread - awscli.clidriver - DEBUG - Exception caught in main() Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 217, in main return command_table[parsed_args.command](remaining, parsed_args) File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 358, in __call__ return command_table[parsed_args.operation](remaining, parsed_globals) File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 530, in __call__ call_parameters, parsed_globals) File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 650, in invoke client, operation_name, parameters, parsed_globals) File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 662, in _make_client_call **parameters) File "/usr/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/lib/python3.6/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (Unknown) when calling the CreateTopic operation: Unknown 2024-07-12 12:07:00,449 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255 An error occurred (Unknown) when calling the CreateTopic operation: Unknown

2 1

Re: CephFS metadata pool size
by Lars Köppel 15 Jul '24

15 Jul '24

Hello everyone, short update to this problem. The zapped OSD is rebuilt and it has now 1.9 TiB (the expected size ~50%). The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They jumped up and down a lot but the higher one has now also reached 'nearfull' status. How is this possible? What is going on? Does anyone have a solution how to fix this without zapping the OSD? Best regards, Lars [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel(a)ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai

5 19

Ceph osd df including block.db size
by Dyweni - Ceph-Users 14 Jul '24

14 Jul '24

Hi, I realize this comes from an end-of-life release (Mimic), but I believe it still applies to current releases. I've just upgraded from Luminous to Mimic and noticed that my 'ceph osd df' output now includes the block.db sizes. This is pushing all OSD's into nearfull status (~70% used in Luminous, ~85% used in Mimic). Each OSD is 1TB HDD (block) and 1TB NVME (block.db). I'd like to understand two things: 1. Why is the reasoning behind including the block.db size/usage in 'ceph osd df'? 2. Is there a way to exclude block.db size/usage from 'ceph osd df'? I've looked through the ceph git comments and noticed the following: * b97f3d6761208a1d317db3d0dba8371181e0b49f (8/21/2016, v11.0.0/master), Sage added: + // include dedicated db, too, if that isn't the shared device. + if (bluefs_shared_bdev != BlueFS::BDEV_DB) { + buf->available += bluefs->get_free(BlueFS::BDEV_DB); + buf->total += bluefs->get_total(BlueFS::BDEV_DB); + } * f798b13a043523e1677abdcb3df0d9ced8774098 (1/31/2018, v13.0.1/master), Igor removed: - // include dedicated db, too, if that isn't the shared device. - if (bluefs_shared_bdev != BlueFS::BDEV_DB) { - buf->total += bluefs->get_total(BlueFS::BDEV_DB); - } * 266d1649cf1d57ab32a71a39347bb418a4490c78 (2/20/2018, v13.0.1/master), Sage re-added: + if (bluefs) { + // include dedicated db, too, if that isn't the shared device. + if (bluefs_shared_bdev != BlueFS::BDEV_DB) { + buf->total += bluefs->get_total(BlueFS::BDEV_DB); } I've checked the main branch (as of right now, commit d622325c17bb98672ad22e60a372becfbeb04cff) and the calculation still exists: File: src/os/bluestore/BlueStore.cc Line: 11480 // include dedicated db, too, if that isn't the shared device. if (bluefs_layout.shared_bdev != BlueFS::BDEV_DB) { buf->total += bluefs->get_total(BlueFS::BDEV_DB); } Thanks, Dyweni

1 0

ceph orch osd rm --zap --replace leaves cluster in odd state
by Matthew Vernon 14 Jul '24

14 Jul '24

Hi, I want to prepare a failed disk for replacement. I did: ceph orch osd rm 35 --zap --replace and it's now in the state "Done, waiting for purge", with 0 pgs, and REPLACE and ZAP set to true. It's been like this for some hours, and now my cluster is unhappy: [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.35 on host moss-be1002 not managed by cephadm (the OSD is down & out) ...and also neither the disk nor the relevant NVME LV has been zapped. I have my OSDs deployed via a spec: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe" And before issuing the ceph orch osd rm I set that to be unmanaged (ceph orch set-unmanaged osd.rrd_single_NVMe), as obviously I don't want ceph to just try and re-make a new OSD on the sad disk. I'd expected from the docs[0] that what I did would leave me with a system ready for the failed disk to be swapped (and that I could then mark osd.rrd_single_NVMe as managed again, and a new OSD built), including removing/wiping the NVME lv so it can be removed. What did I do wrong? I don't much care about the OSD id (but obviously it's neater to not just incrementally increase OSD numbers every time a disk died), but I thought that telling ceph orch not to make new OSDs then using ceph orch osd rm to zap the disk and NVME lv would have been the way to go... Thanks, Matthew [0] https://docs.ceph.com/en/reef/cephadm/services/osd/#replacing-an-osd

3 3

Help needed please ! Filesystem became read-only !
by nbarbier＠deltaonline.net 14 Jul '24

14 Jul '24

Hello, First of all, thanks for reading my message. I set up a Ceph version 18.2.2 cluster with 4 nodes, everything went fine for a while, but after copying some files, the storage showed a warning status and the following message : "HEALTH_WARN: 1 MDSs are read only mds.PVE-CZ235007SH(mds.0): MDS in read-only mode". The logs are showing : Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9999> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x1000000039c object, pool 5, errno -2 Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9998> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 mds.0.189541 unhandled write error (2) No such file or directory, force readonly... After googling for a while, I did not find a hint to understand more precisely the root cause. Any help would we greatly appreciated, or even a link to post this request elsewhere if this is not the place to. Please find below additional details if needed. Thanks a lot ! Nicolas --- # ceph osd dump [...] pool 5 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 292 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.51 [...] # ceph osd lspools 1 .mgr 4 cephfs_data 5 cephfs_metadata 18 ec-pool-001-data 19 ec-pool-001-metadata # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 633 TiB 633 TiB 61 GiB 61 GiB 0 TOTAL 633 TiB 633 TiB 61 GiB 61 GiB 0 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 119 MiB 31 357 MiB 0 200 TiB cephfs_data 4 32 71 KiB 8.38k 240 KiB 0 200 TiB cephfs_metadata 5 32 329 MiB 6.56k 987 MiB 0 200 TiB ec-pool-001-data 18 32 42 GiB 15.99k 56 GiB 0 451 TiB ec-pool-001-metadata 19 32 0 B 0 0 B 0 200 TiB # ceph status cluster: id: f16f53e1-7028-440f-bf48-f99912619c33 health: HEALTH_WARN 1 MDSs are read only services: mon: 4 daemons, quorum PVE-CZ235007SG,PVE-CZ2341016V,PVE-CZ235007SH,PVE-CZ2341016T (age 35h) mgr: PVE-CZ235007SG(active, since 2d), standbys: PVE-CZ235007SH, PVE-CZ2341016T, PVE-CZ2341016V mds: 1/1 daemons up, 3 standby osd: 48 osds: 48 up (since 2d), 48 in (since 3d) data: volumes: 1/1 healthy pools: 5 pools, 129 pgs objects: 30.97k objects, 42 GiB usage: 61 GiB used, 633 TiB / 633 TiB avail pgs: 129 active+clean

4 6

Questions about the usage of space in Ceph
by 2644294460＠qq.com 13 Jul '24

13 Jul '24

My Ceph cluster has a CephFS file system, using an erasure-code data pool (k=8, m=2), which has used 14TiB of space. My CephFS has 19 subvolumes, and each subvolume automatically creates a snapshot every day and keeps it for 3 days. The problem is that when I manually calculate the disk space usage of each subvolume directory in CephFS, the total amount is only 8.4TiB. I don't know why this is happening. Do snapshots take up a lot of space? ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Thank you all!

2 1