May 2020 - ceph-users - lists.ceph.io

PGS INCONSISTENT - read_error - replace disk or pg repair then replace disk
by Peter Lewis 24 May '20

24 May '20

Hello, I came across a section of the documentation that I don't quite understand. In the section about inconsistent PGs it says if one of the shards listed in `rados list-inconsistent-obj` has a read_error the disk is probably bad. Quote from documentation: https://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/… `If read_error is listed in the errors attribute of a shard, the inconsistency is likely due to disk errors. You might want to check your disk used by that OSD.` I determined that the disk is bad by looking at the output of smartctl. I would think that replacing the disk by removing the OSD from the cluster and allowing the cluster to recover would fix this inconsistency error without having to run `ceph pg repair`. Can I just replace the OSD and the inconsistency will be resolved by the recovery? Or would it be better to run `ceph pg repair` and then replace the OSD associated with that bad disk? Thanks!

3 2

Re: 15.2.2 Upgrade - Corruption: error in middle of record
by Chris Palmer 23 May '20

23 May '20

I'm getting similar errors after rebooting a node. Cluster was upgraded 15.2.1 -> 15.2.2 yesterday. No problems after rebooting during upgrade. On the node I just rebooted, 2/4 OSDs won't restart. Similar logs from both. Logs from one below. Neither OSDs have compression enabled, although there is a compression-related error in the log. Both are replicated x3. One has data on HDD & separate WAL/DB on NVMe partition, the other is everything on NVMe partition only. Feeling kinda nervous here - advice welcomed!! Thx, Chris 2020-05-20T13:14:00.837+0100 7f2e0d273700 3 rocksdb: [table/block_based_table_reader.cc:1117] Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch: expected 0, got 3423870535 in db/000304.sst offset 18446744073709551615 size 18446744073709551615 2020-05-20T13:14:00.841+0100 7f2e1957ee00 4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:db/MANIFEST-000312 succeeded,manifest_file_number is 312, next_file_number is 314, last_sequence is 22320582, log_number is 309,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2020-05-20T13:14:00.841+0100 7f2e1957ee00 4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 309 2020-05-20T13:14:00.841+0100 7f2e1957ee00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1589976840843199, "job": 1, "event": "recovery_started", "log_files": [313]} 2020-05-20T13:14:00.841+0100 7f2e1957ee00 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #313 mode 0 2020-05-20T13:14:00.937+0100 7f2e1957ee00 3 rocksdb: [db/db_impl_open.cc:518] db.wal/000313.log: dropping 9044 bytes; Corruption: error in middle of record 2020-05-20T13:14:00.937+0100 7f2e1957ee00 3 rocksdb: [db/db_impl_open.cc:518] db.wal/000313.log: dropping 86 bytes; Corruption: missing start of fragmented record(2) 2020-05-20T13:14:00.937+0100 7f2e1957ee00 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work 2020-05-20T13:14:00.937+0100 7f2e1957ee00 4 rocksdb: [db/db_impl.cc:563] Shutdown complete 2020-05-20T13:14:00.937+0100 7f2e1957ee00 -1 rocksdb: Corruption: error in middle of record 2020-05-20T13:14:00.937+0100 7f2e1957ee00 -1 bluestore(/var/lib/ceph/osd/ceph-9) _open_db erroring opening db: 2020-05-20T13:14:00.937+0100 7f2e1957ee00 1 bluefs umount 2020-05-20T13:14:00.937+0100 7f2e1957ee00 1 fbmap_alloc 0x55daf2b3a900 shutdown 2020-05-20T13:14:00.937+0100 7f2e1957ee00 1 bdev(0x55daf3838700 /var/lib/ceph/osd/ceph-9/block) close 2020-05-20T13:14:01.093+0100 7f2e1957ee00 1 bdev(0x55daf3838000 /var/lib/ceph/osd/ceph-9/block) close 2020-05-20T13:14:01.341+0100 7f2e1957ee00 -1 osd.9 0 OSD:init: unable to mount object store 2020-05-20T13:14:01.341+0100 7f2e1957ee00 -1 ESC[0;31m ** ERROR: osd init failed: (5) Input/output errorESC[0m

4 12

Re: question on ceph node count
by tim taler 23 May '20

23 May '20

Thnx Anthony on a 5 node cluster, replica 5, 5 MON, failure domain is host: > > any two nodes might fail without leaving the cluster > > unresponsive? > > No. Assuming again that your failure domain is “host”, some PGs will have two copies on these two nodes, so they will be undersized until redundancy is restored. > ouch, right, but aren't PGs checksummed so from the remaining PG (given the checksum would be right) two new copies could be created?

1 0

question on ceph node count
by tim taler 22 May '20

22 May '20

Hi all, stumbling over a new ceph cluster setup, I got a basic question regarding the behaviour of ceph. The cluster I found runs 4 hardware nodes as hyper convergent instances - 3 nodes running MON and several OSD instances while one node only runs several OSD. At the same time, all nodes serve as KVM hosts, running several VMs. Replication is set to three. 1.) Am I right that in this setup only one node may fail without leaving the cluster unresponsive? (i.e. rejecting further IO)? if two nodes fail, the cluster will stop serving IO (but probably won't lose data if at least one of the two failing nodes can recover) 2.) Am I further right, that this setup with four nodes might provide a little better IO to the VMs than if it would run on only three nodes (all of whom run OSD and OSD and KVM), but 3.) in a setup with 5 nodes, the IO performance delivered to the VMs would be even better and if all of the 5 nodes would run MON and OSDs and KVM, any two nodes might fail without leaving the cluster unresponsive? (sure one would have to move/restart the VMs on the failing nodes ... just from the underlying data integrity point of view) Sorry if it sounds dumb, seems sort of obvious to me, but I have to nail this cristal clear. Short, comprehensive answer would be most appreciated! Best regards!

1 0

S3 key prefixes and performance impact on Ceph?
by malinsk＠databento.com 22 May '20

22 May '20

I've just set up a Ceph cluster and I'm accessing it via object gateway with S3 API. One thing I don't see documented anywhere is - how does Ceph performance scale with S3 key prefixes? In AWS S3, performance scales linearly with key prefix (see: https://docs.aws.amazon.com/AmazonS3/latest/dev/optimizing-performance.html) I see the keys as a nested hash table or nodes of a prefix tree, where each prefix is stored in closer proximity at a hardware level - you want to spread reads evenly over prefixes to avoid parallel I/O being concentrated on the same hot spots. So for example if my access pattern regularly involves scanning data through multiple dates for a single city, this key structure will be more effective: `yyyymmdd/city/data.csv`. Whereas if my access pattern involves scanning through different cities on a single date, `city/yyyymmdd/data.csv` would be more effective. How about Ceph? Does naming convention of the key prefixes have an effect on Ceph's object gateway performance or does it treat the full object "paths" as a completely flat namespace?

3 2

Bluestore config recommendations
by Adrian Nicolae 22 May '20

22 May '20

Hi, I'm planning to install a new Ceph cluster (Nautilus) using 8+3 EC, SATA-only storage. We want to store here only big files (from 40-50MB to 200-300GB each). The write load will be higher than the read load . I was thinking at the following Bluestore config to reduce the load on the drives as much as possible and also to have a good storage efficiency : - stripe_unit for Jerasure EC profile = 256K - bluestore_min_alloc_size_hdd = bluefs_alloc_size = 1M - rados obj size = 4M - rgw_max_chunk_size=4M = rgw_obj_stripe_size What do you think ? Is there any gain in using a higher rgw_max_chunk_size and/or rgw_obj_stripe_size (for example 20M instead of 4M) ? What about bluestore_min_alloc_size and bluefs_alloc_size ? Should I lower them to 512K or 256K ? Thanks, Adrian

1 0

Pool full but the user cleaned it up already
by Szabo, Istvan (Agoda) 22 May '20

22 May '20

Hi, I have a health warn regarding pool full: health: HEALTH_WARN 1 pool(s) full This is the pool that is complaining: Ceph df: NAME ID USED %USED MAX AVAIL OBJECTS k8s 8 200GiB 0.22 90.5TiB 51860 Rados df: POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR k8s-infraops 200GiB 51860 0 155580 0 0 0 1256337 126GiB 19012858 944GiB What needs to be cleaned to avoid the pool full? Ceph version luminous 12.2.8. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 9

Re: ceph orch upgrade stuck at the beginning.
by Gencer W. Genç 21 May '20

21 May '20

Hi Sebastian, I did not get your reply via e-mail. I am very sorry for this. I hope you can see this message... I've re-run the upgrade and attached the log. Thanks, Gencer.

1 0

Setting up first cluster on proxmox - a few questions
by CodingSpiderFox 21 May '20

21 May '20

Hello everyone :) When I try to create an OSD, Proxmox UI asks for * Data disk * DB disk * WAL disk What disk will be the limiting factor in terms of storage size for my OSD - the data disk? How large do I need to make the other two? Is there a risk of them running over capacity before the capacity of the OSD is reached? If so, how to avoid this (in terms of usage pattern)? Have a nice day :)

2 1

diskprediction_local prediction granularity
by Vytenis A 21 May '20

21 May '20

Hi list, Looking into diskprediction_local module, and I see that it only predicts a few states: good, warning and bad: ceph/src/pybind/mgr/diskprediction_local/predictor.py: if score > 10: return "Bad" if score > 4: return "Warning" return "Good" The predicted fail date is just a derivative based on that: ceph/src/pybind/mgr/diskprediction_local/module.py if result.lower() == 'good': life_expectancy_day_min = (TIME_WEEK * 6) + TIME_DAYS life_expectancy_day_max = None elif result.lower() == 'warning': life_expectancy_day_min = (TIME_WEEK * 2) life_expectancy_day_max = (TIME_WEEK * 6) elif result.lower() == 'bad': life_expectancy_day_min = 0 life_expectancy_day_max = (TIME_WEEK * 2) - TIME_DAYS Is it possible to get any finer prediction date? -- Vytenis

2 2

Large omap
by Szabo, Istvan (Agoda) 21 May '20

21 May '20

Hi, I have in one of my cluster a large omap object under luminous 12.2.8. HEALTH_WARN 1 large omap objects LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'default.rgw.log' Search the cluster log for 'Large omap object found' for more details. In my setup the ha-proxy load balancer is running on 3 osd nodes which is monitor and manager also. When I look for this large omap object, this is the one: for i in `ceph pg ls-by-pool default.rgw.log | tail -n +2 | awk '{print $1}'`; do echo -n "$i: "; ceph pg $i query |grep num_large_omap_objects | head -1 | awk '{print $2}'; done | grep ": 1" 4.d: 1 I found only this way to reduce the size: radosgw-admin usage trim --end-date=2019-05-01 --yes-i-really-mean-it And I ran this many times like months by months. However when this is running the RGW became completely unreachable, the loadbalancer started to flapping and users started to complain because they can't do anything. Is there any other way to fix it, or any suggestion why this issue happens? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 3

Possible bug in op path?
by Robert LeBlanc 20 May '20

20 May '20

We upgraded our Jewel cluster to Nautilus a few months ago and I've noticed that op behavior has changed. This is an HDD cluster (NVMe journals and NVMe CephFS metadata pool) with about 800 OSDs. When on Jewel and running WPQ with the high cut-off, it was rock solid. When we had recoveries going on it barely dented the client ops and when the client ops on the cluster went down the backfills would run as fast as the cluster could go. I could have max_backfills set to 10 and the cluster performed admirably. After upgrading to Nautilus the cluster struggles with any kind of recovery and if there is any significant client write load the cluster can get into a death spiral. Even heavy client write bandwidth (3-4 GB/s) can cause the heartbeat checks to raise, blocked IO and even OSDs becoming unresponsive. As the person who wrote the WPQ code initially, I know that it was fair and proportional to the op priority and in Jewel it worked. It's not working in Nautilus. I've tweaked a lot of things trying to troubleshoot the issue and setting the recovery priority to 1 or zero barely makes any difference. My best estimation is that the op priority is getting lost before reaching the WPQ scheduler and is thus not prioritizing and dispatching ops correctly. It's almost as if all ops are being treated the same and there is no priority at all. Unfortunately, I do not have the time to set up the dev/testing environment to track this down and we will be moving away from Ceph. But I really like Ceph and want to see it succeed. I strongly suggest that someone look into this because I think it will resolve a lot of problems people have had on the mailing list. I'm not sure if a bug was introduced with the other queues that touches more of the op path or if something in the op path restructuring that changed how things work (I know that was being discussed around the time that Jewel was released). But my guess is that it is somewhere between the op being created and being received into the queue. I really hope that this helps in the search for this regression. I spent a lot of time studying the issue to come up with WPQ and saw it work great when I switched this cluster from PRIO to WPQ. I've also spent countless hours studying how it's changed in Nautilus. Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 3

Reweighting OSD while down results in undersized+degraded PGs
by Andras Pataki 20 May '20

20 May '20

In a recent cluster reorganization, we ended up with a lot of undersized/degraded PGs and a day of recovery from them, when all we expected was moving some data around. After retracing my steps, I found something odd. If I crush reweight an OSD to 0 while it is down - it results in the PGs of that OSD ending up degraded even after the OSD is restarted. If I do the same reweighting while the OSD is up - data gets moved without any degraded/undersized states. I would not expect this - so I wonder if this is a bug or is somehow intended. This is on ceph Nautilus 14.2.8. Below are the details. Andras First the case that works as I would expect: # Healthy cluster ... [root@xorphosd00 ~]# ceph -s cluster: id: 86d8a1b9-761b-4099-a960-6a303b951236 health: HEALTH_WARN noout,nobackfill,noscrub,nodeep-scrub flag(s) set services: mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d) mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00 mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby osd: 270 osds: 270 up (since 2m), 270 in (since 4h) flags noout,nobackfill,noscrub,nodeep-scrub data: pools: 4 pools, 5312 pgs objects: 75.87M objects, 287 TiB usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 5312 active+clean # Reweight an OSD to 0 [root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0 reweighted item id 0 name 'osd.0' to 0 in crush map # Crush map changes - data movement is set up, no degraded PGs: [root@xorphosd00 ~]# ceph -s cluster: id: 86d8a1b9-761b-4099-a960-6a303b951236 health: HEALTH_WARN noout,nobackfill,noscrub,nodeep-scrub flag(s) set services: mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d) mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00 mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby osd: 270 osds: 270 up (since 10m), 270 in (since 5h); 175 remapped pgs flags noout,nobackfill,noscrub,nodeep-scrub data: pools: 4 pools, 5312 pgs objects: 75.87M objects, 287 TiB usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 2562045/232996662 objects misplaced (1.100%) 5137 active+clean 172 active+remapped+backfilling 3 active+remapped+backfill_wait # Reweight it back to the original weight [root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0 # Cluster goes back to clean reweighted item id 0 name 'osd.0' to 8 in crush map [root@xorphosd00 ~]# ceph -s cluster: id: 86d8a1b9-761b-4099-a960-6a303b951236 health: HEALTH_WARN noout,nobackfill,noscrub,nodeep-scrub flag(s) set services: mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d) mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00 mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby osd: 270 osds: 270 up (since 11m), 270 in (since 5h) flags noout,nobackfill,noscrub,nodeep-scrub data: pools: 4 pools, 5312 pgs objects: 75.87M objects, 287 TiB usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 5312 active+clean # # Now the problematic case # # Stop an OSD [root@xorphosd00 ~]# systemctl stop ceph-osd@0 # We get degraded PGs - as expected [root@xorphosd00 ~]# ceph -s cluster: id: 86d8a1b9-761b-4099-a960-6a303b951236 health: HEALTH_WARN noout,nobackfill,noscrub,nodeep-scrub flag(s) set 1 osds down Degraded data redundancy: 873964/232996662 objects degraded (0.375%), 82 pgs degraded services: mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d) mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00 mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby osd: 270 osds: 269 up (since 16s), 270 in (since 5h) flags noout,nobackfill,noscrub,nodeep-scrub data: pools: 4 pools, 5312 pgs objects: 75.87M objects, 287 TiB usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 873964/232996662 objects degraded (0.375%) 5230 active+clean 82 active+undersized+degraded # Reweight the OSD to 0: [root@xorphosd00 ~]# ceph osd crush reweight osd.0 0.0 # Still degraded - as expected reweighted item id 0 name 'osd.0' to 0 in crush map [root@xorphosd00 ~]# ceph -s cluster: id: 86d8a1b9-761b-4099-a960-6a303b951236 health: HEALTH_WARN noout,nobackfill,noscrub,nodeep-scrub flag(s) set 1 osds down Degraded data redundancy: 873964/232996662 objects degraded (0.375%), 82 pgs degraded services: mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d) mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00 mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby osd: 270 osds: 269 up (since 59s), 270 in (since 5h); 175 remapped pgs flags noout,nobackfill,noscrub,nodeep-scrub data: pools: 4 pools, 5312 pgs objects: 75.87M objects, 287 TiB usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 873964/232996662 objects degraded (0.375%) 1688081/232996662 objects misplaced (0.725%) 5137 active+clean 93 active+remapped+backfilling 82 active+undersized+degraded+remapped+backfilling # Restarting the OSD [root@xorphosd00 ~]# systemctl start ceph-osd@0 # And the PGs still stay degraded - THIS IS UNEXPECTED!!! [root@xorphosd00 ~]# ceph -s cluster: id: 86d8a1b9-761b-4099-a960-6a303b951236 health: HEALTH_WARN noout,nobackfill,noscrub,nodeep-scrub flag(s) set Degraded data redundancy: 873964/232996662 objects degraded (0.375%), 82 pgs degraded services: mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d) mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00 mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby osd: 270 osds: 270 up (since 14s), 270 in (since 5h); 175 remapped pgs flags noout,nobackfill,noscrub,nodeep-scrub data: pools: 4 pools, 5312 pgs objects: 75.87M objects, 287 TiB usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 873964/232996662 objects degraded (0.375%) 1688081/232996662 objects misplaced (0.725%) 5137 active+clean 93 active+remapped+backfilling 82 active+undersized+degraded+remapped+backfilling # Now for something even more odd - reweight the OSD back to its original weigh # and all the data gets magically FOUND again on that OSD!!! [root@xorphosd00 ~]# ceph osd crush reweight osd.0 8.0 reweighted item id 0 name 'osd.0' to 8 in crush map [root@xorphosd00 ~]# ceph -s cluster: id: 86d8a1b9-761b-4099-a960-6a303b951236 health: HEALTH_WARN noout,nobackfill,noscrub,nodeep-scrub flag(s) set services: mon: 3 daemons, quorum xorphmon00,xorphmon01,xorphmon02 (age 11d) mgr: xorphmon01(active, since 6w), standbys: xorphmon02, xorphmon00 mds: cephfs:1 {0=xorphmon02=up:active} 1 up:standby osd: 270 osds: 270 up (since 51s), 270 in (since 5h) flags noout,nobackfill,noscrub,nodeep-scrub data: pools: 4 pools, 5312 pgs objects: 75.87M objects, 287 TiB usage: 864 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 5312 active+clean

3 6

OSD crashes regularely
by Thomas 20 May '20

20 May '20

Hello, I have a pool of +300 OSDs that are identical model (Seagate model: ST1800MM0129 size: 1.64 TiB). Only 1 OSD crashes regularely, however I cannot identify a root cause. Based on the output of smartctl the disk is ok. # smartctl -a -d megaraid,1 /dev/sda [47/1833] smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.3.18-2-pve] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: LENOVO-X Product: ST1800MM0129 Revision: L2B6 Compliance: SPC-4 User Capacity: 1,800,360,124,416 bytes [1.80 TB] Logical block size: 512 bytes Physical block size: 4096 bytes LU is fully provisioned Rotation Rate: 10500 rpm Form Factor: 2.5 inches Logical Unit id: 0x5000c500bb7822cf Serial number: WBN0QHX80000E852944J Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Mon May 18 09:19:41 2020 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled === START OF READ SMART DATA SECTION === SMART Health Status: HARDWARE IMPENDING FAILURE GENERAL HARD DRIVE FAILURE [asc=5d, ascq=10] [22/1833] Grown defects during certification <not available> Total blocks reassigned during format <not available> Total new blocks reassigned = 68 Power on minutes since format <not available> Current Drive Temperature: 33 C Drive Trip Temperature: 65 C Manufactured in week 31 of year 2018 Specified cycle count over device lifetime: 10000 Accumulated start-stop cycles: 21 Specified load-unload count over device lifetime: 300000 Accumulated load-unload cycles: 709 Elements in grown defect list: 18 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 3278853896 1 0 3278853897 32 83933.567 19 write: 0 0 0 0 0 24093.894 0 verify: 3080361880 0 0 3080361880 0 12630.494 0 Non-medium error count: 244 SMART Self-test log Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ] Description number (hours) # 1 Background short Completed - 3761 - [- - -] # 2 Background short Completed - 3737 - [- - -] # 3 Background short Completed - 3713 - [- - -] # 4 Background short Completed - 3689 - [- - -] # 5 Background short Completed - 3665 - [- - -] # 6 Background short Completed - 3641 - [- - -] # 7 Background short Completed - 3617 - [- - -] # 8 Background short Completed - 3593 - [- - -] # 9 Background long Completed - 3569 - [- - -] #10 Background short Completed - 3545 - [- - -] #11 Background short Completed - 3521 - [- - -] #12 Background short Completed - 3497 - [- - -] #13 Background short Completed - 3473 - [- - -] #14 Background short Completed - 3449 - [- - -] #15 Background short Completed - 3425 - [- - -] #16 Background short Completed - 3401 - [- - -] #17 Background short Completed - 3377 - [- - -] #18 Background short Completed - 3353 - [- - -] #19 Background short Completed - 3329 - [- - -] #20 Background short Completed - 3305 - [- - -] Long (extended) Self-test duration: 9459 seconds [157.7 minutes] I have attached the log of the affected OSD. THX Thomas Ich habe 1 zu dieser E-Mail gehörende Datei hochgeladen: ceph-osd.92.log.1.gz <https://we.tl/t-7DzNCDP3iZ>(578 KB)WeTransferhttps://we.tl/t-7DzNCDP3iZ Mozilla Thunderbird <https://www.thunderbird.net> macht es einfach, große Dateien über E-Mails zu teilen.

2 1

Aging in S3 or Moving old data to slow OSDs
by Khodayar Doustar 20 May '20

20 May '20

Hi, I'm using Nautilus and I'm using the whole cluster mainly for a single bucket in RadosGW. There is a lot of data in this bucket (Petabyte scale) and I don't want to waste all of SSD on it. Is there anyway to automatically set some aging threshold for this data and e.g. move any data older than a month to HDD OSDs? Does anyone has experience with this: Pool Placement and Storage Classes: https://docs.ceph.com/docs/master/radosgw/placement/ But something automatic would be much better for me in this case. Any help would be appreciated. Thanks a lot, Khodayar

3 5

15.2.2 Upgrade - Corruption: error in middle of record
by Ashley Merrick 20 May '20

20 May '20

I just upgraded a cephadm cluster from 15.2.1 to 15.2.2. Everything went fine on the upgrade, however after restarting one node that has 3 OSD's for ecmeta two of the 3 ODS's now wont boot with the following error: May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+0000 7fbcc46f7ec0 4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:db/MANIFEST-002768 succeeded,manifest_file_number is 2768, next_file_number is 2775, last_sequence is 188026749, log_number is 2767,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+0000 7fbcc46f7ec0 4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 2767 May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+0000 7fbcc46f7ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1589963382599157, "job": 1, "event": "recovery_started", "log_files": [2769]} May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+0000 7fbcc46f7ec0 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #2769 mode 0 May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+0000 7fbcc46f7ec0 3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 537526 bytes; Corruption: error in middle of record May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+0000 7fbcc46f7ec0 3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 bytes; Corruption: missing start of fragmented record(1) May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+0000 7fbcc46f7ec0 3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 bytes; Corruption: missing start of fragmented record(1) May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+0000 7fbcc46f7ec0 3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 bytes; Corruption: missing start of fragmented record(1) May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+0000 7fbcc46f7ec0 3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 bytes; Corruption: missing start of fragmented record(1) May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+0000 7fbcc46f7ec0 3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 bytes; Corruption: missing start of fragmented record(1) May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+0000 7fbcc46f7ec0 3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 bytes; Corruption: missing start of fragmented record(1) May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+0000 7fbcc46f7ec0 3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 23263 bytes; Corruption: missing start of fragmented record(2) May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+0000 7fbcc46f7ec0 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+0000 7fbcc46f7ec0 4 rocksdb: [db/db_impl.cc:563] Shutdown complete May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+0000 7fbcc46f7ec0 -1 rocksdb: Corruption: error in middle of record May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+0000 7fbcc46f7ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db erroring opening db: May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+0000 7fbcc46f7ec0 1 bdev(0x558a28dd0700 /var/lib/ceph/osd/ceph-0/block) close May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.870+0000 7fbcc46f7ec0 1 bdev(0x558a28dd0000 /var/lib/ceph/osd/ceph-0/block) close May 20 08:29:43 sn-m01 bash[6833]: debug 2020-05-20T08:29:43.118+0000 7fbcc46f7ec0 -1 osd.0 0 OSD:init: unable to mount object store May 20 08:29:43 sn-m01 bash[6833]: debug 2020-05-20T08:29:43.118+0000 7fbcc46f7ec0 -1 ** ERROR: osd init failed: (5) Input/output error Have I hit a bug, or is there something I can do to try and fix these OSD's? Thanks

3 11

[ceph-users][ceph-dev] Upgrade Luminous to Nautilus 14.2.8 mon service crash
by Amit Ghadge 20 May '20

20 May '20

Hi All, While we enable *ceph mon enable-msgr2 *after gateway service upgrade, the one of the mon service getting crash and never come back, it shows, /usr/bin/ceph-mon -f --cluster ceph --id mon01 --setuser ceph --setgroup ceph --debug_monc 20 --debug_ms 5 global_init: error reading config file. Thanks AmitG

1 0

[ceph][nautilus] prformances with db/wal on nvme
by Ignazio Cassano 20 May '20

20 May '20

Hello All, We have 6 servers. Configuration for each server: 1 ssd for mon (only on three servers) 1 ssd 1.9 TB for db/wal 1 nvme 1.6 TB for db/wal 10 SAS hdd 3.6 TB for osd We decided to create a pool of 30 osd (5x6) with db/wal on ssd and a pool of 30 (5x6) osd with db/wal on nvme. So we create a vm on the pool with db.wal on ssd and a vm on the pool with db/wal on nvme. Fio performances are almost the same on both . What do you think about it ? I expect better performance on pool with db/wal on pci express nvme PS SSD are under SAS controller NVMe pcie Samsung PM1725B Best Regards Ignazio

2 4

Re: total ceph outage again, need help
by Frank Schilder 20 May '20

20 May '20

Looks like the immediate danger has passed by: [root@gnosis ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN nodown,noout flag(s) set 735 slow ops, oldest one blocked for 3573 sec, daemons [mon.ceph-02,mon.ceph-03] have slow ops. services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 288 osds: 268 up, 268 in flags nodown,noout data: pools: 10 pools, 2545 pgs objects: 86.76 M objects, 218 TiB usage: 277 TiB used, 1.5 PiB / 1.8 PiB avail pgs: 2537 active+clean 8 active+clean+scrubbing+deep io: client: 34 MiB/s rd, 24 MiB/s wr, 954 op/s rd, 1.01 kop/s wr I will prepare a new case with info we have collected so far. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Amit Ghadge <amitg.b14(a)gmail.com> Sent: 20 May 2020 09:44 To: Frank Schilder Subject: Re: [ceph-users] total ceph outage again, need help look like ceph-01 shows in starting, so I think that why command not executed and you also try to disable to scrubbing for temporary. On Wed, May 20, 2020 at 12:57 PM Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> wrote: Dear cephers, I'm sitting with a major ceph outage again. The mon/mgr hosts suffer from a packet storm of ceph traffic between ceph fs clients and the mons. No idea why this is happening. Main problem is, that I can't get through to the cluster. Admin commands hang forever: [root@gnosis ~]# ceph osd set nodown However, "ceph status" returns and shows me that I need to do something: [root@gnosis ~]# ceph status cluster: id: --- health: HEALTH_WARN 2 MDSs report slow metadata IOs 1 MDSs report slow requests 8 osds down services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active, starting), standbys: ceph-02, ceph-03 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 288 osds: 208 up, 216 in; 153 remapped pgs data: pools: 10 pools, 2545 pgs objects: 86.71 M objects, 218 TiB usage: 277 TiB used, 1.5 PiB / 1.8 PiB avail pgs: 2542 active+clean 3 active+clean+scrubbing+deep io: client: 152 MiB/s rd, 72 MiB/s wr, 854 op/s rd, 796 op/s wr Is there any way to get admin commands to the mons with higher priority? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

total ceph outage again, need help
by Frank Schilder 20 May '20

20 May '20

Dear cephers, I'm sitting with a major ceph outage again. The mon/mgr hosts suffer from a packet storm of ceph traffic between ceph fs clients and the mons. No idea why this is happening. Main problem is, that I can't get through to the cluster. Admin commands hang forever: [root@gnosis ~]# ceph osd set nodown However, "ceph status" returns and shows me that I need to do something: [root@gnosis ~]# ceph status cluster: id: --- health: HEALTH_WARN 2 MDSs report slow metadata IOs 1 MDSs report slow requests 8 osds down services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active, starting), standbys: ceph-02, ceph-03 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 288 osds: 208 up, 216 in; 153 remapped pgs data: pools: 10 pools, 2545 pgs objects: 86.71 M objects, 218 TiB usage: 277 TiB used, 1.5 PiB / 1.8 PiB avail pgs: 2542 active+clean 3 active+clean+scrubbing+deep io: client: 152 MiB/s rd, 72 MiB/s wr, 854 op/s rd, 796 op/s wr Is there any way to get admin commands to the mons with higher priority? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

What is a pgmap?
by Bryan Henderson 20 May '20

20 May '20

I'm surprised I couldn't find this explained anywhere (I did look), but ... What is the pgmap and why does it get updated every few seconds on a tiny cluster that's mostly idle? I do know what a placement group (PG) is and that when documentation talks about placement group maps, it is talking about something else -- mapping of PGs to OSDs by CRUSH and OSD maps. -- Bryan Henderson San Jose, California

4 6

Ceph Dashboard suddenly gone and primary remote is not accessible [CEPHADM_HOST_CHECK_FAILED, CEPHADM_REFRESH_FAILED]
by Gencer W. Genç 19 May '20

19 May '20

Hi, I was browsing dashboard today. Then suddently it stopped working and i got 502 errors. I checked via root login and see thet ceph health is down to WARN. I can access all rdb devices and CephFS. They work. All OSDs in server-1 is up. health: HEALTH_WARN 1 hosts fail cephadm check failed to probe daemons or devices I even restarted server-1. No luck. I'm on server-1. cephadm complains it cannot access to server-1. In basic term, server-1 cannot access server-1 (192.168.0.1) server-1: 192.168.0.1 server-2: 192.168.0.3 $ ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1 > Success. cephadm says server-1 is not reachable. Why cephadm/ceph complain about this and what happend to my dashboard? I also redirected logs to file. I'm observing for any weird log input but unfortunately nothing helpful. Where should I look for? What am i missing? So, when I ran this command: $ ceph health detail Output was: HEALTH_WARN 1 hosts fail cephadm check; failed to probe daemons or devices [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check host server-1 failed check: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: > ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1 [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host server-1 scrape failed: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: > ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1 host server-1 ceph-volume inventory failed: Failed to connect to server-1 (server-1). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: > ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@server-1 Thanks, Gencer.

2 2

Clarification of documentation
by CodingSpiderFox 19 May '20

19 May '20

In the docs: https://docs.ceph.com/docs/master/radosgw/multisite/ - in the section Requirements and Assumptions There is this warning hint: "Running a single Ceph storage cluster is NOT recommended unless you have low latency WAN connections." What exactly does "single Ceph storage cluster" mean? * Just having one site? * Do I need to run multiple independent clusters across a number of n sites before I can do multi-site - so it would not be possible to start with one site and add further sites step by step? And what exactly happens with higher latency WAN connections?

6 10

Re: Zeroing out rbd image or volume
by Jason Dillaman 19 May '20

19 May '20

I would also like to add that the OSDs can (and will) use redirect on write techniques (not to mention the physical device hardware as well). Therefore, your zeroing of the device might just cause the OSDs to allocate new extents of zeros while the old extents remain intact (albeit unreferenced and available for future writes). The correct solution would be to layer LUKS/dm-crypt on top of the RBD device if you need a strong security guarantee about a specific image, or use encrypted OSDs if the concern is about the loss of the OSD physical device. On Tue, May 12, 2020 at 6:58 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > dd if=/dev/zero of=rbd ???? :) but if you have encrypted osd's, what > would be the use of this? > > > > -----Original Message----- > From: huxiaoyu(a)horebdata.cn [mailto:huxiaoyu@horebdata.cn] > Sent: 12 May 2020 12:55 > To: ceph-users > Subject: [ceph-users] Zeroing out rbd image or volume > > Hi， Ceph folks, > > Is there a rbd command, or any other way, to zero out rbd images or > volume? I would like to write all zero data to an rbd image/volume > before remove it. > > Any comments would be appreciated. > > best regards, > > samuel > Horebdata AG > Switzerland > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Jason

5 4

Re: v15.2.2 Octopus released
by David Orman 19 May '20

19 May '20

The updated images have not been pushed to Dockerhub yet. I ran into the same problem yesterday trying to update. Hopefully updated images will be pushed on release (at the same time as the tarball release/prior to announcement) moving forward in order to avoid this issue. See here for latest tags: https://hub.docker.com/r/ceph/ceph/tags

1 0

Resources for multisite deployment
by Coding SpiderFox 19 May '20

19 May '20

Hello everyone, I'd like to setup a multisite ceph cluster. Are there any sample setups that you can recommend studying? I want to achieve fault tolerance but also I want to avoid split brain scenarios. I'm not that familiar with systems like ceph, so I would consider myself as a beginner. Thank you and have a great day

1 0

v15.2.2 Octopus released
by Abhishek Lekshmanan 19 May '20

19 May '20

We're happy to announce the second bugfix release of Ceph Octopus stable release series, we recommend that all Octopus users upgrade. This release has a range of fixes across all components and a security fix. Notable Changes --------------- * CVE-2020-10736: Fixed an authorization bypass in mons & mgrs (Olle SegerDahl, Josh Durgin) For the complete changelog please refer to the full release blog at https://ceph.io/releases/v15-2-2-octopus-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.2.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 0c857e985a29d90501a285f242ea9c008df49eb8 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

2 1

nfs migrate to rgw
by Zhenshi Zhou 19 May '20

19 May '20

Hi all, We have several nfs servers providing file storage. There is a nginx in front of nfs servers in order to serve the clients. The files are mostly small files and nearly about 30TB in total. I'm gonna use ceph rgw as the storage. I wanna know if it's appropriate to do so. The data migrating from nfs to rgw is a huge job. Besides I'm not sure whether ceph rgw is suitable in this scenario or not. Thanks

3 5

Re: Dealing with non existing crush-root= after reclassify on ec pools
by Dan 18 May '20

18 May '20

I think I did a bad job explaining my issue: I have a fairly old cluster which had a crush map with two trees, one for hdds and one for ssd, like root hdd {..} and root ssd {...} now with the newer class based rules I used crushtool —reclassify to merge those two trees into root default {...} So I already downloaded, edited and Reuploaded the crush map, which resulted in a very minor data movement, which crushtool —compare predicted. One of my pools is an ec pool with an ec profile with crush-root=hdd. I can not, I think, change the ec-profile of an existing pool. But since the pool runs on that profile, with the now non existing crush-root=hdd, I am wondering if I can expect to run into trouble down the line or does the cluster use some internal id, and the string displayed only matters on creation. Basically am I safe or am I hosed? On Mon 18. May 2020 at 19:05, Eric Smith <Eric.Smith(a)vecima.com> wrote: > You'll probably have to decompile, hand edit, recompile, and reset the > crush map pointing at the expected root. The EC profile is only used during > pool creation and will not change the crush map if you change the EC > profile. I think you can expect some data movement if you change the root > but either way I would test it in a lab if you have one available. > > -----Original Message----- > From: Dan <dan34567(a)gmail.com> On Behalf Of Dan > Sent: Monday, May 18, 2020 9:14 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Dealing with non existing crush-root= after > reclassify on ec pools > > I have reclassified a CRUSH map, using the crushtool to a class based > ruleset. > I still have an ec pool with an older ec profile with a new non existing > crush-root=hdd > > I already switched the pool’s ruleset over to a newer rule with a newer > ec-profile with a correct crush-root But pool ls detail still shows: > > > pool 9 'data' erasure profile jerasure-3-1 size 4 min_size 3 ….. > > Jerasure-3-1 being the old profile with non existing crush-root > > So what do I do now? Switching over the pool ruleset does not change the > ec-profile, can I switch the ec-profile over? > What can I expect having a pool with a ec-profile with a non existing > crush-root key? > > Please advise. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io >

2 1

feature mask: why not use HAVE_FEATURE macro in Connection::has_feature()?
by Xinying Song 18 May '20

18 May '20

Hi, everyone: Why don't we use HAVE_FEATURE macro in Connection::has_feature()? Do the features in a Connection not need to care about incarnation things? Missing the macro in Connection is really confusing. Would anyone like to give some explains on this? Thanks!

1 0

Dealing with non existing crush-root= after reclassify on ec pools
by Dan 18 May '20

18 May '20

I have reclassified a CRUSH map, using the crushtool to a class based ruleset. I still have an ec pool with an older ec profile with a new non existing crush-root=hdd I already switched the pool’s ruleset over to a newer rule with a newer ec-profile with a correct crush-root But pool ls detail still shows: pool 9 'data' erasure profile jerasure-3-1 size 4 min_size 3 ….. Jerasure-3-1 being the old profile with non existing crush-root So what do I do now? Switching over the pool ruleset does not change the ec-profile, can I switch the ec-profile over? What can I expect having a pool with a ec-profile with a non existing crush-root key? Please advise.

1 0

how to restart daemons on 15.2 on Debian 10
by Ml Ml 18 May '20

18 May '20

Hello List, how do you restart daemons (mgr, mon, osd) on 15.2.1? It used to be something like: systemctl stop ceph-osd@10 Or: systemctl start ceph-mon@ceph03 however, those command do nothing on my setup. Is this because i use cephadm and that docker stuff? The Logs also seem to be missing. /var/log/ceph/5436dd5d-83d4-4dc8-a93b-60ab5db145df is pretty empty. I feel like i am missing a lot of documentation here? Can anyone point me to my missing parts? Thanks a lot. Cheers, Michael

3 6

Re: Cephadm and rados gateways
by brendan＠tobolaski.com 18 May '20

18 May '20

I'm experiencing a similar thing with Cephadm when attempting to deploy rgw. Following the instructions at https://ceph.readthedocs.io/en/latest/cephadm/install/#deploy-rgws results in the container failing to start up immediately. Digging through the logs, it is because it can't bind to port 80. There is no configuration on my machines that would prevent it from doing so nor is there an existing service bound to 0.0.0.0:80. Checking within the container, I see that the ceph binary does not have cap_net_bind_service which would be required to bind to port 80 from a non-root user. In addition, I see that rgw_frontends is not editable and is set to port 80. This seems like a bug in cephadm / the containers it uses. Either the containers should allow ceph to bind to port 80 or rgw_frontends should remain at the default of 7480 to allow the frontends to be deployed. Is there a way for me to work around this?

2 1

Luminous to Nautilus mon upgrade oddity - failed to decode mgrstat state; luminous dev version? buffer::end_of_buffer
by Thomas Byrne - UKRI STFC 18 May '20

18 May '20

Hi all, We're upgrading a cluster from luminous to nautilus. The monitors and managers are running a non-release version of luminous (12.2.12-642-g5ff3e8e) and we're upgrading them to 14.2.9. We've upgraded one monitor and it's happily in quorum as a peon. However, when a ceph status hits the nautilus mon it has trouble talking to the manager apparently, and it returns a status output with no pg stats and garbage usage numbers. From the mon log: 2020-05-13 10:41:43.121 7fa1e6fdf700 0 mon.ceph-mon5@4(peon) e25 handle_command mon_command({"prefix": "status"} v 0) v1 2020-05-13 10:41:43.121 7fa1e6fdf700 0 log_channel(audit) log [DBG] : from='client.? v1:130.246.x.x:0/3261311028' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2020-05-13 10:41:43.443 7fa1e6fdf700 -1 mon.ceph-mon5(a)4(peon).mgrstat failed to decode mgrstat state; luminous dev version? buffer::end_of_buffer 2020-05-13 10:41:44.397 7fa1e6fdf700 1 mon.ceph-mon5@4(peon) e25 dropping unexpected mon_health( e 0 r 0 ) v1 Is this expected for a luminous to nautilus upgrade or could this be due to the odd luminous version we are running, or something else entirely? Cheers, Tom

2 2

Re: Ceph as a Fileserver for 3D Content Production
by Janne Johansson 18 May '20

18 May '20

Den sön 17 maj 2020 kl 14:33 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: > outs, and you are more likely to shoot yourself in the foot. At least > ask first. Eg this bcache, I am not 100% sure what it is, but if it is > sitting between the osd process and the disk, it could be getting nasty > with a reset/power outage, when ceph thinks data is written to disk, > while it is not. > Bcache is a neat way in linux to stack devices, working basically like the ceph layered pools, but on a per-host basis of course. The main advantage, and why some ceph admins use if instead of putting WAL/DB on the fast device, is that a single bcache device can be placed in front of several other devices, and the kernel will decide which writes are going to use the write cache, so in effect you are sharing the full size of your fast device speed on all OSDs, instead of having to split your NVME into pieces or preset a journal size. If you at any point expect to add more drives to a box, this becomes even more interesting, since knowing beforehand how many preallocations to leave room for is very hard and leaves out potential gains until you get it right. -- May the most significant bit of your life be positive.

2 1

RGW issue with containerized ceph
by Szabo, Istvan (Agoda) 18 May '20

18 May '20

Hi Gents, We are having strange problem with demo ceph container. Any write operation we do from the RGW fail with a NameResolutionFailure Read operation work fine. If we use the ceph UI we can create objects fine though. This is the env varibles we are using to configure it: This is the full compose: version: '3.3' services: ceph: ports: - 8080:8080 # S3 - 5000:5000 # Web UI - 8003:8003 environment: - NETWORK_AUTO_DETECT=4 - CEPH_DAEMON=demo - RGW_NAME=ceph - CEPH_DEMO_UID=owncloud - CEPH_DEMO_ACCESS_KEY=G1EZ5R4K6IJ7XUQKMAED - CEPH_DEMO_SECRET_KEY=cNmUrqpBKjCMzcfqG8fg4Qk07Xkoyau52OmvnSsz - DEBUG=verbose image: ceph/daemon Is there anybody has some idea what is going on? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

OSDs taking too much memory, for pglog
by Harald Staub 18 May '20

18 May '20

Several OSDs of one of our clusters are down currently because RAM usage has increased during the last days. Now it is more than we can handle on some systems. Frequently OSDs get killed by the OOM killer. Looking at "ceph daemon osd.$OSD_ID dump_mempools", it shows that nearly all (about 8.5 GB) is taken by osd_pglog, e.g. "osd_pglog": { "items": 461859, "bytes": 8445595868 }, We tried to reduce it, with "osd memory target" and even with "bluestore cache autotune = false" (together with "bluestore cache size hdd"), but there was no effect at all. I remember the pglog_hardlimit parameter, but that is already set by default with Nautilus I read. I.e. this is on Nautilus, 14.2.8. Is there a way to limit this pglog memory? Cheers Harry

4 6

Need help on cache tier monitoring
by icy chan 18 May '20

18 May '20

Hi, I had configured a cache tier with below parameters: cache_target_dirty_ratio: 0.1 cache_target_dirty_high_ratio: 0.7 cache_target_full_ratio: 0.9 The cache tier did improved the performance much. And I am targeted to keep the cache tier with only 10% of dirty data. Remaining data ( 80% ) are clean and ready for evict. I don't want the cache tier enter high flushing state which impacting the performance. Are there any methods to keep track the dirty ratio of target cache tier? Which I can plan for adding more OSDs if the dirty ratio keep floating around 50~70% Regs, Icy

1 1

Ceph-mgr wont start, cant find rook module
by Jeff Welling 17 May '20

17 May '20

Hello my ceph-using comrades! I've been using ceph for awhile at home but wanted to update to the latest, Octopus. I got it installed on a single node, added a second node and some OSDs, and have been migrating from my original Jewel cluster. When I installed Octopus, I wiped the systems and installed Debian Buster, added the ceph apt repos instead of using the packages in Debian, and installed manually using ceph-vol to create Bluestore OSDs, using the ceph docs as my guide. Now though, one of the two Octopus nodes (the one running ceph-mgr and ceph-mon) are crashing weekly. I haven't been able to look into the cause of the crashes in detail yet as these are hobbyist systems and work has been exceptionally busy lately, but now after the most recent crash, I'm unable to start ceph-mgr and the syslog has ceph-mgr messages complaining of not being able to find the 'rook' module. This is rather confusing because though I'm aware of rook, to my knowledge I've never used it on my systems, and there's no mention of it in my config. I tried applying pending upgrades but that hasn't changed the behavior. I normally wouldn't dare ask for help this early in my adventure but I find myself in a bit of a pinch. By any chance have you hit this before, or know what may be causing it? Ceph is awesome. Keep up the good work, stay safe, and Thank You Kindly in advance! My ceph version root@zim:~# ceph --version ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable) This is my ceph.config [global] fsid = 495d7f30-CCCC-BBBB-AAAA-ddf6ffe063d0 mon initial members = zim.internal.justdev.ca mon host = 192.168.0.11 public network = 192.168.0.0/24 cluster network = 192.168.42.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 1024 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 333 osd pool default pgp num = 333 osd crush chooseleaf type = 1 rbd_default_features = 7 These are the syslogs that show up when trying to restart ceph-mgr May 4 01:35:13 zim ceph-mgr[21602]: 2020-05-04T01:35:13.065-0700 7fcdccaa2f40 -1 mgr[py] Module not found: 'rook' May 4 01:35:13 zim ceph-mgr[21602]: 2020-05-04T01:35:13.065-0700 7fcdccaa2f40 -1 mgr[py] Traceback (most recent call last): May 4 01:35:13 zim ceph-mgr[21602]: File "/usr/share/ceph/mgr/rook/__init__.py", line 2, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from .module import RookOrchestrator May 4 01:35:13 zim ceph-mgr[21602]: File "/usr/share/ceph/mgr/rook/module.py", line 16, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from kubernetes import client, config May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/kubernetes/__init__.py", line 22, in <module> May 4 01:35:13 zim ceph-mgr[21602]: import kubernetes.stream May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/kubernetes/stream/__init__.py", line 15, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from .stream import stream May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/kubernetes/stream/stream.py", line 13, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from . import ws_client May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/kubernetes/stream/ws_client.py", line 19, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from websocket import WebSocket, ABNF, enableTrace May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/websocket/__init__.py", line 22, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from ._abnf import * May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/websocket/_abnf.py", line 34, in <module> May 4 01:35:13 zim ceph-mgr[21602]: import numpy May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/numpy/__init__.py", line 142, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from . import core May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/numpy/core/__init__.py", line 40, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from . import multiarray May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/numpy/core/multiarray.py", line 12, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from . import overrides May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/numpy/core/overrides.py", line 65, in <module> May 4 01:35:13 zim ceph-mgr[21602]: """) May 4 01:35:13 zim ceph-mgr[21602]: RuntimeError: _get_implementing_args method already has a docstring May 4 01:35:13 zim ceph-mgr[21602]: 2020-05-04T01:35:13.069-0700 7fcdccaa2f40 -1 mgr[py] Class not found in module 'rook' May 4 01:35:13 zim ceph-mgr[21602]: 2020-05-04T01:35:13.069-0700 7fcdccaa2f40 -1 mgr[py] Error loading module 'rook': (2) No such file or directory May 4 01:35:13 zim ceph-mgr[21602]: 2020-05-04T01:35:13.673-0700 7fcdccaa2f40 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook

2 2

Ceph as a Fileserver for 3D Content Production
by Moritz Wilhelm 17 May '20

17 May '20

Hi Martin, The Samsung Evo is just my system drive on my test host. This is my server setup 3 Nodes: (original those where render workstations that are not in use right now) Each Node is MON MGM OSD Mainboard: ASRock TRX40 Creator CPU: AMD Ryzen Threadripper 3960X, 24 Cores, 3.8Ghz RAM: 2 x Samsung 32 GB 2 x 8 DDR4 2666 MHz 288-pin DIMM, Unregistered, ECC (64 GB Total) NIC Public: OnBoard Aquantia 107, 10Gbit NIC Ceph: Intel XXV710-DA2, 2x SFP28, 25Gbit System Drive: 2x Samsung SSD 860 PRO 256GB, SATA, ZFS Raid 1 System: Proxmox VE 6.2, Debian Buster, Ceph Nautilus HBA: Broadcom SAS 9305-16i OSDs: 6x Seagate Exos, 16TB, 7.200 rpm, 12Gb SAS Cache: 1x Micron 9300 MAX 3.2TB U.2 NVME __________________________________ EventArt GmbH Kirchenstraße 13, 3243 St. Leonhard am Forst mobile: +43(0)6641951196 mail: office(a)eventart.at<mailto:office@eventart.at> UID: ATU 64417269 http://www.eventart.at II http://www.clamlive.at Von: Martin Verges <martin.verges(a)croit.io> Gesendet: Sonntag, 17. Mai 2020 18:28 An: Moritz Wilhelm <moritz(a)eventart.at> Cc: Marc Roos <M.Roos(a)f1-outsourcing.eu>; ceph-users <ceph-users(a)ceph.io> Betreff: Re: [ceph-users] Re: Ceph as a Fileserver for 3D Content Production Hello Moritz, drop the EVO disk and use a SSD that is working with Ceph. For example just use PM883 / PM983 from the same vendor and you will have a huge performance increase. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io<mailto:martin.verges@croit.io> Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am So., 17. Mai 2020 um 15:12 Uhr schrieb Moritz Wilhelm <moritz(a)eventart.at<mailto:moritz@eventart.at>>: Hi Marc, thank you very much for your feedback, actually that is what I am looking for (Design advices and feedback). I also wanted to get in touch with the community because right now I am on my own with this project with no experience at all. But I also wanted to get into it first and learn the basics and setup a system before bothering other people with it and to be able to even hold a conversation about it. So we are a 3D Content Production Company with around 10-12 Workstations and 6 Render stations. We work with software like Autodesk Maya or Adobe After effects. All our Projects are stored on the NAS and we work directly with those files on the NAS. This is our current System: Synology RS4017xs+ Intel Xeon D-1541 2,1 Ghz, 8 Cores 32GB Ram 16x 8TB WD Red Drives 2 TB SSD Cache 2 x 10 Gb SFP+ Nics About the Bcache I just did tests with it as I found the idea very interesting and also performance was better. If this project is ever getting into real-life production I will stick to the most common setups for sure. Here are some Results of Diskmark on one Computer. (I Had only 5 OSDs per Node up and running at this test because I am still waiting for the new Seagate drives) For Sequential Reads and Writes I sometimes maxed out the 10Gbe host connection so the data is not very useful but the other numbers are quite interesting: For Comparison: Hosts Local NVME Drive (Samsung 970 EVO Pro 1TB): Read: Write: Sequential 1MiB (Q= 8, T= 1): 3507.700 MB/s [ 3345.2 IOPS] < 2389.61 us> Sequential 1MiB (Q= 8, T= 1): 2548.063 MB/s [ 2430.0 IOPS] < 3284.79 us> Sequential 1MiB (Q= 1, T= 1): 2368.308 MB/s [ 2258.6 IOPS] < 442.07 us> Sequential 1MiB (Q= 1, T= 1): 2456.471 MB/s [ 2342.7 IOPS] < 426.10 us> Random 4KiB (Q= 32, T=16): 1555.565 MB/s [ 379776.6 IOPS] < 1312.45 us> Random 4KiB (Q= 32, T=16): 16.580 MB/s [ 4047.9 IOPS] <124267.88 us> Random 4KiB (Q= 1, T= 1): 51.666 MB/s [ 12613.8 IOPS] < 78.82 us> Random 4KiB (Q= 1, T= 1): 108.983 MB/s [ 26607.2 IOPS] < 37.13 us> Current Synology NAS (SMB): Read: Write: Sequential 1MiB (Q= 8, T= 1): 1045.594 MB/s [ 997.2 IOPS] < 7990.33 us> Sequential 1MiB (Q= 8, T= 1): 1101.007 MB/s [ 1050.0 IOPS] < 7588.25 us> Sequential 1MiB (Q= 1, T= 1): 953.709 MB/s [ 909.5 IOPS] < 1098.57 us> Sequential 1MiB (Q= 1, T= 1): 847.847 MB/s [ 808.6 IOPS] < 1235.26 us> Random 4KiB (Q= 32, T=16): 4.198 MB/s [ 1024.9 IOPS] <380158.65 us> Random 4KiB (Q= 32, T=16): 188.827 MB/s [ 46100.3 IOPS] < 11076.14 us> Random 4KiB (Q= 1, T= 1): 2.486 MB/s [ 606.9 IOPS] < 1637.08 us> Random 4KiB (Q= 1, T= 1): 7.177 MB/s [ 1752.2 IOPS] < 570.16 us> Ceph With WAL/DB on NVME total of 5 old SATA HDD OSDs (SMB): Read: Write: Sequential 1MiB (Q= 8, T= 1): 534.050 MB/s [ 509.3 IOPS] < 15628.63 us> Sequential 1MiB (Q= 8, T= 1): 198.420 MB/s [ 189.2 IOPS] < 42020.67 us> Sequential 1MiB (Q= 1, T= 1): 340.580 MB/s [ 324.8 IOPS] < 2921.17 us> Sequential 1MiB (Q= 1, T= 1): 184.329 MB/s [ 175.8 IOPS] < 5603.99 us> Random 4KiB (Q= 32, T=16): 3.172 MB/s [ 774.4 IOPS] <398622.73 us> Random 4KiB (Q= 32, T=16): 8.639 MB/s [ 2109.1 IOPS] <222699.43 us> Random 4KiB (Q= 1, T= 1): 1.907 MB/s [ 465.6 IOPS] < 2139.08 us> Random 4KiB (Q= 1, T= 1): 7.294 MB/s [ 1780.8 IOPS] < 560.91 us> Ceph With Bcache total of 5 old SATA HDD OSDs (SMB): Sequential 1MiB (Q= 8, T= 1): 967.386 MB/s [ 922.6 IOPS] < 8660.59 us> Sequential 1MiB (Q= 8, T= 1): 882.882 MB/s [ 842.0 IOPS] < 9444.23 us> Sequential 1MiB (Q= 1, T= 1): 418.157 MB/s [ 398.8 IOPS] < 2448.00 us> Sequential 1MiB (Q= 1, T= 1): 793.417 MB/s [ 756.7 IOPS] < 1320.37 us> Random 4KiB (Q= 32, T=16): 39.756 MB/s [ 9706.1 IOPS] < 51788.10 us> Random 4KiB (Q= 32, T=16): 128.303 MB/s [ 31324.0 IOPS] < 16298.94 us> Random 4KiB (Q= 1, T= 1): 29.527 MB/s [ 7208.7 IOPS] < 138.36 us> Random 4KiB (Q= 1, T= 1): 32.334 MB/s [ 7894.0 IOPS] < 126.25 us> I will do an Update as soon as I have all the 18 OSDs running. I could also need some advice on how to do proper testing. __________________________________ EventArt GmbH Kirchenstraße 13, 3243 St. Leonhard am Forst mobile: +43(0)6641951196 mail: office(a)eventart.at<mailto:office@eventart.at> UID: ATU 64417269 http://www.eventart.at II http://www.clamlive.at -----Ursprüngliche Nachricht----- Von: Marc Roos <M.Roos(a)f1-outsourcing.eu<mailto:M.Roos@f1-outsourcing.eu>> Gesendet: Sonntag, 17. Mai 2020 14:33 An: ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>>; Moritz Wilhelm <moritz(a)eventart.at<mailto:moritz@eventart.at>> Betreff: RE: [ceph-users] Ceph as a Fileserver for 3D Content Production I was just reading your post, and started wondering why you posted it. I do not see clear question, and you also do not share test results (from your nas vs cephfs smb). So maybe you like some attention in this covid social distancing time? ;) Anyway, I have been 'testing' with ceph for 2,5 years or so. (I am telling you so, you know how to value my info) If you start playing with ceph keep it simple and stick to what is recommended. Do not start looking for unconventional ways making ceph faster, because you are not able to do it better than the developers. You do not know the ins and outs, and you are more likely to shoot yourself in the foot. At least ask first. Eg this bcache, I am not 100% sure what it is, but if it is sitting between the osd process and the disk, it could be getting nasty with a reset/power outage, when ceph thinks data is written to disk, while it is not. I would start by identifying what your minimum performance requirements are, maybe post them, and ask if someone has realized them with a ceph setup. -----Original Message----- Sent: 17 May 2020 13:42 To: ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Ceph as a Fileserver for 3D Content Production Hi, my Name is Moritz and I am working for a 3D production company. Because of the corona virus I have too much time left and also to much unused hardware. That is why I started playing around with Ceph as a fileserver for us. Here I want to share my experience for all those who are interested. To start of here is my actual running test system. I am interested in the thoughts of the community and also on more suggestions on what to try out with my available Hardware. I don’t know how to test it right now because I am a newbie to ceph and our production file server is a super user-friendly but high performance Synology NAS 😉. All I have done so far was running Crystal disk benchmark on 1 Windows machine on the SMB Share. 3 Nodes: (original those where render workstations that are not in use right now) Each Node is MON MGM OSD Mainboard: ASRock TRX40 Creator CPU: AMD Ryzen Threadripper 3960X, 24 Cores, 3.8Ghz RAM: 2 x Samsung 32 GB 2 x 8 DDR4 2666 MHz 288-pin DIMM, Unregistered, ECC (64 GB Total) NIC Public: OnBoard Aquantia 107, 10Gbit NIC Ceph: Intel XXV710-DA2, 2x SFP28, 25Gbit System Drive: 2x Samsung SSD 860 PRO 256GB, SATA, ZFS Raid 1 System: Proxmox VE 6.2, Debian Buster, Ceph Nautilus HBA: Broadcom SAS 9305-16i OSDs: 6x Seagate Exos, 16TB, 7.200 rpm, 12Gb SAS Cache: 1x Micron 9300 MAX 3.2TB U.2 NVME I Played around with setting it up as a WAL/DB Device. Right now I have configured the Micron NVME as a BCache Infront of the 6 Seagate Drives in writeback mode. Because in this configuration BCache takes care of translating random writes to sequential ones for the HDDs I turned the Ceph WAL LOG off. I think Bcache gives more options to tune the System for my use case instead of just putting WAL/DB on the NVME. And also I can easily add cache drives or remove them without touching osds. I set up SMB Shares with the vfs_ceph module. I still have to add CTDB to distribute Samba to all nodes. My Next steps are to keep playing around in tuning the system and testing stability and performance. After that I want to put the Ceph cluster infront of our production NAS. Because our data is not super critical I thought of setting the replicas to 2 and running Rsync overnight to our NAS. That way I can switch to the old NAS at any time and wouldn’t loose more than 1 Day of work which is acceptable for us. This is how I could compare the two solutions side by side with real-life workload. I know that ceph might not be the best solution right now but if I am able to get at least similar performance to our Synology HDD NAS out of it, it would give a super scalable Solution in size and performance to grow with our needs. And who knows what performance improvements we get with ceph in the next 3 years. I am happy to hear your thoughts and ideas. And please I know this might be kind of a crazy setup but I have fun with it and I learned a lot the last few weeks. If my experiment fails I will go back to my original plan: Put FreeNas on two of the Nodes with overnight replication and put the third Node back to his render-friends. 😃 By the way I also have a spare Dell Server: 2x Xeon E5-2630 v3 2,40GHz, 128G Ram. I just don’t have an idea on how to utilize it. Maybe as extra OSD Node or as a separate Samba Server to get the SMB traffic away from the Public Ceph Network. Moritz Wilhelm _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

Re: Ceph as a Fileserver for 3D Content Production
by Martin Verges 17 May '20

17 May '20

Hello Moritz, drop the EVO disk and use a SSD that is working with Ceph. For example just use PM883 / PM983 from the same vendor and you will have a huge performance increase. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am So., 17. Mai 2020 um 15:12 Uhr schrieb Moritz Wilhelm <moritz(a)eventart.at >: > Hi Marc, > > thank you very much for your feedback, actually that is what I am looking > for (Design advices and feedback). I also wanted to get in touch with the > community because right now I am on my own with this project with no > experience at all. But I also wanted to get into it first and learn the > basics and setup a system before bothering other people with it and to be > able to even hold a conversation about it. > > So we are a 3D Content Production Company with around 10-12 Workstations > and 6 Render stations. We work with software like Autodesk Maya or Adobe > After effects. All our Projects are stored on the NAS and we work directly > with those files on the NAS. > > This is our current System: > > Synology RS4017xs+ > Intel Xeon D-1541 2,1 Ghz, 8 Cores > 32GB Ram > 16x 8TB WD Red Drives > 2 TB SSD Cache > 2 x 10 Gb SFP+ Nics > > > About the Bcache I just did tests with it as I found the idea very > interesting and also performance was better. If this project is ever > getting into real-life production I will stick to the most common setups > for sure. > > Here are some Results of Diskmark on one Computer. (I Had only 5 OSDs per > Node up and running at this test because I am still waiting for the new > Seagate drives) For Sequential Reads and Writes I sometimes maxed out the > 10Gbe host connection so the data is not very useful but the other numbers > are quite interesting: > > For Comparison: > > Hosts Local NVME Drive (Samsung 970 EVO Pro 1TB): > Read: > Write: > Sequential 1MiB (Q= 8, T= 1): 3507.700 MB/s [ 3345.2 IOPS] < > 2389.61 us> Sequential 1MiB (Q= 8, T= 1): 2548.063 MB/s [ > 2430.0 IOPS] < 3284.79 us> > Sequential 1MiB (Q= 1, T= 1): 2368.308 MB/s [ 2258.6 IOPS] < > 442.07 us> Sequential 1MiB (Q= 1, T= 1): 2456.471 MB/s [ > 2342.7 IOPS] < 426.10 us> > Random 4KiB (Q= 32, T=16): 1555.565 MB/s [ 379776.6 IOPS] < > 1312.45 us> Random 4KiB (Q= 32, T=16): 16.580 MB/s [ > 4047.9 IOPS] <124267.88 us> > Random 4KiB (Q= 1, T= 1): 51.666 MB/s [ 12613.8 IOPS] < > 78.82 us> Random 4KiB (Q= 1, T= 1): 108.983 MB/s [ > 26607.2 IOPS] < 37.13 us> > > Current Synology NAS (SMB): > Read: > Write: > Sequential 1MiB (Q= 8, T= 1): 1045.594 MB/s [ 997.2 IOPS] < > 7990.33 us> Sequential 1MiB (Q= 8, T= 1): 1101.007 MB/s [ > 1050.0 IOPS] < 7588.25 us> > Sequential 1MiB (Q= 1, T= 1): 953.709 MB/s [ 909.5 IOPS] < > 1098.57 us> Sequential 1MiB (Q= 1, T= 1): 847.847 MB/s [ > 808.6 IOPS] < 1235.26 us> > Random 4KiB (Q= 32, T=16): 4.198 MB/s [ 1024.9 IOPS] > <380158.65 us> Random 4KiB (Q= 32, T=16): 188.827 MB/s [ > 46100.3 IOPS] < 11076.14 us> > Random 4KiB (Q= 1, T= 1): 2.486 MB/s [ 606.9 IOPS] < > 1637.08 us> Random 4KiB (Q= 1, T= 1): 7.177 MB/s > [ 1752.2 IOPS] < 570.16 us> > > Ceph With WAL/DB on NVME total of 5 old SATA HDD OSDs (SMB): > Read: > Write: > Sequential 1MiB (Q= 8, T= 1): 534.050 MB/s [ 509.3 IOPS] < > 15628.63 us> Sequential 1MiB (Q= 8, T= 1): 198.420 MB/s [ > 189.2 IOPS] < 42020.67 us> > Sequential 1MiB (Q= 1, T= 1): 340.580 MB/s [ 324.8 IOPS] < > 2921.17 us> Sequential 1MiB (Q= 1, T= 1): 184.329 MB/s [ > 175.8 IOPS] < 5603.99 us> > Random 4KiB (Q= 32, T=16): 3.172 MB/s [ 774.4 IOPS] > <398622.73 us> Random 4KiB (Q= 32, T=16): 8.639 MB/s [ > 2109.1 IOPS] <222699.43 us> > Random 4KiB (Q= 1, T= 1): 1.907 MB/s [ 465.6 IOPS] < > 2139.08 us> Random 4KiB (Q= 1, T= 1): 7.294 MB/s > [ 1780.8 IOPS] < 560.91 us> > > Ceph With Bcache total of 5 old SATA HDD OSDs (SMB): > Sequential 1MiB (Q= 8, T= 1): 967.386 MB/s [ 922.6 IOPS] < > 8660.59 us> Sequential 1MiB (Q= 8, T= 1): 882.882 MB/s [ > 842.0 IOPS] < 9444.23 us> > Sequential 1MiB (Q= 1, T= 1): 418.157 MB/s [ 398.8 IOPS] < > 2448.00 us> Sequential 1MiB (Q= 1, T= 1): 793.417 MB/s [ > 756.7 IOPS] < 1320.37 us> > Random 4KiB (Q= 32, T=16): 39.756 MB/s [ 9706.1 IOPS] < > 51788.10 us> Random 4KiB (Q= 32, T=16): 128.303 MB/s [ > 31324.0 IOPS] < 16298.94 us> > Random 4KiB (Q= 1, T= 1): 29.527 MB/s [ 7208.7 IOPS] < > 138.36 us> Random 4KiB (Q= 1, T= 1): 32.334 MB/s [ > 7894.0 IOPS] < 126.25 us> > > I will do an Update as soon as I have all the 18 OSDs running. > > I could also need some advice on how to do proper testing. > > > > > > __________________________________ > > EventArt GmbH > Kirchenstraße 13, > 3243 St. Leonhard am Forst > mobile: +43(0)6641951196 > mail: office(a)eventart.at > UID: ATU 64417269 > > http://www.eventart.at II http://www.clamlive.at > > -----Ursprüngliche Nachricht----- > Von: Marc Roos <M.Roos(a)f1-outsourcing.eu> > Gesendet: Sonntag, 17. Mai 2020 14:33 > An: ceph-users <ceph-users(a)ceph.io>; Moritz Wilhelm <moritz(a)eventart.at> > Betreff: RE: [ceph-users] Ceph as a Fileserver for 3D Content Production > > > > I was just reading your post, and started wondering why you posted it. I > do not see clear question, and you also do not share test results (from > your nas vs cephfs smb). So maybe you like some attention in this covid > social distancing time? ;) > > Anyway, I have been 'testing' with ceph for 2,5 years or so. (I am telling > you so, you know how to value my info) If you start playing with ceph keep > it simple and stick to what is recommended. Do not start looking for > unconventional ways making ceph faster, because you are not able to do it > better than the developers. You do not know the ins and outs, and you are > more likely to shoot yourself in the foot. At least ask first. Eg this > bcache, I am not 100% sure what it is, but if it is sitting between the osd > process and the disk, it could be getting nasty with a reset/power outage, > when ceph thinks data is written to disk, while it is not. > > I would start by identifying what your minimum performance requirements > are, maybe post them, and ask if someone has realized them with a ceph > setup. > > > > > -----Original Message----- > Sent: 17 May 2020 13:42 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Ceph as a Fileserver for 3D Content Production > > Hi, > > my Name is Moritz and I am working for a 3D production company. Because of > the corona virus I have too much time left and also to much unused > hardware. That is why I started playing around with Ceph as a fileserver > for us. Here I want to share my experience for all those who are > interested. To start of here is my actual running test system. I am > interested in the thoughts of the community and also on more suggestions on > what to try out with my available Hardware. I don’t know how to test it > right now because I am a newbie to ceph and our production file server is a > super user-friendly but high performance Synology NAS 😉. > All I have done so far was running Crystal disk benchmark on 1 Windows > machine on the SMB Share. > > 3 Nodes: (original those where render workstations that are not in use > right now) > > Each Node is MON MGM OSD > > Mainboard: ASRock TRX40 Creator > CPU: AMD Ryzen Threadripper 3960X, 24 Cores, 3.8Ghz > RAM: 2 x Samsung 32 GB 2 x 8 DDR4 2666 MHz 288-pin DIMM, Unregistered, ECC > (64 GB Total) NIC Public: OnBoard Aquantia 107, 10Gbit NIC Ceph: > Intel XXV710-DA2, 2x SFP28, 25Gbit System Drive: 2x Samsung SSD 860 PRO > 256GB, SATA, ZFS Raid 1 > System: Proxmox VE 6.2, Debian Buster, Ceph Nautilus > HBA: Broadcom SAS 9305-16i > > OSDs: > 6x Seagate Exos, 16TB, 7.200 rpm, 12Gb SAS > > Cache: > 1x Micron 9300 MAX 3.2TB U.2 NVME > > I Played around with setting it up as a WAL/DB Device. Right now I have > configured the Micron NVME as a BCache Infront of the 6 Seagate Drives in > writeback mode. > Because in this configuration BCache takes care of translating random > writes to sequential ones for the HDDs I turned the Ceph WAL LOG off. I > think Bcache gives more options to tune the System for my use case instead > of just putting WAL/DB on the NVME. And also I can easily add cache drives > or remove them without touching osds. > > I set up SMB Shares with the vfs_ceph module. I still have to add CTDB to > distribute Samba to all nodes. > > My Next steps are to keep playing around in tuning the system and testing > stability and performance. After that I want to put the Ceph cluster > infront of our production NAS. Because our data is not super critical I > thought of setting the replicas to 2 and running Rsync overnight to our > NAS. That way I can switch to the old NAS at any time and wouldn’t loose > more than 1 Day of work which is acceptable for us. > This is how I could compare the two solutions side by side with real-life > workload. > > I know that ceph might not be the best solution right now but if I am able > to get at least similar performance to our Synology HDD NAS out of it, it > would give a super scalable Solution in size and performance to grow with > our needs. And who knows what performance improvements we get with ceph in > the next 3 years. > > I am happy to hear your thoughts and ideas. And please I know this might > be kind of a crazy setup but I have fun with it and I learned a lot the > last few weeks. If my experiment fails I will go back to my original > plan: Put FreeNas on two of the Nodes with overnight replication and put > the third Node back to his render-friends. 😃 > > By the way I also have a spare Dell Server: 2x Xeon E5-2630 v3 2,40GHz, > 128G Ram. I just don’t have an idea on how to utilize it. Maybe as extra > OSD Node or as a separate Samba Server to get the SMB traffic away from the > Public Ceph Network. > > > > > Moritz Wilhelm > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Ceph as a Fileserver for 3D Content Production
by Marc Roos 17 May '20

17 May '20

I was just reading your post, and started wondering why you posted it. I do not see clear question, and you also do not share test results (from your nas vs cephfs smb). So maybe you like some attention in this covid social distancing time? ;) Anyway, I have been 'testing' with ceph for 2,5 years or so. (I am telling you so, you know how to value my info) If you start playing with ceph keep it simple and stick to what is recommended. Do not start looking for unconventional ways making ceph faster, because you are not able to do it better than the developers. You do not know the ins and outs, and you are more likely to shoot yourself in the foot. At least ask first. Eg this bcache, I am not 100% sure what it is, but if it is sitting between the osd process and the disk, it could be getting nasty with a reset/power outage, when ceph thinks data is written to disk, while it is not. I would start by identifying what your minimum performance requirements are, maybe post them, and ask if someone has realized them with a ceph setup. -----Original Message----- Sent: 17 May 2020 13:42 To: ceph-users(a)ceph.io Subject: [ceph-users] Ceph as a Fileserver for 3D Content Production Hi, my Name is Moritz and I am working for a 3D production company. Because of the corona virus I have too much time left and also to much unused hardware. That is why I started playing around with Ceph as a fileserver for us. Here I want to share my experience for all those who are interested. To start of here is my actual running test system. I am interested in the thoughts of the community and also on more suggestions on what to try out with my available Hardware. I don’t know how to test it right now because I am a newbie to ceph and our production file server is a super user-friendly but high performance Synology NAS 😉. All I have done so far was running Crystal disk benchmark on 1 Windows machine on the SMB Share. 3 Nodes: (original those where render workstations that are not in use right now) Each Node is MON MGM OSD Mainboard: ASRock TRX40 Creator CPU: AMD Ryzen Threadripper 3960X, 24 Cores, 3.8Ghz RAM: 2 x Samsung 32 GB 2 x 8 DDR4 2666 MHz 288-pin DIMM, Unregistered, ECC (64 GB Total) NIC Public: OnBoard Aquantia 107, 10Gbit NIC Ceph: Intel XXV710-DA2, 2x SFP28, 25Gbit System Drive: 2x Samsung SSD 860 PRO 256GB, SATA, ZFS Raid 1 System: Proxmox VE 6.2, Debian Buster, Ceph Nautilus HBA: Broadcom SAS 9305-16i OSDs: 6x Seagate Exos, 16TB, 7.200 rpm, 12Gb SAS Cache: 1x Micron 9300 MAX 3.2TB U.2 NVME I Played around with setting it up as a WAL/DB Device. Right now I have configured the Micron NVME as a BCache Infront of the 6 Seagate Drives in writeback mode. Because in this configuration BCache takes care of translating random writes to sequential ones for the HDDs I turned the Ceph WAL LOG off. I think Bcache gives more options to tune the System for my use case instead of just putting WAL/DB on the NVME. And also I can easily add cache drives or remove them without touching osds. I set up SMB Shares with the vfs_ceph module. I still have to add CTDB to distribute Samba to all nodes. My Next steps are to keep playing around in tuning the system and testing stability and performance. After that I want to put the Ceph cluster infront of our production NAS. Because our data is not super critical I thought of setting the replicas to 2 and running Rsync overnight to our NAS. That way I can switch to the old NAS at any time and wouldn’t loose more than 1 Day of work which is acceptable for us. This is how I could compare the two solutions side by side with real-life workload. I know that ceph might not be the best solution right now but if I am able to get at least similar performance to our Synology HDD NAS out of it, it would give a super scalable Solution in size and performance to grow with our needs. And who knows what performance improvements we get with ceph in the next 3 years. I am happy to hear your thoughts and ideas. And please I know this might be kind of a crazy setup but I have fun with it and I learned a lot the last few weeks. If my experiment fails I will go back to my original plan: Put FreeNas on two of the Nodes with overnight replication and put the third Node back to his render-friends. 😃 By the way I also have a spare Dell Server: 2x Xeon E5-2630 v3 2,40GHz, 128G Ram. I just don’t have an idea on how to utilize it. Maybe as extra OSD Node or as a separate Samba Server to get the SMB traffic away from the Public Ceph Network. Moritz Wilhelm _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Ceph as a Fileserver for 3D Content Production
by Moritz Wilhelm 17 May '20

17 May '20

Hi, my Name is Moritz and I am working for a 3D production company. Because of the corona virus I have too much time left and also to much unused hardware. That is why I started playing around with Ceph as a fileserver for us. Here I want to share my experience for all those who are interested. To start of here is my actual running test system. I am interested in the thoughts of the community and also on more suggestions on what to try out with my available Hardware. I don’t know how to test it right now because I am a newbie to ceph and our production file server is a super user-friendly but high performance Synology NAS 😉. All I have done so far was running Crystal disk benchmark on 1 Windows machine on the SMB Share. 3 Nodes: (original those where render workstations that are not in use right now) Each Node is MON MGM OSD Mainboard: ASRock TRX40 Creator CPU: AMD Ryzen Threadripper 3960X, 24 Cores, 3.8Ghz RAM: 2 x Samsung 32 GB 2 x 8 DDR4 2666 MHz 288-pin DIMM, Unregistered, ECC (64 GB Total) NIC Public: OnBoard Aquantia 107, 10Gbit NIC Ceph: Intel XXV710-DA2, 2x SFP28, 25Gbit System Drive: 2x Samsung SSD 860 PRO 256GB, SATA, ZFS Raid 1 System: Proxmox VE 6.2, Debian Buster, Ceph Nautilus HBA: Broadcom SAS 9305-16i OSDs: 6x Seagate Exos, 16TB, 7.200 rpm, 12Gb SAS Cache: 1x Micron 9300 MAX 3.2TB U.2 NVME I Played around with setting it up as a WAL/DB Device. Right now I have configured the Micron NVME as a BCache Infront of the 6 Seagate Drives in writeback mode. Because in this configuration BCache takes care of translating random writes to sequential ones for the HDDs I turned the Ceph WAL LOG off. I think Bcache gives more options to tune the System for my use case instead of just putting WAL/DB on the NVME. And also I can easily add cache drives or remove them without touching osds. I set up SMB Shares with the vfs_ceph module. I still have to add CTDB to distribute Samba to all nodes. My Next steps are to keep playing around in tuning the system and testing stability and performance. After that I want to put the Ceph cluster infront of our production NAS. Because our data is not super critical I thought of setting the replicas to 2 and running Rsync overnight to our NAS. That way I can switch to the old NAS at any time and wouldn’t loose more than 1 Day of work which is acceptable for us. This is how I could compare the two solutions side by side with real-life workload. I know that ceph might not be the best solution right now but if I am able to get at least similar performance to our Synology HDD NAS out of it, it would give a super scalable Solution in size and performance to grow with our needs. And who knows what performance improvements we get with ceph in the next 3 years. I am happy to hear your thoughts and ideas. And please I know this might be kind of a crazy setup but I have fun with it and I learned a lot the last few weeks. If my experiment fails I will go back to my original plan: Put FreeNas on two of the Nodes with overnight replication and put the third Node back to his render-friends. 😃 By the way I also have a spare Dell Server: 2x Xeon E5-2630 v3 2,40GHz, 128G Ram. I just don’t have an idea on how to utilize it. Maybe as extra OSD Node or as a separate Samba Server to get the SMB traffic away from the Public Ceph Network. Moritz Wilhelm

1 0

Re: Zeroing out rbd image or volume
by ceph＠elchaka.de 17 May '20

17 May '20

Hello Samuel, Why you want to do that? Just remove the rbd - as long your Image is distributet over random OSDs there is no way to recovery a delete Image.. ...2 cents Mehmet Am 12. Mai 2020 12:55:22 MESZ schrieb "huxiaoyu(a)horebdata.cn" <huxiaoyu(a)horebdata.cn>: >Hi， Ceph folks, > >Is there a rbd command, or any other way, to zero out rbd images or >volume? I would like to write all zero data to an rbd image/volume >before remove it. > >Any comments would be appreciated. > >best regards, > >samuel >Horebdata AG >Switzerland > > > > >huxiaoyu(a)horebdata.cn >_______________________________________________ >ceph-users mailing list -- ceph-users(a)ceph.io >To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

[ceph-users]
by Amit Ghadge 16 May '20

16 May '20

Hi All, I have question regarding Ceph Nautilus upgrade. In our test environment upgrading Luminous to Nautilus 14.2.8, and after enable msgr2, we seen one of the mon node restarted, my question is this normal process of restart mon service and 2nd question is, we using below mon_host format that can be correct or move to v2? mon_host = 10.44.172.181,10.44.172.182,10.44.172.183 Thanks, AmitG

1 0

Cephfs - NFS Ganesha
by Amudhan P 16 May '20

16 May '20

Hi, I am trying to setup NFS ganesh in Ceph Nautilus. In a ubuntu 18.04 system i have installed nfs-ganesha (v2.6) and nfs-ganesha-ceph pkg and followed the steps in the link https://docs.ceph.com/docs/nautilus/cephfs/nfs/ but i am not able to export my cephfs volume there is no error msg in nfs-ganesha, also i doubt whether its loading nfs-ganesha-ceph config file from "/etc/ganesha" folder. From same system i am able to mount thru ceph kernel client without any issue? How do i make this work? regards Amudhan

3 4

Re: EC Plugins Benchmark with Current Intel/AMD CPU
by Lazuardi Nasution 15 May '20

15 May '20

Hi Marc, I read https://blog.dachary.org/2015/05/12/ceph-jerasure-and-isa-plugins-benchmark… and it seem there is significant performance difference between benchmarked plugins. But, it was with old Intel E3-1200v2 series. I have read some others EC benchmark result improvements with current CPU but I cannot find it for Ceph implementation. Best regards, On Fri, May 15, 2020 at 9:03 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > How many % of the latency is even CPU related? > > > > -----Original Message----- > From: Lazuardi Nasution [mailto:mrxlazuardin@gmail.com] > Sent: 15 May 2020 16:00 > To: ceph-users(a)ceph.io > Subject: [ceph-users] EC Plugins Benchmark with Current Intel/AMD CPU > > Hi, > > Is there any EC plugins benchmark with current Intel/AMD CPU? It seem > there are new instructions which may accelerate EC. Let's say we want to > benchmark plugins using Intel 6200 or AMD 7002 series. I hope there is > better result than what have been benchmarked some years ago. > > Best regards, > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > >

1 0

EC Plugins Benchmark with Current Intel/AMD CPU
by Lazuardi Nasution 15 May '20

15 May '20

Hi, Is there any EC plugins benchmark with current Intel/AMD CPU? It seem there are new instructions which may accelerate EC. Let's say we want to benchmark plugins using Intel 6200 or AMD 7002 series. I hope there is better result than what have been benchmarked some years ago. Best regards,

2 1

Ceph modules
by Alfredo De Luca 15 May '20

15 May '20

Hi all. Just a quick one. Are there any ansible modules for ceph around? Cheers -- */Alfredo*

2 1

ACL for user in another teant
by Vishwas Bm 15 May '20

15 May '20

> Hi, > > I have two users both belong to different tenant. > > Can I give permission for the user in another tenant to access the bucket > using setacl or setPolicy command ? > I tried the setacl command and setpolicy command, but it was not working ? > It used to say bucket not found, when the grantee tried to access. > > Is this supported ? > > *Thanks & Regards,* > *Vishwas * > >

2 5

stale+active+clean PG
by tomislav.raseta＠gmail.com 14 May '20

14 May '20

Dear all, We're running Ceph Luminous and we've recently hit an issue with some OSD's (autoout states, IO/CPU overload) which unfortunately resulted with one placement group with the state "stale+active+clean", it's a placement group from .rgw.root pool: 1.15 0 0 0 0 0 0 1 1 stale+active+clean 2020-05-11 23:22:51.396288 40'1 2142:152 [3,2,6] 3 [3,2,6] 3 40'1 2020-04-22 00:46:05.904418 40'1 2020-04-20 20:18:13.371396 0 I guess there is no active replica of that object anywhere on the cluster. Restarting osd.3, osd.2 or osd.6 daemons does not help. I've used ceph-objectstore-tool and successfully exported placement group from osd.3, osd.2 and osd.6 and tried to import it on a completely different OSD, the exports differ in filesize slightly, but the osd.3 wihch was the latest primary is the biggest so I've tried to import it on a different OSD, when starting up I see the following (this is from osd.1): 2020-05-14 21:43:19.779740 7f7880ac3700 1 osd.1 pg_epoch: 2459 pg[1.15( v 40'1 (0'0,40'1] local-lis/les=2073/2074 n=0 ec=73/39 lis/c 2073/2073 les/c/f 2074/2074/633 2145/39/2145) [] r=-1 lpr=2455 crt=40'1 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray I see from previous pg dumps (several weeks before while it was still active+clean) that it was 115 bytes with zero objects in it but I am not sure how to interpret that. As this is a pg from .rgw.root pool, I cannot get any response from the cluster when accessing (everything timeouts). What is the correct course of action with this pg? Any help would be greatly appriciated. Thanks, Tomislav

1 0

Using rbd-mirror in existing pools
by Kees Meijs | Nefos 14 May '20

14 May '20

Hi list, Thanks again for pointing me towards rbd-mirror! I've read documentation, old mailing list posts, blog posts and some additional guides. Seems like the tool to help me through my data migration. Given one-way synchronisation and image-based (so, not pool based) configuration, it's still unclear to me how the mirroring will cope with an existing target pool, already consisting of (a lot of) images. Has someone done this already? It feels quite scary with doom scenario's like "cleaning up" the target pool and such in mind... To sum up: my goal is to mirror clusterA/somepool with some specific images to clusterB/someotherpool where already other images reside. The mirrored images should be kept in sync and the other images should be left alone completely. Cheers, Kees -- https://nefos.nl/contact Nefos IT bv Ambachtsweg 25 (industrienummer 4217) 5627 BZ Eindhoven Nederland KvK 66494931 /Aanwezig op maandag, dinsdag, woensdag en vrijdag/

5 7

Re: Cluster network and public network
by Nghia Viet Tran 14 May '20

14 May '20

Hi Martin, Thanks for your response. You mean one network interface for only MON hosts or for the whole cluster including OSD hosts? I’m confusing now because there are some projects that only use one public network for the whole cluster. That means the rebalancing, replicating objects and heartbeats from OSD hosts would affects the performance of Ceph client. From: Martin Verges <martin.verges(a)croit.io> Date: Friday, May 8, 2020 at 16:20 To: Nghia Viet Tran <Nghia.Viet.Tran(a)mgm-tp.com> Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Cluster network and public network Hello Nghia, just use one network interface card and use frontend and backend traffic on the same. No problem with that. If you have a dual port card, use both ports as an LACP channel and maybe separate it using VLANs if you want to, but not required as well. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io<mailto:martin.verges@croit.io> Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Fr., 8. Mai 2020 um 09:29 Uhr schrieb Nghia Viet Tran <Nghia.Viet.Tran(a)mgm-tp.com<mailto:Nghia.Viet.Tran@mgm-tp.com>>: Hi everyone, I have a question about the network setup. From the document, It’s recommended to have 2 NICs per hosts as described in below picture [Diagram] In the picture, OSD hosts will connect to the Cluster network for replicate and heartbeat between OSDs, therefore, we definitely need 2 NICs for it. But seems there are no connections between Ceph MON and Cluster network. Can we install 1 NIC on Ceph MON then? I appreciated any comments! Thank you! -- Nghia Viet Tran (Mr) _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

12 20

ceph-ansible replicated crush rule
by Marc Boisis 14 May '20

14 May '20

Hello, With ceph-ansible the default replicated crush rule is : { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } And I would like to have this: { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 2, "max_size": 4, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "rack" }, { "op": "emit" } ] } How can I do this within ceph-ansible playbook ? I try : crush_rule_replicated: name: replicated_rule root: default ruleset: 0 type: replicated min_size: 2 max_size: 4 step: - take default - chooseleaf firstn 0 type rack - emit default: true crush_rules: - "{{ crush_rule_replicated }}" task path: roles/ceph-osd/tasks/crush_rules.yml:32 try to change it but it fail: stdout_lines: - '' - '{"rule_id":0,"rule_name":"replicated_rule","ruleset":0,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-1,"item_name":"default"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}' Anyone have an idea ? Regards Marc

1 0

Re: virtual machines crashes after upgrade to octopus
by Erwin Lubbers 14 May '20

14 May '20

Hi, Did anyone found a way to resolve the problem? I'm seeing the same on a clean Octopus Ceph installation on Ubuntu 18 with an Octopus compiled KVM server running on CentOS 7.8. The KVM machine shows: [ 7682.233684] fn-radosclient[6060]: segfault at 2b19 ip 00007f8165cc0a50 sp 00007f81397f6490 error 4 in librbd.so.1.12.0[7f8165ab4000+537000] Ceph is healthy and stable for a few weeks and I did not get these messages while running on KVM compiled with Luminous libraries. Regards, Erwin

4 6

Bucket - radosgw-admin reshard process
by CUZA Frédéric 14 May '20

14 May '20

Hi everyone, I am facing an issue with bucket resharding. It started with a warning on my ceph cluster health : [root@ceph_monitor01 ~]# ceph -s cluster: id: 2da0734-2521-1p7r-8b4c-4a265219e807 health: HEALTH_WARN 1 large omap objects Finds out I had a problem with a bucket : "buckets": [ { "bucket": "bucket-elementary-1", "tenant": "", "num_objects": 615915, "num_shards": 3, "objects_per_shard": 205305, "fill_status": "OVER 100.000000%" }, I was wondering why the dynamic sharding wasn't doing its job but find out it was in the sharding queue : [root@ceph_monitor01 ~]# radosgw-admin reshard list [ { "time": "2020-05-14 09:42:10.905080Z", "tenant": "", "bucket_name": " bucket-elementary-1", "bucket_id": "97c1cfac-009f-4f7d-8d9d-9097c322c606.51988974.133", "new_instance_id": "", "old_num_shards": 3, "new_num_shards": 12 } ] As i wanted to process on taks in this queue I tried to run : radosgw-admin reshard process But turns out in an error : [root@ceph_monitor01 ~]# radosgw-admin reshard process ERROR: failed to process reshard logs, error=(22) Invalid argument 2020-05-14 14:15:10.225362 7f99b0437dc0 0 RGWReshardLock::lock failed to acquire lock on bucket-college-35:97c1cfac-009f-4f7d-8d9d-9097c322c606.51988974.133 ret=-22 2020-05-14 14:15:10.225376 7f99b0437dc0 0 process_single_logshardERROR in reshard_bucket bucket-elementary-1:(22) Invalid argument Tried to cancel it tu do it manually but had the same error [root@ceph_monitor01 ~]# radosgw-admin reshard cancel --bucket bucket-elementary-1 Error canceling bucket bucket-elementary-1 resharding: (22) Invalid argument 2020-05-14 14:16:42.196023 7fa0b1655dc0 0 RGWReshardLock::lock failed to acquire lock on bucket-elementary-1:97c1cfac-009f-4f7d-8d9d-9097c322c606.51988974.133 ret=-22 I found this post searching for an answer : https://tracker.ceph.com/issues/39970 But it seems that it will not help me since my whole cluster (monitors / data nodes / rgw) is on : [root@ceph_monitor01 ~]# ceph --version ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable) This is what I have on the status of this bucket : [root@ceph_monitor01 ~]# radosgw-admin reshard status --bucket= bucket-elementary-1 [ { "reshard_status": "CLS_RGW_RESHARD_NONE", "new_bucket_instance_id": "", "num_shards": 18446744073709551615 }, { "reshard_status": "CLS_RGW_RESHARD_NONE", "new_bucket_instance_id": "", "num_shards": 18446744073709551615 }, { "reshard_status": "CLS_RGW_RESHARD_NONE", "new_bucket_instance_id": "", "num_shards": 18446744073709551615 } ] Any help would be appreciated. Regards,

1 0

Re: RGW and the orphans
by EDH - Manuel Rios 14 May '20

14 May '20

Hi Eric Any update about that? Cluster status are critical and there's no a simple tool or cli provided in current releases that help to maintain our S3 Clusters Healthy. Right now with the multipart/sharding bugs looks like a bunch of scrap. Regards Manuel -----Mensaje original----- De: EDH - Manuel Rios <mriosfer(a)easydatahost.com> Enviado el: martes, 5 de mayo de 2020 15:34 Para: Katarzyna Myrek <katarzyna(a)myrek.pl>; Eric Ivancich <ivancich(a)redhat.com> CC: ceph-users(a)ceph.io Asunto: [ceph-users] Re: RGW and the orphans Hi Eric, Expected version to be included your tool in Nautilus? Maybe next reléase? Best Regards Manuel -----Mensaje original----- De: Katarzyna Myrek <katarzyna(a)myrek.pl> Enviado el: lunes, 20 de abril de 2020 12:19 Para: Eric Ivancich <ivancich(a)redhat.com> CC: EDH - Manuel Rios <mriosfer(a)easydatahost.com>; ceph-users(a)ceph.io Asunto: Re: [ceph-users] RGW and the orphans Hi Eric, I will try your tool this week on lab clusters. Will get back to you when I get the results. Kind regards / Pozdrawiam, Katarzyna Myrek pt., 17 kwi 2020 o 21:12 Eric Ivancich <ivancich(a)redhat.com> napisał(a): > > On Apr 17, 2020, at 9:38 AM, Katarzyna Myrek <katarzyna(a)myrek.pl> wrote: > > Hi Eric, > > Would it be possible to use it with an older cluster version (like > running new radosgw-admin in the container, connecting to the cluster > on 14.2.X)? > > Kind regards / Pozdrawiam, > Katarzyna Myrek > > > I did mention the nautilus backport PR in a separate reply (https://github.com/ceph/ceph/pull/34127) > > You can try the master version and see. To the best of my recollection the code porting did not involve any at-rest data structures. Instead it involved internal reorganization of the code. I suspect it would work, but if you try it, please report back what you find. Of course this is currently an experimental feature and care (e.g., sanity checking) should be taken before using the list produced to feed into a massive delete process. > > Eric > > -- > J. Eric Ivancich > > he / him / his > Red Hat Storage > Ann Arbor, Michigan, USA > > > > czw., 16 kwi 2020 o 19:58 EDH - Manuel Rios > <mriosfer(a)easydatahost.com> napisał(a): > > > Hi Eric, > > > > Are there any ETA for get those script backported maybe in 14.2.10? > > > > Regards > > Manuel > > > > > > De: Eric Ivancich <ivancich(a)redhat.com> Enviado el: jueves, 16 de > abril de 2020 19:05 > Para: Katarzyna Myrek <katarzyna(a)myrek.pl>; EDH - Manuel Rios > <mriosfer(a)easydatahost.com> > CC: ceph-users(a)ceph.io > Asunto: Re: [ceph-users] RGW and the orphans > > > > There is currently a PR for an “orphans list” capability. I’m currently working on the testing side to make sure it’s part of our teuthology suite. > > > > See: https://github.com/ceph/ceph/pull/34148 > > > > Eric > > > > > > On Apr 16, 2020, at 9:26 AM, Katarzyna Myrek <katarzyna(a)myrek.pl> wrote: > > > > Hi > > Thanks for the quick response. > > To be honest my cluster is getting full because of that trash and I am > at the point where I have to do the removal manually ;/. > > Kind regards / Pozdrawiam, > Katarzyna Myrek > > czw., 16 kwi 2020 o 13:09 EDH - Manuel Rios > <mriosfer(a)easydatahost.com> napisał(a): > > > Hi, > > From my experience orphans find didn't work since several releases ago, and command should be re-coded or deprecated because its not running. > > Im our cases it loops over generated shards until RGW daemon crash. > > Interested into this post, in our case orphans find takes more than 24 hours into start loop over shards, but never pass the shard 0 or 1. > > CEPH RGW devs, should provide any workaround script/ new tool or something to maintain our rgw clusters. Because with the last bugs all rgw cluster got a ton of trash, wasting resources and money. > > And manual cleaning is not trivial and easy. > > Waiting for more info, > > Manuel > > > -----Mensaje original----- > De: Katarzyna Myrek <katarzyna(a)myrek.pl> Enviado el: jueves, 16 de > abril de 2020 12:38 > Para: ceph-users(a)ceph.io > Asunto: [ceph-users] RGW and the orphans > > Hi > > Is there any new way to find and remove orphans from RGW pools on Nautilus? I have found info that "orphans find" is now deprecated? > > I can see that I have tons of orphans in one of our clusters. Was wondering how to safely remove them - make sure that they are really orphans. > Does anyone have a good method for that? > > My cluster mostly has orphans from multipart uploads. > > > Kind regards / Pozdrawiam, > Katarzyna Myrek > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph meltdown, need help
by Frank Schilder 14 May '20

14 May '20

Hi all, a lot of OSDs crashed in our cluster. Mimic 13.2.8. Current status included below. All daemons are running, no OSD process crashed. Can I start marking OSDs in and up to get them back talking to each other? Please advice on next steps. Thanks!! [root@gnosis ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN 2 MDSs report slow metadata IOs 1 MDSs report slow requests nodown,noout,norecover flag(s) set 125 osds down 3 hosts (48 osds) down Reduced data availability: 2221 pgs inactive, 1943 pgs down, 190 pgs peering, 13 pgs stale Degraded data redundancy: 5134396/500993581 objects degraded (1.025%), 296 pgs degraded, 299 pgs undersized 9622 slow ops, oldest one blocked for 2913 sec, daemons [osd.0,osd.100,osd.101,osd.112,osd.118,osd.133,osd.136,osd.142,osd.144,osd.145]... have slow ops. services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-02(active), standbys: ceph-03, ceph-01 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 288 osds: 90 up, 215 in; 230 remapped pgs flags nodown,noout,norecover data: pools: 10 pools, 2545 pgs objects: 62.61 M objects, 144 TiB usage: 219 TiB used, 1.6 PiB / 1.8 PiB avail pgs: 1.729% pgs unknown 85.540% pgs not active 5134396/500993581 objects degraded (1.025%) 1796 down 226 active+undersized+degraded 147 down+remapped 140 peering 65 active+clean 44 unknown 38 undersized+degraded+peered 38 remapped+peering 17 active+undersized+degraded+remapped+backfill_wait 12 stale+peering 12 active+undersized+degraded+remapped+backfilling 4 active+undersized+remapped 2 remapped 2 undersized+degraded+remapped+peered 1 stale 1 undersized+degraded+remapped+backfilling+peered io: client: 26 KiB/s rd, 206 KiB/s wr, 21 op/s rd, 50 op/s wr [root@gnosis ~]# ceph health detail HEALTH_WARN 2 MDSs report slow metadata IOs; 1 MDSs report slow requests; nodown,noout,norecover flag(s) set; 125 osds down; 3 hosts (48 osds) down; Reduced data availability: 2219 pgs inactive, 1943 pgs down, 188 pgs peering, 13 pgs stale; Degraded data redundancy: 5214696/500993589 objects degraded (1.041%), 298 pgs degraded, 299 pgs undersized; 9788 slow ops, oldest one blocked for 2953 sec, daemons [osd.0,osd.100,osd.101,osd.112,osd.118,osd.133,osd.136,osd.142,osd.144,osd.145]... have slow ops. MDS_SLOW_METADATA_IO 2 MDSs report slow metadata IOs mdsceph-08(mds.0): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 2940 secs mdsceph-12(mds.0): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 2942 secs MDS_SLOW_REQUEST 1 MDSs report slow requests mdsceph-08(mds.0): 100 slow requests are blocked > 30 secs OSDMAP_FLAGS nodown,noout,norecover flag(s) set OSD_DOWN 125 osds down osd.0 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-21) is down osd.6 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.7 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.8 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.16 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.18 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.19 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.21 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.31 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.37 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.38 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-07) is down osd.48 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.51 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-22) is down osd.53 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-21) is down osd.55 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-19) is down osd.62 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.67 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.72 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-21) is down osd.75 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.78 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.79 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.80 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.81 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.82 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-14) is down osd.83 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.88 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.89 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.92 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.93 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.95 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.96 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.97 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.100 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.104 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.105 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.107 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.108 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.109 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.111 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-14) is down osd.113 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.114 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.116 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.117 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.119 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.122 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.123 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.124 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.125 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.126 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.128 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.131 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.134 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.139 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.140 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.141 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.145 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.149 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.151 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.152 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.153 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.154 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-14) is down osd.155 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.156 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.157 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-05) is down osd.159 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-07) is down osd.161 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.162 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.164 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.165 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.166 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.167 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-14) is down osd.171 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.172 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-07) is down osd.174 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.176 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.177 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-13) is down osd.179 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.182 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-06) is down osd.183 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-07) is down osd.184 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.186 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.187 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.190 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-14) is down osd.191 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-15) is down osd.194 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.195 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.196 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.199 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.200 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.201 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.202 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.203 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.204 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.208 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.210 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-08) is down osd.212 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.213 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.214 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.215 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-10) is down osd.216 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.218 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-09) is down osd.219 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-11) is down osd.221 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-12) is down osd.224 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-16) is down osd.226 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1,host=ceph-17) is down osd.228 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-20) is down osd.230 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-20) is down osd.233 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-19) is down osd.236 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-19) is down osd.238 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.247 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-21) is down osd.248 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.254 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.256 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-04) is down osd.259 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.260 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-20) is down osd.262 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-19) is down osd.266 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.267 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-18) is down osd.272 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-20) is down osd.274 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-21) is down osd.275 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-19) is down osd.276 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-22) is down osd.281 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-22) is down osd.285 (root=DTU,region=Risoe,datacenter=ServerRoom,room=SR-113,host=ceph-05) is down OSD_HOST_DOWN 3 hosts (48 osds) down host ceph-11 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1) (16 osds) is down host ceph-10 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1) (16 osds) is down host ceph-13 (root=DTU,region=Risoe,datacenter=ContainerSquare,room=CON-161A1) (16 osds) is down PG_AVAILABILITY Reduced data availability: 2219 pgs inactive, 1943 pgs down, 188 pgs peering, 13 pgs stale pg 14.513 is stuck inactive for 1681.564244, current state down, last acting [2147483647,2147483647,2147483647,2147483647,2147483647,143,2147483647,2147483647,2147483647,2147483647] pg 14.514 is down, acting [193,2147483647,2147483647,2147483647,2147483647,118,2147483647,2147483647,2147483647,2147483647] pg 14.515 is down, acting [2147483647,2147483647,2147483647,211,133,135,2147483647,2147483647,2147483647,2147483647] pg 14.516 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,205,2147483647] pg 14.517 is down, acting [2147483647,2147483647,5,2147483647,2147483647,2147483647,2147483647,2147483647,61,112] pg 14.518 is down, acting [2147483647,198,2147483647,2147483647,2147483647,2147483647,4,185,2147483647,2147483647] pg 14.519 is down, acting [2147483647,2147483647,68,2147483647,2147483647,2147483647,2147483647,185,2147483647,94] pg 14.51a is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,101,2147483647] pg 14.51b is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,197,2147483647,2147483647,2147483647,2147483647] pg 14.51c is down, acting [193,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,197] pg 14.51d is down, acting [2147483647,2147483647,61,2147483647,77,2147483647,2147483647,2147483647,112,2147483647] pg 14.51e is down, acting [2147483647,2147483647,2147483647,2147483647,112,2147483647,2147483647,193,2147483647,2147483647] pg 14.51f is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,94,2147483647,2147483647] pg 14.520 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,207,2147483647,101,133,2147483647] pg 14.521 is down, acting [205,2147483647,133,2147483647,2147483647,2147483647,2147483647,4,2147483647,193] pg 14.522 is down, acting [101,2147483647,2147483647,11,197,2147483647,136,94,2147483647,2147483647] pg 14.523 is down, acting [2147483647,2147483647,2147483647,118,2147483647,71,2147483647,2147483647,2147483647,2147483647] pg 14.524 is down, acting [2147483647,111,2147483647,2147483647,2147483647,8,2147483647,112,2147483647,2147483647] pg 14.525 is down, acting [2147483647,2147483647,2147483647,142,2147483647,61,2147483647,2147483647,2147483647,2147483647] pg 14.526 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,61,193,2147483647,2147483647,2147483647] pg 14.527 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,109,2147483647,2147483647] pg 14.528 is down, acting [2147483647,133,2147483647,2147483647,2147483647,2147483647,4,2147483647,2147483647,2147483647] pg 14.529 is down, acting [2147483647,112,2147483647,2147483647,2147483647,2147483647,185,2147483647,118,2147483647] pg 14.52a is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,136,2147483647,135,2147483647,2147483647] pg 14.52b is down, acting [2147483647,2147483647,2147483647,112,142,211,2147483647,2147483647,2147483647,2147483647] pg 14.52c is down, acting [185,2147483647,198,2147483647,118,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.52d is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,5,2147483647,2147483647,2147483647] pg 14.52e is down, acting [71,101,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,142,2147483647] pg 14.52f is down, acting [198,2147483647,2147483647,2147483647,2147483647,11,2147483647,2147483647,118,2147483647] pg 14.530 is down, acting [142,2147483647,2147483647,2147483647,133,2147483647,2147483647,2147483647,2147483647,112] pg 14.531 is down, acting [2147483647,142,2147483647,2147483647,2147483647,185,2147483647,2147483647,2147483647,2147483647] pg 14.532 is down, acting [135,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,136,118] pg 14.533 is down, acting [2147483647,77,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.534 is down, acting [2147483647,2147483647,2147483647,185,118,2147483647,2147483647,207,2147483647,2147483647] pg 14.535 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,136,142,133,2147483647] pg 14.536 is down, acting [2147483647,11,2147483647,2147483647,136,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.537 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,77,2147483647] pg 14.538 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,205,2147483647,2147483647] pg 14.539 is down, acting [2147483647,2147483647,2147483647,198,2147483647,2147483647,4,2147483647,2147483647,2147483647] pg 14.53a is down, acting [2147483647,11,136,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.53b is down, acting [2147483647,2147483647,2147483647,2147483647,112,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.53c is down, acting [2147483647,2147483647,2147483647,71,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.53d is down, acting [2147483647,2147483647,2147483647,185,2147483647,2147483647,2147483647,2147483647,2147483647,136] pg 14.53e is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,112,185] pg 14.53f is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,185,2147483647,2147483647,2147483647] pg 14.540 is down, acting [205,2147483647,2147483647,2147483647,2147483647,2147483647,142,2147483647,112,77] pg 14.541 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,197,211,2147483647,2147483647,2147483647] pg 14.542 is down, acting [112,2147483647,101,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] pg 14.543 is down, acting [111,2147483647,2147483647,2147483647,2147483647,101,2147483647,2147483647,2147483647,2147483647] pg 14.544 is down, acting [4,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,205] pg 14.545 is down, acting [2147483647,2147483647,2147483647,2147483647,2147483647,142,5,2147483647,2147483647,2147483647] PG_DEGRADED Degraded data redundancy: 5214696/500993589 objects degraded (1.041%), 298 pgs degraded, 299 pgs undersized pg 1.29 is stuck undersized for 2075.633328, current state active+undersized+degraded, last acting [253,258] pg 1.2a is stuck undersized for 1642.864920, current state active+undersized+degraded, last acting [252,255] pg 1.2b is stuck undersized for 2355.149928, current state active+undersized+degraded+remapped+backfill_wait, last acting [240,268] pg 1.2c is stuck undersized for 1459.277329, current state active+undersized+degraded, last acting [241,273] pg 1.2d is stuck undersized for 803.339131, current state undersized+degraded+peered, last acting [282] pg 2.25 is active+undersized+degraded, acting [253,2147483647,2147483647,258,261,273,277,243] pg 2.28 is stuck undersized for 803.340163, current state active+undersized+degraded, last acting [282,241,246,2147483647,273,252,2147483647,268] pg 2.29 is stuck undersized for 803.341160, current state active+undersized+degraded, last acting [240,258,277,264,2147483647,2147483647,271,250] pg 2.2a is stuck undersized for 1447.684978, current state active+undersized+degraded+remapped+backfilling, last acting [252,270,2147483647,261,2147483647,255,287,264] pg 2.2e is stuck undersized for 2030.849944, current state active+undersized+degraded, last acting [264,2147483647,251,245,257,286,261,258] pg 2.51 is stuck undersized for 1459.274671, current state active+undersized+degraded+remapped+backfilling, last acting [270,2147483647,2147483647,265,241,243,240,252] pg 2.52 is stuck undersized for 2030.850897, current state active+undersized+degraded+remapped+backfilling, last acting [240,2147483647,270,265,269,280,278,2147483647] pg 2.53 is stuck undersized for 1459.273517, current state active+undersized+degraded, last acting [261,2147483647,280,282,2147483647,245,243,241] pg 2.61 is stuck undersized for 2075.633140, current state active+undersized+degraded+remapped+backfilling, last acting [269,2147483647,258,286,270,255,2147483647,264] pg 2.62 is stuck undersized for 803.340577, current state active+undersized+degraded, last acting [2147483647,253,258,2147483647,250,287,264,284] pg 2.66 is stuck undersized for 803.341231, current state active+undersized+degraded, last acting [264,280,265,255,257,269,2147483647,270] pg 2.6c is stuck undersized for 963.369539, current state active+undersized+degraded, last acting [286,269,278,251,2147483647,273,2147483647,280] pg 2.70 is stuck undersized for 873.662725, current state active+undersized+degraded, last acting [2147483647,268,255,273,253,265,278,2147483647] pg 2.74 is stuck undersized for 2075.632312, current state active+undersized+degraded+remapped+backfilling, last acting [240,242,2147483647,245,243,269,2147483647,265] pg 3.24 is stuck undersized for 1570.800184, current state active+undersized+degraded, last acting [235,263] pg 3.25 is stuck undersized for 733.673503, current state undersized+degraded+peered, last acting [232] pg 3.28 is stuck undersized for 2610.307886, current state active+undersized+degraded, last acting [263,84] pg 3.2a is stuck undersized for 1214.710839, current state active+undersized+degraded, last acting [181,232] pg 3.2b is stuck undersized for 2075.630671, current state active+undersized+degraded, last acting [63,144] pg 3.52 is stuck undersized for 1570.777598, current state active+undersized+degraded, last acting [158,237] pg 3.54 is stuck undersized for 1350.257189, current state active+undersized+degraded, last acting [239,74] pg 3.55 is stuck undersized for 2592.642531, current state active+undersized+degraded, last acting [157,233] pg 3.5a is stuck undersized for 2075.608257, current state undersized+degraded+peered, last acting [168] pg 3.5c is stuck undersized for 733.674836, current state active+undersized+degraded, last acting [263,234] pg 3.5d is stuck undersized for 2610.307220, current state active+undersized+degraded, last acting [180,84] pg 3.5e is stuck undersized for 1710.756037, current state undersized+degraded+peered, last acting [146] pg 3.61 is stuck undersized for 1080.210021, current state active+undersized+degraded, last acting [168,239] pg 3.62 is stuck undersized for 831.217622, current state active+undersized+degraded, last acting [84,263] pg 3.63 is stuck undersized for 733.674204, current state active+undersized+degraded, last acting [263,232] pg 3.65 is stuck undersized for 1570.790824, current state active+undersized+degraded, last acting [63,84] pg 3.66 is stuck undersized for 733.682973, current state undersized+degraded+peered, last acting [63] pg 3.68 is stuck undersized for 1570.624462, current state active+undersized+degraded, last acting [229,148] pg 3.69 is stuck undersized for 1350.316213, current state undersized+degraded+peered, last acting [235] pg 3.6b is stuck undersized for 783.813654, current state undersized+degraded+peered, last acting [63] pg 3.6c is stuck undersized for 783.819083, current state undersized+degraded+peered, last acting [229] pg 3.6f is stuck undersized for 2610.321349, current state active+undersized+degraded, last acting [232,158] pg 3.72 is stuck undersized for 1350.358149, current state active+undersized+degraded, last acting [229,74] pg 3.73 is stuck undersized for 1570.788310, current state undersized+degraded+peered, last acting [234] pg 11.20 is stuck undersized for 733.682510, current state active+undersized+degraded, last acting [2147483647,239,87,2147483647,158,237,63,76] pg 11.26 is stuck undersized for 1914.334332, current state active+undersized+degraded, last acting [2147483647,237,2147483647,263,158,148,181,180] pg 11.2d is stuck undersized for 1350.365988, current state active+undersized+degraded, last acting [2147483647,2147483647,73,229,86,158,169,84] pg 11.54 is stuck undersized for 1914.398125, current state active+undersized+degraded, last acting [231,169,2147483647,229,84,85,237,63] pg 11.5b is stuck undersized for 2047.980719, current state active+undersized+degraded, last acting [86,237,168,263,144,1,229,2147483647] pg 11.5e is stuck undersized for 873.643661, current state active+undersized+degraded, last acting [181,2147483647,229,158,231,1,169,2147483647] pg 11.62 is stuck undersized for 1144.491696, current state active+undersized+degraded, last acting [2147483647,85,235,74,63,234,181,2147483647] pg 11.6f is stuck undersized for 873.646628, current state active+undersized+degraded, last acting [234,3,2147483647,158,180,63,2147483647,181] SLOW_OPS 9788 slow ops, oldest one blocked for 2953 sec, daemons [osd.0,osd.100,osd.101,osd.112,osd.118,osd.133,osd.136,osd.142,osd.144,osd.145]... have slow ops. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

6 19

Re: Memory usage of OSD
by Igor Fedotov 14 May '20

14 May '20

Rafal, just to mention - stupid allocator is known to cause high memory usage in certain scenarios but it uses bluestore_alloc mempool. Thanks, Igor On 5/13/2020 6:52 PM, Rafał Wądołowski wrote: > Mark, > Unfortunetly I closed terminal with mempool. But there was a lot of bytes used by bluestore_cache_other. That was the highest value (about 85%). The onode cache takes about 10%. PGlog and osdmaps was okey, low values. I saw some ideas that maybe compression_mode force in pool can make a mess. > One more thing, we are running stupid allocator. Right now I am decrease the osd_memory_target to 3GiB and will wait if ram problem occurs. > > > > > Regards, > > Rafał Wądołowski > > ________________________________ > From: Mark Nelson <mnelson(a)redhat.com> > Sent: Wednesday, May 13, 2020 3:30 PM > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Memory usage of OSD > > On 5/13/20 12:43 AM, RafaĹ' WÄ...doĹ'owski wrote: >> Hi, >> I noticed strange situation in one of our clusters. The OSD deamons are taking too much RAM. >> We are running 12.2.12 and have default configuration of osd_memory_target (4GiB). >> Heap dump shows: >> >> osd.2969 dumping heap profile now. >> ------------------------------------------------ >> MALLOC: 6381526944 ( 6085.9 MiB) Bytes in use by application >> MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist >> MALLOC: + 173373288 ( 165.3 MiB) Bytes in central cache freelist >> MALLOC: + 17163520 ( 16.4 MiB) Bytes in transfer cache freelist >> MALLOC: + 95339512 ( 90.9 MiB) Bytes in thread cache freelists >> MALLOC: + 28995744 ( 27.7 MiB) Bytes in malloc metadata >> MALLOC: ------------ >> MALLOC: = 6696399008 ( 6386.2 MiB) Actual memory used (physical + swap) >> MALLOC: + 218267648 ( 208.2 MiB) Bytes released to OS (aka unmapped) >> MALLOC: ------------ >> MALLOC: = 6914666656 ( 6594.3 MiB) Virtual address space used >> MALLOC: >> MALLOC: 408276 Spans in use >> MALLOC: 75 Thread heaps in use >> MALLOC: 8192 Tcmalloc page size >> ------------------------------------------------ >> Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). >> Bytes released to the OS take up virtual address space but no physical memory. >> >> IMO "Bytes in use by application" should be less than osd_memory_target. Am I correct? >> I checked heap dump with google-pprof and got following results. >> Total: 149.4 MB >> 60.5 40.5% 40.5% 60.5 40.5% rocksdb::UncompressBlockContentsForCompressionType >> 34.2 22.9% 63.4% 34.2 22.9% ceph::buffer::create_aligned_in_mempool >> 11.9 7.9% 71.3% 12.1 8.1% std::_Rb_tree::_M_emplace_hint_unique >> 10.7 7.1% 78.5% 71.2 47.7% rocksdb::ReadBlockContents >> >> Does it mean that most of RAM is used by rocksdb? > > It looks like your heap dump is only accounting for 149.4MB of the > memory so probably not representative across the whole ~6.5G. Instead > could you try dumping the mempools via "ceph daemon osd.2969 dump_mempools"? > > >> How can I take a deeper look into memory usage ? > > Beyond looking at the mempools, you can see the bluestore cache > allocation information by either enabling debug bluestore and debug > priority_cache_manager 5, or potentially looking at the PCM perf > counters (I'm not sure if those were in 14.2.12 though). Between the > heap data, mempool data, and priority cache records, it should become > clearer what's going on. > > > Mark > > >> >> Regards, >> >> RafaĹ' WÄ...doĹ'owski >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

iscsi issues with ceph (Nautilus) + tcmu-runner
by Phil Regnauld 14 May '20

14 May '20

So, we've been running with iscsi enabled (tcmu-runner) on our Nautilus ceph cluster for a couple of weeks, and started using it with our vsphere cluster. Things looked good so we put it in production, but yesterday morning we experienced a freeze of all iSCSO I/O one of the ESXi nodes, and the only way to recover was a reboot of the client node (and the VMs with it). At first I thought this was a glitch on the VMware side, as we saw lots of warnings, but after digging in the kernel logs on the ceph side, we saw quite a few of the messages below, leading up to the freeze. We've got two iscsi gateways enabled, and have set things up as per https://docs.ceph.com/docs/mimic/rbd/iscsi-initiators/ Kernel is 4.19 / Debian 10. This is single 10G link for the ESXi nodes, with separate vlan and dedicated vmkernel nic on the storage vlan to talk to the iscsi gateways on the ceph side. Ceph is 10 nodes with a mix of HDD and SSD. The affected pool is replication on HDD w/NVMe block db in front. No overrun / errors on the interfaces, and no mtu mismatch - not that I believe this is connected to the error below. No errors/warnings on the ceph side. Anyone seen this before ? Can provide more details/logs as required. If this happens again, we'll be moving the VMs to an NFS backed vm store (even if iscsi is probably preferred) until we can find a solution to go into full prod. Cheers, Phil [Tue May 12 05:15:45 2020] WARNING: CPU: 8 PID: 2448784 at kernel/workqueue.c:2917 __flush_work.cold.54+0x1f/0x29 [Tue May 12 05:15:45 2020] Modules linked in: fuse cbc ceph libceph libcrc32c crc32c_generic fscache target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_user uio target_core_mod configfs binfmt_misc 8021q garp stp mrp llc bonding intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ast cryptd ipmi_ssif ttm intel_cstate drm_kms_helper intel_uncore mei_me iTCO_wdt drm pcc_cpufreq intel_rapl_perf joydev pcspkr sg iTCO_vendor_support mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter evdev dm_mod sunrpc ip_tables x_tables autofs4 squashfs zstd_decompress xxhash loop overlay hid_generic usbhid hid sd_mod ahci xhci_pci ehci_pci libahci ehci_hcd xhci_hcd libata ixgbe igb nvme mxm_wmi lpc_ich [Tue May 12 05:15:45 2020] dca usbcore i2c_i801 scsi_mod mfd_core mdio usb_common nvme_core crc32c_intel i2c_algo_bit wmi button [Tue May 12 05:15:45 2020] CPU: 8 PID: 2448784 Comm: kworker/u32:0 Tainted: G W 4.19.0-8-amd64 #1 Debian 4.19.98-1 [Tue May 12 05:15:45 2020] Hardware name: Supermicro SYS-6018R-TD8/X10DDW-i, BIOS 3.2 12/16/2019 [Tue May 12 05:15:45 2020] Workqueue: tmr-user target_tmr_work [target_core_mod] [Tue May 12 05:15:45 2020] RIP: 0010:__flush_work.cold.54+0x1f/0x29 [Tue May 12 05:15:45 2020] Code: 69 2c 04 00 0f 0b e9 4a d3 ff ff 48 c7 c7 d8 f9 83 be e8 56 2c 04 00 0f 0b e9 41 d6 ff ff 48 c7 c7 d8 f9 83 be e8 43 2c 04 00 <0f> 0b 45 31 ed e9 2b d6 ff ff 49 8d b4 24 b0 00 00 00 48 c7 c7 b8 [Tue May 12 05:15:45 2020] RSP: 0018:ffffa0180b537d38 EFLAGS: 00010246 [Tue May 12 05:15:45 2020] RAX: 0000000000000024 RBX: ffff94f619c3fad0 RCX: 0000000000000006 [Tue May 12 05:15:45 2020] RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff94f77fa166b0 [Tue May 12 05:15:45 2020] RBP: ffff94f619c3fad0 R08: 00000000000013ea R09: 0000000000aaaaaa [Tue May 12 05:15:45 2020] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000 [Tue May 12 05:15:45 2020] R13: 0000000000000001 R14: ffffffffbda98400 R15: ffff94f5a3ecc088 [Tue May 12 05:15:45 2020] FS: 0000000000000000(0000) GS:ffff94f77fa00000(0000) knlGS:0000000000000000 [Tue May 12 05:15:45 2020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Tue May 12 05:15:45 2020] CR2: 0000555d72f0f000 CR3: 000000048240a005 CR4: 00000000003606e0 [Tue May 12 05:15:45 2020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [Tue May 12 05:15:45 2020] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [Tue May 12 05:15:45 2020] Call Trace: [Tue May 12 05:15:45 2020] ? __irq_work_queue_local+0x50/0x60 [Tue May 12 05:15:45 2020] ? irq_work_queue+0x46/0x50 [Tue May 12 05:15:45 2020] ? wake_up_klogd+0x30/0x40 [Tue May 12 05:15:45 2020] ? vprintk_emit+0x215/0x270 [Tue May 12 05:15:45 2020] ? get_work_pool+0x40/0x40 [Tue May 12 05:15:45 2020] __cancel_work_timer+0x10a/0x190 [Tue May 12 05:15:45 2020] ? printk+0x58/0x6f [Tue May 12 05:15:45 2020] core_tmr_abort_task+0xd6/0x130 [target_core_mod] [Tue May 12 05:15:45 2020] target_tmr_work+0xc4/0x140 [target_core_mod] [Tue May 12 05:15:45 2020] process_one_work+0x1a7/0x3a0 [Tue May 12 05:15:45 2020] worker_thread+0x30/0x390 [Tue May 12 05:15:45 2020] ? create_worker+0x1a0/0x1a0 [Tue May 12 05:15:45 2020] kthread+0x112/0x130 [Tue May 12 05:15:45 2020] ? kthread_bind+0x30/0x30 [Tue May 12 05:15:45 2020] ret_from_fork+0x1f/0x40 [Tue May 12 05:15:45 2020] ---[ end trace 3835b5fe0aa99038 ]--- [Tue May 12 05:15:45 2020] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 94585265 [Tue May 12 05:15:45 2020] ABORT_TASK: Found referenced iSCSI task_tag: 94585266 [Tue May 12 05:15:45 2020] ------------[ cut here ]------------

2 2

all VMs in compute node openstack connecting to this ceph cluster error connect after run command ceph osd set-require-min-compat-client luminus
by luuvuong91＠gmail.com 14 May '20

14 May '20

Hi all, My Cluster include 4 node runing in version luminus 12.2.12 I use command ceph osd set-require-min-compat-client luminus in my cluster then all VMs in compute node openstack connecting to this ceph cluster error connect I want update ceph osd set-require-min-compat-client jewel to rollback config or @all have plan fix other ? Please support for me Thanks

3 10

why don't ceph daemon output their log to /var/log/ceph
by 展荣臻（信泰） 14 May '20

14 May '20

Hi.all, when ceph run in container why not let ceph daemon output their log to /var/log/ceph, I build ceph image with ceph-container and deploy ceph via ceph-ansible. I found no logs under /var/log/ceph.why don't ceph daemon output logs to /var/log/ceph?

1 0

Migrating clusters (and versions)
by Kees Meijs 14 May '20

14 May '20

Hi list, I'm in the middle of an OpenStack migration (obviously Ceph backed) and stumble into some huge virtual machines. To ensure downtime is kept to a minimum, I'm thinking of using Ceph's snapshot features using rbd export-diff and import-diff. However, is it safe (or even supported) to do this across versions? The source cluster is running 10.2.11 and destination is 12.2.11 Thanks in advance! Regards, Kees -- https://nefos.nl/contact Nefos IT bv Ambachtsweg 25 (industrienummer 4217) 5627 BZ Eindhoven Nederland KvK 66494931 /Aanwezig op maandag, dinsdag, woensdag en vrijdag/

5 11

Remove or recreate damaged PG in erasure coding pool
by Francois Legrand 14 May '20

14 May '20

Hello, We run nautilus 14.2.8 ceph cluster. After a big crash in which we lost some disks we had a PG down (Erasure coding 3+2 pool) and trying to fix it we followed this https://medium.com/opsops/recovering-ceph-from-reduced-data-availability-3-… As the PG was reported with 0 objects we first marked a shard as complete with ceph-objectstore-tool and restart the osd. The pg thus went active but reported lost objects ! As we consider the datas on this pg as lost, we try to get rid of this with ceph pg 30.3 mark_unfound_lost delete. This produced some logs like (~3 lines/hour): 2020-05-12 14:45:05.251830 osd.103 (osd.103) 886 : cluster [ERR] 30.3s0 Unexpected Error: recovery ending with 41: {30:c000e27d:::rbd_data.34.c963b6314efb84.000000000 0000100:head=435293'2 flags = delete,30:c01f1248:::rbd_data.34.7f0c0d1df22f45.0000000000000325:head=435293'3 flags = delete,30:c05e82b2:::rbd_data.34.674d063bdc66d2.0 000000000000015:head=435293'4 flags = delete,30:c0b2d8e7:::rbd_data.34.6bc88749c741cb.00000000000007d0:head=435293'5 flags = delete,30:c0c3e20e:::rbd_data.34.674d063b dc66d2.00000000000000fb:head=435293'6 flags = delete,30:c0c89740:::rbd_data.34.a7f2202210bb39.0000000000000bbc:head=435293'7 flags = delete,30:c0e59ffa:::rbd_data.34. 7f0c0d1df22f45.00000000000002fb:head=435293'8 flags = delete,30:c0e72bf4:::rbd_data.34.7f0c0d1df22f45.00000000000000fa:head=435293'9 flags = delete,30:c10ab507:::rbd_ data.34.80695c646d9535.0000000000000327:head=435293'10 flags = delete,30:c219e412:::rbd_data.34.a7f2202210bb39.0000000000000fa0:head=435293'11 flags = delete,30:c29ae ba3:::rbd_data.34.8038585a0eb9f6.0000000000000eb2:head=435293'12 flags = delete,30:c29fae09:::rbd_data.34.674d063bdc66d2.000000000000148a:head=435293'13 flags = delet e,30:c2b77a99:::rbd_data.34.7f0c0d1df22f45.000000000000031d:head=435293'14 flags = delete,30:c2c8598f:::rbd_data.34.674d063bdc66d2.00000000000002f5:head=435293'15 fla gs = delete,30:c2dd39fe:::rbd_data.34.6494fb1b0f88bf.000000000000030b:head=435293'16 flags = delete,30:c2f6ce39:::rbd_data.34.806ab864459ae5.0000000000000109:head=435 293'17 flags = delete,30:c2f8a62f:::rbd_data.34.ed0c58ebdc770f.000000000000002a:head=435293'18 flags = delete,30:c306cd86:::rbd_data.34.ed0c58ebdc770f.000000000000020 5:head=435293'19 flags = delete,30:c30f5230:::rbd_data.34.7f0c0d1df22f45.00000000000002f5:head=435293'20 flags = delete,30:c32b81df:::rbd_data.34.c79f6d1f78a707.00000 00000000100:head=435293'21 flags = delete,30:c3374080:::rbd_data.34.7f217e33dd742c.00000000000007d0:head=435293'22 flags = delete,30:c3cdbeb5:::rbd_data.34.674dcefe97 f606.0000000000000109:head=435293'23 flags = delete,30:c3cdd149:::rbd_data.34.674dcefe97f606.0000000000000019:head=435293'24 flags = delete,30:c40946c0:::rbd_data.34. ded8d21a9d3d8f.00000000000002a8:head=435293'25 flags = delete,30:c42ed4fd:::rbd_data.34.a6985314ad8dad.0000000000000200:head=435293'26 flags = delete,30:c483a99b:::rb d_data.34.ed0c58ebdc770f.0000000000000a00:head=435293'27 flags = delete,30:c49f09d6:::rbd_data.34.7e1c1abf436885.0000000000000bb8:head=435293'28 flags = delete,30:c51 5a4e8:::rbd_data.34.ed0c58ebdc770f.0000000000000106:head=435293'29 flags = delete,30:c5181a8e:::rbd_data.34.9385d45172fa0f.000000000000020c:head=435293'30 flags = del ete,30:c531de44:::rbd_data.34.6bc88749c741cb.0000000000000102:head=435293'31 flags = delete,30:c5427518:::rbd_data.34.806ab864459ae5.00000000000006db:head=435293'32 f lags = delete,30:c5693b53:::rbd_data.34.6494fb1b0f88bf.000000000000148a:head=435293'33 flags = delete,30:c5804bc9:::rbd_data.34.ed0cb8730e020c.0000000000000105:head=4 35293'34 flags = delete,30:c598117e:::rbd_data.34.7f0811fbac0b9d.0000000000000327:head=435293'35 flags = delete,30:c5a64fbd:::rbd_data.34.c963b6314efb84.0000000000000 010:head=435293'36 flags = delete,30:c5f9e0e5:::rbd_data.34.ed0c58ebdc770f.0000000000000f01:head=435293'37 flags = delete,30:c5ffe1d8:::rbd_data.34.6bc88749c741cb.000 0000000000abe:head=435293'38 flags = delete,30:c6ecfaa1:::rbd_data.34.9385d45172fa0f.0000000000000002:head=435293'39 flags = delete,30:c755550f:::rbd_data.34.6494fb1b 0f88bf.0000000000000106:head=435293'40 flags = delete,30:c7a730f4:::rbd_data.34.7f217e33dd742c.00000000000006e1:head=435293'41 flags = delete,30:c7aa79f7:::rbd_data.3 4.674dcefe97f606.0000000000000108:head=435293'42 flags = delete} But yesterday it started to flood the logs (~9 GB of logs/day !) with lines like : 2020-05-14 10:36:03.851258 osd.29 [ERR] Error -2 reading object 30:c24a0173:::rbd_data.34.806ab864459ae5.000000000000022d:head 2020-05-14 10:36:03.851333 osd.29 [ERR] Error -2 reading object 30:c4a41972:::rbd_data.34.6bc88749c741cb.0000000000000320:head 2020-05-14 10:36:03.851382 osd.29 [ERR] Error -2 reading object 30:c543da6f:::rbd_data.34.80695c646d9535.0000000000000dce:head 2020-05-14 10:36:03.859900 osd.29 [ERR] Error -2 reading object 30:c24a0173:::rbd_data.34.806ab864459ae5.000000000000022d:head 2020-05-14 10:36:03.859979 osd.29 [ERR] Error -2 reading object 30:c4a41972:::rbd_data.34.6bc88749c741cb.0000000000000320:head We think that the best would probably to completely delete this pg. Is that possible without totally breaking the pool ? How ? Do we need to recreate the pg manually (or ceph will do it automatically) ? Thanks for you help. F.

1 0

ceph orch ps => osd <unknown> (Octopus 15.2.1)
by Ml Ml 14 May '20

14 May '20

Hello, any idea what´s wrong with my osd.34+35? root@ceph01:~# ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID (...) osd.34 ceph04 running - - <unknown> <unknown> <unknown> <unknown> osd.35 ceph04 running - - <unknown> <unknown> <unknown> <unknown> (...) root 9411 0.3 0.1 4471132 55732 ? Ssl May04 42:43 /usr/sbin/dockerd -H fd:// root 9429 0.2 0.0 4644008 28456 ? Ssl May04 29:15 \_ docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level info root 15536 0.0 0.0 774832 3980 ? Sl May04 0:48 \_ docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/a2f6 167 15553 4.7 12.4 5905888 4068064 ? Ssl May04 654:21 | \_ /usr/bin/ceph-osd -n osd.34 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true root 17168 0.0 0.0 848628 3736 ? Sl May04 0:50 \_ docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/5135 167 17186 4.6 12.7 6006116 4179644 ? Ssl May04 632:40 | \_ /usr/bin/ceph-osd -n osd.35 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true ceph osd tree -11 10.68115 host ceph04 34 hdd 2.67029 osd.34 up 0.90002 1.00000 35 hdd 2.67029 osd.35 up 0.80005 1.00000 44 hdd 2.67029 osd.44 up 0.95001 1.00000 45 hdd 2.67029 osd.45 up 1.00000 1.00000 Thanks, Michael

1 0

OSD weight on Luminous
by Florent B. 14 May '20

14 May '20

Hi, I have something strange on a Ceph Luminous cluster. All OSDs have the same size, the same weight, and one of them is used at 88% by Ceph (osd.3) while others are around 40 to 50% usage : # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS 2 hdd 0.49179 1.00000 504GiB 264GiB 263GiB 63.7MiB 960MiB 240GiB 52.34 1.14 81 13 hdd 0.49179 1.00000 504GiB 267GiB 266GiB 55.7MiB 1.37GiB 236GiB 53.09 1.16 94 20 hdd 0.49179 1.00000 504GiB 235GiB 234GiB 62.5MiB 962MiB 268GiB 46.70 1.02 99 21 hdd 0.49179 1.00000 504GiB 306GiB 305GiB 65.2MiB 991MiB 198GiB 60.75 1.32 87 22 hdd 0.49179 1.00000 504GiB 185GiB 184GiB 51.9MiB 972MiB 318GiB 36.83 0.80 73 23 hdd 0.49179 1.00000 504GiB 167GiB 166GiB 60.9MiB 963MiB 337GiB 33.07 0.72 80 24 hdd 0.49179 1.00000 504GiB 235GiB 234GiB 67.5MiB 956MiB 268GiB 46.74 1.02 90 25 hdd 0.49179 1.00000 504GiB 183GiB 182GiB 68.8MiB 955MiB 321GiB 36.32 0.79 100 3 hdd 0.49179 1.00000 504GiB 442GiB 440GiB 77.5MiB 1.15GiB 61.9GiB 87.70 1.91 103 26 hdd 0.49179 1.00000 504GiB 220GiB 219GiB 61.2MiB 963MiB 283GiB 43.78 0.95 80 29 hdd 0.49179 1.00000 504GiB 298GiB 296GiB 77.4MiB 1013MiB 206GiB 59.09 1.29 106 30 hdd 0.49179 1.00000 504GiB 183GiB 182GiB 60.2MiB 964MiB 321GiB 36.32 0.79 88 10 hdd 0.49179 1.00000 504GiB 176GiB 175GiB 56.5MiB 968MiB 327GiB 35.02 0.76 85 11 hdd 0.49179 1.00000 504GiB 209GiB 208GiB 62.5MiB 961MiB 295GiB 41.42 0.90 89 0 hdd 0.49179 1.00000 504GiB 253GiB 252GiB 55.7MiB 968MiB 251GiB 50.18 1.09 76 1 hdd 0.49179 1.00000 504GiB 199GiB 198GiB 60.4MiB 964MiB 305GiB 39.51 0.86 92 16 hdd 0.49179 1.00000 504GiB 219GiB 218GiB 58.2MiB 966MiB 284GiB 43.51 0.95 85 17 hdd 0.49179 1.00000 504GiB 231GiB 230GiB 69.0MiB 955MiB 272GiB 45.97 1.00 97 14 hdd 0.49179 1.00000 504GiB 210GiB 209GiB 61.0MiB 963MiB 293GiB 41.72 0.91 74 15 hdd 0.49179 1.00000 504GiB 182GiB 181GiB 50.7MiB 973MiB 322GiB 36.10 0.79 72 18 hdd 0.49179 1.00000 504GiB 297GiB 296GiB 53.7MiB 978MiB 206GiB 59.03 1.29 87 19 hdd 0.49179 1.00000 504GiB 125GiB 124GiB 61.9MiB 962MiB 379GiB 24.81 0.54 82 TOTAL 10.8TiB 4.97TiB 4.94TiB 1.33GiB 21.4GiB 5.85TiB 45.91 MIN/MAX VAR: 0.54/1.91 STDDEV: 12.80 Is it a normal situation ? Is there any way to let Ceph handle this alone or am I forced to reweight the OSD manually ? Thank you. Florent

2 1

Re: Memory usage of OSD
by Mark Nelson 14 May '20

14 May '20

Coincidentally Adam on our core team just reported this morning that he saw extremely high bluestore_cache_other memory usage while running compression performance tests as well. That may indicate we have a memory leak related to the compression code. I doubt setting the memory_target to 3GiB will help in the long run as that will just attempt to compensate by decreasing the other caches until nothing else can be shrunk. Adam said he's planning to investigate so hopefully we will know more soon. Mark On 5/13/20 10:52 AM, Rafał Wądołowski wrote: > Mark, > Unfortunetly I closed terminal with mempool. But there was a lot of > bytes used by bluestore_cache_other. That was the highest value (about > 85%). The onode cache takes about 10%. PGlog and osdmaps was okey, low > values. I saw some ideas that maybe compression_mode force in pool can > make a mess. > One more thing, we are running stupid allocator. Right now I am > decrease the osd_memory_target to 3GiB and will wait if ram problem > occurs. > > > > Regards, > > */Rafał Wądołowski/* > > ------------------------------------------------------------------------ > *From:* Mark Nelson <mnelson(a)redhat.com> > *Sent:* Wednesday, May 13, 2020 3:30 PM > *To:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> > *Subject:* [ceph-users] Re: Memory usage of OSD > On 5/13/20 12:43 AM, RafaĹ‚ WÄ…doĹ‚owski wrote: > > Hi, > > I noticed strange situation in one of our clusters. The OSD deamons > are taking too much RAM. > > We are running 12.2.12 and have default configuration of > osd_memory_target (4GiB). > > Heap dump shows: > > > > osd.2969 dumping heap profile now. > > ------------------------------------------------ > > MALLOC: 6381526944 ( 6085.9 MiB) Bytes in use by application > > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist > > MALLOC: + 173373288 ( 165.3 MiB) Bytes in central cache freelist > > MALLOC: + 17163520 ( 16.4 MiB) Bytes in transfer cache freelist > > MALLOC: + 95339512 ( 90.9 MiB) Bytes in thread cache freelists > > MALLOC: + 28995744 ( 27.7 MiB) Bytes in malloc metadata > > MALLOC: ------------ > > MALLOC: = 6696399008 ( 6386.2 MiB) Actual memory used (physical + > swap) > > MALLOC: + 218267648 ( 208.2 MiB) Bytes released to OS (aka unmapped) > > MALLOC: ------------ > > MALLOC: = 6914666656 ( 6594.3 MiB) Virtual address space used > > MALLOC: > > MALLOC: 408276 Spans in use > > MALLOC: 75 Thread heaps in use > > MALLOC: 8192 Tcmalloc page size > > ------------------------------------------------ > > Call ReleaseFreeMemory() to release freelist memory to the OS (via > madvise()). > > Bytes released to the OS take up virtual address space but no > physical memory. > > > > IMO "Bytes in use by application" should be less than > osd_memory_target. Am I correct? > > I checked heap dump with google-pprof and got following results. > > Total: 149.4 MB > > 60.5 40.5% 40.5% 60.5 40.5% > rocksdb::UncompressBlockContentsForCompressionType > > 34.2 22.9% 63.4% 34.2 22.9% > ceph::buffer::create_aligned_in_mempool > > 11.9 7.9% 71.3% 12.1 8.1% > std::_Rb_tree::_M_emplace_hint_unique > > 10.7 7.1% 78.5% 71.2 47.7% rocksdb::ReadBlockContents > > > > Does it mean that most of RAM is used by rocksdb? > > > It looks like your heap dump is only accounting for 149.4MB of the > memory so probably not representative across the whole ~6.5G. Instead > could you try dumping the mempools via "ceph daemon osd.2969 > dump_mempools"? > > > > > > How can I take a deeper look into memory usage ? > > > Beyond looking at the mempools, you can see the bluestore cache > allocation information by either enabling debug bluestore and debug > priority_cache_manager 5, or potentially looking at the PCM perf > counters (I'm not sure if those were in 14.2.12 though). Between the > heap data, mempool data, and priority cache records, it should become > clearer what's going on. > > > Mark > > > > > > > > Regards, > > > > RafaĹ‚ WÄ…doĹ‚owski > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 3

ceph osd set-require-min-compat-client jewel failure
by 潘东元 14 May '20

14 May '20

hi,every one, my ceph version 12.2.12，I want to set require min compat client luminous,I use command #ceph osd set-require-min-compat-client luminous but ceph report:Error EPERM: cannot set require_min_compat_client to luminous: 4 connected client(s) look like jewel (missing 0xa00000000200000); add --yes-i-really-mean-it to do it anyway [root@node-1 ~]# ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 15 } }, "client": { "group": { "features": "0x40106b84a842a52", "release": "jewel", "num": 4 }, "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 168 } } } so,I run command: [root@node-1 gyt]# ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it set require_min_compat_client to luminous but now,I want to set require min compat client jewel,I use command： [root@node-1 gyt]# ceph osd set-require-min-compat-client jewel Error EPERM: osdmap current utilizes features that require luminous; cannot set require_min_compat_client below that to jewel what‘s the way we are set luminous chang to jewel？

3 2

What is ceph doing after sync
by Zhenshi Zhou 14 May '20

14 May '20

Hi, I deployed a multi-site in order to sync data from a cluster to anther. The data is fully synced(I suppose) and the cluster has no traffic at present. Everything seems fine. However, the sync status is not what I expected. Is there any step after data transfer? Can I change the master zone to my new zone? Can I stop the old cluster? sudo radosgw-admin sync status realm bde4bb56-fbca-4ef8-a979-935dbf109b78 (cn) zonegroup d25ae683-cdb8-4227-be45-ebaf0aed6050 (beijing) zone 313c8244-fe4d-4d46-bf9b-0e33e46be041 (newzone) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: f70a5eb9-d88d-42fd-ab4e-d300e97094de (oldzone) syncing full sync: 1/128 shards full sync: 0 buckets to sync incremental sync: 127/128 shards data is behind on 14 shards behind shards: [3,21,42,54,55,62,71,75,92,95,104,106,108,122]

1 0

Re: Memory usage of OSD
by Mark Nelson 13 May '20

13 May '20

On 5/13/20 12:43 AM, RafaĹ WÄdoĹowski wrote: > Hi, > I noticed strange situation in one of our clusters. The OSD deamons are taking too much RAM. > We are running 12.2.12 and have default configuration of osd_memory_target (4GiB). > Heap dump shows: > > osd.2969 dumping heap profile now. > ------------------------------------------------ > MALLOC: 6381526944 ( 6085.9 MiB) Bytes in use by application > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist > MALLOC: + 173373288 ( 165.3 MiB) Bytes in central cache freelist > MALLOC: + 17163520 ( 16.4 MiB) Bytes in transfer cache freelist > MALLOC: + 95339512 ( 90.9 MiB) Bytes in thread cache freelists > MALLOC: + 28995744 ( 27.7 MiB) Bytes in malloc metadata > MALLOC: ------------ > MALLOC: = 6696399008 ( 6386.2 MiB) Actual memory used (physical + swap) > MALLOC: + 218267648 ( 208.2 MiB) Bytes released to OS (aka unmapped) > MALLOC: ------------ > MALLOC: = 6914666656 ( 6594.3 MiB) Virtual address space used > MALLOC: > MALLOC: 408276 Spans in use > MALLOC: 75 Thread heaps in use > MALLOC: 8192 Tcmalloc page size > ------------------------------------------------ > Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). > Bytes released to the OS take up virtual address space but no physical memory. > > IMO "Bytes in use by application" should be less than osd_memory_target. Am I correct? > I checked heap dump with google-pprof and got following results. > Total: 149.4 MB > 60.5 40.5% 40.5% 60.5 40.5% rocksdb::UncompressBlockContentsForCompressionType > 34.2 22.9% 63.4% 34.2 22.9% ceph::buffer::create_aligned_in_mempool > 11.9 7.9% 71.3% 12.1 8.1% std::_Rb_tree::_M_emplace_hint_unique > 10.7 7.1% 78.5% 71.2 47.7% rocksdb::ReadBlockContents > > Does it mean that most of RAM is used by rocksdb? It looks like your heap dump is only accounting for 149.4MB of the memory so probably not representative across the whole ~6.5G. Instead could you try dumping the mempools via "ceph daemon osd.2969 dump_mempools"? > > How can I take a deeper look into memory usage ? Beyond looking at the mempools, you can see the bluestore cache allocation information by either enabling debug bluestore and debug priority_cache_manager 5, or potentially looking at the PCM perf counters (I'm not sure if those were in 14.2.12 though). Between the heap data, mempool data, and priority cache records, it should become clearer what's going on. Mark > > > Regards, > > RafaĹ WÄdoĹowski > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Erasure coded pool queries
by Biswajeet Patra 13 May '20

13 May '20

Hi, I have created an erasure coded pool and the below default parameters related to stripe sizes are present. "osd_pool_erasure_code_stripe_width": "4096" --> 4KB "rgw_obj_stripe_size": "4194304" --> 4MB Let say the k+m values are 10+5 for the erasure pool, and we upload an object of let say, size <4MB & another object of size >4MB. How will ceph break the object into chunks and store it & how many stripes will be created? Regards, Biswajeet -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

2 1

Disproportionate Metadata Size
by Denis Krienbühl 13 May '20

13 May '20

Hi On one of our Ceph clusters, some OSDs have been marked as full. Since this is a staging cluster that does not have much data on it, this is strange. Looking at the full OSDs through “ceph osd df” I figured out that the space is mostly used by metadata: SIZE: 122 GiB USE: 118 GiB DATA: 2.4 GiB META: 116 GiB We run mimic, and for the affected OSDs we use a db device (nvme) in addition to the primary device (hdd). In the logs we see the following errors: 2020-05-12 17:10:26.089 7f183f604700 1 bluefs _allocate failed to allocate 0x400000 on bdev 1, free 0x0; fallback to bdev 2 2020-05-12 17:10:27.113 7f183f604700 1 bluestore(/var/lib/ceph/osd/ceph-8) _balance_bluefs_freespace gifting 0x180a000000~400000 to bluefs 2020-05-12 17:10:27.153 7f183f604700 1 bluefs add_block_extent bdev 2 0x180a000000~400000 We assume it is an issue with Rocksdb, as the following call will quickly fix the problem: ceph daemon osd.8 compact The question is, why is this happening? I would think that “compact" is something that runs automatically from time to time, but I’m not sure. Is it on us to run this regularly? Any pointers are welcome. I’m quite new to Ceph :) Cheers, Denis

3 3

Difficulty creating a topic for bucket notifications
by Alexis Anand 13 May '20

13 May '20

Hi, I am trying to create a topic so that I can use it to listen for object creation notifications on a bucket. If I make my API call without supplying AWS authorization headers, the topic creation succeeds, and it can be seen by using a ListTopics call. However, in order to attach a topic to a bucket, the topic and bucket must have the same owner. So I tried creating a topic using AWS auth. The credential header I tried was the same as what I use for get/put items to a bucket: Credential=<access key id>/20200512/us-east-1/s3/aws4_request However in this case rather than succeeding I get a NotImplemented error. If I tried changing the service in the credential to something other than s3, like "Credential=<access key id>/20200512/us-east-1/s3/aws4_request" I instead get a SignatureDoesNotMatch error. What is the right way to authenticate a CreateTopic request? Thanks, Alexis

2 1

Memory usage of OSD
by Rafał Wądołowski 13 May '20

13 May '20

Hi, I noticed strange situation in one of our clusters. The OSD deamons are taking too much RAM. We are running 12.2.12 and have default configuration of osd_memory_target (4GiB). Heap dump shows: osd.2969 dumping heap profile now. ------------------------------------------------ MALLOC: 6381526944 ( 6085.9 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 173373288 ( 165.3 MiB) Bytes in central cache freelist MALLOC: + 17163520 ( 16.4 MiB) Bytes in transfer cache freelist MALLOC: + 95339512 ( 90.9 MiB) Bytes in thread cache freelists MALLOC: + 28995744 ( 27.7 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 6696399008 ( 6386.2 MiB) Actual memory used (physical + swap) MALLOC: + 218267648 ( 208.2 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 6914666656 ( 6594.3 MiB) Virtual address space used MALLOC: MALLOC: 408276 Spans in use MALLOC: 75 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. IMO "Bytes in use by application" should be less than osd_memory_target. Am I correct? I checked heap dump with google-pprof and got following results. Total: 149.4 MB 60.5 40.5% 40.5% 60.5 40.5% rocksdb::UncompressBlockContentsForCompressionType 34.2 22.9% 63.4% 34.2 22.9% ceph::buffer::create_aligned_in_mempool 11.9 7.9% 71.3% 12.1 8.1% std::_Rb_tree::_M_emplace_hint_unique 10.7 7.1% 78.5% 71.2 47.7% rocksdb::ReadBlockContents Does it mean that most of RAM is used by rocksdb? How can I take a deeper look into memory usage ? Regards, Rafał Wądołowski

1 0

Read speed low in cephfs volume exposed as samba share using vfs_ceph
by Amudhan P 13 May '20

13 May '20

Hi, I am running a small 3 node Ceph Nautilus 14.2.8 cluster on Ubuntu 18.04. I am testing cluster to expose cephfs volume in samba v4 share for the user to access from windows for latter use. Samba Version 4.7.6-Ubuntu and mount.cifs version: 6.8. When I did a test with DD Write (600 MB/s) and md5sum file Read speed is (300 - 400 MB/s) from ceph kernel mount. The same volume I have exposed in samba using "vfs_ceph" and mounted it through CIFS in another ubuntu18.04 as client. Now, when I perform DD write I get the speed of 600 MB/s and md5sum of file Read speed is only 65 MB/s. There is a different result when I try to read the same file using smbclinet getting the speed of 101 MB/s. Why is this difference what could be the issue?

1 0

Re: RGW STS Support in Nautilus ?
by Pritha Srivastava 13 May '20

13 May '20

app_id must match with the 'aud' field in the token introspection result (In the example the value of 'aud' is 'customer-portal') Thanks, Pritha On Tue, May 12, 2020 at 8:16 PM Wyllys Ingersoll < wyllys.ingersoll(a)keepertech.com> wrote: > > Running Nautilus 14.2.9 and trying to follow the STS example given here: > https://docs.ceph.com/docs/master/radosgw/STS/ to setup a policy > for AssumeRoleWithWebIdentity using KeyCloak (8.0.1) as the OIDC provider. > I am able to see in the rgw debug logs that the token being passed from the > client is passing the introspection check, but it always ends up failing > the final authorization to access the requested bucket resource and is > rejected with a 403 status "AccessDenied". > > I configured my policy as described in the 2nd example on the STS page > above. I suspect the problem is with the "StringEquals" condition statement > in the AssumeRolePolicy document (I could be wrong though). > > The example shows using the keycloak URI followed by ":app_id" matching > with the name of the keycloak client application ("customer-portal" in the > example). My keycloak setup does not have any such field in the > introspection result and I can't seem to figure out how to make this all > work. > > I cranked up the logging to 20/20 and still did not see any hints as to > what part of the policy is causing the access to be denied. > > Any suggestions? > > -Wyllys Ingersoll > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io >

3 5

data increase after multisite syncing
by Zhenshi Zhou 13 May '20

13 May '20

Hi, I deployed a multisite in order to sync data from a mimic cluster zone to a nautilus cluster zone. The data sync well at present. However, I check the cluster status and I find something strange. The data in my new cluster seems larger than that in old ones. The data is far from full synced while the space used is nearly the same. Does that normal? 'ceph df ' on old cluster: GLOBAL: SIZE AVAIL RAW USED %RAW USED 82 TiB 41 TiB 41 TiB 50.37 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS .rgw.root 1 6.0 KiB 0 10 TiB 19 default.rgw.control 2 0 B 0 10 TiB 8 default.rgw.meta 3 3.5 KiB 0 10 TiB 19 default.rgw.log 4 8.4 KiB 0 10 TiB 1500 default.rgw.buckets.index 5 0 B 0 10 TiB 889 default.rgw.buckets.non-ec 6 0 B 0 10 TiB 497 default.rgw.buckets.data 7 14 TiB 56.96 10 TiB 3968545 testpool 8 0 B 0 10 TiB 0 'ceph df ' on new cluster: RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 137 TiB 98 TiB 38 TiB 38 TiB 28.02 TOTAL 137 TiB 98 TiB 38 TiB 38 TiB 28.02 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL .rgw.root 1 6.4 KiB 21 3.8 MiB 0 26 TiB shubei.rgw.control 13 0 B 8 0 B 0 26 TiB shubei.rgw.meta 14 4.1 KiB 20 3.2 MiB 0 26 TiB shubei.rgw.log 15 9.9 MiB 1.64k 47 MiB 0 26 TiB default.rgw.meta 16 0 B 0 0 B 0 26 TiB shubei.rgw.buckets.index 17 2.7 MiB 889 2.7 MiB 0 26 TiB shubei.rgw.buckets.data 18 11 TiB 2.90M 33 TiB 29.37 26 TiB 'radosgw-admin sync status' on new cluster: realm bde4bb56-fbca-4ef8-a979-935dbf109b78 (new-oriental) zonegroup d25ae683-cdb8-4227-be45-ebaf0aed6050 (beijing) zone 313c8244-fe4d-4d46-bf9b-0e33e46be041 (shubei) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: f70a5eb9-d88d-42fd-ab4e-d300e97094de (oldzone) syncing full sync: 106/128 shards full sync: 350 buckets to sync incremental sync: 22/128 shards data is behind on 115 shards behind shards: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,23,24,25,26,27,28,29,30,32,35,37,38,39,40,41,42,43,44,45,46,47,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,96,97,98,99,100,101,102,103,104,105,107,108,109,110,111,112,113,114,116,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2020-05-11 10:46:41.0.60179s [80] 5 shards are recovering recovering shards: [21,31,95,104,106]

1 1

OSD corruption and down PGs
by Kári Bertilsson 12 May '20

12 May '20

Hello I had an incidence where 3 OSD's crashed at once completely and won't power up. And during recovery 3 OSD's in another host have somehow become corrupted. I am running erasure coding with 8+2 setup using crush map which takes 2 OSDs per host, and after losing the other 2 OSD i have few PG's down. Unfortunately these PG's seem to overlap almost all data on the pool, so i believe the entire pool is mostly lost after only these 2% of PG's down. I am running ceph 14.2.9. OSD 92 log https://pastebin.com/5aq8SyCW OSD 97 log https://pastebin.com/uJELZxwr ceph-bluestore-tool repair without --deep showed "success" but OSD's still fail with the log above. Log from trying ceph-bluestore-tool repair --deep which is still running, not sure if it will actually fix anything and log looks pretty bad. https://pastebin.com/gkqTZpY3 Trying "ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-97 --op list" gave me input/output error. But everything in SMART looks OK, and i see no indication of hardware read error in any logs. Same for both OSD. The OSD's with corruption have absolutely no bad sectors and likely have only a minor corruption but at important locations. Any ideas on how to recover this kind of scenario ? Any tips would be highly appreciated. Best regards, Kári Bertilsson

4 6

DocuBetter Meeting -- EMEA 13 May 2020
by John Zachary Dover 12 May '20

12 May '20

There is a general documentation meeting called the "DocuBetter Meeting", and it is held every two weeks. The next DocuBetter Meeting will be on 13 May 2020 at 0830 PST, and will run for thirty minutes. Everyone with a documentation-related request or complaint is invited. The meeting will be held here: https://bluejeans.com/908675367 Send documentation-related requests and complaints to me by replying to this email and CCing me at zac.dover(a)gmail.com. The next DocuBetter meeting is scheduled for: 13 May 2020 0830 PST 13 May 2020 1630 UTC 14 May 2020 0230 AEST Etherpad: https://pad.ceph.com/p/Ceph_Documentation Meeting: https://bluejeans.com/908675367 Thanks, everyone. Zac Dover

2 1

Unable to reshard bucket
by Timothy Geier 12 May '20

12 May '20

Hello all, I'm having an issue with a bucket that refuses to be resharded..for the record, the cluster was recently upgraded from 13.2.4 to 13.2.10. # radosgw-admin reshard add --bucket foo --num-shards 3300 ERROR: the bucket is currently undergoing resharding and cannot be added to the reshard list at this time # radosgw-admin reshard list [] # radosgw-admin reshard status --bucket=foo [ { "reshard_status": "not-resharding", "new_bucket_instance_id": "", "num_shards": -1 }, <snip> # radosgw-admin reshard cancel --bucket foo ERROR: failed to remove entry from reshard log, oid=reshard.0000000009 tenant= bucket=foo # radosgw-admin reshard stale-instances list [] Is there anything else I should check to troubleshoot this? I was able to reshard another bucket since the upgrade, so I suspect there's something lingering that's blocking this.

2 2

Ceph Apply/Commit vs Read/Write Op Latency
by John Petrini 12 May '20

12 May '20

Hello, I was hoping someone could clear up the difference between these metrics. In filestore the difference between Apply and Commit Latency was pretty clear and these metrics gave a good representation of how the cluster was performing. High commit usually meant our journals were performing poorly while high apply pointed to an OSD issue. With bluestore Apply & Commit are now tied to the same metric and it's not as clear to me what that metric is. In addition new metrics such as Read and Write Op Latency have been added. I'm led to believe that these are similar to what Apply Latency used to represent but is that actually the case? If anyone who has a better understanding of this than I do can enlighten me I'd appreciate it! Thanks, John

1 1

Add lvm in cephadm
by Simon Sutter 12 May '20

12 May '20

Hello Everyone, The new cephadm is giving me a headache. I'm setting up a new testenvironment, where I have to use lvm partitions, because I don't have more Hardware. I could't find any information about the compatibility of existing lvm partitions and cephadm/octopus. I tried the old method, where I had to use ceph-volume to use an existing lvm:

2 3

rgw user access questions
by Vishwas Bm 12 May '20

12 May '20

Hi, I am a newbie to ceph. I have gone through the ceph docs, we are planning to use rgw for object storage. From the docs, what I have understood is that there are two types of users: 1) ceph storage user 2) radosgw user I am able to create user of both the types. But I am not able to understand how to restrict the rgw user access to a pool. My questions are below: 1) How to restrict the access of a rgw user to a particular pool ? Can this be done using placement groups ? 2) Is it possible to restrict rgw user access to a particular namespace in a pool ? 3) I can understand the flow till he is able to write to a bucket using the .index pool object. But I am not able to understand the flow how the rgw user can write objects in pool. Where can I check the permissions ? *Thanks & Regards,* *Vishwas *

2 2

Re: Zeroing out rbd image or volume
by Marc Roos 12 May '20

12 May '20

dd if=/dev/zero of=rbd ???? :) but if you have encrypted osd's, what would be the use of this? -----Original Message----- From: huxiaoyu(a)horebdata.cn [mailto:huxiaoyu@horebdata.cn] Sent: 12 May 2020 12:55 To: ceph-users Subject: [ceph-users] Zeroing out rbd image or volume Hi， Ceph folks, Is there a rbd command, or any other way, to zero out rbd images or volume? I would like to write all zero data to an rbd image/volume before remove it. Any comments would be appreciated. best regards, samuel Horebdata AG Switzerland huxiaoyu(a)horebdata.cn _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Zeroing out rbd image or volume
by huxiaoyu＠horebdata.cn 12 May '20

12 May '20

Hi， Ceph folks, Is there a rbd command, or any other way, to zero out rbd images or volume? I would like to write all zero data to an rbd image/volume before remove it. Any comments would be appreciated. best regards, samuel Horebdata AG Switzerland huxiaoyu(a)horebdata.cn

1 0

Re: Write Caching to hot tier not working as expected
by Steve Hughes 12 May '20

12 May '20

Thanks Eric. Using your command for SET reported that the OSD may need a restart (which sets it back to default anyway) but the below seems to work: ceph tell osd.24 config set objecter_inflight_op_bytes 1073741824 ceph tell osd.24 config set objecter_inflight_ops 10240 reading back the settings looks right: [root@ceph00 ~]# ceph daemon osd.24 config show | grep objecter "debug_objecter": "0/1", "objecter_completion_locks_per_session": "32", "objecter_debug_inject_relock_delay": "false", "objecter_inflight_op_bytes": "1073741824", "objecter_inflight_ops": "10240", "objecter_inject_no_watch_ping": "false", "objecter_retry_writes_after_first_reply": "false", "objecter_tick_interval": "5.000000", "objecter_timeout": "10.000000", "osd_objecter_finishers": "1", Ive done that for the three OSDs that are in the cache tier. But the performance is unchanged - the writes still spill over to the HDD pool. Still, your idea sounds close - it does feel like something in the cache tier is hitting a limit. Regards, Steve -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Monday, 11 May 2020 9:11 PM To: Steve Hughes <steveh(a)scalar.com.au>; ceph-users(a)ceph.io Subject: RE: [ceph-users] Re: Write Caching to hot tier not working as expected Reading and setting them should be pretty easy: READ (Run from the host where OSD <id> is hosted): ceph daemon osd.<id> config show | grep objecter SET (Assuming these can be set in memory): ceph tell osd.<id> injectargs "--objecter-inflight-op-bytes=1073741824" (Change to 1GB/sec throttle) To persist these you should add them to the ceph.conf (I'm not sure what section though - you might have to test this). And yes - the information is sketchy I agree - I don't really have any input here. That's the best I can do for now 😊 Eric -----Original Message----- From: Steve Hughes <steveh(a)scalar.com.au> Sent: Monday, May 11, 2020 6:44 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users(a)ceph.io Subject: RE: [ceph-users] Re: Write Caching to hot tier not working as expected Thank you Eric. That 'sounds like' exactly my issue. Though I'm surprised to bump into something like that on such a small system and at such low bandwidth. But the information I can find on those parameters is sketchy to say the least. Can you point me at some doco that explains what they do, how to read the current values and how to set them? Cheers, Steve -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Monday, 11 May 2020 8:00 PM To: Steve Hughes <steveh(a)scalar.com.au>; ceph-users(a)ceph.io Subject: RE: [ceph-users] Re: Write Caching to hot tier not working as expected It sounds like you might be bumping up against the default objecter_inflight_ops (1024) and/or objecter_inflight_op_bytes (100MB). -----Original Message----- From: steveh(a)scalar.com.au <steveh(a)scalar.com.au> Sent: Monday, May 11, 2020 5:48 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Write Caching to hot tier not working as expected Interestingly, I have found that if I limit the rate at which data is written the tiering behaves as expected. I'm using a robocopy job from a Windows VM to copy large files from my existing storage array to a test Ceph volume. By using the /IPG parameter I can roughly control the rate at which data is written. I've found that if I limit the write rate to around 30MBytes/sec the data all goes to the hot tier, zero data goes to the HDD tier, and the observed write latency is about 5msec. If I go any higher than this I see data being written to the HDDs and the observed write latency goes way up. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io --

1 0

Cifs slow read speed
by Amudhan P 12 May '20

12 May '20

Hi, I am running a small Ceph Nautilus cluster on Ubuntu 18.04. I am testing cluster to expose cephfs volume in samba v4 share for user to access from windows. When i do test with DD Write (600 MB/s) and md5sum file Read speed is (700 - 800 MB/s) from ceph kernel mount. Same volume i have exposed in samba using "vfs_ceph" and mounted it thru cifs in another ubuntu18.04 as client. Now, when i perform DD write i get speed of 600 MB/s and md5sum of file Read speed is only 65 MB/s. What could be the problem? Any one faced similar issue?

1 0

Re: Elasticsearch Sync module bug ?
by Cervigni, Luca (Pawsey, Kensington WA) 12 May '20

12 May '20

Thanks for the reply I moved the 6.8.8 and the JSON parsing error seems gone. I am trying to do a simple request though I have very weird results: --- Request URL = https://xxx.xxx.xxx.xxx:6666/test2?query=name%3D%3Dfile-020 {'Host': 'xxx.xxx.xxx', 'Content-length': '0', 'X-Amz-Content-SHA256': 'UNSIGNED-PAYLOAD', 'X-Amz-Date': '20200512T053711Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=xxxxxxxxxxxxxxxxxx/20200512/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'} Response Response code: 200 <SearchMetadataResponse><Marker></Marker><IsTruncated>false</IsTruncated></SearchMetadataResponse> SearchMetadataResponse: None Marker: None IsTruncated: false --- ________________________________ From: Cervigni, Luca (Pawsey, Kensington WA) <Luca.Cervigni(a)csiro.au> Sent: Monday, 11 May 2020 4:34 PM To: Peter Parker <346415320(a)qq.com>; Luca Cervigni <luca.cervigni(a)pawsey.org.au> Subject: Re: Re:[ceph-users] Elasticsearch Sync module bug ? Thanks Peter for your reply. Could you point me out to the correct settings to put? I tried as well ES 6.8 without success. I just use the default docker container that the ES documentations points out. Could you give me a little more details? Many thanks Regards, Luca Cervigni ________________________________ From: Peter Parker <346415320(a)qq.com> Sent: Monday, 11 May 2020 4:00 PM To: Luca Cervigni <luca.cervigni(a)pawsey.org.au> Subject: Re:[ceph-users] Elasticsearch Sync module bug ? Hi, Luca In my env, ES 7.x is not a valid metadata tier by default for reasons such as index type. but it could be fixed by changing default setting. (I haven't tried) I use ES 6.8 instead and it works fine. ------------------ Original ------------------ From: "Luca Cervigni"<luca.cervigni(a)pawsey.org.au>; Date: Mon, May 11, 2020 12:27 PM To: "ceph-users"<ceph-users(a)ceph.io>; Subject: [ceph-users] Elasticsearch Sync module bug ? Hello, I have a two zones cluster. Default zone + a ES sync zone where I would like to index metadata and do searches via the ES tier gateway. Struggling to get this service running (documentation is bad and very, very minimal) I thought that in the end I made it work. The requests are getting correctly to the ES zone, I can see the payload sent to ES with debug rgw = 20, ES returns OK then an error is trigger, apparently a JSON parsing error taking the ES search and giving it back to the user. I tried with Nautilus, Octopus, and different versions of ES, from 6 to 7. I have no possibilities of downgrades as our production cluster is on nautilus. Here the logs: https://pastebin.com/hnj8YysK I would think this is a bug, but would like your opinion. If not a bug how shall I solve this? I am using a handmade S3 tool to query as boto is not supported anymore, obo just does not work, and boto3 cannot send any arbitrary requests anymore. Also I would like to know if those Sync modules are still supported or not. The documentation is very poor and if they still are in the roadmap, it should be improved. Thanks Cheers, Luca Cervigni Pawsey Supercomputing centre _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

1 pg unknown (from cephfs data pool)
by Marc Roos 11 May '20

11 May '20

I had a 1x replicated fs data test pool, when a osd died I had '1 pg stale+active+clean of cephfs'[1] after a cluster reboot this turned into '1 pg unknown' ceph pg repair did not fix anything (for stale and unknown state) I recreated the pg with: ceph osd force-create-pg pg.id --yes-i-really-mean-it Question now is: Say I had one or two files in this pool/pg, is this still administered in the mds server? Do I need to fix something in the mds? PS. This was just performance testing pool, so at most there could be a few testing images on it, nothing important. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg03147.html

1 0

MDS_CACHE_OVERSIZED warning
by jesper＠krogh.cc 11 May '20

11 May '20

Hi. I have a cluster that has been running for close to 2 years now - pretty much with the same setting, but over the past day I'm seeing this warning. (and the cache seem to keep growing) - Can I figure out which clients is accumulating the inodes? Ceph 12.2.8 - is it ok just to "bump" the memory to say 128GB - any negative sideeffects? jk@ceph-mon1:~$ sudo ceph health detail HEALTH_WARN 1 MDSs report oversized cache; 3 clients failing to respond to cache pressure MDS_CACHE_OVERSIZED 1 MDSs report oversized cache mdsceph-mds1(mds.0): MDS cache is too large (91GB/32GB); 34400070 inodes in use by clients, 3293 stray files Thanks - Jesper

2 1

Yet another meltdown starting
by Frank Schilder 11 May '20

11 May '20

Hi all, another client-load induced meltdown. It is just starting and I hope we get it under control. This time, its the MGRs failing under the load. It looks like thay don't manage to get their beacons to the mons and are kicked out as unresponsive. However, the processes are fine and up. Its just an enormous load. I'm trying to increase # ceph config set global mon_mgr_beacon_grace 90 but the command doesn't complete. I guess because all the MGRs are out. Is there any way to force the MONs *not* to mark MGRs as unresponsive? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 4

Recover datas from pg incomplete
by Francois Legrand 11 May '20

11 May '20

Hi, After a major crash in which we lost few osds, we are stucked with incomplete pgs. At first, peering was blocked with peering_blocked_by_history_les_bound. Thus we set osd_find_best_info_ignore_history_les true for all osds involved in the pg and set the primary osd down to force repeering. It worked for one pg which is in a replica 3 pool, but for the 2 others pgs which are in a erasurce coding (3+2) pool, it didn't worked... and the pgs are still incomplete. We know that we will have data lost, but we would like to minimize it and save as much as possible. Also because this pg is part of the data pool of a cephfs filesystem and it seems that files are spread among a lot of pgs and loosing objects in a pg of the datapool means the loss of a huge number of files ! According to https://www.spinics.net/lists/ceph-devel/msg41665.html a way would be to : - stop each osd involved in that pg - export the shards with ceph-objectstore-tool - compare the size of the shards and select the biggest one (alternatively maybe we can also look at the num_objects returned by ceph pg query ?) - Mark it as complete - restart the osd - Wait for recover and finally get rid of the missing objects with ceph pg 10.2 mark_unfound_lost delete But on this other source https://github.com/TheJJ/ceph-cheatsheet/blob/master/README.md or here https://medium.com/opsops/recovering-ceph-from-reduced-data-availability-3-… it's suggested to remove the other parts (but I am not sure these threads are really related to EC pools). Could you confirm that we could follow this procedure (or correct it or suggests anything else) ? Thanks for your advices. F. PS: Here is a part of the ceph pg 10.2 query return : "state": "incomplete", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 434321, "up": [ 78, 105, 90, 4, 41 ], "acting": [ 78, 105, 90, 4, 41 ], "info": { "pgid": "10.2s0", "state": "incomplete", "last_peered": "2020-04-22 09:58:42.505638", "last_became_peered": "2020-04-20 11:06:07.701833", "num_objects": 161314, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 161314, "num_objects_recovered": 1290285, "peer_info": [ "peer": "4(3)", "pgid": "10.2s3", "state": "active+undersized+degraded+remapped+backfilling", "last_peered": "2020-04-25 13:25:12.860435", "last_became_peered": "2020-04-22 10:45:45.520125", "num_objects": 162869, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 85071, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 162869, "num_objects_recovered": 1368082, "peer": "9(2)", "pgid": "10.2s2", "state": "down", "last_peered": "2020-04-25 13:25:12.860435", "last_became_peered": "2020-04-22 10:45:45.520125", "num_objects": 162869, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 162869, "num_objects_recovered": 1368082, "peer": "41(4)", "pgid": "10.2s4", "state": "unknown", "last_peered": "0.000000", "last_became_peered": "0.000000", "num_objects": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_objects_recovered": 0, "peer": "46(4)", "pgid": "10.2s4", "state": "down", "last_peered": "2020-04-25 13:25:12.860435", "last_became_peered": "2020-04-22 10:45:45.520125", "num_objects": 162869, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 162869, "num_objects_recovered": 1368082, "peer": "52(3)", "pgid": "10.2s3", "state": "unknown", "last_peered": "0.000000", "last_became_peered": "0.000000", "num_objects": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_objects_recovered": 0, "peer": "69(0)", "pgid": "10.2s0", "state": "unknown", "last_peered": "0.000000", "last_became_peered": "0.000000", "num_objects": 84063, "num_objects_missing_on_primary": 0, "num_objects_missing": 78807, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 84063, "num_objects_recovered": 0, "peer": "90(2)", "pgid": "10.2s2", "state": "down", "last_peered": "0.000000", "last_became_peered": "0.000000", "num_objects": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_objects_recovered": 0, "peer": "105(1)", "pgid": "10.2s1", "state": "incomplete", "last_peered": "0.000000", "last_became_peered": "0.000000", "num_objects": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_objects_recovered": 0, "recovery_state": [ "peering_blocked_by": [] "agent_state": {}

1 0

adding block.db to OSD
by Stefan Priebe - Profihost AG 11 May '20

11 May '20

Hello, is there anything else needed beside running: ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-${OSD} bluefs-bdev-new-db --dev-target /dev/vgroup/lvdb-1 I did so some weeks ago and currently i'm seeing that all osds originally deployed with --block-db show 10-20% I/O waits while all those got converted using ceph-bluestore-tool show 80-100% I/O waits. Also is there some tuning available to use more of the SSD? The SSD (block-db) is only saturated at 0-2%. Greets, Stefan

2 21

Re: Write Caching to hot tier not working as expected
by Steve Hughes 11 May '20

11 May '20

Thank you Eric. That 'sounds like' exactly my issue. Though I'm surprised to bump into something like that on such a small system and at such low bandwidth. But the information I can find on those parameters is sketchy to say the least. Can you point me at some doco that explains what they do, how to read the current values and how to set them? Cheers, Steve -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Monday, 11 May 2020 8:00 PM To: Steve Hughes <steveh(a)scalar.com.au>; ceph-users(a)ceph.io Subject: RE: [ceph-users] Re: Write Caching to hot tier not working as expected It sounds like you might be bumping up against the default objecter_inflight_ops (1024) and/or objecter_inflight_op_bytes (100MB). -----Original Message----- From: steveh(a)scalar.com.au <steveh(a)scalar.com.au> Sent: Monday, May 11, 2020 5:48 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Write Caching to hot tier not working as expected Interestingly, I have found that if I limit the rate at which data is written the tiering behaves as expected. I'm using a robocopy job from a Windows VM to copy large files from my existing storage array to a test Ceph volume. By using the /IPG parameter I can roughly control the rate at which data is written. I've found that if I limit the write rate to around 30MBytes/sec the data all goes to the hot tier, zero data goes to the HDD tier, and the observed write latency is about 5msec. If I go any higher than this I see data being written to the HDDs and the observed write latency goes way up. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io --

1 0

ceph-volume/batch fails in non-interactive mode
by Michał Nasiadka 11 May '20

11 May '20

Hello, I stumbled across that weird behaviour today - after initially creating disks and DB LVs on NVMe (using ceph-ansible) - second run of ceph-volume ends up with an error: --> RuntimeError: 4 devices were filtered in non-interactive mode, bailing out What is interesting - the same command in interactive mode (without —yes) ends up without that error. I raised a bug in Tracker - https://tracker.ceph.com/issues/45461 <https://tracker.ceph.com/issues/45461> I found a PR fixing a similar bug here: https://github.com/ceph/ceph/pull/33202 <https://github.com/ceph/ceph/pull/33202> - is that related? Kind regards, Michal

1 0

Write Caching to hot tier not working as expected
by steveh＠scalar.com.au 11 May '20

11 May '20

Hi all, I'm a newbie to Ceph. I'm an MSP and a small-scale cloud hoster. I'm intending to use Ceph as production storage for a small-scale private hosting cloud. We run ESXi as our HVs so we want to present Ceph as iSCSI. We've got Ceph Nautilus running on a 3-node cluster. Each node contains a pair of Bronze XEONs, 128G RAM, 6 x 10G NICs, 8 x 10TB spinners, 2 x 2TB SATA SSDs and a 4TB NVME. The HDDs and the 2TB SSDs give me an rbd pool of 24 OSDs (1024 PGs) with the SSDs partitioned and used to hold the DB and WAL. Each SSD hold the DB/WAL for 4 HDDs. The NVMes give me a cache pool of 3 OSDs (128 PGs) which I want to use as a hot tier. As far as I can tell I have followed the guidance given here: https://docs.ceph.com/docs/nautilus/rados/operations/cache-tiering/ The cluster is working, the iSCSI is working, and generally everything is looking pretty good. My only problem at this stage is that the tiering is not handling writes in the way I expect, and I really can't get my brain around why. For my test I start with the hot tier empty. I drained it by setting dirty_ratio=dirty_high_ratio=full_ratio = 0. I then set dirty_ratio = 0.5, dirty_high_ratio = 0.6 and full_ratio = 0.7 and start writing data to it at high speed (using a simple file copy) from a VM on ESXi. My expectation is that all inbound writes will land initially on the hot tier, resulting in low latency writes as seen by the ESXi hosts. I expect that once the hot tier fills to 50% that it will start to flush these writes down to the HDD storage. What I actually see is that as soon as I start throwing data at the cluster, the Ceph dashboard shows writes going to both the NVMEs and to the HDDs, and the write latency seen by ESX hits several hundred milliseconds. It seems that the hot tier is absorbing only a fraction of the writes. Here are my pool settings. [root@ceph00 ~]# ceph osd pool ls detail pool 4 'rbd' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 6971 lfor 6971/6971/6971 flags hashpspool,selfmanaged_snaps tiers 11 read_tier 11 write_tier 11 stripe_width 0 application rbd removed_snaps [1~3] pool 11 'cache' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 7065 lfor 6971/6971/6971 flags hashpspool,incomplete_clones,selfmanaged_snaps tier_of 4 cache_mode writeback target_bytes 3078632557772 hit_set bloom{false_positive_probability: 0.001, target_size: 0, seed: 0} 3600s x12 decay_rate 0 search_last_n 0 min_read_recency_for_promote 2 stripe_width 0 application rbd removed_snaps [1~3] pool 12 'pure_hdd' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 7059 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~3] Is anyone able to point me towards a solution? Thanks, Steve

1 1

radosgw Swift bulk upload
by Martin Zurowietz 11 May '20

11 May '20

Hi, I'm interested in the Swift extract archive/bulk upload feature [1]. First I thought that this feature is not implemented in radosgw, as it is not mentioned anywhere in the docs [2] and some test requests of mine were not successful. However, while browsing the issue tracker, I found some issues that mention a bulk upload feature [3, 4]. Can someone please clarify whether this feature exists in radosgw or not? Thanks a lot, Martin [1]: https://docs.openstack.org/swift/latest/middleware.html#module-swift.common… [2]: https://docs.ceph.com/docs/master/radosgw/swift/ [3]: https://tracker.ceph.com/issues/19645 [4]: https://tracker.ceph.com/issues/8945 -- Biodata Mining Group Faculty of Technology Bielefeld University https://biodatamining.cebitec.uni-bielefeld.de/ Office: V10-103 +49 521 106 60 58

1 0

OSD Inbalance - upmap mode
by Ashley Merrick 11 May '20

11 May '20

I have a cluster running 15.2.1, was originally running 14.x, the cluster is running the balance module in upmap mode (I have tried crush-compat in the past) Most OSD's are around the same & used give or take 0.x, however there is one OSD that is down a good few % and a few that are above average by 1 or 2 %, I have been trying to get the balance to fix this. I have tried running a manual osdmaptool command on an export of my map, but it lists no fixed however does display the underfall OSD in it's output (overfull 3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,20 underfull [36]) The debug output is just lots of: 2020-05-05T06:15:39.172+0000 7f3dfb0c3c40 10 trying 2.55 2020-05-05T06:15:39.172+0000 7f3dfb0c3c40 10 2.55 [12,3,7,6,33,34,30,35,21,18] -> [12,3,7,6,33,34,30,35,21,16] 2020-05-05T06:15:39.172+0000 7f3dfb0c3c40 10 will try adding new remapping pair 18 -> 16 for 2.55 NOT selected osd 2020-05-05T06:15:39.172+0000 7f3dfb0c3c40 10 stddev 528.667 -> 528.667 2020-05-05T06:15:39.172+0000 7f3dfb0c3c40 10 Overfull search osd.7 target 170.667 deviation 9.33327 Is there anything I can to try and balance the overfull onto the underful OSDs to balance out the last bit.

1 1

Elasticsearch Sync module bug ?
by Luca Cervigni 11 May '20

11 May '20

Hello, I have a two zones cluster. Default zone + a ES sync zone where I would like to index metadata and do searches via the ES tier gateway. Struggling to get this service running (documentation is bad and very, very minimal) I thought that in the end I made it work. The requests are getting correctly to the ES zone, I can see the payload sent to ES with debug rgw = 20, ES returns OK then an error is trigger, apparently a JSON parsing error taking the ES search and giving it back to the user. I tried with Nautilus, Octopus, and different versions of ES, from 6 to 7. I have no possibilities of downgrades as our production cluster is on nautilus. Here the logs: https://pastebin.com/hnj8YysK I would think this is a bug, but would like your opinion. If not a bug how shall I solve this? I am using a handmade S3 tool to query as boto is not supported anymore, obo just does not work, and boto3 cannot send any arbitrary requests anymore. Also I would like to know if those Sync modules are still supported or not. The documentation is very poor and if they still are in the roadmap, it should be improved. Thanks Cheers, Luca Cervigni Pawsey Supercomputing centre

1 0

RGW crashed
by Zhenshi Zhou 11 May '20

11 May '20

Hi all, I deployed a multi-site cluster in order to sync object from an old cluster to a brand new cluster. It seems good cause I can see the data syncing. However, when I check the cluster health, it shows warn messages "2 daemons have recently crashed". I get the crash info by 'sudo ceph crash info $id': { "os_version_id": "7", "utsname_release": "3.10.0-957.27.2.el7.x86_64", "os_name": "CentOS Linux", "entity_name": "client.rgw.ceph-node7", "timestamp": "2020-05-09 15:17:59.482502Z", "process_name": "radosgw", "utsname_machine": "x86_64", "utsname_sysname": "Linux", "os_version": "7 (Core)", "os_id": "centos", "utsname_version": "#1 SMP Mon Jul 29 17:46:05 UTC 2019", "backtrace": [ "(()+0xf5f0) [0x7f32b1bdf5f0]", "(RGWCoroutine::set_sleeping(bool)+0xc) [0x555eeb1351ac]", "(RGWOmapAppend::flush_pending()+0x2d) [0x555eeb13acad]", "(RGWOmapAppend::finish()+0x10) [0x555eeb13acd0]", "(RGWDataSyncShardCR::stop_spawned_services()+0x2b) [0x555eeb0a185b]", "(RGWDataSyncShardCR::incremental_sync()+0x72a) [0x555eeb0a9baa]", "(RGWDataSyncShardCR::operate()+0x9d) [0x555eeb0ab33d]", "(RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x60) [0x555eeb136520]", "(RGWCoroutinesManager::run(std::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x236) [0x555eeb137196]", "(RGWCoroutinesManager::run(RGWCoroutine*)+0x78) [0x555eeb138098]", "(RGWRemoteDataLog::run_sync(int)+0x1cf) [0x555eeb08851f]", "(RGWDataSyncProcessorThread::process()+0x46) [0x555eeb1e71a6]", "(RGWRadosThread::Worker::entry()+0x115) [0x555eeb1b6195]", "(()+0x7e65) [0x7f32b1bd7e65]", "(clone()+0x6d) [0x7f32b10e188d]" ], "utsname_hostname": "ceph-node7", "crash_id": "2020-05-09_15:17:59.482502Z_b80d7bee-faa0-4d2f-9d86-a1b3f4d4802e", "ceph_version": "14.2.8" } AND { "os_version_id": "7", "utsname_release": "3.10.0-957.27.2.el7.x86_64", "os_name": "CentOS Linux", "entity_name": "client.rgw.ceph-node7", "timestamp": "2020-05-10 16:23:13.375063Z", "process_name": "radosgw", "utsname_machine": "x86_64", "utsname_sysname": "Linux", "os_version": "7 (Core)", "os_id": "centos", "utsname_version": "#1 SMP Mon Jul 29 17:46:05 UTC 2019", "backtrace": [ "(()+0xf5f0) [0x7f409f42e5f0]", "(RGWCoroutine::set_sleeping(bool)+0xc) [0x55e3f45e01ac]", "(RGWOmapAppend::flush_pending()+0x2d) [0x55e3f45e5cad]", "(RGWOmapAppend::finish()+0x10) [0x55e3f45e5cd0]", "(RGWDataSyncShardCR::stop_spawned_services()+0x2b) [0x55e3f454c85b]", "(RGWDataSyncShardCR::incremental_sync()+0x72a) [0x55e3f4554baa]", "(RGWDataSyncShardCR::operate()+0x9d) [0x55e3f455633d]", "(RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x60) [0x55e3f45e1520]", "(RGWCoroutinesManager::run(std::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x236) [0x55e3f45e2196]", "(RGWCoroutinesManager::run(RGWCoroutine*)+0x78) [0x55e3f45e3098]", "(RGWRemoteDataLog::run_sync(int)+0x1cf) [0x55e3f453351f]", "(RGWDataSyncProcessorThread::process()+0x46) [0x55e3f46921a6]", "(RGWRadosThread::Worker::entry()+0x115) [0x55e3f4661195]", "(()+0x7e65) [0x7f409f426e65]", "(clone()+0x6d) [0x7f409e93088d]" ], "utsname_hostname": "ceph-node7", "crash_id": "2020-05-10_16:23:13.375063Z_9e70a0c0-929e-445f-b4cd-8d29e909fe2f", "ceph_version": "14.2.8" } So I fetch and check the file "ceph-client.rgw.ceph-node7.log". The log has huge amount of errors like: -732> 2020-05-09 23:17:53.476 7f328b7ff700 0 RGW-SYNC:data:sync:shard[98]:entry[harbor-registry:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4620.94:23]:bucket[harbor-registry:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4620.94:23]:inc_sync[harbor-registry:f70a5 eb9-d88d-42fd-ab4e-d300e97094de.4620.94:23]: ERROR: lease is not taken, abort AND -723> 2020-05-09 23:17:56.388 7f328b7ff700 5 RGW-SYNC:data:sync:shard[88]:entry[harbor-registry:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4620.94:13]:bucket[harbor-registry:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4620.94:13]: incremental sync on bucket fa iled, retcode=-125 AND -215> 2020-05-09 23:17:58.809 7f328b7ff700 5 RGW-SYNC:data:sync:shard[10]:entry[pf2-harbor-swift:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4608.101:113]:bucket[pf2-harbor-swift:f70a5eb9-d88d-42fd-ab4e-d300e97094de.4608.101:113]: full sync on bucket failed, retcode=-125 AND 2020-05-09 23:18:24.048 7f4085867700 1 robust_notify: If at first you don't succeed: (110) Connection timed out 2020-05-09 23:18:24.048 7f4083863700 0 ERROR: failed to distribute cache for shubei.rgw.log:datalog.sync-status.shard.f70a5eb9-d88d-42fd-ab4e-d300e97094de.5 2020-05-09 23:28:49.181 7f407e859700 1 heartbeat_map reset_timeout 'RGWAsyncRadosProcessor::m_tp thread 0x7f407e859700' had timed out after 600 2020-05-10 03:12:01.905 7f409708a700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror And finally it crashed. I'm not sure where the problem is. Were the crashes caused by the network? Thanks

1 0