- ceph-users - lists.ceph.io

diskprediction_local fails with python3-sklearn 0.22.2
by Eric Dold 22 Dec '20

22 Dec '20

Hello the mgr module diskprediction_local fails under ubuntu 20.04 focal with python3-sklearn version 0.22.2 Ceph version is 15.2.3 when the module is enabled i get the following error: File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 112, in serve self.predict_all_devices() File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 279, in predict_all_devices result = self._predict_life_expentancy(devInfo['devid']) File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 222, in _predict_life_expentancy predicted_result = obj_predictor.predict(predict_datas) File "/usr/share/ceph/mgr/diskprediction_local/predictor.py", line 457, in predict pred = clf.predict(ordered_data) File "/usr/lib/python3/dist-packages/sklearn/svm/_base.py", line 585, in predict if self.break_ties and self.decision_function_shape == 'ovo': AttributeError: 'SVC' object has no attribute 'break_ties' Best Regards Eric

2 1

Failing OSD RocksDB Corrupt
by Ashley Merrick 22 Dec '20

22 Dec '20

Hello,I had some faulty power cables on some OSD's in one server which caused lots of IO issues/disks appearing/disappearing.This has been corrected now, 2 of the 10 OSD's are working, however 8 are failing to start due to what looks to be a corrupt DB.When running a ceph-bluestore-tool fsck I get the following output:rocksdb: [db/db_impl_open.cc:516] db.wal/002221.log: dropping 1302 bytes; Corruption: missing start of fragmented record(2)2020-12-22T16:21:52.715+0100 7f7b6a1500c0 4 rocksdb: [db/db_impl.cc:389] Shutdown: canceling all background work2020-12-22T16:21:52.715+0100 7f7b6a1500c0 4 rocksdb: [db/db_impl.cc:563] Shutdown complete2020-12-22T16:21:52.715+0100 7f7b6a1500c0 -1 rocksdb: Corruption: missing start of fragmented record(2)2020-12-22T16:21:52.715+0100 7f7b6a1500c0 -1 bluestore(/var/lib/ceph/b1db6b36-0c4c-4bce-9cda-18834be0632d/osd.28) opendb erroring opening db:Trying to start the OSD leads to:ceph_abort_msg("Bad table magic number: expected 9863518390377041911, found 9372993859750765257 in db/002442.sst")It looks like the last write to these OSD's never fully completed, sadly as I was adding this new node to move from OSD to Host redundancy (EC Pool) I have 20% down PG's currently, is there anything I can do to remove the last entry in the DB or somehow clean up the rocksDB to get these OSD's atleast started? Understand may end up with some corrupted files.Thanks Sent via MXlogin

1 0

PGs down
by Jeremy Austin 22 Dec '20

22 Dec '20

I could use some input from more experienced folks… First time seeing this behavior. I've been running ceph in production (replicated) since 2016 or earlier. This, however, is a small 3-node cluster for testing EC. Crush map rules should sustain the loss of an entire node. Here's the EC rule: rule cephfs425 { id 6 type erasure min_size 3 max_size 6 step set_chooseleaf_tries 40 step set_choose_tries 400 step take default step choose indep 3 type host step choose indep 2 type osd step emit } I had actual hardware failure on one node. Interestingly, this appears to have resulted in data loss. OSDs began to crash in a cascade on other nodes (i.e., nodes with no known hardware failure). Not a low RAM problem. I could use some pointers about how to get the down PGs back up — I *think* there are enough EC shards, even disregarding the OSDs that crash on start. nautilus 14.2.15 ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 54.75960 root default -10 16.81067 host sumia 1 hdd 5.57719 osd.1 up 1.00000 1.00000 5 hdd 5.58469 osd.5 up 1.00000 1.00000 6 hdd 5.64879 osd.6 up 1.00000 1.00000 -7 16.73048 host sumib 0 hdd 5.57899 osd.0 up 1.00000 1.00000 2 hdd 5.56549 osd.2 up 1.00000 1.00000 3 hdd 5.58600 osd.3 up 1.00000 1.00000 -3 21.21844 host tower1 4 hdd 3.71680 osd.4 up 0 1.00000 7 hdd 1.84799 osd.7 up 1.00000 1.00000 8 hdd 3.71680 osd.8 up 1.00000 1.00000 9 hdd 1.84929 osd.9 up 1.00000 1.00000 10 hdd 2.72899 osd.10 up 1.00000 1.00000 11 hdd 3.71989 osd.11 down 0 1.00000 12 hdd 3.63869 osd.12 down 0 1.00000 cluster: id: d0b4c175-02ba-4a64-8040-eb163002cba6 health: HEALTH_ERR 1 MDSs report slow requests 4/4239345 objects unfound (0.000%) Too many repaired reads on 3 OSDs Reduced data availability: 7 pgs inactive, 7 pgs down Possible data damage: 4 pgs recovery_unfound Degraded data redundancy: 95807/24738783 objects degraded (0.387%), 4 pgs degraded, 3 pgs undersized 7 pgs not deep-scrubbed in time 7 pgs not scrubbed in time services: mon: 3 daemons, quorum sumib,tower1,sumia (age 4d) mgr: sumib(active, since 7d), standbys: sumia, tower1 mds: cephfs:1 {0=sumib=up:active} 2 up:standby osd: 13 osds: 11 up (since 3d), 10 in (since 4d); 3 remapped pgs data: pools: 5 pools, 256 pgs objects: 4.24M objects, 15 TiB usage: 24 TiB used, 24 TiB / 47 TiB avail pgs: 2.734% pgs not active 95807/24738783 objects degraded (0.387%) 47910/24738783 objects misplaced (0.194%) 4/4239345 objects unfound (0.000%) 245 active+clean 7 down 3 active+recovery_unfound+undersized+degraded+remapped 1 active+recovery_unfound+degraded+repair progress: Rebalancing after osd.12 marked out [============================..] Rebalancing after osd.4 marked out [=============================.] An snipped from an example down pg: "up": [ 3, 2, 5, 1, 8, 9 ], "acting": [ 3, 2, 5, 1, 8, 9 ], <snip> ], "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 11, 12 ], "peering_blocked_by": [ { "osd": 11, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 12, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } ] }, { Oddly, these OSDs possibly did NOT experience hardware failure. However, they won't start -- see pastebin for ceph-osd.11.log https://pastebin.com/6U6sQJuJ HEALTH_ERR 1 MDSs report slow requests; 4/4239345 objects unfound (0.000%); Too many repaired reads on 3 OSDs; Reduced data availability : 7 pgs inactive, 7 pgs down; Possible data damage: 4 pgs recovery_unfound; Degraded data redundancy: 95807/24738783 objects degraded (0 .387%), 4 pgs degraded, 3 pgs undersized; 7 pgs not deep-scrubbed in time; 7 pgs not scrubbed in time MDS_SLOW_REQUEST 1 MDSs report slow requests mdssumib(mds.0): 42 slow requests are blocked > 30 secs OBJECT_UNFOUND 4/4239345 objects unfound (0.000%) pg 19.5 has 1 unfound objects pg 15.2f has 1 unfound objects pg 15.41 has 1 unfound objects pg 15.58 has 1 unfound objects OSD_TOO_MANY_REPAIRS Too many repaired reads on 3 OSDs osd.9 had 9664 reads repaired osd.7 had 9665 reads repaired osd.4 had 12 reads repaired PG_AVAILABILITY Reduced data availability: 7 pgs inactive, 7 pgs down pg 15.10 is down, acting [3,2,5,1,8,9] pg 15.1e is down, acting [5,1,9,8,2,3] pg 15.40 is down, acting [7,10,1,5,3,2] pg 15.4a is down, acting [0,3,5,6,9,10] pg 15.6a is down, acting [3,2,6,1,10,8] pg 15.71 is down, acting [3,2,1,6,8,10] pg 15.76 is down, acting [2,0,6,5,10,9] PG_DAMAGED Possible data damage: 4 pgs recovery_unfound pg 15.2f is active+recovery_unfound+undersized+degraded+remapped, acting [5,1,0,3,2147483647,7], 1 unfound pg 15.41 is active+recovery_unfound+undersized+degraded+remapped, acting [5,1,0,3,2147483647,2147483647], 1 unfound pg 15.58 is active+recovery_unfound+undersized+degraded+remapped, acting [10,2147483647,2,3,1,5], 1 unfound pg 19.5 is active+recovery_unfound+degraded+repair, acting [3,2,5,1,8,10], 1 unfound PG_DEGRADED Degraded data redundancy: 95807/24738783 objects degraded (0.387%), 4 pgs degraded, 3 pgs undersized pg 15.2f is stuck undersized for 635305.932075, current state active+recovery_unfound+undersized+degraded+remapped, last acting [5,1,0,3,2147483647,7] pg 15.41 is stuck undersized for 364298.836902, current state active+recovery_unfound+undersized+degraded+remapped, last acting [5,1,0,3,2147483647,2147483647] pg 15.58 is stuck undersized for 384461.110229, current state active+recovery_unfound+undersized+degraded+remapped, last acting [10,2147483647,2,3,1,5] pg 19.5 is active+recovery_unfound+degraded+repair, acting [3,2,5,1,8,10], 1 unfound PG_NOT_DEEP_SCRUBBED 7 pgs not deep-scrubbed in time pg 15.76 not deep-scrubbed since 2020-10-21 14:30:03.935228 pg 15.71 not deep-scrubbed since 2020-10-21 12:20:46.235792 pg 15.6a not deep-scrubbed since 2020-10-21 07:52:33.914083 pg 15.10 not deep-scrubbed since 2020-10-22 03:24:40.465367 pg 15.1e not deep-scrubbed since 2020-10-22 10:37:36.169959 pg 15.40 not deep-scrubbed since 2020-10-23 05:33:35.208748 pg 15.4a not deep-scrubbed since 2020-10-22 05:14:06.981035 PG_NOT_SCRUBBED 7 pgs not scrubbed in time pg 15.76 not scrubbed since 2020-10-24 08:12:40.090831 pg 15.71 not scrubbed since 2020-10-25 05:22:40.573572 pg 15.6a not scrubbed since 2020-10-24 15:03:09.189964 pg 15.10 not scrubbed since 2020-10-24 16:25:08.826981 pg 15.1e not scrubbed since 2020-10-24 16:05:03.080127 pg 15.40 not scrubbed since 2020-10-24 11:58:04.290488 pg 15.4a not scrubbed since 2020-10-24 11:32:44.573551 -- Jeremy Austin jhaustin(a)gmail.com

4 14

osd_pglog memory hoarding - another case
by Kalle Happonen 22 Dec '20

22 Dec '20

Hello all, wrt: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7IMIWCKIHXN… Yesterday we hit a problem with osd_pglog memory, similar to the thread above. We have a 56 node object storage (S3+SWIFT) cluster with 25 OSD disk per node. We run 8+3 EC for the data pool (metadata is on replicated nvme pool). The cluster has been running fine, and (as relevant to the post) the memory usage has been stable at 100 GB / node. We've had the default pg_log of 3000. The user traffic doesn't seem to have been exceptional lately. Last Thursday we updated the OSDs from 14.2.8 -> 14.2.13. On Friday the memory usage on OSD nodes started to grow. On each node it grew steadily about 30 GB/day, until the servers started OOM killing OSD processes. After a lot of debugging we found that the pg_logs were huge. Each OSD process pg_log had grown to ~22GB, which we naturally didn't have memory for, and then the cluster was in an unstable situation. This is significantly more than the 1,5 GB in the post above. We do have ~20k pgs, which may directly affect the size. We've reduced the pg_log to 500, and started offline trimming it where we can, and also just waited. The pg_log size dropped to ~1,2 GB on at least some nodes, but we're still recovering, and have a lot of ODSs down and out still. We're unsure if version 14.2.13 triggered this, or if the osd restarts triggered this (or something unrelated we don't see). This mail is mostly to figure out if there are good guesses why the pg_log size per OSD process exploded? Any technical (and moral) support is appreciated. Also, currently we're not sure if 14.2.13 triggered this, so this is also to put a data point out there for other debuggers. Cheers, Kalle Happonen

4 16

kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production)
by Marc Roos 22 Dec '20

22 Dec '20

I have a vm on a osd node (which can reach host and other nodes via the macvtap interface (used by the host and guest)). I just did a simple bonnie++ test and everything seems to be fine. Yesterday however the dovecot procces apparently caused problems (only using cephfs for an archive namespace, inbox is on rbd ssd, fs meta also on ssd) How can I recover from such lock-up. If I have a similar situation with an nfs-ganesha mount, I have the option to do a umount -l, and clients recover quickly without any issues. Having to reset the vm, is not really an option. What is best way to resolve this? Ceph cluster: 14.2.11 (the vm has 14.2.16) I have in my ceph.conf nothing special, these 2x in the mds section: mds bal fragment size max = 120000 # maybe for nfs-ganesha problems? # http://docs.ceph.com/docs/master/cephfs/eviction/ #mds_session_blacklist_on_timeout = false #mds_session_blacklist_on_evict = false mds_cache_memory_limit = 17179860387 All running: CentOS Linux release 7.9.2009 (Core) Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

2 4

Can big data use Ceph?
by fantastic2085 22 Dec '20

22 Dec '20

Can big data use Ceph?For example, can Hive Hbase Spark use Ceph? https://github.com/ceph/cephfs-hadoop is no longer maintain?

4 3

friendly warning about death by container versions
by Philip Brown 21 Dec '20

21 Dec '20

I was having horrible problems getting my test ceph clusterj reinitialized. All kinds of annoying things were happening. including things like getting differing output from ceph orch device ls vs ceph device ls Being new-ish to ceph, i was going nuts, wondering what kind of init options I was missing. Turns out, nothing i was doing was wrong, per se. I had ended up with differing container versions. Even after doing "cephadm rm-cluster", the old versions were sticking. The really annoying thing is, the difference was tiny. 15.2.8 on the master. 15.2.5 on other nodes. but device related things were failing, with 2020-12-21 09:37:09,134 INFO /bin/podman:stderr ceph-volume inventory: error: unrecognized arguments: --filter-for-batch in /var/log/ceph/cephadm.log Sighhh... To save anyone else some research, the end-user fix is: run ceph orch upgrade start --ceph-version 15.2.8 -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

1 0

Cephfs mount hangs
by Marc Roos 21 Dec '20

21 Dec '20

How to recover from this? It is possible to have a vm with a cephfs mount on a osd server?

1 0

guide to multi-homed hosts, for Octopus?
by Philip Brown 21 Dec '20

21 Dec '20

In my experiments with ceph so far, setting up a new cluster goes fairly well... so long as i only use a single network. But when I try to use separate networks, things stop functioning in various ways. (For example, I can " SO I thought I'd ask for pointers to any multi-network setup guide. My goal: * have a 3+ node ceph cluster that each has local SSD storage only. * have an RBD mapped on each node, which will then have a non-cephfs filesystem on it, shared out via NFS * have each node share out NFS on one interface, but communicate to the cluster on a separate interface In the old ways, it was theoretically straightforward, in that you could specify "public" interfaces vs other ones. But in the new cephadm driven world, I havent found the magic that works. For example, in my current iteration, I have successfully added all three hosts, and have 3 "mon"s... but "ceph orch device ls --refresh" only shows the dev from the node I'm running it on. Im still trying to cycle through different bootstrap options. and Im experimenting with overriding naming in /etc/hosts, for things like which IP addresses get mapped to the real hostname, vs which get given "hostname-datainterface" type naming. For example: On the one hand, Im wondering if I need to name ALL IP addresses for a host, with the same hostname. But on the other hand, my sysadmin instincts whisper to me that sounds like a terrible idea. SO, tips from people who have done multi homing under octopus, would be appreciated. Note that my initial proof-of-concept cluster is just 3 physical nodes, so everything needs to live on them. -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

1 0

Ceph with Firewall
by Seena Fallah 21 Dec '20

21 Dec '20

Hi, I want to enable the firewall on my ceph nodes with ufw. Does anyone have any experience with any performance regression in it? Is there any solution for blocking exporter ports without a firewall in a Ceph cluster like node exporter and ceph exporter? Thanks.

1 0