August 2020 - ceph-users

by Cloud Guy

Hello TL;DR We have a Nautilus cluster which has been operating without issue for quite some time. Recently one OSD experienced a relatively slow and painful death. The OSD was purged (via dashboard), replaced and added as a new OSD (same ID). Upon rebuild, we notice the node hosting the replaced OSD to experience extraordinary memory consumption (upwards of 95%). Remaining nodes healthy, operating normally. No adverse performance observations within the cluster. Looking for community assistance in helping troubleshoot this anomaly and resolving. Detail - Ceph 14.2.11 - Originally Deployed using ceph-ansible 4.0 stable; - OSD nodes run CentOS 7, have 10 drives, 256G RAM, no swap; - Problematic OSD ramps memory consumption to 95-99% available RAM; - Problematic OSD node appears to ramp RAM utilization until OSDs crash - and restart – resembling a contiguous saw-tooth pattern / memory leak??; - Other OSD nodes average around 30-40G steady state consumption. Cluster Logs: 2020-08-23 08:36:59.458072 [INF] Cluster is now healthy 2020-08-23 08:36:59.458032 [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2274696/100424847 objects degraded (2.265%), 85 pgs degraded) 2020-08-23 08:36:56.352104 [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 4 pgs down, 2 pgs peering, 15 pgs incomplete) 2020-08-23 08:36:56.352062 [WRN] Health check update: Degraded data redundancy: 2320455/100424847 objects degraded (2.311%), 87 pgs degraded (PG_DEGRADED) 2020-08-23 08:36:55.401933 [INF] osd.16 [v2:10.1.10.7:6831/204520,v1:10.1.10.7:6833/204520] boot 2020-08-23 08:36:54.744590 [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2020-08-23 08:36:53.688833 [WRN] Health check update: 1 osds down (OSD_DOWN) 2020-08-23 08:36:49.714790 [WRN] Health check update: Degraded data redundancy: 6617521/100424847 objects degraded (6.590%), 298 pgs degraded (PG_DEGRADED) 2020-08-23 08:36:49.714769 [WRN] Health check update: Reduced data availability: 17 pgs inactive, 5 pgs down, 2 pgs peering, 41 pgs incomplete (PG_AVAILABILITY) 2020-08-23 08:36:49.701966 [INF] osd.32 [v2:10.1.10.7:6812/204477,v1:10.1.10.7:6813/204477] boot 2020-08-23 08:36:49.701868 [INF] osd.28 [v2:10.1.10.7:6820/204478,v1:10.1.10.7:6821/204478] boot 2020-08-23 08:36:48.613586 [INF] osd.24 [v2:10.1.10.7:6836/204523,v1:10.1.10.7:6837/204523] boot 2020-08-23 08:36:48.540225 [WRN] Health check update: 3 osds down (OSD_DOWN) 2020-08-23 08:36:43.755645 [WRN] Health check failed: Degraded data redundancy: 5700895/100424847 objects degraded (5.677%), 266 pgs degraded (PG_DEGRADED) 2020-08-23 08:36:43.755611 [WRN] Health check failed: Reduced data availability: 12 pgs inactive, 5 pgs down, 37 pgs incomplete (PG_AVAILABILITY) 2020-08-23 08:36:40.049275 [INF] osd.16 failed (root=default,host=otr2817e02stor23) (connection refused reported by osd.39) 2020-08-23 08:36:40.047058 [INF] osd.24 failed (root=default,host=otr2817e02stor23) (connection refused reported by osd.6) 2020-08-23 08:36:39.452024 [WRN] Health check failed: 2 osds down (OSD_DOWN) 2020-08-23 08:36:39.399920 [INF] osd.32 failed (root=default,host=otr2817e02stor23) (connection refused reported by osd.33) 2020-08-23 08:36:39.399255 [INF] osd.28 failed (root=default,host=otr2817e02stor23) (connection refused reported by osd.18) 2020-08-23 08:04:08.089438 [INF] Cluster is now healthy 2020-08-23 08:04:08.089399 [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1549004/100424847 objects degraded (1.542%), 62 pgs degraded) 2020-08-23 08:04:05.154839 [WRN] Health check update: Degraded data redundancy: 1873878/100424847 objects degraded (1.866%), 68 pgs degraded (PG_DEGRADED) 2020-08-23 08:04:02.728402 [INF] osd.24 [v2:10.1.10.7:6820/202960,v1:10.1.10.7:6821/202960] boot 2020-08-23 08:04:02.572426 [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2020-08-23 08:03:53.138693 [WRN] Health check update: Degraded data redundancy: 2507272/100424847 objects degraded (2.497%), 88 pgs degraded (PG_DEGRADED) 2020-08-23 08:03:48.049569 [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 59 pgs peering) 2020-08-23 08:03:45.170204 [WRN] Health check failed: Degraded data redundancy: 281695/100424847 objects degraded (0.281%), 15 pgs degraded (PG_DEGRADED) 2020-08-23 08:03:45.170175 [WRN] Health check failed: Reduced data availability: 7 pgs inactive, 59 pgs peering (PG_AVAILABILITY) 2020-08-23 08:03:41.355393 [WRN] Health check failed: 1 osds down (OSD_DOWN)

3 years, 8 months

2
1
0 0

SEO Services Company

by kseoagency＠gmail.com

K SEO Agency an SEO Company in Bangalore focused on providing full-service Search Engine Optimization Solutions that include specialized Local SEO, National SEO, Ecommerce SEO services for businesses in Bangalore, India. https://kseo.agency/

3 years, 8 months

1
0
0 0

seo for small business

by kseoagency＠gmail.com

K SEO Agency an SEO Company in Bangalore focused on providing full-service Search Engine Optimization Solutions that include specialized Local SEO, National SEO, Ecommerce SEO services for businesses in Bangalore, India. https://kseo.agency/

3 years, 8 months

1
0
0 0

ISO Certification in Kolkata

by kseoagency＠gmail.com

ISO Certification in Kolkata – Genveritas a global ISO Certification Consulting firm represents considerable authority in tweaked and result-situated answers for assisting organizations to actualize change and improve business execution. https://genveritas.in/iso-certification-in-kolkata/

3 years, 8 months

1
0
0 0

How to change wal block in bluestore?

by Xu Xiao

Hi, I want to change my wal device to another device. I search much time and write a shell script, but finally, when I start the osd, the process is crashed. I think something is wrong, please help me check the script. Thanks! ceph version: ceph version 14.2.9 (bed944f8c45b9c98485e99b70e11bbcec6f6659a) nautilus (stable). The shell script is below: #! /bin/bash id=$1 # osd id vg=$2 # vg name osdPath="/var/lib/ceph/osd/ceph-$id" osdService="ceph-osd(a)${id}.service" blockPath=$(ls -lh ${osdPath}/block|awk '{print $11}') if [ -z "$blockPath" ]; then echo "no block path" exit 1; fi walPath=$(ls -lh ${osdPath}/block.wal|awk '{print $11}') if [ -z "$walPath" ]; then echo "no wal path" exit 2; fi walSize=$(blockdev --getsize64 $walPath) if [ -z "$walSize" ]; then echo "no wal size" exit 3; fi walUUID=$(lvdisplay ${walPath}|grep "LV UUID"|awk '{print $3}') if [ -z "$walUUID" ]; then echo "no wal uuid" exit 3; fi newWalLv="journal-osd${id}" newWalPath="/dev/${vg}/${newWalLv}" lvcreate -n "${newWalLv}" -L${walSize}b $vg || exit 4 newWalUUID=$(lvdisplay ${newWalPath}|grep "LV UUID"|awk '{print $3}') if [ -z "$newWalUUID" ]; then echo "no new wal uuid, new wal path:${newWalPath}" exit 3; fi ceph osd set noout || exit 5 systemctl stop ${osdService} || exit 6 ceph-osd -i $id --flush-journal || exit 7 dd if=${walPath} of=${newWalPath} bs=4M || exit 8 chown -h ceph:ceph $(realpath ${newWalPath}) || exit 9 ceph-bluestore-tool bluefs-bdev-migrate --path ${osdPath} --devs-source ${osdPath}/block.wal --dev-target ${newWalPath} chown -h ceph:ceph ${osdPath}/block.wal || exit 11 # device lvchange --deltag "ceph.wal_device=${walPath}" ${blockPath} || exit 12 lvchange --addtag "ceph.wal_device=${newWalPath}" ${blockPath} || exit 13 # for uuid lvchange --deltag "ceph.wal_uuid=${walUUID}" ${blockPath} || exit 12 lvchange --addtag "ceph.wal_uuid=${newWalUUID}" ${blockPath} || exit 13 # for wal lvs for tag in $(lvs -o lv_tags ${blockPath}|grep ceph|sed 's/,/\n/g') do lvchange --addtag "$tag" ${newWalPath} done lvchange --deltag "ceph.type=block" ${newWalPath} lvchange --addtag "ceph.type=wal" ${newWalPath} systemctl start ceph-osd(a)${id}.service || exit 14 ceph osd unset noout || exit 15

3 years, 8 months

1
0
0 0

Add OSD with primary on HDD, WAL and DB on SSD

by Tony Liu

Hi, I am able to add OSD with primary on HDD, WAL and DB on SSD on dashboard. LVs are created on SSD for OSDs. How can I achieve the same by CLI? I tried with drivegroups, which adds OSD, but not WAL and DB. Should I handle LV for WAL and DB separately? Anything I am missing here? Thanks! Tony

3 years, 8 months

1
0
0 0

How to Find the Cheap & Best Assignment Help Sydney Available Online?

by maxwillor＠gmail.com

I have been a part of Myassignmenthelpau I am working as a fully functional academic consultant with the team. We offer affordable pricing solutions for assignments and homework across all course subjects. I hold excellent writing skills and offer Assignment help Sydney of the best form. https://www.myassignmenthelpau.com/assignment-help-sydney

3 years, 8 months

1
0
0 0

does ceph RBD have the ability to load balance?

by 差很多先生

Hello Cephers, I am a newcomer to Ceph.I know that RBD is a distributed block storage device and libvirt supports rbd pool type. Suppose I have a ceph cluster and use it to build a rbd pool to store virtual machine images. When multiple qemu-kvm clients use the image in the rbd pool to start virtual machines at the same time, will rbd perform load balancing? That is, different hosts in the cluster would send images to different clients at the same time. Or only one host will send images, and the other hosts will only send images when the current host is down? Would someone explain it to me please? Thanks consteven

3 years, 8 months

2
1
0 0

Get Instant HELP From Epson support For Epson Error Code 0x97.

by mary smith

The Epson Error Code 0x97 is a kind of caution at whatever point you experience an inside equipment issue with your printer. It very well may be a result of a motherboard glitch or some other sort of inner segment breakdown. You should choose the guide from Epson support , straightforwardly from comfort of your home. https://www.epsonprintersupportpro.net/epson-error-code-0x97/

3 years, 8 months

1
0
0 0

[doc] drivegroups advanced case

by Tony Liu

Hi, Regarding to the YAML in this section, https://ceph.readthedocs.io/en/latest/cephadm/drivegroups/#the-advanced-case Is the "rotational" supposed to be "1" meaning spinning HDD? Thanks! Tony

3 years, 8 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users August 2020