Hi All
I've been worried that I did not have a good backup of my cluster and
having looked around I could not find anything which did not require
local storage.
I found Rhian script while looking for a backup solution before a major
version upgrade and found that it worked very well.
I'm looking for opinions on this method below.
This uses stdio to pipe the data directly from rbd to a s3 host, while
encrypting the data.
I have a few worries.
1. The snap shots, I think I should be deleting the older one and only
keeping the the last one or two.
2. Having only one full backup and diff from there seems wrong to me but
the amount of data would seem to preclude creating a new full back up on
a regular biases.
I'm sure that others will pipe up with more issues.
Note. The sum.c program is needed because s3 needs to be told about
files which are larger than 5Gig and the standard tools I tested all had
a limit of 32bits when printing out the size in bytes
Cheers
Mike
#!/bin/bash
###
# Original Author: Rhian Resnick - Updated a lot by Mike O'Connor
# Purpose: Backup CEPH RBD using snapshots, the files that are created
should be stored off the ceph cluster, but you can use ceph storage
during the process of backing them up.
###
export AWS_PROFILE=wasabi
PUBKEY='PublicKey'
pool=$1
if [ “$pool” == “” ]
then
echo Usage: $0 pool
exit 1
fi
rbd ls $pool | while read vol
do
if [ $vol == ISO ]
then
continue
fi
# Look up latest backup file
echo BACKUP ${vol}
LASTSNAP=`aws s3 ls s3://dcbackup/$vol/ | sort | tail -n 1 | awk
'{print $4}' | cut -d "." -f 1`
echo "Last Snap: $vol/$LASTSNAP"
# Create a snap, we need this to do the diff
NEWSNAP=`date +%y%m%d%H%M`
echo "New Snap: $NEWSNAP"
echo rbd snap create $pool/$vol@$NEWSNAP
rbd snap create $pool/$vol@$NEWSNAP
if [ "$LASTSNAP" == "" ]
then
RBD_SIZE=`rbd diff $pool/$vol | ./sum`
echo "rbd export-diff $pool/$vol@$NEWSNAP - | seccure-encrypt
${PUBKEY} | aws s3 cp --expected-size ${RBD_SIZE} -
s3://dcbackup/$vol/$NEWSNAP.diff"
rbd export-diff $pool/$vol@$NEWSNAP - | seccure-encrypt
${PUBKEY} | aws s3 cp --expected-size ${RBD_SIZE} -
s3://dcbackup/$vol/$NEWSNAP.diff
else
RBD_SIZE=`rbd diff --from-snap $LASTSNAP $pool/$vol | ./sum`
echo "rbd export-diff --from-snap $LASTSNAP $pool/$vol@$NEWSNAP
- | seccure-encrypt ${PUBKEY} | aws s3 cp --expected-size ${RBD_SIZE} -
s3://dcbackup/$vol/$NEWSNAP.diff"
rbd export-diff --from-snap $LASTSNAP $pool/$vol@$NEWSNAP - |
seccure-encrypt ${PUBKEY} | aws s3 cp - s3://dcbackup/$vol/$NEWSNAP.diff
fi
echo
echo
done
#include <stdio.h>
#include <inttypes.h>
#include <string.h>
#include <stdlib.h>
/* sum column 2 of input with lines like: number number word
* skipping the first (header) row
*
* usage: sum [-v] file
*
* if -v is specified a count of rows processed (including the header)
is also output
*/
int main(int argc, char *argv[]) {
uint64_t sum = 0L;
size_t n = 100;
unsigned int count=1;
char *buf = (char*)malloc(n);
getline(&buf,&n,stdin);
while (1) {
uint64_t offset, bytes;
n = fscanf(stdin, "%lld %lld %s\n", &offset, &bytes, buf);
if (n != 3) break;
++count;
sum += bytes;
}
if (argc>1 && strcmp(argv[1],"-v")==0)
printf("%d %lld\n", count, sum);
else
printf("%lld\n", sum);
}
A couple of weeks ago, I sent a request to the mailing list asking
whether anyone was using the inline_data support in cephfs:
https://docs.ceph.com/docs/mimic/cephfs/experimental-features/#inline-data
I got exactly zero responses, so I'm going to formally propose that we
move to start deprecating this feature for Octopus.
Why deprecate this feature?
===========================
While the userland clients have support for both reading and writing,
the kernel only has support for reading, and aggressively uninlines
everything as soon as it needs to do any writing. That uninlining has
some rather nasty potential race conditions too that could cause data
corruption.
We could work to fix this, and maybe add write support for the kernel,
but it adds a lot of complexity to the read and write codepaths in the
clients, which are already pretty complex. Given that there isn't a lot
of interest in this feature, I think we ought to just pull the plug on
it.
How should we do this?
======================
We should start by disabling this feature in master for Octopus.
In particular, we should stop allowing users to call "fs set inline_data
true" on filesystems where it's disabled, and maybe throw a loud warning
about the feature being deprecated if the mds is started on a filesystem
that has it enabled.
We could also consider creating a utility to crawl an existing
filesystem and uninline anything there, if there was need for it.
Then, in a few release cycles, once we're past the point where someone
can upgrade directly from Nautilus (release Q or R?) we'd rip out
support for this feature entirely.
Thoughts, comments, questions welcome.
--
Jeff Layton <jlayton(a)redhat.com>
Hi Everyone,
There's been a few threads around about small HDD (spinning disk)
clusters and performance on Bluestore.
One recently from Christian
(http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036385.html)
was particularly interesting to us as we have a very similar setup to
what Christian has and we see similar performance.
We have a 6 node cluster each with 12x 4TB SATA HDD, IT mode LSI 3008, wal/db on
33GB NVMe partitions. Each node has a single Xeon Gold 6132 CPU @
2.60GHz and dual 10GB network.
We also use bcache with 1 180GB NVMe partition shared between 6 osd's.
Workload is via KVM (Proxmox)
I did the same benchmark fio tests as Christian. Here's my results (M
for me, C for Christian)
direct=0
========
M -- read : io=6008.0MB, bw=203264KB/s, iops=49, runt= 30267msec
C -- read: IOPS=40, BW=163MiB/s (171MB/s)(7556MiB/46320msec)
direct=1
========
M -- read : io=32768MB, bw=1991.4MB/s, iops=497, runt= 16455ms
C -- read: IOPS=314, BW=1257MiB/s (1318MB/s)(32.0GiB/26063msec)
direct=0
========
M -- write: io=32768MB, bw=471105KB/s, iops=115, runt= 71225msec
C -- write: IOPS=119, BW=479MiB/s (503MB/s)(32.0GiB/68348msec
direct=1
========
M -- write: io=32768MB, bw=479829KB/s, iops=117, runt= 69930msec
C -- write: IOPS=139, BW=560MiB/s (587MB/s)(32.0GiB/58519msec)
I should probably mention that there was some active workload on the
cluster at that time also, around 500iops write and 100MB/s
throughput.
The main problem that we're having with this cluster is how easy it is
for it to hit slow requests and we have one particular vm that ends up
doing scsi resets because of the latency.
So we're considering switching these osd's to filestore.
We have two other clusters using filestore/bcache/ssd journal and the
performance seems to be much better on those - taking into account the
different sizes.
What are peoples thoughts on this size cluster? Is it just not a good
fit with bluestore and our type of workload?
Also, does anyone have any knowledge on future support for filestore?
I'm concerned that we may have to migrate our other clusters off
filestore sometime in the future and that'll hurt us with the current
performance.
Rich
Hi,
> Am 07.08.2019 um 20:20 schrieb Nerurkar, Ruchir (Nokia - US/Mountain View) <ruchir.nerurkar(a)nokia.com>:
>
> Hello,
>
> I work in Nokia as a Software QA engineer and I am trying to install Ceph on centOS 7.4 version.
Do you have to stick at 7.4?
> But I am getting this error with the following output: -
>
> vmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-common-12.2.12-0.el7.x86_64 (Ceph)
> [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit)
> [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-osd-12.2.12-0.el7.x86_64 (Ceph)
> [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit)
> [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-base-12.2.12-0.el7.x86_64 (Ceph)
> [mvmgoscephcontrollerluminous][WARNIN] Requires: gperftools-libs >= 2.6.1
> [mvmgoscephcontrollerluminous][WARNIN] Available: gperftools-libs-2.4-8.el7.i686 (base)
> [mvmgoscephcontrollerluminous][DEBUG ] You could try using --skip-broken to work around the problem
> [mvmgoscephcontrollerluminous][WARNIN] gperftools-libs = 2.4-8.el7
> [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-mon-12.2.12-0.el7.x86_64 (Ceph)
> [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit)
> [mvmgoscephcontrollerluminous][WARNIN] Error: Package: 2:ceph-base-12.2.12-0.el7.x86_64 (Ceph)
> [mvmgoscephcontrollerluminous][WARNIN] Requires: liblz4.so.1()(64bit)
> [mvmgoscephcontrollerluminous][DEBUG ] You could try running: rpm -Va --nofiles --nodigest
> [mvmgoscephcontrollerluminous][ERROR ] RuntimeError: command returned non-zero exit status: 1
> [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: yum -y install ceph ceph-radosgw
>
> Does anyone know about this issue like how can I resolve package dependencies?
My first shot: Update Centos to min. 7.5 (which includes gperftools-libs-2.6.1)
And install lz4 too.
Regards . Götz
I have a fairly dormant ceph luminous cluster on centos7 with stock
kernel, and thought about upgrading it before putting it to more use.
I can remember some page on the ceph website that had specific
instructions mentioning upgrading from luminous. But I can't find it
anymore, this page[0] even says I am viewing an unsupported version of
ceph??
Any pointers aside from a default upgrade procedure?
- I am using rbd, rgw and cephfs (only one mds)
- Still using direct block devices (not lvm)
- have one 1 pg active+clean+inconsistent since a year or so, that I am
getting sentimentally attached to ;)
[0]
https://docs.ceph.com/docs/mimic/install/upgrading-ceph/#
Pfff, you are right, I don't even know which one is the newest latest,
indeed Nautilus
-----Original Message-----
Subject: Re: [ceph-users] Upgrad luminous -> mimic , any pointers?
Why would you go to Mimic instead of Nautilus?
>
>
>
> I have a fairly dormant ceph luminous cluster on centos7 with stock
> kernel, and thought about upgrading it before putting it to more use.
> I can remember some page on the ceph website that had specific
> instructions mentioning upgrading from luminous. But I can't find it
> anymore, this page[0] even says I am viewing an unsupported version of
> ceph??
>
> Any pointers aside from a default upgrade procedure?
> - I am using rbd, rgw and cephfs (only one mds)
> - Still using direct block devices (not lvm)
> - have one 1 pg active+clean+inconsistent since a year or so, that I
> am getting sentimentally attached to ;)
>
>
> [0]
> https://docs.ceph.com/docs/mimic/install/upgrading-ceph/#
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> email to ceph-users-leave(a)ceph.io
