29 Jun
2020
29 Jun
'20
10:59 a.m.
Hi,
We have already installed a new Debian (10.4) server and I need put it in a
Ceph cluster.
When I execute the command to install ceph on this node:
ceph-deploy install --release nautilus node1
It starts to install a version 12.x in my node...
(...)
[serifos][DEBUG ] After this operation, 183 MB of additional disk space will
be used.
[serifos][DEBUG ] Selecting previously unselected package python-cephfs.
(Reading database ... 30440 files and directories currently installed.)
[serifos][DEBUG ] Preparing to unpack
.../python-cephfs_12.2.11+dfsg1-2.1+b1_amd64.deb ...
[serifos][DEBUG ] Unpacking python-cephfs (12.2.11+dfsg1-2.1+b1) ...
[serifos][DEBUG ] Selecting previously unselected package ceph-common.
[serifos][DEBUG ] Preparing to unpack
.../ceph-common_12.2.11+dfsg1-2.1+b1_amd64.deb ...
[serifos][DEBUG ] Unpacking ceph-common (12.2.11+dfsg1-2.1+b1) ...
(...)
How do I upgrade this packages?
Even installed packages in this version, the installation
completes without erros.
The question is due to an error message that I'm recieving
when deploy a new osd.
ceph-deploy osd create --data /dev/sdb node1
At this point:
[ceph_deploy.osd][INFO ] Distro info: debian 10.4 buster
[ceph_deploy.osd][DEBUG ] Deploying osd to node1
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node1][DEBUG ] find the location of an executable
[node1][INFO ] Running command: sudo /usr/sbin/ceph-volume --cluster ceph
lvm create --bluestore --data /dev/sdb
[node1][WARNIN] --> RuntimeError: Unable to create a new OSD id
[node1][DEBUG ] Running command: /bin/ceph-authtool --gen-print-key
[node1][DEBUG ] Running command: /bin/ceph --cluster ceph --name
client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i -
osd new 76da6c51-8385-4ffc-9a8e-0dfc11e31feb
[node1][DEBUG ] stderr:
/build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: In function 'void
MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char>,
entity_addr_t>&)' thread 7f2bc7fff700 time 2020-06-29 06:56:17.331350
[node1][DEBUG ] stderr:
/build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: 77: FAILED
assert(mon_info[p.first].public_addr == p.second)
[node1][DEBUG ] stderr: ceph version 12.2.11
(26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)
[node1][DEBUG ] stderr: 1: (ceph::__ceph_assert_fail(char const*, char
const*, int, char const*)+0xf5) [0x7f2bdaff5f75]
[node1][DEBUG ] stderr: 2:
(MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >, entity_addr_t,
std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const, entity_addr_t>
>
&)+0x568) [0x7f2bdb050038]
[node1][DEBUG ] stderr: 3:
(MonMap::decode(ceph::buffer::list::iterator&)+0x4da) [0x7f2bdb05500a]
[node1][DEBUG ] stderr: 4: (MonClient::handle_monmap(MMonMap*)+0x216)
[0x7f2bdb042a06]
[node1][DEBUG ] stderr: 5: (MonClient::ms_dispatch(Message*)+0x4ab)
[0x7f2bdb04729b]
[node1][DEBUG ] stderr: 6: (DispatchQueue::entry()+0xeba) [0x7f2bdb06bf5a]
[node1][DEBUG ] stderr: 7: (DispatchQueue::DispatchThread::entry()+0xd)
[0x7f2bdb1576fd]
[node1][DEBUG ] stderr: 8: (()+0x7fa3) [0x7f2be499dfa3]
[node1][DEBUG ] stderr: 9: (clone()+0x3f) [0x7f2be45234cf]
[node1][DEBUG ] stderr: NOTE: a copy of the executable, or `objdump -rdS
<executable>` is needed to interpret this.
[node1][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume
--cluster ceph lvm create --bluestore --data /dev/sdb
[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs
I think this error occurs because the wrong package that was
installed.
Thanks,
Rafael
29 Jun
29 Jun
4:02 p.m.
New subject: layout help: need chassis local io to minimize net links
Hi
I have a few servers each with 6 or more disks, with a storage workload
that's around 80% done entirely within each server. From a
work-to-be-done perspective there's no need for 80% of the load to
traverse network interfaces, the rest needs what ceph is all about. So
I cooked up a set of crush maps and pools, one map/pool for each server
and one map/pool for the whole. Skipping the long story, the
performance remains network link speed bound and has got to change.
"Chassis local" io is too slow. I even tried putting a mon within each
server. I'd like to avoid having to revert to some other HA
filesystem per server with ceph at the chassis layer if I can help it.
Any notions that would allow 'chassis local' rbd traffic to avoid or
mostly avoid leaving the box?
Thanks!
7:57 p.m.
New subject: *****SPAM***** layout help: need chassis local io to minimize net links
I wonder if you should not have chosen a different product? Ceph is
meant to distribute data across nodes, racks, data centers etc. For a
nail use a hammer, for a screw use a screw driver.
-----Original Message-----
To: ceph-users(a)ceph.io
Subject: *****SPAM***** [ceph-users] layout help: need chassis local io
to minimize net links
Hi
I have a few servers each with 6 or more disks, with a storage workload
that's around 80% done entirely within each server. From a
work-to-be-done perspective there's no need for 80% of the load to
traverse network interfaces, the rest needs what ceph is all about. So
I cooked up a set of crush maps and pools, one map/pool for each server
and one map/pool for the whole. Skipping the long story, the
performance remains network link speed bound and has got to change.
"Chassis local" io is too slow. I even tried putting a mon within each
server. I'd like to avoid having to revert to some other HA
filesystem per server with ceph at the chassis layer if I can help
it.
Any notions that would allow 'chassis local' rbd traffic to avoid or
mostly avoid leaving the box?
Thanks!
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
30 Jun
30 Jun
12:01 a.m.
Hello Rafael,
Can you check the apt sources list that exist from your ceph-deploy
node? Maybe there you have put luminous debian packages version?
Regards,
Anastasios
On Mon, 2020-06-29 at 06:59 -0300, Rafael Quaglio wrote:
Hi,
We have already installed a new Debian (10.4) server and I need put
it in a
Ceph cluster.
When I execute the command to install ceph on this node:
ceph-deploy install --release nautilus node1
It starts to install a version 12.x in my node...
(...)
[serifos][DEBUG ] After this operation, 183 MB of additional disk
space will
be used.
[serifos][DEBUG ] Selecting previously unselected package python-
cephfs.
(Reading database ... 30440 files and directories currently
installed.)
[serifos][DEBUG ] Preparing to unpack
.../python-cephfs_12.2.11+dfsg1-2.1+b1_amd64.deb ...
[serifos][DEBUG ] Unpacking python-cephfs (12.2.11+dfsg1-2.1+b1) ...
[serifos][DEBUG ] Selecting previously unselected package ceph-
common.
[serifos][DEBUG ] Preparing to unpack
.../ceph-common_12.2.11+dfsg1-2.1+b1_amd64.deb ...
[serifos][DEBUG ] Unpacking ceph-common (12.2.11+dfsg1-2.1+b1) ...
(...)
How do I upgrade this packages?
Even installed packages in this version, the
installation
completes without erros.
The question is due to an error message that I'm
recieving
when deploy a new osd.
ceph-deploy osd create --data /dev/sdb node1
At this point:
[ceph_deploy.osd][INFO ] Distro info: debian 10.4 buster
[ceph_deploy.osd][DEBUG ] Deploying osd to node1
[node1][DEBUG ] write cluster configuration to
/etc/ceph/{cluster}.conf
[node1][DEBUG ] find the location of an executable
[node1][INFO ] Running command: sudo /usr/sbin/ceph-volume --cluster
ceph
lvm create --bluestore --data /dev/sdb
[node1][WARNIN] --> RuntimeError: Unable to create a new OSD id
[node1][DEBUG ] Running command: /bin/ceph-authtool --gen-print-key
[node1][DEBUG ] Running command: /bin/ceph --cluster ceph --name
client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-
osd/ceph.keyring -i -
osd new 76da6c51-8385-4ffc-9a8e-0dfc11e31feb
[node1][DEBUG ] stderr:
/build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: In function
'void
MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char>,
entity_addr_t>&)' thread 7f2bc7fff700 time 2020-06-29 06:56:17.331350
[node1][DEBUG ] stderr:
/build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: 77: FAILED
assert(mon_info[p.first].public_addr == p.second)
[node1][DEBUG ] stderr: ceph version 12.2.11
(26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)
[node1][DEBUG ] stderr: 1: (ceph::__ceph_assert_fail(char const*,
char
const*, int, char const*)+0xf5) [0x7f2bdaff5f75]
[node1][DEBUG ] stderr: 2:
(MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >, entity_addr_t,
std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const, entity_addr_t>
&)+0x568) [0x7f2bdb050038]
[node1][DEBUG ] stderr: 3:
(MonMap::decode(ceph::buffer::list::iterator&)+0x4da)
[0x7f2bdb05500a]
[node1][DEBUG ] stderr: 4:
(MonClient::handle_monmap(MMonMap*)+0x216)
[0x7f2bdb042a06]
[node1][DEBUG ] stderr: 5: (MonClient::ms_dispatch(Message*)+0x4ab)
[0x7f2bdb04729b]
[node1][DEBUG ] stderr: 6: (DispatchQueue::entry()+0xeba)
[0x7f2bdb06bf5a]
[node1][DEBUG ] stderr: 7:
(DispatchQueue::DispatchThread::entry()+0xd)
[0x7f2bdb1576fd]
[node1][DEBUG ] stderr: 8: (()+0x7fa3) [0x7f2be499dfa3]
[node1][DEBUG ] stderr: 9: (clone()+0x3f) [0x7f2be45234cf]
[node1][DEBUG ] stderr: NOTE: a copy of the executable, or `objdump
-rdS
<executable>` is needed to interpret this.
[node1][ERROR ] RuntimeError: command returned non-zero exit status:
1
[ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-
volume
--cluster ceph lvm create --bluestore --data /dev/sdb
[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs
I think this error occurs because the wrong package
that was
installed.
Thanks,
Rafael
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:31 a.m.
New subject: Re layout help: need chassis local io to minimize net links
I need exactly what ceph is for a whole lot of work, that work just
doesn't represent a large fraction of the total local traffic. Ceph is
the right choice. Plainly ceph has tremendous support for replication
within a chassis, among chassis and among racks. I just need
intra-chassis traffic to not hit the net much. Seems not such an
unreasonable thing given the intra-chassis crush rules and all. After
all.. ceph's name wasn't chosen for where it can't go....
On 6/29/20 1:57 PM, Marc Roos wrote:
> I wonder if you should not have chosen a different product? Ceph is
> meant to distribute data across nodes, racks, data centers etc. For a
> nail use a hammer, for a screw use a screw driver.
>
>
> -----Original Message-----
> To: ceph-users(a)ceph.io
> Subject: *****SPAM***** [ceph-users] layout help: need chassis local io
> to minimize net links
>
> Hi
>
> I have a few servers each with 6 or more disks, with a storage workload
> that's around 80% done entirely within each server. From a
> work-to-be-done perspective there's no need for 80% of the load to
> traverse network interfaces, the rest needs what ceph is all about. So
> I cooked up a set of crush maps and pools, one map/pool for each server
> and one map/pool for the whole. Skipping the long story, the
> performance remains network link speed bound and has got to change.
> "Chassis local" io is too slow. I even tried putting a mon within each
> server. I'd like to avoid having to revert to some other HA
> filesystem per server with ceph at the chassis layer if I can help
> it.
>
> Any notions that would allow 'chassis local' rbd traffic to avoid or
> mostly avoid leaving the box?
>
> Thanks!
>
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> email to ceph-users-leave(a)ceph.io
>
>
1418
days inactive
1419
days old
4 comments
4 participants
participants (4)
-
Anastasios Dados -
Harry G. Coin -
Marc Roos -
Rafael Quaglio