3 May
2021
3 May
'21
12:15 p.m.
On Mon, May 3, 2021 at 12:24 PM Magnus Harlander <magnus(a)harlan.de> wrote:
Am 03.05.21 um 11:22 schrieb Ilya Dryomov:
There is a 6th osd directory on both machines, but it's empty
[root@s0 osd]# ll
total 0
drwxrwxrwt. 2 ceph ceph 200 2. Mai 16:31 ceph-1
drwxrwxrwt. 2 ceph ceph 200 2. Mai 16:31 ceph-3
drwxrwxrwt. 2 ceph ceph 200 2. Mai 16:31 ceph-4
drwxrwxrwt. 2 ceph ceph 200 2. Mai 16:31 ceph-5
drwxr-xr-x. 2 ceph ceph 6 3. Apr 19:50 ceph-8 <===
drwxrwxrwt. 2 ceph ceph 200 2. Mai 16:31 ceph-9
[root@s0 osd]# pwd
/var/lib/ceph/osd
[root@s1 osd]# ll
total 0
drwxrwxrwt 2 ceph ceph 200 May 2 15:39 ceph-0
drwxr-xr-x. 2 ceph ceph 6 Mar 13 17:54 ceph-1 <===
drwxrwxrwt 2 ceph ceph 200 May 2 15:39 ceph-2
drwxrwxrwt 2 ceph ceph 200 May 2 15:39 ceph-6
drwxrwxrwt 2 ceph ceph 200 May 2 15:39 ceph-7
drwxrwxrwt 2 ceph ceph 200 May 2 15:39 ceph-8
[root@s1 osd]# pwd
/var/lib/ceph/osd
The bogus directories are empty and they are
used on the other machine for a real osd!
How is that?
Should I remove them and restart ceph.target?
I don't think empty directories matter at this point. You may not have
had 12 OSDs at any point in time, but the max_osd value appears to have
gotten bumped when you were replacing those disks.
Note that max_osd being greater than the number of OSDs is not a big
problem by itself. The osdmap is going to be larger and require more
memory but that's it. You can test by setting it back to 12 and trying
to mount -- it should work. The issue is specific to how to those OSDs
were replaced -- something went wrong and the osdmap somehow ended up
with rather bogus addrvec entries. Not sure if it's ceph-deploy's
fault, something weird in ceph.conf (back then) or a an actual ceph
bug.
Thanks,
Ilya
11 May
11 May
9:49 a.m.
New subject: cephfs mount problems with 5.11 kernel - not a ipv6 problem
Hi Ilya,
On 3 May 2021, at 14:15, Ilya Dryomov
<idryomov(a)gmail.com> wrote:
I don't think empty directories matter at this point. You may not have
had 12 OSDs at any point in time, but the max_osd value appears to have
gotten bumped when you were replacing those disks.
Note that max_osd being greater than the number of OSDs is not a big
problem by itself. The osdmap is going to be larger and require more
memory but that's it. You can test by setting it back to 12 and trying
to mount -- it should work. The issue is specific to how to those OSDs
were replaced -- something went wrong and the osdmap somehow ended up
with rather bogus addrvec entries. Not sure if it's ceph-deploy's
fault, something weird in ceph.conf (back then) or a an actual ceph
bug.
What actuality is bug? When max_osds > total_osd_in?
What kernel's was affected?
For example, max_osds is 132, total_osds_in in 126, max osd number is 131 - is affected?
Thanks,
k
12:24 p.m.
New subject: cephfs mount problems with 5.11 kernel - not a ipv6 problem
On Tue, May 11, 2021 at 10:50 AM Konstantin Shalygin <k0ste(a)k0ste.ru> wrote:
Hi Ilya,
On 3 May 2021, at 14:15, Ilya Dryomov <idryomov(a)gmail.com> wrote:
I don't think empty directories matter at this point. You may not have
had 12 OSDs at any point in time, but the max_osd value appears to have
gotten bumped when you were replacing those disks.
Note that max_osd being greater than the number of OSDs is not a big
problem by itself. The osdmap is going to be larger and require more
memory but that's it. You can test by setting it back to 12 and trying
to mount -- it should work. The issue is specific to how to those OSDs
were replaced -- something went wrong and the osdmap somehow ended up
with rather bogus addrvec entries. Not sure if it's ceph-deploy's
fault, something weird in ceph.conf (back then) or a an actual ceph
bug.
What actuality is bug? When max_osds > total_osd_in?
No, as mentioned above max_osds being greater is not a problem per se.
Having max_osds set to 10000 when you only have a few dozen is going to
waste a lot of memory and network bandwidth, but if it is just slightly
bigger it's not something to worry about. Normally these "spare" slots
are ignored, but in Magnus' case they looked rather weird and the kernel
refused the osdmap. See
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?…
for details.
What kernel's was affected?
5.11 and 5.12, backports are on the way.
For example, max_osds is 132, total_osds_in in 126, max osd number is 131 - is affected?
No, max_osds alone is not enough to trigger it.
Thanks,
Ilya
1:01 p.m.
New subject: cephfs mount problems with 5.11 kernel - not a ipv6 problem
On 11 May 2021, at 14:24, Ilya Dryomov
<idryomov(a)gmail.com> wrote:
No, as mentioned above max_osds being greater is not a problem per se.
Having max_osds set to 10000 when you only have a few dozen is going to
waste a lot of memory and network bandwidth, but if it is just slightly
bigger it's not something to worry about. Normally these "spare" slots
are ignored, but in Magnus' case they looked rather weird and the kernel
refused the osdmap. See
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?…
<https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3f1c6f2122fc780560f09735b6d1dbf39b44eb0f>
for details.
Thanks for clarification!
k
What kernel's was affected?
5.11 and 5.12, backports are on the way.
For example, max_osds is 132, total_osds_in in 126, max osd number is 131 - is affected?
No, max_osds alone is not enough to trigger it.
1103
days inactive
1111
days old
3 comments
2 participants
participants (2)
-
Ilya Dryomov -
Konstantin Shalygin