Re: ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)
2 Dec
2019
2 Dec
'19
4:57 p.m.
On Mon, Dec 2, 2019 at 12:48 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote:
Hi Ilya,
4 nodes (osd, mon combined), the 4th node has local cephfs mount, which
is rsync'ing some files from vm's. 'low load' I have sort of test setup,
going to production. Mostly the nodes are below a load of 1 (except when
the concurrent rsync starts)
Is there a reason you are snapshotting each directory individually
instead of just snapshotting a common parent?
If you have thousands of snapshots, you may eventually hit a different
bug:
https://tracker.ceph.com/issues/21420
https://docs.ceph.com/docs/master/cephfs/experimental-features/#snapshots
Be aware that each set of 512 snapshots amplify your writes by 4K in
terms of network consumption. With 14000 snapshots, a 4K write would
need to transfer ~109K worth of snapshot metadata to carry itself out.
Thanks,
Ilya
ISTR there were some anti-spam measures put in place. Is your account
waiting for manual approval? If so, David should be able to help.
Yes if I remember correctly I get waiting approval when I try to log in.
Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9287
ffff911a9a26bd00 fail -12
Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9283
It is failing to allocate memory. "low load" isn't very specific,
can you describe the setup and the workload in more detail? How many snapshots do you have?
Don't know how to count them. I have script running on a 2000 dirs. If
one of these dirs is not empty it creates a snapshot. So in theory I
could have 2000 x 7 days = 14000 snapshots.
(btw the cephfs snapshots are in a different tree than the rsync is
using) 
2 Dec
2 Dec
5:06 p.m.
New subject: ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)
> >
> >ISTR there were some anti-spam measures put in place. Is your
account
> >waiting for manual approval? If so, David
should be able to help.
>
> Yes if I remember correctly I get waiting approval when I try to log
in.
>
> >>
> >>
> >>
> >> Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9287
> >> ffff911a9a26bd00 fail -12
> >> Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9283
> >
> >
> >It is failing to allocate memory. "low load" isn't very
specific,
> >can you describe the setup and the workload in more detail?
>
> 4 nodes (osd, mon combined), the 4th node has local cephfs mount,
which
> is rsync'ing some files from vm's.
'low load' I have sort of test
setup,
> going to production. Mostly the nodes are below a
load of 1 (except
when
> the concurrent rsync starts)
>
> >How many snapshots do you have?
>
> Don't know how to count them. I have script running on a 2000 dirs.
If
one of these
dirs is not empty it creates a snapshot. So in theory I
could have 2000 x 7 days = 14000 snapshots.
(btw the cephfs snapshots are in a different tree than the rsync is
using)
Is there a reason you are snapshotting each directory individually
instead of just snapshotting a common parent? If you have thousands of snapshots, you may eventually
hit a different
bug:
https://tracker.ceph.com/issues/21420
https://docs.ceph.com/docs/master/cephfs/experimental-features/#snapsh
ots
Be aware that each set of 512 snapshots amplify your writes by 4K in
terms of network consumption. With 14000 snapshots, a 4K write would
need to transfer ~109K worth of snapshot metadata to carry itself out.
Also when I am not even writing to a tree with snapshots enabled? I am
rsyncing to dir3
.
├── dir1
│ ├── dira
│ │ └── .snap
│ ├── dirb
│ ├── dirc
│ │ └── .snap
│ └── dird
│ └── .snap
├── dir2
└── dir3
6:29 p.m.
New subject: ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)
I can confirm that removing all the snapshots seems to resolve the
problem.
A - I would propose a redesign of something like that snapshots from
below the mountpoint are only taken into account and not snapshots in
the entire filesystem. That should fix a lot of issues
B - That reminds me about this mv command, that does not move data
across different pools in the fs. I would like to see this. Because it
is the logical thing to expect.
> >
> >ISTR there were some anti-spam measures put in place. Is your
account
>> >waiting for manual approval? If so, David should be able
to help.
>
> Yes if I remember correctly I get waiting approval when I try to log
in.
>
> >>
> >>
> >>
> >> Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9287
> >> ffff911a9a26bd00 fail -12 >> >> Dec 1 03:14:36 c04 kernel:
ceph: build_snap_context 100020c9283 >> > >> > >>
>It is failing
to allocate memory. "low load" isn't very specific, >> >can you
describe the setup and the workload in more detail?
>
> 4 nodes (osd, mon combined), the 4th node has local cephfs mount,
which
>> is rsync'ing some files from vm's. 'low load' I have sort of
test setup, >> going to production. Mostly the nodes are below a load
of 1 (except when >> the concurrent rsync starts) >> >> >How many
snapshots do you have?
>
> Don't know how to count them. I have script running on a 2000 dirs.
If
> one of these dirs is not empty it creates a
snapshot. So in theory I
> could have 2000 x 7 days = 14000 snapshots.
> (btw the cephfs snapshots are in a different tree than the rsync is
> using) > >Is there a reason you are snapshotting each directory
individually >instead of just snapshotting a common parent?
Yes because I am not sure the snapshot frequency on all folders is going
to be the same.
If you have thousands of snapshots, you may eventually
hit a different
bug:
https://tracker.ceph.com/issues/21420
https://docs.ceph.com/docs/master/cephfs/experimental-features/#snapsh
ots
Be aware that each set of 512 snapshots amplify your writes by 4K in
terms of network consumption. With 14000 snapshots, a 4K write would
need to transfer ~109K worth of snapshot metadata to carry itself out.
Also when I am not even writing to a tree with snapshots enabled? I am
rsyncing to dir3
.
├── dir1
│ ├── dira
│ │ └── .snap
│ ├── dirb
│ ├── dirc
│ │ └── .snap
│ └── dird
│ └── .snap
├── dir2
└── dir3
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
1611
days inactive
1611
days old
2 comments
2 participants
participants (2)
-
Ilya Dryomov -
Marc Roos