Increase of osd space usage on cephfs heavy load
hello when I run my borgbackup over cephfs volume (10 subvolumes for 1.5To) I can see a big increase of osd space usage and 2 or 3 osd goes near full, or full, then out and finally the cluster goes in error. Any tips to prevent this ? My cluster is cephv15 with : 9 nodes : each node run : 2x6to hdd and 2x600to ssd the cephfs got data on hdd and metadata on ssd. the cephfs md cache is : 32Go 128pg for data and metadata (this is has been setup by auto balancer) Perhaps I can fix the pg num for each of cephfs pool and prevent autobalancer to run for them. what do you think ? thx you for your help and advices. UPDATE : I increase the pg number to 256 for data and 1024 for metadata Here the df during the backup started since 30min POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs-metadata 12 183 GiB 514.68k 550 GiB 7.16 2.3 TiB Before the backup the stored was 20GiB oau
hello now backup is running since 3hours and cephfs metadata goes from 20G to 479Go... POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs-metadata 12 479 GiB 642.26k 1.4 TiB 18.79 2.0 TiB cephfs-data0 13 2.9 TiB 9.23M 9.4 TiB 10.67 26 TiB is that a normal behaviour ? oau Le lundi 05 avril 2021 à 15:17 +0200, Olivier AUDRY a écrit :
hello
when I run my borgbackup over cephfs volume (10 subvolumes for 1.5To) I can see a big increase of osd space usage and 2 or 3 osd goes near full, or full, then out and finally the cluster goes in error.
Any tips to prevent this ?
My cluster is cephv15 with :
9 nodes :
each node run : 2x6to hdd and 2x600to ssd the cephfs got data on hdd and metadata on ssd. the cephfs md cache is : 32Go
128pg for data and metadata (this is has been setup by auto balancer)
Perhaps I can fix the pg num for each of cephfs pool and prevent autobalancer to run for them.
what do you think ?
thx you for your help and advices.
UPDATE : I increase the pg number to 256 for data and 1024 for metadata
Here the df during the backup started since 30min
POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs-metadata 12 183 GiB 514.68k 550 GiB 7.16 2.3 TiB
Before the backup the stored was 20GiB
oau _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 4/6/21 2:20 PM, Olivier AUDRY wrote:
hello
now backup is running since 3hours and cephfs metadata goes from 20G to 479Go... POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs-metadata 12 479 GiB 642.26k 1.4 TiB 18.79 2.0 TiB cephfs-data0 13 2.9 TiB 9.23M 9.4 TiB 10.67 26 TiB
is that a normal behaviour ?
The MDS maintains a list of open files in the metadata pool. If your backup is scanning a lot of files, and caps are not reclaimed by the MDS, this list will become large. The corresponding objects are called 'mds<rank>_openfiles.<chunk>', e.g. mds0_openfiles.0, mds0_openfiles.1 etc. You can check the size of these objects with the rados command. If this is the reason for the large pool, I would recommend to restrict the number of caps per client, otherwise you might run into out of memory problems if the MDS is restarted during the backup. Regards, Burkhard
hello many thx for the time you take helping me on this. I restarted one of the backup and now the space usage for cephfs meta data goes from 17Go to 70Go but the hints you give me seems to not help here. cephfs-metadata/mds1_openfiles.0 mtime 2021-04-06 18:27:08.000000, size 0 cephfs-metadata/mds0_openfiles.0 mtime 2021-04-06 18:27:10.000000, size 0 cephfs-metadata/mds2_openfiles.1 mtime 2021-04-06 06:31:00.000000, size 0 cephfs-metadata/mds0_openfiles.1 mtime 2021-04-06 06:31:02.000000, size 0 cephfs-metadata/mds2_openfiles.0 mtime 2021-04-06 18:27:08.000000, size 0 uhm. As I'm writting this email the metadata pool goes from 70Go to 39Go as the backup is still running. I don't really get what is going on here ... oau Le mardi 06 avril 2021 à 15:08 +0200, Burkhard Linke a écrit :
Hi,
On 4/6/21 2:20 PM, Olivier AUDRY wrote:
hello
now backup is running since 3hours and cephfs metadata goes from 20G to 479Go... POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs-metadata 12 479 GiB 642.26k 1.4 TiB 18.79 2.0 TiB cephfs-data0 13 2.9 TiB 9.23M 9.4 TiB 10.67 26 TiB
is that a normal behaviour ?
The MDS maintains a list of open files in the metadata pool. If your backup is scanning a lot of files, and caps are not reclaimed by the MDS, this list will become large.
The corresponding objects are called 'mds<rank>_openfiles.<chunk>', e.g. mds0_openfiles.0, mds0_openfiles.1 etc. You can check the size of these objects with the rados command.
If this is the reason for the large pool, I would recommend to restrict the number of caps per client, otherwise you might run into out of memory problems if the MDS is restarted during the backup.
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Burkhard Linke
-
Olivier AUDRY