Large file downloads (~2TB) from CephFS drop at ~100 GB
Hello all, I am experiencing a strange issue with downloads from a CephFS-backed environment and would like to ask if anyone has seen similar behavior. When downloading large files (via wget or even directly through Firefox), the transfer speed is fine until the file size reaches roughly 100 GB. At that point, the session gets disconnected. This happens consistently, independent of the bandwidth available (e.g., whether the link runs at ~80 Mbit/s or ~230 Mbit/s). The only difference is how quickly the 100 GB “border” is reached. After the disconnect, the download can usually be resumed, but the cutoff is very repeatable. Is there a configuration that needs to be done on the ceph site? Best, Mevludin
Is it possible that the software serving the content has a limit you run into? May the most significant bit of your life be positive. Den sön 24 aug. 2025 13:08Mevludin Blazevic <mejdibl@yahoo.com> skrev:
Hello all,
I am experiencing a strange issue with downloads from a CephFS-backed environment and would like to ask if anyone has seen similar behavior.
When downloading large files (via wget or even directly through Firefox), the transfer speed is fine until the file size reaches roughly 100 GB. At that point, the session gets disconnected. This happens consistently, independent of the bandwidth available (e.g., whether the link runs at ~80 Mbit/s or ~230 Mbit/s). The only difference is how quickly the 100 GB “border” is reached.
After the disconnect, the download can usually be resumed, but the cutoff is very repeatable. Is there a configuration that needs to be done on the ceph site?
Best, Mevludin
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I’m a bit uncertain by the OP’s phrasing. The Subject header mentions CephFS, but the description mentions Firefox and wget, which sound more like RGW. If the latter, perhaps this threshold is a function of the load balancing layer having a payload size limit or timeout?
On Aug 24, 2025, at 3:02 PM, Janne Johansson <icepic.dz@gmail.com> wrote:
Is it possible that the software serving the content has a limit you run into?
May the most significant bit of your life be positive.
Den sön 24 aug. 2025 13:08Mevludin Blazevic <mejdibl@yahoo.com> skrev:
Hello all,
I am experiencing a strange issue with downloads from a CephFS-backed environment and would like to ask if anyone has seen similar behavior.
When downloading large files (via wget or even directly through Firefox), the transfer speed is fine until the file size reaches roughly 100 GB. At that point, the session gets disconnected. This happens consistently, independent of the bandwidth available (e.g., whether the link runs at ~80 Mbit/s or ~230 Mbit/s). The only difference is how quickly the 100 GB “border” is reached.
After the disconnect, the download can usually be resumed, but the cutoff is very repeatable. Is there a configuration that needs to be done on the ceph site?
Best, Mevludin
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Indeed, Ceph FS and RBD serves Cloudstack which in turn uses Ceph FS via NFS for "Secondary Storage". Cloudstack provides a download URL where I can download a raw file via browser or wget. What is odd to me is that the download rate shrinks rapidly around 100GB and it maintains for a few minutes around 140 kbit/s before the connection is completely lost. It is also visible in Ceph when the FS pool read ops per seconds drops to zero. After I tried other approaches in Cloudstack, I see that other types of large downloads do not show this behaviour, so I do not think that Ceph is doing any throttling.. Am 24.08.2025 um 21:02 schrieb Janne Johansson:
Is it possible that the software serving the content has a limit you run into?
May the most significant bit of your life be positive.
Den sön 24 aug. 2025 13:08Mevludin Blazevic <mejdibl@yahoo.com> skrev:
Hello all,
I am experiencing a strange issue with downloads from a CephFS-backed environment and would like to ask if anyone has seen similar behavior.
When downloading large files (via wget or even directly through Firefox), the transfer speed is fine until the file size reaches roughly 100 GB. At that point, the session gets disconnected. This happens consistently, independent of the bandwidth available (e.g., whether the link runs at ~80 Mbit/s or ~230 Mbit/s). The only difference is how quickly the 100 GB “border” is reached.
After the disconnect, the download can usually be resumed, but the cutoff is very repeatable. Is there a configuration that needs to be done on the ceph site?
Best, Mevludin
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den mån 25 aug. 2025 kl 08:18 skrev Mevludin Blazevic <mejdibl@yahoo.com>:
Indeed, Ceph FS and RBD serves Cloudstack which in turn uses Ceph FS via NFS for "Secondary Storage". Cloudstack provides a download URL where I can download a raw file via browser or wget. What is odd to me is that the download rate shrinks rapidly around 100GB and it maintains for a few minutes around 140 kbit/s before the connection is completely lost. It is also visible in Ceph when the FS pool read ops per seconds drops to zero.
After I tried other approaches in Cloudstack, I see that other types of large downloads do not show this behaviour, so I do not think that Ceph is doing any throttling..
Interesting to note that the number 100GB comes up in cloudstack docs: https://docs.cloudstack.apache.org/en/latest/adminguide/templates.html#best-... This bug report seem to say others have size-related problems aswell, and log lines you could look for: https://github.com/apache/cloudstack/issues/2820 Looks close to your issue too: https://lists.apache.org/thread/9g5nf3oj5szydojswmzjyv6r786lht9w -- May the most significant bit of your life be positive.
My current opinion is that the problem lies with Cloudstack itself, not Ceph. I opened this thread here to rule out the rather unlikely possibility that the issue is related to Ceph storage. Thanks for the links. The Cloudstack thread is actually mine, which I opened there in parallel. I will continue the discussion there.. Am 25.08.2025 um 09:02 schrieb Janne Johansson:
Den mån 25 aug. 2025 kl 08:18 skrev Mevludin Blazevic <mejdibl@yahoo.com>:
Indeed, Ceph FS and RBD serves Cloudstack which in turn uses Ceph FS via NFS for "Secondary Storage". Cloudstack provides a download URL where I can download a raw file via browser or wget. What is odd to me is that the download rate shrinks rapidly around 100GB and it maintains for a few minutes around 140 kbit/s before the connection is completely lost. It is also visible in Ceph when the FS pool read ops per seconds drops to zero.
After I tried other approaches in Cloudstack, I see that other types of large downloads do not show this behaviour, so I do not think that Ceph is doing any throttling.. Interesting to note that the number 100GB comes up in cloudstack docs:
https://docs.cloudstack.apache.org/en/latest/adminguide/templates.html#best-...
This bug report seem to say others have size-related problems aswell, and log lines you could look for:
https://github.com/apache/cloudstack/issues/2820
Looks close to your issue too:
https://lists.apache.org/thread/9g5nf3oj5szydojswmzjyv6r786lht9w
Hi, On 8/25/25 10:04, Mevludin Blazevic wrote:
My current opinion is that the problem lies with Cloudstack itself, not Ceph. I opened this thread here to rule out the rather unlikely possibility that the issue is related to Ceph storage.
Thanks for the links. The Cloudstack thread is actually mine, which I opened there in parallel. I will continue the discussion there..
CephFS has an upper file size limit which should be set to 1 TB by default. You can inspect the current value in `ceph fs dump`. Maybe Cloudstack is setting this to a lower values? Best regards, Burkhard
participants (4)
-
Anthony D'Atri
-
Burkhard Linke
-
Janne Johansson
-
Mevludin Blazevic