2004-07-01 01:58:34

by Steven Newbury

[permalink] [raw]
Subject: Trouble with the filesize limit

Hi!
I'm been trying to download the DVD ISO for FC2. But I've run into a
problem. It isn't possible for me to download more than 2GiB. Other
people have been having the same problem (see the d4x message board). I
haven't been able to trace the exact cause for the limit "ulimit -f"
reports unlimited and I'm running reiserfs with 3.6 format filesystem
(in theory supports much more than 2GiB). I've tried programs d4x, wget
etc., each of them have received a SIGXFSZ and exited at 2GiB.

Strangely I am able to create much larger files with dd.

This sounds like a similar bug to one that was fixed in 2.4, where the
limit was always 2GiB unless logged in directly as root. However in
this case even that doesn't work.

kernel 2.6.7-ck1
glibc 2.3.3-27 (fedora core)
bash 2.05b-38 (fedora core)

Any other info on request.

--
Steven Newbury <[email protected]>


2004-07-01 03:25:59

by Steven Newbury

[permalink] [raw]
Subject: Re: Trouble with the filesize limit

On Thu, 2004-07-01 at 03:45, Steven Newbury wrote:
> Thanks for the quick response.
>
> On Thu, 2004-07-01 at 03:21, Eric Lammerts wrote:
> > On Thu, 1 Jul 2004, Steven Newbury wrote:
> > > It isn't possible for me to download more than 2GiB.
> >
> > > I've tried programs d4x, wget etc., each of them have received a
> > > SIGXFSZ and exited at 2GiB.
> >
> > Probably none of those apps were compiled with
> > -D_FILE_OFFSET_BITS=64 ...
> >
> Okay I've recompiled wget with -D_FILE_OFFSET_BITS=64. Now I've got:
> get: progress.c:706: create_image: Assertion `insz <= dlsz' failed.
> when I try to continue the download...
This, I guess, is a bug in wget. Are the various types like size_t set
correctly for FILE_OFFSET_BITS=64, or should special care be taken
within each app to ensure types of sufficient size are used?

>
> > > Strangely I am able to create much larger files with dd.
> >
> > That one probably is... (the coreutils configure script enables that
> > automatically)
> >
> > Eric
--
Steven Newbury <[email protected]>

2004-07-01 03:27:14

by Eric Lammerts

[permalink] [raw]
Subject: Re: Trouble with the filesize limit


On Thu, 1 Jul 2004, Steven Newbury wrote:
> > Probably none of those apps were compiled with
> > -D_FILE_OFFSET_BITS=64 ...
> >
> Okay I've recompiled wget with -D_FILE_OFFSET_BITS=64. Now I've got:
> get: progress.c:706: create_image: Assertion `insz <= dlsz' failed.
> when I try to continue the download...

It looks like the progress bar code is not 64-bit clean... Maybe
you'll have better luck if you turn that off with -q or -nv.

Eric

2004-07-01 02:45:33

by Steven Newbury

[permalink] [raw]
Subject: Re: Trouble with the filesize limit

Thanks for the quick response.

On Thu, 2004-07-01 at 03:21, Eric Lammerts wrote:
> On Thu, 1 Jul 2004, Steven Newbury wrote:
> > It isn't possible for me to download more than 2GiB.
>
> > I've tried programs d4x, wget etc., each of them have received a
> > SIGXFSZ and exited at 2GiB.
>
> Probably none of those apps were compiled with
> -D_FILE_OFFSET_BITS=64 ...
>
Okay I've recompiled wget with -D_FILE_OFFSET_BITS=64. Now I've got:
get: progress.c:706: create_image: Assertion `insz <= dlsz' failed.
when I try to continue the download...

> > Strangely I am able to create much larger files with dd.
>
> That one probably is... (the coreutils configure script enables that
> automatically)
>
> Eric
--
Steven Newbury <[email protected]>

2004-07-01 03:35:26

by Eric Lammerts

[permalink] [raw]
Subject: Re: Trouble with the filesize limit


> On Thu, 2004-07-01 at 03:45, Steven Newbury wrote:
>> Okay I've recompiled wget with -D_FILE_OFFSET_BITS=64. Now I've got:
>> get: progress.c:706: create_image: Assertion `insz <= dlsz' failed.
>> when I try to continue the download...
> This, I guess, is a bug in wget. Are the various types like size_t set
> correctly for FILE_OFFSET_BITS=64, or should special care be taken
> within each app to ensure types of sufficient size are used?

I'm seeing the use of 'long' throughout the wget source... I guess you're
out of luck regarding wget... And it looks like it won't be fixed anytime
soon:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=181634
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=240281

Eric

2004-07-01 02:21:23

by Eric Lammerts

[permalink] [raw]
Subject: Re: Trouble with the filesize limit


On Thu, 1 Jul 2004, Steven Newbury wrote:
> It isn't possible for me to download more than 2GiB.

> I've tried programs d4x, wget etc., each of them have received a
> SIGXFSZ and exited at 2GiB.

Probably none of those apps were compiled with
-D_FILE_OFFSET_BITS=64 ...

> Strangely I am able to create much larger files with dd.

That one probably is... (the coreutils configure script enables that
automatically)

Eric