The below patch which was first posted here:
https://lkml.org/lkml/2012/4/3/193
and tested by me and picked up by the maintainer here:
https://lkml.org/lkml/2012/5/1/57
But it seems to not to have made to the linux-next or Linus tree.
I'm resending what it seems like a lost patch.
From: Justin Lecher <[email protected]>
Support the caching of large files.
https://bugzilla.kernel.org/show_bug.cgi?id=31182
Signed-off-by: Justin Lecher <[email protected]>
Tested-by: Suresh Jayaraman <[email protected]>
---
fs/cachefiles/rdwr.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c
index 0e3c092..d136686 100644
--- a/fs/cachefiles/rdwr.c
+++ b/fs/cachefiles/rdwr.c
@@ -919,7 +919,7 @@ int cachefiles_write_page(struct fscache_storage *op, struct page *page)
dget(object->backer);
mntget(cache->mnt);
file = dentry_open(object->backer, cache->mnt, O_RDWR,
- cache->cache_cred);
+ (O_RDWR | O_LARGEFILE), cache->cache_cred);
if (IS_ERR(file)) {
ret = PTR_ERR(file);
} else {
On Mon, 11 Jun 2012 11:08:21 +0530
Suresh Jayaraman <[email protected]> wrote:
> The below patch which was first posted here:
>
> https://lkml.org/lkml/2012/4/3/193
>
> and tested by me and picked up by the maintainer here:
>
> https://lkml.org/lkml/2012/5/1/57
>
> But it seems to not to have made to the linux-next or Linus tree.
> I'm resending what it seems like a lost patch.
>
>
> From: Justin Lecher <[email protected]>
>
> Support the caching of large files.
>
> https://bugzilla.kernel.org/show_bug.cgi?id=31182
>
> Signed-off-by: Justin Lecher <[email protected]>
> Tested-by: Suresh Jayaraman <[email protected]>
> ---
> fs/cachefiles/rdwr.c | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c
> index 0e3c092..d136686 100644
> --- a/fs/cachefiles/rdwr.c
> +++ b/fs/cachefiles/rdwr.c
> @@ -919,7 +919,7 @@ int cachefiles_write_page(struct fscache_storage *op, struct page *page)
> dget(object->backer);
> mntget(cache->mnt);
> file = dentry_open(object->backer, cache->mnt, O_RDWR,
> - cache->cache_cred);
> + (O_RDWR | O_LARGEFILE), cache->cache_cred);
Doesn't compile.
How extensively was this change tested? Please describe the testing
which was performed?
On 06/19/2012 04:34 AM, Andrew Morton wrote:
>> From: Justin Lecher <[email protected]>
>>
>> Support the caching of large files.
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=31182
>>
>> Signed-off-by: Justin Lecher <[email protected]>
>> Tested-by: Suresh Jayaraman <[email protected]>
>> ---
>> fs/cachefiles/rdwr.c | 2 +-
>> 1 files changed, 1 insertions(+), 1 deletions(-)
>>
>> diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c
>> index 0e3c092..d136686 100644
>> --- a/fs/cachefiles/rdwr.c
>> +++ b/fs/cachefiles/rdwr.c
>> @@ -919,7 +919,7 @@ int cachefiles_write_page(struct fscache_storage *op, struct page *page)
>> dget(object->backer);
>> mntget(cache->mnt);
>> file = dentry_open(object->backer, cache->mnt, O_RDWR,
>> - cache->cache_cred);
>> + (O_RDWR | O_LARGEFILE), cache->cache_cred);
>
> Doesn't compile.
Oops, my bad. While refreshing the original patch against the latest
kernel, I seemed to have introduced an extra argument while attempting
to limit the characters to 80. Extremely sorry about that.
> How extensively was this change tested? Please describe the testing
> which was performed?
>
The original patch was tested by mounting a NFS share with fscache
option enabled, do a md5sum on a nfs file larger than 2GB and ensure
that the file is getting cached by fscache (watch cache size growing)
on x86_64.
Please find below the fixed one.
From: Justin Lecher <[email protected]>
Subject: [PATCH] fs: cachefiles: Add support for large files in filesystem caching
Support the caching of large files.
https://bugzilla.kernel.org/show_bug.cgi?id=31182
Tested by mounting a NFS share with fscache option enabled, do a
md5sum on a nfs file larger than 2GB and ensure that the file is getting
cached by fscache (watch cache size growing) on x86_64.
Signed-off-by: Justin Lecher <[email protected]>
Signed-off-by: Suresh Jayaraman <[email protected]>
Tested-by: Suresh Jayaraman <[email protected]>
Cc: David Howells <[email protected]>
Acked-by: David Howells <[email protected]>
---
fs/cachefiles/rdwr.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c
index 0e3c092..cc7ef95 100644
--- a/fs/cachefiles/rdwr.c
+++ b/fs/cachefiles/rdwr.c
@@ -918,7 +918,7 @@ int cachefiles_write_page(struct fscache_storage *op, struct page *page)
* own time */
dget(object->backer);
mntget(cache->mnt);
- file = dentry_open(object->backer, cache->mnt, O_RDWR,
+ file = dentry_open(object->backer, cache->mnt, (O_RDWR | O_LARGEFILE),
cache->cache_cred);
if (IS_ERR(file)) {
ret = PTR_ERR(file);
On Tue, 19 Jun 2012 10:57:27 +0530 Suresh Jayaraman <[email protected]> wrote:
> > How extensively was this change tested? Please describe the testing
> > which was performed?
> >
>
> The original patch was tested by mounting a NFS share with fscache
> option enabled, do a md5sum on a nfs file larger than 2GB and ensure
> that the file is getting cached by fscache (watch cache size growing)
> on x86_64.
Well it will need a lot more coverage testing than that. truncate?
expanding truncate? write, lseek, pwrite, pread(), mmap(MAP_SHARED),
behavior at the new max file size (what is that?), etc.
On 19/06/12 09:52, Andrew Morton wrote:
> On Tue, 19 Jun 2012 10:57:27 +0530 Suresh Jayaraman <[email protected]> wrote:
>
>>> How extensively was this change tested? Please describe the testing
>>> which was performed?
>>>
>>
>> The original patch was tested by mounting a NFS share with fscache
>> option enabled, do a md5sum on a nfs file larger than 2GB and ensure
>> that the file is getting cached by fscache (watch cache size growing)
>> on x86_64.
>
> Well it will need a lot more coverage testing than that. truncate?
> expanding truncate? write, lseek, pwrite, pread(), mmap(MAP_SHARED),
> behavior at the new max file size (what is that?), etc.
>
Hello,
I am using this patch since the 2.6.3* linux, while working with files
>4 Gb on a daily bases. This isn't a testsuite, but it worked fine for
me all the time.
How would such testsuite you request look like? Anybody willing to write it?
Thanks,
Justin
On 06/19/2012 01:22 PM, Andrew Morton wrote:
> On Tue, 19 Jun 2012 10:57:27 +0530 Suresh Jayaraman <[email protected]> wrote:
>
>>> How extensively was this change tested? Please describe the testing
>>> which was performed?
>>>
>>
>> The original patch was tested by mounting a NFS share with fscache
>> option enabled, do a md5sum on a nfs file larger than 2GB and ensure
>> that the file is getting cached by fscache (watch cache size growing)
>> on x86_64.
>
> Well it will need a lot more coverage testing than that. truncate?
> expanding truncate? write, lseek, pwrite, pread(), mmap(MAP_SHARED),
> behavior at the new max file size (what is that?), etc.
>
The changelog could be slightly misleading as the patch doesn't really
add LFS support. It just passes O_LARGEFILE flag to dentry_open() to make
VFS allow cachefiles to open larger files (instead of failing it with
-EOVERFLOW). Cachefiles is just a cache that uses the VFS/VM interfaces to
get another disk filesystem to do the requisite I/O on its behalf.
So, I doubt whether more coverage testing is required for this change.
Anyway, I'm attaching a quick and dirty test program that might help
(not well tested though). I'm unable to to test it as I have modified
my setup and have to redo it.
Justin, do you have the setup and can you get this tested?
/*
* Few tests to ensure LFS (Large File Support) is working correctly.
*
* The requests (seek, truncate, write, mmap etc) that extend beyond the offset
* maximum (off_t) should succeed.
*
* Compile it with -D_FILE_OFFSET_BITS=64
*
* Run it like:
* ./lfs-tests large_file
*
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#define BUF_SZ 32
#define err(msg) \
do { perror(msg); exit(EXIT_FAILURE); } while (0)
void report_size(int fd, struct stat *st)
{
fstat(fd, st);
printf("size of file: %ld\n", st->st_size);
return;
}
int main(int argc, char *argv[])
{
int fd;
int rc = 0;
unsigned long offset = 0;
struct stat stbuf;
char buf[BUF_SZ];
void *file_mem;
if (argc != 2) {
fprintf(stderr, "Usage: %s test_file\n", argv[0]);
exit(EXIT_FAILURE);
}
fd = open(argv[1], O_RDWR | O_APPEND);
if (fd == -1)
err("open");
memset(buf, 0, sizeof(buf));
/* seek to the end of file */
offset = lseek(fd, 0, SEEK_END);
if (offset < 0)
err("lseek");
/* seek past the end of file */
offset = lseek(fd, BUF_SZ, SEEK_END);
printf("%ld\n", offset);
if (offset < 0)
err("lseek");
/* truncate it to < 2GB and report size */
rc = ftruncate(fd, 2147483648 - BUF_SZ);
if (rc != 0)
err("ftruncate");
report_size(fd, &stbuf);
/* truncate it to a size larger than 2GB */
rc = ftruncate(fd, 4294967360);
if (rc != 0)
err("ftruncate");
report_size(fd, &stbuf);
/* seek to the end of file */
offset = lseek(fd, 0, SEEK_END);
if (offset < 0)
err("lseek");
memset(buf, 'a', sizeof(buf));
/* write past 4GB */
rc = write(fd, buf, sizeof(buf));
if (rc == -1)
err("write");
report_size(fd, &stbuf);
offset = lseek(fd, 0, SEEK_END);
if (offset < 0)
err("lseek");
/* pwrite and pread */
rc = pread(fd, buf, BUF_SZ, offset);
if (rc == -1)
err("pread");
rc = pwrite(fd, buf, sizeof(buf), offset);
if (rc == -1)
err("pwrite");
report_size(fd, &stbuf);
/* page align offset */
offset = offset & ~(sysconf(_SC_PAGE_SIZE) - 1);
file_mem = mmap(0, BUF_SZ, PROT_WRITE, MAP_SHARED, fd, offset);
if (file_mem == MAP_FAILED)
err("mmap");
munmap(file_mem, 64);
close(fd);
return EXIT_SUCCESS;
}
On 20.06.2012 15:35, Suresh Jayaraman wrote:
> On 06/19/2012 01:22 PM, Andrew Morton wrote:
>> On Tue, 19 Jun 2012 10:57:27 +0530 Suresh Jayaraman <[email protected]> wrote:
>>
>>>> How extensively was this change tested? Please describe the testing
>>>> which was performed?
>>>>
>>>
>>> The original patch was tested by mounting a NFS share with fscache
>>> option enabled, do a md5sum on a nfs file larger than 2GB and ensure
>>> that the file is getting cached by fscache (watch cache size growing)
>>> on x86_64.
>>
>> Well it will need a lot more coverage testing than that. truncate?
>> expanding truncate? write, lseek, pwrite, pread(), mmap(MAP_SHARED),
>> behavior at the new max file size (what is that?), etc.
>>
>
> The changelog could be slightly misleading as the patch doesn't really
> add LFS support. It just passes O_LARGEFILE flag to dentry_open() to make
> VFS allow cachefiles to open larger files (instead of failing it with
> -EOVERFLOW). Cachefiles is just a cache that uses the VFS/VM interfaces to
> get another disk filesystem to do the requisite I/O on its behalf.
> So, I doubt whether more coverage testing is required for this change.
>
> Anyway, I'm attaching a quick and dirty test program that might help
> (not well tested though). I'm unable to to test it as I have modified
> my setup and have to redo it.
>
> Justin, do you have the setup and can you get this tested?
>
>
>
> /*
> * Few tests to ensure LFS (Large File Support) is working correctly.
> *
> * The requests (seek, truncate, write, mmap etc) that extend beyond the offset
> * maximum (off_t) should succeed.
> *
> * Compile it with -D_FILE_OFFSET_BITS=64
> *
> * Run it like:
> * ./lfs-tests large_file
> *
> */
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <fcntl.h>
> #include <unistd.h>
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <sys/mman.h>
>
> #define BUF_SZ 32
>
> #define err(msg) \
> do { perror(msg); exit(EXIT_FAILURE); } while (0)
>
>
> void report_size(int fd, struct stat *st)
> {
> fstat(fd, st);
> printf("size of file: %ld\n", st->st_size);
>
> return;
> }
>
> int main(int argc, char *argv[])
> {
> int fd;
> int rc = 0;
> unsigned long offset = 0;
> struct stat stbuf;
> char buf[BUF_SZ];
> void *file_mem;
>
> if (argc != 2) {
> fprintf(stderr, "Usage: %s test_file\n", argv[0]);
> exit(EXIT_FAILURE);
> }
>
> fd = open(argv[1], O_RDWR | O_APPEND);
> if (fd == -1)
> err("open");
>
> memset(buf, 0, sizeof(buf));
>
> /* seek to the end of file */
> offset = lseek(fd, 0, SEEK_END);
> if (offset < 0)
> err("lseek");
>
> /* seek past the end of file */
> offset = lseek(fd, BUF_SZ, SEEK_END);
> printf("%ld\n", offset);
> if (offset < 0)
> err("lseek");
>
> /* truncate it to < 2GB and report size */
> rc = ftruncate(fd, 2147483648 - BUF_SZ);
> if (rc != 0)
> err("ftruncate");
>
> report_size(fd, &stbuf);
>
> /* truncate it to a size larger than 2GB */
> rc = ftruncate(fd, 4294967360);
> if (rc != 0)
> err("ftruncate");
>
> report_size(fd, &stbuf);
>
> /* seek to the end of file */
> offset = lseek(fd, 0, SEEK_END);
> if (offset < 0)
> err("lseek");
>
> memset(buf, 'a', sizeof(buf));
>
> /* write past 4GB */
> rc = write(fd, buf, sizeof(buf));
> if (rc == -1)
> err("write");
>
> report_size(fd, &stbuf);
>
> offset = lseek(fd, 0, SEEK_END);
> if (offset < 0)
> err("lseek");
>
> /* pwrite and pread */
> rc = pread(fd, buf, BUF_SZ, offset);
> if (rc == -1)
> err("pread");
>
> rc = pwrite(fd, buf, sizeof(buf), offset);
> if (rc == -1)
> err("pwrite");
>
> report_size(fd, &stbuf);
>
> /* page align offset */
> offset = offset & ~(sysconf(_SC_PAGE_SIZE) - 1);
>
> file_mem = mmap(0, BUF_SZ, PROT_WRITE, MAP_SHARED, fd, offset);
> if (file_mem == MAP_FAILED)
> err("mmap");
>
> munmap(file_mem, 64);
>
> close(fd);
>
> return EXIT_SUCCESS;
> }
>
Hi,
so here are my test. Both system with fscache enabled.
unpatched kernel (Opensuse 12.1 linux-3.1.10-1.9-default):
$ ./large_file foo
8589934624
size of file: 2147483616
size of file: 4294967360
size of file: 4294967392
size of file: 4294967424
patched kernel (Gentoo Linux linux-3.4.3):
$ ./large_file bar
8589934624
size of file: 2147483616
size of file: 4294967360
size of file: 4294967392
size of file: 4294967424
no crash, everything worked.
Thanks justin