2005-01-03 18:13:35

by Olaf Hering

[permalink] [raw]
Subject: pin files in memory after read


Is there a way to always keep a file (once read from disk) in memory, no
matter how much memory pressure exists?
There are always complains that updatedb and similar tools wipe out all
caches. So I guess there is no such thing yet.

I simply want to avoid the spinup of my ibook harddisk when something
has been 'forgotten' and must be loaded again (like opening a new screen
window after a while).

The best I could do so far was a cramfs image. I copied it to tmpfs
during early boot, then mount -o bind every cramfs file over the real
binary on disk. Of course that will fail as soon as I want to update an
affected package because the binary is busy (readonly). So there must be
a better way to achieve this.

How can one tell the kernel to pin a file in memory once it was read?
Maybe with an xattr or something?
Unfortunately I dont know about the block layer and other things
involved, so I cant attach a patch that does what I want.


2005-01-03 18:30:48

by Arjan van de Ven

[permalink] [raw]
Subject: Re: pin files in memory after read

On Mon, 2005-01-03 at 19:07 +0100, Olaf Hering wrote:
> Is there a way to always keep a file (once read from disk) in memory, no
> matter how much memory pressure exists?
> There are always complains that updatedb and similar tools wipe out all
> caches. So I guess there is no such thing yet.
>
> I simply want to avoid the spinup of my ibook harddisk when something
> has been 'forgotten' and must be loaded again (like opening a new screen
> window after a while).
>
> The best I could do so far was a cramfs image. I copied it to tmpfs
> during early boot, then mount -o bind every cramfs file over the real
> binary on disk. Of course that will fail as soon as I want to update an
> affected package because the binary is busy (readonly). So there must be
> a better way to achieve this.
>
> How can one tell the kernel to pin a file in memory once it was read?
> Maybe with an xattr or something?
> Unfortunately I dont know about the block layer and other things
> involved, so I cant attach a patch that does what I want.

you could write a small userspace daemon that mmaps the file and mlock's
it....

2005-01-04 00:11:07

by Olaf Hering

[permalink] [raw]
Subject: Re: pin files in memory after read

On Mon, Jan 03, Arjan van de Ven wrote:

> you could write a small userspace daemon that mmaps the file and mlock's
> it....

Thanks.
It seems to work ok with this thing. I used this patch to find the files
with an absolute path. Any idea how to get to the relative path like
"./x" and print an absolute path for these files?

--- ../linux-2.6.10.orig/fs/open.c 2004-12-31 09:29:25.000000000 +0100
+++ ./fs/open.c 2005-01-04 00:48:30.000000000 +0100
@@ -961,6 +961,17 @@ asmlinkage long sys_open(const char __us
out:
putname(tmp);
}
+ if (0 && fd >= 0) {
+ if (filename[0] == '/' && filename[1] != '\0' && !(
+ !memcmp(filename,"/home/olaf/Mail",15) ||
+ !memcmp(filename,"/events",7) ||
+ !memcmp(filename,"/proc",5) ||
+ !memcmp(filename,"/sys",4) ||
+ !memcmp(filename,"/dev",4) ||
+ !memcmp(filename,"/var",4)
+ ))
+ printk("OP%s %s\n",current->comm, filename);
+ }
return fd;

out_error:



#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

#define file_list "/home/olaf/x"

size_t total;
void map(unsigned char *file, struct stat *sb)
{
int fd;
register unsigned char *p, c;
if (stat(file, sb) < 0)
return;
fd = open(file, O_RDONLY);
if (fd < 0)
return;
p = mmap(NULL, sb->st_size, PROT_READ, MAP_SHARED | MAP_LOCKED, fd, 0);
if (p != MAP_FAILED && (total += sb->st_size))
while (sb->st_size)
c = p[sb->st_size--];
close(fd);
return;
}

int main(int argc, char *argv[])
{
struct stat sb;
int fd, ret, line_count;
size_t len;
off_t fs;
void *flp;
unsigned char *p1, *p2;
fd = open(file_list, O_RDONLY);
if (fd < 0) {
perror(file_list);
ret = fd;
goto out;
}
ret = stat(file_list, &sb);
if (ret < 0)
goto out;
flp = mmap(NULL, sb.st_size, PROT_READ, MAP_SHARED | MAP_LOCKED, fd, 0);
if (flp == MAP_FAILED) {
perror("mmap");
goto out;
}
total += sb.st_size;
p1 = flp;
line_count = 0;
fs = sb.st_size;
while (fs > 0) {
line_count++;
printf("line %d ", line_count);
len = 0;
while (1) {
// printf("len %d\n", len);
fs--;
if (p1[len] != '\n') {
len++;
continue;
}
p2 = malloc(len);
if (p2) {
memcpy(p2, p1, len);
p2[len] = '\0';
printf("%u %s\n", len, p2);
map(p2, &sb);
free(p2);
p1 = p1 + len + 1;
}
break;
}
}
printf("sleeping ... %u\n", total);
while (1)sleep(123456789);
out:
return ret;
}

2005-01-04 00:37:34

by Chris Wright

[permalink] [raw]
Subject: Re: pin files in memory after read

* Olaf Hering ([email protected]) wrote:
> with an absolute path. Any idea how to get to the relative path like
> "./x" and print an absolute path for these files?

d_path() will give that to you, once you've resolved the path to dentry
and vfsmount pair.

thanks,
-chris
--
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net