Subject: making an in-memory hashing table ["name" -> ino_t] with thousands of entries

dear kernel hackers,

i seek advice on how to do an in-kernel lookup table capable of storing
thousands, potentially hundreds of thousands, of entries.

the reason for this is to move the fuse userspace inode lookup
tables into the kernel.

fuse userspace servers have their own in-memory database of
unique inode numbers which represent the file names, and there is a
communication mechanism between userspace and kernelspace that transfers
those inode numbers, amongst other things.

is there any _sane_ way to do this or should i leave the inode lookup
table where it presently is - in userspace?

bearing in mind that for every file accessed via a fuse
filesystem, a cache entry is created, and therefore the number
of entries could potentially run into hundreds of thousands
of entries.

l.

--
--
Truth, honesty and respect are rare commodities that all spring from
the same well: Love. If you love yourself and everyone and everything
around you, funnily and coincidentally enough, life gets a lot better.
--
<a href="http://lkcl.net"> lkcl.net </a> <br />
<a href="mailto:[email protected]"> [email protected] </a> <br />


2004-10-01 12:30:25

by Martin Waitz

[permalink] [raw]
Subject: Re: making an in-memory hashing table ["name" -> ino_t] with thousands of entries

hi :)

On Fri, Oct 01, 2004 at 01:02:22PM +0100, Luke Kenneth Casson Leighton wrote:
> bearing in mind that for every file accessed via a fuse
> filesystem, a cache entry is created, and therefore the number
> of entries could potentially run into hundreds of thousands
> of entries.

but these can be cleaned if needed.

why do you have to move all inodes into the kernel when the kernel
already cache those that are really needed?
(perhaps I'm misunderstandig fuses role here)

--
Martin Waitz


Attachments:
(No filename) (497.00 B)
(No filename) (189.00 B)
Download all attachments
Subject: Re: making an in-memory hashing table ["name" -> ino_t] with thousands of entries

On Fri, Oct 01, 2004 at 02:30:19PM +0200, Martin Waitz wrote:
> hi :)

hello martin: thanks for responding.

> On Fri, Oct 01, 2004 at 01:02:22PM +0100, Luke Kenneth Casson Leighton wrote:
> > bearing in mind that for every file accessed via a fuse
> > filesystem, a cache entry is created, and therefore the number
> > of entries could potentially run into hundreds of thousands
> > of entries.
>
> but these can be cleaned if needed.
>
> why do you have to move all inodes into the kernel when the kernel
> already cache those that are really needed?
> (perhaps I'm misunderstandig fuses role here)

fuse is a userspace filesystem.

it can be absolutely anything.

however, the author of fuse, in order to simplify the userspace
interface, manages the userspace-inode <-> userspace-fullpathnames
in a library.

for the sake of argument, let's call this the fuse-inode-cache.

entries in the fuse-inode-cache must be persistent for as long as a
filename exists on the userspace file system.

also, if an entry happens not to exist at the time a lookup
is done, then an entry into the cache is created, associating that full
path name with a newly created unique inode number (in the userspace
fuse-inode-cache).

then, that information (the unique inode number) is communicated _back_
to the fuse module, along with the stat information (yes, a userspace
lstat is done as well), and the kernel's inode structure is updated
from that information.


anyway: for various reasons, inside the kernel, it must
be possible to do a lookup from the full path name to the
inode number.


smbfs suffers from exactly the same problem: inodes don't exist on the
SMB protocol.

hm, i wonder if i should just rip stacks of code from the smbfs kernel
module?

i note with interest the existence of a function smb_refresh_inode()
which gets passed a dentry.

then i can do a getattr and go from there.

oh, man, this is hair-raising stuff.

l.



--
--
Truth, honesty and respect are rare commodities that all spring from
the same well: Love. If you love yourself and everyone and everything
around you, funnily and coincidentally enough, life gets a lot better.
--
<a href="http://lkcl.net"> lkcl.net </a> <br />
<a href="mailto:[email protected]"> [email protected] </a> <br />