[Upfront apologies if this is the wrong place to post this to.Just
give me a gentle redirect to the right place in that case.]
I've been looking into improving the boot time of an embedded linux
system (monta vista based).
While doing so I detected that initramfs initialisation was one of the
places where a lot of time was spent, so I looked into it in some more
detail and came up with the following findings & suggestions.
First proposal:
==========
initramfs is build from a compressed cpio archive.
Proposal is to introduce a build option to make the compression and
decompression optional.
Rationale 1: could be faster as it trades off I/O time (to read the
image) against decompression time
Rationale 2: for architectures that use compressed images (bzImage)
actually we compress twice, which is not really efficient.
I can implement this, but before spending time on it I would like to know if
a) people consider this a good idea
b) no one else already has doen this.
Second proposal:
============
after decompressing the cpio archive all files are made using
sys_open/sys_write/sys_close and friends.
This implies that a lot of system calls and data copying is done.
It would be nice if that could be avoided.
I'm not fully into all details of how ramfs is implemented, but would
it be possible to e.g. dump all blocks of a tmp ram fs into a data
structure (e.g.an array of blocks) while making the kernel, and while
booting the kernel initialise the fs cache with these data? (I guess
this would be around fs/dcache.c; I understand the data here is
kmalloc-ed, but it might be possible to initialise the cache with
pointers to that data structure; due to the nature of ramfs they won't
be deallocated anyway I assume).
Does this sound feasible? Hidden snags? Appreciate your
opinion/feedback/suggestions.
Best regards, Frans.
PS: if anyone has a pointer to a place which explains in full detail
how linux fs internally works (apart from the source :-) ) I
appreciate a reference. FM
Hi Frans
El Tue, Aug 05, 2008 at 09:42:11AM +0200 Frans Meulenbroeks ha dit:
> [Upfront apologies if this is the wrong place to post this to.Just
> give me a gentle redirect to the right place in that case.]
>
> I've been looking into improving the boot time of an embedded linux
> system (monta vista based).
> While doing so I detected that initramfs initialisation was one of the
> places where a lot of time was spent, so I looked into it in some more
> detail and came up with the following findings & suggestions.
>
> First proposal:
> ==========
>
> initramfs is build from a compressed cpio archive.
> Proposal is to introduce a build option to make the compression and
> decompression optional.
> Rationale 1: could be faster as it trades off I/O time (to read the
> image) against decompression time
> Rationale 2: for architectures that use compressed images (bzImage)
> actually we compress twice, which is not really efficient.
>
> I can implement this, but before spending time on it I would like to know if
> a) people consider this a good idea
> b) no one else already has doen this.
>
>
> Second proposal:
> ============
>
> after decompressing the cpio archive all files are made using
> sys_open/sys_write/sys_close and friends.
> This implies that a lot of system calls and data copying is done.
> It would be nice if that could be avoided.
> I'm not fully into all details of how ramfs is implemented, but would
> it be possible to e.g. dump all blocks of a tmp ram fs into a data
> structure (e.g.an array of blocks) while making the kernel, and while
> booting the kernel initialise the fs cache with these data? (I guess
> this would be around fs/dcache.c; I understand the data here is
> kmalloc-ed, but it might be possible to initialise the cache with
> pointers to that data structure; due to the nature of ramfs they won't
> be deallocated anyway I assume).
> Does this sound feasible? Hidden snags? Appreciate your
> opinion/feedback/suggestions.
>
> Best regards, Frans.
>
> PS: if anyone has a pointer to a place which explains in full detail
> how linux fs internally works (apart from the source :-) ) I
> appreciate a reference. FM
you might want to get in touch with Robert P. J. Day
([email protected]>) who is also working on initramfs
optimizations. see a recent thread on the kernelnewbies mailing list:
http://article.gmane.org/gmane.linux.kernel.kernelnewbies/26978
--
Matthias Kaehlcke
Embedded Linux Engineer
Barcelona
You can chain me, you can torture me, you can even
destroy this body, but you will never imprison my mind
(Mahatma Gandhi)
.''`.
using free software / Debian GNU/Linux | http://debian.org : :' :
`. `'`
gpg --keyserver pgp.mit.edu --recv-keys 47D8E5D4 `-
Frans Meulenbroeks wrote:
>
> First proposal:
> ==========
>
> initramfs is build from a compressed cpio archive.
> Proposal is to introduce a build option to make the compression and
> decompression optional.
> Rationale 1: could be faster as it trades off I/O time (to read the
> image) against decompression time
> Rationale 2: for architectures that use compressed images (bzImage)
> actually we compress twice, which is not really efficient.
>
> I can implement this, but before spending time on it I would like to know if
> a) people consider this a good idea
> b) no one else already has doen this.
>
It already is optional. If you don't want to compress it, don't.
Perhaps what you are referring to is the initramfs that is optionally
built out of the kernel tree?
You are (correctly) pointing out that if the image is already
compressed, it doesn't gain from additional compression, but that would
increase the operational memory footprint during expansion.
>
> Second proposal:
> ============
>
> after decompressing the cpio archive all files are made using
> sys_open/sys_write/sys_close and friends.
> This implies that a lot of system calls and data copying is done.
> It would be nice if that could be avoided.
> I'm not fully into all details of how ramfs is implemented, but would
> it be possible to e.g. dump all blocks of a tmp ram fs into a data
> structure (e.g.an array of blocks) while making the kernel, and while
> booting the kernel initialise the fs cache with these data? (I guess
> this would be around fs/dcache.c; I understand the data here is
> kmalloc-ed, but it might be possible to initialise the cache with
> pointers to that data structure; due to the nature of ramfs they won't
> be deallocated anyway I assume).
> Does this sound feasible? Hidden snags? Appreciate your
> opinion/feedback/suggestions.
>
The current code has a lot of advantages in terms of code complexity,
however. Your proposal would come with a dramatic increase in complexity.
-hpa
Peter,
appreciate your feedback. Some answers/clarification below:
2008/8/7 H. Peter Anvin <[email protected]>:
> Frans Meulenbroeks wrote:
>>
>> First proposal:
>> ==========
>>
>> initramfs is build from a compressed cpio archive.
>> Proposal is to introduce a build option to make the compression and
>> decompression optional.
>> Rationale 1: could be faster as it trades off I/O time (to read the
>> image) against decompression time
>> Rationale 2: for architectures that use compressed images (bzImage)
>> actually we compress twice, which is not really efficient.
>>
>> I can implement this, but before spending time on it I would like to know
>> if
>> a) people consider this a good idea
>> b) no one else already has doen this.
>>
>
> It already is optional. If you don't want to compress it, don't.
The initramfs loader (initramfs.c) indeed can deal with uncompressed
cpio files. I saw that in the code and I just did a quick test to
verify that it does. Actually I was unaware that it could until very
recently. The doc & code are not *that* clear (see below)
>
> Perhaps what you are referring to is the initramfs that is optionally built
> out of the kernel tree?
Yes.
I am building my initramfs with the kernel and there the possibility
to have an uncompressed ramfs does not exist.
Kconfig INITRAMFS_SOURCE can point to a pre-cooked .cpio file, but
that one will always be compressed by the gen_initramfs_list.sh
script.
It might be desirable to have a option to specify whether compression
is needed or not. Guess this would also imply renaming the output file
of the script to something like initramfs_data.dat or so instead of
initramfs_data.cpio.gz (as it would be confusing to have a file with
.gz extension that is not compressed). The alternative of course could
be to generate the complete initramfs_data.S file (where the .incbin
"usr/initramfs_data.cpio.gz" is, which hardcodes the name of the
file).
(BTW: please don't say that having the initramfs in the kernel image
is less desirable/builds slower/is less flexible and therefore less
desirable. I work on embedded systems and for me having a kernel with
embedded initramfs is highly convenient. I can just take my image,
load it over jtag in my hardware and boot it, without having to worry
about having a proper initramfs image somewhere on my target)
>
> You are (correctly) pointing out that if the image is already compressed, it
> doesn't gain from additional compression, but that would increase the
> operational memory footprint during expansion.
Agree about the footprint but as this is only during booting the
memory is on most systems available anyway (and I assume the memory
occupied by init.ramfs is reclaimed later on).
Disadvantage of the double compression is the waste of time for the
2nd decompression (which is probably neglectable if you have only a
tiny ramfs and/or a fast processor).
However if the initramfs is bigger and the processor slower this
starts to count in the boot time.
For embedded systems (say a router) it often makes sense to keep all
software in initramfs (as opposite to squashfs/cramfs) as the access
of the data is faster (of course at the expense of some memory as the
squashfs image is likely compressed.
Actually I did a short test on this, and loading a 6.6 MB compressed
ramfs image (24 MB after decompression) using a 200 Mhz MIPS processor
takes about 4.5 seconds. It is exactly this 4.5 seconds I want to get
rid of.
(and as some might say, if I were to use squashfs I would not get this
delay. This is true, but the delay then will happen when the
application(s) are loaded from the squashfs).
>
>>
>> Second proposal:
>> ============
>>
>> after decompressing the cpio archive all files are made using
>> sys_open/sys_write/sys_close and friends.
>> This implies that a lot of system calls and data copying is done.
>> It would be nice if that could be avoided.
>> I'm not fully into all details of how ramfs is implemented, but would
>> it be possible to e.g. dump all blocks of a tmp ram fs into a data
>> structure (e.g.an array of blocks) while making the kernel, and while
>> booting the kernel initialise the fs cache with these data? (I guess
>> this would be around fs/dcache.c; I understand the data here is
>> kmalloc-ed, but it might be possible to initialise the cache with
>> pointers to that data structure; due to the nature of ramfs they won't
>> be deallocated anyway I assume).
>> Does this sound feasible? Hidden snags? Appreciate your
>> opinion/feedback/suggestions.
>>
>
> The current code has a lot of advantages in terms of code complexity,
> however. Your proposal would come with a dramatic increase in complexity.
>
> -hpa
>
Indeed it definitely does not simplify things.
Not sure if the complexity would increase that much though
.
Actually I was only considering the initramfs that is included in the kernel.
Instead of doing things the way they are done now, we could have
initialised data structures in fs and mm.
Of course if the initramfs is loaded from an external file (with
initrd= ) it becomes a different story.
One other possibility I thought of is to keep things roughly as they
are now, but have a way to say (either in sys_write or through a new
system call) that the data is not to be copied but keep the data at
the same location (and use the memory provided by the caller)
This would of course require a few things:
- the memory contained by init.ramfs cannot be recovered (as mm will
have pointers to it)
- mm must allow that the memory for some pages is provided by the
caller (not sure if it can handle this).
At first it might seem that there is again a footprint penalty as the
init.ramfs section cannot be recovered. However the amount of memory
that mm needs in this scenario is less (as the memory for the ramfs
need not be provided by mm).
Not fully sure yet about the impact and best way forward (if any).
Note that the rationale for this is to improve boot time, especially
for embedded systems with slow processors.
As usual all feedback/suggestions/comments/whatever is appreciated.
Best regards, Frans.
Frans Meulenbroeks wrote:
>
> The initramfs loader (initramfs.c) indeed can deal with uncompressed
> cpio files. I saw that in the code and I just did a quick test to
> verify that it does. Actually I was unaware that it could until very
> recently. The doc & code are not *that* clear (see below)
>
>> Perhaps what you are referring to is the initramfs that is optionally built
>> out of the kernel tree?
>
> Yes.
> I am building my initramfs with the kernel and there the possibility
> to have an uncompressed ramfs does not exist.
> Kconfig INITRAMFS_SOURCE can point to a pre-cooked .cpio file, but
> that one will always be compressed by the gen_initramfs_list.sh
> script.
> It might be desirable to have a option to specify whether compression
> is needed or not. Guess this would also imply renaming the output file
> of the script to something like initramfs_data.dat or so instead of
> initramfs_data.cpio.gz (as it would be confusing to have a file with
> .gz extension that is not compressed). The alternative of course could
> be to generate the complete initramfs_data.S file (where the .incbin
> "usr/initramfs_data.cpio.gz" is, which hardcodes the name of the
> file).
>
Having a CONFIG option to make this image uncompressed would be a good idea.
> (BTW: please don't say that having the initramfs in the kernel image
> is less desirable/builds slower/is less flexible and therefore less
> desirable. I work on embedded systems and for me having a kernel with
> embedded initramfs is highly convenient. I can just take my image,
> load it over jtag in my hardware and boot it, without having to worry
> about having a proper initramfs image somewhere on my target)
>> You are (correctly) pointing out that if the image is already compressed, it
>> doesn't gain from additional compression, but that would increase the
>> operational memory footprint during expansion.
>
> Agree about the footprint but as this is only during booting the
> memory is on most systems available anyway (and I assume the memory
> occupied by init.ramfs is reclaimed later on).
It is, however, due to lowmem restrictions on a lot of systems this can
still hurt.
> Actually I was only considering the initramfs that is included in the kernel.
> Instead of doing things the way they are done now, we could have
> initialised data structures in fs and mm.
> Of course if the initramfs is loaded from an external file (with
> initrd= ) it becomes a different story.
>
> One other possibility I thought of is to keep things roughly as they
> are now, but have a way to say (either in sys_write or through a new
> system call) that the data is not to be copied but keep the data at
> the same location (and use the memory provided by the caller)
> This would of course require a few things:
> - the memory contained by init.ramfs cannot be recovered (as mm will
> have pointers to it)
> - mm must allow that the memory for some pages is provided by the
> caller (not sure if it can handle this).
>
> At first it might seem that there is again a footprint penalty as the
> init.ramfs section cannot be recovered. However the amount of memory
> that mm needs in this scenario is less (as the memory for the ramfs
> need not be provided by mm).
>
> Not fully sure yet about the impact and best way forward (if any).
> Note that the rationale for this is to improve boot time, especially
> for embedded systems with slow processors.
> As usual all feedback/suggestions/comments/whatever is appreciated.
I think the complexity penalty would be prohibitive; plus, on
HIGHMEM-afflicted system, this would move the initramfs from highmem to
lowmem, which would be highly undesirable.
Overall, part of the attractiveness of the current code is that it is
extremely simple. There are reasons it could use to be cleaned up (I
found its producer/consumer model to be a little to implicit to make it
easy to support loading from highmem proper) but I don't believe in the
model of going in and mucking with ramfs internals.
-hpa
[trimmed quoted text heavily; please refer to earlier msg if you need
the full text]
2008/8/7 H. Peter Anvin <[email protected]>:
> Frans Meulenbroeks wrote:
[...]
> Having a CONFIG option to make this image uncompressed would be a good idea.
Ok. Doesn't look too difficult. I'll put it on my TODO list (but don't
expect anything this weekend, fully booked already :-) )
>> Agree about the footprint but as this is only during booting the
>> memory is on most systems available anyway (and I assume the memory
>> occupied by init.ramfs is reclaimed later on).
>
> It is, however, due to lowmem restrictions on a lot of systems this can
> still hurt.
Sorry, I miss what lowmem restrictions you mean.
You're not talking 8088 or so, are you ?
Besides that: my proposed changes still could be an option. If you
have restrictions, then don't use it.
As I wrote before: for embedded systems this could make a lot of difference.
Instead of storing the application in a squashfs, you could directly
store it into the buffer cache (where it also ends up through ramfs).
Note also that I am not saying the proposal is a replacement for
initramfs. I can imagine they can live besides each other.
But for systems where fast boot is important it would help if the
gunzip/parse cpio/copy data steps can be skipped.
An initialised buffer cache (together with some associated data
structures of course) could be a nice alternative.
[...]
>
> I think the complexity penalty would be prohibitive; plus, on
> HIGHMEM-afflicted system, this would move the initramfs from highmem to
> lowmem, which would be highly undesirable.
As said before, I'm not sure what you mean with HIGHMEM. For me this
is something from the 8088/80286 era.
(and wikipedia tells me the same). Please educate me by providing a
pointer to some info.
Having said that I do not understand why this would move at all.
If I have the initramfs build into my kernel it occupies memory when
loaded. If the same data moves from the init.ramfs section to fs or mm
in order to have some initialised data structures, it just moved to a
different place in the image that is loaded.
The key difference is that the initramfs can be compressed, but if we
decide to implement an option to do things uncompressed (which you
agreed to be a good plan), then at least with an uncompressed
initramfs there does not seem too much of a difference.
>
> Overall, part of the attractiveness of the current code is that it is
> extremely simple. There are reasons it could use to be cleaned up (I found
> its producer/consumer model to be a little to implicit to make it easy to
> support loading from highmem proper) but I don't believe in the model of
> going in and mucking with ramfs internals.
The current code is indeed very simple.
The amount of complication added by another solution remains to be
investigated. Definitely it will not simplify the code, but there is
also a gain associated with it. A gain which might not be too
interesting if you run a fast PC which only boots rarely, but which is
very interesting if you have a low performance embedded system that
needs to boot up quickly (I'm thinking here about systems like set-top
boxes, digital camera's, tv's etc).
Best regards, Frans.
Afterthought on my previous mail:
By creating something like initialised buffers the total memory usage
while booting could even go down.
In the current scenario during boot time the (compressed) cpio file is
there, but the buffer memory is also allocated.
If there is a possibility to have initialised caches the amount of
memory used by the (compressed) cpio file is not needed.
This could be beneficial for embedded systems that are low on memory.
(and yes, I know the memory used by the cpio file is reclaimed later).
Frans.