Please consider pulling my linux-next branch
git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-linus.git
The patches have been posted to LKML and linux-fsdevel over the last
couple of months and have been reviewed and changed as requested.
Diffstat below.
Squashfs is a compressed read-only filesystem. It compresses metadata
and data, and uses up to 1 MiB block sizes for greater compression.
This is a comparison of Squashfs against Cramfs:
Squashfs Cramfs
Max filesystem size: 2^64 256 MiB
Max file size: ~ 2 TiB 16 MiB
Max files: unlimited unlimited
Max directories: unlimited unlimited
Max entries per directory: unlimited unlimited
Max block size: 1 MiB 4 KiB
Metadata compression: yes no
Directory indexes: yes no
Sparse file support: yes no
Tail-end packing (fragments): yes no
Exportable (NFS etc.): yes no
Hard link support: yes no
"." and ".." in readdir: yes no
Real inode numbers: yes no
32-bit uids/gids: yes no
File creation time: yes no
Xattr and ACL support: no no
Thanks
Phillip
Documentation/filesystems/squashfs.txt | 225 ++++++++++++++
MAINTAINERS | 7 +
fs/Kconfig | 52 ++++
fs/Makefile | 1 +
fs/squashfs/Makefile | 8 +
fs/squashfs/block.c | 274 +++++++++++++++++
fs/squashfs/cache.c | 412 ++++++++++++++++++++++++++
fs/squashfs/dir.c | 235 +++++++++++++++
fs/squashfs/export.c | 155 ++++++++++
fs/squashfs/file.c | 502 ++++++++++++++++++++++++++++++++
fs/squashfs/fragment.c | 98 +++++++
fs/squashfs/id.c | 94 ++++++
fs/squashfs/inode.c | 346 ++++++++++++++++++++++
fs/squashfs/namei.c | 242 +++++++++++++++
fs/squashfs/squashfs.h | 90 ++++++
fs/squashfs/squashfs_fs.h | 381 ++++++++++++++++++++++++
fs/squashfs/squashfs_fs_i.h | 45 +++
fs/squashfs/squashfs_fs_sb.h | 76 +++++
fs/squashfs/super.c | 440 ++++++++++++++++++++++++++++
fs/squashfs/symlink.c | 118 ++++++++
init/do_mounts_rd.c | 14 +
21 files changed, 3815 insertions(+), 0 deletions(-)
On Thu, Jan 08, 2009 at 04:48:51PM +0000, Phillip Lougher wrote:
> Please consider pulling my linux-next branch
>
> git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-linus.git
>
> The patches have been posted to LKML and linux-fsdevel over the last
> couple of months and have been reviewed and changed as requested.
> Diffstat below.
>
> Squashfs is a compressed read-only filesystem. It compresses metadata
> and data, and uses up to 1 MiB block sizes for greater compression.
That seems pretty fast for something only posted this week and not
even in -next.
Christoph Hellwig wrote:
> On Thu, Jan 08, 2009 at 04:48:51PM +0000, Phillip Lougher wrote:
>> Please consider pulling my linux-next branch
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-linus.git
>>
>> The patches have been posted to LKML and linux-fsdevel over the last
>> couple of months and have been reviewed and changed as requested.
>> Diffstat below.
>>
>> Squashfs is a compressed read-only filesystem. It compresses metadata
>> and data, and uses up to 1 MiB block sizes for greater compression.
>
> That seems pretty fast for something only posted this week and not
> even in -next.
>
>
It is in -next. This is the third set of patches incorporating
all the review comments from earlier patches sent since last October.
Phillip
On Thu, 8 Jan 2009, Christoph Hellwig wrote:
> On Thu, Jan 08, 2009 at 04:48:51PM +0000, Phillip Lougher wrote:
> > Please consider pulling my linux-next branch
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-linus.git
> >
> > The patches have been posted to LKML and linux-fsdevel over the last
> > couple of months and have been reviewed and changed as requested.
> > Diffstat below.
> >
> > Squashfs is a compressed read-only filesystem. It compresses metadata
> > and data, and uses up to 1 MiB block sizes for greater compression.
>
> That seems pretty fast for something only posted this week and not
The patches have been posted before last October, and the recent changes are
according to the review comments.
> even in -next.
Incorrect, as of next-20090106 ;-)
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: [email protected]
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
Christoph Hellwig wrote:
>>
>> Squashfs is a compressed read-only filesystem. It compresses metadata
>> and data, and uses up to 1 MiB block sizes for greater compression.
>
> That seems pretty fast for something only posted this week and not
> even in -next.
>
It is in -next, but, to be fair, it has trivial impact on the rest of
the system, which should also be considered.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
On Thu, 8 Jan 2009 11:50:29 -0500 Christoph Hellwig <[email protected]> wrote:
> On Thu, Jan 08, 2009 at 04:48:51PM +0000, Phillip Lougher wrote:
> > Please consider pulling my linux-next branch
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-linus.git
> >
> > The patches have been posted to LKML and linux-fsdevel over the last
> > couple of months and have been reviewed and changed as requested.
> > Diffstat below.
> >
> > Squashfs is a compressed read-only filesystem. It compresses metadata
> > and data, and uses up to 1 MiB block sizes for greater compression.
>
> That seems pretty fast for something only posted this week and not
> even in -next.
I discussed this with Phillip a bit, and it looks like we'll go for a
2.6.21 merge. This gives a couple of months for testing and review in
linux-next.
Of course, this assumes that someone will actually perform that testing
and review. Chances are that it'll just sit there, getting nothing more
than compilation testing.
Apparently I reviewed the code a while back - that grey cell must have
died. Does anyone else intend to review the code in that timeframe?
If not, we might as well merge it now, if ever..
Andrew Morton wrote:
> On Thu, 8 Jan 2009 11:50:29 -0500 Christoph Hellwig <[email protected]> wrote:
>
>> On Thu, Jan 08, 2009 at 04:48:51PM +0000, Phillip Lougher wrote:
>>> Please consider pulling my linux-next branch
>>>
>>> git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-linus.git
>>>
>>> The patches have been posted to LKML and linux-fsdevel over the last
>>> couple of months and have been reviewed and changed as requested.
>>> Diffstat below.
>>>
>>> Squashfs is a compressed read-only filesystem. It compresses metadata
>>> and data, and uses up to 1 MiB block sizes for greater compression.
>> That seems pretty fast for something only posted this week and not
>> even in -next.
>
> I discussed this with Phillip a bit, and it looks like we'll go for a
> 2.6.21 merge. This gives a couple of months for testing and review in
> linux-next.
>
> Of course, this assumes that someone will actually perform that testing
> and review. Chances are that it'll just sit there, getting nothing more
> than compilation testing.
Yes please review! If people feels it needs a couple more months in
linux-next with more in-depth review then I'll live with that. I'll
fix anything more that comes up, because I'm desperate to get it
mainlined. I've lived the last couple of years with constant
moaning from people that I've not mainlined it, and I took
the last six months off work to work on it full time to get
it into shape to mainline.
Hopefully now it's at least in sufficient shape to go into
Linux-next I can get another job, and get some income which
will be nice.
>
> Apparently I reviewed the code a while back - that grey cell must have
> died. Does anyone else intend to review the code in that timeframe?
> If not, we might as well merge it now, if ever..
>
I'll be very happy for a merge now, but of course it's not up to me :-(
Phillip
>
On Fri, Jan 9, 2009 at 03:11, Phillip Lougher
<[email protected]> wrote:
> Andrew Morton wrote:
>> Apparently I reviewed the code a while back - that grey cell must have
>> died. Does anyone else intend to review the code in that timeframe? If
>> not, we might as well merge it now, if ever..
>>
>
> I'll be very happy for a merge now, but of course it's not up to me :-(
I guess, it is hard to find a larger distro that does not merge the
out-of-tree version to support their live-, and installer-cds. So,
what argument is preventing a mainline merge now?
Thanks,
Kay
On Thu, 2009-01-08 at 17:53 -0800, Andrew Morton wrote:
> On Thu, 8 Jan 2009 11:50:29 -0500 Christoph Hellwig <[email protected]> wrote:
>
> > On Thu, Jan 08, 2009 at 04:48:51PM +0000, Phillip Lougher wrote:
> > > Please consider pulling my linux-next branch
> > >
> > > git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-linus.git
> > >
> > > The patches have been posted to LKML and linux-fsdevel over the last
> > > couple of months and have been reviewed and changed as requested.
> > > Diffstat below.
> > >
> > > Squashfs is a compressed read-only filesystem. It compresses metadata
> > > and data, and uses up to 1 MiB block sizes for greater compression.
> >
> > That seems pretty fast for something only posted this week and not
> > even in -next.
>
> I discussed this with Phillip a bit, and it looks like we'll go for a
> 2.6.21 merge. This gives a couple of months for testing and review in
> linux-next.
akpm time machine? I'll assume that means squashfs is headed for a
2.6.30 merge? Or .29?
Harvey
On Fri, Jan 09, 2009 at 03:24:40AM +0100, Kay Sievers wrote:
> On Fri, Jan 9, 2009 at 03:11, Phillip Lougher
> <[email protected]> wrote:
> > Andrew Morton wrote:
> >> Apparently I reviewed the code a while back - that grey cell must have
> >> died. Does anyone else intend to review the code in that timeframe? If
> >> not, we might as well merge it now, if ever..
> >>
> >
> > I'll be very happy for a merge now, but of course it's not up to me :-(
>
> I guess, it is hard to find a larger distro that does not merge the
> out-of-tree version to support their live-, and installer-cds. So,
> what argument is preventing a mainline merge now?
Agreed...I'm tired of having to pull along a patch to vanilla kernels
for squashfs. Drivers are merged that don't even _work_ these days,
so why is there so much concern about merging something which almost
all distros use which impacts almost nothing if you don't enable it???
Phil
> I discussed this with Phillip a bit, and it looks like we'll go for a
> 2.6.21 merge. This gives a couple of months for testing and review in
> linux-next.
>
> Of course, this assumes that someone will actually perform that testing
> and review. Chances are that it'll just sit there, getting nothing more
> than compilation testing.
As far as I heared from japanese embedded developer, squashfs is already used by
embedded community and some consumer electronics product.
Therefore, I think it can merge now if it's no impact to other subsystem.
> Apparently I reviewed the code a while back - that grey cell must have
> died. Does anyone else intend to review the code in that timeframe?
> If not, we might as well merge it now, if ever..
> Of course, this assumes that someone will actually perform that testing
> and review. Chances are that it'll just sit there, getting nothing more
> than compilation testing.
Almost every distribution ships the updated version and has done for ages
- exactly what kind of testing is going to beat that ?
On Thu, 8 Jan 2009, Andrew Morton wrote:
> On Thu, 8 Jan 2009 11:50:29 -0500 Christoph Hellwig <[email protected]> wrote:
> > On Thu, Jan 08, 2009 at 04:48:51PM +0000, Phillip Lougher wrote:
> > > Please consider pulling my linux-next branch
> > >
> > > git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-linus.git
> > >
> > > The patches have been posted to LKML and linux-fsdevel over the last
> > > couple of months and have been reviewed and changed as requested.
> > > Diffstat below.
> > >
> > > Squashfs is a compressed read-only filesystem. It compresses metadata
> > > and data, and uses up to 1 MiB block sizes for greater compression.
> >
> > That seems pretty fast for something only posted this week and not
> > even in -next.
>
> I discussed this with Phillip a bit, and it looks like we'll go for a
> 2.6.21 merge. This gives a couple of months for testing and review in
^^
(assuming single digit flips :-) 29 or 31? I hope 29...
> linux-next.
>
> Of course, this assumes that someone will actually perform that testing
> and review. Chances are that it'll just sit there, getting nothing more
> than compilation testing.
FWIW, I've been running tests on squashfs during the last few months, on
various platforms (32/64 bit, little/big endian, forgiving/notforgiving
unaligned accesses). I admit not having tested the latest squashfs version on
all of them, as some are partially broken due to the current merge window
churn.
> Apparently I reviewed the code a while back - that grey cell must have
> died. Does anyone else intend to review the code in that timeframe?
> If not, we might as well merge it now, if ever..
And I've kept a close eye on all changes in the squashfs git repository since
the last formal review on lkml.
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: [email protected]
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
On Thu, 8 January 2009 18:36:29 -0800, Phil Oester wrote:
>
> Agreed...I'm tired of having to pull along a patch to vanilla kernels
> for squashfs. Drivers are merged that don't even _work_ these days,
> so why is there so much concern about merging something which almost
> all distros use which impacts almost nothing if you don't enable it???
In general, filesystems and ABI changes are special because stupid
mistakes are eternal. If some device driver has a bug, you can fix it,
reboot and be done with it. Not so with filesystems.
Jörn
--
to show off how geeky they were.
-- Rob Enderle
On Fri, Jan 09, 2009 at 05:54:22PM +0100, J?rn Engel wrote:
>On Thu, 8 January 2009 18:36:29 -0800, Phil Oester wrote:
>>
>> Agreed...I'm tired of having to pull along a patch to vanilla kernels
>> for squashfs. Drivers are merged that don't even _work_ these days,
>> so why is there so much concern about merging something which almost
>> all distros use which impacts almost nothing if you don't enable it???
>
>In general, filesystems and ABI changes are special because stupid
>mistakes are eternal. If some device driver has a bug, you can fix it,
>reboot and be done with it. Not so with filesystems.
Squashfs is readonly from the kernel. The images are created with
userspace tools.
David
On Fri, 9 January 2009 11:37:39 -0800, David Brown wrote:
> On Fri, Jan 09, 2009 at 05:54:22PM +0100, Jörn Engel wrote:
> >
> >In general, filesystems and ABI changes are special because stupid
> >mistakes are eternal. If some device driver has a bug, you can fix it,
> >reboot and be done with it. Not so with filesystems.
>
> Squashfs is readonly from the kernel. The images are created with
> userspace tools.
While true, it doesn't make a difference. If, for example, your
structures members are not naturally aligned, you take a performance hit
for no good reason. Simply moving fields around would make the code
go faster. But the format is fixed and prevents you from making this
change.
You have to get those things right from the beginning or pay for your
mistakes everafter. In general (and I stress "In general") filesystems
want more review than ordinary device drivers. And just to stress that
again, this is not an argument against merging squashfs now.
Jörn
--
Simplicity is prerequisite for reliability.
-- Edsger W. Dijkstra
On Fri, 9 Jan 2009, Alan Cox wrote:
> > Of course, this assumes that someone will actually perform that testing
> > and review. Chances are that it'll just sit there, getting nothing more
> > than compilation testing.
>
> Almost every distribution ships the updated version and has done for ages
> - exactly what kind of testing is going to beat that ?
Yes, if this is really in use by everybody, then not merging it is kind of
pointless.
Linus
On Fri, 9 Jan 2009 13:56:54 -0800 (PST)
Linus Torvalds <[email protected]> wrote:
>
>
> On Fri, 9 Jan 2009, Alan Cox wrote:
>
> > > Of course, this assumes that someone will actually perform that testing
> > > and review. Chances are that it'll just sit there, getting nothing more
> > > than compilation testing.
> >
> > Almost every distribution ships the updated version and has done for ages
> > - exactly what kind of testing is going to beat that ?
>
> Yes, if this is really in use by everybody, then not merging it is kind of
> pointless.
>
Merge it!
We've long needed a filesystem named after a vegetable.
On Fri, 9 Jan 2009, Andrew Morton wrote:
>
> We've long needed a filesystem named after a vegetable.
You make a very compelling case.
Linus
On Fri, 9 Jan 2009 14:12:34 -0800 (PST)
Linus Torvalds <[email protected]> wrote:
>
>
> On Fri, 9 Jan 2009, Andrew Morton wrote:
> >
> > We've long needed a filesystem named after a vegetable.
>
> You make a very compelling case.
And vegetables are much healtheir than FAT ...
On Fri, 2009-01-09 at 22:26 +0000, Alan Cox wrote:
> On Fri, 9 Jan 2009 14:12:34 -0800 (PST)
> Linus Torvalds <[email protected]> wrote:
>
> >
> >
> > On Fri, 9 Jan 2009, Andrew Morton wrote:
> > >
> > > We've long needed a filesystem named after a vegetable.
> >
> > You make a very compelling case.
>
> And vegetables are much healtheir than FAT ...
I always preferred my squash with butter, it appears 2.6.30 will be a very
tasty kernel.
Harvey
* J?rn Engel <[email protected]> wrote:
> On Fri, 9 January 2009 11:37:39 -0800, David Brown wrote:
> > On Fri, Jan 09, 2009 at 05:54:22PM +0100, J?rn Engel wrote:
> > >
> > > In general, filesystems and ABI changes are special because stupid
> > > mistakes are eternal. If some device driver has a bug, you can fix
> > > it, reboot and be done with it. Not so with filesystems.
> >
> > Squashfs is readonly from the kernel. The images are created with
> > userspace tools.
>
> While true, it doesn't make a difference. If, for example, your
> structures members are not naturally aligned, you take a performance hit
> for no good reason. Simply moving fields around would make the code go
> faster. But the format is fixed and prevents you from making this
> change.
>
> You have to get those things right from the beginning or pay for your
> mistakes everafter. In general (and I stress "In general") filesystems
> want more review than ordinary device drivers. And just to stress that
> again, this is not an argument against merging squashfs now.
What does a performance hit have to do with an ABI? Absolutely nothing -
if such a bug is noticed it is fixed, that's it. Your argument does not
parse and makes absolutely zero technical sense.
Your "ABI is forever" objection against a _read only_ filesystem is a
newbie mistake worthy of cookie file inclusion - i had a real good laugh
when i read it ;-)
Ingo
On Sat, 10 January 2009 13:43:35 +0100, Ingo Molnar wrote:
>
> What does a performance hit have to do with an ABI? Absolutely nothing -
> if such a bug is noticed it is fixed, that's it. Your argument does not
> parse and makes absolutely zero technical sense.
>
> Your "ABI is forever" objection against a _read only_ filesystem is a
> newbie mistake worthy of cookie file inclusion - i had a real good laugh
> when i read it ;-)
Thank you, glad to be of service. Should I have picked an example where
the code becomes horribly convoluted and there is nothing you can do
about it?
But since I am clearly the newbie, could you try to teach my stupid ass
instead of just ridiculing it? What is the thing that makes a read only
filesystem special? And why does everyone believe that I am arguing
against merging squashfs when I'm not?
Jörn
--
The only real mistake is the one from which we learn nothing.
-- John Powell
On Sat, 10 Jan 2009 17:50:33 +0100 J__rn Engel <[email protected]> wrote:
> What is the thing that makes a read only
> filesystem special?
Nothing, really.
The filesystem has mount options (I think). That interface needs to be
maintained.
More importantly, the filesystem driver has to be able to read older
filesystem instances. This is a userspace-visible binary interface!
A really complex one.
If for some reason we wish to change the on-disk format then that could
be done now. But once the code is merged, such changes could only be
done in a back-compatible way.
And the day-one code (ie: this code) would need to be designed so that
such on-disk changes can be made - we don't want old kernels exploding
when asked to read new-layout filesystem instances. This is what `grep
-i compat include/linux/ext2_fs.h' is all about.
On Sat, 10 Jan 2009, Andrew Morton wrote:
>
> More importantly, the filesystem driver has to be able to read older
> filesystem instances. This is a userspace-visible binary interface!
> A really complex one.
Well, the good news is that read-only filesystems are _sooo_ much simpler
than any real filesystem that quite frankly, on a "complexity" scale it's
still way way down there.
Also, if it's not used for backup (and I don't think anybody would),
there's actually less reason to be back-wards compatible. I know I changed
cramfs a few times incompatibly, simply becaus "you might as well just
re-run the user tools to generate the image". It was for a similar need,
and the image really always goes along with the kernel.
I think squashfs usage would be similar - you'd not have squashfs as a
standalone media, it would be a "installation medium" thing.
Linus
On Sat, Jan 10, 2009 at 10:12:35AM -0800, Andrew Morton wrote:
> More importantly, the filesystem driver has to be able to read older
> filesystem instances. This is a userspace-visible binary interface!
> A really complex one.
There already have been format changes in squashfs. My experience has
been that the developpers consider problems with the support of the
older formats bugs that have to be fixed (it happened with a
combination of lack of luck and integrated buggy alpha in an old
Fedora with the 3.x transition).
OG.
Linus Torvalds wrote:
> I think squashfs usage would be similar - you'd not have squashfs as a
> standalone media, it would be a "installation medium" thing.
Plenty of existing, active squashfs use on router firewall standalone
media. Here is my home router running unmodified OpenWRT:
root@gw:~# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro)
none on /dev type devfs (rw)
none on /proc type proc (rw)
none on /tmp type tmpfs (rw,nosuid,nodev)
none on /dev/pts type devpts (rw)
/dev/mtdblock/4 on /jffs type jffs2 (rw)
mini_fo:/jffs on / type mini_fo (rw)
The squashfs area stores the vast majority of the system; log files and
other writable updates to the router go into the smaller jffs2
filesystem. Symlinks stitch the whole mess together.
It will be nice to finally get squashfs in the kernel, since it is
actively used all over the place in embedded projects like OpenWRT.
Regards,
Jeff
Hello,
On Sat, Jan 10, 2009 at 8:57 PM, Jeff Garzik <[email protected]> wrote:
>> I think squashfs usage would be similar - you'd not have squashfs as a
>> standalone media, it would be a "installation medium" thing.
>
> Plenty of existing, active squashfs use on router firewall standalone media.
> Here is my home router running unmodified OpenWRT:
>
> It will be nice to finally get squashfs in the kernel, since it is actively
> used all over the place in embedded projects like OpenWRT.
>
Indeed, in embedded projects squashfs is" Used Everywhere Where Size Matters".
The adds read: "Even my girlfriend recommends it for everyday use."
(At least, that's what I've heard.)
Thanks for the people who could review and make the upcoming merger happen,
and kudos to Phillip.
Regards,
--
Leon
* Andrew Morton <[email protected]> wrote:
> More importantly, the filesystem driver has to be able to read older
> filesystem instances. This is a userspace-visible binary interface! A
> really complex one.
>
> If for some reason we wish to change the on-disk format then that could
> be done now. But once the code is merged, such changes could only be
> done in a back-compatible way.
IMHO what makes squashfs special is that:
1) it's read-only: i.e. we dont actually _generate_ this data structure.
It comes from the outside.
2) it's a the "cat is already out of the bag" situation.
It's in the field, it's used.
And as such it's pretty much an externality already to Linux: the
filesystem is in use, distros patch it in, devices ship with it, etc. etc.
So the main technical question at this stage is whether we want to support
that binary format _at all_, whether we want to support that kind of data
protocol.
Also, i think your interpretation of 'ABI' stretches it quite far: we dont
control the lowlevel bits at all - we dont generate the data, we just
interpret it in the kernel in a read-only way. We dont provide the raw
lowlevel bits to user-space either - we provide VFS bindings to it and
that is generalized and not touched by squashfs.
As such squashfs has little to no classic ABI bindings - other than the
trivial "do we support this data format" question. (which is not a classic
ABI question.)
[ Often such read-only formats can even be iterated slightly incompatibly
without much fuss, as the userspace side goes together with the kernel
anyway so it's easy to stay in touch. ]
And the thing is, i think we never before in the kernel said "no" to any
support for a data format externality. [Let me qualify that: 'unpure' data
protocols with active 'code' components were always an exception to that:
ACPI, reiser4, etc. - but read-only access to a passive medium is far more
clear-cut.]
This is a nice filesystem, and we support far, far worse data formats
already. Far, far, far, far worse data formats, formats that are less used
and were developed completely regardless of Linux. We support data formats
and physical transports that should not even be mentioned with squashfs in
the same paragraph.
And the thing is, if in the mainline kernel we were at the forefront of FS
R&D, we would and could prototype such filesystems in the mainline kernel
and we would have efficient and prompt influence over lowlevel format
decisions.
We are not at the forefront - distros tend to apply new filesystem patches
far earlier than we do and it takes forever to get stuff upstream - for
better or worse. And we (the upstream community) had little active role in
developing this thing _at all_.
( This is not a criticism it's just an observation of current reality: it
is _fundamentally hard_ to do active high-flux R&D for persistent
formats upstream and still be reasonably compatible at the same time.
So developing out-of-tree - or via FUSE - might as well be the right
model here. )
Generally when a filesystem driver comes to us, its lowlevel format is
pretty much a done deal already - it's out in the wild and we should say
'no' only as an exception mechanism for clearly unacceptable crap.
Instead of trying to flex our muscle and steer the big red firetruck way
after the fire has been put out already - by others.
Saying 'no' at this stage comes at a great and largely unnecessary cost to
everyone involved. I believe we force ourselves into the R&D flow at an
inappropriately late stage - while at the same time we are unreceptive to
early adopter projects who'd like to avoid that. We cannot have the cake
and eat it too.
At least IMHO.
( What could _perhaps_ change the picture a bit IMO is drivers/staging/ i
think - we could take a far more active role in certain types of
projects that have been done out of tree typically, with no formal
promise for compatibility - or something like that. )
Ingo
Linus Torvalds wrote:
>
> On Fri, 9 Jan 2009, Alan Cox wrote:
>
>>> Of course, this assumes that someone will actually perform that testing
>>> and review. Chances are that it'll just sit there, getting nothing more
>>> than compilation testing.
>> Almost every distribution ships the updated version and has done for ages
>> - exactly what kind of testing is going to beat that ?
>
> Yes, if this is really in use by everybody, then not merging it is kind of
> pointless.
>
> Linus
>
Great news... Thanks everyone for reviewing Squashfs and helping
to get it merged.
Phillip
Phillip Lougher wrote:
>
> Great news... Thanks everyone for reviewing Squashfs and helping
> to get it merged.
>
Excellent news!
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
On Sat, 10 Jan 2009 10:30:23 PST, Linus Torvalds said:
> I think squashfs usage would be similar - you'd not have squashfs as a
> standalone media, it would be a "installation medium" thing.
Actually, for some of us, squashfs would *usually* be a standalone - my biggest
application for it is when I end up having to can-opener some wonky install
medium that one of my users is having a problem with. And of course, I
have to use some *other* kernel/system to do the can-opener trick with, because
if the one on the medium worked, the user wouldn't be standing in my office :)
Fortunately, that usually means I'm using a recent -mm kernel on my laptop,
and the kernel/medium I'm trying to debug is almost always older, so I don't
have to worry much about a too-new medium. Anybody comes in my office with
something even more bleeding edge than what I have, I damn well expect them
to be able to debug the issue themselves. ;)
On Sat, 10 Jan 2009, Ingo Molnar wrote:
> * Andrew Morton <[email protected]> wrote:
>
> > More importantly, the filesystem driver has to be able to read older
> > filesystem instances. This is a userspace-visible binary interface! A
> > really complex one.
> >
> > If for some reason we wish to change the on-disk format then that could
> > be done now. But once the code is merged, such changes could only be
> > done in a back-compatible way.
>
> IMHO what makes squashfs special is that:
>
> 1) it's read-only: i.e. we dont actually _generate_ this data structure.
> It comes from the outside.
This is not 100% true: the squashfs layout _has_ been changed in V4, in
response to review comments from the Linux kernel community on V3.3/V3.4.
> 2) it's a the "cat is already out of the bag" situation.
> It's in the field, it's used.
AFAIK, all existing users use pre-V4. Let's hope this is going to change soon.
Apart from the use for backups (old ones have to be read using e.g. unsquashfs
now), I think this is OK: in most situations (embedded, Live CDs), the kernel
and the file system image are generated together (this is what e.g. OpenWRT
does).
> Generally when a filesystem driver comes to us, its lowlevel format is
> pretty much a done deal already - it's out in the wild and we should say
> 'no' only as an exception mechanism for clearly unacceptable crap.
>
> Instead of trying to flex our muscle and steer the big red firetruck way
> after the fire has been put out already - by others.
>
> Saying 'no' at this stage comes at a great and largely unnecessary cost to
> everyone involved. I believe we force ourselves into the R&D flow at an
> inappropriately late stage - while at the same time we are unreceptive to
> early adopter projects who'd like to avoid that. We cannot have the cake
> and eat it too.
>
> At least IMHO.
>
> ( What could _perhaps_ change the picture a bit IMO is drivers/staging/ i
> think - we could take a far more active role in certain types of
> projects that have been done out of tree typically, with no formal
> promise for compatibility - or something like that. )
So if staging would have existed +one year ago, we should probably have
included squashfs 3.3 at that time, and just have moved it to fs/ once the V4
layout was finished?
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
* Geert Uytterhoeven <[email protected]> wrote:
> > At least IMHO.
> >
> > ( What could _perhaps_ change the picture a bit IMO is drivers/staging/ i
> > think - we could take a far more active role in certain types of
> > projects that have been done out of tree typically, with no formal
> > promise for compatibility - or something like that. )
>
> So if staging would have existed +one year ago, we should probably have
> included squashfs 3.3 at that time, and just have moved it to fs/ once
> the V4 layout was finished?
might have been a possibility - although there's a notable absence of any
filesystem drivers in the latest drivers/staging/ set.
Greg, is that just accidental (no one has submitted one yet) or is it some
sort of policy?
Ingo
On Sun, Jan 11, 2009 at 04:39:20PM +0100, Ingo Molnar wrote:
>
> * Geert Uytterhoeven <[email protected]> wrote:
>
> > > At least IMHO.
> > >
> > > ( What could _perhaps_ change the picture a bit IMO is drivers/staging/ i
> > > think - we could take a far more active role in certain types of
> > > projects that have been done out of tree typically, with no formal
> > > promise for compatibility - or something like that. )
> >
> > So if staging would have existed +one year ago, we should probably have
> > included squashfs 3.3 at that time, and just have moved it to fs/ once
> > the V4 layout was finished?
>
> might have been a possibility - although there's a notable absence of any
> filesystem drivers in the latest drivers/staging/ set.
>
> Greg, is that just accidental (no one has submitted one yet) or is it some
> sort of policy?
I was wanting to stick with drivers to start with, but I really have no
objection to adding filesystems, if they are self-contained, to the
drivers/staging/ directory.
I looked at adding squashfs, but at the time, it touched other portions
of the kernel which wouldn't have made it a good canidate for staging.
This was later resolved, and now that it is merged, it's a moot issue :)
So, if anyone wants to send me filesystems, I'll be glad to take them
into drivers/staging, as long as they are self-contained (novfs for
example would fit this category.)
thanks,
greg k-h
On Sun, Jan 11, 2009 at 08:30:18AM -0800, Greg KH wrote:
> I was wanting to stick with drivers to start with, but I really have no
> objection to adding filesystems, if they are self-contained, to the
> drivers/staging/ directory.
>
> I looked at adding squashfs, but at the time, it touched other portions
> of the kernel which wouldn't have made it a good canidate for staging.
> This was later resolved, and now that it is merged, it's a moot issue :)
>
> So, if anyone wants to send me filesystems, I'll be glad to take them
> into drivers/staging, as long as they are self-contained (novfs for
> example would fit this category.)
Filesystems in staging worries me.
* The number of people who competently review filesystem code
(and I mean real review here, not checkpatch & codingstyle crap)
is significantly less than those who review drivers.
I foresee stuff just lingering there for years.
(Look how long fs stuff hangs around unmerged in -mm for example).
* The fallout of staging is already starting to drift into distros.
Within a week of Fedora shipping a kernel that had staging/
we had requests to enable drivers from it.
And of course, those drivers were garbage.
This is only going to increase as time goes on.
* For crap drivers that a minority cares about, this isn't a big deal
to tell the users "build it yourself, we don't support it when stuff breaks".
(And a lot of that crap will break. NetworkManager won't work properly
with some of the wireless crap in staging for example), but by
continually adding to the shitpile the potential for review dramatically
gets reduced, and for something as critical as a filesystem, I find this
absolutely terrifying from a support perspective.
I don't mean to piss on your parade, but from my viewpoint, staging
is a trainwreck so far, and I'd hate to see it get worse.
We've already demonstrated "look how much stuff we can merge" time and
time again, but no-one ever seems to have a proposal for how we increase
the amount of review code gets before it's merged.
There's lowering the barrier for entry, and there's not having a barrier at all.
The latter is what I'm concerned that staging/ has become.
Dave
--
http://www.codemonkey.org.uk
Dave Jones wrote:
> On Sun, Jan 11, 2009 at 08:30:18AM -0800, Greg KH wrote:
>
> > I was wanting to stick with drivers to start with, but I really have no
> > objection to adding filesystems, if they are self-contained, to the
> > drivers/staging/ directory.
> >
> > I looked at adding squashfs, but at the time, it touched other portions
> > of the kernel which wouldn't have made it a good canidate for staging.
> > This was later resolved, and now that it is merged, it's a moot issue :)
> >
> > So, if anyone wants to send me filesystems, I'll be glad to take them
> > into drivers/staging, as long as they are self-contained (novfs for
> > example would fit this category.)
>
> Filesystems in staging worries me.
>
> * The number of people who competently review filesystem code
> (and I mean real review here, not checkpatch & codingstyle crap)
> is significantly less than those who review drivers.
> I foresee stuff just lingering there for years.
> (Look how long fs stuff hangs around unmerged in -mm for example).
>
> * The fallout of staging is already starting to drift into distros.
> Within a week of Fedora shipping a kernel that had staging/
> we had requests to enable drivers from it.
> And of course, those drivers were garbage.
> This is only going to increase as time goes on.
>
> * For crap drivers that a minority cares about, this isn't a big deal
> to tell the users "build it yourself, we don't support it when stuff breaks".
> (And a lot of that crap will break. NetworkManager won't work properly
> with some of the wireless crap in staging for example), but by
> continually adding to the shitpile the potential for review dramatically
> gets reduced, and for something as critical as a filesystem, I find this
> absolutely terrifying from a support perspective.
>
> I don't mean to piss on your parade, but from my viewpoint, staging
> is a trainwreck so far, and I'd hate to see it get worse.
>
> We've already demonstrated "look how much stuff we can merge" time and
> time again, but no-one ever seems to have a proposal for how we increase
> the amount of review code gets before it's merged.
>
> There's lowering the barrier for entry, and there's not having a barrier at all.
> The latter is what I'm concerned that staging/ has become.
I agree. Alexey D. asked about that a few days ago and the Greg's answer
about what he would accept was "anything". Ugh ugh ugh. I did not like
that reply at all.
I agree that crap is the right name for lots of it. For the ones that
people & distros care about, someone should step up and do some real
work on them.
~Randy
On Thu, Jan 22, 2009 at 04:50:41PM -0500, Dave Jones wrote:
> On Sun, Jan 11, 2009 at 08:30:18AM -0800, Greg KH wrote:
>
> > I was wanting to stick with drivers to start with, but I really have no
> > objection to adding filesystems, if they are self-contained, to the
> > drivers/staging/ directory.
> >
> > I looked at adding squashfs, but at the time, it touched other portions
> > of the kernel which wouldn't have made it a good canidate for staging.
> > This was later resolved, and now that it is merged, it's a moot issue :)
> >
> > So, if anyone wants to send me filesystems, I'll be glad to take them
> > into drivers/staging, as long as they are self-contained (novfs for
> > example would fit this category.)
>
> Filesystems in staging worries me.
>
> * The number of people who competently review filesystem code
> (and I mean real review here, not checkpatch & codingstyle crap)
> is significantly less than those who review drivers.
> I foresee stuff just lingering there for years.
> (Look how long fs stuff hangs around unmerged in -mm for example).
If it lingers, it has a better chance of getting cleaned up here, than
if it lives in -mm, right?
> * The fallout of staging is already starting to drift into distros.
> Within a week of Fedora shipping a kernel that had staging/
> we had requests to enable drivers from it.
> And of course, those drivers were garbage.
> This is only going to increase as time goes on.
That's up to you as a distro to handle, not much I can do there.
But, if you want a recommendation, some of the drivers in staging came
from the Fedora kernel tree, so you should be enabling them :)
> * For crap drivers that a minority cares about, this isn't a big deal
> to tell the users "build it yourself, we don't support it when stuff breaks".
> (And a lot of that crap will break. NetworkManager won't work properly
> with some of the wireless crap in staging for example), but by
> continually adding to the shitpile the potential for review dramatically
> gets reduced, and for something as critical as a filesystem, I find this
> absolutely terrifying from a support perspective.
>From a support perspective, if I see the TAINT_CRAP flag set, we just
instantly close the bug out. Already done that a number of times for
openSUSE bugs, with no problems.
> I don't mean to piss on your parade, but from my viewpoint, staging
> is a trainwreck so far, and I'd hate to see it get worse.
What is wrong with it? Bugs are getting fixed, people are getting use
out of their hardware (hell, Linus is even using one of the drivers),
and lots of developers are cutting their teeth on helping out.
If you don't like it, just disable it in your kernel packages, or
instantly close out the bugs. The drivers in staging has already helped
out some distros by virtue of including newer drivers than they were
mistakenly using at the time (Ubuntu, I'm looking at you...)
And again, it's helped out users, which is the most important thing
here.
thanks,
greg k-h
On Thu, Jan 22, 2009 at 01:57:53PM -0800, Randy Dunlap wrote:
> I agree that crap is the right name for lots of it. For the ones that
> people & distros care about, someone should step up and do some real
> work on them.
Are people ignoring the fact that there _are_ people stepping up and
doing real work on these drivers? Look at all of the cleanup patches
that went into 2.6.29-rc1 in drivers/staging. There were hundreds.
And I already have 75+ staging cleanup patches in -next right now.
This work is being done right now, is it somehow being missed?
Yeah, compiler warnings are annoying, I totally agree, that is why it is
very hard to enable the staging tree. To do so, you really have to want
to, and I'm eagerly taking any patch sent to me to do this work.
thanks,
greg k-h
* Dave Jones <[email protected]> wrote:
> We've already demonstrated "look how much stuff we can merge" time and
> time again, but no-one ever seems to have a proposal for how we increase
> the amount of review code gets before it's merged.
I think i agree with you in some ways (more review is always good), but
there's a very important aspect where i think you are quite wrong:
The first step towards increasing review activity is to increase the
readability of the code - so that is _can_ be reviewed quickly and
efficiently.
But talking down such efforts with:
> (and I mean real review here, not checkpatch & codingstyle crap)
is a shoot-own-foot excercise.
You are attacking the very foundation of proper high-level review. Yes,
there can be bad types of syntactic cleanups but most of the efforts add
real value to the code and should not be dismissed just because they stay
on the local level.
"real review" only becomes easy if the code _is reviewable to begin with_:
if it is written in standard coding patterns where we almost
sub-consciously recognize bad constructs and bad practices.
I've seen it time and again that if the code is cleaned up visually, real
review and real improvements follow eventually. It's a gradual process and
you just cannot do "real review" efficiently without what you call the
"checkpatch crap".
In fact, i claim that doing "real review" on butt-ugly code is a waste of
time and resources, and that it is also _harmful_. By doing real review on
something that is not even right stylistically, you insert value into it.
That way you _encourage_ that author of that ugly piece of code to
contribute more code in the same fashion. You indirectly harm Linux that
way because you encourage bad taste.
I strongly support the notion that high-level review is only warranted on
code that is reviewable and looks tasteful, and that code which doesnt
meet basic style should not be merged at all.
Code that 'looks good' can of course still be utter crap - but that crap
is usually easily noticed, and often the crap portion does not permeate
the whole code base. But if we have code that _does not even look good_
then crap can hide for a long time. (obscured by filth, so to speak.)
I know plenty of in-tree examples of 5-10 years old code that has remained
butt-ugly and unhackable for 5-10 years and which has an above-average
proportion of bugs and regressions.
Ingo
Hi Ingo,
On Fri, Jan 23, 2009 at 12:16 AM, Ingo Molnar <[email protected]> wrote:
> "real review" only becomes easy if the code _is reviewable to begin with_:
> if it is written in standard coding patterns where we almost
> sub-consciously recognize bad constructs and bad practices.
>
> I've seen it time and again that if the code is cleaned up visually, real
> review and real improvements follow eventually. It's a gradual process and
> you just cannot do "real review" efficiently without what you call the
> "checkpatch crap".
Agreed. When I looked at the pile of crap in drivers/staging/winbond
at first, it was more or less impossible to even say what the hell the
driver was doing. That was because (a) there were tons of obfuscating
wrappers carried over from Windows driver APIs and (b) there's so much
dead code it's not even funny!
So yeah, doing a "real review" on that monster before we get the
basics right would probably be a complete waste of time.
Pekka
* Greg KH <[email protected]> wrote:
> > I don't mean to piss on your parade, but from my viewpoint, staging is
> > a trainwreck so far, and I'd hate to see it get worse.
>
> What is wrong with it? Bugs are getting fixed, people are getting use
> out of their hardware (hell, Linus is even using one of the drivers),
> and lots of developers are cutting their teeth on helping out.
>
> If you don't like it, just disable it in your kernel packages, or
> instantly close out the bugs. The drivers in staging has already helped
> out some distros by virtue of including newer drivers than they were
> mistakenly using at the time (Ubuntu, I'm looking at you...)
>
> And again, it's helped out users, which is the most important thing
> here.
yes.
Firstly, a distro can disable CONFIG_STAGING just fine and then there will
be no 'crappy' drivers in that distro.
The thing is, the past decade has taught us that distros are willing to
apply just about any crap if it helps out a significant proportion of
users. Utrace crashes in Fedora dominated kerneloops.org stats for months.
Special ACPI patches and hacks, experimental wireless and DRI drivers in
Fedora, etc.
Why should the mainline kernel be any different? Treating it differently
would be a double standard. If a distro can apply crappy patches in sake
of utility, why shouldnt the upstream kernel have a staging area where
useful but not fully upstream-worthy drivers can hang around?
For years the upstream kernel was a lot less useful to testers in practice
because all the crappy but useful patches were in the distro kernels but
not in the mainline kernel.
Now that the upstream kernel has such an area, exactly what has changed -
besides making the kernel more useful, more testable, more hackable and
more viable? In fact i claim that crap gets cleaned up much faster if it's
out in the open for all to see - instead of hidden in distro SRPMs.
Ingo
Playing devil's advocate here...
On Thu, Jan 22, 2009 at 01:58:17PM -0800, Greg KH wrote:
> > * The fallout of staging is already starting to drift into distros.
> > Within a week of Fedora shipping a kernel that had staging/
> > we had requests to enable drivers from it.
> > And of course, those drivers were garbage.
> > This is only going to increase as time goes on.
>
> That's up to you as a distro to handle, not much I can do there.
>
> But, if you want a recommendation, some of the drivers in staging came
> from the Fedora kernel tree, so you should be enabling them :)
>
Just at76, I think.
> What is wrong with it? Bugs are getting fixed, people are getting use
> out of their hardware (hell, Linus is even using one of the drivers),
> and lots of developers are cutting their teeth on helping out.
>
Why does it need to be upstream for someone to cut their teeth helping
out?
> If you don't like it, just disable it in your kernel packages, or
> instantly close out the bugs. The drivers in staging has already helped
> out some distros by virtue of including newer drivers than they were
> mistakenly using at the time (Ubuntu, I'm looking at you...)
>
> And again, it's helped out users, which is the most important thing
> here.
>
What concerns me is the precedent this sets. If "getting something
upstream" now means "getting something into staging" then we've failed
our users since there's no longer any motivation for a vendor to invest
in all but the most cursory work on a Linux driver.
I think we have higher standards to live up to than that.
I agree that this is a much better plan than all the distros
individually collecting all the shit drivers themselves and (well, for
some of them) fixing the most egregious of crap and not getting fixes
back upstream because they've all frozen on different versions. But
still.
(I also think TAINT_CRAP is kind of an insulting name for things which
are really Linux-targetted features that just haven't had thorough
enough review. Evgeniy Polyakov's work comes to mind... it's really
comparing apples to a bunch of festering pieces of turd. While I'm sure
he's happy to have gotten his stuff in for more review, is it likely to
actually get more review than it would with weekly mailing list
postings? Maybe, who am I to say... I do think labelling his work crap
by virtue of the directory it resides in is fairly silly.)
</rant off>
Yeah, I realize it's fairly hypocritical for me to criticize given I
used to work on Ubuntu, but if it makes anyone feel any better, I didn't
like the idea of shipping things we wouldn't possibly support there
either.
While it is nice to justify things as "but users want it" they also
don't want a driver that reads files out of /etc and panics their kernel
or trashes their data. Which is why we won't enable this in Fedora. We
have enough maintenance burden with the utter horrowshow drivers we
claim as "maintained" in mainline already. If people really think having
something in mainline increases the odds something will get cleaned up,
I urge them to look at any scsi, network, or isdn driver from more than
five years ago. Make sure you bring a barf bag.
In summary, I don't know, this is one of those damned if you do, damned
if you don't paradoxes. ;-) But if you suck in a driver that barfs all
over your filesystem, because it was allowed to be turned in with zero
review, are you going to be the one to tell the user "ha ha, sucks to be
you?" I sure wouldn't want to be.
regards, Kyle
On Thu, Jan 22, 2009 at 05:50:26PM -0500, Kyle McMartin wrote:
> Playing devil's advocate here...
>
> On Thu, Jan 22, 2009 at 01:58:17PM -0800, Greg KH wrote:
> > > * The fallout of staging is already starting to drift into distros.
> > > Within a week of Fedora shipping a kernel that had staging/
> > > we had requests to enable drivers from it.
> > > And of course, those drivers were garbage.
> > > This is only going to increase as time goes on.
> >
> > That's up to you as a distro to handle, not much I can do there.
> >
> > But, if you want a recommendation, some of the drivers in staging came
> > from the Fedora kernel tree, so you should be enabling them :)
> >
>
> Just at76, I think.
Yes.
> > What is wrong with it? Bugs are getting fixed, people are getting use
> > out of their hardware (hell, Linus is even using one of the drivers),
> > and lots of developers are cutting their teeth on helping out.
> >
>
> Why does it need to be upstream for someone to cut their teeth helping
> out?
Because people don't know where to look if it is out-of-the tree.
Seriously, if it can't be easily found, it's not fixed up. Proof of
that is the hundreds of out-of-tree drivers that I have found over the
past months.
> > If you don't like it, just disable it in your kernel packages, or
> > instantly close out the bugs. The drivers in staging has already helped
> > out some distros by virtue of including newer drivers than they were
> > mistakenly using at the time (Ubuntu, I'm looking at you...)
> >
> > And again, it's helped out users, which is the most important thing
> > here.
> >
>
> What concerns me is the precedent this sets. If "getting something
> upstream" now means "getting something into staging" then we've failed
> our users since there's no longer any motivation for a vendor to invest
> in all but the most cursory work on a Linux driver.
Woah, you are changing the conversation here totally.
This has nothing to do with "put pressure on a vendor to get their code
into upstream so that a distro will enable it." We have vendors today
working with the staging tree to get their code cleaned up better to get
it into mergable shape to move over to the main portion of the kernel
tree.
Other vendors throw us code and run away. We handle that as well, by
doing the work on our own, taking our time. While that cleanup happens,
users can use their hardware with Linux without having to find the
latest version of a driver on a random google code site, and figure out
how to patch things to handle api changes that have happened since then
(true story for one of the drivers in staging right now.)
> I think we have higher standards to live up to than that.
The staging tree has NOTHING to do with our coding and acceptance
standards. And anyone that thinks otherwise is totally mistaken.
> (I also think TAINT_CRAP is kind of an insulting name for things which
> are really Linux-targetted features that just haven't had thorough
> enough review. Evgeniy Polyakov's work comes to mind... it's really
> comparing apples to a bunch of festering pieces of turd.
It was his choice to put the code into there, I'll let him handle the
mental issues of having that taint flag associated with it for a few
releases :)
> In summary, I don't know, this is one of those damned if you do, damned
> if you don't paradoxes. ;-) But if you suck in a driver that barfs all
> over your filesystem, because it was allowed to be turned in with zero
> review, are you going to be the one to tell the user "ha ha, sucks to be
> you?" I sure wouldn't want to be.
I'll take responsibility if such a thing happens. Fear of such a thing
is no reason to prevent users from having their hardware work properly
with Linux.
Again, I'll point to Linus's laptop that now works properly due to the
drivers/staging/ tree as a very visible example of this effort actually
working properly.
thanks,
greg k-h
On Thu, Jan 22, 2009 at 03:04:57PM -0800, Greg KH ([email protected]) wrote:
> > (I also think TAINT_CRAP is kind of an insulting name for things which
> > are really Linux-targetted features that just haven't had thorough
> > enough review. Evgeniy Polyakov's work comes to mind... it's really
> > comparing apples to a bunch of festering pieces of turd.
>
> It was his choice to put the code into there, I'll let him handle the
> mental issues of having that taint flag associated with it for a few
> releases :)
If you could know haw bad is my sleep since then...
--
Evgeniy Polyakov
Hi Kyle.
On Thu, Jan 22, 2009 at 05:50:26PM -0500, Kyle McMartin ([email protected]) wrote:
> (I also think TAINT_CRAP is kind of an insulting name for things which
> are really Linux-targetted features that just haven't had thorough
> enough review. Evgeniy Polyakov's work comes to mind... it's really
> comparing apples to a bunch of festering pieces of turd. While I'm sure
> he's happy to have gotten his stuff in for more review, is it likely to
> actually get more review than it would with weekly mailing list
> postings? Maybe, who am I to say... I do think labelling his work crap
> by virtue of the directory it resides in is fairly silly.)
Reviews really do not happen based on the fact that code is in the tree
or not. Most of the time it is submission time and maintainer's look at
bugs. People like to talk about reviews, but when it does not happen at
submission time, let's just move further :)
--
Evgeniy Polyakov
On Thu, Jan 22, 2009 at 03:04:57PM -0800, Greg KH wrote:
> > Why does it need to be upstream for someone to cut their teeth helping
> > out?
> Because people don't know where to look if it is out-of-the tree.
> Seriously, if it can't be easily found, it's not fixed up. Proof of
> that is the hundreds of out-of-tree drivers that I have found over the
> past months.
What about the hundreds of utterly crap drivers we have *right now* in
the kernel? Just because something is distributed with the kernel does
*not* mean it will get cleaned up. There's hundreds of counterexamples
to that. If you think moving some of them to drivers/staging to increase
the "visibility" to people looking for low hanging fruit, I can generate
a list...
> > What concerns me is the precedent this sets. If "getting something
> > upstream" now means "getting something into staging" then we've failed
> > our users since there's no longer any motivation for a vendor to invest
> > in all but the most cursory work on a Linux driver.
>
> Woah, you are changing the conversation here totally.
>
What if vendor X has decided that they can now save some money by simply
dumping the code at the "hey, I'll take anything" tree instead of
continuing to work with the community, or following the example of people
who do. They can still tell their users "we're upstream, just enable
this CONFIG option and our stuff will work."
> This has nothing to do with "put pressure on a vendor to get their code
> into upstream so that a distro will enable it." We have vendors today
> working with the staging tree to get their code cleaned up better to get
> it into mergable shape to move over to the main portion of the kernel
> tree.
>
> Other vendors throw us code and run away. We handle that as well, by
> doing the work on our own, taking our time. While that cleanup happens,
> users can use their hardware with Linux without having to find the
> latest version of a driver on a random google code site, and figure out
> how to patch things to handle api changes that have happened since then
> (true story for one of the drivers in staging right now.)
>
My concern is what was stated above, that vendors who have historically
done the right thing may have less motivation to continue in the future.
I defer to your judgement though.
> > I think we have higher standards to live up to than that.
>
> The staging tree has NOTHING to do with our coding and acceptance
> standards. And anyone that thinks otherwise is totally mistaken.
>
I don't know how you can say this, given. Nothing in there would
make the cut at all... obviously it has a lower barrier to entry. Is
there a maintenance burden imposed on someone for staging? I mean, to
get something in there is someone agreeing to take point on all the
issues?
If its purpose isn't "the staging point for things to get (in theory)
cleaned up because people want convenience now" then what is it?
> > In summary, I don't know, this is one of those damned if you do, damned
> > if you don't paradoxes. ;-) But if you suck in a driver that barfs all
> > over your filesystem, because it was allowed to be turned in with zero
> > review, are you going to be the one to tell the user "ha ha, sucks to be
> > you?" I sure wouldn't want to be.
>
> I'll take responsibility if such a thing happens. Fear of such a thing
> is no reason to prevent users from having their hardware work properly
> with Linux.
>
> Again, I'll point to Linus's laptop that now works properly due to the
> drivers/staging/ tree as a very visible example of this effort actually
> working properly.
>
I assume this is the eee wireless?
As I said, I'm not (deliberately) trying to be negative. I'm simply
trying to understand the rationale. I'm sorry if I sound that way, but
working on distros has made me horribly pragmatic about these things.
I guess time will tell.
cheers, Kyle
> thanks,
>
> greg k-h
>
* Kyle McMartin <[email protected]> wrote:
> On Thu, Jan 22, 2009 at 03:04:57PM -0800, Greg KH wrote:
> > > Why does it need to be upstream for someone to cut their teeth helping
> > > out?
> > Because people don't know where to look if it is out-of-the tree.
> > Seriously, if it can't be easily found, it's not fixed up. Proof of
> > that is the hundreds of out-of-tree drivers that I have found over the
> > past months.
>
> What about the hundreds of utterly crap drivers we have *right now* in
> the kernel? Just because something is distributed with the kernel does
> *not* mean it will get cleaned up. There's hundreds of counterexamples
> to that. If you think moving some of them to drivers/staging to increase
> the "visibility" to people looking for low hanging fruit, I can generate
> a list...
that is true and it does not contradict the purpose and intention of
drivers/staging/ though - it extends it.
We could create drivers/staging/in/ and drivers/staging/out/.
So instead of marking drivers CONFIG_BROKEN we could move them to
drivers/staging/out/ - and if they dont get 'saved' within a kernel
release they will be zapped for good. That is more gradual than a sudden
'remove a driver completely' action.
Ingo
> On Thu, 22 Jan 2009 16:50:41 -0500 Dave Jones <[email protected]> wrote:
> I don't mean to piss on your parade, but from my viewpoint, staging
> is a trainwreck so far, and I'd hate to see it get worse.
I must say that I have intense "wtf were we thinking of" moments each time
something in staging/ sticks an ugly head out of its cave.
My question is: what is the process for moving things out of staging/ and
into the kernel proper? We don't want to be merging ex-crap code which has
had its whitespace and wrappers cleaned up by a person who is not suitably
experienced in the relevant subsystem.
It _should_ (I think) be "OK, this is ready for us to start reviewing".
ie: treat it as if it had just come in from a random new submitter.
a) If that happens, who will be responding to the review comments? Who
was the submitter?
b) What happens if it is then decided that we just don't want that code
in the tree at all?
c) Has any code yet made the transition out of staging and into the
kernel proper? If so, which?
On Thu, Jan 22, 2009 at 04:16:59PM -0800, Andrew Morton wrote:
> My question is: what is the process for moving things out of staging/ and
> into the kernel proper? We don't want to be merging ex-crap code which has
> had its whitespace and wrappers cleaned up by a person who is not suitably
> experienced in the relevant subsystem.
>
> It _should_ (I think) be "OK, this is ready for us to start reviewing".
> ie: treat it as if it had just come in from a random new submitter.
I agree, and I will do just that.
> a) If that happens, who will be responding to the review comments? Who
> was the submitter?
For drivers that don't have an active developer, I'll handle the review
commends and fix things up. Some of these drivers do have active
developers, and I'm sure they will be the ones handling the review as
well.
> b) What happens if it is then decided that we just don't want that code
> in the tree at all?
As it's "just a driver", the odds of that happening are probably going
to be very slim. We can cross that bridge when we come to it.
> c) Has any code yet made the transition out of staging and into the
> kernel proper? If so, which?
Not yet, but some of the network drivers are looking much better and I
think they will move out soon.
One of the v4l drivers should move out for .30, I need to get some time
to send the patch to those developers.
thanks,
greg k-h