Alan Cox wrote:
> Greg Felix wrote:
> > Right. I get the output at bootup time. It reads that the HPA is
> > 20MB. Which is exactly the size of how far off the metadata is in
> > Linux (once the HPA is disabled).
>
> So your actual problem is nothing to do with the kernel or with the HPA
> behaviour ? Whatever tool you are using for raid set up isn't reading
> and processing the right fields.
If the formula is to fix all the userspace apps to take into account a
potential HPA, then eg. FDISK + SFDISK + Disk Druid et al should also
be fixed. Because if you create a partition spanning your entire
disk, including the HPA area, and your boot files by some coincidence
ends up in the HPA part of the disk, the BIOS won't be able to read
them, and your system will not boot. And if you fix those tools now,
what about users that use older Linux distributions? They'll have a
parade of problems coming to them with their new HPA-enabled disks
because every userspace tool assumes that <BIOS sector count == Linux
sector count>.
> It isnt the kernels fault if you compute from of end of disk rather than
> from end of non reserved area is it ?
I agree that the entire disk should be visible under Linux. But
instead of fixing every userspace app that assumes <BIOS sector count
== Linux sector count> (I'm guessing that's a lot), how about simply
exporting the HPA as /dev/ide/hostX/busY/targetZ/lunA/hpa, next to the
'disc' device ?
> If the formula is to fix all the userspace apps to take into account a
> potential HPA, then eg. FDISK + SFDISK + Disk Druid et al should also
> be fixed. Because if you create a partition spanning your entire
> disk, including the HPA area, and your boot files by some coincidence
> ends up in the HPA part of the disk, the BIOS won't be able to read
> them, and your system will not boot. And if you fix those tools now,
The distribution vendors already put the boot area low down the disk to
handle a whole variety of problems like this and the 1024 cylinder limit
on old BIOSes.
> what about users that use older Linux distributions? They'll have a
> parade of problems coming to them with their new HPA-enabled disks
> because every userspace tool assumes that <BIOS sector count == Linux
> sector count>.
I work for a vendor. I get most of our IDE layer bugs. And do you know
HPA doesn't really feature. The broken locking does, the hangs on
changing down speed by CRC do, numerous fascinating platform specific
bugs do, people with cables backwards do. HPA doesn't. It just works out
for people.
> > It isnt the kernels fault if you compute from of end of disk rather than
> > from end of non reserved area is it ?
>
> I agree that the entire disk should be visible under Linux. But
> instead of fixing every userspace app that assumes <BIOS sector count
> == Linux sector count> (I'm guessing that's a lot), how about simply
> exporting the HPA as /dev/ide/hostX/busY/targetZ/lunA/hpa, next to the
> 'disc' device ?
A lot of applications assume Linux reported size == real disk size. It
also wouldn't solve the case of a file system that spans both inside and
outside the HPA.
Alan
Alan Cox wrote:
> > If the formula is to fix all the userspace apps to take into account a
> > potential HPA, then eg. FDISK + SFDISK + Disk Druid et al should also
> > be fixed. Because if you create a partition spanning your entire
> > disk, including the HPA area, and your boot files by some coincidence
> > ends up in the HPA part of the disk, the BIOS won't be able to read
> > them, and your system will not boot. And if you fix those tools now,
>
> The distribution vendors already put the boot area low down the disk to
> handle a whole variety of problems like this and the 1024 cylinder limit
> on old BIOSes.
Only as an option - it's not always possible.
For example one might have a Windows partition already in place, etc..
> > what about users that use older Linux distributions? They'll have a
> > parade of problems coming to them with their new HPA-enabled disks
> > because every userspace tool assumes that <BIOS sector count == Linux
> > sector count>.
>
> I work for a vendor. I get most of our IDE layer bugs. And do you know
> HPA doesn't really feature.
Fair enough. It took 5 seconds of googling to find a guy whose GRUB
stopped working because of the HPA, so at least *some* people are
already having problems...
> > > It isnt the kernels fault if you compute from of end of disk rather than
> > > from end of non reserved area is it ?
> >
> > I agree that the entire disk should be visible under Linux. But
> > instead of fixing every userspace app that assumes <BIOS sector count
> > == Linux sector count> (I'm guessing that's a lot), how about simply
> > exporting the HPA as /dev/ide/hostX/busY/targetZ/lunA/hpa, next to the
> > 'disc' device ?
>
> A lot of applications assume Linux reported size == real disk size.
I can't think of any, but you might be right.
> It also wouldn't solve the case of a file system that spans both inside and
> outside the HPA.
If HPA were exposed as /dev/.../hpa then it wouldn't be possible to
create such a filesystem. I'm guessing it's not possible with Windows
either, or with any BIOS-based OS.
Creating filesystems that include the HPA defeats the entire idea of
the HPA in the first place.
If one does not care to use the HPA, one should disable it in the BIOS
entirely, so that everywhere (!) the entire disk is seen.
The inclusion of the HPA in /dev/.../disc should be a special case for
those unfortunate souls with laptops where the HPA cannot be disabled
because their BIOS is crap, IMHO.
On Gwe, 2005-09-02 at 15:33 +0200, Molle Bestefich wrote:
> > It also wouldn't solve the case of a file system that spans both inside and
> > outside the HPA.
>
> If HPA were exposed as /dev/.../hpa then it wouldn't be possible to
> create such a filesystem. I'm guessing it's not possible with Windows
> either, or with any BIOS-based OS.
It is on many windows systems which include extra drivers that come with
large disks.
> Creating filesystems that include the HPA defeats the entire idea of
> the HPA in the first place.
See previous discussion, this is untrue. It may not have been the intent
of the standards authors but its certainly the idea of a large number of
disk and system vendors that the HPA is a handy way to deal with old
BIOSes and other items of that nature.
> If one does not care to use the HPA, one should disable it in the BIOS
> entirely, so that everywhere (!) the entire disk is seen.
And in the real world BIOSes don't get updated often by vendors let
alone by users.
Molle Bestefich <[email protected]> wrote:
> If HPA were exposed as /dev/.../hpa then it wouldn't be possible to
> create such a filesystem. I'm guessing it's not possible with Windows
> either, or with any BIOS-based OS.
Such filesystems already exist. Changing this behaviour now would break
existing setups.
--
Matthew Garrett | [email protected]
Alan Cox wrote:
> > If one does not care to use the HPA, one should disable it in the
> > BIOS entirely, so that everywhere (!) the entire disk is seen.
>
> And in the real world BIOSes don't get updated often
> by vendors let alone by users.
I meant the BIOS setup screen, not a firmware update...
Supposedly the BIOS can change the bounds of the HPA with special ATA commands..
Matthew Garrett wrote:
> Molle Bestefich wrote:
> > If HPA were exposed as /dev/.../hpa then it wouldn't be possible to
> > create such a filesystem. I'm guessing it's not possible with Windows
> > either, or with any BIOS-based OS.
>
> Such filesystems already exist. Changing this behaviour now would break
> existing setups.
Not if, as proposed, there was a kernel switch to enable including the
HPA in the disc area.
Also those who has such filesystems {c,sh}ould disable HPA in their
BIOS (hopefully).
On Gwe, 2005-09-02 at 18:24 +0200, Molle Bestefich wrote:
> I meant the BIOS setup screen, not a firmware update...
> Supposedly the BIOS can change the bounds of the HPA with special ATA commands..
I've yet to see a BIOS that exposed the functionality
> Not if, as proposed, there was a kernel switch to enable including the
> HPA in the disc area.
And users magically knew about it - thats why it has to default the
other way.
Alan Cox wrote:
> Molle Bestefich wrote:
> > Not if, as proposed, there was a kernel switch to enable including the
> > HPA in the disc area.
>
> And users magically knew about it - thats why it has to default the
> other way.
Ok, so just to reiterate..
The current default is causing grief because dmraid doesn't work, grub
doesn't work and other userspace apps probably breaks too. Users have
to google and post to mailing lists just to get things to work (... if
they could, which would require eg. a kernel option, but anyway).
The other way round, users would have to google to find the kernel
option that claims the HPA area (if they felt the need to overwrite
the BIOS's backup area), but those that felt the need would then be
rewarded with eg. 10 GB extra disk space. And if they didn't feel
like it, their userspace apps would still work just fine.
>From my POV it's hard to see why the current default is sensible.
(We'll probably just have to agree on disagreeing, unless you can enlighten me.)
Related matters:
If you decide to include the HPA in one of your filesystems, is there
not a big risk that the BIOS will overwrite something there?
On Fri, Sep 02, 2005 at 06:05:12PM +0100, Alan Cox wrote:
> On Gwe, 2005-09-02 at 18:24 +0200, Molle Bestefich wrote:
> > I meant the BIOS setup screen, not a firmware update...
> > Supposedly the BIOS can change the bounds of the HPA with special ATA commands..
>
> I've yet to see a BIOS that exposed the functionality
ThinkPads do let the user enable/disable it.
--
Vojtech Pavlik
SuSE Labs, SuSE CR
Molle Bestefich <[email protected]> wrote:
> The other way round, users would have to google to find the kernel
> option that claims the HPA area (if they felt the need to overwrite
> the BIOS's backup area), but those that felt the need would then be
> rewarded with eg. 10 GB extra disk space. And if they didn't feel
> like it, their userspace apps would still work just fine.
Unless they already have a filesystem that stretches over the HPA, in
which case the change in default behaviour would result in kernel
upgrades breaking everything horribly. It may well be the case that the
current situation is undesirable, but just changing it back *will break*
things.
--
Matthew Garrett | [email protected]
On Fri, 2005-09-02 at 19:44 +0200, Molle Bestefich wrote:
> Related matters:
> If you decide to include the HPA in one of your filesystems, is there
> not a big risk that the BIOS will overwrite something there?
Isn't the bigger risk that if you include the HPA in your block device,
you'll overwrite your BIOS there?
At least that's what happens on some thinkpads.
It'd be nice if we left it as the BIOS leaves it by default, but make a
straightforward way to enable/disable/alter the HPA region from
software.
(if there's already a straightforward way, feel free to clue me in --
but the default should almost certainly be to assume the HPA is set up
correctly, shouldn't it?)
--
Peter
On Gwe, 2005-09-02 at 19:44 +0200, Molle Bestefich wrote:
> The current default is causing grief because dmraid doesn't work, grub
> doesn't work and other userspace apps probably breaks too. Users have
> to google and post to mailing lists just to get things to work (... if
> they could, which would require eg. a kernel option, but anyway).
You've show what - one obscure case with grub of an issue installers
already know about. dmraid is experimental anyway so submit fixes.
> The other way round, users would have to google to find the kernel
> option that claims the HPA area
And to find why their fs just choked, their computer no longer runs and
their files are lost. At which point they may not even be able to use
google.
On Gwe, 2005-09-02 at 14:09 -0400, Peter Jones wrote:
> (if there's already a straightforward way, feel free to clue me in --
> but the default should almost certainly be to assume the HPA is set up
> correctly, shouldn't it?)
The normal use of HPA is to clip drives to get them past BIOS boot
checks. The thinkpads come with a pre-installed partition table which
will protect the HPA unless the user goes to town removing it.
The ideal case would be that the partition table is considered at boot
to see if the HPA matches the partitiont table or not. You'd also then
need dynamic HPA enable/disable for installers and other tools to go
with that.
Send patches.
On Fri, 2005-09-02 at 19:59 +0100, Alan Cox wrote:
> On Gwe, 2005-09-02 at 14:09 -0400, Peter Jones wrote:
> > (if there's already a straightforward way, feel free to clue me in --
> > but the default should almost certainly be to assume the HPA is set up
> > correctly, shouldn't it?)
>
> The normal use of HPA is to clip drives to get them past BIOS boot
> checks.
Ugh. So some BIOSes use it for legitimate reasons (like thinkpads), and
some use it to work around BIOS bugs. Great.
> The thinkpads come with a pre-installed partition table which
> will protect the HPA unless the user goes to town removing it.
Mine didn't, but it does have an HPA. Thankfully we weren't disabling
it yet when I installed my laptop -- I know others who weren't so lucky.
So this partitioning scheme hasn't always been the case...
And it seems broken anyway. The point of the HPA is to make the OS see
a smaller/different disk layout, unless it's actually trying to update
things that are "protected", right? If so, the partition table pointing
outside of the the disk when the HPA configuration hasn't been changed
from the bootup default totally broken :/
It really sounds better (to my naive mind, at least) to whitelist the
known-broken BIOSes.
> The ideal case would be that the partition table is considered at boot
> to see if the HPA matches the partitiont table or not. You'd also then
> need dynamic HPA enable/disable for installers and other tools to go
> with that.
Well, installers probably should be aware, yes -- that's why I mentioned
userland interfaces to enabling/disabling. But to me it still seems
like we want to disable the HPA during installation and bootup, but only
if your BIOS is doing things wrong.
> Send patches.
Point taken.
--
Peter
On Gwe, 2005-09-02 at 15:14 -0400, Peter Jones wrote:
> Ugh. So some BIOSes use it for legitimate reasons (like thinkpads), and
> some use it to work around BIOS bugs. Great.
All are legitimate uses. The partition table tells you which.
> Mine didn't, but it does have an HPA. Thankfully we weren't disabling
> it yet when I installed my laptop -- I know others who weren't so lucky.
> So this partitioning scheme hasn't always been the case...
You installed it on Red Hat 7 ? I think 7, may have been 6.x or earlier.
This behaviour goes back pretty much to the creation of the ATA spec for
HPA. In fact if it was that long ago IBM shipped it with Windows so it
did have a partition table!
> It really sounds better (to my naive mind, at least) to whitelist the
> known-broken BIOSes.
Not really practical. You'd have to list most older PC systems.
> Well, installers probably should be aware, yes -- that's why I mentioned
> userland interfaces to enabling/disabling. But to me it still seems
> like we want to disable the HPA during installation and bootup, but only
> if your BIOS is doing things wrong.
"Wrong" is a poor term here.
If the system has a partition table that doesn't cover the post HPA area
and its about the right size we can be fairly sure the right choice is
to honour the HPA, if its a randomly different size its a fair bet the
disk got moved
If the partition table exceeds the no HPA area of disk but not the full
disk then its almost certainly right the HPA should be disabled post
boot. If it exceeds both its a raid 0 volume of some form...
On Fri, 2005-09-02 at 21:22 +0100, Alan Cox wrote:
> On Gwe, 2005-09-02 at 15:14 -0400, Peter Jones wrote:
> > Ugh. So some BIOSes use it for legitimate reasons (like thinkpads), and
> > some use it to work around BIOS bugs. Great.
>
> All are legitimate uses. The partition table tells you which.
>
> > Mine didn't, but it does have an HPA. Thankfully we weren't disabling
> > it yet when I installed my laptop -- I know others who weren't so lucky.
> > So this partitioning scheme hasn't always been the case...
>
> You installed it on Red Hat 7 ? I think 7, may have been 6.x or earlier.
You may be right -- it's likely that I shrank my windows partition on
some other OS or Distro that wasn't designed with to screw up the disk.
> This behaviour goes back pretty much to the creation of the ATA spec for
> HPA. In fact if it was that long ago IBM shipped it with Windows so it
> did have a partition table!
Yes, it did have a partition table -- but the partition table did (and
still does) not include partitions which overlap the HPA. Right now it
still appears as unused space.
> > It really sounds better (to my naive mind, at least) to whitelist the
> > known-broken BIOSes.
>
> Not really practical. You'd have to list most older PC systems.
Most older PC systems use HPA? Really?
The alternatives seem to be:
a) whitelist systems (in the kernel or userland) which use it the
opposite way, where the space is being used for something
relatively important (at least conceptually), or
b) bad heuristics to figure out which is which.
Both of these suck. Have I missed something?
> > Well, installers probably should be aware, yes -- that's why I mentioned
> > userland interfaces to enabling/disabling. But to me it still seems
> > like we want to disable the HPA during installation and bootup, but only
> > if your BIOS is doing things wrong.
>
> "Wrong" is a poor term here.
>
> If the system has a partition table that doesn't cover the post HPA area
> and its about the right size we can be fairly sure the right choice is
> to honour the HPA, if its a randomly different size its a fair bet the
> disk got moved
>
> If the partition table exceeds the no HPA area of disk but not the full
> disk then its almost certainly right the HPA should be disabled post
> boot. If it exceeds both its a raid 0 volume of some form...
So where would you envision this code to check the partition table, the
HPA/host default disk size, and guess how things should be set up?
>From a userland perspective, it's very difficult to let users know
they'll be screwing themselves by partitioning the entire disk, so we
really should be leaving HPA enabled if the protected area is indeed not
for consumption.
Also, the heuristic is harder than this -- if we reexamine the fakeraid
case, then it's clear we have to look for raid metadata, figure out if
the raid includes stuff inside the HPA or not, and then if it doesn't we
don't care -- but that's assuming there _is_ raid metadata.
Long term, many people hope, possibly unrealistically, that we'll be
able to write out raid metadata for people creating raids on cards which
support fakeraid, and have the BIOS grok it appropriately. So in that
case, we may well have a blank (or garbage) disk, and we can't check the
partition table or any raid metadata. Correct me if I'm wrong, but I
don't see a simple heuristic for this case.
(as a side note, I know one user who, at OLS, noticed we fail to
re-initialize HPA after unsuspend, so on at least t40 the disk gets
smaller when you suspend. This may or may not be fixed, I haven't
checked. But it's one more sort of pain we get into by disabling it,
whether justified or not.)
I think if we go the heuristic route, then the *safest* option is to
leave it enabled by default and let userland installers/initrd do fixups
by telling the kernel to change the state. That way it can keep it
correct on suspend/unsuspend, but we don't give users the opportunity to
really screw themselves unless they try pretty hard, and we don't
actually screw the users unless we simply get the heuristic totally
wrong. It's also consistent with the more general policy of leaving
policy up to the userland.
But your point about how I should send you a patch certainly still
applies.
--
Peter
On Gwe, 2005-09-02 at 17:14 -0400, Peter Jones wrote:
> > You installed it on Red Hat 7 ? I think 7, may have been 6.x or earlier.
>
> You may be right -- it's likely that I shrank my windows partition on
> some other OS or Distro that wasn't designed with to screw up the disk.
If you shrink existing partitions it won't ever screw you up. The
geometry data for the partition table spans only the non HPA area.
> Yes, it did have a partition table -- but the partition table did (and
> still does) not include partitions which overlap the HPA. Right now it
> still appears as unused space.
But they are also on the IBM I looked at are obvious because the
geometry in the partition table does not span the HPA so the problem
doesn't arise as confusion.
> > Not really practical. You'd have to list most older PC systems.
>
> Most older PC systems use HPA? Really?
Many of those "magic windows drive/bios fixup" type programs work by
having the jumper on the drive set the HPA and the drive report a
smaller size, then the windows magic driver undoes this.
> Both of these suck. Have I missed something?
I fear not.
> So where would you envision this code to check the partition table, the
> HPA/host default disk size, and guess how things should be set up?
fdisk and friends already have to parse and process the existing
partition tables.
> they'll be screwing themselves by partitioning the entire disk, so we
> really should be leaving HPA enabled if the protected area is indeed not
> for consumption.
Define "not for consumption". It should be *hard* to use it, and it
should not occur by accident. Deliberately is a different matter. And
that should be a run time not boot time action.
> (as a side note, I know one user who, at OLS, noticed we fail to
> re-initialize HPA after unsuspend, so on at least t40 the disk gets
> smaller when you suspend. This may or may not be fixed, I haven't
> checked. But it's one more sort of pain we get into by disabling it,
> whether justified or not.)
Known problem. ACPI provides the correct infrastructure for much of this
but the IDE layer doesn't support it. Send patches to Bartlomiej. The
core infrastructure is there because Andre saw the need for the ACPI
taskfile support coming. The HPA restore is just another step in the
state machine for resume and quite doable. Good little project for
someone wanting to play with the IDE layer.
> I think if we go the heuristic route, then the *safest* option is to
> leave it enabled by default and let userland installers/initrd do fixups
> by telling the kernel to change the state.
Assuming we are talking about hda1/2/... then the partitions are already
clipped by the OS to the volume size. We could conceivably make the size
of the disk itself writable. We don't need to get into programming drive
HPA when we can do it ourselves, and we can clip non HPA capable drives
too should someone find a cause for it.
On Fri, Sep 02, 2005 at 09:22:58PM +0100, Alan Cox wrote:
> You installed it on Red Hat 7 ? I think 7, may have been 6.x or earlier.
> This behaviour goes back pretty much to the creation of the ATA spec for
> HPA. In fact if it was that long ago IBM shipped it with Windows so it
> did have a partition table!
FC3 happily ignored the HPA on IBM X31. FC4 did not. I won't vow about the
original partitioning, but a "worked for just about everything" FC3
kickstart slightly updated to FC4 started breaking horribly after suspend.
As in messed up filesystems since parts of the disk just vanished when you
resumed. (FC3 could have been broken too, but CONFIG_IDE_STROKE or whatnot
wasn't enabled, so it worked as expected). It probably didn't work
as expected for people with broken bioses that didn't do >32GB ether, but those
people required additional hacks for competing OS's too, so it wasn't such a
biggie for them, I suppose.
Some sort of BIOS bug, completely IBM's (or rather some subcontractor)
fault, happens on one X31 laptop I know of, where the HPA just can't be
disabled. At all. The BIOS setting gets ignored. On the one I personally use
disabling it works, so losing the recovery Windows XP was enough to have a
functional system. Not optimal, but I don't really need the recovery stuff
for anything, so might as well use the entire disk.
For the one where disabling the HPA just didn't work the solution was to
manually partition, and just not using the area the HPA would appear on.
There goes automatic kickstart installs, but at least the laptop now is usable.
Peter Jones wrote:
> So where would you envision this code to check the partition table, the
> HPA/host default disk size, and guess how things should be set up?
>
>>From a userland perspective, it's very difficult to let users know
> they'll be screwing themselves by partitioning the entire disk, so we
> really should be leaving HPA enabled if the protected area is indeed not
> for consumption.
>
> Also, the heuristic is harder than this -- if we reexamine the fakeraid
> case, then it's clear we have to look for raid metadata, figure out if
> the raid includes stuff inside the HPA or not, and then if it doesn't we
> don't care -- but that's assuming there _is_ raid metadata.
>
> Long term, many people hope, possibly unrealistically, that we'll be
> able to write out raid metadata for people creating raids on cards which
> support fakeraid, and have the BIOS grok it appropriately. So in that
> case, we may well have a blank (or garbage) disk, and we can't check the
> partition table or any raid metadata. Correct me if I'm wrong, but I
> don't see a simple heuristic for this case.
>
> (as a side note, I know one user who, at OLS, noticed we fail to
> re-initialize HPA after unsuspend, so on at least t40 the disk gets
> smaller when you suspend. This may or may not be fixed, I haven't
> checked. But it's one more sort of pain we get into by disabling it,
> whether justified or not.)
It seems to me that one should write an ATA-specific Device Mapper
driver, which layers on top of an ATA disk. The driver obtains the
starting location of HPA, then exports two block devices: one for the
primary data area, and one for the HPA.
For situations where we want the start Linux philosophy -- Linux exports
100% of the hardware capability -- no DM layer needs to be used. For
situations where its better to treat the HPA as a separate and distinct
area, the DM driver would come in handy.
This follows the same philosophy as fakeraid (BIOS RAID): we simply
export the entire disk, and Device Mapper (google for 'dmraid') handles
the vendor-proprietary RAID metadata.
Jeff
Jeff Garzik wrote:
> It seems to me that one should write an ATA-specific Device Mapper
> driver, which layers on top of an ATA disk. The driver obtains the
> starting location of HPA, then exports two block devices: one for the
> primary data area, and one for the HPA.
I've stayed out of this, but that sounds like a perfect solution to move
the choice back to the user. However, installers still need to be aware
of it at initial Linux install, and give the user some rational options:
- ignore it
- leave alone but visible
- blow it away and use the whole drive
It feels as if that's where the future disposition needs to be made. I
do like treating the HPA as a separate drive though.
>
> For situations where we want the start Linux philosophy -- Linux exports
> 100% of the hardware capability -- no DM layer needs to be used. For
> situations where its better to treat the HPA as a separate and distinct
> area, the DM driver would come in handy.
>
> This follows the same philosophy as fakeraid (BIOS RAID): we simply
> export the entire disk, and Device Mapper (google for 'dmraid') handles
> the vendor-proprietary RAID metadata.
--
-bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
last possible moment - but no longer" -me