2006-03-23 15:06:18

by Bill Waddington

[permalink] [raw]
Subject: [RFCLUE2] 64 bit driver 32 bit app ioctl

Apologies for dashing this off without the proper homework. My
customer is out of country doing an installation, and didn't test
this configuration first :(

Customer is running RHEL3 on a 64 bit PC. Running the 64 bit kernel
and my 64 bit driver. They are calling the driver from their 32 bit
app. The driver supports a whole mess of ioctls.

It seems that the kernel is trapping the 32-bit ioctl call and returning
an error to the app w/out calling the driver. It looks like
register_ioctl32_conversion() can convice the kernel that the driver can
handle 32-bit calls, but it has to be called for each ioctl cmd (??)

Putting aside (please) discussion of whether the kernel should presume
to hijack private ioctls, and whether I should be using the ioctl
interface at all (compatibility with app interface going back to 2.0
and SunOS) is there some way to make _one_ register call to indicate
that all my cmds are safe, or maybe an alternate ioctl entry point
that the kernel won't trap?

Yours in desperation,
Bill

--
--------------------------------------------
William D Waddington
Bainbridge Island, WA, USA
[email protected]
--------------------------------------------
"Even bugs...are unexpected signposts on
the long road of creativity..." - Ken Burtch


2006-03-23 16:43:05

by Mikael Pettersson

[permalink] [raw]
Subject: Re: [RFCLUE2] 64 bit driver 32 bit app ioctl

William D Waddington writes:
> Apologies for dashing this off without the proper homework. My
> customer is out of country doing an installation, and didn't test
> this configuration first :(
>
> Customer is running RHEL3 on a 64 bit PC. Running the 64 bit kernel
> and my 64 bit driver. They are calling the driver from their 32 bit
> app. The driver supports a whole mess of ioctls.
>
> It seems that the kernel is trapping the 32-bit ioctl call and returning
> an error to the app w/out calling the driver. It looks like
> register_ioctl32_conversion() can convice the kernel that the driver can
> handle 32-bit calls, but it has to be called for each ioctl cmd (??)

In these old pre-compat_ioctl kernels you have to register each
ioctl command individually. Yes that sucks. Live with it.

> Putting aside (please) discussion of whether the kernel should presume
> to hijack private ioctls, and whether I should be using the ioctl
> interface at all (compatibility with app interface going back to 2.0
> and SunOS) is there some way to make _one_ register call to indicate
> that all my cmds are safe, or maybe an alternate ioctl entry point
> that the kernel won't trap?

Not as long as you're stuck with old 2.4 kernels. 2.6 kernels since
2.6.11-rc2 allow you to set up a single ->compat_ioctl() method,
but not even RHEL4 has that yet.

2006-03-23 16:50:01

by Arjan van de Ven

[permalink] [raw]
Subject: Re: [RFCLUE2] 64 bit driver 32 bit app ioctl

On Thu, 2006-03-23 at 07:06 -0800, William D Waddington wrote:
> Apologies for dashing this off without the proper homework. My
> customer is out of country doing an installation, and didn't test
> this configuration first :(
>
> Customer is running RHEL3 on a 64 bit PC. Running the 64 bit kernel
> and my 64 bit driver. They are calling the driver from their 32 bit
> app. The driver supports a whole mess of ioctls.
>
> It seems that the kernel is trapping the 32-bit ioctl call and returning
> an error to the app w/out calling the driver. It looks like
> register_ioctl32_conversion() can convice the kernel that the driver can
> handle 32-bit calls, but it has to be called for each ioctl cmd (??)

you forgot to attach you code btw or post the url to it..


2006-03-23 16:58:03

by Avi Kivity

[permalink] [raw]
Subject: Re: [RFCLUE2] 64 bit driver 32 bit app ioctl

Arjan van de Ven wrote:
> On Thu, 2006-03-23 at 07:06 -0800, William D Waddington wrote:
>
> [...]
> you forgot to attach you code btw or post the url to it..
>
>
is that your new .signature?

:)

--
error compiling committee.c: too many arguments to function

2006-03-23 18:33:42

by Bill Waddington

[permalink] [raw]
Subject: Re: [RFCLUE2] 64 bit driver 32 bit app ioctl

Arjan van de Ven wrote:
> On Thu, 2006-03-23 at 07:06 -0800, William D Waddington wrote:
>
>>Apologies for dashing this off without the proper homework. My
>>customer is out of country doing an installation, and didn't test
>>this configuration first :(
>>
>>Customer is running RHEL3 on a 64 bit PC. Running the 64 bit kernel
>>and my 64 bit driver. They are calling the driver from their 32 bit
>>app. The driver supports a whole mess of ioctls.
>>
>>It seems that the kernel is trapping the 32-bit ioctl call and returning
>>an error to the app w/out calling the driver. It looks like
>>register_ioctl32_conversion() can convice the kernel that the driver can
>>handle 32-bit calls, but it has to be called for each ioctl cmd (??)
>
>
> you forgot to attach you code btw or post the url to it..

No I didn't :) It's just too ugly for public view. And I notice it
needs some other fix-ups like fixed width data types in the ioctl
routine...

Thanks to all for the info. I've got some typing to do.

Bill

2006-03-24 19:00:48

by Bill Waddington

[permalink] [raw]
Subject: Re: [RFCLUE2] 64 bit driver 32 bit app ioctl



Mikael Pettersson wrote:
> William D Waddington writes:
> > Apologies for dashing this off without the proper homework. My
> > customer is out of country doing an installation, and didn't test
> > this configuration first :(
> >
> > Customer is running RHEL3 on a 64 bit PC. Running the 64 bit kernel
> > and my 64 bit driver. They are calling the driver from their 32 bit
> > app. The driver supports a whole mess of ioctls.
> >
> > It seems that the kernel is trapping the 32-bit ioctl call and returning
> > an error to the app w/out calling the driver. It looks like
> > register_ioctl32_conversion() can convice the kernel that the driver can
> > handle 32-bit calls, but it has to be called for each ioctl cmd (??)
>
> In these old pre-compat_ioctl kernels you have to register each
> ioctl command individually. Yes that sucks. Live with it.
>
> > Putting aside (please) discussion of whether the kernel should presume
> > to hijack private ioctls, and whether I should be using the ioctl
> > interface at all (compatibility with app interface going back to 2.0
> > and SunOS) is there some way to make _one_ register call to indicate
> > that all my cmds are safe, or maybe an alternate ioctl entry point
> > that the kernel won't trap?
>
> Not as long as you're stuck with old 2.4 kernels. 2.6 kernels since
> 2.6.11-rc2 allow you to set up a single ->compat_ioctl() method,
> but not even RHEL4 has that yet.

Thanks,

It's working OK in my test cases: FC1/64 and the customer's RHEL3. I
just #include <asm/ioctl32.h> and register all my ioctls. Ugh.

The location of ioctl32.h seems to move from 2.4 kernel to kernel (and
distro to distro??). Any suggestion how to include in a universal way
and how to detect all the appropriate 64 bit configs for conditional
inclusion w/a 2.4 kernel? I detest #ifdef'd code but I guess I have to
do that too, or just keep one version around for this specific case :(

Thanks again,
Bill
--------------------------------------------
William D Waddington
Bainbridge Island, WA, USA
[email protected]
--------------------------------------------
"Even bugs...are unexpected signposts on
the long road of creativity..." - Ken Burtch