Hi,
It looks like the random driver in 2.4test will return a
short read, rather than blocking. This is breaking vpnd
(http://sunsite.dk/vpnd/) which breaks with "failed to
gather random data" or similar.
Here's a sample strace:
open("/dev/random", O_RDONLY) = 3
read(3, "q\321Nu\204\251^\234i\254\350\370\363\"\305\366R\2708V"..., 72) = 29
close(3) = 0
Have the semantics of the device changed, or is vpnd doing
something wrong?
Matthew.
I've seen that happen with kernel version 2.2.16!
> Hi,
>
> It looks like the random driver in 2.4test will return a
> short read, rather than blocking. This is breaking vpnd
> (http://sunsite.dk/vpnd/) which breaks with "failed to
> gather random data" or similar.
>
> Here's a sample strace:
>
> open("/dev/random", O_RDONLY) = 3
> read(3, "q\321Nu\204\251^\234i\254\350\370\363\"\305\366R\2708V"..., 72) = 29
> close(3) = 0
>
> Have the semantics of the device changed, or is vpnd doing
> something wrong?
>
> Matthew.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
Followup to: <Pine.LNX.4.10.10012011720050.23462-100000@sphinx.mythic-beasts.com>
By author: Matthew Kirkwood <[email protected]>
In newsgroup: linux.dev.kernel
>
> Hi,
>
> It looks like the random driver in 2.4test will return a
> short read, rather than blocking. This is breaking vpnd
> (http://sunsite.dk/vpnd/) which breaks with "failed to
> gather random data" or similar.
>
> Here's a sample strace:
>
> open("/dev/random", O_RDONLY) = 3
> read(3, "q\321Nu\204\251^\234i\254\350\370\363\"\305\366R\2708V"..., 72) = 29
> close(3) = 0
>
> Have the semantics of the device changed, or is vpnd doing
> something wrong?
>
vpnd is doing something wrong: it's assuming short reads won't
happen. /dev/random will block if there is NOTHING to be read;
however, if there are bytes to be read, it will return them even if it
is less than the user asked for.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
On Fri, 1 Dec 2000 [email protected] wrote:
> > open("/dev/random", O_RDONLY) = 3
> > read(3, "q\321Nu\204\251^\234i\254\350\370\363\"\305\366R\2708V"..., 72) = 29
> > close(3) = 0
> I've seen that happen with kernel version 2.2.16!
Indeed, you are correct. Is vpnd broken then, for assuming
that it can gather the required randomness in one read?
Matthew.
On 1 Dec 2000, H. Peter Anvin wrote:
> > open("/dev/random", O_RDONLY) = 3
> > read(3, "q\321Nu\204\251^\234i\254\350\370\363\"\305\366R\2708V"..., 72) = 29
> > close(3) = 0
> >
> > Have the semantics of the device changed, or is vpnd doing
> > something wrong?
>
> vpnd is doing something wrong: it's assuming short reads won't happen.
> /dev/random will block if there is NOTHING to be read; however, if
> there are bytes to be read, it will return them even if it is less
> than the user asked for.
Ah, many thanks for the clarification. I'll fix vpnd then.
Matthew.
> Indeed, you are correct. Is vpnd broken then, for assuming
> that it can gather the required randomness in one read?
Yep. It assumes that if the required randommness numbers aren't met a read
to /dev/random will block.
And it's not the only program that assumes this : I also did.
/dev/random is called a blocking random device, which more or less implies
that it will totally block. I suggest we put this somewhere in the kernel
docs, since lots of people out there assume that it totally blocks.
Means I've got to updates some sources of mine :)
> Matthew.
Igmar
On Sat, 2 Dec 2000, Igmar Palsenberg wrote:
> > Indeed, you are correct. Is vpnd broken then, for assuming
> > that it can gather the required randomness in one read?
>
> Yep. It assumes that if the required randommness numbers aren't met a read
> to /dev/random will block.
>
> And it's not the only program that assumes this : I also did.
>
> /dev/random is called a blocking random device, which more or less implies
> that it will totally block. I suggest we put this somewhere in the kernel
> docs, since lots of people out there assume that it totally blocks.
"totally block"?
For a blocking fd, read(2) has always blocked until some data is
available. There has never been a guarantee, for any driver, that
a read(2) will return the full amount of bytes requested.
There is no need to document this... man read(2) ;-)
Jeff
> For a blocking fd, read(2) has always blocked until some data is
> available. There has never been a guarantee, for any driver, that
> a read(2) will return the full amount of bytes requested.
I know. Still leaves lot's of people that assume that reading /dev/random
will return data, or will block.
I've seen lots of programs that will assume that if we request x bytes
from /dev/random it will return x bytes.
> There is no need to document this... man read(2) ;-)
>
> Jeff
Igmar
> "totally block"?
>
> For a blocking fd, read(2) has always blocked until some data is
> available. There has never been a guarantee, for any driver, that
> a read(2) will return the full amount of bytes requested.
Hmm.. Some came to mind :
Making /dev/random block if the amount requirements aren't met makes sense
to me. If I request x bytes of random stuff, and get less, I probably
reread /dev/random. If it's entropy pool is exhausted it makes sense to be
to block.
Just some mind spin.
> There is no need to document this... man read(2) ;-)
>
> Jeff
Igmar
Igmar Palsenberg wrote:
> > For a blocking fd, read(2) has always blocked until some data is
> > available. There has never been a guarantee, for any driver, that
> > a read(2) will return the full amount of bytes requested.
>
> I know. Still leaves lot's of people that assume that reading /dev/random
> will return data, or will block.
>
> I've seen lots of programs that will assume that if we request x bytes
> from /dev/random it will return x bytes.
I find this really humorous honestly. I see a lot of people assuming that if
you write N bytes or read N bytes that you will have done N bytes. There are
return values for these functions that tell you clearly how many bytes were
done.
Any programmer who has evolved sufficiently from a scriptie should take
necessary precautions to check how much data was transferred. Those who
don't..well, there is still tomorrow.
There is no reason to add any additional documentation. If we did, we'd be
starting the trend of documenting the direction a mouse moves when it's
pushed and not to be alarmed if you turn the mouse sideways and the result is
90 degrees off.
-d
> Making /dev/random block if the amount requirements aren't met makes sense
> to me. If I request x bytes of random stuff, and get less, I probably
> reread /dev/random. If it's entropy pool is exhausted it makes sense to be
> to block.
This is the job of the program accessing /dev/random. Open it blocking or
non-blocking and read until you satisfy your read buffer.
-d
David Ford writes:
> Igmar Palsenberg wrote:
>> I've seen lots of programs that will assume that if we request x bytes
>> from /dev/random it will return x bytes.
>
> I find this really humorous honestly. I see a lot of people
> assuming that if you write N bytes or read N bytes that you will
> have done N bytes. There are return values for these functions
> that tell you clearly how many bytes were done.
>
> Any programmer who has evolved sufficiently from a scriptie
> should take necessary precautions to check how much data was
> transferred. Those who don't..well, there is still tomorrow.
Yeah, people write annoying little wrapper functions that
bounce right back into the kernel until the job gets done.
This is slow, it adds both source and object bloat, and it
is a source of extra bugs. What a lovely API, eh?
The fix: write_write_write_damnit() and read_read_read_damnit(),
which go until they hit a fatal error or complete the job.
From: "Albert D. Cahalan" <[email protected]>
Date: Sat, 2 Dec 2000 17:00:32 -0500 (EST)
> Any programmer who has evolved sufficiently from a scriptie
> should take necessary precautions to check how much data was
> transferred. Those who don't..well, there is still tomorrow.
Yeah, people write annoying little wrapper functions that
bounce right back into the kernel until the job gets done.
This is slow, it adds both source and object bloat, and it
is a source of extra bugs. What a lovely API, eh?
Well, that's the Unix interface you. I you don't like it, why don't you
become a Windows programmer and try your hand at the Win32 interface? :-)
Seriously, doing something different for /dev/random compared to all
other read(2) calls is a bad idea; it will get people confused. The
answer is whenever you call read(2), you must check the return values.
People who don't are waiting to get themselves into a lot of trouble,
particularly people who writing network programs. The number of people
who assume that they can get an entire (variable-length) RPC packet by
doing a single read() call astounds me. TCP doesn't provide message
boundaries, never did and never will. The problem is that such program
will work on a LAN, and then blow up when you try using them across the
real Internet.
Secondly, the number of times that you end up going into a kernel is
relatively rare; I doubt you'd be able to notice a performance
difference in the real world using a real-world program. As far as
source/object code bloat, well, how much space does a while loop take?
And I usyally write a helper function which takes care of the while
loop, checks for errors, calls read again if EINTR is returned, etc.
- Ted
On Sat, 2 Dec 2000, Theodore Y. Ts'o wrote:
> particularly people who writing network programs. The number of people
> who assume that they can get an entire (variable-length) RPC packet by
> doing a single read() call astounds me. TCP doesn't provide message
> boundaries, never did and never will. The problem is that such program
> will work on a LAN, and then blow up when you try using them across the
> real Internet.
Erm... Not that ignoring the return values was a bright idea, but the
lack of reliable ordered datagram protocol in IP family is not a good
thing. It can be implemented over TCP, but it's a big overkill. IL is a
nice thing to have...
Date: Sat, 2 Dec 2000 18:34:44 -0500 (EST)
From: Alexander Viro <[email protected]>
Erm... Not that ignoring the return values was a bright idea, but the
lack of reliable ordered datagram protocol in IP family is not a good
thing. It can be implemented over TCP, but it's a big overkill. IL is a
nice thing to have...
You mean RDP (Reliable Data Protocol), RFC-908 and RFC-1151? It
provides reliable (i.e., will handle retransmission and dropping
duplicate) datagrams with optional sequencing guarantees if requested by
the application (but if you don't need sequencing, you don't have to pay
for it). It's built on top of IP. It's specified, but it never become
popular. But it *is* a member of the IP family, however neglected and
ignored it might be.
- Ted
Followup to: <[email protected]>
By author: Igmar Palsenberg <[email protected]>
In newsgroup: linux.dev.kernel
>
> > Indeed, you are correct. Is vpnd broken then, for assuming
> > that it can gather the required randomness in one read?
>
> Yep. It assumes that if the required randommness numbers aren't met a read
> to /dev/random will block.
>
> And it's not the only program that assumes this : I also did.
>
> /dev/random is called a blocking random device, which more or less implies
> that it will totally block. I suggest we put this somewhere in the kernel
> docs, since lots of people out there assume that it totally blocks.
>
That's pretty much ALWAYS wrong -- for pipes, sockets, you name it. A
blocking read() will only block if there is nothing to read at all.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
Followup to: <[email protected]>
By author: Igmar Palsenberg <[email protected]>
In newsgroup: linux.dev.kernel
>
> > For a blocking fd, read(2) has always blocked until some data is
> > available. There has never been a guarantee, for any driver, that
> > a read(2) will return the full amount of bytes requested.
>
> I know. Still leaves lot's of people that assume that reading /dev/random
> will return data, or will block.
>
> I've seen lots of programs that will assume that if we request x bytes
> from /dev/random it will return x bytes.
>
Again, that's wrong even when you replace /dev/random with something
else. After all, you could be getting EINTR at any time, too, or get
interrupted by a signal in the middle (in which case you'd get a short
read.)
SUCH CODE IS BROKEN. PERIOD. FULL STOP.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
Followup to: <[email protected]>
By author: Igmar Palsenberg <[email protected]>
In newsgroup: linux.dev.kernel
>
> Hmm.. Some came to mind :
>
> Making /dev/random block if the amount requirements aren't met makes sense
> to me. If I request x bytes of random stuff, and get less, I probably
> reread /dev/random. If it's entropy pool is exhausted it makes sense to be
> to block.
>
Yes, it does, but it doesn't make any sense to block if there are data
to be read. If you need a larger read then you should advance your
pointer and try again with the residual size, or use fread() which
does this for you.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
Followup to: <[email protected]>
By author: "Albert D. Cahalan" <[email protected]>
In newsgroup: linux.dev.kernel
>
> Yeah, people write annoying little wrapper functions that
> bounce right back into the kernel until the job gets done.
> This is slow, it adds both source and object bloat, and it
> is a source of extra bugs. What a lovely API, eh?
>
> The fix: write_write_write_damnit() and read_read_read_damnit(),
> which go until they hit a fatal error or complete the job.
>
RTFM(fwrite), RTFM(fread).
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
H. Peter Anvin writes:
>> Yeah, people write annoying little wrapper functions that
>> bounce right back into the kernel until the job gets done.
>> This is slow, it adds both source and object bloat, and it
>> is a source of extra bugs. What a lovely API, eh?
>>
>> The fix: write_write_write_damnit() and read_read_read_damnit(),
>> which go until they hit a fatal error or complete the job.
>
> RTFM(fwrite), RTFM(fread).
Those sure look like "annoying little wrapper functions that
bounce right back into the kernel until the job gets done".
They don't even operate directly on file descriptors.
Actually they look like they come from VMS... records???
Alexander Viro wrote:
>
> Erm... Not that ignoring the return values was a bright idea, but the
> lack of reliable ordered datagram protocol in IP family is not a good
> thing. It can be implemented over TCP, but it's a big overkill. IL is a
> nice thing to have...
Pet peeve? There are about five "reliable UDPs" floating
around. Take a look at
http://www.faqs.org/rfcs/rfc2960.html
SCTP is mainly designed as a way of transporting telephony signalling
information across IP. But it is now a quite general purpose protocol.
Culturally, this is "Telephony industry comes to IP. Telephony
industry is appalled. IP industry gets a clue".
SCTP provides the reliable delivery of messages to which you refer.
It's slightly more efficient than TCP for a given set of network
characteristics - there's no statement about implementation
efficiency here. No head-of-line blocking issues.
One very interesting part of SCTP is that transport endpoints are
explicitly set up between *hosts*, not between IP addresses. The
protocol is designed around multihomed hosts.
I don't know if anyone has looked into mapping SCTP capabilities onto
the BSD socket API. It may be hard.
The reference implementation is for userland Linux. It's at
ftp://standards.nortelnetworks.com/sigtran/
A good kernel-mode implementation of SCTP for Linux would be
a very big project. But also a very big contribution.
> > I know. Still leaves lot's of people that assume that reading /dev/random
> > will return data, or will block.
> >
> > I've seen lots of programs that will assume that if we request x bytes
> > from /dev/random it will return x bytes.
>
> I find this really humorous honestly. I see a lot of people assuming that if
> you write N bytes or read N bytes that you will have done N bytes. There are
> return values for these functions that tell you clearly how many bytes were
> done.
Of course. Lesson one : check return values
> Any programmer who has evolved sufficiently from a scriptie should take
> necessary precautions to check how much data was transferred. Those who
> don't..well, there is still tomorrow.
>
> There is no reason to add any additional documentation. If we did, we'd be
> starting the trend of documenting the direction a mouse moves when it's
> pushed and not to be alarmed if you turn the mouse sideways and the result is
> 90 degrees off.
random devices are different. If it request 10 bytes on random stuff, I
want 10 bytes. Anything less is a waste of the read, because I need 10
bytes.
At least, in my opinion.
Anyone has an insight how other *NIX'es handle this ?
> -d
Igmar
> > Making /dev/random block if the amount requirements aren't met makes sense
> > to me. If I request x bytes of random stuff, and get less, I probably
> > reread /dev/random. If it's entropy pool is exhausted it makes sense to be
> > to block.
>
> This is the job of the program accessing /dev/random. Open it blocking or
> non-blocking and read until you satisfy your read buffer.
Agree, if randomness is guaranteed in that case. I usually bail out in
that case. Time to change that :)
> -d
Igmar
> Well, that's the Unix interface you. I you don't like it, why don't you
> become a Windows programmer and try your hand at the Win32 interface? :-)
>
> Seriously, doing something different for /dev/random compared to all
> other read(2) calls is a bad idea; it will get people confused. The
> answer is whenever you call read(2), you must check the return values.
> People who don't are waiting to get themselves into a lot of trouble,
> particularly people who writing network programs. The number of people
> who assume that they can get an entire (variable-length) RPC packet by
> doing a single read() call astounds me. TCP doesn't provide message
> boundaries, never did and never will. The problem is that such program
> will work on a LAN, and then blow up when you try using them across the
> real Internet.
>
> Secondly, the number of times that you end up going into a kernel is
> relatively rare; I doubt you'd be able to notice a performance
> difference in the real world using a real-world program. As far as
> source/object code bloat, well, how much space does a while loop take?
> And I usyally write a helper function which takes care of the while
> loop, checks for errors, calls read again if EINTR is returned, etc.
Agree. I thought that en exhausted entropy pool gave less random numbers
on the next read. After having a look at the source I realized I was
taking nonsense.
> - Ted
Igmar
On Sun, 3 Dec 2000, Igmar Palsenberg wrote:
> > Any programmer who has evolved sufficiently from a scriptie should take
> > necessary precautions to check how much data was transferred. Those who
> > don't..well, there is still tomorrow.
> >
> > There is no reason to add any additional documentation. If we did, we'd be
> > starting the trend of documenting the direction a mouse moves when it's
> > pushed and not to be alarmed if you turn the mouse sideways and the result is
> > 90 degrees off.
>
> random devices are different. If it request 10 bytes on random stuff, I
> want 10 bytes. Anything less is a waste of the read, because I need 10
> bytes.
>
> At least, in my opinion.
>
> Anyone has an insight how other *NIX'es handle this ?
This is standard stuff... You are really pissing into the wind here ;)
> This is standard stuff... You are really pissing into the wind here ;)
Guess I am. Still isn't an explaination why I see a lot of broken code out
there regarding this issue.
Igmar
Igmar Palsenberg wrote:
> > This is standard stuff... You are really pissing into the wind here ;)
>
> Guess I am. Still isn't an explaination why I see a lot of broken code out
> there regarding this issue.
>
> Igmar
Broken code due to broken programmers.
-d
[email protected] (H. Peter Anvin) wrote on 02.12.00 in <[email protected]>:
> Again, that's wrong even when you replace /dev/random with something
> else. After all, you could be getting EINTR at any time, too, or get
> interrupted by a signal in the middle (in which case you'd get a short
> read.)
>
> SUCH CODE IS BROKEN. PERIOD. FULL STOP.
Well, one might argue that for some applications, it's sufficient to
detect and abort such a situation.
But not checking is *never* right. Except *maybe* for a throwaway program
whose source you erase after one use.
MfG Kai