Partial blocking reads throw EOFError instead of blocking until read complete (e.g., reading Int from /dev/random)

[hessammehr]

(Look further down for what seems to be the real problem)
Using Julia 0.4.0 on Linux (Ubuntu 14.10 as well as the latest Manjaro) rand(RandomDevice(false), 100) almost always fails with EOFError after the first few numbers. RandomDevice(true) works flawlessly.

The following fails similarly and seems to be how rand(RandomDevice(false), n) gets its random data.

a = open("/dev/random")
for i in 1:100
    read(a, Int64)
end

ERROR: EOFError: read end of file
 in read at iostream.jl:170

[nalimilan]

BTW, with latest git master, ?RandomDevice does not mention the unlimited::Bool argument. How did you find out that you could use it?

[hessammehr]

It's still there in master at random.jl:35. Interestingly, you can't call it as

RandomDevice(unlimited=false)

=> ERROR: MethodError: `call` has no method matching call(::Type{RandomDevice})

even though RandomDevice(false) works as expected.

RandomDevice(unlimited=false)

=> RandomDevice(IOStream(<file /dev/random>))

There're definitely things about Julia constructors that I have yet to learn .

[pao]

Is the expected behavior a different error?

An error is almost certainly assured: "When the entropy pool is empty, reads from /dev/random will block until additional environmental noise is gathered." (http://man7.org/linux/man-pages/man4/random.4.html)

[pao]

@hessammehr, that's because specifying the name of the parameter only works when the argument is a keyword argument (after a semicolon in the formal parameter list).

[hessammehr]

@pao I was expecting the function to block until all 100 random numbers were ready, and it certainly does block for a while. In fact, it will pause indefinitely if you don't give it any entropy, i.e., don't touch the mouse and keyboard, no network requests, etc. Pressing random keys on the keyboard will cause the function to fail (and sometimes not) and return a partially filled array of random numbers.

[pao]

Ah, thanks, that's much clearer. Wonder if that makes this a libuv-related thing?

[hessammehr]

That's what I think. It seems to come from iostream.jl:170, which is a ccall, presumably to libuv.

[hessammehr]

Correction, Julia's read{T<:Union{UInt16, Int16, UInt32, Int32, UInt64, Int64}}(s::IOStream, ::Type{T}) calls into src/sys.c:301, where the error is thrown

DLLEXPORT uint64_t jl_ios_get_nbyte_int(ios_t *s, const size_t n)
{
    assert(n <= 8);
    size_t ret = ios_readprep(s, n);
    if (ret < n)
        throw_eof_error();
.
.
.

ios_readprep(s,n) in turns calls into support/ios.c:321, which calls _os_read in the same file and ultimately the system read function. Now, according to the read(2) manual page, a return value smaller than expected doesn't necessarily mean EOF. However, the above snippet will throw EOFError if anything less than n bytes is read.

[wildart]

unlimeted flag provides distinction between /dev/urandom and /dev/random. /dev/random is blocking when entropy pool is exhausted, while /dev/urandom behaves as pseudo-random generator if there is not sufficient entropy. I guess access to both devices is non-blocking, so you see an error for /dev/random on empty pool. See http://man7.org/linux/man-pages/man4/random.4.html.

[pao]

@wildart, I already quoted the man page.

[wildart]

My bad, I guess a solution should be in different reading mode (blocking and non-blocking) for different deceives.

[hessammehr]

The following works fine.

rand(RandomDevice(false),Int8, 100)

and so does this:

a = open("/dev/random", "r")
for i in 1:100
    read(a, Int8)
end

Both also work with Int16, Int32, and Float32, even when the function call blocks to wait for entropy.

[hessammehr]

This might help understand the problem:

#include <unistd.h>
#include <stdio.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>

int main() {
        printf("sizeof(unsigned long int) = %d\n", sizeof(unsigned long int));
        int a = open("/dev/random", O_RDONLY), cnt, ret;
        unsigned long int res;
        for (cnt=0;cnt<1000;cnt++) {
                ret = read(a, (void*) &res, sizeof(long int));
                printf("Returned: %d Read:%lu Errno:%s\n", ret, res, strerror(errno));
        }
}

A typical run of the above code will correctly read 8 bytes per iteration for the first few iterations:

sizeof(unsigned long int) = 8
Returned: 8 Read:6129408719018679362 Errno:Success
Returned: 8 Read:7851765565466219663 Errno:Success

then exhaust /dev/random and start returning 6 bytes per iteration indefinitely, but never less.

Returned: 7 Read:7790287781257302767 Errno:Success
Returned: 6 Read:7790267631181720159 Errno:Success
Returned: 6 Read:7790290698167501351 Errno:Success
Returned: 6 Read:7790370641659074752 Errno:Success

A similar thing happens when dev/tty? are used instead of /dev/random. The C code happily returns 1 byte per loop iteration; Julia fails (after blocking for input) with EOFError if anything other than Int8 or UInt8 is used:

a = open("/dev/tty1", "r")
for i in 1:100
    read(a, Int8)
end

[pao]

Is this a valid summary: If Julia performs a partial read, it produces EOFError, even when the underlying read call should block? (At least with /dev/random. Just reread and saw you also tried /dev/ttys.) That seems consistent with the idea with large reads failing. I'm going to tentatively retitle and relabel based on that; let us know if I got it wrong.

[hessammehr]

Thanks, here's a small revision: "If Julia performs a partial read from a blocking source (e.g. /dev/random or /dev/tty) it fails with EOFError when the number of bytes read is smaller than the datatype Julia is trying to reading."

So far I have found that /dev/random, when exhausted and blocking, yields data in 6-byte chunks. /dev/tty on the other hand, gives single bytes per read.

[hessammehr]

@pao Do you think I should revise my first comment so it doesn't confuse people?

[StefanKarpinski]

No, I think having the issue history intact is good. Maybe add a note indicating to look further down for the real problem.

[hessammehr]

Making the following change at sys.c:305 fixed the problem on my Linux machine.

DLLEXPORT uint64_t jl_ios_get_nbyte_int(ios_t *s, const size_t n)
{
    assert(n <= 8);
    size_t ret = ios_readprep(s, n);
    if (ret < n)
        throw_eof_error();

to

DLLEXPORT uint64_t jl_ios_get_nbyte_int(ios_t *s, const size_t n)
{
    assert(n <= 8);
    size_t ret = ios_readprep(s, n);
    if (ret == 0)
        throw_eof_error();

Julia now blocks until the requested number of bytes has been read from the stream, i.e., read(open("/dev/tty1"), UInt64) will block until 8 bytes have been read from tty1.

If ios_readprep is expected to follow the UNIX read(2) semantics, reading less than the expected number of bytes does not imply EOF. file tests are still passing and Julia correctly complains on actual EOFs in input.

[nalimilan]

Could you make a pull request with this change? That would run the full testsuite on it. Can you also check that blocking waiting for more input is consistent with the behaviour described in the docs?

Could be good to add a test too.

[hessammehr]

@nalimilan The docs are pretty sparse for read:

read(stream, type, dims)
Read a series of values of the given type from a stream, in canonical binary representation. dims is either a tuple or a series of integer arguments specifying the size of Array to return.

readbytes is a little more explicit

readbytes(stream, nb=typemax(Int); all=true)
Read at most nb bytes from the stream, returning a Vector{UInt8} of the bytes read.
If all is true (the default), this function will block repeatedly trying to read all requested bytes, until an error or end-of-file occurs. If all is false, at most one read call is performed, and the amount of data returned is device-dependent. Note that not all stream types support the all option.

readbytes(open("/dev/random","r"), 10, all=true)

The above has desired property of blocking until all 10 bytes have been read. Perhaps the random number generator can use readbytes instead of read, then?

[nalimilan]

While it doesn't sound unreasonable to expect both read and readbytes to behave the same and block until enough bytes can be read, somebody more knowledgeable than me will have to comment.

[hessammehr]

I am no expert, but to me the reasonable (least surprising) thing would be for read to block until at least one element of the requested type has been read. That is, read(stream, Int64, 5) should read between 8 and 40 bytes, blocking if less than 8 bytes are available. It seems like the current behavior is to block until the stream makes any number of device-dependent bytes available for reading. This might be less than 8 (6 for /dev/random and 1 for /dev/tty on my computer), hence the EOFError.

I think we can agree that an EOFError should not be thrown unless the stream is definitely terminated, i.e., closed socket, detached tty, or end of a regular file.

[JeffBezanson]

Looks like a fix for this has been merged? Plz reopen if I'm missing something.