File::readdir fails for filename longer than 128 bytes

[harmic]

If a directory contains a file with a name longer than 128 bytes, that filename is not returned when iterating over the directory contents using readdir().

Example program:

use std::{net::TcpStream, path::Path};
use ssh2::Session;

fn main() {

    let tcp = TcpStream::connect("127.0.0.1:22").unwrap();
    let mut sess = Session::new().unwrap();
    sess.set_tcp_stream(tcp);
    sess.handshake().unwrap();
    sess.userauth_password("xxxx","xxxx").unwrap();
    let sftp = sess.sftp().unwrap();

    let mut dir = sftp.opendir(Path::new("/home/someuser/tst")).unwrap();
    loop {
        match dir.readdir() {
            Ok((dir,..)) => println!("Found: {}", dir.display()),
            Err(e) => {
                println!("Err: {}", e);
                break;
            },
        }
    }
}

In the test directory, create a file with name 128 chars long:

mkdir tst
touch tst/12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678

Output:

Found: .
Found: ..
Err: [Session(-16)] no more files

The problem appears to be the way in which the buffer is managed:

let mut buf = Vec::<u8>::with_capacity(128);
...
if rc == raw::LIBSSH2_ERROR_BUFFER_TOO_SMALL {
    let cap = buf.capacity();
    buf.reserve(cap);
}  ...

Ie. create a Vec with capacity 128, then pass it in to libssh2 readdir_ex. If we get back ERROR_BUFFER_TOO_SMALL, then double the buffer's capacity.

The problem is that Vec::reserve reserves enough capacity in the vec for at least additional elements, and does nothing if capacity is already sufficient. Since the actual length of the vec is still 0, buf.reserve(cap) does nothing because the Vec already has enough space for that.

A simple solution would be:

buf.reserve(cap * 2);

Unfortunately, in the process of testing this, we found a bug in libssh2, where if it returns LIBSSH2_ERROR_BUFFER_TOO_SMALL, then it returns one less that the actual number of directories. See this libssh2 bug report.
Until this is fixed, perhaps we should be allocating a bigger buffer from the start.

[harmic]

Any feedback on this? It is blocking us at the moment.

I have not had much movement on the associated libssh2 bug report but I was thinking that maybe in the interim we could improve the situation by

1. Changing the way the buffer is being resized, as described above, which would prepare us if libssh2 fixes their bug
2. Allocating a bigger bufffer to begin with, which would lessen the chance of having this issue, at the cost of more memory being used for all cases. Maybe double the initial buffer to 256.

Would a PR that implemented these two suggestions be accepted?

[yodaldevoid]

I think this is a duplicate of #217. To answer your final question, yet a PR fixing both issues would be accepted. Right now there is a PR for the second proposal at #245.

[sourcefrog]

I think this is not yet published to crates.io? If there are now new active maintainers who can push a new crate release may I ask that this be pushed?