Integration tests

[killercup]

We need to ensure that we don't regress the main guarantee we give: That we can serve files without corruption.

To test as much of the code as possible, I propose this test:

1. From a random directory structure, build an archive and index.
2. Then, start the server for these files.
3. For each file in the original directory, request it from the server, and assert that the response has the same content as the original file (make sure to decompress the response).

I'd write this in a tests/roundtrip.rs using tempfile to generate files, escargot and duct to run the main binary of this crate, and reqwest to get the data from the server.

[Michael-Overall]

I'd like to see what I can do with this.

[killercup]

Awesome! Let me know if you have any questions!

…

On Sun, 21 Oct 2018, 22:51 Michael Overall, ***@***.***> wrote: I'd like to see what I can do with this. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABOXx_5eujJ8lNINBSCJU1BtA7Uuyzjks5unN5ogaJpZM4Xyj8o> .

[killercup]

@Michael-Overall how's it going? Anything I can do to help?

[Michael-Overall]

A couple of things (sorry if I was holding things up):

I was running into an issue where the server would display the text "not found" when building the original static-filez repo via the instructions in README.md. This is on a VM running Ubuntu 18.04, and trying to open the page in Firefox.

from the static-filez directory:

cargo install --path .

• I then created a directory on the same level as the static-filez repo directory, and added 2 files files: (touch file1 file2), then added some text to them
• ran the command:

static-filez build ./temp ./docs.archive

• tried to serve the files:

static-filez serve -p 3000 docs.archive

• I then tried navigating to http://127.0.0.1:3000, as well as to the '/regex/' URL, both of which served the message "not found".

• on the assumption the server didn't have the file permissions to read certain data, I also tried to give all of the files (both the ones being archived, and the the code repo) rwx permissions for all users:

chmod -R 777 [parent directory name holding static-filez and file archives]

• this still didn't work.

Full disclosure: I was also being a bit ambitious trying to use this issue as an excuse to learn Rust, so I'm running into some scope issues trying to generate temporary files with tempfiles_in() inside a for loop. Although I try to push the file handles to a vector, the files don't seem to persist in the temporary directory (tempfiles claims it will clean up files after their handles go out of scope).
I can post what I have in that respect, or I'll keep plugging away at the problem with the help of the Rust documentation.

[killercup]

sorry if I was holding things up

Oh sorry, didn't mean to imply that -- I'm not actively working this project this week!

I was running into an issue where the server would display the text "not found" when building the original static-filez repo via the instructions in README.md.
[… well documented steps you did …]

Let me try and see if this works on a fresh container with nothing but Debian and Rust 1.29 in it:

pascal@~/P/static-filez> docker run -it --rm -v (pwd):/source rust:1.29.2
root@b2f94a80c295:/# cd /source/
root@b2f94a80c295:/source# cargo install --path .
[… a minute or five later (in which I really regret giving the docker vm only two cores) …]
root@b2f94a80c295:/source# mkdir temp
root@b2f94a80c295:/source# echo 'foo' > temp/foo
root@b2f94a80c295:/source# echo 'bar' > temp/bar
root@b2f94a80c295:/source# static-filez build temp temp.archive
root@b2f94a80c295:/source# static-filez serve temp.archive -p 3000 & # & to run in background which also makes the next prompt look weird
[1] 6643
root@b2f94a80c295:/source# Server listening on http://127.0.0.1:3000
root@b2f94a80c295:/source# curl http://127.0.0.1:3000/foo | gunzip # curl doesn't handle compressed responses by default
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0  10371      0 --:--:-- --:--:-- --:--:-- 12000
foo
root@b2f94a80c295:/source#

Looks okay? (Heh, this is actually an okay-ish integration test!)

Ohhhh, did you only try / and /regex/ with your temp dir? / only works when there is a /index.html and /regex/ when there is a /regex/index.html (by usual http server convention). If your temp dir only has file1 and file2, try http://127.0.0.1:3000/file1!

I was also being a bit ambitious trying to use this issue as an excuse to learn Rust

That's awesome! I can't promise I have much time this week, but if you like, open a PR with what you have -- including broken code, a text file full of questions, whatever you have right now -- and we can chat about that and I'll try to help answer your questions! (Feel free to contact me in your preferred way, my username is killercup on most platform, be it Twitter, IRC, Discord, or GMail :))

[Michael-Overall]

Thanks for the feedback, and thanks especially for the open and supportive stance you're taking to your contributors.

I'll try to post a pull request tomorrow at the latest (I'd still like to make an effort to clean some things up slightly), as having code up would probably also make it easier to discuss the details of the change.

I actually have more questions the more I look at the code. Questions like:

• What should the temporary file structure be like for the integration test? Based on your above comment about things not working without index.html pages, should one be added to the temp files so a file/folder list would be rendered when checking it with reqwest? How difficult would it be to crawl the folder structure via http if the structure is large/complex?

• How do we want escargot to build the project? I'm guessing a debug build will be sufficient since release takes forever?

[Michael-Overall]

As for your tips on serving the files via static-filez serve, I can indeed get the content on the files:

[killercup]

I'll try to post a pull request tomorrow at the latest

Sound good. Don't feel pressured to get something done by then, though. I'm in UTC+2, and I'll be out tomorrow evening anyway :)

What should the temporary file structure be like for the integration test? Based on your above comment about things not working without index.html pages, should one be added to the temp files so a file/folder list would be rendered when checking it with reqwest?

It works without index.html files -- they are only needed when request a directory. E.g., you can load /foo.text but when trying to access /sub/subdirectory/ it will look for an index.html file (and give you a 404 if there is none).

So you don't need to create index.html files, if the test only tries to fetch the files. It would, however, be great to have a test that checks that mathcing index.html files are returned when requesting a directory.

How do we want escargot to build the project? I'm guessing a debug build will be sufficient since release takes forever?

Debug sounds fine. Btw, since this will just call out to cargo, it'll have the same cached files available, so it should actually build instantly in most cases. (Assuming you have ever build the project before.)