Add basic tests

[rkitover]

We need to start adding some basic tests and having travis run them, stuff gets constantly broken.

Things like, does it build, does the built version run, etc. would be a good start.

[lucc]

What kind of test framework do we want to use?

1. We add a makefile target test.
2. We can add random shell snippets to that target?
3. We add a tests folder with random shell scripts and make test executes them all?
4. We use some fancy testsuite?

I am against 2. For 1 and 3 I can send PRs or code very easily.

[rkitover]

@lucc Take a look here:

http://stackoverflow.com/questions/971945/unit-testing-for-shell-scripts

I don't like having to depend on bash for anything, but I'm leaning towards BATS right now since it seems like the nicest framework. Also depending on bash just for tests doesn't seem like too big a deal.

For testing the viml we'd use some vim TAP compatible framework I guess.

I think having a test suite in a t/ directory that is a mix of shell and viml test files will be good in general. Also generating TAP output would probably be a good idea for integration with Travis and other tools.

Other ideas:

1. we can set VIMPAGER_SOURCE to allow just sourcing vimpager without running anything to test the functions.
2. we can set VIMPAGER_TEST to disable some checks like being connected to a terminal for easier testing.
3. likewise for VIMCAT_SOURCE and VIMCAT_TEST

[lucc]

I had a look at bats and tried to write some tests. The first problem I encountered is that normally you do something like this in bats:

@test "-v prints the version of vimpager" {
  run standalone/vimpager -v
  [[ $status -eq 0 ]]
  [[ "$output" = "vimpager*standalone*" ]]
}

But that hangs as vimpager just execs cat because run tries to capture the output and hence vimpager has no terminal on stdout. See https://github.com/sstephenson/bats/blob/master/libexec/bats-exec-test#L58 .

So either we have to redesign vimpager to be testable with bats or we have to hack our way around this problem (I don't know how). Or we have to look for another test framework or write our own.

A simple idea for redesigning vimpager would be to wrap most shell code of vimpager into functions, put them all into functions.sh and then source functions.sh from vimpager and from the test suite. Similar to this (I am against putting the test suite into functions.sh).

[lucc]

I also had a look at some of the other test suits mentioned in that SO thread.

• shelltestrunner depends on haskell so I think it's out
• shunit2 has two websites
• shunit seems to be very old and has no documentation linked on the main page, as far as I can see
• I don't like the interface (syntax) of assert.sh and it has the same problem that bats has
• roundup looks very similar to bats but it appears to me that it requires a little more boilerplate code to wite tests (catch stdout ourself, this might also be an advantage in our situation, not sure jet). It claims to support any POSIX shell, not just bash.
• the examples of jshu look like a lot of boiler plate code

So here is my opinion:

1. Use bats or roundup (not sure jet).
2. Move all functions into a new file. This file must only contain function definitions, it should not run a single line of code! Source it from vimpager, vimcat and the test suite.
3. As a bonus we can split the vimpager code into smaller functions to make it easier to test some things and easier to understand/maintain/develop them.
4. Come up with an idea to test vim script code directly (not only if called from shell code).

So after step 2 vimpager could look like this:

#!/bin/sh

# Script for using ViM as a PAGER.
# Based on Bram's less.sh.
# git://github.com/rkitover/vimpager.git

# FIND REAL PARENT DIRECTORY
link="$0"

while true; do
    ls0="`ls -l \"$link\"`"
    new_link="`expr \"$ls0\" : '.* -> \(.*\)$'`"

    [ -z "$new_link" ] && break

    link="$new_link"
done

project_dir="`dirname \"$link\"`"
# END OF FIND REAL PARENT DIRECTORY

. "$project_dir/inc/prologue.sh"
. "$project_dir/inc/functions.sh"
main "$@"
# Copyright (c) 2016, Rafael Kitover <[email protected]> and
# Contributors (below.)
# ...

And we would wrap some code that currently is at the top level of the script into functions. For example a set_version or find_awk function.

[lucc]

As #196 is going to be merged soon (I hope :) I was looking for possibilities to test our vim code.

A general discussion on stackoverflow.

Here is what test suits and helpers I found so far, I will add more and some comments when I go on evaluating them:

• vimrunner: controls a vim instance from ruby code. depends on ruby, +clientserver and needs a X server. Some examples are in the Readme but I feel that they look very involved and long. In my opinion this is out.
• vim-ultimate-test-utility: pure vimscript, I don't like the report format, a little old
• vim-unittest: seems old, pure vimscript, object oriented
• spec_vim: vimscript, old, no examples, it tries to be very readable but I don't quite like that
• tassert_vim
• vimunit
• robot-vim: ruby
• vim-vspec: vimscript but syntax for test similar to some ruby stuff
• vader: vimscript, unique syntax for test
• VimTAP
• runVimTests
• vimtap: old, vimscript, same output as bats, looks simple