Add the ability to install test dependencies and run tests to Pkg

[burrowsa]

Pkg.test("PackageName") will install a package's test dependencies
(marked with @test in the REQUIRES file) then run its tests. If the
tests fail or there is no run_tests.jl provided then an error is raised.

[mlubin]

@IainNZ

[ivarne]

I think this would do better as a keyword argument. It is not obvious what parse(lines, true) means, but parse(lines, test=true) makes it obvious.

[burrowsa]

I agree and will change this

[ivarne]

This also forces a convention on the name of the run_tests.jl file, which in my opinion is much better than the long list in https://github.com/IainNZ/PackageEvaluator.jl/blob/master/src/package.jl#L146-159. There is no reason to not make a standard.

[IainNZ]

I've been in the trenches with package testing for about 12 months, initially in favour of a "standard" like this, and I've learnt that making it a standard is indeed a pretty big point of contention. Even if this does go through, there will still be many packages that don't comply or don't want to comply so PackageEvaluator isn't going anywhere.

[IainNZ]

Having said that, I think the @test thing is great, and I'd still like there to be a standard - but its not clear that this is the right way to do it.

In fact, I think a declarative list of files to be run that could be run independently might be a better idea, rather than making a main script file. But again, we're getting to the infinite discussion we've already had on julia-dev and "Julep 2" (https://gist.github.com/IainNZ/6086173)

[IainNZ]

See also: https://github.com/johnmyleswhite/PackageTesting.jl for more ideas

[burrowsa]

My motivator for this was wanting to be able to add test dependencies. Previously I just put my test dependencies in the travis config but that broke on http://status.julialang.org/ (and anywhere else the tests were run that wasn't travis) so I had to put my test dependencies in REQUIRES which seemed nasty. Most other package tools I've used in other languages allow you define test dependencies and link them to running the tests. I seem to recall that some (maven?) might even uninstall the test dependencies after the tests are run but that felt over the top.

It seems disappointing if there can't be a standard way to run tests for all Julia packages.

[IainNZ]

Yeah... maybe I'm just jaded. I kinda gave up, and just went with doing whatever it takes to make it work, I mean, check out the exceptions here:
https://github.com/IainNZ/PackageEvaluator.jl/blob/master/extra/genresults.jl#L115
and the logic in the link above.

I think people would be able to get behind the @test but the other part is tougher. Maybe we can do some sort of strawpoll on julia-dev for either a "run_tests.jl" style master file or declarative file format, and go with that.

[mlubin]

What about splitting out the @test dependency part and leaving an API to access it, removing Pkg.test? This will raise less objections and should be easy to merge. We can move standardizing the testing script to another issue.

[Keno]

I'm wondering whether making the REQUIRE format more complicated is worth it for that purpose. Why not a seperate REQUIRES.test?

[mlubin]

That's also reasonable. The testing dependencies don't need to be stored in METADATA.

[IainNZ]

If you have a declarative format for the testing files, you can include it in there as well, e.g. as json

{
"dependencies" : ["FactChecker"],
"tests" : [["test/runfirst.jl", "test/runsecond.jl"],
              "test/foo.jl",
              "test/bar.jl"]
}

where things have to be run in sequence if nested, otherwise any order

[IainNZ]

Theres also this: https://github.com/IainNZ/PackageEvaluator.jl/blob/master/extra/genresults.jl#L134-L138

    if pkg_name == "TimeSeries" || pkg_name == "MarketTechnicals"  # SUPER HACKY
        try
            Pkg.clone("git://github.com/JuliaQuant/MarketData.jl.git")
        end
    end

for non-registered dependencies that could be handled in a declaritive format, but doesn't fit in the proposed @test format (well, I guess it could, but might be confusing)

[burrowsa]

I guess I could split out the test dependency stuff and forget the test running bit but I would expect these two things to go together. I could add an API to Pkg that installs the test deps and then make PackageTesting.jl or something similar call it, except PackageTesting.jl isn't in METADATA so I guess no one uses it.

When I was first introduced to Julia (by Stefan Karpinski's talk at codemesh 2013) one of the selling points was a package manager that was git based so that any user of a package was already in a position to hack on it and create pull requests. This can't be wholly true if the user can't easily run the tests on the package.

I know I'm new around here and I'm sorry for going over old ground. I don't care what the standard is but I do believe there should be a standard way of running tests. I also believe that test dependencies should be supported.

[milktrader]

I like the way Gemfiles are configured where you could do something along the lines of this pseudocode in the REQUIRE:

DateTime

group :test do
FactCheck, version="0.3.0"
MarketData, clone="git://github.com/JuliaQuant/MarketData.jl.git"
end

[IainNZ]

yaml

--- 
dependencies: 
  - FactChecker
  - git://github.com/JuliaQuant/MarketData.jl.git
tests: 
  - 
    - test/runfirst.jl
    - test/runsecond.jl
  - test/foo.jl
  - test/bar.jl

[StefanKarpinski]

How about just having a test/REQUIRE file? Same format but it's additional requirements for testing. That seems straightforward enough and gets the job done.

[StefanKarpinski]

I do like the idea of having this – it seems like something we really need. I'm fine with standardizing on a script name for running tests, but I do prefer test/runtests.jl over test/run_tests.jl.

[porterjamesj]

+1 for test/runtests.jl and test/REQUIRE

[johnmyleswhite]

Another +1

[StefanKarpinski]

I'm a bit ambivalent about whether we should just list tests or have a script, but since you're going to be running code anyway, having a runtests.jl script does seem like it provides maximum flexibility. In the future, we can always have something where we look for runtests.jl first but if it doesn't exist, look for some declarative test list.

[mlubin]

I'd expect some testing packages to pop up that let you easily implement runtests.jl with a one-liner instead of explicitly looping through the files and including them. So I don't see the harm in the extra flexibility.

[burrowsa]

@mlubin funny you should say that, one has already popped up :)

https://github.com/burrowsa/RunTests.jl

[IainNZ]

test/runtests.jl is 👍 for me and has the benefit of being the most common current set up.

[IainNZ]

Does REQUIRE/test/REQUIRE handle non-METADATA packages?

[burrowsa]

@IainNZ could that be a separate issue to expand the format of REQUIRE (and hence test/REQUIRE) to support a URL instead of a version?

[IainNZ]

Oh definitely, I was more inquiring as to whether it currently supports it (I thought it might for some reason, which would make life even easier!)

[StefanKarpinski]

I think it's pretty reasonable to only support registered packages as requirements. It's not like registering a package is particularly onerous – just register it and tag a version.

[Keno]

This issue is hilarious with all the references from PackageEvaluator

[IainNZ]

Hah wow, didn't think of that. Was only on that first one-off-mass filing, thank goodness, otherwise it'd be bumped every day!