Continuous Regression Testing for Incremental Compilation

[michaelwoerister]

I'd like to setup some infrastructure that does continuous regression testing
for incremental compilation with the purpose of discovering issues and test
cases that can then be added to Rust's regular auto-test suite. I'd like to
discuss a few things before I start implementing something.

Requirements

I imagine the basic functionality of the testing infrastructure somewhat like
this:

• There is a curated list of Rust codebases available as git repositories
• There is a tool that will regularly probe those codebase for new commits
• The tool will walk the history of a given branch of each project, compiling in
  regular and in incremental mode at each step and checking that the result is
  the same
  (see cargo-incremental).
• When there is something wrong, it should post some kind of notification
  somewhere, containing enough information to reproduce the problem.
• It will re-run whenever there is a new nightly version of the compiler
• The tool should run on as many platforms as possible but at least the tier 1
  platforms.

Questions

I'm looking for input from people already dealing with our automation and
experience with this kind of thing.
Specifically I'd be interested in the following questions:

• Do we have (virtual) machines that could host this and that prospective
  maintainers have access to?
• Could some of this be hosted on travis-ci or similar services?
• Is there something in this plan that sounds like a bad idea?
• Is there some better way of achieving the given goals?

cc @rust-lang/tools @edunham @nikomatsakis

[alexcrichton]

I believe we have a Linux/Windows/Mac machine available to do tests like this, but ideally yeah I'd prefer to use Travis+Appveyor for this. We can use http://nightli.es/ to schedule Travis builds every night at least and cron schedules for Appveyor (for nightly builds) and then we could perhaps give maintainers a script to run that would email us a bug report whenever one happens?

That is, perhaps we could allow authors to opt-in easily to this infrastructure and it'd auto-report failures back to us?

[michaelwoerister]

Thanks, Alex! Yes, that more I look at it, the less sense it makes not to use a service like Travis.

That is, perhaps we could allow authors to opt-in easily to this infrastructure and it'd auto-report failures back to us?

You mean allowing library authors to sign their library up for testing? Yeah, maybe at some point, when things are really stable. In the beginning, I'd like to start out with a list of codebases compiled by us though.

[alexcrichton]

In some sense yeah, I figure that it'd be easiest if we just said "please copy these 10 lines to your .travis.yml". If we had that then we could only do that for a few projects initially but eventually ask anyone who's willing to do that.

[michaelwoerister]

I'm closing this as fixed. We have a suite of tests under at https://github.com/rust-icci that is being continuously tested on Travis (https://travis-ci.org/rust-icci) and appveyor. We'll probably want to add more projects under test in the future, but for the purposes of the incr. comp. beta milestone, the suite is already pretty good.