Add OpenerTarget Support.

[KyleJamesWalker]

Allow a single luigi target to open multiple file system types. This can allow for easier local testing, unit test, and greater flexibility of your luigi tasks.

Example:

import luigi

from luigi.contrib.opener import OpenerTarget

class ConfigurableTask(luigi.Task):
    output_file = luigi.parameter(default='s3://foo/bar/baz.txt')

    def run(self):
        payload = {'foo': 'bar'}
        with self.output().open('w') as fp:
            json.dump(payload, fp)

    def output(self):
        return OpenerTarget(self.output_file)

example1 = ConfigurableTask(output_file='/tmp/local_file.txt')
example2 = ConfigurableTask(output_file='mock://tmp/memory_file.txt')

{% endhighlight %}

I've only add three target types so far to make sure this pattern is accepted into the code base (and this is all I needed to run our pipeline in all environments needed).

[KyleJamesWalker]

Not sure what's going on with the error, seems to be the same error in #1547, #1543, #1538, and others.

Warning, treated as error:
./luigi/doc/api/luigi.rst:11: WARNING: Could not lex literal_block as "python3". Highlighting skipped.

[KyleJamesWalker]

Note this is fixed if you edit ./luigi/.tox/docs/lib/python2.7/site-packages/sphinx/config.py and change highlight_language = ('python3', 'env'), to highlight_language = ('python', 'env'),

Maybe a config was changed, or the sphinx version isn't pinned and it had a breaking change added?

[erikbern]

minor thing but shouldn't it be just "mock:foo/bar.txt"? double slash usually indicates a hostname

not blocking

[KyleJamesWalker]

I set it up to use a URI-like syntax (inspired by the syntax of http://commons.apache.org/vfs/filesystems.html).

[erikbern]

that's great – thanks for making sure we adhere to standards!

[Tarrasch]

@sisidra didn't Spotify want something like this once?

@KyleJamesWalker, out of curiosity, are you still using this module. Is everything working out good?

[sisidra]

@Tarrasch not sure. We are in JVM land and Luigi is only orchestration layer. Anyway thanks for pinging ;)

[Tarrasch]

I think luigi only for orchestration is the case for many luigi players these days. I also don't use luigi MR today based on my experience from Spotify. Anyway I thought Spotify previously had migration issues when it came to HDFS --> GFS migration. And that this Opener factory could smooth things. :)