Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate other exchange formats for config files #67

Open
YPares opened this issue Oct 16, 2019 · 0 comments
Open

Investigate other exchange formats for config files #67

YPares opened this issue Oct 16, 2019 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@YPares
Copy link
Owner

YPares commented Oct 16, 2019

For now, our config file can be in JSON/YAML which the pipeline can automatically generate.
That is nice, but porcupine (due to its inclusive philosophy) could attract other people by supporting other formats, I'm thinking about:

  • TOML (would be easy to add, and nice for pipelines with light configuration. Although it wouldn't fit workflows where we want to embed arbitrary input data in the config file --see Uniformize the resource tree and the record of options types #47 for some details--, because TOML is too flat. So it would require some thinking)
  • Avro/Thrift/Protobuf (would require more work, but given the virtual tree contains all the information about types it is already possible to do so, and it would really enhance a pipeline's capacity to be called from an external tool)
  • Apache Arrow (related to Improve integration with Haskell's and other ecosystems #9, this could be useful for pipelines using a lot of data which could actually be packed in one big parquet/arrow dataset)
@YPares YPares added enhancement New feature or request good first issue Good for newcomers labels Oct 16, 2019
@YPares YPares changed the title Investigate other formats for config file Investigate other representations for config file Oct 16, 2019
@YPares YPares changed the title Investigate other representations for config file Investigate other representations/exchange formats for config file Oct 16, 2019
@YPares YPares changed the title Investigate other representations/exchange formats for config file Investigate other exchange formats for config files Oct 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

1 participant