Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support MapReduce executor #8

Merged
merged 2 commits into from
Mar 4, 2016
Merged

Support MapReduce executor #8

merged 2 commits into from
Mar 4, 2016

Conversation

sakama
Copy link
Contributor

@sakama sakama commented Mar 4, 2016

Embulk supports distributed executor such as an embulk-executor-mapreduce.

This executor run tasks on remote server and couldn't read local file path that is written at
 config.yml.
Because file is not there.

I changed secret_key_file parameter to accept embedded secret key.

out:
  type: sftp
  ...
  secret_key_file:
    content |
      -----BEGIN RSA PRIVATE KEY-----
      ABCDEFG...
      HIJKLMN...
      OPQRSTU...
      -----END RSA PRIVATE KEY-----
  ...

This changes keeps a backward compatibility.
So we can set local file path as before.

out:
  type: sftp
  ...
  secret_key_file: /path/to/id_rsa
  ...

@sakama
Copy link
Contributor Author

sakama commented Mar 4, 2016

CI failed. This failure will be fixed by #7.

@sakama
Copy link
Contributor Author

sakama commented Mar 4, 2016

#7 was merged. And I merged these changes.

I guess building environment for MR-Executor is little hard.
I welcome that you assign issue to me when something wrong behavior is reported.

@civitaspo
Copy link
Member

np!

civitaspo added a commit that referenced this pull request Mar 4, 2016
@civitaspo civitaspo merged commit 74a62b5 into embulk:master Mar 4, 2016
@sakama sakama deleted the support-mapreduce-executor branch March 4, 2016 12:40
@sakama
Copy link
Contributor Author

sakama commented Mar 4, 2016

Thank you!

@civitaspo civitaspo mentioned this pull request Mar 9, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants