Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Housekeeping failing due to missing credentials #23

Closed
massdosage opened this issue Feb 1, 2018 · 1 comment
Closed

Housekeeping failing due to missing credentials #23

massdosage opened this issue Feb 1, 2018 · 1 comment

Comments

@massdosage
Copy link
Contributor

When HK module is configured with credentials like so

security:
  credential-provider: jceks://hdfs/hcom/etl/bdp/secure/hcom-data-lab/bdp-circustrain.jceks

it is failing after a Circus Train run with an error like so

2018-02-01 08:14:19,097 ERROR com.hotels.bdp.circustrain.housekeeping.HousekeepingRunner:66 - Housekeeping at instant 2018-01-29T16:14:18.843Z has failed
java.lang.RuntimeException: Unable to execute housekeeping at instant 1517242458843
at com.hotels.housekeeping.service.impl.FileSystemHousekeepingService.cleanUp(FileSystemHousekeepingService.java:82)
at com.hotels.bdp.circustrain.housekeeping.HousekeepingRunner.run(HousekeepingRunner.java:62)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:791)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:781)
at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:771)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
at com.hotels.bdp.circustrain.CircusTrain.main(CircusTrain.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).

which indicates that the credentials aren't being passed through. This is occurring with CT 11.0.0 with no other changes made to the job in question so is most likely related to the refactoring of the HK into a separate module.

@massdosage
Copy link
Contributor Author

Fixed in Circus Train 11.1.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant