- OpenSearch Continuous Integration
- Getting Started
- Deployment
- Contributing
- Getting Help
- Code of Conduct
- Security
- License
- Copyright
OpenSearch Continuous Integration is an open source CI system for OpenSearch and its plugins.
- Requires NPM to be installed
- Install project dependencies using
npm install
from this project directory - Configure aws credentials
- Deploy stacks with
npm run cdk deploy
- Create another cdk project and depend on this package
- Import the config / ci stacks alongside the other resources
new CIConfigStack(app, 'CI-Config-Beta', {}); new CIStack(app, 'CI-Beta', ciSettings, {});
- Update the
ciSettings
according to the environment needs such as SSL or strict deployment, see CIStackProps for details. - Update the
assetsSettings
according to the environment needs such as SSL or strict deployment, see deployAwsAssetProps for details. - Deploy using the CI system of your choice.
-
Setup your local machine to credentials to deploy to the AWS Account
-
Deploy the bootstrap stack by running the following command that sets up required resources to create the stacks. More info
npm run cdk bootstrap -- -c useSsl=false -c runWithOidc=false
-
Deploy the ci-config-stack using the following (takes ~1 minute to deploy) -
npm run cdk deploy OpenSearch-CI-Config-Dev -- -c useSsl=false -c runWithOidc=false
-
Optional Configure the elements of the config stack for SSL configuration
-
Optional Configure the elements setting up oidc via federate
-
Deploy the ci-stack, takes ~10 minutes to deploy (parameter values depend on step 2 and step 3)
npm run cdk deploy OpenSearch-CI-Dev -- -c useSsl=false -c runWithOidc=false
-
When OIDC is disabled, this set up will enforce the user to secure jenkins by adding first admin user on deployment. Create admin user and password, fill in all other details like name and email id to start using jenkins.
-
Go to the
OpenSearch-CI-Dev.JenkinsExternalLoadBalancerDns
url returned by CDK output to access the jenkins host. -
If you want to destroy the stack make sure you delete the agent nodes manually (via jenkins UI or AWS console) so that shared resources (like vpc, security groups, etc) can be deleted.
- Locate the secret manager arns in the ci-config-stack outputs
- Update the secret value (see docs) for the
certContentsSecret
with the certificate contents
$aws secretsmanager put-secret-value \
--secret-id MyTestDatabaseSecret_or_ARN \
--secret-string file://mycreds.json_or_value
- Update the secret value (see docs) for the
privateKeySecret
with the certificate private key - Upload the certificate to IAM see docs
- Update the secret value for the
certificateArnSecret
with the certificate arn generated by IAM - Update the secret value for
redirectUrlSecret
with a dummy or valid redirect URL. eg: https://dummyJenkinsUrl.com - Run with parameter using one of the following (refer this for value of
runWithOidc
)npm run cdk deploy OpenSearch-CI-Dev -- -c useSsl=true -c runWithOidc=true
or,cdk deploy OpenSearch-CI-Dev -c useSsl=true -c runWithOidc=true
- Continue with next steps
- Locate the secret manager arns in the ci-config-stack outputs
- Update the secret value (see docs) for the
OIDCClientIdSecret
with the credentials as json as follows:- JSON format
{ "clientId": "example_id", "clientSecret": "example_password", "wellKnownOpenIDConfigurationUrl": "https://www.example.com", "tokenServerUrl": "https://example.com/token", "authorizationServerUrl": "https://example.com/authorize", "userInfoServerUrl": "https://example.com/userinfo" }
- Command Eg: see docs
$aws secretsmanager put-secret-value \ --secret-id MyTestDatabaseSecret_or_ARN \ --secret-string file://mycreds.json_or_value
- Add additional
adminUsers
for role based authentication according to your needs, see CIStackProps for details. - Run with parameter with one of the following (refer this for value of
useSsL
) -npm run cdk deploy OpenSearch-CI-Dev -- -c runWithOidc=true -c useSsl=true
or,cdk deploy OpenSearch-CI-Dev -c runWithOidc=true -c useSsl=true
- Continue with next steps
Change in any EC2 config (specially init config) leads to replacement of EC2. The jenkins configuration is managed via code using configuration as code plugin. More details. See inital jenkins.yaml If you want to retain all the jobs and its build history,
- Update the
dataRetention
property inciSettings
to true (defaults to false) see CIStackProps for details. This will create an EFS (Elastic File System) and mount it on/var/lib/jenkins
which will retain all jobs and its build history.
Users can add global level environment variables using configuration as code as follows:
Update the envVars
property in ciSettings
to an array of key value pairs. See CIStackProps for details.
Example:
envVars = { s3Bucket: 'artifactBucket', account: 1234, isStaging: true }
The Created jenkins agent role can assume cross account role by passing agentAssumeRole
parameter
Example:
npm run cdk deploy OpenSearch-CI-Dev -- -c useSsl=false -c runWithOidc=false -c agentAssumeRole=arn:aws:iam::522XXX13897:role/sample-assume-role
NOTE: The assume role has to be pre-created for the agents to assume. Once CDK stack is deployed with -c agentAssumeRole
flag, make sure this flag is passed for next CDK operations to make sure this created policy that assumes cross-account role is not removed.
In cases where you need to run additional logic/commands, such as adding a cron to emit ssl cert expiry metric, you can pass the commands as a script using additionalCommands
context parameter.
Below sample will write the python script to $HOME/hello-world path on jenkins master node and then execute it once the jenkins master node has been brought up.
cat << EOF > $HOME/hello-world && chmod 755 $HOME/hello-world && $HOME/hello-world
#!/usr/bin/env python3
def print_hello():
print("Hello World")
if __name__ == "__main__":
print_hello()
EOF
To use above example, you need to write the contents of the script to a file, say example.txt and pass the path of example.txt to additionalCommands
paramter.
Usage:
npm run cdk deploy OpenSearch-CI-Dev -- -c useSsl=false -c runWithOidc=false -c additionalCommands='./example.txt'
Useful links
- Log are found in Cloud Watch Logs
- Need to access the host, ssh via Session Manager in EC2 Console
- Instance instance isn't coming up, get the system log in EC2 Console
npm run build
compile typescript to js, run lint, run testsnpm run watch
watch for changes and compilenpm run cdk deploy
deploy this stack to your default AWS account/regionnpm run cdk diff
compare deployed stack with current statenpm run cdk synth
emits the synthesized CloudFormation template
Built using AWS Cloud Development Kit the configuration of the CI systems will be available for replication in your own accounts. The Jenkins instance will be hardened and publically visible, connected to GitHub to make build notifications easy for everyone to see.
See developer guide and how to contribute to this project.
If you find a bug, or have a feature request, please don't hesitate to open an issue in this repository.
For more information, see project website and documentation. If you need help and are unsure where to open an issue, try forums.
This project has adopted the Amazon Open Source Code of Conduct. For more information see the Code of Conduct FAQ, or contact [email protected] with any additional questions or comments.
If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page. Please do not create a public GitHub issue.
This project is licensed under the Apache v2.0 License.
Copyright OpenSearch Contributors. See NOTICE for details.