Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow users to specify base path with elastic-agent install #2500

Merged
merged 17 commits into from
Apr 21, 2023
Merged

Allow users to specify base path with elastic-agent install #2500

merged 17 commits into from
Apr 21, 2023

Conversation

ycombinator
Copy link
Contributor

@ycombinator ycombinator commented Apr 13, 2023 •
edited
Loading

What does this PR do?

This PR allows users to specify a new option, --base-path <custom-base-path> with the elastic-agent install command. An Elastic Agent that's installed with this option will be installed under <custom-base-path>/Elastic/Agent (Linux/MacOS) or <custom-base-path>\Elastic\Agent (Windows).

For example, on Linux, the default base path where Elastic Agent is installed is /opt. Thus, Elastic Agent's files get installed under /opt/Elastic/Agent. However, with the enhancement in this PR, if a user runs elastic-agent install --basepath /foo/bar, Elastic Agent's files will be installed under /foo/bar/Elastic/Agent.

The thinking behind the implementation is as follows:

  • For the install sub-command code path, the topPath — the path to the root folder where Elastic Agent's files are installed — is determined dynamically by combining the given --base-path and Elastic/Agent (on Linux/Mac) or Elastic\Agent (on Windows). The default value of --base-path is OS-specific: /Library for Mac, /opt for Linux, and C:\Program Files for Windows.
  • For every other code path, the topPath is determined from the running executable's path. This makes it irrelevant which topPath was used at installation time or how to "persist" it across restarts and upgrades.

Why is it important?

This enhancement allows users to install Elastic Agent under a different base path than the default one for their OS, to be compliant with requirements in certain organizations.

Checklist

How to test this PR locally

There are several scenarios to test with this change, detailed below.

For each scenario MacOS + Apple Silicon is assumed so please make the necessary adjustments for your test platform.

Install Elastic Agent into custom base path

  1. Uninstall Elastic Agent to start with clean slate.

    sudo elastic-agent uninstall

  2. Build Elastic Agent package from this PR.

    DEV=true SNAPSHOT=true EXTERNAL=true PLATFORMS="darwin/arm64" PACKAGES="tar.gz" mage -v clean package

  3. Install Elastic Agent into custom base path.

    cd /tmp
rm -rf elastic-agent-8.8.0-SNAPSHOT-*
tar xzf $OLDPWD/build/distributions/elastic-agent-*.tar.gz
cd elastic-agent-*
sudo ./elastic-agent install --base-path ~/foobar

  4. Check that Elastic Agent files are installed under custom base path.

    • If you have tree installed: 
      sudo tree -L 3 ~/foobar/Elastic/Agent
sudo tree -L 3 /Library/Elastic/Agent      # There should be no files here
    • Else: 
      sudo find ~/foobar/Elastic/Agent -maxdepth 3

  5. Check that symlink is correctly created (MacOS and Linux only).

    ls -l /usr/local/bin/elastic-agent

  6. Check that version and status subcommands work and return expected results.

    sudo elastic-agent version
sudo elastic-agent status

  7. Uninstall Elastic Agent.

    sudo elastic-agent uninstall

  8. Check that Elastic Agent files are removed from custom base path.

    sudo find ~/foobar -maxdepth 5

Install Elastic Agent into custom base path and upgrade it

  1. Uninstall Elastic Agent to start with clean slate.

    sudo elastic-agent uninstall

  2. Build Elastic Agent package from this PR.

    DEV=true SNAPSHOT=true EXTERNAL=true PLATFORMS="darwin/arm64" PACKAGES="tar.gz" mage -v clean package

  3. Install Elastic Agent into custom base path.

    cd /tmp
rm -rf elastic-agent-8.8.0-SNAPSHOT-*
tar xzf $OLDPWD/build/distributions/elastic-agent-*.tar.gz
cd elastic-agent-*
sudo ./elastic-agent install --base-path ~/foobar

  4. Note the installed Elastic Agent's version, particularly the SHA.

    sudo elastic-agent version

  5. Note the installed Elastic Agent's status, particularly the number of components running and their PIDs.

    sudo elastic-agent status

  6. Add a no-op commit (e.g. whitespace-only change) and rebuild Elastic Agent package so we have a new version.

    cd -
echo >> main.go
git commit -m 'Adding whitespace to generate new commit SHA in Agent version' main.go
DEV=true SNAPSHOT=true EXTERNAL=true PLATFORMS="darwin/arm64" PACKAGES="tar.gz" mage -v clean package

    Note the commit SHA of the latest commit.

  7. Upgrade the installed Elastic Agent to the new one you just built.

    sudo elastic-agent upgrade --source-uri file:///$(pwd)/build/distributions/ 8.8.0-SNAPSHOT

  8. Check that upgraded Elastic Agent files are installed under custom base path.

    • If you have tree installed: 
      sudo tree -L 3 ~/foobar/Elastic/Agent
sudo tree -L 3 /Library/Elastic/Agent      # There should be no files here
    • Else: 
      sudo find ~/foobar/Elastic/Agent -maxdepth 3

  9. Check that symlink is still correct (MacOS and Linux only).

    ls -l /usr/local/bin/elastic-agent

  10. Check the installed Elastic Agent's version, particularly the SHA, and ensure that a) it's different from the previously installed Elastic Agent's SHA from step 4 and b) it corresponds to the SHA of the latest commit from step 6.

    sudo elastic-agent version

  11. Check the installed Elastic Agent's status, particularly the number of components running and their PIDs, and ensure that the number of components is the same as before (from step 5) but their PIDs have changed.

    sudo elastic-agent status

  12. Uninstall Elastic Agent.

    sudo elastic-agent uninstall

  13. Check that Elastic Agent files are removed from custom base path.

    sudo find ~/foobar -maxdepth 5

  14. Cleanup: make sure to remove the test commit.

    git reset --hard HEAD~1

Install Elastic Agent into custom base path and check that diagnostics work

  1. Uninstall Elastic Agent to start with clean slate.

    sudo elastic-agent uninstall

  2. Build Elastic Agent package from this PR.

    DEV=true SNAPSHOT=true EXTERNAL=true PLATFORMS="darwin/arm64" PACKAGES="tar.gz" mage -v clean package

  3. Install Elastic Agent into custom base path.

    cd /tmp
rm -rf elastic-agent-8.8.0-SNAPSHOT-*
tar xzf $OLDPWD/build/distributions/elastic-agent-*.tar.gz
cd elastic-agent-*
sudo ./elastic-agent install --base-path ~/foobar

  4. Gather Elastic Agent diagnostics and check that they are as expected, particularly the paths in the variables.yaml and pre-config.yaml files.

    sudo elastic-agent diagnostics
unzip elastic-agent-diagnostics-*.zip -d diag/
cd diag
tree .       # or find .
grep -A4 'path:' variables.yaml pre-config.yaml

Install Elastic Agent into default base path (regression test)

  1. Uninstall Elastic Agent to start with clean slate.

    sudo elastic-agent uninstall

  2. Build Elastic Agent package from this PR.

    DEV=true SNAPSHOT=true EXTERNAL=true PLATFORMS="darwin/arm64" PACKAGES="tar.gz" mage -v clean package

  3. Install Elastic Agent into the default base path.

    cd /tmp
rm -rf elastic-agent-8.8.0-SNAPSHOT-*
tar xzf $OLDPWD/build/distributions/elastic-agent-*.tar.gz
cd elastic-agent-*
sudo ./elastic-agent install     # no --base-path option specified

  4. Check that Elastic Agent files are installed under default base path.

    • If you have tree installed: 
      sudo tree -L 3 /Library/Elastic/Agent
    • Else: 
      sudo find /Library/Elastic/Agent -maxdepth 3

  5. Check that symlink is correctly created (MacOS and Linux only).

    ls -l /usr/local/bin/elastic-agent

  6. Check that version and status subcommands work and return expected results.

    sudo elastic-agent version
sudo elastic-agent status

  7. Uninstall Elastic Agent.

    sudo elastic-agent uninstall

  8. Check that Elastic Agent files are removed from default base path.

    sudo find /Library/ -maxdepth 5

Install Elastic Agent into default base path and upgrade it (regression test)

  1. Uninstall Elastic Agent to start with clean slate.

    sudo elastic-agent uninstall

  2. Build Elastic Agent package from this PR.

    DEV=true SNAPSHOT=true EXTERNAL=true PLATFORMS="darwin/arm64" PACKAGES="tar.gz" mage -v clean package

  3. Install Elastic Agent into custom base path.

    cd /tmp
rm -rf elastic-agent-8.8.0-SNAPSHOT-*
tar xzf $OLDPWD/build/distributions/elastic-agent-*.tar.gz
cd elastic-agent-*
sudo ./elastic-agent install     # no --base-path option specified

  4. Note the installed Elastic Agent's version, particularly the SHA.

    sudo elastic-agent version

  5. Note the installed Elastic Agent's status, particularly the number of components running and their PIDs.

    sudo elastic-agent status

  6. Add a no-op commit (e.g. whitespace-only change) and rebuild Elastic Agent package so we have a new version.

    cd -
echo >> main.go
git commit -m 'Adding whitespace to generate new commit SHA in Agent version' main.go
DEV=true SNAPSHOT=true EXTERNAL=true PLATFORMS="darwin/arm64" PACKAGES="tar.gz" mage -v clean package

    Note the commit SHA of the latest commit.

  7. Upgrade the installed Elastic Agent to the new one you just built.

    sudo elastic-agent upgrade --source-uri file:///$(pwd)/build/distributions/ 8.8.0-SNAPSHOT

  8. Check that upgraded Elastic Agent files are installed under custom base path.

    • If you have tree installed: 
      sudo tree -L 3 ~/foobar/Elastic/Agent
sudo tree -L 3 /Library/Elastic/Agent      # There should be no files here
    • Else: 
      sudo find ~/foobar/Elastic/Agent -maxdepth 3

  9. Check that symlink is still correct (MacOS and Linux only).

    ls -l /usr/local/bin/elastic-agent

  10. Check the installed Elastic Agent's version, particularly the SHA, and ensure that a) it's different from the previously installed Elastic Agent's SHA from step 4 and b) it corresponds to the SHA of the latest commit from step 6.

    sudo elastic-agent version

  11. Check the installed Elastic Agent's status, particularly the number of components running and their PIDs, and ensure that the number of components is the same as before (from step 5) but their PIDs have changed.

    sudo elastic-agent status

  12. Uninstall Elastic Agent.

    sudo elastic-agent uninstall

  13. Check that Elastic Agent files are removed from custom base path.

    sudo find ~/foobar -maxdepth 5

  14. Cleanup: make sure to remove the test commit.

    git reset --hard HEAD~1

Run Elastic Agent in container (regression test)

  1. Build Elastic Agent Docker image from this PR.

    DEV=true SNAPSHOT=true EXTERNAL=true PLATFORMS="+all linux/arm64" mage -v clean package

  2. Run Elastic Agent Docker container.

    docker run --rm --name elastic-agent -d docker.elastic.co/beats/elastic-agent:8.8.0-SNAPSHOT

  3. Check that Elastic Agent files are installed under default base path within the container.

    docker exec elastic-agent find /usr/share/elastic-agent -maxdepth 3

  4. Check that version and status subcommands work and return expected results.

    docker exec elastic-agent elastic-agent version
docker exec elastic-agent elastic-agent status

  5. Cleanup: stop the Elastic Agent Docker container.

    docker stop elastic-agent

Run Elastic Agent without installing it (regression test)

  1. Build Elastic Agent package from this PR.

    DEV=true SNAPSHOT=true EXTERNAL=true PLATFORMS="darwin/arm64" PACKAGES="tar.gz" mage -v clean package

  2. Run Elastic Agent.

    cd /tmp
rm -rf elastic-agent-8.8.0-SNAPSHOT-*
tar xzf $OLDPWD/build/distributions/elastic-agent-*.tar.gz
cd elastic-agent-*
sudo ./elastic-agent run

  3. Check that version and status subcommands work and return expected results.

    cd /tmp/elastic-agent-*
sudo ./elastic-agent version
sudo ./elastic-agent status

@ycombinator ycombinator added enhancement New feature or request backport-skip labels Apr 13, 2023
@elasticmachine
Copy link
Contributor

elasticmachine commented Apr 13, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-04-20T19:44:56.333+0000

  • Duration: 17 min 56 sec

Test stats 🧪

Test Results
Failed 0
Passed 5547
Skipped 19
Total 5566

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages.

  • run integration tests : Run the Elastic Agent Integration tests.

  • run end-to-end tests : Generate the packages and run the E2E Tests.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@elasticmachine
Copy link
Contributor

elasticmachine commented Apr 13, 2023 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 98.529% (67/68) 👍
Files 68.644% (162/236) 👍 0.424
Classes 68.009% (304/447) 👍 0.224
Methods 54.106% (929/1717) 👎 -0.005
Lines 39.586% (10508/26545) 👎 -0.024
Conditionals 100.0% (0/0) 💚

@ycombinator ycombinator force-pushed the set-install-base-path branch 4 times, most recently from e47bb32 to e09b799 Compare April 18, 2023 00:30
@ycombinator ycombinator mentioned this pull request Apr 18, 2023
Merged
7 tasks
@ycombinator ycombinator force-pushed the set-install-base-path branch 2 times, most recently from 5d48b77 to 3886be1 Compare April 18, 2023 22:46
ycombinator added a commit that referenced this pull request Apr 19, 2023
@ycombinator
Cleanup in preparation for #2500 (#2507)
5fb9985 
* Check error

* Try a fix for the subprocess linting error

* Use dbus.NewWithContext

* Use dbus.GetServicePropertyContext

* Handle error

* Trying to get rid of tainted exec input linter false positive
@ycombinator ycombinator force-pushed the set-install-base-path branch from 3886be1 to 7423dd2 Compare April 19, 2023 02:50
@ycombinator ycombinator force-pushed the set-install-base-path branch from 7423dd2 to 09a1495 Compare April 19, 2023 18:30
@ycombinator ycombinator mentioned this pull request Apr 19, 2023
Merged
@ycombinator ycombinator force-pushed the set-install-base-path branch from b316812 to 09a1495 Compare April 19, 2023 19:26
@ycombinator ycombinator added the Team:Elastic-Agent Label for the Agent team label Apr 19, 2023
@ycombinator ycombinator force-pushed the set-install-base-path branch from 1852e7c to c3b4a1a Compare April 19, 2023 21:34
@ycombinator ycombinator marked this pull request as ready for review April 19, 2023 21:37
@ycombinator ycombinator requested a review from a team as a code owner April 19, 2023 21:37
@ycombinator ycombinator requested review from michalpristas and pchila and removed request for a team April 19, 2023 21:37
@ycombinator ycombinator changed the title WIP: Allow users to specify base path with elastic-agent install Allow users to specify base path with elastic-agent install Apr 19, 2023
@ycombinator ycombinator mentioned this pull request Apr 19, 2023
Closed
@ycombinator ycombinator requested a review from blakerouse April 20, 2023 02:35
blakerouse
blakerouse requested changes Apr 20, 2023
View reviewed changes
Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have read through this PR and was surprise on how mechanical most of the changes are, which is great to see and makes me feel some what better about us having the support this as a team.

There is a key question that is inline that we need to verify and understand what it means with changing the results of the RunningInstalled function.

I have not tested this either and with a change like this I really want to have time to build locally and test it.

internal/pkg/agent/application/info/state.go Outdated Show resolved Hide resolved
@ycombinator ycombinator requested a review from blakerouse April 20, 2023 20:08
blakerouse
blakerouse approved these changes Apr 21, 2023
View reviewed changes
Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets go with what you have with the .installed file, you have talked me into it. ;-)

Overall most of the changes are mechanical and I am happy with how this turned out and looks. Honestly thought it would have been a lot worse.

@ycombinator ycombinator merged commit dca7d62 into elastic:main Apr 21, 2023
@ycombinator ycombinator deleted the set-install-base-path branch April 21, 2023 19:23
@ycombinator ycombinator added the QA:Ready For Testing Code is merged and ready for QA to validate label Apr 21, 2023
This was referenced Apr 25, 2023
Open
Merged
Merged
@amolnater-qasource
Copy link

Hi Team,

We have created 12 testcases for this feature under Fleet test suite at links:

Below testcases will be executed after 8.8.0 release:

Further testcases for the agent installation without the --base-path flag and related tests already exists.

Known limitations:

  • Elastic Defend integration not supported.
  • RPM/DEB agents not supported.

Please let us know if we are missing any scenario to be covered here.

Thanks!

@ycombinator ycombinator mentioned this pull request May 10, 2023
Merged
7 tasks
@amolnater-qasource
Copy link

Hi Team,
We have executed 12 testcases under the Feature test run for the 8.8.0 release at the link:

Status:

  • PASS: 09
  • SKIP: 03[Related to agent upgrades, will be validated from 8.8.1 and further]

Build details:
VERSION: 8.8 BC3
BUILD: 62994
COMMIT: 85b22d307ab93fca95c1698ede4cb61d85f3d314

As the testing is completed on this feature, we are marking this as QA:Validated.

Please let us know if anything else is required from our end.
Thanks

@amolnater-qasource amolnater-qasource added QA:Validated Validated by the QA Team and removed QA:Ready For Testing Code is merged and ready for QA to validate labels May 12, 2023
@ycombinator ycombinator mentioned this pull request Oct 4, 2023
Merged
7 tasks
@pierrehilbert pierrehilbert mentioned this pull request Jan 10, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@blakerouse blakerouse blakerouse approved these changes

@michalpristas michalpristas Awaiting requested review from michalpristas michalpristas is a code owner automatically assigned from elastic/elastic-agent-control-plane

@pchila pchila Awaiting requested review from pchila pchila is a code owner automatically assigned from elastic/elastic-agent-control-plane

Assignees

@ycombinator ycombinator

Labels
backport-skip enhancement New feature or request QA:Validated Validated by the QA Team Team:Elastic-Agent Label for the Agent team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ycombinator @elasticmachine @amolnater-qasource @blakerouse