Agent MSI support #212

amitkanfer · 2023-11-19T20:37:00Z

Support basic functionality e2e (copy files and execute agent install from the target folder)
Support uninstall
Check return code from agent install and fail execution if code is different than 0
Support custom installation folder (EDIT: not sure this is needed. See comment here)
Implement the buildkite logic to trigger an MSI build whenever there's a new Agent version (similar to the logic we have with beats)
Cleanup all temp files after agent install is executed successfully
check installation with endpoint
check upgrade flow

To test, run from a command prompt with administrator permissions:

elastic-agent.msi INSTALLARGS="--url=<URL> --enrollment-token=<TOKEN>"

Another option, with logs collection:

msiexec -i elastic-agent.msi INSTALLARGS="--url=<URL> --enrollment-token=<TOKEN>" -L*V "log.txt"

Few notes:

If INSTALLARGS are not provided, the MSI will copy the files to a temp folder and finish.
If INSTALLARGS are provided, the MSI will copy the files to a temp folder and then run the elastic-agent install command with the provided args. If the install flow is successful, the temp folder is deleted.
If INSTALLARGS are provided but the elastic-agent install command fails - the top folder is NOT deleted to allow further troubleshooting.
MSI uninstall will call elatic-agent uninstall on a "best effort" basis and will always finish successfully (if not, a 3rd party tool will be needed to remove the windows service)
upgrades are not supported, and in fact prevented as implemented here. Upgrades shall happen via fleet.
if the elastic-agent install command fails (for any reason), the MSI will rollback all changes.

nimarezainia · 2023-11-19T23:05:49Z

@amitkanfer is it possible to have the MSI set the --base-path option for the agent install command? (as @cmacknz mentions here)?

Setting the path was a big requirement from enterprises and I can't see any reason why it wouldn't also be a requirement for the same set of users who would want to utilize the MSI (aside from Defend users of course). it would be great not to have diverging product behaviors if possible. (of course I am saying this not knowing what the amount of effort involved exactly is)

amitkanfer · 2023-11-20T07:44:05Z

i'll look into that @nimarezainia . My concern is more around the uninstall flow. Not sure how the new base path will be available to the "add/remove features" and the other native Windows applications. Probably a solved problem, but i just need to understand them.

amitkanfer · 2023-11-20T18:54:14Z

I also need to check and implement upgrade flows. editing the description

cmacknz

When this gets merged, at what point is it included in the stack release process? Does it show up in the snapshots for the next minor and would be released what that version is released? Or is there more work to do to have this actually show up in the downloads page for agent?

Following from having a way to build this, we need a way to test it. Likely we can do this as an agent integration test, but since this isn't actually packaged from the agent repository we'd only want the test to run every time there is a new snapshot (or once per day or something).

src/installer/BeatPackageCompiler/AgentCustomAction.cs

src/installer/BeatPackageCompiler/BeatPackageCompiler.cs

src/installer/BeatPackageCompiler/Properties/launchSettings.json

cmacknz · 2023-11-24T15:24:34Z

src/shared/ArtifactPackage.cs

@@ -68,6 +68,12 @@ static bool FromFilenameOrUrl(string fileName, string url, out ArtifactPackage a
            Architecture = rxGroups["arch"].Value.ToLower();
            IsSnapshot = !rxGroups["snapshot"].Value.IsEmpty();

+            // HACK


Is there a better way to do this, or will this live on forever :)?

that's a very good question :)

src/installer/BeatPackageCompiler/AgentCustomAction.cs

Co-authored-by: Craig MacKenzie <[email protected]>

amitkanfer · 2023-12-02T13:37:06Z

@cmacknz

When this gets merged, at what point is it included in the stack release process?

There will be a follow up PR to make the necessary changes in the pipelines. @dliappis is will be reviewing the work. We should be able to first merge this PR, make sure there's no impact on existing (beats) MSIs, and then trigger the agent MSI flow.

At the moment, i'm focused on testing the install / upgrade / uninstall flows. Once they work 100% i'll cleanup the PR and go through the review cycle

amitkanfer · 2023-12-02T13:44:18Z

Will try following https://stackoverflow.com/a/17790056 to show error messages to the end user. Note that this is valid only when running the installer w/o MDMs

src/installer/BeatPackageCompiler/AgentCustomAction.cs

src/installer/BeatPackageCompiler/BeatPackageCompiler.cs

…ack-installers into agent_support

src/installer/BeatPackageCompiler/AgentCustomAction.cs

strawgate · 2023-12-04T17:46:57Z

src/installer/BeatPackageCompiler/AgentCustomAction.cs

+            process.StartInfo.CreateNoWindow = true;
+            session.Log("Running command: " + process.StartInfo.FileName + " " + process.StartInfo.Arguments);
+            process.Start();
+            session.Log(process.StandardOutput.ReadToEnd());


Should we also log process.standarderror?

it's trickier to do both and didn't succeed on the first try... so yes, we should, will work on it

for now i just switched to stderr instead of stdout

strawgate · 2023-12-04T17:47:33Z

src/installer/BeatPackageCompiler/AgentCustomAction.cs

+                System.Diagnostics.Process process = new System.Diagnostics.Process();
+                process.StartInfo.WorkingDirectory = install_folder;
+                process.StartInfo.FileName = Path.Combine(install_folder, "elastic-agent.exe");
+                process.StartInfo.Arguments = "install -f " + install_args;


Should we consider always adding --delay-enroll so that installation success doesn't depend on fleet service reachability + valid API token?

one of the tasks was to make sure the service was up and running right after the installation flow... Don't you think the admin would want to know if the token is wrong when rolling out agent installations?

As the release manager can't trigger the workflow introduced in 566b9e4 , we maintain the old way of building snapshot/staging beats artifacts by keeping the trigger pipeline only for that branch. Note: this means that this PR **MUST NOT** be backported to the elastic-stack-installers 7.17 branch. Details: https://elasticco.atlassian.net/browse/REL-1004?focusedCommentId=107973

…ack-installers into agent_support

Add Install, Upgrade, and Uninstall MSI tests using Pester w/ Powershell Core

…uild on test failure for now.

* first draft * 6 * 6 * Preventing agent upgrade via MSI * Update src/installer/BeatPackageCompiler/BeatPackageCompiler.cs Co-authored-by: Craig MacKenzie <[email protected]> * Update README.md * Update README.md * Fixing uninstall flow and install cleanup flow * rolling back if agent install command fails * redirecting stdout to MSI log * Failing the uninstall flow in case agent uninstall command fails * Don't attempt calling agent install if the file doesn't exist * adding BK logic (#219) * Update build.ps1 * Clean up path length Instead of re-cloning to a checkout with a shorter path, we rename the existing BK checkout. * Fix 798ff24 * Typo * Build from MANIFEST_URL and don't sign This commit implements the refactoring to allow a short term integration with the unified release. It's meant to be triggered by getting passed a $MANIFEST_URL and $DRA_WORKFLOW env var. * Update README.md Co-authored-by: Craig MacKenzie <[email protected]> * Update README.md Co-authored-by: Craig MacKenzie <[email protected]> * Update README.md Co-authored-by: Craig MacKenzie <[email protected]> * Update README.md Co-authored-by: Craig MacKenzie <[email protected]> * Redirecting stderr and removing PATH manipulation for Agent MSI * Remove cron schedule As discussed in https://elasticco.atlassian.net/browse/REL-1004?focusedCommentId=107598 * Trigger 7.17 beats DRA using schedule As the release manager can't trigger the workflow introduced in 566b9e4 , we maintain the old way of building snapshot/staging beats artifacts by keeping the trigger pipeline only for that branch. Note: this means that this PR **MUST NOT** be backported to the elastic-stack-installers 7.17 branch. Details: https://elasticco.atlassian.net/browse/REL-1004?focusedCommentId=107973 * Making sure Agent MSI runs as an administrator * Add MSI tests for Agent Pipeline (#220) Add Install, Upgrade, and Uninstall MSI tests using Pester w/ Powershell Core * Enabling agent tests * Disable "Default" test case as it doesn't exist anymore. Don't fail build on test failure for now. * Default mode clean-up * Update build.ps1 --------- Co-authored-by: Craig MacKenzie <[email protected]> Co-authored-by: Dimitrios Liappis <[email protected]> Co-authored-by: William Easton <[email protected]> Co-authored-by: William Easton <[email protected]>

This commit fixes a bug introduced in PR elastic#212 which missed the switch from WORKFLOW -> DRA_WORKFLOW from the publish step, resulting in failures[^1]. [^1]: https://buildkite.com/elastic/elastic-stack-installers/builds/3641#018d5b0b-5ae5-404c-b79f-211468f45413/51-52

This commit fixes a bug introduced in PR #212 which missed the switch from WORKFLOW -> DRA_WORKFLOW from the publish step, resulting in failures[^1]. [^1]: https://buildkite.com/elastic/elastic-stack-installers/builds/3641#018d5b0b-5ae5-404c-b79f-211468f45413/51-52

This commit fixes the downloaded artifacts path to the expected path by the release manager image (`bin/out`). The original path was changed in #212 and the previous step because path lengths exceeded a limit imposed by the msi builder.

This commit fixes the downloaded artifacts path to the expected path by the release manager image (bin/out). The original path was changed in #212 and the previous step because path lengths exceeded a limit imposed by the msi builder.

This reverts commit ffffea6.

* Revert "Fix artifacts path for dra publish (#227)" This reverts commit 2739036. * Revert "Fix dra publish step (#226)" This reverts commit 98f830a. * Revert "Agent MSI support (#212)" This reverts commit ffffea6.

This commit fixes the downloaded artifacts path to the expected path by the release manager image (bin/out). The original path was changed in #212 and the previous step because path lengths exceeded a limit imposed by the msi builder.

PR #212 made some changes on `main` that require the presence of `MANIFEST_URL`. However, the pipeline is still configured to run the same script when a PR (from an upstream branch) is raised and it currently fails. This commit skips tests when triggered from another pipeline (unified release), and allows running tests when triggered by PR creation by picking up a suitable `MANIFEST_URL`.

PR #212 made some changes on `main` that require the presence of `MANIFEST_URL`. However, the pipeline is still configured to run the same script when a PR (from an upstream branch) is raised and it currently fails. This commit skips tests when triggered from another pipeline (unified release), and allows running tests when triggered by PR creation by picking up a suitable `MANIFEST_URL`. DRA publishing is skipped unless we are triggered from the unified release. Closes elastic/ingest-dev#2895

mbudge · 2024-04-17T14:08:58Z

When will this be GA with supporting documentation? Says it's beta.

nimarezainia · 2024-04-18T01:50:34Z

@mbudge we are looking at 8.14 for GA and docs will be ready then.

kilfoyle · 2024-04-18T15:00:31Z

@amitkanfer , @nimarezainia I've opened the issue above for the docs.

Amit, I'd really appreciate if you, or anyone, can provide the install steps and any other details users will need. Also, are there any UI changes planned? Thanks!

amitkanfer added 3 commits November 19, 2023 21:02

first draft

f7f4e23

6

0a38799

6

98a893b

amitkanfer requested a review from a team as a code owner November 19, 2023 20:37

amitkanfer requested a review from leehinman November 19, 2023 20:37

amitkanfer mentioned this pull request Nov 19, 2023

Agent MSI support #206

Closed

7 tasks

cmacknz requested a review from a team November 23, 2023 19:02

cmacknz reviewed Nov 24, 2023

View reviewed changes

leehinman requested changes Nov 27, 2023

View reviewed changes

src/installer/BeatPackageCompiler/AgentCustomAction.cs Show resolved Hide resolved

pierrehilbert mentioned this pull request Nov 30, 2023

Improve Agent Install/Uninstall/Upgrade elastic/elastic-agent#835

Open

amitkanfer and others added 2 commits December 1, 2023 20:32

Preventing agent upgrade via MSI

67cd709

Update src/installer/BeatPackageCompiler/BeatPackageCompiler.cs

947738e

Co-authored-by: Craig MacKenzie <[email protected]>

amitkanfer added 2 commits December 3, 2023 14:44

Update README.md

46c394a

Update README.md

093e120

strawgate reviewed Dec 3, 2023

View reviewed changes

src/installer/BeatPackageCompiler/AgentCustomAction.cs Show resolved Hide resolved

strawgate reviewed Dec 3, 2023

View reviewed changes

src/installer/BeatPackageCompiler/AgentCustomAction.cs Outdated Show resolved Hide resolved

strawgate reviewed Dec 3, 2023

View reviewed changes

src/installer/BeatPackageCompiler/BeatPackageCompiler.cs Outdated Show resolved Hide resolved

amitkanfer added 4 commits December 3, 2023 21:42

Fixing uninstall flow and install cleanup flow

1c32ac2

Merge branch 'agent_support' of https://github.com/elastic/elastic-st…

0a40567

…ack-installers into agent_support

rolling back if agent install command fails

add4b02

redirecting stdout to MSI log

04c589b

strawgate reviewed Dec 4, 2023

View reviewed changes

src/installer/BeatPackageCompiler/AgentCustomAction.cs Show resolved Hide resolved

strawgate reviewed Dec 4, 2023

View reviewed changes

Failing the uninstall flow in case agent uninstall command fails

818cb23

dliappis and others added 8 commits January 18, 2024 16:49

Making sure Agent MSI runs as an administrator

41444d5

Merge branch 'agent_support' of https://github.com/elastic/elastic-st…

7832f16

…ack-installers into agent_support

Add MSI tests for Agent Pipeline (#220)

a245c94

Add Install, Upgrade, and Uninstall MSI tests using Pester w/ Powershell Core

Enabling agent tests

07d0d86

Disable "Default" test case as it doesn't exist anymore. Don't fail b…

622fd2b

…uild on test failure for now.

Default mode clean-up

00dc987

Update build.ps1

7c6a596

alpar-t merged commit 887274a into main Jan 29, 2024
2 of 3 checks passed

alpar-t deleted the agent_support branch January 29, 2024 18:00

dliappis mentioned this pull request Jan 30, 2024

Fix dra publish step #226

Merged

This was referenced Jan 31, 2024

Fix artifacts path for dra publish #227

Merged

Selectively skip and fix tests after PR#212 #225

Closed

dliappis added a commit to dliappis/elastic-stack-installers that referenced this pull request Jan 31, 2024

Revert "Agent MSI support (elastic#212)"

efad2c8

This reverts commit ffffea6.

dliappis mentioned this pull request Feb 2, 2024

Selectively skip and fix tests after PR#212 #235

Merged

kilfoyle mentioned this pull request Apr 18, 2024

Add docs for MSI support elastic/ingest-docs#1036

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Agent MSI support #212

Agent MSI support #212

amitkanfer commented Nov 19, 2023 •

edited

Loading

nimarezainia commented Nov 19, 2023

amitkanfer commented Nov 20, 2023

amitkanfer commented Nov 20, 2023

cmacknz left a comment

cmacknz Nov 24, 2023

amitkanfer Dec 1, 2023

amitkanfer commented Dec 2, 2023

amitkanfer commented Dec 2, 2023

strawgate Dec 4, 2023

amitkanfer Dec 4, 2023

amitkanfer Jan 17, 2024

strawgate Dec 4, 2023

amitkanfer Dec 5, 2023

mbudge commented Apr 17, 2024

nimarezainia commented Apr 18, 2024

kilfoyle commented Apr 18, 2024

Agent MSI support #212

Agent MSI support #212

Conversation

amitkanfer commented Nov 19, 2023 • edited Loading

nimarezainia commented Nov 19, 2023

amitkanfer commented Nov 20, 2023

amitkanfer commented Nov 20, 2023

cmacknz left a comment

Choose a reason for hiding this comment

cmacknz Nov 24, 2023

Choose a reason for hiding this comment

amitkanfer Dec 1, 2023

Choose a reason for hiding this comment

amitkanfer commented Dec 2, 2023

amitkanfer commented Dec 2, 2023

strawgate Dec 4, 2023

Choose a reason for hiding this comment

amitkanfer Dec 4, 2023

Choose a reason for hiding this comment

amitkanfer Jan 17, 2024

Choose a reason for hiding this comment

strawgate Dec 4, 2023

Choose a reason for hiding this comment

amitkanfer Dec 5, 2023

Choose a reason for hiding this comment

mbudge commented Apr 17, 2024

nimarezainia commented Apr 18, 2024

kilfoyle commented Apr 18, 2024

amitkanfer commented Nov 19, 2023 •

edited

Loading