Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filebeat setup error on machine-learning #19964

Closed
liza-mae opened this issue Jul 15, 2020 · 10 comments
Closed

Filebeat setup error on machine-learning #19964

liza-mae opened this issue Jul 15, 2020 · 10 comments
Assignees
@kvch
Labels
bug Filebeat Filebeat Team:Services (Deprecated) Label for the former Integrations-Services team

Comments

@liza-mae
Copy link

liza-mae commented Jul 15, 2020 •
edited
Loading

  • Version: 7.9.0 BC1
  • Operating System: Ubuntu 16
  • Steps to Reproduce:
    Install and run ES/Kibana.
    Install Filebeat.
    Run
./filebeat setup 
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/machine-learning/current/index.html
Exiting: 1 error: Error setting up ML for nginx_ecs: cannot set up ML with prefix: filebeat-nginx_ecs-access-

Not sure if this related to another issue I opened #12998 and it is now just throwing a different error in 7.9 or if it is a new error.
@liza-mae liza-mae added the Filebeat Filebeat label Jul 15, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 15, 2020
@liza-mae liza-mae added Team:Integrations Label for the Integrations team and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 15, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@liza-mae liza-mae added the bug label Jul 15, 2020
@liza-mae liza-mae mentioned this issue Jul 16, 2020
Closed
@kvch
Copy link
Contributor

kvch commented Jul 16, 2020

--setup has been removed from Beats 1.5 years ago: #10138

I have also downloaded the latest BC for 7.9, but the binary does not provide a --setup flag. Are you sure you are running 7.9?

@liza-mae
Copy link
Author

liza-mae commented Jul 16, 2020 •
edited
Loading

@kvch that was a typo in my description it's not filebeat --setup it's filebeat setup.

@kvch
Copy link
Contributor

kvch commented Jul 16, 2020

Have you modified the configuration files? If yes, could you please share it?
Are you connecting to an OSS version of ES and Kibana?

I have tried reproducing the issue multiple times with different configurations. So far no success.

@liza-mae
Copy link
Author

@kvch it is with security enabled, machine learning is not part of OSS maybe that is why it is not seen there? I have not tried it. The config only has my security settings, as follows:

output.elasticsearch:
  username: elastic
  password: changeme
  protocol: https
  hosts: ["10.0.2.15:9200"]
  ssl.certificate_authorities: /home/vagrant/filebeat-7.9.0-linux-x86_64/ca/ca.crt
  ssl.certificate: /home/vagrant/filebeat-7.9.0-linux-x86_64/node/node.crt
  ssl.key: /home/vagrant/filebeat-7.9.0-linux-x86_64/node/node.key
setup.kibana:
  host: "10.0.2.15:5601"
  protocol: https
  ssl.certificate_authorities: /home/vagrant/filebeat-7.9.0-linux-x86_64/ca/ca.crt
  ssl.certificate: /home/vagrant/filebeat-7.9.0-linux-x86_64/node/node.crt
  ssl.key: /home/vagrant/filebeat-7.9.0-linux-x86_64/node/node.key

@andresrc andresrc added Team:Services (Deprecated) Label for the former Integrations-Services team and removed Team:Integrations Label for the Integrations team labels Jul 17, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@liza-mae liza-mae mentioned this issue Jul 17, 2020
Closed
@kvch
Copy link
Contributor

kvch commented Jul 20, 2020 •
edited
Loading

I have retried with security enabled with both trial and basic licence. Still nothing. I also tested with OSS and not OSS version. No error at all.

Could you please share your debug logs?

@kvch
Copy link
Contributor

kvch commented Jul 27, 2020

@liza-mae What exactly did you do when got the error. The output you have pasted here seems inconsistent with the command you used. Did you run setup with only --machine-learning flag?

@kvch kvch mentioned this issue Jul 27, 2020
Merged
3 tasks
@liza-mae
Copy link
Author

liza-mae commented Jul 27, 2020 •
edited
Loading

@kvch I ran ./filebeat setup no additional flags

@kvch
Copy link
Contributor

kvch commented Jul 27, 2020

Thanks.

kvch added a commit that referenced this issue Jul 28, 2020
@kvch
Disable the option of running setup --machine-learning on its own (#2…

Verified

This commit was signed with the committer’s verified signature.
msfjarvis Harsh Shandilya
GPG key ID: 366D7BBAD1031E80
Verified
Learn about vigilant mode
ca7ce29 
…0241)

## What does this PR do?

This PR disables running `setup --machine-learning` on its own. From now on `setup --machine-learning --dashboards` has to be specified to enable ML jobs.

## Why is it important?

ML jobs require an index template pattern in Kibana. If it is not configured, loading the jobs fails. However, loading this asset is done by the dashboards step of setup.

Loading index template patterns are too tightly coupled with Kibana and dashboards and it needs refactoring. As this code is way too complex and going to be removed soon, I opted for a workaround instead of a fix.

## Related issues

Closes #19964
Closes #12998
Closes #20022
@kvch kvch closed this as completed Jul 28, 2020
@kvch kvch mentioned this issue Jul 28, 2020
Merged
3 tasks
kvch added a commit to kvch/beats that referenced this issue Jul 28, 2020
@kvch
Disable the option of running setup --machine-learning on its own (el…
69368de 
…astic#20241)

This PR disables running `setup --machine-learning` on its own. From now on `setup --machine-learning --dashboards` has to be specified to enable ML jobs.

ML jobs require an index template pattern in Kibana. If it is not configured, loading the jobs fails. However, loading this asset is done by the dashboards step of setup.

Loading index template patterns are too tightly coupled with Kibana and dashboards and it needs refactoring. As this code is way too complex and going to be removed soon, I opted for a workaround instead of a fix.

Closes elastic#19964
Closes elastic#12998
Closes elastic#20022
(cherry picked from commit ca7ce29)
kvch added a commit that referenced this issue Jul 29, 2020
@kvch
Disable the option of running setup --machine-learning on its own (#2…
91b84e3 
…0241) (#20266)

This PR disables running `setup --machine-learning` on its own. From now on `setup --machine-learning --dashboards` has to be specified to enable ML jobs.

ML jobs require an index template pattern in Kibana. If it is not configured, loading the jobs fails. However, loading this asset is done by the dashboards step of setup.

Loading index template patterns are too tightly coupled with Kibana and dashboards and it needs refactoring. As this code is way too complex and going to be removed soon, I opted for a workaround instead of a fix.

Closes #19964
Closes #12998
Closes #20022
(cherry picked from commit ca7ce29)
leweafan pushed a commit to leweafan/beats that referenced this issue Apr 28, 2023
@kvch
Disable the option of running setup --machine-learning on its own (el…
4d3b96b 
…astic#20241) (elastic#20266)

This PR disables running `setup --machine-learning` on its own. From now on `setup --machine-learning --dashboards` has to be specified to enable ML jobs.

ML jobs require an index template pattern in Kibana. If it is not configured, loading the jobs fails. However, loading this asset is done by the dashboards step of setup.

Loading index template patterns are too tightly coupled with Kibana and dashboards and it needs refactoring. As this code is way too complex and going to be removed soon, I opted for a workaround instead of a fix.

Closes elastic#19964
Closes elastic#12998
Closes elastic#20022
(cherry picked from commit 4ebf884)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kvch kvch

Labels
bug Filebeat Filebeat Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@andresrc @kvch @elasticmachine @liza-mae