GPL-892 Look at gains from using centralized ELK logging from FCE (C=M,V=2*)

[rl15]

Description
Look at gains from using centralized ELK logging from FCE

Who the primary contacts are for this work
Brett H (ICT)

Additional context or information
This backlog item was raised in response to presentation to ICT town hall on 2nd February - see Rich L, (Friday, 12 February 2021 at 12:22) Subject: FW: Recording was Re: Todays Town Hall

Brett H wrote (Monday, 15 February 2021 at 09:25) Abridged

See; https://ssg-confluence.internal.sanger.ac.uk/display/OPENSTACK/Using+ELK+to+centralise+logging

Gains

1. remove duplication of effort and quota, but we are
2. Sanger looking at getting a Platinum licence for this deployment which would give us new features: https://www.elastic.co/subscriptions

Costs

The centralised elk does have the disadvantage that we can't give the manage_index_templates permission to users (because it's per-cluster, so you could theoretically change anyone's templates), so if you have frequently changing templates, it may not be the best answer.

I did report this to elastic last year, but it's had very little movement: elastic/elasticsearch#53110

Questions

• How do you use your ELK stack?
• Beats, syslog, individual index access?
• If you would like an account, let me know the name and email you want associated with it and I'll set it up for you.

[KatyTaylor]

Find out if there are limits on the amount of logging allowed - we are worried we might flood it

[ashaith]

We cannot give manage_index_templates role to the user coz it has very broad access (like template delete privileges). Our requirement is the user needs to list the available template using "indices:admin/template/get".