Add Elasticsearch Clients as non-Elastic Agent integrations (MVP)

[alexfrancoeur]

Updated with 7.16 deliverables on 2021-09-28

We would like to unify all ingestion capabilities from a single integrations page (#93084). The approach we are taking will include new cards in the integrations view that will navigate directly to other areas in Kibana's UI (sample data, file upload, beats modules, etc.). Each integration will be as searchable, filterable and discoverable as the Elastic Agent ones. This also allows us to simplify the ingestion call to action across the board.

It is worth noting to support Elasticsearch language clients, we play to add a "deployment details" pop over (#113285) to simplify the experience in this MVP, with more curated experiences to be delivered in the future.

For 7.16, we are keeping things simple and plan to add / update the following.

• Titles and descriptions for integrations cards for each language. This will adhere to content guidelines that we plan to finalize soon.
• URL routes to the best onboarding documentation for each language client. Since these are external, they should probably open a new tab
• Documentation improvements focused on onboarding if needed and should call out the new "deployment details" pop-over for the Kibana and Cloud onboarding experiences

Open questions

• What is the most appropriate documentation for each language?
  • Currently https://www.elastic.co/guide/en/elasticsearch/client/[CLIENT]/current/introduction.html
• What can be updated to improve this experience?
  • Cloud has put together a few examples for using clients on Cloud, can / should we borrow some of these ideas? (example 1, example 2)
  • Would connect a client documentation be better with deep linking?

cc: @snide @VijayDoshi @mostlyjason @philkra @sethpayne @linyaru @technige @gchaps @mriley @dborodyansky

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ezimuel]

@alexfrancoeur and @bytebilly I really like the idea of having a "add data tutorial" for each language.
I have two questions about this approach:

• how we can pass the host and the CloudID in the example? Are you using a template system (you mentioned a YAML implementation)?
• which data set do we want to use as example for ingestion?

[joshdover]

I think this request generally makes sense, and we'd likely want a top-level category in the integrations UI for language clients.

Timing wise, we would like to target this work for 7.last. Language clients aren't necessarily part of the requirements today, primarily because we do not provide them in Kibana now, but it'd be a great addition. I think we can treat it as a nice to have for the unified integrations view, but if it's low effort I believe we can provide a better experience for our net new users.

At this point, the Fleet team is targeting 8.x for delivering the unified integrations view, so I just want to set clear expectations here that this very likely will not be shipping in 7.x given the team's current roadmap. We are planning to work on the technical definition in the coming months and I think we should include this request in our plans for this UI.

[bytebilly]

I'd suggest to create a single integration for Language Clients, and use a dropdown or something similar to switch between different languages and automatically update instructions.

This is more aligned with the long term view of having Apps as the first-class entity (rather than the language client itself), and may remove some complexity of grouping multiple integrations in a good way.

[bytebilly]

which data set do we want to use as example for ingestion?

@ezimuel since language clients are very generic, I don't think we should do something very specific here.
Ingestion of a simple doc with a few fields could be enough and fit well into a textbox.
See https://www.elastic.co/guide/en/cloud/current/ec-getting-started-python.html#ec_ingest_data_2 or something along those lines.

[mostlyjason]

@bytebilly we've put some work recently into supporting integrations that can surface multiple tiles in the integrations browse tab. This is to provide better discoverability for different keywords and use cases. We believe some users will search keywords with the name of the system they want to integrate with. An example of that is AWS which surfaces tiles for EC2, RDS, etc. I'm also talking to the APM team about providing tiles for each language. For example if the user searches "Python" they might see tiles for APM agents and ES language clients for Python. I'd be happy to tell you more about that capability.

Also, in the first release of the unified integrations view we can only support a direct link from the tile to another URL. It could be a link to docs or an internal link to a page created by your team. It looks like @alexfrancoeur is suggesting an add data tutorial page. I think that could work as a near term option but I imagine we plan to deprecate those later in 8.x? The integrations app has the capability to generate tutorial pages from YAML and even embed custom react components. It would require some effort to support non-fleet integrations, but it might provide a cleaner experience and have a longer shelf life.

[bytebilly]

Thanks @mostlyjason, I'm glad to see that we are iterating so fast on this project!
I like the idea of discoverability by keyword, is that something achievable with a single-tile integration by using multiple tags, or is it based on the display name only?

Do you have a preview environment to play with, or some animation of the multi-tile AWS integration and search flow? 🙂

@philkra and @sethpayne will define the best next steps and the involvement of the Clients team.

[mostlyjason]

I just met with @philkra and we discussed several options to add language clients. I sounded like his preference for the first phase is adding a tile for language clients overall along with tiles for each language. They would link to the documentation. In a later phase, we'd add in-product tutorials pages. He's going to add the content to our tracking spreadsheet.

@bytebilly you see the multi-tile AWS integration in any 7.15 environment. There is a main tile for AWS and "virtual" tiles for each AWS service like EC2, RDS, etc.

[alexfrancoeur]

After speaking with @snide, we've added this as a requirement for #111374. Keeping scope minimal, let's focus on a link to external documentation and a copy end point CTA in the integrations view.

cc: @mostlyjason @thomasneirynck @dborodyansky

[alexfrancoeur]

@bytebilly @sethpayne @philkra we are tracking the progress of this as part of the required work for the next release. Minimally, we will plan to make language clients discoverable in the integrations UI, link out to the docs and a UX for copying the elasticsearch endpoing and / or cloud ID.

We'll likely need some input on titles, descriptions, doc endpoint and potentially some updates to the documentation. Who would be the best person to work with?

[philkra]

We'll likely need some input on titles, descriptions, doc endpoint and potentially some updates to the documentation. Who would be the best person to work with?

@alexfrancoeur that would be me. The individual links to the docs are already in the tracking spreadsheet. Can you point me to the best person to assist with:

a UX for copying the elasticsearch endpoing and / or cloud ID.

Pinging @technige for awareness.

[thomasneirynck]

just for awareness, so the issues are linked. This PR #112481 is adding API to actually do this addition. When it merges, the Elasticsearch-clients can be added as a non-Elastic Agent integrations in a follow-on PR.

[alexfrancoeur]

Great, thanks @philkra! We're working to finalize title & description content guidelines so we may ping you for some feedback / help.

For the elasticsearch endpoint (endpoing 🤦 ) and cloud ID, @dborodyansky is going to working through some designs for how we may generically provide this capability in 7.16. It might be worth updating client docs to include this step as part of onboarding. Ideally, this would be a quick one stop shop for your elasticsearch endpoint and cloud ID when you first spin up the stack.

[alexfrancoeur]

@philkra @clintandrewhall @joshdover @technige @sethpayne @thomasneirynck @mostlyjason I updated the description of this issue and opened #113285 based on our discussion today. Please feel free to add anything I may have missed.

@linyaru I know you've received positive feedback on the Cloud documentation for language clients, we'd love to hear more about that process and ideas for improving onboarding documentation

For the title and description line item, I can share some guidelines in the next couple of days. For the documentation, we need to make sure we have the endpoints decided and can iterate as needed before and after FF.