Troubleshoot common error "apm-$version-$type' exists, but it is not an alias"

[graphaelli]

Connection marked as failed because the onConnect callback failed: resource 'apm-7.6.2-metric' exists, but it is not an alias

This appears to be related to removing the ILM write alias - perhaps APM Server can detect this. At least, there should be documentation for how to recover manually for now.

[graphaelli]

Confirmed this is reproduceable with DELETE apm-*-metric* (or any type) while apm-server is running. apm-server will then do bulk writes to what it thinks is a write alias but instead will auto_create_index a real index where the write alias used to be. Writes will continue to succeed until apm-server reestablishes a connection with Elasticsearch, like after a restart. At that point the connection callback fails with exists, but it is not an alias.

That's it for troubleshooting from me. Happy to discuss solution considerations with anyone that picks this up.

[graphaelli]

For those coming across this issue, the recovery procedure described by @simitt works great and can be used on cloud as the stopping ingestion step is not required. For example, assuming someone deleted apm-7.7.0-transaction with DELETE apm*transaction*.

1. Block writes to the index:

PUT apm-7.7.0-transaction/_settings
{
  "settings": {
    "index.blocks.write": true
  }
}

This should also cause any APM Servers still writing to the underlying index to stop ingestion and allow the rest of the process to complete

2. [optional] Clone the index to retain data

POST apm-7.7.0-transaction/_clone/apm-7.7.0-transaction-original

and wait for clone to complete - stage: done

GET _cat/recovery/apm*transaction*?s=index&v=true&h=index,stage

3. Delete the index that should be a write alias

DELETE apm-7.7.0-transaction

4. Confirm an APM Server recreated the write alias on the next connection attempt:

GET _cat/aliases/apm*transaction*?s=index&v=true&h=alias,index,is_write_index

should return:

alias                 index                        is_write_index
apm-7.7.0-transaction apm-7.7.0-transaction-000001 true

[graphaelli]

Datastreams should solve this problem, removing the aliasing issues altogether. Let's resolve this issue by documenting these steps under common problems.

[axw]

I wonder if we should disable auto_create_index for APM indices when ILM is enabled, as a bandaid solution? Probably depends on when we expect to have datastreams support implemented.

[simitt]

That's definitly worth looking into it. We would need to set the auto_create_index accordingly when users switch between unmanaged and managed indices.
Additionally, even when using ILM we do have indices that are not managed (sourcemap, onboarding, apm fallback index). We could either ensure to manage all of the created indices and then apply the setting simply for apm* or take the concrete index prefixes into account and ensure the auto creation is still respected for unmanaged indices. I can take a stab on that.

[graphaelli]

Closing this out as docs are up now.