[DOC] Segment replication changes with remote store GA and 2.10 release

[dreamer-89]

What do you want to do?

• Request a change to existing documentation
• Add new documentation
• Report a technical problem with the documentation
• Other

Tell us about your request.
Created this issue to track documentation changes GA release of remote store feature which works with only SEGMENT replication strategy and other core changes related to SEGMENT replication. With introduction of remote store, SEGMENT replication is available in two flavours.

1. With remote store. Primary uploads segments to remote store, replica downloads from remote store.
2. Without remote store. Replica shard copies directly syncs segments from primary shard via node-node communication.

Existing segment replication documentation is applicable only for 2 above and thus needs amendments to incorporate remote store availability. Below section shows the changes and new documentation that can be added/udpated.

Change request

Considerations

Update segment replication considerations documentation

1. Integration with remote-backed storage as the source of replication is currently not supported. --> Delete this as remote backed storage as replication source is now supported
2. Segment replication leads to increased network congestion on primary shards. --> This is only applicable for node-node communication. With remote store, primary shard uploads on remote store while replica copies downloads directly from remote store.
3. Read-after-write guarantees: The wait_until refresh policy is not compatible with segment replication. If you use the wait_until refresh policy while ingesting documents, you’ll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. We are exploring other mechanisms for providing read-after-write guarantees. For more information, see the corresponding GitHub issue. --> Read after write (RAW) guarantees are now supported using get/mget or using _primary shard preference based search. Related github issue [DISCUSS - Segment Replication] SegRep consistency limitations OpenSearch#8700, [DISCUSS] Add back preference for searching _primaries or _replicas OpenSearch#6046
4. System indexes will continue to use document replication internally until read-after-write guarantees are available. In this case, document replication does not hinder the overall performance because there are few system indexes. --> System indices are now supported with segment replication.

Configurations

Update segment replication configuration documentation

1. In segment replication, the primary shard is usually generating more network traffic than the replicas because it copies segment files to the replicas. Thus, it’s beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic cluster.routing.allocation.balance.prefer_primary setting to true. --> This is applicable only for node-node communication.
2. This setting is not applied to system indexes and hidden indexes. By default, all system and hidden indexes in OpenSearch will still use document replication even if this setting is enabled. --> Delete this note, as this is not true anymore.

Add new documentation

Add references to remote store feature in segment replication documentation as alternate mechanism for segment syncs. Along with existing example on index creation, we can

1. Add another example which shows remote store related settings used during index creation.
2. (Better) Refer to existing remote store documentation which already contains create index example

PUT /my-index1
{
  "settings": {
    "index": {
      "replication.type": "SEGMENT" 
    }
  }
}

OS Version. 2.10.0+

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.

[Naarcha-AWS]

@dreamer-89: Is this in addition to or a duplicate of #4875, #4984, and #4405?

[dreamer-89]

@dreamer-89: Is this in addition to or a duplicate of #4875, #4984, and #4405?

Thanks @Naarcha-AWS for the update. Yes, #4405 seems to be duplicate (parent) of this issue. We can use details/changes mentioned in this issue and close this issue in favour of #4405. The other issues #4875 - seems specific to remote store GA release while #4984 is about adding new consideration for get/mget, term-vector APIs.