[Remote Store] Support refresh level durability

[sachinpkale]

Is your feature request related to a problem? Please describe.

• As part of remote store feature that was introduced in OpenSearch 2.3, segments are uploaded to remote store post each refresh.
• This means remote segment store contains segments that were part of last refresh.
• But the _remotestore/_restore API only supports commit level durability.
• This feature will add refresh level durability support to the restore API.

[sachinpkale]

Currently, I am thinking of using the same flow that segment replication uses when replica receives the publish checkpoint.

OpenSearch/server/src/main/java/org/opensearch/index/store/Store.java

Lines 869 to 884 in b43e985

	public void commitSegmentInfos(SegmentInfos latestSegmentInfos, long maxSeqNo, long processedCheckpoint) throws IOException {
	assert indexSettings.isSegRepEnabled();
	metadataLock.writeLock().lock();
	try {
	final Map<String, String> userData = new HashMap<>(latestSegmentInfos.getUserData());
	userData.put(LOCAL_CHECKPOINT_KEY, String.valueOf(processedCheckpoint));
	userData.put(SequenceNumbers.MAX_SEQ_NO, Long.toString(maxSeqNo));
	latestSegmentInfos.setUserData(userData, true);
	latestSegmentInfos.commit(directory());
	directory.sync(latestSegmentInfos.files(true));
	directory.syncMetaData();
	cleanupAndPreserveLatestCommitPoint("After commit", latestSegmentInfos);
	} finally {
	metadataLock.writeLock().unlock();
	}
	}

[mch2]

@sachinpkale Why do we need to support refresh level durability with node-node replication? Could remote store use commit level durability until we implement remote store as the source of replication for segrep? There is a lot of overlap here with that effort.

My concern here is that remote store now is coupled to segrep's node-node implementation, and this will add further network load on primaries that have to copy to remote store & all replicas. I think this is ok to do if we are using it as the first half of using remote store as a src of replication, but doesn't make much sense with node-node.

note also on the commitSegmentInfos method you linked above - We can skip this commit on replicas entirely if those files are already in remote store & we can recover from it.

Edit -Added more context.