Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Container Rehydration from a snapshot taken from detached container. #3061

Closed
wants to merge 39 commits into from
Closed
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
59f1e16
final changes for rehydration
jatgarg Aug 4, 2020
4bffb77
final changes for rehydration
jatgarg Aug 5, 2020
5573ffa
merge conflict
jatgarg Aug 5, 2020
59d66b5
mertge conflict
jatgarg Aug 5, 2020
d240a3a
remove
jatgarg Aug 5, 2020
f99f68e
Merge branch 'master' of https://github.com/microsoft/FluidFramework …
jatgarg Aug 5, 2020
217aacc
fix tiny
jatgarg Aug 5, 2020
1bb9da0
remove unloaded map
jatgarg Aug 5, 2020
6fcce12
remove unrealized map
jatgarg Aug 5, 2020
3f903b7
add comment
jatgarg Aug 5, 2020
e255f54
remove loadlocal
jatgarg Aug 5, 2020
59a4ff2
pr sugg
jatgarg Aug 5, 2020
737111d
pr sugg
jatgarg Aug 5, 2020
cfde78c
pr sugg
jatgarg Aug 5, 2020
0a38dff
merge conflict
jatgarg Aug 5, 2020
acc535a
fix name
jatgarg Aug 5, 2020
918a5c4
merge conflict
jatgarg Aug 6, 2020
84bc42e
pr sugg
jatgarg Aug 6, 2020
808d530
merge conflict
jatgarg Aug 12, 2020
7150af4
separate the 2 apis
jatgarg Aug 12, 2020
60f977f
resolve m,erge
jatgarg Aug 12, 2020
4bd5d7e
merge conflict
jatgarg Aug 13, 2020
f93b1df
set existing as true if loaded from rehydration snapshopt
jatgarg Aug 17, 2020
79592b1
set existing as true if loaded from rehydration snapshopt
jatgarg Aug 17, 2020
2876424
merge conflict
jatgarg Aug 17, 2020
acd860d
remove
jatgarg Aug 17, 2020
e52d2ac
remove
jatgarg Aug 17, 2020
3f46d22
localdatastore
jatgarg Aug 17, 2020
a19116f
rename comp toi datta store
jatgarg Aug 19, 2020
eae3ec6
merge conflict
jatgarg Aug 19, 2020
562b29b
merge conflict
jatgarg Aug 19, 2020
55d76e2
cast
jatgarg Aug 20, 2020
d18fd5f
cast
jatgarg Aug 20, 2020
0525797
merge conflict
jatgarg Aug 24, 2020
7c3e821
merge conflict
jatgarg Aug 28, 2020
c395773
pr sugg
jatgarg Aug 28, 2020
cf97acd
merge conflict
jatgarg Sep 4, 2020
a1a07a8
pr sugg
jatgarg Sep 8, 2020
545f304
merge conflict
jatgarg Sep 8, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/dds/cell/src/cell.ts
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ export class SharedCell<T extends Serializable = any> extends SharedObject<IShar
* @returns - promise that resolved when the load is completed
*/
protected async loadCore(
branchId: string,
branchId: string | undefined,
jatgarg marked this conversation as resolved.
Show resolved Hide resolved
jatgarg marked this conversation as resolved.
Show resolved Hide resolved
storage: IChannelStorageService): Promise<void> {
const rawContent = await storage.read(snapshotFileName);

Expand Down
2 changes: 1 addition & 1 deletion packages/dds/cell/src/cellFactory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ export class CellFactory implements IChannelFactory {
runtime: IFluidDataStoreRuntime,
id: string,
services: IChannelServices,
branchId: string,
branchId: string | undefined,
attributes: IChannelAttributes): Promise<ISharedCell> {
const cell = new SharedCell(id, runtime, attributes);
await cell.load(branchId, services);
Expand Down
2 changes: 1 addition & 1 deletion packages/dds/counter/src/counter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ export class SharedCounter extends SharedObject<ISharedCounterEvents> implements
* @returns - promise that resolved when the load is completed
*/
protected async loadCore(
branchId: string,
branchId: string | undefined,
storage: IChannelStorageService): Promise<void> {
const rawContent = await storage.read(snapshotFileName);

Expand Down
2 changes: 1 addition & 1 deletion packages/dds/ink/src/ink.ts
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ export class Ink extends SharedObject<IInkEvents> implements IInk {
* {@inheritDoc @fluidframework/shared-object-base#SharedObject.loadCore}
*/
protected async loadCore(
branchId: string,
branchId: string | undefined,
storage: IChannelStorageService,
): Promise<void> {
const header = await storage.read(snapshotFileName);
Expand Down
2 changes: 1 addition & 1 deletion packages/dds/map/src/directory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -653,7 +653,7 @@ export class SharedDirectory extends SharedObject<ISharedDirectoryEvents> implem
* {@inheritDoc @fluidframework/shared-object-base#SharedObject.loadCore}
*/
protected async loadCore(
branchId: string,
branchId: string | undefined,
storage: IChannelStorageService) {
const header = await storage.read(snapshotFileName);
const data = JSON.parse(fromBase64ToUtf8(header));
Expand Down
2 changes: 1 addition & 1 deletion packages/dds/map/src/map.ts
Original file line number Diff line number Diff line change
Expand Up @@ -333,7 +333,7 @@ export class SharedMap extends SharedObject<ISharedMapEvents> implements IShared
* {@inheritDoc @fluidframework/shared-object-base#SharedObject.loadCore}
*/
protected async loadCore(
branchId: string,
branchId: string | undefined,
storage: IChannelStorageService) {
const header = await storage.read(snapshotFileName);

Expand Down
2 changes: 1 addition & 1 deletion packages/dds/matrix/src/matrix.ts
Original file line number Diff line number Diff line change
Expand Up @@ -551,7 +551,7 @@ export class SharedMatrix<T extends Serializable = Serializable>
debug(`${this.id} is now disconnected`);
}

protected async loadCore(branchId: string, storage: IChannelStorageService) {
protected async loadCore(branchId: string | undefined, storage: IChannelStorageService) {
try {
await this.rows.load(this.runtime, new ObjectStoragePartition(storage, SnapshotPath.rows), branchId);
await this.cols.load(this.runtime, new ObjectStoragePartition(storage, SnapshotPath.cols), branchId);
Expand Down
1 change: 1 addition & 0 deletions packages/dds/merge-tree/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
"dependencies": {
"@fluidframework/common-definitions": "^0.18.1",
"@fluidframework/common-utils": "^0.22.1",
"@fluidframework/container-definitions": "^0.26.0",
"@fluidframework/core-interfaces": "^0.26.0",
"@fluidframework/datastore-definitions": "^0.26.0",
"@fluidframework/protocol-definitions": "^0.1011.1",
Expand Down
38 changes: 20 additions & 18 deletions packages/dds/merge-tree/src/snapshotLoader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { ChildLogger } from "@fluidframework/telemetry-utils";
import { ISequencedDocumentMessage } from "@fluidframework/protocol-definitions";
import { IFluidDataStoreRuntime, IChannelStorageService } from "@fluidframework/datastore-definitions";
import { ITelemetryLogger } from "@fluidframework/common-definitions";
import { AttachState } from "@fluidframework/container-definitions";
jatgarg marked this conversation as resolved.
Show resolved Hide resolved
import { Client } from "./client";
import { NonCollabClient, UniversalSequenceNumber } from "./constants";
import { ISegment, MergeTree } from "./mergeTree";
Expand Down Expand Up @@ -134,24 +135,25 @@ export class SnapshotLoader {
if (chunk.headerMetadata === undefined) {
throw new Error("header metadata not available");
}
// specify a default client id, "snapshot" here as we
// should enter collaboration/op sending mode if we load
// a snapshot in any case (summary or attach message)
// once we get a client id this will be called with that
// clientId in the connected event
// TODO: this won't support rehydrating a detached container
// we need to think more holistically about the dds state machine
// now that we differentiate attached vs local
this.client.startOrUpdateCollaboration(
this.runtime.clientId ?? "snapshot",

// TODO: Make 'minSeq' non-optional once the new snapshot format becomes the default?
// (See https://github.com/microsoft/FluidFramework/issues/84)
/* minSeq: */ chunk.headerMetadata.minSequenceNumber !== undefined
? chunk.headerMetadata.minSequenceNumber
: chunk.headerMetadata.sequenceNumber,
/* currentSeq: */ chunk.headerMetadata.sequenceNumber,
branching);
// If we load a detached container from snapshot, then we don't supply a default clientId
// because we don't want to start collaboration.
if (this.runtime.attachState !== AttachState.Detached) {
// specify a default client id, "snapshot" here as we
// should enter collaboration/op sending mode if we load
// a snapshot in any case (summary or attach message)
// once we get a client id this will be called with that
// clientId in the connected event
this.client.startOrUpdateCollaboration(
this.runtime.clientId ?? "snapshot",

// TODO: Make 'minSeq' non-optional once the new snapshot format becomes the default?
// (See https://github.com/microsoft/FluidFramework/issues/84)
/* minSeq: */ chunk.headerMetadata.minSequenceNumber !== undefined
? chunk.headerMetadata.minSequenceNumber
: chunk.headerMetadata.sequenceNumber,
/* currentSeq: */ chunk.headerMetadata.sequenceNumber,
branching);
}

return chunk;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ export class ConsensusOrderedCollection<T = any>
}

protected async loadCore(
branchId: string,
branchId: string | undefined,
storage: IChannelStorageService): Promise<void> {
assert(this.jobTracking.size === 0);
const rawContentTracking = await storage.read(snapshotFileNameTracking);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ export class ConsensusRegisterCollection<T>
}

protected async loadCore(
branchId: string,
branchId: string | undefined,
storage: IChannelStorageService,
): Promise<void> {
const header = await storage.read(snapshotFileName);
Expand Down
2 changes: 1 addition & 1 deletion packages/dds/sequence/src/sequence.ts
Original file line number Diff line number Diff line change
Expand Up @@ -487,7 +487,7 @@ export abstract class SharedSegmentSequence<T extends MergeTree.ISegment>
}

protected async loadCore(
branchId: string,
branchId: string | undefined,
storage: IChannelStorageService) {
const header = await storage.read(snapshotFileName);

Expand Down
2 changes: 1 addition & 1 deletion packages/dds/sequence/src/sharedIntervalCollection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ export class SharedIntervalCollection<TInterval extends ISerializableInterval =
}

protected async loadCore(
branchId: string,
branchId: string | undefined,
storage: IChannelStorageService) {
const header = await storage.read(snapshotFileName);

Expand Down
16 changes: 10 additions & 6 deletions packages/dds/shared-object-base/src/sharedObject.ts
Original file line number Diff line number Diff line change
Expand Up @@ -130,14 +130,18 @@ export abstract class SharedObject<TEvent extends ISharedObjectEvents = ISharedO
* @param services - Services used by the shared object
*/
public async load(
branchId: string,
services: IChannelServices): Promise<void> {
this.services = services;

branchId: string | undefined,
services: IChannelServices,
): Promise<void> {
if (this.runtime.attachState !== AttachState.Detached) {
this.services = services;
}
await this.loadCore(
branchId,
services.objectStorage);
this.attachDeltaHandler();
if (this.runtime.attachState !== AttachState.Detached) {
this.attachDeltaHandler();
}
}

/**
Expand Down Expand Up @@ -200,7 +204,7 @@ export abstract class SharedObject<TEvent extends ISharedObjectEvents = ISharedO
* @param services - Storage used by the shared object
*/
protected abstract loadCore(
branchId: string,
branchId: string | undefined,
services: IChannelStorageService): Promise<void>;

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ export class SharedSummaryBlock extends SharedObject implements ISharedSummaryBl
* {@inheritDoc @fluidframework/shared-object-base#SharedObject.loadCore}
*/
protected async loadCore(
branchId: string,
branchId: string | undefined,
storage: IChannelStorageService): Promise<void> {
const rawContent = await storage.read(snapshotFileName);
const contents = JSON.parse(fromBase64ToUtf8(rawContent)) as ISharedSummaryBlockDataSerializable;
Expand Down
9 changes: 8 additions & 1 deletion packages/loader/container-definitions/src/loader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import {
IDocumentMessage,
IQuorum,
ISequencedDocumentMessage,
ISnapshotTree,
} from "@fluidframework/protocol-definitions";
import { IResolvedUrl } from "@fluidframework/driver-definitions";
import { IEvent, IEventProvider } from "@fluidframework/common-definitions";
Expand Down Expand Up @@ -148,7 +149,13 @@ export interface ILoader {
* Creates a new container using the specified chaincode but in an unattached state. While unattached all
* updates will only be local until the user explicitly attaches the container to a service provider.
*/
createDetachedContainer(source: IFluidCodeDetails): Promise<IContainer>;
createDetachedContainer(codeDetails: IFluidCodeDetails): Promise<IContainer>;

/**
* Creates a new container using the specified snapshot but in an unattached state. While unattached all
* updates will only be local until the user explicitly attaches the container to a service provider.
*/
rehydrateDetachedContainerFromSnapshot(snapshot: ISnapshotTree): Promise<IContainer>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rehydrateDetachedContainerFromSnapshot [](start = 4, length = 38)

nit: Consider rehydrateContainer or rehydrateContainerFromSnapshot. I don't think we should expose detached keyword here.

Not related to this PR, but I think createDetachedContainer should also be just called createContainer. Now that detached container is (almost) used everywhere, the word "detached" doesn't add any extra value.

}

export enum LoaderHeader {
Expand Down
2 changes: 1 addition & 1 deletion packages/loader/container-definitions/src/runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ export interface IContainerContext extends IMessageScheduler, IDisposable {
readonly storage: IDocumentStorageService | undefined | null;
readonly connected: boolean;
readonly branch: string;
readonly baseSnapshot: ISnapshotTree | null;
readonly baseSnapshot: ISnapshotTree | undefined;
readonly submitFn: (type: MessageType, contents: any, batch: boolean, appData?: any) => number;
readonly submitSignalFn: (contents: any) => void;
readonly snapshotFn: (message: string) => Promise<void>;
Expand Down
Loading