Skip to content

Commit

Permalink
feat(migration-tools): Include Migrator in container code, make it si…
Browse files Browse the repository at this point in the history 
…ngle use, use callbacks instead of taking a full loader (microsoft#23274)
  • Loading branch information
@ChumpChief
ChumpChief authored Dec 12, 2024
1 parent 02b2e94 commit 94d7f5e
Show file tree
Hide file tree
Showing 38 changed files with 1,130 additions and 1,363 deletions.
131 changes: 90 additions & 41 deletions examples/utils/migration-tools/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,94 +2,143 @@

This package contains tools for migrating data from one version to another, used by Fluid examples. They are not currently intended for use in production scenarios.

Use of the migration tools imposes several requirements on the container code and application, detailed here.
## `IMigrator`

## Implementing `IMigratableModel`
Migration is performed by using the `IMigrator`. The `IMigrator` can be inspected to discover the state of migration (`migrationState`), provides events to listen to as the state transitions, and has the method to kick off a migration (`proposeVersion()`).

Your data model must implement `IMigratableModel` to be migrated using the migration tools.
```ts
if (migrator.migrationState === "collaborating") {
migrator.proposeVersion("2.0");
}
```

This includes:
1. A `version` string to identify the model version.
1. Methods to export and import data, and to detect if the model supports a given data format:
1. `importData: (initialData: ImportType) => Promise<void>`
1. `exportData: () => Promise<ExportType>`
1. `supportsDataFormat: (initialData: unknown) => initialData is ImportType`
1. A `dispose` method to clean up the container - most likely calling `IContainer.dispose`.
To ensure no data is lost when moving between containers, you should stop making edits after the `"stopping"` event is raised. After this point, it's no longer guaranteed that changes will be included in the migration.

## Implementing the composite runtime pattern
```ts
migrator.events.on("stopping", () => {
// ...disable input in your UI
});
```

See documentation for the composite runtime pattern [here](./src/compositeRuntime/README.md).
Once the `"migrated"` event is raised, you can inspect the `migrationResult` property to find the result of the migration. If the container author used the `makeSeparateContainerMigrationCallback()` helper, this will contain the container ID of the new, migrated container.

The migration tools expect to find an `IMigratableModel` by accessing and calling a `getModel()` function provided on the `entryPoint`. They also expect to find an `IMigrationTool` by accessing a `migrationTool` member of the `entryPoint`. These requirements are most easily satisfied by using the composite runtime pattern.
```ts
migrator.events.on("migrated", () => {
console.log(`The new container ID is: ${migrator.migrationResult}`);
});
```

## Requirements for use

Accessing and using the `IMigrator` imposes several requirements on the container code and application, detailed in the following sections.

### Implementing the composite runtime pattern as the container code author

See documentation for the composite runtime pattern [here](./src/compositeRuntime/README.md).

`getModel()` is a function that takes an `IContainer` to aid in producing the `IMigratableModel`. This is because the contract of `IMigratableModel` likely requires functionality from `IContainer` (especially `IContainer.dispose()`).
The migrator is provided via the composite runtime pattern using the provided `makeMigratorEntryPointPiece()`. When using this tool,
the host will be able to access the `IMigrator` by calling `getMigrator()` on the container entryPoint (`container.getEntryPoint()`).

### Defining the entry point piece
#### Defining an example model entry point piece

```ts
const rootDatastoreAlias = "my-root-datastore";

export const getModelEntryPointPiece: IEntryPointPiece = {
name: "getModel",
name: "model", // This is the name that the host will find the root datastore under in the entrypoint
registryEntries: [MyRootDatastoreFactory.registryEntry],
onCreate: async (runtime: IContainerRuntime): Promise<void> => {
const rootDatastore = await runtime.createDataStore(MyRootDatastoreFactory.type);
await rootDatastore.trySetAlias(rootDatastoreAlias);
},
onLoad: async (runtime: IContainerRuntime): Promise<void> => {},
createPiece: async (runtime: IContainerRuntime): Promise<(container: IContainer) => Promise<FluidObject>> => {
createPiece: async (runtime: IContainerRuntime): Promise<FluidObject> => {
const entryPointHandle = await containerRuntime.getAliasedDataStoreEntryPoint(rootDatastoreAlias);

if (entryPointHandle === undefined) {
throw new Error(`Default dataStore [${rootDatastoreAlias}] must exist`);
}

// Entry points are typed as FluidObject and must be cast. Here we know it's a MyRootDatastore since
// we created it just above. Type validation can be added here if desired.
const rootDatastore = entryPointHandle.get() as Promise<MyRootDatastore>;
// MigratableAppModel (defined by the container code author) must implement IMigratableModel.
// Note that we're returning a function of type (container: IContainer) => Promise<FluidObject>,
// where the FluidObject is expected to be an IMigratableModel.
return async (container: IContainer) => new MigratableAppModel(rootDatastore, container);
return entryPointHandle.get();
},
};
```

#### Using `makeMigratorEntryPointPiece()`

```ts
// Entry points are typed as FluidObject and must be cast. Here we know it's a MyRootDatastore since
// we (the container code author) created it just above. Type validation can be added here if desired.
const getModelFromContainer = async <ModelType>(container: IContainer): Promise<ModelType> => {
const entryPoint = (await container.getEntryPoint()) as {
model: ModelType; // Note "model" matches up with the name we defined on the entry point piece above.
};

return entryPoint.model;
};

const exportDataCallback = async (container: IContainer): Promise<unknown> => {
const rootDatastore = await getModelFromContainer<MyRootDatastore>(container);
// This assumes that we've implemented an exportData() method on MyRootDatastore.
return rootDatastore.exportData();
};

// In the IRuntimeFactory
public async instantiateRuntime(
context: IContainerContext,
existing: boolean,
): Promise<IRuntime> {
const compositeEntryPoint = new CompositeEntryPoint();
compositeEntryPoint.addEntryPointPiece(getModelEntryPointPiece);
// migrationToolEntryPointPiece is provided by the migration-tools package
compositeEntryPoint.addEntryPointPiece(migrationToolEntryPointPiece);
// makeMigratorEntryPointPiece is provided by the migration-tools package
const migratorEntryPointPiece = makeMigratorEntryPointPiece(exportDataCallback);
compositeEntryPoint.addEntryPointPiece(migratorEntryPointPiece);
return loadCompositeRuntime(context, existing, compositeEntryPoint, this.runtimeOptions);
}
```

### `migrationToolEntryPointPiece`

This package additionally provides a `migrationToolEntryPointPiece` which is an off-the-shelf implementation of the piece to provide the `IMigrationTool`. With these provided pieces, you're only responsible for implementing the `IMigratableModel` piece with your data model.
### Calling `getMigrator()` as the host

## `Migrator`
The host must provide certain functionality to the migrator (as callback functions) that the container code author doesn't have access to or knowledge
about. In particular, these portions require access to the loader layer, and also knowledge about the future container code
that is being migrated to.

Finally, to actually execute the migration we provide the `Migrator` class. This takes a `SimpleLoader` (see below), the initially loaded model, migration tool, and container ID (TODO: can we simplify this handoff), as well as an optional `DataTransformationCallback` (see below). The migrator provides a collection of APIs to observe the state of the migration, as well as to acquire the new container after migration completes. (TODO: should the migrate() API also live here?)
The migration-tools package makes helper functions available to simplify creation of these callback functions in basic scenarios. Calling `getMigrator()` then returns an `IMigrator`.

TODO: Detail usage of the Migrator

### `SimpleLoader`

See documentation for `SimpleLoader` [here](./src/simpleLoader/README.md). `SimpleLoader` is used in place of a `Loader` and is used by the `Migrator`.

### Code loader
```ts
// makeCreateDetachedContainerCallback is provided by the migration-tools package
const createDetachedCallback = makeCreateDetachedContainerCallback(
loader,
createTinyliciousCreateNewRequest,
);

const importDataCallback: ImportDataCallback = async (
destinationContainer: IContainer,
exportedData: unknown,
) => {
const destinationModel = await getModelFromContainer<MyRootDatastore2>(destinationContainer);
// Note that if the data needs to be transformed from the old export format to some new import format,
// this is where it could be done.
// This assumes that we've implemented an importData() method on MyRootDatastore2.
await destinationModel.importData(exportedData);
};

To migrate between two different code versions, you must also provide a code loader to the `SimpleLoader` that is capable of loading those two respective code versions. This uses the usual `ICodeDetailsLoader` interface.
// makeSeparateContainerMigrationCallback is provided by the migration-tools package
const migrationCallback = makeSeparateContainerMigrationCallback(
createDetachedCallback,
importDataCallback,
);

const { getMigrator } = (await container.getEntryPoint()) as IMigratorEntryPoint;
const migrator: IMigrator = await getMigrator(
async () => loader.resolve({ url: id }),
migrationCallback,
);
```

### `DataTransformationCallback`
### Providing a code loader as the host

If your old and new code share an import/export format, you don't need a `DataTransformationCallback`. But if the import/export format has changed between versions, you can provide this callback to the `Migrator` and it will be called with the old exported data. This callback is responsible for transforming the data to the new format and returning the transformed data.
To migrate between two different code versions, the host must also provide a code loader that is capable of loading those two respective code versions. There is nothing new here, but if you've been statically loading your code (i.e. via `StaticCodeLoader`) you'll need to start performing real code loading.

<!-- AUTO-GENERATED-CONTENT:START (README_FOOTER) -->

Expand Down
158 changes: 32 additions & 126 deletions examples/utils/migration-tools/api-report/migration-tools.alpha.api.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,178 +4,84 @@
```ts

// @alpha (undocumented)
// @alpha
export class CompositeEntryPoint {
// (undocumented)
readonly addEntryPointPiece: (entryPointPiece: IEntryPointPiece) => void;
// (undocumented)
readonly onCreate: (runtime: IContainerRuntime) => Promise<void>;
// (undocumented)
readonly onLoad: (runtime: IContainerRuntime) => Promise<void>;
// (undocumented)
readonly provideEntryPoint: (runtime: IContainerRuntime) => Promise<Record<string, FluidObject>>;
// (undocumented)
get registryEntries(): NamedFluidDataStoreRegistryEntries;
}

// @alpha
export type DataTransformationCallback = (exportedData: unknown, modelVersion: string) => Promise<unknown>;
export type CreateDetachedContainerCallback = (version: string) => Promise<{
container: IContainer;
attach: () => Promise<string>;
}>;

// @alpha
export type ExportDataCallback = (sourceContainer: IContainer) => Promise<unknown>;

// @alpha
export interface IAcceptedMigrationDetails {
migrationSequenceNumber: number;
newVersion: string;
}

// @alpha (undocumented)
// @alpha
export interface IEntryPointPiece {
// (undocumented)
readonly createPiece: (runtime: IContainerRuntime) => Promise<FluidObject>;
// (undocumented)
readonly name: string;
// (undocumented)
readonly onCreate: (runtime: IContainerRuntime) => Promise<void>;
// (undocumented)
readonly onLoad: (runtime: IContainerRuntime) => Promise<void>;
// (undocumented)
readonly registryEntries: NamedFluidDataStoreRegistryEntries;
}

// @alpha
export interface IImportExportModel<ImportType, ExportType> {
exportData: () => Promise<ExportType>;
importData: (initialData: ImportType) => Promise<void>;
supportsDataFormat: (initialData: unknown) => initialData is ImportType;
}

// @alpha
export interface IMigratableModel extends IVersionedModel, IImportExportModel<unknown, unknown> {
dispose(): void;
}

// @alpha (undocumented)
export interface IMigrationTool {
export interface IMigrator {
readonly acceptedMigration: IAcceptedMigrationDetails | undefined;
completeMigrationTask(): void;
// (undocumented)
readonly connected: boolean;
// (undocumented)
readonly events: IEventProvider<IMigrationToolEvents>;
finalizeMigration(id: string): Promise<void>;
haveMigrationTask(): boolean;
readonly events: IEventProvider<IMigratorEvents>;
readonly migrationResult: unknown | undefined;
readonly migrationState: MigrationState;
readonly newContainerId: string | undefined;
readonly proposedVersion: string | undefined;
proposeVersion: (newVersion: string) => void;
volunteerForMigration(): Promise<boolean>;
}

// @alpha (undocumented)
export interface IMigrationToolEvents extends IEvent {
// (undocumented)
(event: "stopping" | "migrating" | "migrated", listener: () => void): any;
// (undocumented)
(event: "connected" | "disconnected", listener: () => void): any;
// (undocumented)
(event: "disposed", listener: () => void): any;
}

// @alpha (undocumented)
export interface IMigrator {
readonly currentModel: IMigratableModel;
readonly currentModelId: string;
// (undocumented)
readonly events: IEventProvider<IMigratorEvents>;
readonly migrationState: MigrationState;
// @alpha
export interface IMigratorEntryPoint {
getMigrator: (loadSourceContainerCallback: LoadSourceContainerCallback, migrationCallback: MigrationCallback) => Promise<IMigrator>;
}

// @alpha (undocumented)
// @alpha
export interface IMigratorEvents extends IEvent {
// (undocumented)
(event: "migrated" | "migrating", listener: () => void): any;
// (undocumented)
(event: "migrationNotSupported", listener: (version: string) => void): any;
}

// @alpha (undocumented)
export interface ISimpleLoader {
createDetached(version: string): Promise<{
container: IContainer;
attach: () => Promise<string>;
}>;
loadExisting(id: string): Promise<IContainer>;
supportsVersion(version: string): Promise<boolean>;
(event: "stopping" | "migrating" | "migrated", listener: () => void): void;
}

// @alpha
export interface IVersionedModel {
readonly version: string;
}
export type ImportDataCallback = (destinationContainer: IContainer, exportedData: unknown) => Promise<void>;

// @alpha
export const loadCompositeRuntime: (context: IContainerContext, existing: boolean, compositeEntryPoint: CompositeEntryPoint, runtimeOptions?: IContainerRuntimeOptions) => Promise<IContainerRuntime & IRuntime>;

// @alpha
export type MigrationState = "collaborating" | "stopping" | "migrating" | "migrated";
export type LoadSourceContainerCallback = () => Promise<IContainer>;

// @alpha (undocumented)
export const migrationToolEntryPointPiece: IEntryPointPiece;

// @alpha (undocumented)
export class MigrationToolFactory implements IFluidDataStoreFactory {
// (undocumented)
get IFluidDataStoreFactory(): IFluidDataStoreFactory;
// (undocumented)
instantiateDataStore(context: IFluidDataStoreContext, existing: boolean): Promise<IFluidDataStoreChannel>;
// (undocumented)
get type(): string;
}
// @alpha
export const makeCreateDetachedContainerCallback: (loaderProps: ILoaderProps, generateCreateNewRequest: () => IRequest) => CreateDetachedContainerCallback;

// @alpha
export class Migrator implements IMigrator {
constructor(simpleLoader: ISimpleLoader, initialMigratable: IMigratableModel, initialMigrationTool: IMigrationTool, initialId: string, dataTransformationCallback?: DataTransformationCallback | undefined);
// (undocumented)
get connected(): boolean;
// (undocumented)
get currentMigrationTool(): IMigrationTool;
// (undocumented)
get currentModel(): IMigratableModel;
// (undocumented)
get currentModelId(): string;
// (undocumented)
get events(): IEventProvider<IMigratorEvents>;
// (undocumented)
get migrationState(): MigrationState;
}
export const makeMigratorEntryPointPiece: (exportDataCallback: ExportDataCallback) => IEntryPointPiece;

// @alpha (undocumented)
export class SessionStorageSimpleLoader implements ISimpleLoader {
constructor(codeLoader: ICodeDetailsLoader, logger?: ITelemetryBaseLogger | undefined);
// (undocumented)
createDetached(version: string): Promise<{
container: IContainer;
attach: () => Promise<string>;
}>;
// (undocumented)
loadExisting(id: string): Promise<IContainer>;
// (undocumented)
supportsVersion(version: string): Promise<boolean>;
}
// @alpha
export const makeSeparateContainerMigrationCallback: (createDetachedContainerCallback: CreateDetachedContainerCallback, importDataCallback: ImportDataCallback) => MigrationCallback;

// @alpha (undocumented)
export class SimpleLoader implements ISimpleLoader {
constructor(props: Pick<ILoaderProps, "urlResolver" | "documentServiceFactory" | "codeLoader" | "logger"> & {
generateCreateNewRequest: () => IRequest;
});
// (undocumented)
createDetached(version: string): Promise<{
container: IContainer;
attach: () => Promise<string>;
}>;
// (undocumented)
loadExisting(id: string): Promise<IContainer>;
// (undocumented)
supportsVersion(version: string): Promise<boolean>;
}
// @alpha
export type MigrationCallback = (version: string, exportedData: unknown) => Promise<unknown>;

// @alpha
export type MigrationState = "collaborating" | "stopping" | "migrating" | "migrated";

// @alpha
export type SeparateContainerMigrationResult = string;

```
Loading

0 comments on commit 94d7f5e

Please sign in to comment.