Skip to content

Commit

Permalink
Add the Fix for MongoPoolClearedError (#15764)
Browse files Browse the repository at this point in the history
Recently, we had a MongoPoolClearedError. That means the Mongo pool
would be destroyed and cleared whenever the scribe crashes off.
![Screenshot 2023-05-30 at 8 18 04
PM](https://github.com/microsoft/FluidFramework/assets/66701969/36169b1e-0722-4610-ba81-e91e0c1f9ea0)

To prevent this issue, we runWithRetry with 6 times for 30 seconds
intervals to read the document from the document repository. Therefore,
once the scribe pod is resumed, we can connect to the document
collection again.
  • Loading branch information
tianzhu007 authored May 31, 2023
1 parent 3e95c94 commit f634660
Showing 1 changed file with 10 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import {
ITenantManager,
LambdaName,
MongoManager,
runWithRetry,
} from "@fluidframework/server-services-core";
import {
IDocumentSystemMessage,
Expand Down Expand Up @@ -112,7 +113,15 @@ export class ScribeLambdaFactory
const lumberProperties = getLumberBaseProperties(documentId, tenantId);

try {
document = await this.documentRepository.readOne({ documentId, tenantId });
document = (await runWithRetry(
async () => this.documentRepository.readOne({ documentId, tenantId }),
"readIDocumentInScribeLambdaFactory",
3 /* maxRetries */,
1000 /* retryAfterMs */,
lumberProperties,
undefined /* shouldIgnoreError */,
(error) => true /* shouldRetry */,
)) as IDocument;

if (!isDocumentValid(document)) {
// Document sessions can be joined (via Alfred) after a document is functionally deleted.
Expand Down

0 comments on commit f634660

Please sign in to comment.