Skip to content

Commit

Permalink
fix: Scheduling the heavy database ops on bounded elastic thread pool…
Browse files Browse the repository at this point in the history
… for Ms-sql (#39176)

## Description
Production is seeing high number of pod restarts causing downtime. It
seems to be correlated with user MS-SQL query triggers. On the
hypothesis that its indeed the plugin executions causing this, it could
be the datasource create taking up redis thread pool (possibly because
of an upstream scheduling of a task on it)

Sample log lines displaying the behaviour (only user email removed from
the logs) :
traceId=bda95acf0e182c48037214d5173d0ac9 spanId=00fdca579a517b66 -
boundedElastic-13: datasourceCreate() called for MSSQL plugin.
Feb 11 05:38:05 appsmith-d6d86999-6djmx appsmith backend stdout |
[2025-02-11 00:08:05,357] [lettuce-epollEventLoop-10-3]
requestId=dd547055-d9bc-41ed-b7b4-03b0de691984 userEmail=<>
traceId=bda95acf0e182c48037214d5173d0ac9 spanId=00fdca579a517b66 -
lettuce-epollEventLoop-10-3: Connecting to SQL Server db
Feb 11 05:38:05 appsmith-d6d86999-6djmx appsmith backend stdout |
[2025-02-11 00:08:05,381] [lettuce-epollEventLoop-10-3]
requestId=dd547055-d9bc-41ed-b7b4-03b0de691984 userEmail=<>
traceId=bda95acf0e182c48037214d5173d0ac9 spanId=00fdca579a517b66 -
HikariPool-3 - Starting...
Feb 11 05:38:15 appsmith-d6d86999-6djmx appsmith backend stdout |
[2025-02-11 00:08:15,346] [parallel-2]
requestId=dd547055-d9bc-41ed-b7b4-03b0de691984 userEmail=<>
traceId=bda95acf0e182c48037214d5173d0ac9 spanId=88ee6f7444f698dc -
parallel-2: Action Query3 with id 67aa94d7b95fcb7ea3d5512c execution
time : 10002 ms

This was possibly brought on by the following PR : 
#38818

As part of the above PR, scheduleOn statement moved from Mono.callable()
which creates its own subscription chain to outside. This could be
causing the heavy database ops moving to event loop instead of bounded
elastic.

Fixes #`Issue Number`  
_or_  
Fixes `Issue URL`
> [!WARNING]  
> _If no issue exists, please create an issue first, and check with the
maintainers if the issue is valid._

## Automation

/test sanity

### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results  -->
> [!TIP]
> 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/13265007600>
> Commit: 3e758fb
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=13265007600&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
> Spec:
> <hr>Tue, 11 Feb 2025 14:53:05 UTC
<!-- end of auto-generated comment: Cypress test results  -->


## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [ ] No


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Refactor**
- Optimized the connection creation workflow to defer resource
initialization until required. This enhancement improves the scheduling
and responsiveness of establishing connections, ensuring that system
resources are utilized more efficiently during runtime.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
  • Loading branch information
trishaanand authored Feb 11, 2025
1 parent c66512e commit 9aa065b
Showing 1 changed file with 7 additions and 8 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -363,15 +363,14 @@ private Set<String> populateHintMessages(List<String> columnNames) {
@Override
public Mono<HikariDataSource> datasourceCreate(DatasourceConfiguration datasourceConfiguration) {
log.debug(Thread.currentThread().getName() + ": datasourceCreate() called for MSSQL plugin.");
return connectionPoolConfig
.getMaxConnectionPoolSize()
.flatMap(maxPoolSize -> {
return Mono.defer(
() -> connectionPoolConfig.getMaxConnectionPoolSize().flatMap(maxPoolSize -> {
return Mono.fromCallable(() -> {
log.debug(Thread.currentThread().getName() + ": Connecting to SQL Server db");
return createConnectionPool(datasourceConfiguration, maxPoolSize);
});
})
.subscribeOn(scheduler);
log.debug(Thread.currentThread().getName() + ": Connecting to SQL Server db");
return createConnectionPool(datasourceConfiguration, maxPoolSize);
})
.subscribeOn(scheduler);
}));
}

@Override
Expand Down

0 comments on commit 9aa065b

Please sign in to comment.