From 9aa065b195d664059f2ce61f1e735b64b8410b25 Mon Sep 17 00:00:00 2001
From: Trisha Anand <trisha@appsmith.com>
Date: Tue, 11 Feb 2025 20:24:31 +0530
Subject: [PATCH] fix: Scheduling the heavy database ops on bounded elastic
 thread pool for Ms-sql (#39176)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## Description
Production is seeing high number of pod restarts causing downtime. It
seems to be correlated with user MS-SQL query triggers. On the
hypothesis that its indeed the plugin executions causing this, it could
be the datasource create taking up redis thread pool (possibly because
of an upstream scheduling of a task on it)

Sample log lines displaying the behaviour (only user email removed from
the logs) :
traceId=bda95acf0e182c48037214d5173d0ac9 spanId=00fdca579a517b66 -
boundedElastic-13: datasourceCreate() called for MSSQL plugin.
Feb 11 05:38:05 appsmith-d6d86999-6djmx appsmith backend stdout |
[2025-02-11 00:08:05,357] [lettuce-epollEventLoop-10-3]
requestId=dd547055-d9bc-41ed-b7b4-03b0de691984 userEmail=<>
traceId=bda95acf0e182c48037214d5173d0ac9 spanId=00fdca579a517b66 -
lettuce-epollEventLoop-10-3: Connecting to SQL Server db
Feb 11 05:38:05 appsmith-d6d86999-6djmx appsmith backend stdout |
[2025-02-11 00:08:05,381] [lettuce-epollEventLoop-10-3]
requestId=dd547055-d9bc-41ed-b7b4-03b0de691984 userEmail=<>
traceId=bda95acf0e182c48037214d5173d0ac9 spanId=00fdca579a517b66 -
HikariPool-3 - Starting...
Feb 11 05:38:15 appsmith-d6d86999-6djmx appsmith backend stdout |
[2025-02-11 00:08:15,346] [parallel-2]
requestId=dd547055-d9bc-41ed-b7b4-03b0de691984 userEmail=<>
traceId=bda95acf0e182c48037214d5173d0ac9 spanId=88ee6f7444f698dc -
parallel-2: Action Query3 with id 67aa94d7b95fcb7ea3d5512c execution
time : 10002 ms

This was possibly brought on by the following PR :
https://github.com/appsmithorg/appsmith/pull/38818

As part of the above PR, scheduleOn statement moved from Mono.callable()
which creates its own subscription chain to outside. This could be
causing the heavy database ops moving to event loop instead of bounded
elastic.

Fixes #`Issue Number`
_or_
Fixes `Issue URL`
> [!WARNING]
> _If no issue exists, please create an issue first, and check with the
maintainers if the issue is valid._

## Automation

/test sanity

### :mag: Cypress test results
<!-- This is an auto-generated comment: Cypress test results  -->
> [!TIP]
> 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/13265007600>
> Commit: 3e758fb8bbe536a653228d9d61c59eb0e9f1b0b1
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=13265007600&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.Sanity`
> Spec:
> <hr>Tue, 11 Feb 2025 14:53:05 UTC
<!-- end of auto-generated comment: Cypress test results  -->


## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [ ] No


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Refactor**
- Optimized the connection creation workflow to defer resource
initialization until required. This enhancement improves the scheduling
and responsiveness of establishing connections, ensuring that system
resources are utilized more efficiently during runtime.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---
 .../java/com/external/plugins/MssqlPlugin.java    | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/app/server/appsmith-plugins/mssqlPlugin/src/main/java/com/external/plugins/MssqlPlugin.java b/app/server/appsmith-plugins/mssqlPlugin/src/main/java/com/external/plugins/MssqlPlugin.java
index 5f47a5cf9d42..3677eba0fdab 100644
--- a/app/server/appsmith-plugins/mssqlPlugin/src/main/java/com/external/plugins/MssqlPlugin.java
+++ b/app/server/appsmith-plugins/mssqlPlugin/src/main/java/com/external/plugins/MssqlPlugin.java
@@ -363,15 +363,14 @@ private Set<String> populateHintMessages(List<String> columnNames) {
         @Override
         public Mono<HikariDataSource> datasourceCreate(DatasourceConfiguration datasourceConfiguration) {
             log.debug(Thread.currentThread().getName() + ": datasourceCreate() called for MSSQL plugin.");
-            return connectionPoolConfig
-                    .getMaxConnectionPoolSize()
-                    .flatMap(maxPoolSize -> {
+            return Mono.defer(
+                    () -> connectionPoolConfig.getMaxConnectionPoolSize().flatMap(maxPoolSize -> {
                         return Mono.fromCallable(() -> {
-                            log.debug(Thread.currentThread().getName() + ": Connecting to SQL Server db");
-                            return createConnectionPool(datasourceConfiguration, maxPoolSize);
-                        });
-                    })
-                    .subscribeOn(scheduler);
+                                    log.debug(Thread.currentThread().getName() + ": Connecting to SQL Server db");
+                                    return createConnectionPool(datasourceConfiguration, maxPoolSize);
+                                })
+                                .subscribeOn(scheduler);
+                    }));
         }
 
         @Override