Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: refactor datasource import flow to add support for transaction in pg #34514

Merged
merged 13 commits into from
Jul 3, 2024
Merged

chore: refactor datasource import flow to add support for transaction in pg #34514

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
0853b38
refactor datasource import flow to add support for dryOps queries
AnaghHegde Jun 26, 2024
c41ef62
add dryOps repository class to execute the queries
AnaghHegde Jun 27, 2024
1f13f37
fix null check
AnaghHegde Jun 27, 2024
4ab0ed3
fix map issue
AnaghHegde Jun 27, 2024
a61b08c
fix issue with map and partial import query execution
AnaghHegde Jun 30, 2024
4304142
Merge branch 'release' of https://github.com/appsmithorg/appsmith int…
AnaghHegde Jun 30, 2024
d3f21eb
fix spotless failures
AnaghHegde Jun 30, 2024
51dd46d
Merge branch 'release' into pg-transaction-datsoure-import-flow
AnaghHegde Jul 1, 2024
c5daa09
review comments
AnaghHegde Jul 1, 2024
3a0653f
refactor dryOps repo class to execute all the entities in one go
AnaghHegde Jul 1, 2024
239a42f
update add to map logic
AnaghHegde Jul 1, 2024
bf4278a
null check for datasource id
AnaghHegde Jul 1, 2024
1f5ab41
update dryOps method in partial import
AnaghHegde Jul 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 2 additions & 16 deletions ...psmith-server/src/main/java/com/appsmith/server/imports/internal/ImportServiceCEImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -505,22 +505,8 @@ private Mono<Artifact> importArtifactInWorkspace(
new AppsmithException(AppsmithError.GENERIC_JSON_IMPORT_ERROR, workspaceId, errorMessage));
})
// execute dry run for datasource
.flatMap(importableArtifact -> Flux.fromIterable(mappedImportableResourcesDTO
.getDatasourceDryRunQueries()
.keySet())
.flatMap(key -> dryOperationRepository.saveDatasourceToDb(mappedImportableResourcesDTO
.getDatasourceDryRunQueries()
.get(key)))
.collectList()
.thenReturn(importableArtifact))
// execute dryOps for datasourceStorage
.flatMap(importableArtifact -> Flux.fromIterable(mappedImportableResourcesDTO
.getDatasourceStorageDryRunQueries()
.keySet())
.flatMap(key -> dryOperationRepository.saveDatasourceStorageToDb(mappedImportableResourcesDTO
.getDatasourceStorageDryRunQueries()
.get(key)))
.collectList()
.flatMap(importableArtifact -> dryOperationRepository
.executeAllDbOps(mappedImportableResourcesDTO)
.thenReturn(importableArtifact))
.as(transactionalOperator::transactional);

Expand Down
26 changes: 26 additions & 0 deletions ...ppsmith-server/src/main/java/com/appsmith/server/repositories/DryOperationRepository.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,12 @@

import com.appsmith.external.models.Datasource;
import com.appsmith.external.models.DatasourceStorage;
import com.appsmith.server.dtos.MappedImportableResourcesDTO;
import jakarta.annotation.PostConstruct;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Component;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;

import java.util.Collections;
import java.util.HashMap;
Expand Down Expand Up @@ -41,4 +43,28 @@ public Flux<Datasource> saveDatasourceToDb(List<Datasource> datasources) {
public Flux<DatasourceStorage> saveDatasourceStorageToDb(List<DatasourceStorage> datasourceStorage) {
return datasourceStorageRepository.saveAll(datasourceStorage);
}

public Mono<Void> executeAllDbOps(MappedImportableResourcesDTO mappedImportableResourcesDTO) {

Flux<List<Datasource>> datasourceFLux = Flux.fromIterable(mappedImportableResourcesDTO
.getDatasourceStorageDryRunQueries()
.keySet())
.flatMap(key -> {
List<Datasource> datasourceList = mappedImportableResourcesDTO
.getDatasourceDryRunQueries()
.get(key);
return saveDatasourceToDb(datasourceList).collectList();
});

Flux<List<DatasourceStorage>> datasourceStorageFLux = Flux.fromIterable(mappedImportableResourcesDTO
.getDatasourceStorageDryRunQueries()
.keySet())
.flatMap(key -> {
List<DatasourceStorage> datasourceStorageList = mappedImportableResourcesDTO
.getDatasourceStorageDryRunQueries()
.get(key);
return saveDatasourceStorageToDb(datasourceStorageList).collectList();
});
return Flux.merge(datasourceFLux, datasourceStorageFLux).then();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's update this logic to perform the ops in generic way as discussed over the call. Feel free to implement this while working on pages, apps refactor.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will work on this once i get more clarity on the page and actions refactor. These flows might require some changes hence i am going to stick to this approach for now.

}
}