fix(medusa): Product repo typeorm issues #4084

adrien2p · 2023-05-12T10:41:37Z

What
Type orm query strategy executes each separate query in a separate connection which exhaust the max connections allowed. Now it works again as it use to work. The perf should not be impacted

FIXES CORE-1388

changeset-bot · 2023-05-12T10:41:40Z

🦋 Changeset detected

Latest commit: 4aa42bb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 10 packages

Name	Type
@medusajs/medusa	Patch
@medusajs/admin-ui	Patch
@medusajs/admin	Patch
@medusajs/medusa-js	Patch
medusa-payment-paypal	Patch
medusa-payment-stripe	Patch
medusa-plugin-restock-notification	Patch
medusa-react	Patch
@medusajs/medusa-oas-cli	Patch
@medusajs/oas-github-ci	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2023-05-12T10:41:41Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
medusa-docs	⬜️ Ignored (Inspect)	Visit Preview		May 16, 2023 7:07am

packages/medusa/src/repositories/product.ts

…x/product-repo

adrien2p · 2023-05-14T09:59:50Z

Need to do some little cleanup and fix the batch job integration

SGFGOV · 2023-05-15T05:10:20Z

Hi.. is this done? Will this solve the too many connections issue?

…

On Sun, 14 May, 2023, 15:30 Adrien de Peretti, ***@***.***> wrote: Need to do some little cleanup and fix the batch job integration — Reply to this email directly, view it on GitHub <#4084 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXEQJHBPZ6YWZXKSTB3IEKDXGCUKPANCNFSM6AAAAAAX7LBXGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- This message (including any attachments) may contain confidential, proprietary, privileged and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.

adrien2p · 2023-05-15T07:30:16Z

Hi.. is this done? Will this solve the too many connections issue?
…
On Sun, 14 May, 2023, 15:30 Adrien de Peretti, @.> wrote: Need to do some little cleanup and fix the batch job integration — Reply to this email directly, view it on GitHub <#4084 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXEQJHBPZ6YWZXKSTB3IEKDXGCUKPANCNFSM6AAAAAAX7LBXGM . You are receiving this because you are subscribed to this thread.Message ID: @.>
-- This message (including any attachments) may contain confidential, proprietary, privileged and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.

@SGFGOV It is almost done, I need to do some cleanup and fix the batch jobs. It should fix the idle in transaction queries that typeorm seams to not handle properly due to some internal bugs in their package

…x/product-repo

adrien2p · 2023-05-15T08:31:13Z

I ve succeeded to maintain nice perf while not blocking the db

riqwan

Nice cleanup!

Looks good, overall. I'll refrain from approving since I worked on this partially as well.

integration-tests/api/__tests__/store/products/ff-product-categories.ts

packages/medusa/src/repositories/product.ts

adrien2p · 2023-05-15T12:22:37Z

/snapshot-this

github-actions · 2023-05-15T12:31:44Z

🚀 A snapshot release has been made for this PR

Test the snapshots by updating your package.json with the newly published versions:

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add [email protected]

yarn add [email protected]

yarn add [email protected]

yarn add [email protected]

yarn add [email protected]

yarn add [email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

yarn add @medusajs/[email protected]

Latest commit: 7da6cc9

olivermrbl

Very nice work! Added a couple of comments

packages/medusa/src/services/product.ts

packages/medusa/src/repositories/product.ts

olivermrbl

Very nice work! Added a couple of comments

olivermrbl

LGTM. Just need the CI to turn green 💪

packages/medusa/src/repositories/product.ts

adrien2p · 2023-05-15T15:11:39Z

@olivermrbl eveything is green now :)

adrien2p · 2023-05-15T15:13:52Z

just running some more test before we merge

adrien2p · 2023-05-15T15:19:09Z

after some more deep tests, I decided to remove the query strategy completely. We loose in performance but at least we can be sure that it works no matter the concurrency. Te metrics below does not include the work on the prices in another pr

olivermrbl · 2023-05-15T16:21:07Z

@adrien2p – feel free to merge as you please :)

SGFGOV · 2023-05-16T09:53:54Z

I know this isn't a medusa issue, but if the transaction fails and a rollback is issued the server crashes here inside the entity as it
doesn't get a connection
here
try {
await queryRunner.startTransaction(isolation);
const result = await runInTransaction(queryRunner.manager);
await queryRunner.commitTransaction();
return result;
}
catch (err) {
try {
// we throw original error even if rollback thrown an error
await queryRunner.rollbackTransaction(); <----- IT CRASHES HERE
}
catch (rollbackError) { }
throw err;
}
finally {
if (!this.queryRunner)
// if we used a new query runner provider then release it
await queryRunner.release();
}

adrien2p · 2023-05-16T09:56:56Z

@SGFGOV if you have a query runner already released issue it means that you are having a flow that is not wrapped in a transaction and you don’t have called the service using withTransaction. Therefore in concurrent requests your requests qre sharing the same manager and therefore once it is released the next one can’t use it anymore. Is that your case or is it about something else?

SGFGOV · 2023-05-16T10:03:24Z

@adrien2p , thanks for the prompt reply, here is the full error.

sorry, too many clients already
error: sorry, too many clients already
at Parser.parseErrorMessage (/node_modules/pg-protocol/dist/parser.js:287:98)
at Parser.handlePacket (/node_modules/pg-protocol/dist/parser.js:126:29)
at Parser.parse (/node_modules/pg-protocol/dist/parser.js:39:38)
at Socket. (/node_modules/pg-protocol/dist/index.js:11:42)
at Socket.emit (node:events:526:28)
at Socket.emit (node:domain:475:12)
at addChunk (node:internal/streams/readable:315:12)
at readableAddChunk (node:internal/streams/readable:289:9)
at Socket.Readable.push (node:internal/streams/readable:228:10)
at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
length: 85,
severity: 'FATAL',
code: '53300',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'proc.c',
line: '357',
routine: 'InitProcess'
}

adrien2p · 2023-05-16T10:45:23Z

@SGFGOV have you been able to test with the develop branch ? I didn’t ran i to this issue. What is the max number of connections you have configured?

SGFGOV · 2023-05-16T10:47:23Z

I have configured a pool size of 100 which I think is the default

adrien2p · 2023-05-16T10:50:55Z

Could you give a try with develop?

SGFGOV · 2023-05-16T10:56:26Z

I will try and let you know

…

On Tue, 16 May, 2023, 16:21 Adrien de Peretti, ***@***.***> wrote: Could you give a try with develop? — Reply to this email directly, view it on GitHub <#4084 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXEQJHC3JGQEHVHYOOAZPD3XGNLZVANCNFSM6AAAAAAX7LBXGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- This message (including any attachments) may contain confidential, proprietary, privileged and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.

SGFGOV · 2023-05-16T12:21:23Z

I get a different error syntax error near .[0] in the query sent error: syntax error at or near ".0" QueryFailedError: syntax error at or near ".0" at PostgresQueryRunner.query (<path to application folder>/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async SelectQueryBuilder.loadRawResults (<path to application folder>/node_modules/typeorm/query-builder/SelectQueryBuilder.js:2182:25) at async SelectQueryBuilder.executeEntitiesAndRawResults (<path to application folder>/node_modules/typeorm/query-builder/SelectQueryBuilder.js:2034:26) at async SelectQueryBuilder.getRawAndEntities (<path to application folder>/node_modules/typeorm/query-builder/SelectQueryBuilder.js:683:29) at async SelectQueryBuilder.getMany (<path to application folder>/node_modules/typeorm/query-builder/SelectQueryBuilder.js:749:25) at async Promise.all (index 0) { query: 'SELECT "variants"."id" AS "variants_id", "variants"."created_at" AS "variants_created_at", "variants"."updated_at" AS "variants_updated_at", "variants"."deleted_at" AS "variants_deleted_at", "variants"."title" AS "variants_title", "variants"."product_id" AS "variants_product_id", "variants"."sku" AS "variants_sku", "variants"."barcode" AS "variants_barcode", "variants"."ean" AS "variants_ean", "variants"."upc" AS "variants_upc", "variants"."variant_rank" AS "variants_variant_rank", "variants"."inventory_quantity" AS "variants_inventory_quantity", "variants"."allow_backorder" AS "variants_allow_backorder", "variants"."manage_inventory" AS "variants_manage_inventory", "variants"."hs_code" AS "variants_hs_code", "variants"."origin_country" AS "variants_origin_country", "variants"."mid_code" AS "variants_mid_code", "variants"."material" AS "variants_material", "variants"."weight" AS "variants_weight", "variants"."length" AS "variants_length", "variants"."height" AS "variants_height", "variants"."width" AS "variants_width", "variants"."metadata" AS "variants_metadata", "variants__prices"."id" AS "variants__prices_id", "variants__prices"."created_at" AS "variants__prices_created_at", "variants__prices"."updated_at" AS "variants__prices_updated_at", "variants__prices"."deleted_at" AS "variants__prices_deleted_at", "variants__prices"."currency_code" AS "variants__prices_currency_code", "variants__prices"."amount" AS "variants__prices_amount", "variants__prices"."min_quantity" AS "variants__prices_min_quantity", "variants__prices"."max_quantity" AS "variants__prices_max_quantity", "variants__prices"."price_list_id" AS "variants__prices_price_list_id", "variants__prices"."variant_id" AS "variants__prices_variant_id", "variants__prices"."region_id" AS "variants__prices_region_id", "variants__options"."id" AS "variants__options_id", "variants__options"."created_at" AS "variants__options_created_at", "variants__options"."updated_at" AS "variants__options_updated_at", "variants__options"."deleted_at" AS "variants__options_deleted_at", "variants__options"."value" AS "variants__options_value", "variants__options"."option_id" AS "variants__options_option_id", "variants__options"."variant_id" AS "variants__options_variant_id", "variants__options"."metadata" AS "variants__options_metadata", product.0, product.1, product.2, product.3, product.4, product.5, product.6, product.7, product.8, product.9, product.10, product.11, product.12, product.13, product.14, product.15, product.16, product.17, product.18, product.19, product.20, product.21, product.22, product.23, product.24 FROM "public"."product" "product" LEFT JOIN "public"."product_variant" "variants" ON "variants"."product_id"="product"."id" AND ( "variants"."deleted_at" IS NULL AND "variants"."deleted_at" IS NULL) LEFT JOIN "public"."money_amount" "variants__prices" ON "variants__prices"."variant_id"="variants"."id" AND ("variants__prices"."deleted_at" IS NULL) LEFT JOIN "public"."product_option_value" "variants__options" ON "variants__options"."variant_id"="variants"."id" AND ("variants__options"."deleted_at" IS NULL) WHERE ( "product"."deleted_at" IS NULL AND "product"."id" IN ($1) ) AND ( "product"."deleted_at" IS NULL ) ORDER BY "variants"."variant_rank" ASC', parameters: [ 'prod_01GZZ778SY3BV12NCQWA181A39' ], driverError: error: syntax error at or near ".0" at Parser.parseErrorMessage (<path to application folder>/node_modules/pg-protocol/dist/parser.js:287:98) at Parser.handlePacket (<path to application folder>/node_modules/pg-protocol/dist/parser.js:126:29) at Parser.parse (<path to application folder>/node_modules/pg-protocol/dist/parser.js:39:38) at Socket.<anonymous> (<path to application folder>/node_modules/pg-protocol/dist/index.js:11:42) at Socket.emit (node:events:526:28) at Socket.emit (node:domain:475:12) at addChunk (node:internal/streams/readable:315:12) at readableAddChunk (node:internal/streams/readable:289:9) at Socket.Readable.push (node:internal/streams/readable:228:10) at TCP.onStreamRead (node:internal/stream_base_commons:190:23) at TCP.callbackTrampoline (node:internal/async_hooks:130:17) { length: 93, severity: 'ERROR', code: '42601', detail: undefined, hint: undefined, position: '2321', internalPosition: undefined, internalQuery: undefined, where: undefined, schema: undefined, table: undefined, column: undefined, dataType: undefined, constraint: undefined, file: 'scan.l', line: '1188', routine: 'scanner_yyerror' }, length: 93, severity: 'ERROR', code: '42601', detail: undefined, hint: undefined, position: '2321', internalPosition: undefined, internalQuery: undefined, where: undefined, schema: undefined, table: undefined, column: undefined, dataType: undefined, constraint: undefined, file: 'scan.l', line: '1188', routine: 'scanner_yyerror' }

…

I will try and let you know On Tue, 16 May, 2023, 16:21 Adrien de Peretti, ***@***.***> wrote: > Could you give a try with develop? > > — > Reply to this email directly, view it on GitHub > <#4084 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AXEQJHC3JGQEHVHYOOAZPD3XGNLZVANCNFSM6AAAAAAX7LBXGM> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

-- This message (including any attachments) may contain confidential, proprietary, privileged and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.

adrien2p · 2023-05-16T14:38:26Z

@SGFGOV what is the request you made? It seams that there is weird select that have been inserted in the query
product.0, product.1, product.2, product.3, product.4, product.5, product.6, product.7, product.8, product.9, product.10, product.11, product.12, product.13, product.14, product.15, product.16, product.17, product.18, product.19, product.20, product.21, product.22, product.23, product.24

SGFGOV · 2023-05-16T14:47:16Z

Can you be a little more specific about weird select

…

On Tue, 16 May, 2023, 20:08 Adrien de Peretti, ***@***.***> wrote: @SGFGOV <https://github.com/SGFGOV> what is the request you made? It seams that there is weird select that have been inserted in the query — Reply to this email directly, view it on GitHub <#4084 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXEQJHFE47BKC3RU6PLYY53XGOGO3ANCNFSM6AAAAAAX7LBXGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- This message (including any attachments) may contain confidential, proprietary, privileged and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.

adrien2p · 2023-05-16T15:25:48Z

When you look at your error, you have the query that fail, and in that query, in the SELECT statement you can see that there is those product.0 etc in it which might come from somewhere but I don't know where. What are you running to get this error?

SGFGOV · 2023-05-16T15:29:17Z

Thanks, let me trace it and get back to you

…

On Tue, 16 May, 2023, 20:55 Adrien de Peretti, ***@***.***> wrote: When you look at your error, you have the query that fail, and in that query, in the SELECT statement you can see that there is those product.0 etc in it which might come from somewhere but I don't know where. What are you running to get this error? — Reply to this email directly, view it on GitHub <#4084 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXEQJHEYQPMQVKGY3MJHZRTXGOMAPANCNFSM6AAAAAAX7LBXGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- This message (including any attachments) may contain confidential, proprietary, privileged and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.

SGFGOV · 2023-05-21T12:49:26Z

@adrien2p ..thanks a ton. I got it to work.the issue was the way the query was being prepared.

fix(medusa): Product repo typeorm issues

39ac174

adrien2p commented May 12, 2023

View reviewed changes

packages/medusa/src/repositories/product.ts Outdated Show resolved Hide resolved

riqwan and others added 6 commits May 12, 2023 15:33

chore: fixed category scopes

30601df

WIP fix categories

16f467c

fix product repo to attach categories

803848e

fix uni tests

dfb2964

Merge branch 'develop' into fix/product-repo

9c2afa1

Create eighty-icons-exercise.md

2b99e20

adrien2p marked this pull request as ready for review May 12, 2023 16:21

adrien2p requested a review from a team as a code owner May 12, 2023 16:21

adrien2p added 4 commits May 12, 2023 18:22

revert package.json

c11721c

Merge branch 'fix/product-repo' of github.com:medusajs/medusa into fi…

8ad84c4

…x/product-repo

fix change set

27e469e

last fixes

86e7e8a

riqwan and others added 3 commits May 15, 2023 09:44

Merge branch 'develop' into fix/product-repo

656314e

cleanup iteration

42f1048

Merge branch 'fix/product-repo' of github.com:medusajs/medusa into fi…

9b27312

…x/product-repo

vercel bot deployed to Preview May 15, 2023 07:57 View deployment

adrien2p added 2 commits May 15, 2023 10:10

fix repository deep relations joining aliasing

c9e7cc5

improve response time

4670898

riqwan reviewed May 15, 2023

View reviewed changes

integration-tests/api/__tests__/store/products/ff-product-categories.ts Outdated Show resolved Hide resolved

integration-tests/api/__tests__/store/products/ff-product-categories.ts Show resolved Hide resolved

packages/medusa/src/repositories/product.ts Outdated Show resolved Hide resolved

Merge branch 'develop' into fix/product-repo

6828cc8

vercel bot deployed to Preview May 15, 2023 08:49 View deployment

improve category test case

eb02393

adrien2p added 2 commits May 15, 2023 12:50

update changeset

7a374bd

fix customer group

f41ceed

olivermrbl reviewed May 15, 2023

View reviewed changes

include feedback

61d31e5

olivermrbl approved these changes May 15, 2023

View reviewed changes

packages/medusa/src/repositories/product.ts Show resolved Hide resolved

fix repo

3268da8

remove query strategy

0ba8603

Merge branch 'develop' into fix/product-repo

4aa42bb

adrien2p merged commit 9518efc into develop May 16, 2023

github-actions bot mentioned this pull request May 16, 2023

Version Packages #4074

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(medusa): Product repo typeorm issues #4084

fix(medusa): Product repo typeorm issues #4084

adrien2p commented May 12, 2023 •

edited

Loading

changeset-bot bot commented May 12, 2023 •

edited

Loading

vercel bot commented May 12, 2023 •

edited

Loading

adrien2p commented May 14, 2023

SGFGOV commented May 15, 2023 via email

adrien2p commented May 15, 2023

adrien2p commented May 15, 2023

riqwan left a comment

adrien2p commented May 15, 2023

github-actions bot commented May 15, 2023

olivermrbl left a comment

olivermrbl left a comment

olivermrbl left a comment

adrien2p commented May 15, 2023

adrien2p commented May 15, 2023

adrien2p commented May 15, 2023 •

edited

Loading

olivermrbl commented May 15, 2023

SGFGOV commented May 16, 2023

adrien2p commented May 16, 2023

SGFGOV commented May 16, 2023

adrien2p commented May 16, 2023

SGFGOV commented May 16, 2023

adrien2p commented May 16, 2023

SGFGOV commented May 16, 2023 via email

SGFGOV commented May 16, 2023 via email •

edited

Loading

adrien2p commented May 16, 2023 •

edited

Loading

SGFGOV commented May 16, 2023 via email •

edited

Loading

adrien2p commented May 16, 2023

SGFGOV commented May 16, 2023 via email

SGFGOV commented May 21, 2023

fix(medusa): Product repo typeorm issues #4084

fix(medusa): Product repo typeorm issues #4084

Conversation

adrien2p commented May 12, 2023 • edited Loading

changeset-bot bot commented May 12, 2023 • edited Loading

🦋 Changeset detected

vercel bot commented May 12, 2023 • edited Loading

adrien2p commented May 14, 2023

SGFGOV commented May 15, 2023 via email

adrien2p commented May 15, 2023

adrien2p commented May 15, 2023

riqwan left a comment

Choose a reason for hiding this comment

adrien2p commented May 15, 2023

github-actions bot commented May 15, 2023

🚀 A snapshot release has been made for this PR

olivermrbl left a comment

Choose a reason for hiding this comment

olivermrbl left a comment

Choose a reason for hiding this comment

olivermrbl left a comment

Choose a reason for hiding this comment

adrien2p commented May 15, 2023

adrien2p commented May 15, 2023

adrien2p commented May 15, 2023 • edited Loading

olivermrbl commented May 15, 2023

SGFGOV commented May 16, 2023

adrien2p commented May 16, 2023

SGFGOV commented May 16, 2023

adrien2p commented May 16, 2023

SGFGOV commented May 16, 2023

adrien2p commented May 16, 2023

SGFGOV commented May 16, 2023 via email

SGFGOV commented May 16, 2023 via email • edited Loading

adrien2p commented May 16, 2023 • edited Loading

SGFGOV commented May 16, 2023 via email • edited Loading

adrien2p commented May 16, 2023

SGFGOV commented May 16, 2023 via email

SGFGOV commented May 21, 2023

adrien2p commented May 12, 2023 •

edited

Loading

changeset-bot bot commented May 12, 2023 •

edited

Loading

vercel bot commented May 12, 2023 •

edited

Loading

adrien2p commented May 15, 2023 •

edited

Loading

SGFGOV commented May 16, 2023 via email •

edited

Loading

adrien2p commented May 16, 2023 •

edited

Loading

SGFGOV commented May 16, 2023 via email •

edited

Loading