This repository has been archived by the owner on Sep 23, 2024. It is now read-only.
Logical replication failing due to too many sessions being opened with the database #235
Labels
bug
Something isn't working
Describe the bug
When running with log replication, if the extractor encounters array data, it will open multiple connections to Postgres in order to figure out the array data's schema. When there are large tables that contain array fields, the number of connections being opened to Postgres will cause new connections to be denied. This results in a connection timeout error.
To Reproduce
Steps to reproduce the behavior:
Timeout error:
name=tap_postgres level=CRITICAL message=connection to server at "XXXXXX" (XXXXXX), port 5432 failed: timeout expired
Spikes in connections count when running Meltano. Behaviour is shown being turned on and off by including / excluding the table containing the array field. Session counts drop after the timeout error is hit.
![Screenshot 2024-04-22 at 16 04 11](https://private-user-images.githubusercontent.com/42941066/324510908-292fc12d-ed1e-494a-872b-b9dd1b7ffc28.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkzMTg3MzAsIm5iZiI6MTczOTMxODQzMCwicGF0aCI6Ii80Mjk0MTA2Ni8zMjQ1MTA5MDgtMjkyZmMxMmQtZWQxZS00OTRhLTg3MmItYjlkZDFiN2ZmYzI4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjEyVDAwMDAzMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE5YzQ5YmE5MGU4MTcyZjM3YmE1M2FiM2ViMmQzOWE2OTc3ZGMxZWQ1YzE3NGIxZTY2NzlhYmU4NzhkNTdmMGYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.fX8gT0bd0sWPn0kxNoK2Vfryz_Ge-U7B2xUGxWRUKh8)
Expected behavior
There should be options to allow for the array fields to be handled as string or parsed as JSON. I would be hassle to have to write a potentially have to write a UDF in a downstream platform to parse the string but this is better than a process that fails and also degrades the production database or the read replica database.
Your environment
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: