[Feature]: speed up discovery, just discover selected tables

[s7clarke10]

Feature scope

Taps (catalog, state, stream maps, etc.)

Description

I'm not sure if this is a capability of the SDK for database taps or not.

We were finding that the Discovery of Tables running in an ephemeral container deployment where you always have to discover not satisfactory when you have a large number of tables in the source system. On a particular database, this was taking 2-5 minutes due to the slowness of the Oracle Data Dictionary and the large number of tables.

We decided that we would add a feature to just discover selected tables i.e., tables set in the config or in the MELTANO __SELECT environment variable. If that config item was set, then the discovery was limited to just the selected tables. With that in place discover went from 2-5 minutes to a few seconds.

This was the particular change that we placed in pipelinewise-tap-oracle to support this feature.

s7clarke10/pipelinewise-tap-oracle@d348c4e
s7clarke10/pipelinewise-tap-oracle@d545a7d

Just sharing this idea as I believe it is a helpful feature.

[kgpayne]

+1 Ran into slow discovery with tap-snowflake too, which would benefit from this feature.

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Duplicate of #1234