You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The way we create sample lookup optimized is inefficient, consider the following typical workflow:
Run VT for the first batch of VCF files (with --sample_lookup_optimized_output_table set and without --append).
Run VT for the second batch of VCF files (with --sample_lookup_optimized_output_table and --append set).
Run VT for the third batch of VCF files (with --sample_lookup_optimized_output_table and --append set).
...
Currently the way we load data into sample optimized tables is by querying variant optimized tables, flattening the call column, and then copying the result into sample optimized tables. In this implementation (#606), with each new run of VT (with --append set) we read all rows of variant optimized tables and load the result into sample optimized tables with write_disposition='WRITE_TRUNCATE'.
A more efficient implementation would be to flatten and add only newly added rows with write_disposition='WRITE_APPEND'.
The text was updated successfully, but these errors were encountered:
The way we create sample lookup optimized is inefficient, consider the following typical workflow:
--sample_lookup_optimized_output_table
set and without--append
).--sample_lookup_optimized_output_table
and--append
set).--sample_lookup_optimized_output_table
and--append
set)....
Currently the way we load data into sample optimized tables is by querying variant optimized tables, flattening the
call
column, and then copying the result into sample optimized tables. In this implementation (#606), with each new run of VT (with--append
set) we read all rows of variant optimized tables and load the result into sample optimized tables withwrite_disposition='WRITE_TRUNCATE'
.A more efficient implementation would be to flatten and add only newly added rows with
write_disposition='WRITE_APPEND'
.The text was updated successfully, but these errors were encountered: