Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sqlite-utils bulk command #377

Merged
merged 11 commits into from
Jan 11, 2022
Merged

sqlite-utils bulk command #377

merged 11 commits into from
Jan 11, 2022

Conversation

simonw
Copy link
Owner

@simonw simonw commented Jan 10, 2022

Refs #375

Still needs:

  • Refactor @insert_upsert_options so that it doesn't duplicate @import_options
  • Tests
  • Documentation
  • Try it against a really big file

@simonw simonw added this to the 3.21 milestone Jan 10, 2022
@codecov
Copy link

codecov bot commented Jan 11, 2022

Codecov Report

Merging #377 (f4ea0d3) into main (b6dad08) will decrease coverage by 0.01%.
The diff coverage is 96.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #377      +/-   ##
==========================================
- Coverage   96.52%   96.50%   -0.02%     
==========================================
  Files           6        6              
  Lines        2330     2378      +48     
==========================================
+ Hits         2249     2295      +46     
- Misses         81       83       +2     
Impacted Files Coverage Δ
sqlite_utils/cli.py 95.49% <94.28%> (-0.06%) ⬇️
sqlite_utils/db.py 97.68% <100.00%> (+0.03%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6dad08...f4ea0d3. Read the comment docs.

@simonw
Copy link
Owner Author

simonw commented Jan 11, 2022

@simonw simonw marked this pull request as ready for review January 11, 2022 02:03
@simonw
Copy link
Owner Author

simonw commented Jan 11, 2022

I tested this like so:

% wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
% sqlite-utils create-database test.db
% sqlite-utils create-table test.db power_plants url text owner text
% sqlite-utils schema test.db 
CREATE TABLE [power_plants] (
   [url] TEXT,
   [owner] TEXT
);
% sqlite-utils bulk test.db 'insert into power_plants (url, owner) values (:url, :owner)' global_power_plant_database.csv --csv
  [------------------------------------]    0%
  [###################################-]   99%
% sqlite-utils tables --counts test.db -t
table           count
------------  -------
power_plants    33643

@simonw simonw merged commit 1291415 into main Jan 11, 2022
@simonw simonw deleted the bulk branch January 11, 2022 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant