Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Minimise files #74

Merged
merged 25 commits into from
May 3, 2016
Merged

Minimise files #74

merged 25 commits into from
May 3, 2016

Conversation

martinghunt
Copy link
Collaborator

@martinghunt martinghunt commented May 3, 2016 •
edited
Loading

This minimises the files on disk during a run.

  • reads and refs fastaas are not written for each cluster until needed
  • each cluster directory is deleted as soon as that cluster is finished.

Now the number of files during a run is one log file per cluster, plus whatever files are there while a cluster is running.

@martinghunt
Start writing ReadStore, new method _sort_file
b3d00f8
@martinghunt
Add read_store
c1128a5
@martinghunt
New method _compress_and_index_file
a73a986
@martinghunt
sort and compress in __init__
8fc3dcb
@martinghunt
New method get_reads
5ae61dc
@martinghunt
remove one import pysam (was there twice)
1dc4745
@martinghunt
New method clean
b9808be
@martinghunt
Make input files if root_dir does not exist
39a20f5
@martinghunt
Protect against overwriting infile
b8fafa0
@martinghunt
Make read store file instead of fastq files for each cluster
a43c7ee
@martinghunt
Use read store in each cluster
c58a855
@martinghunt
Write message when getting reads
40ed0fc
@martinghunt
Add missing @ in fastq file name lines
1032409
@martinghunt
Add missing test file
c41b05c
@martinghunt
Pass clean option
97d66e6
@martinghunt
extract reads and references in run(), not __init__()
7176e80
@martinghunt
Store assembled ref seqs in memory instead writing to file
a842008
@martinghunt
Fix function call to use new syntax
cf08815
@martinghunt
Get assembled seqs from memory, not files
70f43db
@martinghunt
Typo in log message
5d5868b
@martinghunt
Delete read store files
727e61a
@martinghunt
gzip assembled seqs fasta file
b59c19d
@martinghunt
Delete cluster dir asap
bf7d9bb
@martinghunt
Remove self.log_fh (breaks multithreading)
d3bee7a
@martinghunt
Pass clean option when multithreading
4d3ea9d
@martinghunt martinghunt merged commit 4588cba into sanger-pathogens:master May 3, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@martinghunt