You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When running mmseqs prefilter on Mac ARM, I've noticed some performance issues.
Here's a summary of what I've noticed:
this issue doesn't seem to happen on linux, but my linux machine is much more powerful than my laptop, so I can't say for sure
it seems to consistently happen when a kmer length of 7 is chosen (parameter -k 7), even with very small searches (e.g. 5,000 queries vs. 5,000 targets).
for our use case, we have been running with -k 0, which seems to usually choose a kmer length of 6, but when it decides to choose 7, we start to run into the performance issues
it seems like, for now, we can just default to an explicit -k 6 to avoid the slowdown, but it would be nice to have the option to use different kmer lengths
when the performance degrades, I've noticed that the memory & CPU usage ends up being roughly half of what is predicted & expected
my initial tests used MMseqs v15, but I see the same issues with using -k 7 on the latest release (v17)
when running with -k 0, it seems like v17 now chooses to use a kmer length of 6 on my laptop instead of 7
When running with -k 0, MMseqs chose a kmer size of 7, and it estimated roughly 20 hours to completion for the first database split.
[jack@manami tmp (dev)]$ time mmseqs prefilter queryDB targetDB prefilterDB --k-score 80 --min-ungapped-score 15 --max-seqs 1000 --threads 8
prefilter queryDB targetDB prefilterDB --k-score 80 --min-ungapped-score 15 --max-seqs 1000 --threads 8
MMseqs Version: 6f45232ac8daca14e354ae320a4359056ec524c2
Substitution matrix aa:blosum62.out,nucl:nucleotide.out
Seed substitution matrix aa:VTML80.out,nucl:nucleotide.out
Sensitivity 4
k-mer length 0
Target search mode 0
k-score seq:80,prof:80
Alphabet size aa:21,nucl:5
Max sequence length 65535
Max results per query 1000
Split database 0
Split mode 2
Split memory limit 0
Coverage threshold 0
Coverage mode 0
Compositional bias 1
Compositional bias 1
Diagonal scoring true
Exact k-mer matching 0
Mask residues 1
Mask residues probability 0.9
Mask lower case residues 0
Minimum diagonal score 15
Selected taxa
Include identical seq. id. false
Spaced k-mers 1
Preload mode 0
Pseudo count a substitution:1.100,context:1.400
Pseudo count b substitution:4.100,context:5.800
Spaced k-mer pattern
Local temporary path
Threads 8
Compressed 0
Verbosity 3
Query database size: 5000 type: Aminoacid
Target split mode. Searching through 4 splits
Estimated memory consumption: 13G
Target database size: 5050000 type: Aminoacid
Process prefiltering step 1 of 4
Index table k-mer threshold: 80 at k-mer size 7
Index table: counting k-mers
[=================================================================] 100.00% 1.27M 3m 21s 337ms
Index table: Masked residues: 6750387
Index table: fill
[=================================================================] 100.00% 1.27M 18m 47s 317ms
Index statistics
Entries: 477737682
DB size: 12499 MB
Avg k-mer size: 0.3
73233
Top 10 k-mers
RAARQGG 3256
LLNPKRH 2641
VGPGTST 2338
LTKSGGV 1370
LTKAGGV 1269
TTGGNLL 1106
KGGEGLV 1086
KGGPGLV 961
LELVGYV 796
EDAHGDN 686
Time for index table init: 0h 22m 21s 129ms
k-mer similarity threshold: 80
Starting prefiltering scores calculation (step 1 of 4)
Query db start 1 to 5000
Target db start 1 to 1270722
[> ] 1.00% 51 eta 19h 30m 45s
\\ *******************
\\ process killed here
\\ *******************
real 44m31.212s
user 11m15.878s
sys 144m50.782s
Interestingly, when I ran it again, it started to run a bit faster, estimating about 13 hours to completion for the first split.
[jack@manami tmp (dev)]$ time mmseqs prefilter queryDB targetDB prefilterDB -k 0 --k-score 80 --min-ungapped-score 15 --max-seqs 1000 --threads 8
prefilter queryDB targetDB prefilterDB -k 0 --k-score 80 --min-ungapped-score 15 --max-seqs 1000 --threads 8
MMseqs Version: 6f45232ac8daca14e354ae320a4359056ec524c2
Substitution matrix aa:blosum62.out,nucl:nucleotide.out
Seed substitution matrix aa:VTML80.out,nucl:nucleotide.out
Sensitivity 4
k-mer length 0
Target search mode 0
k-score seq:80,prof:80
Alphabet size aa:21,nucl:5
Max sequence length 65535
Max results per query 1000
Split database 0
Split mode 2
Split memory limit 0
Coverage threshold 0
Coverage mode 0
Compositional bias 1
Compositional bias 1
Diagonal scoring true
Exact k-mer matching 0
Mask residues 1
Mask residues probability 0.9
Mask lower case residues 0
Minimum diagonal score 15
Selected taxa
Include identical seq. id. false
Spaced k-mers 1
Preload mode 0
Pseudo count a substitution:1.100,context:1.400
Pseudo count b substitution:4.100,context:5.800
Spaced k-mer pattern
Local temporary path
Threads 8
Compressed 0
Verbosity 3
Query database size: 5000 type: Aminoacid
Target split mode. Searching through 4 splits
Estimated memory consumption: 13G
Target database size: 5050000 type: Aminoacid
Process prefiltering step 1 of 4
Index table k-mer threshold: 80 at k-mer size 7
Index table: counting k-mers
[=================================================================] 100.00% 1.27M 11s 922ms
Index table: Masked residues: 6750387
Index table: fill
[=================================================================] 100.00% 1.27M 5m 51s 733ms
Index statistics
Entries: 477737682
DB size: 12499 MB
Avg k-mer size: 0.373233
Top 10 k-mers
RAARQGG 3256
LLNPKRH 2641
VGPGTST 2338
LTKSGGV 1370
LTKAGGV 1269
TTGGNLL 1106
KGGEGLV 1086
KGGPGLV 961
LELVGYV 796
EDAHGDN 686
Time for index table init: 0h 6m 12s 589ms
k-mer similarity threshold: 80
Starting prefiltering scores calculation (step 1 of 4)
Query db start 1 to 5000
Target db start 1 to 1270722
[> ] 1.00% 51 eta 13h 9m 44s
\\ *******************
\\ process killed here
\\ *******************
real 15m27.988s
user 6m38.978s
sys 50m15.786s
In both of these runs, I noticed that the actual memory usage was roughly half of the amount predicted by MMseqs, and the CPU usage stayed at ~400% instead of the ~800% I would expect when using 8 threads.
So, then I decided to use -k 7, and found that the prefilter also runs here with no problems.
[jack@kei tmp]$ time mmseqs prefilter queryDB targetDB prefilterDB -k 7 --k-score 80 --min-ungapped-score 15 --max-seqs 1000 --threads 8
prefilter queryDB targetDB prefilterDB -k 7 --k-score 80 --min-ungapped-score 15 --max-seqs 1000 --threads 8
MMseqs Version: 6f45232ac8daca14e354ae320a4359056ec524c2
Substitution matrix aa:blosum62.out,nucl:nucleotide.out
Seed substitution matrix aa:VTML80.out,nucl:nucleotide.out
Sensitivity 4
k-mer length 7
Target search mode 0
k-score seq:80,prof:80
Alphabet size aa:21,nucl:5
Max sequence length 65535
Max results per query 1000
Split database 0
Split mode 2
Split memory limit 0
Coverage threshold 0
Coverage mode 0
Compositional bias 1
Compositional bias 1
Diagonal scoring true
Exact k-mer matching 0
Mask residues 1
Mask residues probability 0.9
Mask lower case residues 0
Minimum diagonal score 15
Selected taxa
Include identical seq. id. false
Spaced k-mers 1
Preload mode 0
Pseudo count a substitution:1.100,context:1.400
Pseudo count b substitution:4.100,context:5.800
Spaced k-mer pattern
Local temporary path
Threads 8
Compressed 0
Verbosity 3
Query database size: 5000 type: Aminoacid
Estimated memory consumption: 25G
Target database size: 5050000 type: Aminoacid
Index table k-mer threshold: 80 at k-mer size 7
Index table: counting k-mers
[=================================================================] 100.00% 5.05M 27s 170ms
Index table: Masked residues: 27302335
Index table: fill
[=================================================================] 100.00% 5.05M 30s 779ms
Index statistics
Entries: 1911040173
DB size: 20700 MB
Avg k-mer size: 1.493000
Top 10 k-mers
RAARQGG 13310
LLNPKRH 10560
VGPGTST 9543
LTKSGGV 5571
LTKAGGV 5106
TTGGNLL 4631
KGGEGLV 4449
KGGPGAV 4006
KGGPGLV 3995
LELVGYV 3260
Time for index table init: 0h 1m 3s 914ms
Process prefiltering step 1 of 1
k-mer similarity threshold: 80
Starting prefiltering scores calculation (step 1 of 1)
Query db start 1 to 5000
Target db start 1 to 5050000
[=================================================================] 100.00% 5.00K 34m 14s 264ms
130477.492231 k-mers per position
92674480 DB matches per sequence
4896 overflows
1000 sequences passed prefiltering per query sequence
1000 median result list length
0 sequences with 0 size result lists
Time for merging to prefilterDB: 0h 0m 0s 0ms
Time for processing: 0h 35m 27s 604ms
real 35m27.605s
user 282m17.467s
sys 0m5.550s
MMseqs2 v17 | 5,000 queries vs. 5,000,000 targets | macos - 18g memory | arm M3 pro
Last week, @milot-mirdita suggested that I try the v16 release to see if this was somehow related to some prefilter memory issues that were addressed in the new release. I noticed that v17 was released over the past few days, so I did the same test on my laptop with v17.
When running
mmseqs prefilteron Mac ARM, I've noticed some performance issues.Here's a summary of what I've noticed:
-k 7), even with very small searches (e.g. 5,000 queries vs. 5,000 targets).-k 0, which seems to usually choose a kmer length of6, but when it decides to choose7, we start to run into the performance issues-k 6to avoid the slowdown, but it would be nice to have the option to use different kmer lengthsv15, but I see the same issues with using-k 7on the latest release (v17)-k 0, it seems likev17now chooses to use a kmer length of6on my laptop instead of7You should be able to download my benchmarks here
I've been running some tests on my laptop (18gb memory, mac ARM M3 pro) and on a linux workstation (128gb memory, intel i9 14900k).
Here's some of the test runs I've done:
MMseqs2 v15 | 5,000 queries vs. 5,000,000 targets | macos - 18g memory | arm M3 pro
These tests were run on my laptop.
-k 0 --k-score 80 --min-ungapped-score 15 --max-seqs 1000
When running with
-k 0, MMseqs chose a kmer size of7, and it estimated roughly20hours to completion for the first database split.Interestingly, when I ran it again, it started to run a bit faster, estimating about
13hours to completion for the first split.In both of these runs, I noticed that the actual memory usage was roughly half of the amount predicted by MMseqs, and the CPU usage stayed at ~400% instead of the ~800% I would expect when using
8threads.-k 6 --k-score 80 --min-ungapped-score 15 --max-seqs 1000
When running with an explicit
-k 6, there's no performance issues at all:MMseqs2 v15 | 5,000 queries vs. 5,000,000 targets | linux - 128g memory | intel i9 14900kf
These tests were run on my linux workstation.
-k 0 --k-score 80 --min-ungapped-score 15 --max-seqs 1000
Interestingly, MMseqs automatically chooses a kmer size of
6on my linux machine, and the prefilter runs with no problems.-k 7 --k-score 80 --min-ungapped-score 15 --max-seqs 1000
So, then I decided to use
-k 7, and found that the prefilter also runs here with no problems.MMseqs2 v17 | 5,000 queries vs. 5,000,000 targets | macos - 18g memory | arm M3 pro
Last week, @milot-mirdita suggested that I try the v16 release to see if this was somehow related to some prefilter memory issues that were addressed in the new release. I noticed that v17 was released over the past few days, so I did the same test on my laptop with v17.
-k 7 --k-score 80 --min-ungapped-score 15 --max-seqs 1000
It seems that the issue still persists on the latest release:
MMseqs2 v17 | 5,000 queries vs. 100,000 targets | macos - 18g memory | arm M3 pro
Next, I decided I'd use a much smaller target database (100,000 targets).
-k 7 --k-score 80 --min-ungapped-score 15 --max-seqs 1000
It seems to still happen:
MMseqs2 v17 | 5,000 queries vs. 5,000 targets | macos - 18g memory | arm M3 pro
Finally, I figured I'd try a tiny target database (5,000 targets).
-k 7 --k-score 80 --min-ungapped-score 15 --max-seqs 1000
Seems like the issue still persists even at this tiny database size:
Let me know if I can provide any more information to help figure out what's going on here.
The text was updated successfully, but these errors were encountered: