forked from ssadedin/bpipe
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathReleaseNotes.txt
executable file
·1394 lines (916 loc) · 46.2 KB
/
ReleaseNotes.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Changes for Bpipe Version 0.9.9.2
===================================
This release has focused primarily on performance, especially
for pipelines involving high concurrency and large numbers
of inputs and outputs. To achieve this, Bpipe now depends
on new Java APIs available in Java 1.7. This means that
Bpipe now requires Java 1.7 and will not work any longer
with Java 1.6.
Additionally, a range of small bugs have been fixed and
smaller features added.
## Backwards Compatibility
Of note, there is one change that can
affect backwards compatibility. This relates to use of the
'transform' construct when multiple files of the same type are
transformed to multiple files of a different type. In the past
you could write a single transform such as:
```
transform('.txt') to('.csv') {
...
}
```
Previously this would transform every text file in the inputs
to a corresponding CSV file as output. From 0.9.9.2 forward
this will only transform a single input file. For multiple files, the
transform must now be specified as:
```
transform('*.txt') to('.csv') {
...
}
```
This change has been made to resolve ambiguity, since there was no
way previously to specify that a stage transforms exactly 1 input file
of a given type.
## Change list
- Large scale refactoring for performance with
large numbers of inputs / outputs.
Bpipe now requires Java 1.7!
- Issue #180: bpipe doesn't allow empty branch list
- Issue #184: chr not passed to nested parallel pipeline
- fix wrong output referenced for 2nd double ext in
produce eg: $output.bar.txt
- support for modules in TorqueCommandExecutor
- fix potential failure to cleanup files on error
- Issue #175: Check state can be confused b/w branches
- Handle errors in log init so that log fail doesn't prevent
ability to run Bpipe (java logging tries to obtain exclusive
locks which aren't supported on some file systems.
- Update to groovy 2.4.6 (helps performance)
- Fix ambiguous behavior of transform() when multiple inputs
match transform pattern. New behavior requires transform('*.ext') to
match upstream multiple files ending with .ext (minor possibilty for
backwards incompatibility)
- Fix output resolved as input for produce+double ext + numbered input
- Fix exception when check saved without pass/fail set
- Fix leak of tail / sed processes under OSX in some
situations
- Fix bootstrap theme for docs not recognised
- Support to open a debugger Groovy console at any point
in a pipeline using debug() function
- Add settings.gradle to stop build failing if
checked out to directory other than 'bpipe'
- Fix possible hang when multi command used inside
produce(...)
- support for "mem_param" for torque executor
this allows to choose which parameter Bpipe should use
between "mem", "pvmem", etc. by setting mem_param="mem"
in a command config
- Issue #176: support multiple commands per config in multi
- Various documentation fixes and cleanups
- fix double file extension on input not resolving second and
later input (eg: when referenced in form $input2.foo.bar)
Changes for Bpipe Version 0.9.9
==============================
**NOTE:** see 0.9.9_beta_1 changes for further changes since previous release
- SGE: Merge branch 'sge_fixes', including:
* Fix #156, #142, Bpipe may hang or ignore job cancel / delete
using SGE executor
* add -terse to sge default command options
* change to support separate options:
'sge_pe' and 'procs' rather than combined.
(Old combined form still supported for back compat.)
* jobs are created using user modifiable command template,
specified via jobTemplate configuration parameter
which can be object with string value of path to template,
or a closure which will be executed dynamically to obtain the
path
* fix incorrect reporting that stop command
failed in sge
* use Bpipe resolved working directory instead of $PWD since
it is not guaranteed to be defined
* fix cases of failure due to string procs variable
getting overridden with integer (dynamic calculation
of $threads variable)
* fix incorrectly sending slots as well as pe for some
configurations
* support individual parameters for SGE options instead of compiling
them into 'additional_options'
- Fix choosing "terminate" not working for some executors
(issue #162)
- PBS-pro: combine multiple resource limits into single
statement, add support for procs, memory requests
- Experimental option for some executors to disable buffering of
standard output, applies to torque, slurm, pbs-pro. Set "stdbuf" as
option to have bpipe run commands using 'stdbuf' command to disable
output buffering, when output from commands is delayed in appearing
in Bpipe log.
- Performance optimisations for large / complex pipelines:
* Cache output graph in serialised form to avoid reading property
files at startup
* Reduce massive logging of outputs when large number of
inputs and outputs in single stage
- Add 'bpipe errors' and 'bpipe log -e' command to
show output for failed commands
- Add support to display per command output using the
bpipe log -c <cmd id>
- Fix 'bpipe test' printing exceptions and hanging on 'multi'
statement
- Fix regression causing possible failure when stage has
outputs but no inputs
- update gradle wrapper script, include changes omitted from
2.0 upgrade
- Fix files in subdirectory not resolved when whole file name
matches input extension
- Fix variable specified with 'using' not overriding global value
- Fix 'check' results possibly leaking between branches
- Fix reading output.dir causing potential modification of
output directory
=========== Changes for Bpipe Version 0.9.9_beta_1 ============
- Fix content type not set causing errors with some smtp
connections
- Issue #155: Update to gradle 2.0 to avoid compile failure with JDK9
- Performance optimisations for large numbers of inputs /
outputs
- Should report error if is referenced and
finds no inputs
- Give a hint about which 'from' clause
pattern did not find a match in error message
- Fix typo causing exception while logging error message
- Fix: setting branch name from within stage causes incorrect
branches downstream
- Fix input resolved as output in rare circumstances
- Fix transform ... to followed by double
output variable extensions (.foo.bar) failing with confusing
error message
- Fix procs not set in torque job when using variable
- Make bpipe return exit code reflecting whether pipeline
succeeded or failed
- Fix: bpipe may fail to cleanup files created in subdirectories
- Fix hang / exception when using multi command (regression)
- Fix regression caused by incorrect synchronization on wrong object. Refactored
Concurrency resource auctions for clarity
- Fix regression / failure when no inputs specify any fixed thread
requirements
- Better / fairer scheduling for dynamically allocated $thread variables
- Support for ranges in procs parameter, eg: procs = 1..10, or
uses(threads:1..10) { ... }
- Fix: failure to parallelize stages inside a list added to
another list, eg: [foo, bar] + [fubar,fug]
- Added explicit license file for clarity of
licensing
- Issue #148: bpipe parsing commandline memory
specification incorrectly
- Issue #150: Cannot pass blank parameter via command line
- Various documentation fixes
- Fix: junit tests were not executed
- Additional SGE options, SGE options moved inside script
- fix: commands run using torque / pbs not using -e flag for bash,
thus only treated as failed if last command fails
- support for aliasing inputs to outputs via 'forward <x> to
<y>' to allow an input to be virtually treated as a named
output within the pipeline
- fix: bpipe terminating if ssh session terminates while showing bpipe log
in foreground directly after initial launch
- save output from LSF (to assist in debugging failed LSF commands)
- fix stages not found when defined in bpipe lib folder and referenced in
segment declaration if used after load
- Switch to use html format for emails by default
- allow user to configure email format with 'format' attribute in
configuration
- allow multiple templates per notification type with different formats
- fix problems in using 'send'
and 'file' to send explicit files based on templates
- Fix high thread concurrency used when large number of concurrent branches
created by one pipeline branch
- Explicitly destroy process after waiting
until streams closed to ensure file handles not leaked
- fix some cases where Bpipe could report file as missing when
created in slow syncing distributed file system
- Allow more screen width when using 'bpipe checks', better formatting
of check display
- Add option to list checks rather than enter interactive mode
- fix some cases where commands might not
be stopped, and outputs not cleaned up when using bpipe
stop. Update saved command info after cleanup.
- ensure stop time is always set for commands
- close possible file handle leak in (undocumented) capture command
- more robust checking for file existance
- refactor file existance checking into common utils method, to allow
wider application
- Avoid misleading termination message when user deliberately stops
pipeline
- Incorrect input used as basis for output file name in transforms inside
'from', if competing input available from previous stage
- fix: rare deadlocks - use dedicated thread pool per pipeline tier to
avoid possibility of deadlocks in heavily nested pipelines
=========== Changes for Bpipe Version 0.9.8.7 ============
- check /proc on linux rather than relying on kill
0 to see if pipeline is running (enables cross
user support for "bpipe log")
- Addition of documentation to Bpipe distro
- fix failure to cleanup files with no parent directory
(regression)
- Add __future__ import for Python 2 and 3 compatibility
(fix failure on Mac when Python3 is default)
- switch to HTML format for default email. to switch back
to text, use notifications { smtp { format="text" } } in
bpipe.config.
- Fix presence of subdirectory in bpipe output meta data folder
causing bpipe to spew errors
- Fix files sometimes not cleaned up on distributd file system
where lag exists in synchronizing b/w nodes
- Issue #128: use qsub -N to name job according to stage/filename
- support for overriding a stage with an alternative implementation
within a particular branch
- fix: authentication failures on passwordless SMTP
server
- fix failure in cleanup if output properties file
did not get optional canonicalPath property added
- comment out verbose log message creating large bpipe
logfiles for long runs
- avoid encoding whole directory path into internal property file
names (fix 'file path too long errors' on file systems allowing only
256 chars in file names)
- fix parent branches not visible in children via branch property
- fix method missing error when multiplying by map by list
- Reformatting and summarizing of errors at end of pipeline to make
it easier to see where & why pipelines failed
- Aggregating identical errors from
multiple branches together in output so that failures from
many branches are less verbose
- Support for 'load' within a pipeline stage. Loads into the
context of current branch rather than global. Pipeline stages
not supported, only variables.
- prevent error if files used by Bpipe are not writable but
are deletable
- prevent premature wildcard expansion of
quoted arguments passed to Bpipe
- support for cleanup as command in pipeline stage
- prevent error if files used by Bpipe are not writable but
are deletable
- attempt to ensure log messages written
even if exit due to outofmemory error
- log message if file is not cleaned up because it doesn't
exist: help to diagnose problems with files not cleaned up
- better log info about where config files are
- allow memory to be integer for torque/slurm/etc, fix error
querying status in torque/slurm/etc, include queuing system
command id in bpipe status output
- show pending and waiting jobs with bpipe status, don't throw ugly error
if no commands ever run
- Fix: downstream references to inputs
by index throwing "expected more inputs" error incorrectly
- avoid confusing stack overflow when user defines branch
variable with same name as pipeline stage
- accept 'MB' or 'GB' in command line option for memory,
cache runtime directory instead of creating new File objects
all the time to figure it out
- prevent null pointer exception when list containing null
provided to 'from'
- fix multiple inputs not
forwarding correctly when referenced using "$inputs"
- Bug fixes for lsf, added walltime option for Lsf
- caching of DAG lookups by
canonical path to speed up large, complex pipelines
- caching of canonical path with check of basePath to ensure
consistency
- fix dependency issues after
cleanup when non-default output directory used
- Performance optimisations for systems where canonicalPath is
very slow, better logging to resolve dependency issues
=========== Changes for Bpipe Version 0.9.8.6 ============
- fix: errors may printed about files not existing when attempting
to move to trash
- fix bpipe jobs not printing jobs after user ctrl-c's out of
run command
- fix incorrect default for procs_mode variable for torque
- fix ugly stack trace error when transform called without arguments
- fix failing to escape trailing . in split patterns
- branch.dir now ha a default value so that it is easier to put in reusable
pipeline stages
- Ensure walltime gets interpreted as
integer even if passed as String / BigDecimal / Long etc
- fix trash option not working in bpipe cleanup
- Fix default output directory not applied to second
produce argument if first specifies a directory
- fix produce() with reference only to output.dir causing
retry test to incorrectly indicate commands would run again
- Fix incorrect file reported for errors when multiple
external files loaded, added some comments, removed hard
tabs
- fix branch variable not overriding default value of
variable specified with var
- Fix files not checked after 'produce' if previously output and then
removed outside of bpipe
- Fix files not checked after 'produce' if
previously output and then removed outside of bpipe
- fix 'inputs' variable with double file extension not always resolving
only files matching both extensions
- fix output() function causing spurious errors about incompatible inputs
when combined with output variable extensions
- fix command not running when explicitly specified by produce() if another
implicit output referenced by command
- Support for produce using wildcard based on regex (via ~)
- Add information about total run time to report and command log
- support to specify branch filter on command line, to limit
running to a specific branch (for testing, usually)
- fix explicit output() combined with produce(...) resulting in
inconsistent behavior
- Fix stack traces printed out when errors occur to use actual name of user
scripts instead of groovy generated ones
- fix build.gradle not working with existing gradle wrapper
- Implement default behavior for transform with
more to() values than from() values instead of failing
- Fix output referenced inside check causing check to be
skipped / behave incorrectly
=========== Changes for Bpipe Version 0.9.8.6_rc2 ============
- Issue 100: report does not accurately reflect repeated and
parallel stages
- fix: inputs passed in wrong order to parallel stage after 'produce' when
lexical sorting differs to input order
- enhance: migrated all of Bpipe to use groovy 2.3.4 for JDK1.8 compatibility
- Refactoring of basis for pipeline diagrams to solve longstanding bugs
- fix: commands not appearing in html report
- fix: bad log statement causing error if log
file fails to appear, better behavior for distributed
file systems where command may finish before output file becomes
visible on remote node
- fix: transform() from() to() form can result in Bpipe not detecting
command failure
- fix: pipeline fails if pipeline stage is referenced both with and without
'using'
- fix: internal errors (eg: outofmemeory) possibly leaving the Pipeline in
corrupted internal state
- enhance: Make it possible to set custom R executable via
bpipe.config
- fix: some status codes from PBS/torque occasionally causing errors in
logs
- fix: local commands don't show pid with bpipe status
- fix: status command showing old jobs and /
or failing in situations where command objects are missing /
corrupt
- enhance: add thread id to log file (debug)
- fix: not all outputs from branches split by file pattern forwarded to
downstream stage when number of files differs in each branch
- fix: take into account whole parent hierarchy in calculating
total parallelism for computing variable
- enhance: support for sending proc_mode config param to custom executors and
CUSTOM executor config attribute
- enhance: Torque/PBS: support for sending procs request using nodes/ppn attribute
instead of procs
- fix: 'forward' failing with cryptic errors when attempting to
forward multiple files
- enhance: enabled matching of directory name in split pattern when
directory present in pattern
- fix: second and later outputs possibly not cleaned up in some
situations of command failure
- fix: branch variables not accepted as required parameters
- fix: output does not include branch if no input as base and output
extension syntax used
- fix: R aborting when user hits control-c in some situations
- fix: some complex pipelines starting with list failing with errors
- fix: fixes for PBS Pro executor from Davide
- enhance: support for PBS Pro thanks to Davide Rambaldi
=========== Changes for Bpipe Version 0.9.8.6_rc ============
- support for 'load' to work with URLs for loading
scripts remotely
- fix chr variable created by chr() not considered as satisfying
requires chr:...
- start and stop time not set by local command
exector
- restrict use of auto filter to cases where chr actually
in operation
- file() construct sometimes returning paths with .. in them
- Add pipeline stage name to printout in bpipe query
- support for multiple file extensions when using $output variable
($output.foo.bar.gz, etc)
- fix incorrect outputs resolved for filter when inputs are
resolved from a relative path leading with "." character
- fix preserve not resolving files specified by different absolute /
relative paths to Bpipe's form, (thx Anthony)
- Not serializable exception on legacy directories
- Remove legacy feature where output may
be inferred from created files: does not behave reliably in
parallelized pipelines
- support for multiple "preserve" patterns
- fix preserved status not displayed
correctly by bpipe query
- support for grouping commands in tool database
into 'modules'
- fix syntax errors in tool database
- added igv tools, fix GATK probe, add VEP to tools database
- fix binary report files copied as text causing corruption
- add stage name to meta data properties saved for each
output file
- fix filter sometimes expecting wrong input when
file extension supplied
- fix "bpipe run" leaking a tail process each time on OSX
- support for generating reports using arbitrary groovy
code in a separate script
- add tools and versions to meta data stored in
output properties
- fix status for SLURM displaying Torque in output
- fix possible conflicting output names on multiple branches
when nested branches cascade without intervening stages
- add support for referencing output graph in reports
- transform should be able to
match whole file name, fix incorrect input can be resolved
inside transform() to() construct
- display running time and pending output in response to
bpipe query
- Add procs to LSF executor
- refactored a lot of executor stuff to remove duplicated
code, plus to allow for recording start of execution time
separate to start of queuing time
- fix GB ignored as key for memory
- support to use ${output(...)} in commands instead of produce
- support for adding jars to classpath via bpipe.config
(libs="foo.jar:bar.jar...")
- fix requires at pipeline stage level no longer working because
overriden by top level requires function
- experimental: multithread startup to ammeliorate long delays
when file system is slow
- make it possible to remove the cwd option from lsf
executor (see google group)
- use -e instead of -eo for LSF since it is compatible with OpenLava
- ability to use multiple extensions for 'input' variable file
extensions, eg: $input.foo.bar.csv returns files ending in
'.foo.bar.csv'
- fix input with file extension not requiring match on first
character in some cases
- fix problem in some weird cases where object has null class
- prevent errors when running in test mode with explicit
notifications sent in pipeline to 'file' channel
- switch to SimpleTemplateEngine because GStringTemplateEngine is not
observing classpath of added libraries, only core groovy
libraries
- fix output directory set by command line
resulting in inputs not correctly recognised in 2nd and
later stages of pipeline
- fix required inputs using exact path instead of file name
========== Changes for Bpipe Version 0.9.8.6_beta_3 ============
- support for setting check message without failing or succeeding
- explicitly (check.message = "...." in otherwise clause)
- new check report to send / write a summary of all checks
as HTML (bpipe run -R checks ...,
send report('checks.html') ...)
- documentation now available to reports based on user
templates
- "utils" now provided to templates for generic
utilities (escape, todo: documentation)
- Ensure full stack trace is
logged in bpipe log when missing function encountered
- make produce() with directory use that directory rather
than coerce to default output dir
- added new supported commands to default help display
- fix parameters passed in by command line not resolvable in
templates
- support 'send report(...) to file: <file name>'
syntax
- fix subject line used incorrect in SMTP notifications
- inputs not being correctly checked to be provided when specified
at root level
- don't display exception info when failure is due to
value missing from specified inputs via 'inputs ...' statement
- support for 'requires' at top level outside of pipeline
definition
- improve wording and display of error when required value is
missing
- fix templates not able to see all classes available on
classpath
- removed all forms of throttling of R sessions (should now be
unnecessary) as it can form a bottleneck in certain
scenarios
========== Changes for Bpipe Version 0.9.8.6_beta_2 ============
- avoid warnings due to project properties (thanks Davide)
- refactored email functionality to use user configurable
templates
- fix notifications for pipeline finished saying
pipeline failed when it didn't
- support for displaying information about checks at end of pipeline
- more information in SMTP notifications, including checks
- Added option to override all checks for cases where there are
very many that have failed
- changed unknown communication provider to be a warning instead
of a pipeline terminating exception.
- fix relative symbolic link to Bpipe not resolving properly
under Mac/OSX
- fix possibility of commands sometimes
not being cleaned up (add pause before terminating Bpipe,
after terminating child processes)
- Fix Slurm not returning failure exit code when
job cancelled via bpipe stop (causing outputs not to be
cleaned up)
- Added "checks" command and "override" command to display and
override checks respectively
- Fix transforms sometimes applying in such a way
as to add branch name with no period separating them from
the rest of the file name (regression)
- fix transforms producing null outputs when no period in
input file name (regression)
- support for rendering pipeline diagrams as SVG
- support for specifying pipeline stages to ignore
in diagrams via noDiagram="..." in bpipe.config (annotations to
come in the future)
- fix diagrams generated incorrectly if 'segment'
command used
- fix documentation failing to generate unless script has a
relative or absolute path prefixing it
- some tweaks to avoid rare cases double
dots appearing in file names with transform
- 'succeed' should terminate parent branch when all child
branches terminated via succeed
- support for running a pipeline "until" a specified stage
( -u)
- support for send to use HTML templates instead of
inline HTML
- support for 'succeed' and 'fail' to share
'send' syntax to generate an email or IM along with the
termination of the branch
- fix slurm executor sometimes
treating slurm stdout and stderr temp files
as real outputs of pipeline
- Fix space inserted when withFlag used with flags
ending with "="
- fix variables and functions loaded via 'load' not available until
pipeline launched (inside run { } )
- allow global resource limits (as supported by -l flag
to bpipe) to be specified in bpipe.config as well
- support for sending emails and
instant message notifications explicitly
from script, using syntax: send text 'hello' to gtalk,
and more advanced forms (documentation to come)
- support for using
explicit regex pattern to split files to parallel
paths (as opposed to simplified pattern syntax)
just add ~ to pattern
- fix (very rare) possibility for duplicate file
names when referencing 'inputs'
variable
- added support for matching numbers
in input splitting patterns ('#')
- added support for gradle wrapper
- added gradle wrapper for easier building
(now just use gradlew instead - of gradle, and it
self downloads)
- fix custom resources not enforced when
provided as map
- fix default build to autodetect Gridgain
jars and compile without if not
found
- get mail.jar from Maven so default checkout compiles
- changes to allow "single" queue to handle
multiple tasks (vlsci)
- fix 'bpipe status' displaying incorrect command for SLURM, PBS
possibly others
========== Changes for Bpipe Version 0.9.8.6_beta_1 ============
- fixed file(...) function sometimes returning file object
for "<file>/."
- experimental feature: supply closure with ".using(...)" for lazy
evaluation, allowing reference to runtime pipeline variables in
configuration of pipeline stages
- initial implementation of a 'bpipe status' command, to show exactly
what is currently running
- Ability to explicitly cause a
branch to either fail or succeed prematurely
(via "fail" and "succeed" commands)
- added support to transform a
single file pattern to multiple targets
- ability to configure and customize reports from bpipe.config
(custom names, etc.), and to send by email when email notifications
configured
- allow "procs" to be specified as integer in
resource manager config
- fix first line of log file sometimes not showing when using
"bpipe log ..."
- "bpipe query" not working on files in non default output
directories (eg: specified with output.dir=....)
- more graceful / better handling of errors
that cause the script to not parse or load at all
- support for finding bpipe.config in the same directory as
the pipeline script (which might be different to the
directory that bpipe is running in)
- better handling of exception when probing tools in tool database
- new feature: allow for muliply by Map to set an explicit
mapping of branch names to inputs
- special behavior for accompanies when dealing with
transforms: always adds
extension to end of file rather than replacing
- allow requires to specify complete file name instead of just
extension
- allow resolution of 'load' files relative to the pipeline
script itself so that it is easy to maintain a pipeline
file and ancillary files in the same directory
========== Changes for Bpipe Version 0.9.8.5 ============
- fix output.dir not working if transform annotation used
- fix silly mistake causing load external stages to fail
- fix loading a pipeline stage from within a loaded pipeline stage not
working
- added ability to set branch level variables that are carried forward
through the pipeline via branch.<name> = "something" syntax
- Fix output sometimes not fully flushed when running pipelines using
resource managers
- support for using arbitrary lists for parallelization over any
list of items
- fix msg writing output during probes (causing double output in
certain situations)
- when an output that is specified as an absolute / relative file that exists,
fix not seeing it in dependency graph
- better file names when output extension occurs in combination with
introduction of branch name (foo.chr1.csv instead of foo.txt.chr1.csv)
NOTE: in rare situations this might cause output file names to change,
thus may cause outputs to get recomputed for existing pipelines (in rare cases)
- allow command line region to override chr() within pipeline
- fix command configuration sometimes not displaying name
- slightly better description of command for specifying report
template (see Google Group discussion)
- fix R code failing when last line contains certain syntax
- Enable specifying file name for reports (Patch from Ido Tamir,
thank you!)
- fix problems when multiple implicit outputs computed from same input file
(output1.foo and output2.foo end up the same file)
- handle incorrect report template much more nicely than a giant stack
trace
========== Changes for Bpipe Version 0.9.8.4.1 ============
- fix middle parallel section causing spurious error if no inputs provided
at start of pipeline
- fix error when pipeline starts with parallel section and adds a
subsequent stage (regression)
- fix file splitting patterns not working properly for very specific case
of a split wildcard (%) followed by 2 chars
========== Changes for Bpipe Version 0.9.8.4 ============
- fix __bpipe_lazy_resource_threads__ appearing in
output when commands fail instead of real value of $threads
- fix __bpipe_lazy_resource_threads__ appearing in
command log instead of real value of $thread
- prevented tools being probed more often than needed
- moved throttling of command execution into separate
class, applied throttling to tool probing so as to lessen chances
of exhausting threads or file handles when running with
large numbers of inputs and highly parallel pipelines
- fix log.warn causing any UNKNOWN status code return
from PBS/Torque to exit with error instead of retrying
- fix filter transforms sometimes basing output file
on wrong input
- preliminary SLURM support via Andrew Lonsdale (thanks!)
- when a command is configured with to use a specific