op-batcher: Wait for queue to drain before shutdown

[mslipper]

Tests are flaking because the batcher's txmgr still polls for receipts after shutdown. Those polls emit logs, which then cause the tests to panic. To fix, I updated the batcher shutdown code to wait for all pending transactions in the tx queue to complete.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Project coverage is 42.80%. Comparing base (9345537) to head (4d350e8).
Report is 19 commits behind head on develop.

Files with missing lines	Patch %	Lines
op-batcher/batcher/driver.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop   #13172      +/-   ##
===========================================
- Coverage    44.38%   42.80%   -1.59%     
===========================================
  Files          801      745      -56     
  Lines        72026    67364    -4662     
===========================================
- Hits         31972    28836    -3136     
+ Misses       37453    36097    -1356     
+ Partials      2601     2431     -170     

Flag	Coverage Δ
cannon-go-tests-32	?
cannon-go-tests-64	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
op-batcher/batcher/driver.go	3.79% <0.00%> (-0.02%)	⬇️

... and 64 files with indirect coverage changes

[axelKingsley]

Do you think we could make the tests permissive instead of making the TxMgr wait? We eliminated a lot of batcher waiting ~6mo ago because it doesn't need to wait (and in fact the waiting sometimes caused operational challenges in restart)

cc @sebastianst

[mslipper]

My comment was a bit misleading. The transactions won't actually be sent in the case of shutdown - the context gets cancelled, which will in turn cancel any pending sends. This just waits for the goroutines to exit, which will help with test stability and is a best practice.