Refactor bulk API to minimize allocations and increase speed.

[scunningham]

What does this PR do?

Cleans up the bulk API. Implement some missing methods. Minimizes heap allocations and memory size.

Why is it important?

Fleet Server must run in small memory environments. Resource usage must be minimized in those enviroments.

Checklist

• [x ] My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [x ] I have made corresponding changes to the documentation
• [x ] I have made corresponding change to the default configuration files
• [x ] I have added tests that prove my fix is effective or that my feature works

Logs

Before change (benchmarks were back ported):

$ go test --bench .  ./internal/pkg/bulk/ --test.run wee
goos: darwin
goarch: amd64
pkg: github.com/elastic/fleet-server/v7/internal/pkg/bulk
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
BenchmarkMockBulk/1-16     	   52951	     21825 ns/op	   12925 B/op	      62 allocs/op
BenchmarkMockBulk/8-16     	   14673	     81697 ns/op	   30801 B/op	     167 allocs/op
BenchmarkMockBulk/64-16    	    1796	    666980 ns/op	  197584 B/op	     969 allocs/op
BenchmarkMockBulk/4096-16  	      27	  50131378 ns/op	13495313 B/op	   57770 allocs/op
BenchmarkMockBulk/32768-16 	       3	 388034080 ns/op	111144600 B/op	  480312 allocs/op
BenchmarkMultiUpdate/1-16  	  877872	      1364 ns/op	     464 B/op	       7 allocs/op
BenchmarkMultiUpdate/8-16  	  166984	      6944 ns/op	    2672 B/op	      14 allocs/op
BenchmarkMultiUpdate/64-16 	   28059	     43555 ns/op	   21874 B/op	      70 allocs/op
BenchmarkMultiUpdate/4096-16         	     471	   2556287 ns/op	 1310981 B/op	    4102 allocs/op
BenchmarkMultiUpdate/32768-16        	      61	  21555759 ns/op	10486973 B/op	   32776 allocs/op
BenchmarkMultiUpdate/131072-16       	      14	  79094447 ns/op	41945282 B/op	  131081 allocs/op
PASS
ok  	github.com/elastic/fleet-server/v7/internal/pkg/bulk	17.244s

After this PR:

$ go test --bench .  ./internal/pkg/bulk/ --test.run wee
goos: darwin
goarch: amd64
pkg: github.com/elastic/fleet-server/v7/internal/pkg/bulk
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
BenchmarkMockBulk/1-16     	   69457	     16172 ns/op	    5828 B/op	      30 allocs/op
BenchmarkMockBulk/8-16     	   20311	     58576 ns/op	   14468 B/op	      53 allocs/op
BenchmarkMockBulk/64-16    	    3462	    342910 ns/op	   61187 B/op	     223 allocs/op
BenchmarkMockBulk/4096-16  	      31	  42822239 ns/op	 3637939 B/op	   13101 allocs/op
BenchmarkMockBulk/32768-16 	       4	 309354163 ns/op	33611394 B/op	  147244 allocs/op
BenchmarkMultiUpdate/1-16  	  955185	      1182 ns/op	     352 B/op	       6 allocs/op
BenchmarkMultiUpdate/8-16  	  242350	      5032 ns/op	    1744 B/op	       6 allocs/op
BenchmarkMultiUpdate/64-16 	   45374	     26689 ns/op	   12817 B/op	       6 allocs/op
BenchmarkMultiUpdate/4096-16         	     765	   1637018 ns/op	  803055 B/op	       6 allocs/op
BenchmarkMultiUpdate/32768-16        	     100	  12849581 ns/op	 6423245 B/op	       7 allocs/op
BenchmarkMultiUpdate/131072-16       	      22	  48685988 ns/op	25691537 B/op	       8 allocs/op
PASS
ok  	github.com/elastic/fleet-server/v7/internal/pkg/bulk	17.129s

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Branch indexing

• Start Time: 2021-05-10T22:29:06.756+0000

• Duration: 5 min 19 sec

• Commit: ef6ea61

Test stats 🧪

Test	Results
Failed	0
Passed	174
Skipped	0
Total	174

Trends 🧪

[aleksmaus]

one leaked ticker, otherwise just nits