feature: filter fifoWithBody

[szuecs]

feature: filter fifoWithBody that works similar to fifo(), but release deferred until body streaming to client was finished

[szuecs]

@AlexanderYastrebov maybe better than #2675 wdyt?

[AlexanderYastrebov]

Its not clear how multiple fifoWithBody would chain and how they would interplay with regular fifo.
I think you'd need to have the same logic stored under different keys and then run chain of hooks in two places in proxy depending on the key at proxy.do and proxy.ServeHTTP:

...
func (f *fifoFilter) createResponse(ctx filters.FilterContext, typ string) {
        pending, ok := ctx.StateBag()[typ].([]func())
        if !ok {
	        return
        }
        last := len(pending) - 1
        if last < 0 {
	        return
        }
        pending[last]()
        ctx.StateBag()[fifoKey] = pending[:last]
}

...
// existing in proxy.do
pendingFIFO, _ := stateBag[scheduler.FIFOKey].([]func())
for _, done := range pendingFIFO {
	done()
}
...
// in proxy.ServeHTTP
pendingFIFO, _ := stateBag[scheduler.FIFOWithBodyKey].([]func())
for _, done := range pendingFIFO {
	done()
}

[szuecs]

Yes that's a good point. I also thought about that I need to do something like that.
What do you like more to support the copystream part? this one or the other?

[AlexanderYastrebov]

I think we can remove existing pendingFIFO cleanup from proxy.go see #2680

[AlexanderYastrebov]

You can use filter name as spec.typ and have withBody bool in filter similar to

skipper/filters/diag/histogram.go

Lines 56 to 57 in 2a202c4

	f := &histFilter{
	response: s.typ == filters.HistogramResponseLatencyName,

[szuecs]

I like the enum style better than string.

[szuecs]

close to create a PR to cw2023freeze