Access to raw post body

[nscharfe]

I'm implementing a webhook handler and need to perform a checksum on the raw post body that is received but cannot figure out how to access it.

Sinatra Equivalent:

post '/my/webhook/url' do
    body = request.body.read
    ...
end

Django Equivalent:

def my_webhook_view(request):
  body = request.body
  ...

Node Equivalent:

app.use(require('body-parser').raw({type: '*/*'}));

app.post('/my/webhook/url', function(request, response) {
  body = request.body;
  ...
});

---

My original implementation was:

message WebhookRequest {
    string body = 1;
}

rpc HandleWebhook(WebhookRequest) returns (google.protobuf.Empty) {
    option (google.api.http) = {
        post: "/public/processWebhook"
        body: "body"
    };
}

This works as expected if the post body is explicitly a string. However, if the post body is a json object, it tries to unmarshal it into body, which is a string, and fails.

Ex post bodies:
"test" => is unmarshaled into WebhookRequest.body appropriately.
{"test": "val"} => "error": "json: cannot unmarshal object into Go value of type string"
"{\"test\": \"val\"}" => is unmarshaled into WebhookRequest.body

Since this is a webhook, I have no control over how the post body is constructed -- its not possible to encode it as a string before sending it over the wire.

I've tried updating WebhookRequest to hold a google.protobuf.Struct, google.protobuf.Any etc but none of them are able to preserve the post body in its raw form.

I'd imagine its possible to accomplish what I want by customizing message serialization but that feels like overkill.

Any help would be greatly appreciated. I'd imagine I'm overlooking something obvious.

Thanks in advance!

[achew22]

Unfortunately I think you're right that this is probably better handled by a custom marshaller. The proto API, as I understand it, isn't supposed to be able to form to every REST request, only to allow an RPC to be exported in a way that any browser can connect to.

If you make a custom marshaller you will have access to the full body message for hashing. I would be interested in adding that as an example marshaller if you would like to share that code.

PS: Why is there a message checksum in the web hook? Is this a common pattern that I've not yet run into?

[nscharfe]

Thanks for the help. Now that I'm looking a bit closer, it may make sense for me to simply attach the body to the context -- headers, metadata etc are already being attached so it may be a natural extension to add body as well for this specific purpose rather than try to shoe horn a new marshaling scheme. That seem reasonable?

As far as verifying signatures, checksums etc. in a webhook handler, I believe it is a common practice to ensure the webhook event was sent by the correct party. For example:
https://stripe.com/docs/webhooks/signatures
In this case, the official stripe lib handles the signing verification internally but expects to receive the raw post payload as input.

[achew22]

Okay, so it is more about signing than hashing. That makes sense for message integrity.

If I were going to implement this, I think I would implement it as a HTTP interceptor in the grpc-gateway, look for the Stripe-Signature header and if it has it, verify it is correct. If it fails that check, immediately return a 400 error (bad request) or a 412 (precondition failed) to the caller and not pass the request into the stack.

[hiromis]

I came across this issue while trying to create an endpoint that takes an arbitrary JSON, modifies one field, and forwards if to another RESTful service.

I considered creating a custom marshaller, but it seems like it's at MIME type level and not for a particular endpoint. All the other endpoints should use the default marshaller, but for one endpoint I would like an access to the body where I do not the structure of in advance. Any advice on how I might able to achieve that?

[rogchap]

Create a wrapper for your endpoint to set a custom MIME Content-Type then have a custom Marshaler for that MIME.

eg:

func customMimeWrapper(h http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if strings.Contains(r.URL.Path, "webhook") {
			r.Header.Set("Content-Type", "application/raw-webhook")
		}
		h.ServeHTTP(w, r)
	})
}

mux = runtime.NewServeMux(
	runtime.WithMarshalerOption("application/raw-webhook", & rawJSONPb{jsonpb}),
	runtime.WithMarshalerOption(runtime.MIMEWildcard, jsonpb),
	runtime.WithProtoErrorHandler(runtime.DefaultHTTPProtoErrorHandler),
)

restHandler = http.NewServeMux()
restHandler.Handle("/", customMimeWrapper(restServer))

[rogchap]

For those trying to do the same think this is what my marshaler consists of:

var typeOfBytes = reflect.TypeOf([]byte(nil))

type rawJSONPb struct {
	*gateway.JSONPb
}

func (*rawJSONPb) NewDecoder(r io.Reader) runtime.Decoder {
	return runtime.DecoderFunc(func(v interface{}) error {
		rawData, err := ioutil.ReadAll(r)
		if err != nil {
			return err
		}
		rv := reflect.ValueOf(v)

		if rv.Kind() != reflect.Ptr {
			return fmt.Errorf("%T is not a pointer", v)
		}

		rv = rv.Elem()
		if rv.Type() != typeOfBytes {
			return fmt.Errorf("Type must be []byte but got %T", v)
		}

		rv.Set(reflect.ValueOf(rawData))
		return nil
	})
}

expects a proto definition that uses bytes as the body:

service WebhookService {
  rpc HandleWebhook(WebhookRequest) returns (google.protobuf.Empty) {
    option (google.api.http) = {
      post: "/myapi/v1/webhook"
      body: "rawData"
    };
  }
}
message WebhookRequest {
  bytes rawData = 1;
}

[hiromis]

@rogchap, it worked like a charm. Thank you!!

[johanbrandhorst]

Can we close this issue?

[nscharfe]

yep, feel free. thank you!

[arturgspb]

@rogchap

I got error with same code on compile:

kind: ERROR
message: "http: body field path \'rawData\' must be a non-repeated message."

Have you had such problems?

[goulashify]

@arturgspb: Having the same issue w/ Cloud Endpoints, did you figure in the end?

[arturgspb]

@goulashify hi, unfortunately I don’t remember the outcome of the problem at all. 😢

[goulashify]

@arturgspb Thanks for the reply! Don't worry, I figured it out. :) For others who use Cloud Endpoints (Envoy-based), this works:

import "google/api/annotations.proto";
import "google/api/httpbody.proto";

service Webhook {
    rpc StripeWebhook (google.api.HttpBody) returns (google.protobuf.Empty) {
        option (google.api.http) = {
            post: "/v1/webhook-stripe",
            body: "*"
        };
    }
}