m3coordinator with grafana

[tejasbhosale009]

I installed a cluster with 3 node and 1 m3 coordinator
m3coordinator is working fine it can write data point to m3db but i setup grafana with m3coordinator to see data but its not working ..
added data source graphite in grafana as per document in m3db
Note that you'll need to set the URL to: http://<M3_COORDINATOR_HOST_NAME>:7201/api/v1/graphite

Can any one please help why i am not able to view data in grafana ... if i manually pass query in grafana it works but when i try to create any dashboard drop down list is blank and i attached debug tool screen shot with this
carbon debug log
Oct 11 20:20:27 mtc01-tej grafana-server[23913]: t=2019-10-11T20:20:27+1100 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/datasources/proxy/1/metrics/find status=400 remote_addr=192.168.5.81 time_ms=15 size=27 referer="http://mtc01-tej.ops:3000/dashboard/new?panelId=2&edit&fullscreen&orgId=1"
it giving 400 as status

[lz006]

we experienced the same problem. We bypassed this by setting up haproxy as reverse proxy:

Browser -> haproxy -> grafana

where haproxy as kind of man-in-the-middler modifies incoming requests by using following config:

global
    maxconn 50000
    log stdout format raw daemon
    user root
    group root
    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    timeout connect 10s
    timeout client 30s
    timeout server 30s
    log global
    mode http
    log-format "frontend:%f/%H/%fi:%fp\ GMT:%T\  body:%[capture.req.hdr(0)]\ request:%r"
    maxconn 3000

frontend grafana_proxy
    bind 0.0.0.0:30000
    use_backend not_modified if !{ path -i -m end /metrics/find } #do not modify if request concerns other topic
    option http-buffer-request #avoids chunking requests
    declare capture request len 10000 #allocates space (first alloc = id 0)
    http-request capture req.body id 0 #writes available body as a block of data to capture slot 0
    use_backend modified

backend modified
    http-request set-query %[capture.req.hdr(0)]&%[query] if { query -m sub -i = }
    http-request set-query %[capture.req.hdr(0)] if !{ query -m sub -i = }
    server grafana <your_grafana_host:port>

backend not_modified
    server grafana <your_grafana_host:port>

hope this helps you out for the moment.

But of course there shouldn't be necessity for such a workaround.
Hoping this will be fixed in the future

[SUL-sj]

I did some research on this issue. It actually turned out that the Graphite HTTP API implementation of M3DB is not compliant. POST Requests are implemented in the API since 2009 and therefore should be implemented by M3DB, too.
Although the graphite guys agreed that the POST Implementation was completly missing from the documentation so no one could have known...

[ikod]

i have same problem. Is there any chance that this will be fixed?

UPD: looks like this fixed few days ago.

Thanks!

[robskillington]

Thank you for fixing this with #2153 !