[Metricbeat/Filebeat] mapping exception in golang module

[andrewkroh]

The metricbeat golang module is emitting mapping exceptions:

mapper [golang.filebeat.memstats.BySize.Frees] cannot be changed from type [float] to [long]

For confirmed bugs, please report:

• Version: master (2020-06-25)
• Operating System: MacOS connected to Filebeat running on Windows built with Go 1.13
• Discuss Forum URL:
• Steps to Reproduce:

2020-06-25T18:45:19.374-0400 WARN [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfb5681f9338ce90, ext:57865987, loc:(*time.Location)(0x83d8900)}, Meta:null, Fields:{"agent":{"ephemeral_id":"600ded17-b8ad-45af-b5ff-422beb7461e0","id":"7fccf1c5-aa19-48b1-a01f-54e761603dc1","name":"mac15","type":"metricbeat","version":"8.0.0"},"ecs":{"version":"1.5.0"},"event":{"dataset":"golang.filebeat","duration":3125846,"module":"golang"},"golang":{"filebeat":{"beat.cpu.system.ticks":133656.000000,"beat.cpu.system.time.ms":133656.000000,"beat.cpu.total.ticks":274781.000000,"beat.cpu.total.time.ms":274781.000000,"beat.cpu.total.value":274781.000000,"beat.cpu.user.ticks":141125.000000,"beat.cpu.user.time.ms":141125.000000,"beat.handles.open":402.000000,"beat.info.ephemeral_id":"d6ffd055-7bfc-4e9f-aecc-dd2ef6b9cd4a","beat.info.uptime.ms":1268257.000000,"beat.memstats.gc_next":46626464.000000,"beat.memstats.memory_alloc":39269104.000000,"beat.memstats.memory_total":25175470512.000000,"beat.memstats.rss":80547840.000000,"beat.runtime.goroutines":165.000000,"cmdline":"C:\\Gopath\\src\\github.com\\elastic\\beats\\filebeat\\filebeat.exe -e --httpprof 0.0.0.0:6060 -c .\\filebeat.tcp.yml","filebeat.events.active":4257.000000,"filebeat.events.added":2234616.000000,"filebeat.events.done":2230359.000000,"filebeat.harvester.closed":0.000000,"filebeat.harvester.open_files":0.000000,"filebeat.harvester.running":0.000000,"filebeat.harvester.skipped":0.000000,"filebeat.harvester.started":0.000000,"filebeat.input.log.files.renamed":0.000000,"filebeat.input.log.files.truncated":0.000000,"libbeat.config.module.running":0.000000,"libbeat.config.module.starts":0.000000,"libbeat.config.module.stops":0.000000,"libbeat.config.reloads":0.000000,"libbeat.config.scans":0.000000,"libbeat.output.events.acked":2230359.000000,"libbeat.output.events.active":2048.000000,"libbeat.output.events.batches":1211.000000,"libbeat.output.events.dropped":0.000000,"libbeat.output.events.duplicates":0.000000,"libbeat.output.events.failed":0.000000,"libbeat.output.events.toomany":0.000000,"libbeat.output.events.total":2232407.000000,"libbeat.output.read.bytes":0.000000,"libbeat.output.read.errors":0.000000,"libbeat.output.type":"console","libbeat.output.write.bytes":930264450.000000,"libbeat.output.write.errors":0.000000,"libbeat.pipeline.clients":1.000000,"libbeat.pipeline.events.active":4117.000000,"libbeat.pipeline.events.dropped":0.000000,"libbeat.pipeline.events.failed":0.000000,"libbeat.pipeline.events.filtered":0.000000,"libbeat.pipeline.events.published":2234475.000000,"libbeat.pipeline.events.retry":0.000000,"libbeat.pipeline.events.total":2234476.000000,"libbeat.pipeline.queue.acked":2230359.000000,"memstats":{"Alloc":39354592.000000,"BuckHashSys":1479026.000000,"BySize":[{"Frees":0.000000,"Mallocs":0.000000,"Size":0.000000},{"Frees":2236416.000000,"Mallocs":2239952.000000,"Size":8.000000},{"Frees":15653283.000000,"Mallocs":15684874.000000,"Size":16.000000},{"Frees":2250546.000000,"Mallocs":2259889.000000,"Size":32.000000},{"Frees":13396167.000000,"Mallocs":13426476.000000,"Size":48.000000},{"Frees":2237944.000000,"Mallocs":2243115.000000,"Size":64.000000},{"Frees":3679.000000,"Mallocs":3766.000000,"Size":80.000000},{"Frees":11172004.000000,"Mallocs":11183302.000000,"Size":96.000000},{"Frees":7562.000000,"Mallocs":7739.000000,"Size":112.000000},{"Frees":2859.000000,"Mallocs":2965.000000,"Size":128.000000},{"Frees":762.000000,"Mallocs":784.000000,"Size":144.000000},{"Frees":438.000000,"Mallocs":671.000000,"Size":160.000000},{"Frees":75.000000,"Mallocs":133.000000,"Size":176.000000},{"Frees":32.000000,"Mallocs":50.000000,"Size":192.000000},{"Frees":1027.000000,"Mallocs":1202.000000,"Size":208.000000},{"Frees":405.000000,"Mallocs":419.000000,"Size":224.000000},{"Frees":1261.000000,"Mallocs":1306.000000,"Size":240.000000},{"Frees":1063.000000,"Mallocs":1163.000000,"Size":256.000000},{"Frees":13385962.000000,"Mallocs":13411777.000000,"Size":288.000000},{"Frees":471.000000,"Mallocs":589.000000,"Size":320.000000},{"Frees":372.000000,"Mallocs":445.000000,"Size":352.000000},{"Frees":33.000000,"Mallocs":324.000000,"Size":384.000000},{"Frees":130.000000,"Mallocs":148.000000,"Size":416.000000},{"Frees":10.000000,"Mallocs":33.000000,"Size":448.000000},{"Frees":10.000000,"Mallocs":16.000000,"Size":480.000000},{"Frees":4764.000000,"Mallocs":4793.000000,"Size":512.000000},{"Frees":376.000000,"Mallocs":446.000000,"Size":576.000000},{"Frees":199.000000,"Mallocs":1182.000000,"Size":640.000000},{"Frees":26.000000,"Mallocs":38.000000,"Size":704.000000},{"Frees":22.000000,"Mallocs":30.000000,"Size":768.000000},{"Frees":113.000000,"Mallocs":155.000000,"Size":896.000000},{"Frees":898.000000,"Mallocs":1064.000000,"Size":1024.000000},{"Frees":77.000000,"Mallocs":122.000000,"Size":1152.000000},{"Frees":131.000000,"Mallocs":182.000000,"Size":1280.000000},{"Frees":27.000000,"Mallocs":38.000000,"Size":1408.000000},{"Frees":40.000000,"Mallocs":64.000000,"Size":1536.000000},{"Frees":104.000000,"Mallocs":134.000000,"Size":1792.000000},{"Frees":114464.000000,"Mallocs":114570.000000,"Size":2048.000000},{"Frees":48.000000,"Mallocs":116.000000,"Size":2304.000000},{"Frees":134.000000,"Mallocs":207.000000,"Size":2688.000000},{"Frees":9.000000,"Mallocs":37.000000,"Size":3072.000000},{"Frees":6.000000,"Mallocs":10.000000,"Size":3200.000000},{"Frees":7.000000,"Mallocs":10.000000,"Size":3456.000000},{"Frees":116115.000000,"Mallocs":116359.000000,"Size":4096.000000},{"Frees":101.000000,"Mallocs":116.000000,"Size":4864.000000},{"Frees":114301.000000,"Mallocs":114379.000000,"Size":5376.000000},{"Frees":120.000000,"Mallocs":125.000000,"Size":6144.000000},{"Frees":7.000000,"Mallocs":9.000000,"Size":6528.000000},{"Frees":114274.000000,"Mallocs":114341.000000,"Size":6784.000000},{"Frees":1.000000,"Mallocs":1.000000,"Size":6912.000000},{"Frees":25.000000,"Mallocs":31.000000,"Size":8192.000000},{"Frees":114605.000000,"Mallocs":114677.000000,"Size":9472.000000},{"Frees":13.000000,"Mallocs":13.000000,"Size":9728.000000},{"Frees":9.000000,"Mallocs":10.000000,"Size":10240.000000},{"Frees":26.000000,"Mallocs":30.000000,"Size":10880.000000},{"Frees":113340.000000,"Mallocs":113403.000000,"Size":12288.000000},{"Frees":40.000000,"Mallocs":942.000000,"Size":13568.000000},{"Frees":18.000000,"Mallocs":28.000000,"Size":14336.000000},{"Frees":226681.000000,"Mallocs":226807.000000,"Size":16384.000000},{"Frees":121.000000,"Mallocs":122.000000,"Size":18432.000000},{"Frees":5.000000,"Mallocs":6.000000,"Size":19072.000000}],"DebugGC":false,"EnableGC":true,"Frees":63868494.000000,"GCCPUFraction":0.002516,"GCSys":2148096.000000,"HeapAlloc":39354592.000000,"HeapIdle":14131200.000000,"HeapInuse":41410560.000000,"HeapObjects":122194.000000,"HeapReleased":5570560.000000,"HeapSys":55541760.000000,"LastGC":1593125117996026624.000000,"Lookups":0.000000,"MCacheInuse":6816.000000,"MCacheSys":16384.000000,"MSpanInuse":368696.000000,"MSpanSys":425984.000000,"Mallocs":63990688.000000,"NextGC":46626464.000000,"NumForcedGC":0.000000,"NumGC":1523.000000,"OtherSys":837510.000000,"PauseEnd":[1593124971836623104.000000,1593124972363464704.000000,1593124972920140032.000000,1593124973435265280.000000,1593124974010950400.000000,1593124974513875200.000000,1593124975048547840.000000,1593124975587118336.000000,1593124976095432704.000000,1593124976643744512.000000,1593124977142941696.000000,1593124977701356800.000000,1593124978216510976.000000,1593124978798532096.000000,1593124979296092160.000000,1593124979878152704.000000,1593124980397174016.000000,1593124980990457344.000000,1593124981506568192.000000,1593124982136449536.000000,1593124982653213440.000000,1593124983220930816.000000,1593124983726279936.000000,1593124984336633856.000000,1593124984882545152.000000,1593124985524641024.000000,1593124986125227776.000000,1593124986678936320.000000,1593124987271714304.000000,1593124987824915712.000000,1593124988408438272.000000,1593124988968964864.000000,1593124989538321664.000000,1593124990093490944.000000,1593124990679458816.000000,1593124991266357248.000000,1593124991827876608.000000,1593124992382544896.000000,1593124992981219328.000000,1593124993590085632.000000,1593124994189705216.000000,1593124994784944896.000000,1593124995382079488.000000,1593124996018318336.000000,1593124996563232512.000000,1593124997129637888.000000,1593124997659408128.000000,1593124998257588224.000000,1593124998799057664.000000,1593124999363255808.000000,1593124999880676096.000000,1593125000407591680.000000,1593125000956799488.000000,1593125001502712576.000000,1593125002119895552.000000,1593125002655517952.000000,1593125003172615424.000000,1593125003696068608.000000,1593125004235113728.000000,1593125004767837184.000000,1593125005305906176.000000,1593125005831770112.000000,1593125006378653184.000000,1593125006933362688.000000,1593125007469504000.000000,1593125008040769536.000000,1593125008576023552.000000,1593125009128172288.000000,1593125009656551168.000000,1593125010198001152.000000,1593125010701435136.000000,1593125011195580160.000000,1593125011790315520.000000,1593125012741974272.000000,1593125013284957952.000000,1593125013861096192.000000,1593125014370904320.000000,1593125014955335936.000000,1593125015459385856.000000,1593125016039406080.000000,1593125016536876288.000000,1593125017115004928.000000,1593125017638459904.000000,1593125018255124480.000000,1593125018832779520.000000,1593125019530559232.000000,1593125020118912000.000000,1593125020653121280.000000,1593125021221478656.000000,1593125021791485440.000000,1593125022379700224.000000,1593125022947551488.000000,1593125023522228480.000000,1593125024111221248.000000,1593125024644822272.000000,1593125025223445760.000000,1593125025776666112.000000,1593125026333289984.000000,1593125026848422656.000000,1593125027388464384.000000,1593125027947541504.000000,1593125028508678656.000000,1593125029062780928.000000,1593125029581834496.000000,1593125030141892608.000000,1593125030684440832.000000,1593125031274258944.000000,1593125031792765952.000000,1593125032369159424.000000,1593125032901672960.000000,1593125033467590144.000000,1593125034027163392.000000,1593125034574597120.000000,1593125035152654336.000000,1593125035744099328.000000,1593125036345544704.000000,1593125036995409664.000000,1593125037536920320.000000,1593125038135549184.000000,1593125038683909632.000000,1593125039306516480.000000,1593125039842384640.000000,1593125040431958784.000000,1593125040960782080.000000,1593125041526177536.000000,1593125042046815488.000000,1593125042631177216.000000,1593125043185404160.000000,1593125043776236288.000000,1593125044294315008.000000,1593125044934903040.000000,1593125045499840000.000000,1593125046140961280.000000,1593125046766029568.000000,1593125047347510784.000000,1593125048030625536.000000,1593125048590193920.000000,1593125049281105664.000000,1593125049922712320.000000,1593125050557128704.000000,1593125051225199360.000000,1593125051921723136.000000,1593125052664043264.000000,1593125053390494208.000000,1593125054119004928.000000,1593125054795856896.000000,1593125055515998208.000000,1593125056160978176.000000,1593125056808454656.000000,1593125057481908736.000000,1593125058139011840.000000,1593125058802940928.000000,1593125059478867456.000000,1593125060134632960.000000,1593125060777698816.000000,1593125061433264384.000000,1593125062119478272.000000,1593125062770205696.000000,1593125063482704384.000000,1593125064313870592.000000,1593125065934935296.000000,1593125067074110208.000000,1593125067961370880.000000,1593125068690804480.000000,1593125069539491328.000000,1593125070388050432.000000,1593125071146073088.000000,1593125072001359872.000000,1593125072806054144.000000,1593125073551964672.000000,1593125074390048512.000000,1593125075201078016.000000,1593125075959907328.000000,1593125076712789504.000000,1593125077494539264.000000,1593125078232309248.000000,1593125078958444544.000000,1593125079686913280.000000,1593125080402728704.000000,1593125081094169088.000000,1593125081822643456.000000,1593125082582434816.000000,1593125083323119616.000000,1593125084047739392.000000,1593125084721566720.000000,1593125085385159936.000000,1593125086075590912.000000,1593125086719620096.000000,1593125087392485888.000000,1593125088089801216.000000,1593125088710839040.000000,1593125089418846720.000000,1593125090084403968.000000,1593125090667451648.000000,1593125091319429888.000000,1593125091918855680.000000,1593125092487703808.000000,1593125093047272960.000000,1593125093626887680.000000,1593125094200811264.000000,1593125094743569920.000000,1593125095305579776.000000,1593125095833384960.000000,1593125096361730304.000000,1593125096880887808.000000,1593125097501622016.000000,1593125098027258624.000000,1593125098567297792.000000,1593125099157125376.000000,1593125099717714176.000000,1593125100305104128.000000,1593125100839311104.000000,1593125101392503552.000000,1593125101959423488.000000,1593125102515046912.000000,1593125103106348800.000000,1593125103749934592.000000,1593125104303641344.000000,1593125104944270336.000000,1593125105399823872.000000,1593125105980401408.000000,1593125106479423744.000000,1593125107048265472.000000,1593125107542179328.000000,1593125108142981120.000000,1593125108644950272.000000,1593125109209888000.000000,1593125109716268288.000000,1593125110295847680.000000,1593125110844637952.000000,1593125111427667712.000000,1593125111947790592.000000,1593125112485771776.000000,1593125113039487232.000000,1593125113595135488.000000,1593125114161062400.000000,1593125114730395648.000000,1593125115289006336.000000,1593125115821720320.000000,1593125116356880640.000000,1593125116942818816.000000,1593125117451120384.000000,1593125117996026624.000000,1593124964487478272.000000,1593124965118825984.000000,1593124965673513472.000000,1593124966296622080.000000,1593124966812226816.000000,1593124967467969792.000000,1593124967996758528.000000,1593124968593947392.000000,1593124969137407232.000000,1593124969684257024.000000,1593124970211133184.000000,1593124970761659392.000000,1593124971275621888.000000],"PauseNs":[0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,957100.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,983400.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,510700.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,1565500.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,978900.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,985600.000000,0.000000,463200.000000,0.000000,0.000000,0.000000,225100.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,1739300.000000,0.000000,163400.000000,0.000000,0.000000,0.000000,1197100.000000,0.000000,0.000000,940300.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,1516700.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,1226700.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,952600.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,1000400.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,931200.000000,1444400.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,964100.000000,979400.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000],"PauseTotalNs":419309900.000000,"StackInuse":3178496.000000,"StackSys":3178496.000000,"Sys":63627256.000000,"TotalAlloc":25175556000.000000},"registrar.states.cleanup":0.000000,"registrar.states.current":0.000000,"registrar.states.update":0.000000,"registrar.writes.fail":0.000000,"registrar.writes.success":0.000000,"registrar.writes.total":0.000000,"system.cpu.cores":4.000000}},"host":{"name":"myhost"},"metricset":{"name":"expvar","period":10000},"service":{"address":"192.168.135.50:6060","type":"golang"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"mapper [golang.filebeat.memstats.BySize.Frees] cannot be changed from type [float] to [long]"}

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[andrewkroh]

The expvar metricset does not have a mapping. Plus it allows for a custom namespace (golang.filebeat in my case) so mapping those fields is even more of a challenge.

- module: golang
  metricsets:
    - expvar
    - heap
  period: 10s
  hosts: ["192.168.135.50:6060"]
  heap.path: "/debug/vars"
  expvar:
    namespace: "filebeat"
    path: "/debug/vars"

[graphaelli]

awkward to +1 this since it's not awesome, but I just ran into this too mapper [golang.apm-server.memstats.BySize.Frees] cannot be changed from type [float] to [long]

[rfraposa]

FWIW - this is still an issue. I am running the Stack version 7.10 in CentOS Docker containers. I get the following error when trying to index metrics using the Metricbeat golang module OOTB:

"host":{"architecture":"x86_64","containerized":true,"hostname":"checkoutservice","ip":["172.19.0.18"],"mac":["02:42:ac:13:00:12"],"name":"checkoutservice","os":{"codename":"Core","family":"redhat","kernel":"4.9.184-linuxkit","name":"CentOS Linux","platform":"centos","version":"7 (Core)"}},"metricset":{"name":"expvar","period":10000},"service":{"address":"http://localhost:6060/debug/vars","type":"golang"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"mapper [golang.example.memstats.PauseEnd] cannot be changed from type [long] to [float]"}

[wangyoucao577]

Met same issue with docker.elastic.co/beats/metricbeat:7.12.0.

{"type":"illegal_argument_exception","reason":"mapper [golang.example.memstats.BySize.Frees] cannot be changed from type [float] to [long]"}

[mdaliyan]

same here

(status=400): {"type":"illegal_argument_exception","reason":"mapper [golang.expvar.memstats.PauseEnd] cannot be changed from type [long] to [float]"}

I had faced it before in my apps. Our solution was to add .0 to the floats without a decimal point when we were generating JSON

[nerophon]

I've now seen this happen in Filebeat as well. Documents come in as floats, then later some come in looking like "5" or "12". These should really be converted to floats. No need to fail or require the user to sanitise manually.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[thalesfsp]

So, any update on this?

[thibleroy]

Met same issue as @mdaliyan with 8.8.2
mapper [golang.expvar.memstats.PauseEnd] cannot be changed from type [float] to [long]\"}, dropping event!

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!