Error when parsing "CC" from "message-headers" with Mailgun adapter

[wingice]

Mailgun adapter: receiving mail without CC field:

It seems params["message-headers"] is a string , not an array.

Started POST "/email_processor" for 127.0.0.1 at 2014-04-01 17:11:46 -1000
Processing by Griddler::EmailsController#create as HTML
  Parameters: {"recipient"=>"[email protected]",
  "sender"=>"[email protected]", "subject"=>"FW: xxx",
  "from"=>"\"R&D Recruit Mail\" ",
  "Received"=>"from aaaaa ([180.16.63.178])\tby 21CN-ent7-3(MEDUSA 10.27.101.7) with ESMTP id 1396407625.17611 for [email protected]\t;\tWed Apr 2 11:00:43 2014",
  "X-Envelope-From"=>"", "Hmm_source_ip"=>"10.27.101.7:36370.1075920735", "Hmm_attache_num"=>"0001",
  "Hmm_source_type"=>"SMTP", "0/x-Total-Score"=>"0:", "3/x-Brightmail-Tracker"=>"AAAAAA==", "X-Filter-Score"=>"to=<9595959594854f828594836195828d868f958d8a9495944f84908e>, score=<1396407643RPt4uhjjjjjTjj2jjLjT2LHEuf5Qbbbbb9bbCbbdb9Cd> ",
  "X-Real-From"=>"[email protected]", "X-Receive-Ip"=>"180.166.63.178 [email protected]",
  "From"=>"\"R&D Recruit Mail\" ",
  "To"=>"",
  "Subject"=>"FW: aaa", "Date"=>"Wed, 2 Apr 2014 11:00:37 +0800",
  "Message-Id"=>"<[email protected]>",
  "Mime-Version"=>"1.0",
  "Content-Type"=>"multipart/mixed; boundary=\"----=_NextPart_000_0013_01CF4E62.D00C1CD0\"",
  "X-Mailer"=>"Microsoft Office Outlook 12.0",
  "Thread-Index"=>"Ac9IL7jpKzuy9FReQkqj6X93yR9k4AF7/k6Q",
  "Content-Language"=>"zh-cn",
  "X-Mailgun-Incoming"=>"Yes",
  "X-Mailgun-Sflag"=>"No",
  "X-Mailgun-Sscore"=>"0.0",
  "X-Mailgun-Spf"=>"Neutral",
  "message-headers"=>"[[\"Received\", \"by luna.mailgun.net with SMTP mgrt 8766208669373; Wed, 02 Apr 2014 03:00:55 +0000\"], [\"X-Envelope-From\", \"\"], [\"Received\", \"from corp.21cn.com (corp.forptr.21cn.com [121.14.129.51]) by mxa.mailgun.org with ESMTP id 533b7d5e.7f508c1bd960-in2; Wed, 02 Apr 2014 03:00:46 -0000 (UTC)\"], ....]"
Completed 500 Internal Server Error in 0.5ms

NoMethodError (private method `select' called for #):
  griddler (0.6.4) lib/griddler/adapters/mailgun_adapter.rb:42:in `extract_header_cc'
  griddler (0.6.4) lib/griddler/adapters/mailgun_adapter.rb:36:in `ccs'
  griddler (0.6.4) lib/griddler/adapters/mailgun_adapter.rb:16:in `normalize_params'
  griddler (0.6.4) lib/griddler/adapters/mailgun_adapter.rb:10:in `normalize_params'
  griddler (0.6.4) app/controllers/griddler/emails_controller.rb:12:in `normalized_params'
  griddler (0.6.4) app/controllers/griddler/emails_controller.rb:3:in `create'
  actionpack (3.2.16) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
  actionpack (3.2.16) lib/abstract_controller/base.rb:167:in `process_action'
  actionpack (3.2.16) lib/action_controller/metal/rendering.rb:10:in `process_action'
  actionpack (3.2.16) lib/abstract_controller/callbacks.rb:18:in `block in process_action'
  activesupport (3.2.16) lib/active_support/callbacks.rb:403:in `_run__3226707876278453871__process_action__698180469685376029__callbacks'
  activesupport (3.2.16) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.16) lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
  activesupport (3.2.16) lib/active_support/callbacks.rb:81:in `run_callbacks'
  actionpack (3.2.16) lib/abstract_controller/callbacks.rb:17:in `process_action'

[wingice]

I fix the the issue by adding extra checking:

 -        headers = params['message-headers'].select do |h|
 +        message_headers = params['message-headers']
 +        message_headers = JSON.parse(message_headers) if message_headers.is_a?(String)
 +
 +        headers = message_headers.select do |h|

I don't know whether it is an acceptable fix. If it is ok, I can make a pull request from my fork.

[wingice]

I saw a warning in the log, it may be the cause: not translating string to Json.

There are some unicodes chars(UTF-8, Chinese) in those fields.

WARNING: Could not parse (and so ignoring) '[["Received", "by luna.mailgun.net with SMTP mgrt 8775751066621; Fri, 04 Apr 2014 08:15:34 +0000"], ["X-Envelope-From", ""], ["Received", "from corp.21cn.com (corp.forptr.21cn.com [121.14.129.38]) by mxa.mailgun.org with ESMTP id 533e6a1d.477bb20-in2; Fri, 04 Apr 2014 08:15:25 -0000 (UTC)"], ["Received", "from ip?180.166.63.178? (entas7.inner-hermes.com [10.27.101.7])\tby corp.21cn.com (HERMES) with ESMTP id 69DF9484033\tfor ; Fri, 4 Apr 2014 16:15:10 +0800 (CST)"], ["Hmm_attache_num", "0001"], ["Hmm_source_ip", "wmail.10.27.101.7.933431481"], ["Hmm_source_type", "WEBMAIL"], ["Received", "from ip<180.166.63.178> ([180.166.63.178])\tby 21CN-ent7(MEDUSA 10.27.101.7) with ESMTP id 1396599307.27191 for [email protected]\t;\tFri Apr 4 16:15:17 2014"], ["1/x-Total-Score", "-62:"], ["X-Filter-Score", "to=<918c959398979890894f89868d8d90958684898f908d90888a86946195828d868f958d8a9495944f84908e>, score=<1396599317Vo8FwwwwwHwwPwwswHPsXzGU60VVVVV7VVBVVYV7BYDa> "], ["X-Real-From", "[email protected]"], ["X-Receive-Ip", "180.166.63.178 [email protected]"], ["Date", "Fri, 4 Apr 2014 16:15:07 +0800 (CST)"], ["From", "[email protected]"], ["To", "\"pktrwvwoh.hellotechnologies\" "], ["Message-Id", "<172189816.6421396599310618.JavaMail.hermes@ent-web5>"], ["Subject", "Fw: (51job.com)\u7533\u8bf7\u8d35\u516c\u53f8C# .Net \u9ad8\u7ea7\u8f6f\u4ef6\u5de5\u7a0b\u5e08\uff08\u4e0a\u6d77\uff09\uff0d\u77f3\u946b"], ["Mime-Version", "1.0"], ["Content-Type", "multipart/mixed; boundary=\"----=_Part_673_1412613781.1396599307337\""], ["Hmm_webcln_ip", "10.27.10.86"], ["X-Hermes-Sendmode", "normal"], ["X-Hermes-Set", "CC2fLruzRNbroHaQFnf1nCpRJnBb8J7v"], ["X-Mailgun-Incoming", "Yes"], ["X-Mailgun-Sflag", "No"], ["X-Mailgun-Sscore", "0.0"], ["X-Mailgun-Spf", "Neutral"]]'

[marine44]

I can also replicate this issue via https://mailgun.com/cp/routes using the "Send A Sample POST" test function.

I have installed tested your changes and they has resolved the problem for me - so thanks very much!!!

I say you should make a pull request.

[calebhearth]

Moved to wingrunr21/griddler-mandrill#1

[wingrunr21]

@calebthompson my adapter is mandrill not mailgun

[calebhearth]

Whoops!

[bradpauly]

mailgun sends the headers as JSON and this is fixed in the extracted version.

[calebhearth]

Great, so should this be closed as fixed for 1.0 and on master?

[calebhearth]

Or, I guess along with the apart to remove the adapter from Griddler.

[bradpauly]

Sorry I didn't get back about this sooner. I'd also like to fix this in the non-extracted version if that would be helpful. Is the best way to do that to check out the version tagged 0.6.4 and make a branch to fix in from there?

[calebhearth]

Yeah, and we could cut 0.6.5 for that, but I'd intended to EOL 0.6.* as soon as 1.0.0 came out, which should be in the next couple of days. If you feel strongly, we can do it and cut a release. Up to you @bradpauly

[bradpauly]

Hmm, it's in a branch but I don't want to create extra work if it isn't needed. @ryanharnwell @wingice do you plan to move to 1.0? Or are there any reasons you don't want to upgrade?

[marine44]

I'd be happy to upgrade to 1.0 when released.

[calebhearth]

This should be good in the released version.