Asterisks not encoded properly and result in error 32 "Could not authenticate you."

[jeremyhaile]

Twitter-text (1.11.0) says that tweets with asterisks are valid. And I am also allowed to post them to Twitter directly. However, when posted through the Twitter gem (6.0.0) I receive the error: "Could not authenticate you." (code 32)

Twitter::Validation.tweet_invalid?("*TEST* Does this work?") # returns false (valid tweet)

client.update("*TEST* Does this work?") # raises Twitter::Error::Unauthorized: Could not authenticate you.

client.update("TEST Does this work?") # works

This forum post indicates that asterisks need to be URL encoded to be valid, so shouldn't the Twitter gem be handling this for me?
https://twittercommunity.com/t/asterisk/6343

[geori]

I'm having the same issue when posting @ or # in my search terms. I can't figure out where the conflict is, so I had to revert to 5.13.0 for my app to run properly.

[iandawg]

I've found that the same results occur using the https://dev.twitter.com/rest/tools/console. This seems to be an issue with the API, not with the gem. Any thoughts?

[dredenba]

Is there a work around for this? I thought I would be able to URL encode it manually if this gem doesn't do it but I don't get * when I use %2A

[renatolond]

Sorry to re-raise this topic, but I'm getting the same on master. I am going around temporarily by replacing asterisk by ＊. I also tried what dredenba suggested of using the url encoded version and it does not work.

[alzeih]

I think I've figured out the cause of "Could not authenticate you" with *.

https://github.com/sferik/twitter/blob/master/lib/twitter/rest/request.rb#L28 contains a call to set_multipart_options!(request_method, options) which does

@headers = Twitter::Headers.new(@client, @request_method, @uri, options).request_headers

This creates the oauth http headers, which signs the contents of the request, including the options.

When the request is sent

response = http_client.headers(@headers).public_send(@request_method, @uri.to_s, options_key => @options)

the http library encodes the form data, which is fine in most cases, but with * (and likely some of the other specials in https://tools.ietf.org/html/rfc3986#section-2.2 the http library encodes the values, which changes the contents of the request.

Since the request is now different, the twitter api rejects the request with a 401 Unauthorized.

[alzeih]

Actually, after reading #888, it's the other way around, it's not being encoded.

It seems related to httprb/http#449 , which suggests a workaround.

This patch works for me.

diff --git a/lib/twitter/rest/request.rb b/lib/twitter/rest/request.rb
index db34abbe..1c12c18a 100644
--- a/lib/twitter/rest/request.rb
+++ b/lib/twitter/rest/request.rb
@@ -32,8 +32,8 @@ module Twitter

       # @return [Array, Hash]
       def perform
-        options_key = @request_method == :get ? :params : :form
-        response = http_client.headers(@headers).public_send(@request_method, @uri.to_s, options_key => @options)
+        @uri.query_values = @options
+        response = http_client.headers(@headers).public_send(@request_method, @uri.to_s)
         response_body = response.body.empty? ? '' : symbolize_keys!(response.parse)
         response_headers = response.headers
         fail_or_return_response_body(response.code, response_body, response_headers)

[CedricBm]

Interesting patch @alzeih. However, this passes the tweet body into the URL as a GET parameter. So there are chances that we receive a 414 URI too long error, especially when sending direct messages which can have up to 10 000 characters.

[alzeih]

@CedricBm it also doesn't work with image uploads - although not because of length, but due to how query_values works. This patch is just demonstrating why asterisks cause the "Could not authenticate you" error.

[christianstanfield]

I'm also running into the same issue and can confirm that URI#encode_www_form inside HTTP::FormData does not convert asterisks, whereas Twitter::Headers is relying on the gem simple_oauth which is using URI::Parser and is encoding asterisks: URI::Parser.new.escape('*', /[^a-z0-9\-\.\_\~]/i) => "%2A". (https://github.com/laserlemon/simple_oauth/blob/be8a895c73a433dce47aa6ca37dd03a0a343fd55/lib/simple_oauth/header.rb#L29) Looks very much like there's an encoding incompatibility between these two.

[FabienChaynes]

Hi,

The 2.1.1 version of http-form_data includes a fix for this case discussed in httprb/form_data#22 and implemented in httprb/form_data@3bf2e83.

It allows to override the http-form_data encoding method.

The following snippet overrides the encoding to act like the simple_oauth gem and thus fix the issue (you can put it in config/initializers/http_form_data.rb for example):

require "uri"

# The encoder method of the http gem needs to be overriden because of the twitter gem.
# Without that, there's an incompatibility between the simple_oauth gem which encodes asterisks and the http one which does not.
# Cf. https://github.com/httprb/form_data/issues/22 and https://github.com/sferik/twitter/issues/677
HTTP::FormData::Urlencoded.encoder = lambda do |enum|
  unescaped_chars = /[^a-z0-9\-\.\_\~]/i
  enum.map do |k, v|
    if v.nil?
      ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars)
    elsif v.respond_to?(:to_ary)
      v.to_ary.map do |w|
        str = ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars)
        unless w.nil?
          str << '='
          str << ::URI::DEFAULT_PARSER.escape(w.to_s, unescaped_chars)
        end
      end.join('&')
    else
      str = ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars)
      str << '='
      str << ::URI::DEFAULT_PARSER.escape(v.to_s, unescaped_chars)
    end
  end.join('&')
end

It's been a few weeks we're using this method and it seems fine so far. Be careful though, it can have some side effects if some of the other gems you're using depend on http-form_data and expect the original encoding way.

[kieraneglin]

Reviving this one to say it's still affecting the latest version

[marckohlbrugge]

Running into this issue as well.

Does anyone know if it's just asterisks leading to problems, or other characters as well?

[reemaqburst]

For me, the issue is only for asteriks. Other chars gets posted.

[Gordin]

Like @FabienChaynes said, the issue here is that oauth has different rules for escaping characters than the standard for application/x-www-form-urlencoded.
For the oauth signature, the * will be escaped, (* is missing from Reserved Characters here): https://developer.twitter.com/en/docs/basics/authentication/oauth-1-0a/percent-encoding-parameters
while for forms it is not: (0x2A is the *) https://www.w3.org/TR/2013/CR-html5-20130806/forms.html#url-encoded-form-data

The http lib and simpleoauth both work as intended, Twitter just still wants x-www-form-urlencoded requests to be encoded by oauth rules, which doesn't work. (Actually it almost always works, except if your Message has an Asterisk...)

I came up with a solution that is basically the snippet from @FabienChaynes but this also checks if you are calling from inside the Twitter gem and decides based on that if it uses the default, or the decoding Twitter wants. That way we won't break the FormData Module for other libraries.

Also, I tried to make simpleoauth build the signature without encoding the *. Building the signature this way worked, but the Twitter API still didn't accept Tweets with * in them. It seems like the only way to get Twitter to accept tweets is to encode the request forms using oauth escaping rules.

I would have opened a pull request for this, but I don't really know where to put this, since I didn't find any obvious initialization that is done by the twitter gem. Maybe someone who is more familiar with the code has some idea about this?

require "uri"

# The encoder method of the http gem needs to be overriden because of the twitter gem.
# Without that, there's an incompatibility between the simple_oauth gem which encodes asterisks and the http one which does not.
# Cf. https://github.com/httprb/form_data/issues/22 and https://github.com/sferik/twitter/issues/6# 77
# I added a check to see if this method has been called from inside the twitter gem, so that other libraries can still use the default behavior
HTTP::FormData::Urlencoded.encoder = lambda do |enum, enc = nil|
  call_regex = Regexp.new(
    'gems/twitter-[^/]+/lib/twitter/rest/request.rb:\d+:in `public_send\'')
  if caller.any? call_regex
    unescaped_chars = /[^a-z0-9\-\.\_\~]/i
    enum.map do |k, v|
      if v.nil?
        ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars)
      elsif v.respond_to?(:to_ary)
        v.to_ary.map do |w|
          str = ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars)
          unless w.nil?
            str << '='
            str << ::URI::DEFAULT_PARSER.escape(w.to_s, unescaped_chars)
          end
        end.join('&')
      else
        str = ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars)
        str << '='
        str << ::URI::DEFAULT_PARSER.escape(v.to_s, unescaped_chars)
      end
    end.join('&')
  else
    ::URI.encode_www_form(enum, enc)
  end
end

[Gordin]

Just for the record, I'm aware that including this snippet in the twitter lib isn't the best solution possible, even with the check for the calling function this could still break if some other lib also decides to override the default encoder, but I think this solution is still better than having this library not being able to send tweets with * in them. It's not great, but I think it's the next best thing to a clean solution.

IMHO a clean solution would require something like Faraday, where you can have separate instances of request handlers that can have their own rules for encoding to not interfere with other libraries. I only started looking at this repository yesterday, so I don't know why exactly the switch away from Faraday was made, but unless Twitter decides to change something about their API, using Faraday (or some other request library) is probably the only clean solution.

[scy]

I’m an experienced developer, but I don’t speak Ruby. However, I went along with @Gordin in a long Twitter conversation (German though) while he dug into the issue and can confirm his conclusions.

Let me try to summarize what everyone has said:

• OAuth 1.0a (used by the Twitter API and implemented in this project using laserlemon/simple_oauth) requires the signature of a request to be computed over the request parameters encoded as described in RFC 3986, which states that * is to be replaced by %2A. (Twitter’s API docs are very clear about this, too.)
• HTTP::FormData (used by this project to do the actual HTTP request) on the other hand encodes according to the rules of Ruby’s encode_www_form, which implements the HTML5 candidate recommendation, and for HTML5, * (0x2a) is not to be escaped anymore.
• Twitter itself, when verifying the request, seems to compute the signature against the raw HTTP POST request, which will contain an unescaped asterisk produced by HTTP::FormData. But the signature we provide was made over a string containing an escaped asterisk as required by OAuth. Thus, the signature check fails and the tweet is rejected.
  • Twitter could of course adhere to Postel’s Law, decode the incoming request in a more liberal way, and then compute its own signature according to OAuth’s rules by re-escaping the data in the RFC 3986 way, but alas, they don’t.

None of the libraries that this project is using is really at fault here: They all do their job correctly. However, the OAuth and the HTTP code escape the parameters according to different standards, which then causes the issue. That makes this library the correct place to fix it, because it is supposed to connect all the threads correctly, and it doesn’t. (Sorry 😉)

Overriding the way in which the parameters are escaped in the HTTP request seems to be the only way to do this, and apart from switching to a different HTTP library, the snippet suggested by @Gordin seems to me to be the best solution.

[summera]

If the http-form_data gem would allow you to customize encoding on a per-instance basis and we could pass in an encoder through the http gem this could solve the problem without having to use caller to determine what's making the request. I've opened httprb/form_data#29 to see if that's a possible direction.

[summera]

With updates from httprb/http#599 and httprb/form_data#29, the http gem can now accept an object with a custom encoder as of v4.4. I just opened #969 to fix this encoding issue. If you all wouldn't mind testing this with your projects and letting me know if it works, that would be greatly appreciated. I'll be doing the same.