[Twitter] Video downloading issue

[DallasWhite]

I set "video" to true in my config.json, but I keep getting an error. Here's the debug:

[gallery-dl][debug] Version 1.10.1
[gallery-dl][debug] Python 3.7.2 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.1
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/tyson_hesse/status/1103767554424598528'
[gallery-dl][debug] Updating urllib3 ciphers
[twitter][debug] Using TwitterTweetExtractor for 'https://twitter.com/tyson_hesse/status/1103767554424598528'
[twitter][debug] Active postprocessor modules: [<gallery_dl.postprocessor.ugoira.UgoiraPP object at 0x04D52070>]
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /tyson_hesse/status/1103767554424598528 HTTP/1.1" 200 485
[twitter][error] An unexpected error occurred: AttributeError - 'NoneType' object has no attribute 'index'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\job.py", line 47, in run
    for msg in self.extractor:
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\extractor\twitter.py", line 43, in items
    data = self._data_from_tweet(tweet)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\extractor\twitter.py", line 98, in _data_from_tweet
    extr = text.extract_from(tweet)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\text.py", line 152, in extract_from
    def extr(begin, end, index=txt.index, txt=txt):
AttributeError: 'NoneType' object has no attribute 'index'

---

[mikf]

There are two problems:

1. You are logged in and gallery-dl can't deal with the new Twitter interface
2. Even if you don't specify username&password to fall back to the legacy Twitter layout, gallery-dl can currently only extract media from a top-level tweet

You are better off using youtube-dl directly to download this video, and I'll see if I can at least get 2. fixed.

(https://twitter.com/tyson_hesse/media as input URL still works if you want to grab everything fom that account, by the way)

[DallasWhite]

So I logged out, and I tried just scraping the media from https://twitter.com/tyson_hesse/media instead, and I got another error:

[twitter][error] An unexpected error occurred: KeyError - 'extension'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
PS C:\Users\ABlack> gallery-dl --verbose  https://twitter.com/tyson_hesse/media
[gallery-dl][debug] Version 1.10.1
[gallery-dl][debug] Python 3.7.2 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.1
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/tyson_hesse/media'
[gallery-dl][debug] Updating urllib3 ciphers
[twitter][debug] Using TwitterMediaExtractor for 'https://twitter.com/tyson_hesse/media'
[twitter][debug] Active postprocessor modules: [<gallery_dl.postprocessor.ugoira.UgoiraPP object at 0x04BA3070>]
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/profiles/show/tyson_hesse/media_timeline?include_available_features=1&include_entities=1&reset_error_state=false&lang=en HTTP/1.1" 200 13721
[twitter][error] An unexpected error occurred: KeyError - 'extension'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\job.py", line 48, in run
    self.dispatch(msg)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\job.py", line 93, in dispatch
    self.handle_url(url, kwds)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\job.py", line 203, in handle_url
    pp.prepare(pathfmt)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\postprocessor\ugoira.py", line 55, in prepare
    if pathfmt.keywords["extension"] != "zip":
KeyError: 'extension'

[mikf]

That error got already fixed (#382).

Either update to version 1.10.2 or disable the ugoira postprocessor for unrelated sites
("whitelist": ["pixiv", "danbooru"])

Edit: You can stay logged in when using https://twitter.com/tyson_hesse/media. It's only the direct Tweet links that don't work. Nevermind, then video downloads don't work, unless you disable cookie forwarding to youtube-dl.

[DallasWhite]

Looks like that did the trick, thanks! Also, would keeping the cookie forwarding to ytdl disabled cause any problems later on?

[mikf]

The goal for this feature was to allow downloading private videos by transferring a login session from gallery-dl to youtube-dl, primarily for Instagram, but unless youtube-dl fixes its Twitter code it's probably better to leave it disabled.

[DallasWhite]

Okay sounds good, thanks again!