Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix youku's bug #450 #390 translate from LiuLang monkey-videos #463

Closed
wants to merge 18 commits into from

Conversation

linhua55
Copy link
Contributor

翻译的 LiuLang 的monkey-videos

Review on Reviewable

@soimort
Copy link
Owner

soimort commented Jan 2, 2015

Thanks.

The new code raises a TypeError for some videos that cannot be streamed outside mainland China:

$ ./you-get -u --format=hd2 http://v.youku.com/v_show/id_XNTc2NDk2MDMy
you_get: [Warning] This video can only be streamed within Mainland China!
you_get: Use '-y' to specify a proxy server for extracting stream data.

Traceback (most recent call last):
  File "./you-get", line 19, in <module>
    you_get.main(repo_path=_filepath)
  File "/home/soimort/Projects/you-get-develop/src/you_get/__main__.py", line 91, in main
    main()
  File "/home/soimort/Projects/you-get-develop/src/you_get/common.py", line 998, in main
    script_main('you-get', any_download, any_download_playlist)
  File "/home/soimort/Projects/you-get-develop/src/you_get/common.py", line 887, in script_main
    download_main(download, download_playlist, args, playlist, stream_id=stream_id, output_dir=output_dir, merge=merge, info_only=info_only)
  File "/home/soimort/Projects/you-get-develop/src/you_get/common.py", line 784, in download_main
    download(url, **kwargs)
  File "/home/soimort/Projects/you-get-develop/src/you_get/common.py", line 991, in any_download
    m.download(url, **kwargs)
  File "/home/soimort/Projects/you-get-develop/src/you_get/extractor.py", line 44, in download_by_url
    self.extract(**kwargs)
  File "/home/soimort/Projects/you-get-develop/src/you_get/extractors/youku.py", line 236, in extract
    m3u8+='?K='+ k
TypeError: Can't convert 'int' object to str implicitly

While the old one can still extract the source:

$ you-get -u --format=hd2 http://v.youku.com/v_show/id_XNTc2NDk2MDMy 
you_get: [Warning] This video can only be streamed within Mainland China!
you_get: Use '-y' to specify a proxy server for extracting stream data.

site:                优酷 (Youku)
title:               疯狂原始人(国语)
stream:
    - format:        hd2
      container:     flv
      video-profile: 超清
      size:          841.9 MiB (882788723 bytes)
    # download-with: you-get --format=hd2 [URL]

audio-languages:
    - lang:          国语
      download-url:  http://v.youku.com/v_show/id_XNTc2NDk2MDMy

    - lang:          英语
      download-url:  http://v.youku.com/v_show/id_XNTc3OTU5NzI4

    - lang:          粤语
      download-url:  http://v.youku.com/v_show/id_XNjg3NzM3ODcy

Real URLs:
['http://122.143.9.207/69732D68DDF4B82B164AC15558/0300011F0051D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://221.204.199.106/6772C7BB5B03982E1CB9652715/0300011F0151D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://175.22.3.210/6973F8C2694397621F0256CC2/0300011F0251D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://218.26.231.40/6572C7BB79E3C77966F8C3D87/0300011F0351D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://218.60.102.172/6773F8C2C54327CB69DDB4144/0300011F0451D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://103.224.233.50/67745E6F96A4D8448EB9D75E5F/0300011F0551D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://113.6.235.178/6573F8C2C554C73CA871B64E3/0300011F0651D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://175.22.3.141/677529C9B4830752593BD5D90/0300011F0751D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://221.204.194.205/6574C41CCED3982E1BB70D2DB8/0300011F0851D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://122.143.9.201/657529C95A44482702E13F3306/0300011F0951D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://221.203.1.202/65758F76F704083370AE323CE9/0300011F0A51D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://61.54.96.110/6575F52357C4C811B1BA2F5403/0300011F0B51D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv', 'http://60.211.205.13/6977F184C7A3B83159993B44C0/0300011F0C51D4A19EE14405CF07DD46D1E90B-DAB6-BA76-3252-99CDEB847AFF.flv']

Not sure why this happens, I'll look into this later. For now, if anyone needs this feature (and doesn't care about overseas usability), you can check out the youku-full-video branch.

@linhua55
Copy link
Contributor Author

linhua55 commented Jan 4, 2015

This video seems to have region limitation.It gives me the string "抱歉,因版权原因无法观看此视频".
Thus the 'k' and 'k2' from the url 'http://v.youku.com/player/getPlayList/VideoIDS/XNTc2NDk2MDMy/Pf/4/ctype/12/ev/1' are both '-1'.While the request need the k value.

$ proxychains bash

$ curl 'http://v.youku.com/player/getPlayList/VideoIDS/XNTc2NDk2MDMy/Pf/4/ctype/12/ev/1' | json_pp|grep error

"error_code" : -26,
"error" : "抱歉,因版权原因无法观看此视频!",

$ curl 'http://v.youku.com/player/getPlayList/VideoIDS/XNTc2NDk2MDMy/Pf/4/ctype/12/ev/1' | json_pp|grep k

@soimort
Copy link
Owner

soimort commented Jan 9, 2015

There are a few issues left in your code: First, it's no longer able to download password-protected videos; Second, while the old m3u8 method does extract metainfo of region-limited videos for oversea users, this one does not. (I was thinking if we can merge two methods together so if the k.youku.com method fails, you-get will fallback to the pl.youku.com one)

I'll look into this further when I catch a break (can't work with code these days).

@soimort
Copy link
Owner

soimort commented Jun 10, 2015

need to bring back the handling of password-protected videos, then it'll be perfect :shipit:

@linhua55
Copy link
Contributor Author

Ok,I'll re-try it.But for the Second,I think we can try direct connect to obtain the url.if fail,then use 'Unblock youku''s proxy.

@soimort
Copy link
Owner

soimort commented Jun 10, 2015

For the 2nd point: it's not important now. As quite a lot of users complain about incomplete videos, we can stick with k.youku.comas long as it fixes the issue. The old m3u8 method is flawed, and I haven't yet figure out a way to extract all video segments from pl.youku.com. For simplicity, whether to use Unblock Youku's proxy (or another proxy) should be left to users to choose, we really shouldn't bother with that in extractors/youku.py.

@linhua55
Copy link
Contributor Author

I think if we want to obtain full video segments from pl.youku.com,we should decompile youku's android app.

@soimort
Copy link
Owner

soimort commented Jun 25, 2015

I noticed a few edge cases not covered by the new approach (don't know if it's just me, but the old method seems to reach more videos)

Using this PR:

$ you-get http://v.youku.com/v_show/id_XMjU1NzMxODE2.html
you_get: Error

Using develop branch:

$ you-get http://v.youku.com/v_show/id_XMjU1NzMxODE2.html
site:                优酷 (Youku)
title:               第001话 天才篮球手!诞生
stream:
    - format:        mp4
      container:     mp4
      video-profile: 高清
      size:          91.2 MiB (95653457 bytes)
    # download-with: you-get --format=mp4 [URL]

Downloading 第001话 天才篮球手!诞生.mp4 ...
  0.8% (  0.8/91.2 MB) [                                        ] 1/4^C%

I'll keep tracking this issue. (I do use you-get for Youku myself, that's why I'm conservative about any change...)

@linhua55
Copy link
Contributor Author

This video has region limitation too.But the error_code doesn't be tackled.And the unblock youku's proxy "proxy.uku.im:8888" doesn't work.You can find some useful proxys on http://www.proxy.com.ru/

Thanks for your patient.

@soimort
Copy link
Owner

soimort commented Jun 25, 2015

No, it's not about religion limits or proxies at all (my IP is located in mainland China) - the video is not downloadable by this PR (or can you do it on your side?), while it should be, as the old code still does job.

@linhua55
Copy link
Contributor Author

Yes,this PR works well on my side.And it works on my vpn with proxy.

@soimort
Copy link
Owner

soimort commented Jun 26, 2015

It seems the video is no longer region limited anymore.

@zhangn1985
Copy link
Contributor

事实上“http://v.youku.com/v_show/id_XMjU1NzMxODE2.html”, 这个url所指的视频,报告出来的就是版权问题。
所以,硬要fix,只有请出原算法了做backup了。
当 k = -1时,直接break,然后调用原算法。

@soimort
Copy link
Owner

soimort commented Oct 4, 2015

两种算法各有限制。现有算法存在某些视频下载不完整的问题;新算法能下载完整,但是有一部分旧算法能解析的视频完全解析不了(个人推测是因为旧算法调用的是优酷移动端的接口,受到的地域限制比浏览器端的新算法更少)。因为这个原因,目前我还不能把这个branch merge进来。

screenshot from 2015-10-04 22-29-52

@s20208413
Copy link

如果 you-get 能够实现多种解析方法共存, 解析时让用户选择一个 (并且设置默认方法) 就好了.

@yan12125
Copy link
Contributor

yan12125 commented Oct 5, 2015

Use the new API first, and fallback to the old API if failed?

@linhua55
Copy link
Contributor Author

This method is useless due to youku's change.

@linhua55 linhua55 closed this Nov 26, 2015
@soimort
Copy link
Owner

soimort commented Nov 26, 2015

We'll see how to get things right for the new Youku API. Thanks for the PR, anyway!

@linhua55 linhua55 reopened this Nov 30, 2015
@linhua55
Copy link
Contributor Author

Thanks to @zhangn1985 .It comes back.
@yan12125 It works for now.
2015-11-30

@zhangn1985
Copy link
Contributor

居然能这样!!!!服了。

@zhangn1985
Copy link
Contributor

#463 (comment) 所示问题已经不存在了,因为那个视频已经404了

@soimort
Copy link
Owner

soimort commented Dec 2, 2015

Would you mind rebasing these commits to squash them into one? It's impossible to see what changes you've made compared to the original code; the diff is almost impossible to review.

Please don't add irrelevant files to your commit; also, don't commit conflict indication lines like <<<<<<< HEAD. Thanks.

@linhua55
Copy link
Contributor Author

linhua55 commented Dec 2, 2015

@soimort
Ok,Would you mind tell me how to rebase?

pikeszfish and others added 4 commits December 3, 2015 01:12
merge sormort/you-get

写错了type...

[qq] fix #548, close #443

merge sormort's you-get

Fix zhanqi

Zhanqi extractor works again...
And added ts merging implementation...

Signed-off-by: liushuyu <[email protected]>

Fix zhanqi again

Modified against the latest update of zhanqi's server...

Signed-off-by: liushuyu <[email protected]>

[Zhanqi]Modified something...

Changed something according to @jackyzy823

Update zhanqi.py

Minor fix

Fix zhanqi

Indentation fix

iqiyi new key,how it works in comment.

more reasonable tm value

sometimes there's some parameters,drop it

[Tumblr] fix for videos with no title

[Google+] fix for non-ASCII custom URLs

fix iqiyi(20150703)

[iqiyi] update: 20150710 (close #569)

[iqiyi] extract tvid & videoid from URL

version 0.3.34

Update salt.

[soundcloud] fix #575

support for python -m you_get

This makes develop-then-test easier and straghtforward

[netease] support for mv urls

example url: http://music.163.com/#/mv?id=440007

[youtube] switch to https (close #564)

[netease] support for 163.fm short url

e.g. http://163.fm/LzbHaQN

update key for iqiyi,thanks to @Freshman585

[iqiyi] new enckey

[iqiyi] new src

[iqiyi] update key

fix #582

 [iqiyi] fix 1080p parse by set um=1

Add support for Qianmo

http://qianmo.com/

Tested on my machine.

Try adding Funshion(fun.tv) support, fix #215

I have made those functions to allow full drama download, but do not
know how to integrate with the programme.

Please offer your help. Many thanks.

Revert "Try adding Funshion(fun.tv) support, fix #215"

This reverts commit c39927e.

fix iqiyi(20150820)

fix duplicate url bug for bilibili

Add Weibo Miaopai support

fix bug in tudou, fix #612

fix dailymotion for #609

Fix problems with videos that do not have 720p mode

[dailymotion] fix title

Add funshion support, fix #215, replace soimort#601, soimort#604

Update cntv.py

Update cntv.py

support embed player for youku/tudou

try sites one by one, to search video.
but not in video order.

Signed-off-by: Zhang Ning <[email protected]>

support multi video in one page

add matchall api

matchall: almost same as match1, but it will return a list
of all matches.

[yinyuetai] switch to JSON API, close #559

rewrit iqiyi using VideoExtractor class

Signed-off-by: Zhang Ning <[email protected]>

Add metacafe support

Add iQilu support

Add Vimeo Channel support

Fix soimort#634 L2

[common] divide the import of extractors into multiple lines

[youku] support multi-page playlists

[common] fix skipping download of file existence in some cases

[IQIYI] 2015/09/18 new salt,describe method in comments of iqiyi.py

[youku] gracefully handle single failure

merge sormort's you-get

let KeyboardInterrupt bypass

[qq] reimplement qq.py, close #657

merge sormort's you-get

[755] new site support

update README.md

update .travis.yml (add 3.5 and nightly)

version 0.3.35

update you-get.json

update Makefile & setup.py

[common] fix infinite redirecting

[archive] new site support

update README.md

[iqiyi] Update enc and authkey

 [iqiyi] simplify mix() function

 add --json option: output video information in json text

 fix extractors not use VideoExtractor after add --json option

[ffmpeg] use subprocess.check_call

This fixes RuntimeError: No active exception to reraise

 --json option now can support more extractors not using VideoExtractor

[soundcloud] update client_id, fix #679

[acfun] fix "unexpected keyword argument 'json_output'"

Revert "update Makefile & setup.py" (fix #685)

This reverts commit c3ac87a.

version 0.3.36

[sina] match vid from the page

fix for http://video.sina.com.cn/view/249851143.html

[iqiyi] update key (close #693)

[flickr] new site support

[mixcloud] fix

[tudou] fix playlists

Revert "fix duplicate url bug for bilibili"

[bilibili] do not download danmaku for dry_run

[bilibili] fix support of partitions (close #688)
- Download only one partition by default
- Support playlist (for downloading all partitions without merging)

Add suntv support

Fix douyu

Took @yan12125 's advice, and thanks to @yan12125 .
This fix closes #580

[common] --cookies loads Mozilla cookies.sqlite instead of Netscape cookies.txt

[bilibili] load cookies

[bilibili] clean-up

[common] support both cookies.txt and cookies.sqlite

[common] if it's not moz_cookies, pass

[common] update help message

update README.md

[bilibili] do not use description as title

update README.md

[ffmpeg] implement ffmpeg_concat_av()

[ffmpeg] set "-loglevel quiet" to suppress verbose FFmpeg output

[common] download_urls: support A/V merge using ffmpeg_concat_av()

[extractor] support dash_streams

[youtube] support dashmpd

[youtube] always parse video page (for DASH)

[travis] add webhooks for gitter integration

add *.srt to .gitignore

[common] update download_urls_chunked()

[extractor] support caption_tracks

[youtube] support caption_tracks

[common] beautify progress bar

[baomihua] fix #501

[yinyuetai] fix for mobile URL m.yinyuetai.com (#648)

[yinyuetai] support playlist

[yinyuetai] add yinyuetai_download_playlist

[bandcamp] new site support

update README.md

[heavymusic] new site support

update README.md

[netease] add netease_lyric_download()

[youtube] support DASH streams for VEVO videos

[youtube] fix AttributeError: 'YouTube' object has no attribute 'js'

[common] use importlib to import modules from dict SITES

[embed] remove duplicated embedded URLs

[universal] universal extractor and direct downloader

[youtube] set html5player when parsing video page for DASH

[common] show download speed on progress bar
- Close #178 (also requested in #284 and #700)
- Close soimort#305

[extractor] fix #713

[common] set the proper width of progress bar

[common] oops

[netease] download lyrics for albums and playlists

[netease] let's pretend we're not using the freakin key
- http://www.v2ex.com/t/131959

[common] add handy get_head(url)

[baidu] support tieba videos and images

[common] "Video Site" -> "Site"

[vimeo] fix #718

[tumblr] download photo / photoset

[instagram] download the image

[twitter] download images

[flickr] download images

[google] download the image

[baidu] support tieba albums

[common] adjust progress bar (shorten display of speed)

[rtmpdump]

[755] download images

[common] print_info: more MIME types

[common] url_info: more MIME types

[universal] download images

[w56] partly solved #720

[vimeo] fix unescape_html(None)

[miomio] quick fix #716

[embed] add more patterns for Tudou
- Example link: http://tieba.baidu.com/p/4114753102

[common] add google_search()

[util.log] update

[you-get] update

redefine version (0.4.x)

[embed] add more patterns for Tudou
- Example link: http://tieba.baidu.com/shipin/bw/video/play?kw=akb48&v_id=a35619448853a42b942231e1

[common] improve error message, display a brief instruction to tell users what to do

[common] add new option: --output-filename (close #425)

Add Yixia-Miaopai support, replace soimort#639

Add Pixnet support, fix #129, replace soimort#633

[youtube] fix VEVO when no 's' field presents

[youku] support show_page, fix #726

[common] update google_search(), fix #727

remove CONTRIBUTING.md, because no one cares

update you-get.json

[universal] improve (more image patterns)

[extractor]

Add Veoh support, fix #79

[veoh] set info_only=info_only

[iqiyi] fix #728

[youtube] show error message when a video is unavailable
- As requested in #720#issuecomment-150197734

[term] fcntl module only available on Unix

Remove support:
* Blip.tv <http://blip.tv>
* Catfun (喵星球) <http://www.catfun.tv>
* Coursera <https://www.coursera.org>
* SongTaste <http://www.songtaste.com>
* VID48 <http://vid48.com>
* VideoBam <http://videobam.com>

See soimort#732 for details.

update LICENSE.txt

Add Interest.me (CJ E&M)  support

Update Readme as soimort#732

[common] option -d is short for --debug

rewrite README.md

update .gitignore

add a separate README.rst

version 0.4.111

[universal] catch jpeg

[util.git] fix get_version()

added new key

[universal] catch links with href to .png and .gif

[google] support URL with "?cfem=1" postfix

[universal] use filename if len(filename) >= 5

[dilidili] Add support

[universal] fix long filenames

added ternary condition on instagram extractor

[tumblr] filter out some non-image urls

version 0.4.125

[netease] fix #743

[youtube] fix #749

[youtube] enable all DASH streams

add referer to fake_headers

[iqiyi] update key (close #750)

[youtube] fix more (follow-up to 5dd7b5f)

[youtube] nope

version 0.4.134

[common] fix division by zero

version 0.4.136

update README.md

[vine] support card URLs

[twitter] support embedded Vine

[netease] switch to m5.music.126.net (as suggested in #743)

[pinterest] new site support (using VideoExtractor)

update README.md

[755] use movieUrlHq

[755] filenames may be duplicated (sometimes)

[acfun] fix #745

[embed] support embedded yinyuetai patterns

[embed] support embedded iqiyi patterns

[youtube] fix #757

[tumblr] support direct URLs

[musicplayon] new site support (using VideoExtractor)

update README.md

[google] fix for notification links

setup.py: set zip_safe to True (close #759)

[vimeo] fix

version 0.4.156

you-get: neat way to get _filepath (close #762)

[twitter] minor fix

add pattern for url like http://tv.cntv.cn/video/VSET100241824133/ee5ed807346c4b6086f105ef10b559f3

[vimeo] fix (for videos not shown public)

[iqiyi] get curid from URL (close soimort#766)

[embed] redundant regex

[youtube] make use of browse_ajax (fix #746)

[common] do not take SystemExit wrongly

[cntv] merge=False (ffmpeg concat demuxer fails on corrupted segments, see #767)

version 0.4.167

[vimeo] support https in URL pattern (a potential improvement for all extractors in the future)

[baidu] support direct URLs (imgsrc.baidu.com)

[letv] fix #770

[sohu] fix #769

[youku] fix #771

merge soimort's you-get

[youku] print error message

merge

[youku] add support for audio languages, fix #369 (again)

merge

[youku] handle password-protected videos, fix #73 (again)

merge

version 0.4.176

[twitter] fix video extracting

[56] the actual hd type is "wvga", close #775

delete tmp.txt

merge
@soimort
Copy link
Owner

soimort commented Dec 2, 2015

You may see https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#Squashing-Commits on how to squash your Git commits. However, I can imagine there are a lot of conflicts against the latest develop in your current branch. If you feel unable to resolve them, I suggest that you do it the simple way:

  1. Close this PR.
  2. Create a new feature branch (for example, youku-fix) based on the HEAD of the current develop. (It is not advisable to use your own develop to create PRs, unless you just want to do a one-time PR.)
  3. Make necessary changes and commit them.
  4. Open a new PR.

@linhua55
Copy link
Contributor Author

linhua55 commented Dec 2, 2015

@soimort
What about to delete my fork linhua55/you-get.And refork yours.If so,Will these conversation in this pull request disappear.

@linhua55 linhua55 closed this Dec 2, 2015
@soimort
Copy link
Owner

soimort commented Dec 2, 2015

Yes, you can do that, but there is really no need to delete a fork and fork it again. git branch -u upstream/develop will just let you track the remote develop branch.

@linhua55
Copy link
Contributor Author

linhua55 commented Dec 2, 2015

Thanks.
Move to #783

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants