Feature/extra #79

gboudreau · 2016-12-06T18:29:43Z

Rebased on latest master (83aebdb) and fixed login command that didn't work anymore.
Also fixed an issue where it would use the current folder instead of $HOME/.cache/toutv/ folder for cache on Mac.

gboudreau · 2016-12-06T20:02:15Z

Added another fix where using --url wouldn't work, because the episode page does not contain the emission ID any more; we now need to specify two URLs, for the emission and episode, when using --url with either fetch or info.

gboudreau · 2016-12-06T20:03:02Z

And finally, I made a first draft of downloading episodes from the Extra section using toutv fetch --url [emission_url] [episode_url].
We still can't load the episode or emission information for Extras, but I think most people just want fetch to work. They can find the URLs to use by browsing ici.tou.tv, and then download the episodes they want from the command line.

simark · 2016-12-06T20:40:06Z

Woohoo, thanks! I won't have time to look into it tonight, maybe tomorrow night. If somebody else wants to take a look, go ahead.

gboudreau · 2016-12-07T03:41:27Z

So I was able to make info|search|list|fetch work with emissions and episodes in Extra.
Hooray!

Code is a little messy, but it works.

Of note: you will need to delete your .toutv-cache for new emissions & episodes to be loaded from the API.

simark

I looked at an login conversation from an Android phone with mitmproxy, it matches pretty much what you added. LGTM.

simark · 2016-12-08T02:25:13Z

Err, I wanted to comment on the first commit. Is there any way to leave a comment on a particular commit?

simark · 2016-12-08T02:29:28Z

Pushed your xdg/cache folder commit to master, since it's unrelated and helpful in any case.

simark · 2016-12-08T02:34:51Z

Instead of requiring the user to enter two URLs (which is a bit tedious), could we solve the problem by having our client do a bit more work? For example, the HTML of an episode contains the show name at least:

            <meta name="rc.emission" content="district-31"/>
            <meta name="rc.codeemission" content="district31"/>

Perhaps we could do another request to look up the show/emission id?

simark · 2016-12-08T02:42:02Z

Actually, given that an episode url looks like this:

http://ici.tou.tv/district-31/S01E02

We could simply deduce the show URL by keeping the first URL component after the domain... or use the info mentioned in the previous comment and append it to http://ici.tou.tv/. What do you think?

gboudreau · 2016-12-08T14:41:41Z

@simark To comment on any commit part of a PR, simply click the commit that appears in the PR, and comment on specific lines there, or you can go in the Files changed tab at the top of the PR, and comment on whatever change was made.

gboudreau · 2016-12-08T14:44:40Z

Could you rebase the current Feature/extra branch on top of the current master, to make this PR easier to read? It should then automatically adjust to only show the changes I made (if not, I'll manually update my branch).

gboudreau

As suggested by @simark, since the episode URLs look like http://ici.tou.tv/[emission-name]/[episode-name], we could allow using a single URL, and deducing the emission URL.

Another suggestion: allow both an episode URL and an emission URL, when using --url.

simark · 2016-12-08T15:08:40Z

Could you rebase the current Feature/extra branch on top of the current master, to make this PR easier to read? It should then automatically adjust to only show the changes I made (if not, I'll manually update my branch).

Ok, I've rebased feature/extra on master and force-pushed.

gboudreau · 2016-12-08T16:23:06Z

Excellent. Much better now; easier to review.

So I added two commits which allows a user to specify an emission by name or URL for list, info and fetch, and also specify an episode by name or URL for info and fetch.
Also, when specifying an episode URL, there is no need to specify an emission URL too; it's all handled automatically.

simark · 2016-12-08T16:26:12Z

Ah good idea about not having to use --url. It makes it even easier to use.

This makes argparse handle automatically the case where the (sub-)command is omitted. For example, using just "toutv" would show the help, but using "toutv --verbose" would result in an exception: Traceback (most recent call last): File "/home/simark/src/pytoutv-env/bin/toutv", line 9, in <module> load_entry_point('pytoutv==2.3.1', 'console_scripts', 'toutv')() File "/home/simark/src/pytoutv/toutvcli/app.py", line 672, in run return app.run() File "/home/simark/src/pytoutv/toutvcli/app.py", line 91, in run if args.build_client: AttributeError: 'Namespace' object has no attribute 'build_client' As a result, we don't return the exit code 10 when bad arguments are given. Signed-off-by: Simon Marchi <[email protected]>

Rename _toutvclient to _toutv_client. Also, initialize it in __init__ to silence PyCharms warning.

... to silence PyCharms warning.

PyCharms complains that toutv.exceptions is not imported. I am not sure if it's necessary or not, but it doesn't hurt to have it.

simark · 2016-12-15T02:50:24Z

@gboudreau, is it possible that the command toutv list doesn't work? I get:

Unknown exception: <class 'AttributeError'>: 'NoneType' object has no attribute 'startswith'
Traceback (most recent call last):
  File "/home/simark/src/pytoutv/toutvcli/app.py", line 99, in run
    args.func(args)
  File "/home/simark/src/pytoutv/toutvcli/app.py", line 381, in _command_list
    emission, episode = self._get_emission_episode_from_args(args)
  File "/home/simark/src/pytoutv/toutvcli/app.py", line 315, in _get_emission_episode_from_args
    if args.emission.startswith('http'):
AttributeError: 'NoneType' object has no attribute 'startswith'

gboudreau · 2016-12-15T11:58:00Z

Indeed. Fixed.

simark · 2016-12-16T14:56:53Z

toutvcli/app.py

@@ -222,6 +222,8 @@ def _build_argparser(self):
        pf.add_argument('-q', '--quality', action='store',
                        default=App.QUALITY_AVG, choices=quality_choices,
                        help='Video quality (default: {})'.format(App.QUALITY_AVG))
+        pf.add_argument('-v', '--verbose', action='store_true',


Do we need to have a --verbose for each command? If we only have one in the top-level parser, it forces the user to put the -v/--verbose between toutv and the subcommand, but is it a problem? It's a flag that affects the whole program, not just the subcommand, so I think it would make sense.

Didn't know that would work. Will remove all --verbose param handlers I added to pf/pi/pl. Thanks.

I don't see any reason why we have to handle the NoMatchExceptions in different places. It will give the same result if we catch it along with the other exceptions, at the top-level (the App.run method).

…tra section

… section

…g both emission and episode URLs; the emission info is not found in the episode HTML any more...

…mmand: toutv fetch --url [emission_url] [episode_url] eg. toutv fetch -q MAX --url 'http://ici.tou.tv/le-show-cache-2' 'http://ici.tou.tv/le-show-cache-2/S2014E01' We still can't load the episode or emission information for Extras, but I think most people just want `fetch` to work. They can find the URLs to use by browsing ici.tou.tv, and then download the episodes they want from the command line.

…rs; it's now possible to `list`, `search`, `info` and `fetch` those emissions/episodes. Examples: toutv list --all # Now returns 3,350 emissions; returned 1,525 before toutv search|info|list "Le show caché 2" toutv info "Le show caché 2" S2014E01 toutv info "Le show caché 2" 2425154669 toutv info --url 'http://ici.tou.tv/le-show-cache-2' 'http://ici.tou.tv/le-show-cache-2/S2014E01' toutv fetch "Le show caché 2" toutv fetch "Le show caché 2" S2014E02

…GET /presentation/...

… instead; no need to specify --url anymore. Also don't need two URLs anymore; only specifying the episode URL is enough. Examples: - toutv fetch|info 'Le show caché 2' - toutv fetch|info http://ici.tou.tv/le-show-cache-2 - toutv fetch|info 'Le show caché 2' 'S2014E02' - toutv fetch|info http://ici.tou.tv/le-show-cache-2/S2014E02 - toutv list 'Le show caché 2' - toutv list http://ici.tou.tv/le-show-cache-2

Ref: #79

simark · 2016-12-19T00:55:24Z

@gboudreau, I rebased your branch and added one commit to allow specifying only an URL to the episode (to fetch and info). Basically, doing toutv fetch http://ici.tou.tv/foo/bar is the same as doing toutv fetch foo bar. If you like it, you can reset --hard this PR with this branch.

It is in my repo: https://github.com/simark/pytoutv/tree/feature/extra

This patch makes it possible to fetch, list or show the info simply by specifying the URL to the resource (show or episode). Therefore, it should now be possible to fetch using one of the following forms: - toutv fetch <show> - toutv fetch <show> <episode> - toutv fetch <show-url> - toutv fetch <episode> <url> Same for info. We suggested some time ago to try to migrate from "emission" to "show", since it's a bit silly (although cute) to use franglais. I am converting the areas I touch to use that name. I removed the current test, since it's not very useful, and added a test for the show/episode argument parsing. It will be stable, since it doesn't depend on external factors.

gboudreau · 2016-12-25T15:51:24Z

Failure: toutv search "LES AVENTURES DU PHARMACHIEN"
This returns 6 emissions, the correct one twice, and the one emission for each episode! Makes no sense...
Also, toutv fetch "LES AVENTURES DU PHARMACHIEN" only tries to download the first episode...

`toutv search` returned episodes as shows because of this!

…n all the information needed to be able to download them (namely, the PID is missing). This commit adds the notion of 'short_version' for episodes: the short version of an episode is enough to display it to the end-user, but not enough to download it. So we either load the short version (which can be cached) or the full version, when needed; for example, for `toutv fetch`.

…sodes are returned by `GET GetEpisodesForEmission`; using `GET /presentation/[show_url]` first fixed that.

gboudreau · 2016-12-25T22:13:16Z

Above issue fixed with those last few commits.
Side-effect: downloaded files are now always called ShowName.SxxEyy.Épisode.yy.bitratekbps.ts, i.e. Épisode.yy instead of the episode name. This is because the new GET /presentation/search/ endpoint used to load the episodes for a specific show doesn't return the real episode names, for some reason. But using that endpoint is required to ensure we get all the episodes for a show.

Ref: bvanheu#79

simark · 2017-01-02T01:15:59Z

@gboudreau, I have rebased the branch and pushed it as master. At least now that it's in master, we'll be able to do fixes in more targetted PRs, instead of doing everything in this one.

Thanks a lot!

This was referenced Dec 6, 2016

Feature/extra - Let the user login & fetch an episode in 720p #61

Closed

Save cache in proper folder instead of working dir on macOS #76

Closed

gboudreau mentioned this pull request Dec 6, 2016

Tou.tv EXTRA fetching #64

Closed

simark approved these changes Dec 8, 2016

View reviewed changes

gboudreau commented Dec 8, 2016

View reviewed changes

simark force-pushed the feature/extra branch from 5df5cc6 to 368ceb8 Compare December 8, 2016 15:07

simark added 8 commits December 11, 2016 12:47

Rename variable in App

681d4c2

Rename _toutvclient to _toutv_client. Also, initialize it in __init__ to silence PyCharms warning.

Initialize _verbose in __init__

713f60c

... to silence PyCharms warning.

Fix spelling mistake

af0cc1c

Import toutv.exceptions in app

44fb844

PyCharms complains that toutv.exceptions is not imported. I am not sure if it's necessary or not, but it doesn't hurt to have it.

app.py: make methods static when possible

7b64d3c

Remove unused exception variables

1bc32b3

Remove unused variable

8bbcb05

simark reviewed Dec 16, 2016

View reviewed changes

simark and others added 19 commits December 18, 2016 16:46

Handle NoMatchException at top-level

566624a

I don't see any reason why we have to handle the NoMatchExceptions in different places. It will give the same result if we catch it along with the other exceptions, at the top-level (the App.run method).

Introduction of authentication module to fetch HD content from the ex…

ffccf04

…tra section

Added a 'login' command to the CLI to fetch HD content from the extra…

5fc22b5

… section

Updated README with the login command

9e9bfda

Bugfix: login command line action didn't work anymore

e4ef3b2

Bugfix: info and fetch using --url param now requires specifyin…

13499ed

…g both emission and episode URLs; the emission info is not found in the episode HTML any more...

When using info with an Extra emission, load its description using …

d2bea00

…GET /presentation/...

Updated README files to detail the latest changes

6e6005e

Bugfix: list command without emission & episode args failed.

8e6ce0d

Ref: #79

Bugfix: episode arguments was ignored, unless specified as a URL

4c8c7fb

Added --verbose (optional) argument to fetch command

cf7711e

Better parsing of emission & episode parameters

4335f17

Delete auth token, when it becomes invalid

03e360e

Rebase-fix: method is now static

a9396e8

Removing --verbose param handlers from command-specific parameters

3b8af74

Quiet warnings

301f458

gboudreau added 3 commits December 25, 2016 16:58

Bugfix: /presentation/search/ returns not only shows, but episodes too!

01a7d35

`toutv search` returned episodes as shows because of this!

Bugfix: for some shows (e.g. Les aventures du pharmacien) not all epi…

12ae0f4

…sodes are returned by `GET GetEpisodesForEmission`; using `GET /presentation/[show_url]` first fixed that.

simark pushed a commit to simark/pytoutv that referenced this pull request Jan 2, 2017

Bugfix: list command without emission & episode args failed.

e4b2fe6

Ref: bvanheu#79

simark closed this Jan 2, 2017

gboudreau deleted the feature/extra branch January 5, 2017 02:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/extra #79

Feature/extra #79

gboudreau commented Dec 6, 2016

gboudreau commented Dec 6, 2016

gboudreau commented Dec 6, 2016

simark commented Dec 6, 2016

gboudreau commented Dec 7, 2016 •

edited

Loading

simark left a comment

simark commented Dec 8, 2016

simark commented Dec 8, 2016

simark commented Dec 8, 2016

simark commented Dec 8, 2016

gboudreau commented Dec 8, 2016

gboudreau commented Dec 8, 2016

gboudreau left a comment •

edited

Loading

simark commented Dec 8, 2016

gboudreau commented Dec 8, 2016

simark commented Dec 8, 2016

simark commented Dec 15, 2016

gboudreau commented Dec 15, 2016

simark Dec 16, 2016

gboudreau Dec 16, 2016

simark commented Dec 19, 2016

gboudreau commented Dec 25, 2016 •

edited

Loading

gboudreau commented Dec 25, 2016

simark commented Jan 2, 2017

Feature/extra #79

Feature/extra #79

Conversation

gboudreau commented Dec 6, 2016

gboudreau commented Dec 6, 2016

gboudreau commented Dec 6, 2016

simark commented Dec 6, 2016

gboudreau commented Dec 7, 2016 • edited Loading

simark left a comment

Choose a reason for hiding this comment

simark commented Dec 8, 2016

simark commented Dec 8, 2016

simark commented Dec 8, 2016

simark commented Dec 8, 2016

gboudreau commented Dec 8, 2016

gboudreau commented Dec 8, 2016

gboudreau left a comment • edited Loading

Choose a reason for hiding this comment

simark commented Dec 8, 2016

gboudreau commented Dec 8, 2016

simark commented Dec 8, 2016

simark commented Dec 15, 2016

gboudreau commented Dec 15, 2016

simark Dec 16, 2016

Choose a reason for hiding this comment

gboudreau Dec 16, 2016

Choose a reason for hiding this comment

simark commented Dec 19, 2016

gboudreau commented Dec 25, 2016 • edited Loading

gboudreau commented Dec 25, 2016

simark commented Jan 2, 2017

gboudreau commented Dec 7, 2016 •

edited

Loading

gboudreau left a comment •

edited

Loading

gboudreau commented Dec 25, 2016 •

edited

Loading