Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Selector.root is not an instance of lxml.html.HtmlElement even if parser is html #40

Closed
kmike opened this issue Jun 16, 2016 · 7 comments
Closed

Selector.root is not an instance of lxml.html.HtmlElement even if parser is html #40

kmike opened this issue Jun 16, 2016 · 7 comments

Comments

@kmike
Copy link
Member

kmike commented Jun 16, 2016

I'm trying to use lxml.Cleaner without parsing response multiple times:

from lxml.html.clean import Cleaner
cleaner = Cleaner()
sel = parsel.Selector("<html><body><style>.p {width:10px}</style>hello</body></html>"
cleaner.clean_html(sel.root)

This doesn't work because Cleaner needs a lxml.html.HtmlElement instance, while Selector.root is always lxml.etree._Element, so it doesn't have a required .rewrite_links method.

Why is lxml.etree.HtmlParser used for html and not lxml.html.HtmlParser?
@redapple
Copy link
Contributor

@kmike , there was some discussion in the past about moving to lxml.html.HtmlParser
scrapy/scrapy#559 (comment)
but it was left at that, needing performance comparison

@kmike kmike mentioned this issue Jun 17, 2016
Closed
@rmax
Copy link

rmax commented Jun 23, 2016

@kmike @redapple Related #41

@eliasdorneles eliasdorneles mentioned this issue Jun 23, 2016
Closed
@kmike
Copy link
Member Author

kmike commented Aug 9, 2016

Another unscientific benchmark results (Python 3.5.1, OS X, latest lxml): https://gist.github.com/kmike/af647777cef39c3d01071905d176c006. It seems there is a 1-5% penalty in using lxml.html.HTMLParser, based on ~3700 random pages. My vote is to set HTMLParser as a default.

@eliasdorneles eliasdorneles mentioned this issue Aug 9, 2016
Closed
@kmike
Copy link
Member Author

kmike commented Aug 10, 2016 •
edited
Loading

Another way to look at it: additional time required for using HTMLParser is 0.0001s per web page on average (3700 pages, total parsing time is increased by 0.4s).

@eliasdorneles
Copy link
Member

yeah, let's just do it! 👍

@eliasdorneles
Copy link
Member

@kmike what do you think? #54

@eliasdorneles eliasdorneles mentioned this issue Nov 14, 2016
Merged
@eliasdorneles
Copy link
Member

This is fixed since #63:

>>> import parsel
>>> sel = parsel.Selector(u'<html><body><h1>oi</h1></body></html>')
>>> type(sel.root)
lxml.html.HtmlElement

Thanks @kmike !

@barrio barrio mentioned this issue Apr 30, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rmax @eliasdorneles @kmike @redapple