Export in CDRv2 format #3
Conversation
Also remove export of found forms, and do not save pages from other domains.
| crawler=self.settings.get('CDR_CRAWLER'), | ||
| extracted_metadata={}, | ||
| extracted_text='\n'.join( | ||
| response.xpath('//body//text()').extract()), |
There was a problem hiding this comment.
Paul suggested string() xpath function here: scrapy/parsel#34; I've tried it, and output is a bit cleaner.
There was a problem hiding this comment.
Yes, this is nicer, and less string joining happening, thanks!
Following @kmike suggestion. This gives cleaner output with less extra newlines.
|
What do you think about making CDR export non-optional and putting all extra info in |
|
Yeah, I like the idea - I'll update the PR. There is also an optional |
What was previously stored in PageItem and FormItem is now stored in extracted_metadata: is_page, depth, forms.
|
Yeah, it is not clear what is |
| url=url, | ||
| text=response.text, | ||
| if not self.link_extractor.matches(url): | ||
| return |
|
This might be for some structured data from the page, like price of an item, etc. So |
Also remove export of found forms, and do not save pages from other domains (we can get there after following redirects).
I've included only required field, and left
extracted_metadataempty. Also, I did not include required_timestampfield, since the docs say it should be autogenerated by elasticsearch.