Spider

[roniemartinez]

Possible ways to implement a simple Spider

### Expose "scraper" object in the handler functions

@select(css="a")
def get_link(element, scraper):  # <-- pass scraper object
    url = element.get_attribute("href")
    scraper.follow(url)  # <-- add to the URLs that will be scraped by the scraper 
    return {"url": url}

### Include the URLs in return

@select(css="a")
def get_link(element):
    url = element.get_attribute("href")
    return {"url": url}, [url, ...]  # <- return a tuple of dict result and list of URLs

Final implementation

Just use --follow-urls or pass follow_urls=True to run(). #90
This is less complicated when compared to managing the URLs to crawl by yourself inside the code.

WARNING: Do not use until #27 is implemented as this option will crawl indefinitely and will not save the data.