Follow dynamically-built URLs

[roniemartinez]

Use case

There are instances wherein IDs or slugs are embedded in other elements and they can be built into URLs that can be "followed".

<div id="project-id">Project: eca514fc</div>

The project ID eca514fc can be extracted and can be built to a URL, for example, https://example.com/projects/eca514fc.
There should be an option to follow this URL from a decorated function.

Solution

Implement the proposed solution in #62

@select(css=".project-id")
def get_link(element, scraper):  # <-- pass scraper object

    project_id = element.text_content().removeprefix("Project:").strip()
    url = f"https://example.com/projects/{project_id}"

    scraper.follow_url(url)  # <-- add to the URLs that will be scraped by the scraper 

    return {"project_id": project_id}

Final solution: #146