[Crawl] Allow users to crawl an entire website

[StanGirard]

Have a recursive crawling over a website, pulls every other website linked in the website

[StanGirard]

The endpoint and options already exist in the backend but it doesn't have the code behind :D

[StanGirard]

#108 adds an input in the frontend for an URL. There should be a setting to add JS rendering and crawling website

[Ruben1701]

Starting on this!

[shashank-crypto]

@Ruben1701 @StanGirard Do you mind if I take over?

[StanGirard]

No go ahead @shashank-crypto

[Ruben1701]

@shashank-crypto if you check the pull requests there is a working version just with a bug I cant seem to figure out!

[shashank-crypto]

@Ruben1701 I saw your PR, but I don't know if the selenium is the right answer for this. It is a heavy tool plus using chromium might take some unnecessary space. I am thinking it as more of a text scraping tool.
Why did you choose selenium as your first choice ? May be I am missing something. Because the solution I am thinking will have issue covering the cases where we have pdf pages or are not simple html pages.

[Ruben1701]

@shashank-crypto it should only use selenium if its a website that loads its content using javascript

[shashank-crypto]

@Ruben1701 Yeah, that's going to be tough to handle but I wanted to avoid using Selenium for its scalability issue. But what bug were you having ? Just this one ?

Works for one url and crawls the rest but somehow doesnt save it into the file, think it might have to do with the endpoint itself in the API.py the Spoolfile to be more specific.

[Ruben1701]

@shashank-crypto yep if you check the console you will see it returns all the different urls it scrapes but the file only contains the original url content. Kinda caught up in my graduation right now so haven't had a chance to look at it yet.

[shashank-crypto]

Sure @Ruben1701, I will try to work with your PR.

[Ruben1701]

@shashank-crypto have discord? join the server and hit me up on there at the office all day but i will have it open!

[shashank-crypto]

Which server ? @Ruben1701

[Ruben1701]

Which server ? @Ruben1701

https://discord.gg/VTgSYEg7

@shashank-crypto

[shashank-crypto]

@StanGirard I have raised a PR #247 for this. I have tested in my local. It's uploading the page contents. Functionality is working. I have tested with langchanin, aws and beautifulsoup documentation pages. There are few edge cases of collecting links and adding to paths. I have covered the most common cases for now.

[github-actions]

Thanks for your contributions, we'll be closing this issue as it has gone stale. Feel free to reopen if you'd like to continue the discussion.