Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

这程序需要运行多长时间啊。是不是把所有信息爬完统一录入数据库啊 #4

Open
AndrewLius opened this issue Jan 6, 2023 · 2 comments
Open

这程序需要运行多长时间啊。是不是把所有信息爬完统一录入数据库啊 #4

AndrewLius opened this issue Jan 6, 2023 · 2 comments

Comments

@AndrewLius
Copy link

AndrewLius commented Jan 6, 2023 •
edited by monkey-soft
Loading

No description provided.

@monkey-soft
Copy link
Owner

是的，检索完网站所有的信息，才会入库。我记得运行一次大概要1-2小时。其实代码可以优化下，做断点续传功能，即定时存储到数据库。

@gjxq
Copy link

gjxq commented Feb 6, 2025

做个判断把数据库里最新的ID获取到，爬的时候新增的再爬，已存在的跳过会越来越快

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@monkey-soft @gjxq @AndrewLius