Version 1.3

[sluceno]

Hi there,

is it possible to bump a 1.3 version? It would be good to have a version with support for Basic Authentication #326

Thanks

[my8100]

For now, simply execute:

pip install -U git+https://github.com/scrapy/scrapyd.git

[sluceno]

Yes sure, this is what we are doing right now

Do you have a rough estimate of when 1.3 will be released?

[Digenis]

Hi @sluceno

See #389

If you tried the master branch,
any feedback is highly appreciated.

I believe @my8100 will soon merge #337.
Make sure to check if this changes anything for your setup.

[my8100]

@Digenis
#337 merged.
Seems we have to update the code for Travis CI: https://travis-ci.org/github/scrapy/scrapyd/builds/766655073
Also, any plan to drop support for Python 2.7 and other old versions?

[Digenis]

Cc: @sluceno

Edit:
any feedback is welcome.
E.g. something along the lines:

scrapyd git version 788ee1b
scrapy so-and-so version
twisted so-and-so version
python so-and-so version
Works as expected

[Digenis]

@my8100, yes.

For 1.3, we can keep it as is.
The only issue I see is bracketing package requirements' versions properly.
Especially twisted and scrapy.

In scrapyd-1.4
we'll definitely drop py2.7 and let's also drop py3.4 and py3.5.
Maybe we can also make the code more python3-idiomatic,
especially if we drop py3.6.

I had happily forgotten about the cryptography update.
Using binary wheels from pypi should be enough for the CI.

[my8100]

Cc: @sluceno

Edit:
any feedback is welcome.
E.g. something along the lines:

scrapyd git version 788ee1b
scrapy so-and-so version
twisted so-and-so version
python so-and-so version
Works as expected

@Digenis
I have done some basic test for the latest code on Windows/Ubuntu, Python2.7/3.8, with basic auth enabled/disabled this weekend.
I think you had better try it out before releasing the new version.

[Digenis]

As I said in the other issue,
I can't do large scale testing anymore
but I'll add some more environments to tox
and bracket requirements' versions.

For alpha and beta releases that would be enough.

---

I fixed the cryptography issue.

There's another problem in test_failed_spider_list
where a deprecation warning
gets in the way of parsing a traceback
from the subprocess's stdout.

I'll try mocking Popen with a wrapper
that adds python arguments that disable warnings.

[Digenis]

There are no changes to be made to the http auth feature and I fixed the build.
Let's roll out 1.3.0-alpha.1

[Digenis]

I'm too busy for a few more days
so I can't review any PRs now.

I'll publish an alpha release with what's already merged.
@my8100, ok?

[my8100]

I think an alpha release is ok.
More end-to-end test cases would be added before releasing V1.3

[jpmckinney]

@Digenis has a good summary in #389

1.3 has one last outstanding feature (#325) so that would be an alpha release.

I think we can drop #140 from 1.3 and give the users full control, probably through #197 although I'm not sure we can do this for 1.3, because #187 would need to be fixed too.

If there there are users willing to give feedback for 1.3-alpha we can go on with it. Then I estimate it would take 1-2 weeks to beta and maybe 2-3 weeks to a release candidate.

Regarding 1.4, we'd postpone all its issues for 1.5 and use 1.4 only to bump and drop requirements' versions, because it's been some time since releasing and they are outdated. Even if we target experimental support for py3.10 (twisted is onto it)
this would probably take a week or less.

• Review Listen on unix socket #325
• Check milestone https://github.com/scrapy/scrapyd/milestone/4

I'm not sure how those issues were selected for inclusion. We might prefer an even more minimal 1.3, to get the already-included features published.

[pawelmhm]

Hi, I'm from Zyte (old Scrapinghub),

My opinion is that we should go with 1.3 this month with the following things:

• all things already merged, not released, such as Support basic authentication #326
• drop Python 2 - it is already long dead
• drop 3.4, 3.5 - both also dead
• add Python 3.7, 3.8. 3,9, 3.10 - tests are passing with all of them [build] switch from travis to GH actions #412
• Linux socket (I'll try to add unit test for it, unless author of PR will do it) and we can merge it Listen on unix socket #325
• asyncio reactor, fix for Scrapyd does not support spiders that use AsyncioSelectorReactor #377 - this is because lack of asyncio reactor breaks scrapyd compatibilty with many newer scrapy tools such as https://github.com/scrapy-plugins/scrapy-playwright or https://github.com/scrapinghub/scrapy-autoextract
• switching to GitHub actions [build] switch from travis to GH actions #412
• possibly more, focusing on fixing critical level things, warnings etc

If there there are users willing to give feedback for 1.3-alpha we can go on with it

I doubt we'll get much feedback and changes won't be that big, so we can just release 1.3 without waiting for user feedback, which may not come. It will delay release without any potential benefit.

Regarding issues in milestone https://github.com/scrapy/scrapyd/milestone/4 clearing this backlog will be hard work, many PR-s are old, many contributors probably forgot about these PR-s, so we will have to rewrite this code again, fix some things, add tests etc. I would leave this backlog for 1.4 or even 2.0. Unless there are volunteers here who can take ownership of these issues and finalize them until end of this month. If yes, please declare who would like to take what.

I see a large problem here with lack of unit tests, and poor test coverage, so I would welcome more pull requests improving test coverage. Also, please, let's stick to the rule of not merging things without unit test coverage.

[jpmckinney]

@pawelmhm Sounds good to me!

[mxdev88]

Unless there are volunteers here who can take ownership of these issues and finalize them until end of this month. If yes, please declare who would like to take what.

Hi @pawelmhm @jpmckinney

I would like to see #359 in 1.3 if possible. Other people have also expressed interest in the feature. Can it be considered and re-reviewed if necessary ?

[jpmckinney]

@mxdev88 It has now been merged as #418

[pawelmhm]

Released here: https://pypi.org/project/scrapyd/ please report any issues you may find, there was a significant pause between this release and last release (5 years) and several changes, so worth testing everything thoroughly.

Now we should think about 1.3.1 with some minor cleanups that need to be done, there are some deprecation warnings from Scrapy and also six package is no longer needed. Then we can move on to 2.0, for this I would suggest doing new fresh less ugly UI, and some security fixes.