botocore version conflict between awscli, boto3, and moto

[matthew-craig-ncino]

Performing a side by side pip install of awscli, boto3, botocore, and moto results in an incompatible conflict.

To Reproduce run this in a terminal with a fresh machine:
pip install boto3 awscli moto botocore

Actual results yeild:
moto 1.3.5 has requirement botocore<1.11,>=1.9.16, but you'll have botocore 1.11.8 which is incompatible.
Performing a pip list confirms that the moto library is not installed after the above command.

Expected results:
Successful install.

I am attempting to use the following versions:

• awscli-1.16.8
• boto3-1.8.8
• botocore-1.11.8
• moto-1.3.5

I have tried this on the following versions of python:

• Python 3.7.0
• Python 2.7.15
• pip 18.0 (for both python2 and python3)

The source of the problem is:

awscli requires: botocore>1.11.8

• https://github.com/aws/aws-cli/blob/4515c962ab8d186e9e76262b2367c845b14d4a64/setup.py#L26

boto3 requires botocore>1.11.8

• https://github.com/boto/boto3/blob/b2e203799aa2dd18bca33ba095934bc75bf12ff0/setup.py#L17

moto requires: botocore>=1.9.16,<1.11

• https://github.com/spulec/moto/blob/e8c65d3d85ba62e857c7448691568ad4b02f4a46/setup.py#L12

This creates a grid lock that prevents moto from being installed.

I am running on Mac OSX High Sierra, Ubuntu Linux, and Alphine Linux.

[matthew-craig-ncino]

I attempted to bump the moto requirement to anything less than 1.12 but this caused numerous NoCredential Exceptions in unit testing. After an hour of looking I was not able pinpoint the cause. At about botocore version 1.11.4(~) they did make some changes to auth and signatures, but I was not familiar enough with moto/botocore to resolve. I may look at when I have more time, but if anyone knows the projects better it would be nice to have additional input.

I also had a question for devs that I didn't see in the documentation, How do the unit tests run locally? I was able to follow the travis setup to the point of running make test but got errors about nosetests not defined. I use nose in my own testing but I'm not sure what I'm missing on this.

Thanks.

[nikolay-saskovets]

We faced with slighty the same issue.
For us the rollback to moto==1.3.4 is a workable workaround.

btw, it seems like the same issue as in #1800. Isn't it, @matthew-craig-ncino?

[joguSD]

See #1793

[aaron-trout]

Also hitting this when trying to install boto3 + moto today. Also as mentioned above tests which previously passed now fail, but I think possibly just due to changes in boto3 and not to do with moto itself?

I mostly get stuff like this:

botocore.exceptions.ClientError: An error occurred (VPCIdNotSpecified) when calling the RunInstances operation: No default VPC for this user

[joguSD]

@aaron-trout moto needs to be updated to accommodate newer versions of boto3/botocore that's why they've pinned the version.

[spulec]

This should be fixed with #1800

[DrNickBailey]

This seems to be a problem again. Installing s3fs with pip results in this warning/error:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

boto3 1.16.24 requires botocore<1.20.0,>=1.19.24, but you'll have botocore 1.17.44 which is incompatible.

[bblommers]

Don't think that's a Moto issue, @DrNickBailey - we require botocore>=1.12.201, but nothing more specific