[AIRFLOW-4113] Unpin boto3

[Fokko]

Make sure you have checked all steps below.

Right now we're pinning boto on <1.8 because of an underlying issue in moto. This issue has been solved, and therefore it would be nice to update again to a more recent version of boto.

#3810

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-4113] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-4113
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-4113], code changes always need a Jira issue.
  • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

• Here are some details about my PR, including screenshots of any UI changes:

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

• Passes flake8

[ashb]

======================================================================
97) ERROR: test_write_raises (tests.utils.log.test_s3_task_handler.TestS3TaskHandler)
----------------------------------------------------------------------
   Traceback (most recent call last):
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/moto/core/models.py line 71 in wrapper
      result = func(*args, **kwargs)
    tests/utils/log/test_s3_task_handler.py line 67 in setUp
      self.conn = boto3.client('s3')
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/boto3/__init__.py line 91 in client
      return _get_default_session().client(*args, **kwargs)
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/boto3/session.py line 263 in client
      aws_session_token=aws_session_token, config=config)
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/botocore/session.py line 885 in create_client
      client_config=config, api_version=api_version)
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/botocore/client.py line 70 in create_client
      cls = self._create_client_class(service_name, service_model)
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/botocore/client.py line 95 in _create_client_class
      base_classes=bases)
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/botocore/hooks.py line 227 in emit
      return self._emit(event_name, kwargs)
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/botocore/hooks.py line 360 in _emit
      aliased_event_name, kwargs, stop_on_response
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/botocore/hooks.py line 210 in _emit
      response = handler(**kwargs)
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/boto3/utils.py line 61 in _handler
      module = import_module(module)
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/boto3/utils.py line 52 in import_module
      __import__(name)
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/boto3/s3/inject.py line 15 in <module>
      from boto3.s3.transfer import create_transfer_manager
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/boto3/s3/transfer.py line 129 in <module>
      from s3transfer.manager import TransferConfig as S3TransferConfig
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/s3transfer/manager.py line 21 in <module>
      from s3transfer.utils import get_callbacks
    .tox/py35-backend_mysql-env_docker/lib/python3.5/site-packages/s3transfer/utils.py line 27 in <module>
      from botocore.exceptions import ReadTimeoutError
   ImportError: cannot import name 'ReadTimeoutError'

Nothing is ever straight forward :(

[bryanyang0528]

same error here

======================================================================
ERROR: test_execute_with_failing_transform_script (tests.operators.test_s3_file_transform_operator.TestS3FileTransformOperator)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/usr/local/lib/python3.7/site-packages/moto/core/models.py", line 71, in wrapper
    result = func(*args, **kwargs)
  File "/airflow/tests/operators/test_s3_file_transform_operator.py", line 98, in test_execute_with_failing_transform_script
    conn = boto3.client('s3')
  File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 91, in client
    return _get_default_session().client(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 885, in create_client
    client_config=config, api_version=api_version)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 70, in create_client
    cls = self._create_client_class(service_name, service_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 95, in _create_client_class
    base_classes=bases)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 360, in _emit
    aliased_event_name, kwargs, stop_on_response
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/boto3/utils.py", line 61, in _handler
    module = import_module(module)
  File "/usr/local/lib/python3.7/site-packages/boto3/utils.py", line 52, in import_module
    __import__(name)
  File "/usr/local/lib/python3.7/site-packages/boto3/s3/inject.py", line 15, in <module>
    from boto3.s3.transfer import create_transfer_manager
  File "/usr/local/lib/python3.7/site-packages/boto3/s3/transfer.py", line 129, in <module>
    from s3transfer.manager import TransferConfig as S3TransferConfig
  File "/usr/local/lib/python3.7/site-packages/s3transfer/manager.py", line 21, in <module>
    from s3transfer.utils import get_callbacks
  File "/usr/local/lib/python3.7/site-packages/s3transfer/utils.py", line 27, in <module>
    from botocore.exceptions import ReadTimeoutError
ImportError: cannot import name 'ReadTimeoutError' from 'botocore.exceptions' (/usr/local/lib/python3.7/site-packages/botocore/exceptions.py)

environment:

>>> import boto3
>>> boto3.__version__
'1.9.120'
>>> import urllib3
>>> urllib3.__version__
'1.24.1'
>>> import botocore
>>> botocore.__version__
'1.10.84'

[bryanyang0528]

I resolved this issue by updating the version of botocore to 1.12.120 and moto to 1.3.7.
By the way, after updating moto to 1.3.7, you have to export AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID to whatever to avoid NoCredentialsError.

Reference: getmoto/moto#1924

[halilduygulu]

@bryanyang0528 @Fokko Any progress on using new version of boto3 ? It is now 1.9.xxx and we are missing some api changes, like sagemaker section. Currently I can only create a notebook instance with 5 GB of EBS disk, because this parameter is not available on your boto3 version, so 5gb disk size is kind of useless for our use-case, which means I can't use airflow for this purpose but we use airflow for everything scheduled in the company.

[halilduygulu]

Anyone thinking about this pr? Looks like big win if it is merged.

[Fokko]

I'm still working on this, but have very little time right now. Feel free to take it over if you're comfortable with boto.

[kaxil]

@halilduygulu Want to take over? Or @Fokko if you have time?

Ash is planning to release 1.10.4 RC1 today. If we can get this within a day or two we can include it in RC2 (if we had to do an RC2, or else it would go in Airflow 2.0)

[halilduygulu]

I do not have enough information about project to take it over.

[james-see]

@Fokko please can we get this done. Lots of people relying on boto and airflow. We now have to manually patch outdated things to continue to operate.

[ashb]

@jamesacampbell You are welcome to take over this work and make a PR that passes tests.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kaxil]

Created a separate PR: #6884