Limit atlassian-python-api to <3.41.6

[Taragolis]

atlassian-python-api 3.41.5 introduce a bug with beautifulsoup (missing dependency) it not failed in our CI because one because we pre-install it for some other package, hotfix version 3.41.6 introduce another bug with import unknown package.

For avoid any issues better to upper bound dependency until it properly fixed
In addition add beautifulsoup as dependency for a provider

Upstream issue/PRs:

• Version: 3.41.5 is failing with error ModuleNotFoundError: No module named 'bs4' atlassian-api/atlassian-python-api#1295
• Use correct name for Beautiful Soup in dependencies atlassian-api/atlassian-python-api#1296

----------------------------------------
Exception when importing: airflow.providers.atlassian.jira.hooks.jira


Traceback (most recent call last):
  File "/opt/airflow/scripts/in_container/verify_providers.py", line 199, in import_all_classes
    _module = importlib.import_module(modinfo.name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name, package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/atlassian/jira/hooks/jira.py", line 24, in <module>
    from atlassian import Jira
  File "/usr/local/lib/python3.8/site-packages/atlassian/__init__.py", line 5, in <module>
    from .confluence import Confluence
  File "/usr/local/lib/python3.8/site-packages/atlassian/confluence.py", line 10, in <module>
    from beautifulsoup4 import BeautifulSoup
ModuleNotFoundError: No module named 'beautifulsoup4'

----------------------------------------
Exception when importing: airflow.providers.atlassian.jira.notifications.jira


Traceback (most recent call last):
  File "/opt/airflow/scripts/in_container/verify_providers.py", line 199, in import_all_classes
    _module = importlib.import_module(modinfo.name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name, package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/atlassian/jira/notifications/jira.py", line 24, in <module>
    from airflow.providers.atlassian.jira.hooks.jira import JiraHook
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/atlassian/jira/hooks/jira.py", line 24, in <module>
    from atlassian import Jira
  File "/usr/local/lib/python3.8/site-packages/atlassian/__init__.py", line 5, in <module>
    from .confluence import Confluence
  File "/usr/local/lib/python3.8/site-packages/atlassian/confluence.py", line 10, in <module>
    from beautifulsoup4 import BeautifulSoup
ModuleNotFoundError: No module named 'beautifulsoup4'

----------------------------------------
Exception when importing: airflow.providers.atlassian.jira.operators.jira


Traceback (most recent call last):
  File "/opt/airflow/scripts/in_container/verify_providers.py", line 199, in import_all_classes
    _module = importlib.import_module(modinfo.name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name, package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/atlassian/jira/operators/jira.py", line 23, in <module>
    from airflow.providers.atlassian.jira.hooks.jira import JiraHook
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/atlassian/jira/hooks/jira.py", line 24, in <module>
    from atlassian import Jira
  File "/usr/local/lib/python3.8/site-packages/atlassian/__init__.py", line 5, in <module>
    from .confluence import Confluence
  File "/usr/local/lib/python3.8/site-packages/atlassian/confluence.py", line 10, in <module>
    from beautifulsoup4 import BeautifulSoup
ModuleNotFoundError: No module named 'beautifulsoup4'

----------------------------------------
Exception when importing: airflow.providers.atlassian.jira.sensors.jira


Traceback (most recent call last):
  File "/opt/airflow/scripts/in_container/verify_providers.py", line 199, in import_all_classes
    _module = importlib.import_module(modinfo.name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name, package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/atlassian/jira/sensors/jira.py", line 22, in <module>
    from airflow.providers.atlassian.jira.hooks.jira import JiraHook
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/atlassian/jira/hooks/jira.py", line 24, in <module>
    from atlassian import Jira
  File "/usr/local/lib/python3.8/site-packages/atlassian/__init__.py", line 5, in <module>
    from .confluence import Confluence
  File "/usr/local/lib/python3.8/site-packages/atlassian/confluence.py", line 10, in <module>
    from beautifulsoup4 import BeautifulSoup
ModuleNotFoundError: No module named 'beautifulsoup4'

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

Suggestion: should we use != instead of < ? This seems like a "popular" issue and the way it is implemented it's quite likely they will fix it in the next patchlevel, if we use != we would not have to make another PR changing < into !=.

[josh-fell]

Suggestion: should we use != instead of < ? This seems like a "popular" issue and the way it is implemented it's quite likely they will fix it in the next patchlevel, if we use != we would not have to make another PR changing < into !=.

+1. We did this with PyGithub when a single release broke XCom pickling. That team immediately fixed it in the next patch release pretty quickly.

[potiuk]

Ah, I see they attempted to fix it and failed in .6, so maybe we should != both versions and assume they learned from .6 and .7 will be fixing it "properly" - if we keep our faith in Atlassian learning from mistakes that is :)

[potiuk]

BTW. They got that rather reasonable comment atlassian-api/atlassian-python-api#1295 (comment) about their CI not doing the right things - which I sympathise with a lot.

While our CI misses things sometimes - this is precisely what we do if we see things like that, we add a test that basicaly tests what we failed to catch before.

[Taragolis]

Yeah I also think that != would be enough if it fixed in one version 🤣
3.41.5 not affected if we manually define beautifulsoup in atlassian-python-api provider

[Taragolis]

Remaining red checks not related to this PR

Non-DB test: new major version of pinecone-client a client incompatible with current provider, need to pin / fix by the separate PR

< pinecone-client==2.2.4
---
> pinecone-client==3.0.0

Tests / Compat - failed on 2.6.0, I guess #36799 "breaks" compatibility with Airflow 2.6.0. As far a I understand custom command was added into the Airflow 2.7, so we might raise AirflowOptionalProviderFeatureException instead. cc: @vincbeck

[potiuk]

yeah. Let's merge it - seems like we have flurry of breaking dependencies (and our mistakes).