Modify AIX platform_tag so it provides PEP425 needs #190

aixtools · 2019-08-25T20:24:26Z

Provide an adequate platform tag for AIX. The generic tag provided by distutils.util.get_platform() is insufficient - besides missing "bitness" it does not provide any information regarding Technology Level, nor build date.

brettcannon · 2019-08-26T18:50:29Z

packaging/tags.py

@@ -310,6 +310,42 @@ def _have_compatible_glibc(required_major, minimum_minor):
    return _check_glibc_version(version_str, required_major, minimum_minor)


+def _aix_tag():
+    '''


Suggested change

'''

"""

brettcannon · 2019-08-26T18:50:48Z

packaging/tags.py

+    The program lslpp is used to gather these values.
+    The pep425 platform tag for AIX becomes:
+    AIX_V_R_M_YYWW_bitness
+    '''


Suggested change

'''

"""

brettcannon · 2019-08-26T18:50:55Z

packaging/tags.py

+    The pep425 platform tag for AIX becomes:
+    AIX_V_R_M_YYWW_bitness
+    '''
+


Suggested change

brettcannon · 2019-08-26T18:51:07Z

packaging/tags.py

+    AIX_V_R_M_YYWW_bitness
+    '''
+
+    from subprocess import Popen, PIPE, DEVNULL


Why the inline import?

I tried to respond to this, days ago, via email. I still do not see my reply - so I'll try again.

Basically, I was thinking - if not _AIX, then why import something additional that noone else is using. Simple enough to move to global imports.

p.s. I will be trying to get an _aix_support (similar to _osx_support) added into CPython, that would make this unneeded.

Imports are generally cheap, so there's no need to hide it unless you're running some extremely performance-sensitive code.

brettcannon · 2019-08-26T18:51:51Z

packaging/tags.py

+    _lslppLqc = p.stdout.read().strip().split(":")
+    p.wait()
+    (lpp, vrmf, bd) = list(_lslppLqc[index] for index in [0, 2, -1])
+    assert lpp == "bos.mp64", "%s != %s" % (lpp, "bos.mp64")


str.format() is preferred.

Thanks for the comment - I'll do this later in the week.

pradyunsg · 2019-08-27T16:13:50Z

Related to #188. That issue also contains a link to a broader discussion happening over on pip's issue tracker.

aixtools · 2019-11-06T13:49:40Z

I'll run tox seperately, to try and figure out why it is failing ci, but better for me would be a pointer to the documentation I am sure is there - on how to test the code.

tox -e lint passes locally, failed just now because of some HTTP server error.

The other tests fail showing this block:



Name                         Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------
packaging/_AIX_platform.py      20     20      2      0     0%   3-50
packaging/__about__.py          10      0      0      0   100%
packaging/__init__.py            3      0      0      0   100%
packaging/_structures.py        41      0      0      0   100%
packaging/_typing.py             3      0      0      0   100%
packaging/markers.py           133      0     45      0   100%
packaging/requirements.py       79      0     24      0   100%
packaging/specifiers.py        275      0    122      0   100%
packaging/tags.py              259      6    123      1    98%   491-499, 490->491
packaging/utils.py              25      0     14      0   100%

Thanks for pointers!

brettcannon · 2019-11-06T19:18:40Z

You can either read the tox.ini file to see how tox does it or the files in https://github.com/pypa/packaging/tree/master/.github/workflows as another example of how to run the tests.

aixtools · 2019-11-06T21:32:40Z

Well, I am getting - read have become - so frustrated trying to figure out how mypi 'works' that I'll just give up on the whole project. A wasted day.

brettcannon · 2019-11-06T22:10:40Z

@aixtools if you're giving up then I'm going to close this PR. If you choose to try again then feel free to open the PR once more.

aixtools · 2019-11-07T18:13:36Z

Reopen? How? Only button I see is delete branch.

Sigh - refrain from other comments re: frustration - as they are too likely to be misunderstood.

p.s. I was hoping for some assistance with dealing with mypi.

xavfernandez · 2019-11-07T22:05:53Z

@aixtools from my understanding mypy considers that sysconfig.get_config_var is an Optional[str] meaning it can return a None so you'll need to do something like:

from sysconfig import get_config_var


def test() -> str:
    value = get_config_var("test")
    if value is None:
        # or raise ValueError ?
        return ""
    return value

aixtools · 2019-11-08T17:09:46Z

@xavfernandez - Many Thanks - that helped a lot.

Python still makes my head spin when it comes to how the "absolute" typing of things constantly changes.

I have modified this little bit many times - as I get into issues with what is available "when".

So, while lint (read mypy) gets further, I am still stuck in what appears to be a circle for me. Or is the only solution to write another helper function to nudge typing.

Currently - stuck with:

   +51      # p = run(["/usr/bin/lslpp", "-Lqc", "bos.mp64"],
   +52      #       capture_output=True, text=True)
   +53      output = check_output(["/usr/bin/lslpp", "-Lqc", "bos.mp64"])
   +54      result = output.decode("utf-8").strip().split(":") # type: str

I have also tried # type: ignore

mypy.....................................................................Passed
mypy for Python 2........................................................Failed
hookid: mypy

packaging/_AIX_platform.py:54: error: Incompatible types in assignment (expression has type "List[unicode]", variable has type "str")

Also, a hint to see what black does not like.

black....................................................................Failed
hookid: black

Files were modified by this hook. Additional output:

reformatted /data/prj/python/git/packaging/packaging/_AIX_platform.py
All done! \u2728 \U0001f370 \u2728
1 file reformatted, 24 files left unchanged.

Merci!

aixtools · 2019-11-10T15:26:23Z

Some help - with mypi - is much appreciated. I just go in circles. If I understand correctly, there is some 'science' that can be done to satisfy mypi - but it is too much 'magic' for me at this point. I just go around in circles - same error, or new error. Very frustrating.

pradyunsg · 2019-11-10T16:24:02Z

@aixtools This should work:

diff --git a/packaging/_AIX_platform.py b/packaging/_AIX_platform.py
index 6d77aa0..64bd31e 100644
--- a/packaging/_AIX_platform.py
+++ b/packaging/_AIX_platform.py
@@ -17,12 +17,11 @@ def aix_tag(v, r, tl, bd):
 
 def aix_buildtag():
     # type: () -> str
-    bd = get_var("AIX_BUILDDATE")  # type: str
-    # because I do not understand howto tell mypi that
-    # get_var() returns a string _ I getting it to a temp variable
-    # and using that as argument to split()
-    # v, r, tl, f = get_var("BUILD_GNU_TYPE").split(".")
-    gt = get_var("BUILD_GNU_TYPE")  # type: str
+    bd = get_var("AIX_BUILDDATE")
+    assert bd
+
+    gt = get_var("BUILD_GNU_TYPE")
+    assert gt
     v, r, tl, f = gt.split(".")
     return aix_tag(v[-1], r, tl, bd)
 
@@ -47,7 +46,7 @@ def aix_pep425():
     # p = run(["/usr/bin/lslpp", "-Lqc", "bos.mp64"],
     #       capture_output=True, text=True)
     result = check_output(["/usr/bin/lslpp", "-Lqc", "bos.mp64"])
-    result = result.decode("utf-8").strip().split(":")  # type : ignore
+    result = result.decode("utf-8").strip().split(":")  # type: ignore
 
     (lpp, vrmf, bd) = list(result[index] for index in [0, 2, -1])
     assert lpp == "bos.mp64", "%s != %s" % (lpp, "bos.mp64")

I don't have much bandwidth to comment further or do a review, but checking locally, this seems to fix the mypy warnings.

aixtools · 2019-11-11T11:34:04Z

@pradyunsg Thx. Shall give it a try. As mypy was still failing I had not bothered to post my latest changes. So. I'll stash and try again.

aixtools · 2019-11-11T12:26:32Z

@pradyunsg - success for mypy, but tox -e py37 fails, I guess on coverage. Again, a result I know nothing about.

So, packaging/_AIX_platform.py has 0% coverage, and packaging/tags.py drops to 98%.

Again, hints welcome (I'll push these changes, even though failure is expected).


py37 run-test: commands[1] | python -m coverage report -m --fail-under 100
Name                         Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------
packaging/_AIX_platform.py      24     24      4      0     0%   3-62
packaging/__about__.py          10      0      0      0   100%
packaging/__init__.py            3      0      0      0   100%
packaging/_structures.py        41      0      0      0   100%
packaging/_typing.py             3      0      0      0   100%
packaging/markers.py           133      0     49      0   100%
packaging/requirements.py       79      0     24      0   100%
packaging/specifiers.py        275      0    124      0   100%
packaging/tags.py              259      6    125      1    98%   491-498, 490->491
packaging/utils.py              25      0     14      0   100%
packaging/version.py           189      0     88      0   100%
------------------------------------------------------------------------
TOTAL                         1041     30    428      1    97%
ERROR: InvocationError for command /data/prj/python/git/packaging/.tox/py37/bin/python -m coverage report -m --fail-under 100 (exited with code 2)

xavfernandez · 2019-11-11T21:01:43Z

@aixtools the packaging project has a policy to have 100% of its code tested so you'll need to write tests in https://github.com/pypa/packaging/blob/master/tests/test_tags.py exercising the code you wrote.

aixtools · 2019-11-11T22:06:01Z

I guessed as much, and have been playing.

But in all honesty, it has taken me 400% more time to try to understand, let alone accomplish, the additional tasks to get it past CI.

My approach has always been to insert as little as possible to the existing code - merely a trigger, such as platform.system == "AIX" and have all my contribution external to the existing project.

I suppose if your main interest in life is Python, you have either had all of these subjects in school, training at work, or just because you have done other projects. It feels (think something very negative) and also becomes extremely distracting. I have actually forgotten some of my ideas re: step by step progression. And now I fear something I had hoped might be accomplished in a few weeks may not be finished in a year - just because I run out of time between (work) trips - to get this past your requirements.

Well, time to get ready of tomorrow. I'll see when I can get back to this. Please do not close just because I lack time and/or skills to work with all these additional requirements. I shall not be updating the PR to test - I test locally as best I can and then push.

update: just checking (read hoping) for suggestions.

aixtools · 2019-11-20T01:39:23Z

I spent hours trying different things to get the last bit of code in packaging/tags.py to be covered, but I cannot find anything to convince the tool that the code is executing.

I can force errors and see that it gets reached.

After hours, stuck with this:

py37 run-test: commands[1] | python -m coverage report -m --fail-under 100
Name                         Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------
packaging/_AIX_platform.py      25      0      2      0   100%
packaging/__about__.py          10      0      0      0   100%
packaging/__init__.py            3      0      0      0   100%
packaging/_structures.py        41      0      0      0   100%
packaging/_typing.py             3      0      0      0   100%
packaging/markers.py           133      0     49      0   100%
packaging/requirements.py       79      0     24      0   100%
packaging/specifiers.py        275      0    124      0   100%
packaging/tags.py              260      1    125      1    99%   459, 454->459
packaging/utils.py              25      0     14      0   100%
packaging/version.py           189      0     88      0   100%
------------------------------------------------------------------------
TOTAL                         1043      1    426      1    99%

aixtools · 2019-11-20T20:27:40Z

OMG!! They all passed! I guess I should celebrate - like when my daughter said - hey grandpa, it's a boy!

aixtools · 2019-11-28T15:26:29Z

Hmm - says I left a review. If so, by accident.

Rebase, etc. etc. is making a big mess. Not a slight change.... Bah conflicts!

aixtools · 2019-12-17T08:36:05Z

CI failures are not related to the changes in this PR.

aixtools · 2019-12-26T16:12:52Z

9 days further. Any comments? I cannot resolve the CI issues - as they have nothing to do with my proposal.

aixtools · 2020-02-02T13:04:31Z

Question: What happens if I press the button "Update Branch"?

aixtools · 2020-02-03T08:46:47Z

@di Thanks for the help (magic!)

Remove all references to old name _AIX_platform.py

Rename module used for monkeypatch

* Make our mypy type-casting magic more robust Newer versions of mypy are better at following code flow, and assign Any types to our reimplemented cast function. This commit significantly restructures our typing.cast re-definition, to make it more robust. * Rename MYPY_CHECK_RUNNING -> TYPE_CHECKING This allows for some significant cleanups in our typing helpers, and makes it much easier to document our... workarounds. * Bump to newer mypy * Improve `canonicalize_name` safety with type hints

aixtools · 2020-04-20T17:37:15Z

Well, changes to codebase - seems what was working no longer works. Will take this up again later when I have more time to investigagte what, exactly, I need to update on my system so I can run these tests locally.

from tests/test_tags.py

Additional conflicts in coding expected due to changes in `master`.

aixtools · 2020-11-03T14:21:38Z

working through typos - slowly.

tox no arguments - passes

nox and nox -s lint both fail as:

nox > Running session lint
nox > Session lint skipped: Python interpreter 3.8 not found.

As I am working - as before - with py36 - what do I need to do extra to use nox et al, locally.

Thx.

aixtools · 2020-11-03T15:24:09Z

OK. Next step:

One of my new tests looks like:

def test_aix_bgt(monkeypatch):
    monkeypatch.setattr(_aix_support, "_have_subprocess", False)
    result = _aix_support._aix_bgt()
    assert result == [6, 1, 7]
    monkeypatch.setattr(_aix_support, "_have_subprocess", True)
    monkeypatch.setattr(sysconfig, "get_config_var", lambda k: "powerpc-ibm-aix5.3.7.0")
    result = _aix_support._aix_bgt()
    assert result == [5, 3, 7]

Something seems to have changed - because a year ago - the sysconfig.get_config_var('GNU_BUILD_TYPE') would return
"powerpc-ibm-aix5.3.7.0" and the strip command would work.

    # type: () -> List[int]
    if _have_subprocess:
        bgt = get_config_var("BUILD_GNU_TYPE")
    else:
        bgt = "powerpc-ibm-aix6.1.7.0"
    return _aix_vrtl(vrmf=str(bgt))

The test now fails - because the monkeypatch value is not getting pickup.

What do I need to change in:

monkeypatch.setattr(sysconfig, "get_config_var", lambda k: "powerpc-ibm-aix5.3.7.0")

so that the correct value is returned.

FYI: the current python returns (when _have_subprocess is TRUE)

Python 3.6.12 (default, Sep 23 2020, 08:27:01) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var("BUILD_GNU_TYPE")
'powerpc-ibm-aix5.3.12.0'
>>>

Not the value shown in the CI test:


tests/test_tags.py:1297: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
packaging/_aix_support.py:86: in _aix_bgt
    return _aix_vrtl(vrmf=str(bgt))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

vrmf = 'x86_64-pc-linux-gnu'

    def _aix_vrtl(vrmf):
        # type: (str) -> List[int]
>       v, r, tl = vrmf.split(".")[:3]
E       ValueError: not enough values to unpack (expected 3, got 1)

Many thanks for your assistence.

brettcannon · 2022-08-14T22:58:56Z

I'm going to close this as I think adding tag support is a rather serious ask since it's one of the hardest things to keep supporting. If there's still a desire to have AIX tags then we should open an issue about it and discuss how we want to manage what platforms we add tag support for when there isn't wide community demand for it (no offence to AIX).

aixtools · 2022-08-17T11:51:57Z

Actually, The better reason for closing this is because tag support is built-in since Python3.9. This was my attempt to integrate THAT code into this package so that bdist would work properly for earlier versions of Python for AIX that does not have Lib/_aix_support.py

brettcannon reviewed Aug 26, 2019

View reviewed changes

packaging/tags.py Outdated

The pep425 platform tag for AIX becomes:

AIX_V_R_M_YYWW_bitness

'''

Copy link

Member

brettcannon Aug 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

brettcannon reviewed Aug 26, 2019

View reviewed changes

aixtools force-pushed the AIX-platform-tag branch from 1ebd62e to 66d7b40 Compare November 6, 2019 13:21

aixtools force-pushed the AIX-platform-tag branch from 66d7b40 to e74cdc6 Compare November 6, 2019 21:30

brettcannon closed this Nov 6, 2019

xavfernandez reopened this Nov 7, 2019

brettcannon added the packaging.tags label Nov 14, 2019

aixtools changed the title ~~Add AIX platform tag support to Tag class~~ Modify AIX platform_tag so it provides PEP425 needs Nov 28, 2019

chrahunt mentioned this pull request Nov 28, 2019

Modify AIX platform_tag so it provides PEP425 needs pypa/pip#7375

Closed

redo PR and force push

26d4fcd

aixtools mentioned this pull request Dec 17, 2019

CI failing due to coverage failures #246

Closed

Merge branch 'master' into AIX-platform-tag

1dc7248

Merge branch 'master' into AIX-platform-tag

5c9166b

aixtools and others added 8 commits March 17, 2020 10:16

Merge branch 'master' into AIX-platform-tag

2f9fc76

rename _AIX_platform to _aix_support - same as in CPython

f9019b2

Merge branch 'master' into AIX-platform-tag

5572136

Sync _aix_support.py with CPython Lib/_aix_support.py

91412e8

Remove all references to old name _AIX_platform.py

Fix file to pass flake8 tests

81ebe3c

Rename module used for monkeypatch

import subprocess no longer needed to support monkeypatch

27ebbb1

Shorten variable names to pass flake8 line-length tests

cf52688

aixtools force-pushed the AIX-platform-tag branch from add56af to 68e9c80 Compare April 20, 2020 17:25

aixtools added 2 commits April 20, 2020 17:27

Merge branch 'master' into AIX-platform-tag

880edc3

typo

48dab36

aixtools added 3 commits November 1, 2020 18:43

Merge branch 'master' into AIX-platform-tag by removing the conflicts

5088c8c

from tests/test_tags.py

Restored the lines removed - in order to restart work.

ae6ee5f

Additional conflicts in coding expected due to changes in `master`.

Fix paste error; add a new-line

56aee30

Fix blank line with spaces

c9544b6

Base automatically changed from master to main January 21, 2021 19:20

brettcannon added the enhancement label Jul 1, 2021

brettcannon closed this Aug 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modify AIX platform_tag so it provides PEP425 needs #190

Modify AIX platform_tag so it provides PEP425 needs #190

aixtools commented Aug 25, 2019

brettcannon Aug 26, 2019

brettcannon Aug 26, 2019

brettcannon Aug 26, 2019

brettcannon Aug 26, 2019

aixtools Sep 1, 2019

brettcannon Sep 9, 2019

brettcannon Aug 26, 2019

aixtools Sep 1, 2019

pradyunsg commented Aug 27, 2019

aixtools commented Nov 6, 2019 •

edited by brettcannon

Loading

brettcannon commented Nov 6, 2019

aixtools commented Nov 6, 2019

brettcannon commented Nov 6, 2019

aixtools commented Nov 7, 2019

xavfernandez commented Nov 7, 2019 •

edited by brettcannon

Loading

aixtools commented Nov 8, 2019 •

edited

Loading

aixtools commented Nov 10, 2019

pradyunsg commented Nov 10, 2019 •

edited

Loading

aixtools commented Nov 11, 2019

aixtools commented Nov 11, 2019 •

edited

Loading

xavfernandez commented Nov 11, 2019

aixtools commented Nov 11, 2019 •

edited

Loading

aixtools commented Nov 20, 2019

aixtools commented Nov 20, 2019

aixtools commented Nov 28, 2019

aixtools commented Dec 17, 2019

aixtools commented Dec 26, 2019

aixtools commented Feb 2, 2020

aixtools commented Feb 3, 2020

aixtools commented Apr 20, 2020

aixtools commented Nov 3, 2020

aixtools commented Nov 3, 2020

brettcannon commented Aug 14, 2022

aixtools commented Aug 17, 2022 •

edited

Loading

Modify AIX platform_tag so it provides PEP425 needs #190

Modify AIX platform_tag so it provides PEP425 needs #190

Conversation

aixtools commented Aug 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradyunsg commented Aug 27, 2019

aixtools commented Nov 6, 2019 • edited by brettcannon Loading

brettcannon commented Nov 6, 2019

aixtools commented Nov 6, 2019

brettcannon commented Nov 6, 2019

aixtools commented Nov 7, 2019

xavfernandez commented Nov 7, 2019 • edited by brettcannon Loading

aixtools commented Nov 8, 2019 • edited Loading

aixtools commented Nov 10, 2019

pradyunsg commented Nov 10, 2019 • edited Loading

aixtools commented Nov 11, 2019

aixtools commented Nov 11, 2019 • edited Loading

xavfernandez commented Nov 11, 2019

aixtools commented Nov 11, 2019 • edited Loading

aixtools commented Nov 20, 2019

aixtools commented Nov 20, 2019

aixtools commented Nov 28, 2019

aixtools commented Dec 17, 2019

aixtools commented Dec 26, 2019

aixtools commented Feb 2, 2020

aixtools commented Feb 3, 2020

aixtools commented Apr 20, 2020

aixtools commented Nov 3, 2020

aixtools commented Nov 3, 2020

brettcannon commented Aug 14, 2022

aixtools commented Aug 17, 2022 • edited Loading

aixtools commented Nov 6, 2019 •

edited by brettcannon

Loading

xavfernandez commented Nov 7, 2019 •

edited by brettcannon

Loading

aixtools commented Nov 8, 2019 •

edited

Loading

pradyunsg commented Nov 10, 2019 •

edited

Loading

aixtools commented Nov 11, 2019 •

edited

Loading

aixtools commented Nov 11, 2019 •

edited

Loading

aixtools commented Aug 17, 2022 •

edited

Loading