-
-
Notifications
You must be signed in to change notification settings - Fork 381
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Tool
base class
#1093
Conversation
64588e3
to
0f3f9fb
Compare
@@ -14,7 +14,7 @@ def verify_tools(self): | |||
# Require XCode 10.0.0. There's no particular reason for this | |||
# specific version, other than it's a nice round number that's | |||
# not *that* old at time of writing. | |||
verify_xcode_install(self.tools, min_version=(10, 0, 0)) | |||
Xcode.verify(self.tools, min_version=(10, 0, 0)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is no longer correct - we definitely need at least Xcode 13.0 at this point, due to file formats.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bumped to 13.0.0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've had an initial look through this; definitely looks like it's heading in the right direction.
Not sure if it was on your todo list or not - but the upgrade
command can be significantly cleaned up once this is in place, because we can iterate through all tools with a consistent upgrade interface, rather than needing to list tools specifically.
a3c63d5
to
87d2272
Compare
b1bc051
to
8b7f6dc
Compare
47a30af
to
61a3080
Compare
Finalizing the In order to blindly loop over all the tools to identify the ones that can be upgraded, each Tool needs to actually be verified. This presents challenges because 1) certain tools are only supported on certain platforms and 2) some require parameters for initialization that won't be available to the upgrade command. I don't think I'm too keen to potentially over-engineer this and add a whole layer of "host verification" like we did for the Commands themselves. Although, this is an option. Alternatively, I can leave it like I have it now....where errors in trying to verify a Tool are interpreted as "the tool cannot be upgraded". I'm not a big fan of this since it isn't being explicit at all...but it works. Some middle ground perhaps? like a class-level flag on each Tool to signify whether a Tool could qualify for being managed; if it's set Any thoughts/insight would be appreciated. |
I agree - this isn't a feature that needs to be overly complicated.
That was my first thought. A bare-bones check with a class attribute to perform a simple OS check (so you don't try to verify Xcode on Linux, or Linuxdeploy on macOS) avoids needless work; if it meets that criterion, try to upgrade, and report the problem if it didn't work. |
Ok; a simple host OS check will be simple enough. However, there are still other issues as long as introspecting whether a Tool is managed is gated behind verification. Some tools require a bit of contextual information to verify (e.g. |
6297f57
to
47f30ea
Compare
While there may be benefit for DockerAppContext to extend Tool, it's not a Tool in the sense of any of the others - it's a binding of Docker to an App. There's not really an "upgrade" process for these, unless you count purging/rebuilding the Docker image - but we don't have any way to track "all images that have been built", I think it's safe to put on the "can't be managed" list. LinuxDeployURLPlugin is a weird one, because it's not a single plug-in. It's a wrapper allowing end-users to specify a plugin that is available at a URL. However, we don't have any specific details of that URL - we only have a unique hash. The same goes with LinuxDeployFilePlugin. I think the best we can do for LinuxDeployPlugins is to ensure that the plugins folder has been purged, so that the next time a plugin is requested, we are certain to have a clean download. Are there any other tools that are problematic? |
47f30ea
to
3527db0
Compare
Just these since I implemented Host OS verification:
This brings up something that's been periodically coming to mind: what is a tool exactly? For instance, I don't have fully formed thoughts on this yet...
hmm...at first thought, this starts to go outside of the scope of the Upgrade command...I'll have to think about it more... |
f95c09c
to
79a8061
Compare
Spent a little more time thinking about the more overarching ideas. Please let me know your thoughts; I've pushed a prototype implementation of this. Delineating Managed ToolsI've created a dedicated What is a tool?The more relevant pragmatic question is simply why subclass
The Tool registry is immediately applicable to the Upgrade command (obviously) to iterate the tools; but anything that would need to arbitrarily to be considered a part of Briefcase's toolset should be a If something needs to implement its own verification before it can be used or if its install needs to be managed, it should be a So, with that in mind, something like This categorization may present some ideological inconsistencies, though. The
|
Name | Is a Tool? | Managable? | FS Footprint? |
---|---|---|---|
Android SDK | X | X | X |
ADB | |||
Docker | X | X | |
DockerAppContext | X | X | |
Download | X | ||
Flatpak | X | X | |
Git | X | X | |
JDK | X | X | X |
LinuxDeployBase | |||
LinuxDeployPluginBase | |||
LinuxDeployGtkPlugin | X | X | X |
LinuxDeployQtPlugin | X | X | X |
LinuxDeployLocalFilePlugin | X | ||
LinuxDeployURLPlugin | X | X | |
LinuxDeploy | X | X | X |
RCEdit | X | X | X |
Subprocess | X | ||
NativeAppContext | X | ||
VisualStudio | X | X | |
WindowsSDK | X | X | |
WiX | X | X | X |
Xcode | X | X | |
XcodeCliTools | X | X |
dd24a5e
to
3256dde
Compare
That sounds like a really solid baseline definition to me.
I think this is the key insight. We need to differentiate between something:
Most of the Tools are case (1).
From a purely theoretical perspective, I'm sure this could be argued either way; ultimately, I think it's the practicalities of whether the abstraction is more help or harm that matters here.
Totally agreed on this one - although I'd also be completely comfortable with "delete all existing plugins" as a viable |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Finally wrapped this up; was waffling on some of the typing stuff. I've added comments to things I wanted to proactively call out. I also updated the top post with a PR summary.
def docker_data_path(self): | ||
def docker_data_path(self) -> PurePosixPath: | ||
"""The briefcase data directory used inside container.""" | ||
return "/home/brutus/.cache/briefcase" | ||
return PurePosixPath("/home/brutus/.cache/briefcase") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Returning a Path
here instead to better align with all the other *_path
command properties.
def verify_install( | ||
cls: type[LinuxDeployT], | ||
tools: ToolCache, | ||
install: bool = True, | ||
**kwargs, | ||
) -> LinuxDeployT: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding explicit typing to cls
helps the IDE figure out which actual LinuxDeploy class will be returned....since it will know which class is calling verify_install()
.
This is also how a type checker in general would figure out which Class will actually come out of this function.
tests/commands/upgrade/test_call.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had two goals for testing:
- Each possible workflow for Upgrade was exercised
- The expected actions (or lack thereof) are taken for all tools that were in scope for the test
# Tools that are intentionally not annotated in ToolCache. | ||
tools_unannotated = {"cookiecutter"} | ||
# Tool names to exclude from the dynamic annotation checks; they are manually checked. | ||
tool_names_skip_dynamic_check = { | ||
"app_context", | ||
"git", | ||
"xcode", | ||
"xcode_cli", | ||
"ETC_OS_RELEASE", | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The xcode_cli
Tool remains in this list because the test is actually checking if all Tools are defined as modules in briefcase.integrations
. That is true for xcode
but not for xcode_cli
. This is all a little wonky but really just an edge case to all this pseudo type checking...
I guess an alternative would be to promote XcodeCliTools
to its own module....but that doesn't seem especially necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed; they're sufficiently closely bound that I don't think it's worth splitting them out for the sake of making a test less confusing; however, it would be worth adding this comment into the code itself so that the context isn't forgotten.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The linuxdeploy tests were refactored to avoid using Dummy classes.
def inner(sub, args, **kwargs): | ||
def inner(*args: SubprocessArgsT, **kwargs): | ||
"""Evaluate whether conditions are met to remove any dynamic elements in the | ||
console before returning control to Subprocess. | ||
|
||
:param sub: Subprocess object | ||
:param args: list of implicit strings that will be run as subprocess command | ||
:return: the return value for the Subprocess method | ||
""" | ||
sub: Subprocess = args[0] | ||
sub_args: SubprocessArgsT = args[1] | ||
pos_args: Sequence[Any] = args[2:] | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Properly typing a method decorator is near impossible...
Nonetheless, exposing the signature this way and manually extracting the arguments like this helps trick the IDE in to showing a more accurate signature when attempting to autocomplete. That's the goal anyway....
This also returns support for positional arguments....since this would've just errored previously.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hrm... not sure how I feel about this. When you have to dramatically change your coding style to appease a tool, my eyebrows start to raise... are we sure the juice is worth the squeeze in this case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's fair; I reverted it (but did add support for positional arguments).
I'll try to get proper typing for the decorator working.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Woof... that's a monster :-)
tl;dr - this mostly looks good to me. The parts that are actually about the Tool/ManagedTool upgrade seem mostly uncontroversial, and a good improvement.
Most of the issues that I saw are with the typing changes. For the most part, they're uncontroversial as well; the problems mostly fall into "what is the right way to type?" gap. It's entirely possible you're doing the right thing, and that it's my own knowledge that is lacking.
My bigger concern with the typing is maintaining the types - unless they're being actively maintained, they're going to decay, at which point they become more harm than good. You've clearly got something enabled in your editor that is checking these; but unless we're all using the same editor, we're not going to benefit. Is there any way to automate type checks as a CI concern? Do we just need to bite the bullet and add Mypy?
@@ -1258,7 +1270,7 @@ def run(self, *arguments, quiet=False): | |||
raise InvalidDeviceError("device id", self.device) from e | |||
raise | |||
|
|||
def install_apk(self, apk_path): | |||
def install_apk(self, apk_path: SubprocessArgT): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this right? Shouldn't it be a str/Path?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SubprocessArgT
is Union[str, Path]
....so, yes. But I think I see your point. I typed this based on the usage of apk_path
within install_apk()
instead of how the signature should be presented to devs.
I'll update it to str | Path
.
@@ -1342,7 +1354,7 @@ def start_app(self, package, activity, passthrough): | |||
f"Unable to start {package}/{activity} on {self.device}" | |||
) from e | |||
|
|||
def logcat(self, pid): | |||
def logcat(self, pid: str): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it a str? seems like it should be an int...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on the current implementation, it must be a string because subprocess.Popen()
does not accept int
s in the command line to pass to the OS.
We can coerce it to a str
, though.
def inner(sub, args, **kwargs): | ||
def inner(*args: SubprocessArgsT, **kwargs): | ||
"""Evaluate whether conditions are met to remove any dynamic elements in the | ||
console before returning control to Subprocess. | ||
|
||
:param sub: Subprocess object | ||
:param args: list of implicit strings that will be run as subprocess command | ||
:return: the return value for the Subprocess method | ||
""" | ||
sub: Subprocess = args[0] | ||
sub_args: SubprocessArgsT = args[1] | ||
pos_args: Sequence[Any] = args[2:] | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hrm... not sure how I feel about this. When you have to dramatically change your coding style to appease a tool, my eyebrows start to raise... are we sure the juice is worth the squeeze in this case?
src/briefcase/integrations/xcode.py
Outdated
|
||
ensure_xcode_is_installed(tools=tools, min_version=min_version) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure it really matters, but any reason for keeping these standalone functions rather than methods on the class?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Originally, I was probably rationalizing leaving them to avoid churn....but then this PR exploded a bit...
Converted them to methods.
# Tools that are intentionally not annotated in ToolCache. | ||
tools_unannotated = {"cookiecutter"} | ||
# Tool names to exclude from the dynamic annotation checks; they are manually checked. | ||
tool_names_skip_dynamic_check = { | ||
"app_context", | ||
"git", | ||
"xcode", | ||
"xcode_cli", | ||
"ETC_OS_RELEASE", | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed; they're sufficiently closely bound that I don't think it's worth splitting them out for the sake of making a test less confusing; however, it would be worth adding this comment into the code itself so that the context isn't forgotten.
|
||
# Ensure defined Tool modules/classes are annotated in ToolCache. | ||
for tool_module_name in briefcase.integrations.__all__: | ||
if tool_module_name not in tools_unannotated: | ||
assert tool_module_name in ToolCache.__annotations__.keys() | ||
for tool_name in tools_for_module(tool_module_name): | ||
for tool_name in tools_for_module(tool_module_name).keys(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is keys actually needed here? Isn't the default iterable over keys?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I distinctly remember something bombing out without this....but it isn't happening anymore... 🤔
I'll remove them.
if tool_name not in tool_klasses_skip_dynamic_checks: | ||
assert tool_name in ToolCache.__annotations__.values() | ||
|
||
# Ensure annotated tools use valid Tool names. | ||
for tool_name, tool_klass_name in ToolCache.__annotations__.items(): | ||
if tool_name not in tool_names_skip_dynamic_check: | ||
assert tool_name in briefcase.integrations.__all__ | ||
assert tool_klass_name in tools_for_module(tool_name) | ||
assert tool_klass_name in tools_for_module(tool_name).keys() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed.
🥳
PyCharm enables type checking by default; so, if you call a method using the wrong type for an argument, it'll add red squiggles under the argument. I'd have to imagine that VSCode supports something similar. Although, your point still stands.
hmm....I can play around with this....but running I can take a stab at it, though. |
If you're curious, I pushed a commit for what type checking support might start looking like. Admittedly, there might be better ways to do some of this. Some complications and barriers:
So, running automated type checking may be possible....but not ultimately without some pain and compromise. |
Hrm... none of those seem especially appealing. I guess that means our options are:
3 seems like overkill, and 1 sounds like it's really not worth the effort (both in terms of the immediate refactoring that is required, and the ongoing headaches of maintaining it). So - I guess that leaves 2. |
This was my conclusion as well. I have reverted the type-checking changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I said in my last review - the core Tool stuff all looks solid; and AFAICT all the typing stuff makes sense now as well (or, at least, as much sense as it's ever going to make :-) I've pushed one minor tweak adding a clarifying docstring, but otherwise I think this is good to go!
Nice work - you've definitely earned a celebratory beverage of choice on this one :-)
Changes
Tool
andManagedTool
base classes for all toolsToolCache
that contains verified Tool instancesUpgradeCommand
to abstract out Tool implementation detailsNotes
mypy
.mypy
returns a torrent of issues...but it was very useful to verify my typing additions.PR Checklist: