Open and load --code file and --py-args arguments in C++ #60134

btzy · 2025-01-13T17:35:37Z

Description

I took a stab at #60120 to see if we can design a better alternative to #60121.

Instead of doing string substitution into Python code, we use the proper Python C API functions to assign to sys.argv and execute a Python script from a file. This avoids the issues that arise due to improper escaping of strings (e.g. ' and \). Unix file names in particular may contain any non-null character (including non-printable characters), and trying to escape such characters manually would otherwise be difficult to get right. String substitution is also possibly vulnerable to code injection attacks, as it is possible to carefully craft a string containing code that will be executed.

There was a hack that converts Windows-style directory separators to Unix-style directory separators. This is no longer necessary, and therefore has been removed.

Fixes #60120.

domi4484 · 2025-01-16T16:30:53Z

@btzy when we introduced --py-args we decided on purpose to make the args available for each python execution in QGIS independently from the --code argument.
But I can't remember exactly what's the use case thought, maybe @m-kuhn do you remember more?

@btzy If you need arguments valid only in the context of the --code script I would suggest to add a new parameter --code-args. But then arise the question what to do if they are both set?

m-kuhn · 2025-01-16T20:55:49Z

No strong opinion. What would be the reason to not make it also available for other python code than --code?

btzy · 2025-01-17T12:58:13Z

@domi4484 I misunderstood the help string of --py-args. I'll modify this PR. Likely we'll need another function in QgsPythonUtils to set the sys.argv variable (separate from runFile).

btzy · 2025-01-18T16:03:17Z

I'm not sure how to trigger the automatic Windows build that other PRs have; can someone help me get the GitHub Action to run?

btzy · 2025-01-18T17:56:37Z

Separately, I made #60183 to fix the grammar error in the help, which was what originally misled me to think that --py-args is only used with --code.

github-actions · 2025-01-19T13:40:23Z

🪟 Windows builds

Download Windows builds of this PR for testing.
Debug symbols for this build are available here.
(Built from commit 3e1671a)

🪟 Windows Qt6 builds

Download Windows Qt6 builds of this PR for testing.
(Built from commit 3e1671a)

btzy · 2025-01-23T16:01:59Z

I've tested that the Windows build works:

Passing --code runs the Python code
Passing --code with --py-args make the extra args available in the Python code in sys.argv
Passing --py-args, and then later opening the Python console and running sys.argv shows the arguments

btzy · 2025-01-28T16:11:42Z

@m-kuhn Can I get a review for this?

github-actions · 2025-02-12T02:46:39Z

The QGIS project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 14 days and is being automatically marked as "stale". If you think this pull request should be merged, please check

that all unit tests are passing
that all comments by reviewers have been addressed
that there is enough information for reviewers, in particular
- link to any issues which this pull request fixes
- add a description of workflows which this pull request fixes
- add screenshots if applicable
that you have written unit tests where possible
In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request.
If there is no further activity on this pull request, it will be closed in a week.

src/core/qgspythonrunner.h

m-kuhn · 2025-02-12T05:59:15Z

src/python/qgspythonutilsimpl.cpp

+  if ( !file.open( QIODevice::ReadOnly | QIODevice::Text ) )
+  {
+    ret = "Cannot open file";
+    goto error;


Please avoid goto

src/python/qgspythonutilsimpl.cpp

m-kuhn · 2025-02-12T06:02:42Z

src/python/qgspythonutilsimpl.cpp

+  if ( !argsobj )
+  {
+    ret = "Error occurred in PyList_New";
+    goto error;


m-kuhn · 2025-02-12T06:02:57Z

src/python/qgspythonutilsimpl.cpp

+      ret = QString( "SetArgvTraceback" ) + getTraceback();
+    else
+      ret = "Error occurred in PyImport_ImportModule";
+    goto error;


How would you like me to rewrite this code? We need to execute the cleanup code at the end whether or not this function succeeds.

The most C++-ic way to do this is to write an RAII wrapper class around PyGILState_Ensure/PyGILState_Release and PyObject/Py_XDECREF. Or for a more ad-hoc mechanism, I might have used the equivalent of std::experimental::scope_exit] (both Folly and Abseil have this in some form, but QGIS uses neither of them). I would have written code in one of the above two ways if this was for my own project, but I do not see existing code in QGIS that leverages either of them.

If the above solutions are unacceptable, we would have to either invert the guard clauses into nested if-statements, which is generally considered to be bad practice. We are thus left with goto statements, which seem to be an idiom for writing cleanup code in C.

What would be most appropriate for the QGIS codebase? Writing my own scope guard?

m-kuhn · 2025-02-12T06:04:55Z

Thanks, using sys.argv does look like the correct approach!

github-actions bot added this to the 3.42.0 milestone Jan 13, 2025

agiudiceandrea mentioned this pull request Jan 15, 2025

Add py-args and -- as command line parameters qgis/QGIS-Documentation#9556

Merged

1 task

btzy force-pushed the codearg branch from 0120199 to 45ae0c0 Compare January 18, 2025 15:53

btzy marked this pull request as ready for review January 18, 2025 15:54

btzy force-pushed the codearg branch from 45ae0c0 to 6dd2003 Compare January 18, 2025 17:44

btzy force-pushed the codearg branch 4 times, most recently from ef2a7f6 to 20e2318 Compare January 19, 2025 13:10

btzy force-pushed the codearg branch 6 times, most recently from 64c5dda to 1f44ac4 Compare January 22, 2025 15:34

btzy force-pushed the codearg branch from 1f44ac4 to 4efb330 Compare January 24, 2025 17:43

btzy changed the title ~~Open and load --code file and arguments in C++~~ Open and load --code file and --py-args arguments in C++ Jan 25, 2025

nyalldawson mentioned this pull request Jan 31, 2025

[App] Allow to use single quotes in file name with the --code command line option (Fix #60120) #60121

Closed

github-actions bot added the stale Uh oh! Seems this work is abandoned, and the PR is about to close. label Feb 12, 2025

m-kuhn reviewed Feb 12, 2025

View reviewed changes

src/core/qgspythonrunner.h Outdated Show resolved Hide resolved

m-kuhn reviewed Feb 12, 2025

View reviewed changes

src/python/qgspythonutilsimpl.cpp Outdated Show resolved Hide resolved

m-kuhn reviewed Feb 12, 2025

View reviewed changes

src/python/qgspythonutilsimpl.cpp

if ( !argsobj )

{

ret = "Error occurred in PyList_New";

goto error;

Copy link

Member

m-kuhn Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

m-kuhn reviewed Feb 12, 2025

View reviewed changes

github-actions bot removed the stale Uh oh! Seems this work is abandoned, and the PR is about to close. label Feb 12, 2025

btzy force-pushed the codearg branch from 4efb330 to 31fbb4a Compare February 12, 2025 15:46

Open and load --code file in C++

3e1671a

btzy force-pushed the codearg branch from 31fbb4a to 3e1671a Compare February 12, 2025 16:10

nyalldawson added Frozen Feature freeze - Do not merge! and removed Frozen Feature freeze - Do not merge! labels Feb 20, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Open and load --code file and --py-args arguments in C++ #60134

Open and load --code file and --py-args arguments in C++ #60134

btzy commented Jan 13, 2025 •

edited

Loading

domi4484 commented Jan 16, 2025

m-kuhn commented Jan 16, 2025

btzy commented Jan 17, 2025 •

edited

Loading

btzy commented Jan 18, 2025

btzy commented Jan 18, 2025

github-actions bot commented Jan 19, 2025 •

edited

Loading

btzy commented Jan 23, 2025 •

edited

Loading

btzy commented Jan 28, 2025

github-actions bot commented Feb 12, 2025

m-kuhn Feb 12, 2025

m-kuhn Feb 12, 2025

m-kuhn Feb 12, 2025

btzy Feb 12, 2025

m-kuhn commented Feb 12, 2025

Open and load --code file and --py-args arguments in C++ #60134

Are you sure you want to change the base?

Open and load --code file and --py-args arguments in C++ #60134

Conversation

btzy commented Jan 13, 2025 • edited Loading

Description

domi4484 commented Jan 16, 2025

m-kuhn commented Jan 16, 2025

btzy commented Jan 17, 2025 • edited Loading

btzy commented Jan 18, 2025

btzy commented Jan 18, 2025

github-actions bot commented Jan 19, 2025 • edited Loading

🪟 Windows builds

🪟 Windows Qt6 builds

btzy commented Jan 23, 2025 • edited Loading

btzy commented Jan 28, 2025

github-actions bot commented Feb 12, 2025

m-kuhn Feb 12, 2025

Choose a reason for hiding this comment

m-kuhn Feb 12, 2025

Choose a reason for hiding this comment

m-kuhn Feb 12, 2025

Choose a reason for hiding this comment

btzy Feb 12, 2025

Choose a reason for hiding this comment

m-kuhn commented Feb 12, 2025

btzy commented Jan 13, 2025 •

edited

Loading

btzy commented Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 19, 2025 •

edited

Loading

btzy commented Jan 23, 2025 •

edited

Loading