Multiple Gill Aquire

[letalboy]

Bug Description

I was trying to create a pool of python interpreters to be able to process thing more fast and call multiple callbacks from inside rust at once in a project here to accelerate the processing, however I find that i can't do it in a easy way, I have done a search here and an exploratory analysis of the PyO3 code and docs related to python acquiring using PyO3 and I find a way to do it using this structure:

let py;

{
    let getting_py = unsafe { Python::assume_gil_acquired() };

    let gil_pool = unsafe { getting_py.clone().new_pool() };

    py = gil_pool.python();

    logger.debug(format!("Aquired python in a process task!"));

    process(py, dow_command);

    logger.debug(format!("Finalize a process task!"));
}
        

Were the rust will call a process function that have multiple mechanisms to call callbacks in certain situations and to not call in others. Since the project is private I can't share more details about the project structures, and even if it is not private I also will have a lot of work to pass it here since is a 30k+ lines code with, lot of macros and complex multithreading functionalities, however the principal point in my analysis here is that when I use this structure that is inside a for loop that process a queue of tasks using multithreading I have random exceptions generated in the event of a gill being acquired in the same time another one is using it, but if I do it in a queue format it works fine using the example above, however I need another a way to process it using multiple python instances to be able to call it from multiple places without having this issues, is it possible to do, or will be needed to develop something new?

I also tryed to find how to do it however I found nothing on the internet and ChatGPT 4 also don't know, so I tryed to develop ways to go around it, I did a custom queue pool for python gill but isn't working 100% and face the same problem that even using that the processing mechanism needs to use only one python instance at time, and I need multiple ones...

I also know that we can't use the rust unsafe tags because it will produce error like this because of other threading using variables that are in use by one thread and also know that this is one of the most complicated subjects in any language like streams data structure in rust case, however I really need to find a solution to use multiple python instances at once in my application without getting these types of errors.

I will be wainting some feedback and while i waiht i will be using the queue configuration that i describe that only uuses one python instance at once, tks by now! ;)

ATT
Cristian Camargo (Poseidon)

Steps to Reproduce

1. Initialize a PyO3 lib
2. Setup some callback mecanisms
3. Then use the code above that get a python temporary instance by gill aquire
4. Then Use multiple gill aquire in diferent threads runing at once to call calback
5. This will make the aplication simple stop doing some of the things or skip or crash not chowing any errors or chowing weird errors like the system memory errors in some cases

Backtrace

No response

Your operating system and version

I tested in Windows and a Linux (ubunto) env

Your Python version (python --version)

Python 3.10

Your Rust version (rustc --version)

rustc 1.70.0 (90c541806 2023-05-31)

Your PyO3 version

i Use the last one (using version = "*")

How did you install python? Did you use a virtualenv?

py -m venv .venv

Additional Info

No response

[mejrs]

Pyo3 does not support using multiple interpreters, and trying to get around it like this will cause undefined behavior.

[letalboy]

Is it possible to implement something like this, or not because it has limitations on the python side? Because if it's possible, I'm really in to help implement something like this, if by chance we can run several python calls at once, that will be a very good thing, imagine a rust library for python that doesn't just manage things in an orderly way , of course, many cases do not need it, but those that do will benefit greatly, as it would greatly speed up the call of several callbacks if it could be called in parallel

[messense]

@letalboy See #576

[davidhewitt]

I'm closing this now we've redirected conversation to #3451

[letalboy]

Good idea! This way the issue will not be split in two different lines and avoid confusions related to duplicated ideas that can be posted here or in the other that we are redirecting simultaneously!