[4.1.0rc2] sqlalchemy InvalidRequestError: This nested transaction is inactive when trying to activate embedding on a dashboard

[ablanchard]

Bug description

On a fresh install of superset 4.1.0rc2, I have imported dashboards from my 4.0.2 instance and I try to activate the embedding feature on it.
No matter if I do it via api or via the UI, every time I end up with the stack:

{"written_at": "2024-09-10T15:28:55.338Z", "written_ts": 1725982135338642000, "msg": "This nested transaction is inactive", "type": "log", "logger": "flask_appbuilder.api", "thread": "ThreadPoolExecutor-0_8", "level": "ERROR", "module": "__init__", "line_no": 115, "exc_info": "Traceback (most recent call last):
  File \"/usr/local/lib/python3.10/site-packages/flask_appbuilder/api/__init__.py\", line 111, in wraps
    return f(self, *args, **kwargs)
  File \"/app/superset/views/base_api.py\", line 119, in wraps
    duration, response = time_function(f, self, *args, **kwargs)
  File \"/app/superset/utils/core.py\", line 1364, in time_function
    response = func(*args, **kwargs)
  File \"/app/superset/utils/log.py\", line 303, in wrapper
    value = f(*args, **kwargs)
  File \"/app/superset/dashboards/api.py\", line 136, in wraps
    return f(self, dash)
  File \"/app/superset/dashboards/api.py\", line 1506, in set_embedded
    with db.session.begin_nested():
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/util.py\", line 235, in __exit__
    with util.safe_reraise():
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py\", line 70, in __exit__
    compat.raise_(
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py\", line 211, in raise_
    raise exception
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/util.py\", line 233, in __exit__
    self.commit()
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py\", line 839, in commit
    trans.commit()
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 2469, in commit
    self._do_commit()
  File \"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py\", line 2787, in _do_commit
    raise exc.InvalidRequestError(
    sqlalchemy.exc.InvalidRequestError: This nested transaction is inactive
", "filename": "__init__.py", "hostname": "superset-7dd85c9446-b8psq"}

To note: I'm activating the embedding without giving a domain.
I have tried reproducing the error on a docker instance in 4.1.0rc2 but I haven't succeeded. I have try with both the examples dashboards and imported ones.
I saw that the code changed recently in this PR
Do you have any clue why this behavior can happen ?

How to reproduce the bug

1. Go to a dashboard
2. Click on "Embed dashboard" in the contextual menu
3. On the modal click on "Enable embedding"

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.9

Node version

16

Browser

Firefox

Additional context

No response

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot]

Hello @ablanchard! I'm here to assist you with any bugs, questions, or contributions related to Superset. Let's dive in and tackle this issue together!

The error "sqlalchemy InvalidRequestError: This nested transaction is inactive" occurs when trying to activate embedding on a dashboard in Superset 4.1.0rc2 because the nested transaction is being used incorrectly or has already been rolled back or committed.

In the provided code, the issue likely arises from the following block:

with db.session.begin_nested():
    embedded = EmbeddedDashboardDAO.upsert(
        dashboard,
        body["allowed_domains"],
    )

If an exception occurs within the begin_nested block, the nested transaction becomes inactive, leading to the InvalidRequestError when further operations are attempted.

To handle this properly, ensure that the nested transaction is correctly managed and that any exceptions are handled appropriately to avoid leaving the transaction in an inactive state. Here is an example of how you might handle this:

try:
    with db.session.begin_nested():
        embedded = EmbeddedDashboardDAO.upsert(
            dashboard,
            body["allowed_domains"],
        )
    result = self.embedded_response_schema.dump(embedded)
    return self.response(200, result=result)
except ValidationError as error:
    return self.response_400(message=error.messages)
except Exception as e:
    db.session.rollback()  # Rollback the session to avoid inactive transaction
    logger.error("Error during embedding activation: %s", str(e))
    return self.response_500(message="An error occurred while activating embedding.")

This ensures that if an exception occurs, the session is rolled back, preventing the nested transaction from becoming inactive ^[1].

To continue talking to Dosu, mention @dosu.

[mujibishola]

@ablanchard Did you ever get this resolved?

[ablanchard]

@mujibishola No. I have tried to reproduce in docker but without success.

[rusackas]

@ablanchard you mean you're unable to reproduce the bug in docker, so we're all good? Or you're unable to resolve the error? Not sure if you or anyone are seeing this as a 4.1 blocker... CC @sadpandajoe as release manager and @michael-s-molina as a reviewer of the linked PR, in case either of them want to track this further.

[ablanchard]

@rusackas On my kubernetes deployed instance I have the stacktrace. When I do the same on a docker instance, I don't have the stacktrace. Giving the stacktrace, is there anything I can do in the step to reproduce, to be sure to have the stacktrace also on the docker one ?

[mujibishola]

@ablanchard see my fix

Location

File: superset/dashboards/api.py

Issue

The current implementation is failing to save embedded dashboards properly.

Fix

Replace the following code:

try:
    body = self.embedded_config_schema.load(request.json)

    with db.session.begin_nested():
        embedded = EmbeddedDashboardDAO.upsert(
            dashboard,
            body["allowed_domains"],
        )

    result = self.embedded_response_schema.dump(embedded)
    return self.response(200, result=result)
except ValidationError as error:
    return self.response_400(message=error.messages)

With this updated version:

try:
    body = self.embedded_config_schema.load(request.json)
    embedded = EmbeddedDashboardDAO.upsert(dashboard, body["allowed_domains"])
    result = self.embedded_response_schema.dump(embedded)
    
    return self.response(200, result=result)
except ValidationError as error:
    return self.response_400(message=error.messages)

Changes

1. Removed the db.session.begin_nested() context manager.
2. Directly call EmbeddedDashboardDAO.upsert() without wrapping it in a transaction.

Rationale

The nested transaction was interfering with the saving process of embedded dashboards. Removing it allows the save operation to complete successfully.

[ablanchard]

@mujibishola Hello, seems to fix my problem. You will make a PR with this changes, this way I can deploy it on my test environment. I can also make the PR if you want

[viveksingh-numerator]

i am facing the same issue. is there a way to fix it manually on k8 setup? can not modify the code and using the pre release images from this repo.