Mismatch of python versions in flow's environment

[anjsam1402]

I have recently started exploring Prefect and Prefect Cloud using a sample code.
Local environment python version : 3.8.5
ECR container python version : 3.8
The flow always gets registered, but throws the same error:

Failed to load and execute Flow's environment: StorageError("An error occurred while unpickling the flow:\n TypeError('code() takes at most 15 arguments (16 given)')\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - python: (flow built with '3.8.5', currently running with '3.7.9')")

[zanieb]

It seems very likely that your ECR container is somehow running on a different version than you think. sys.version does not tend to be wrong. What same code are you trying to use?

[anjsam1402]

@mashun4ek
Got the same logs for using docker image prefecthq/prefect:latest-python3.7 for ECR container

[mashun4ek]

Hmm, I have trouble to reproduce this error. Could you do the following please:
Run prefect version on you local machine, pull and push the image with specific version prefecthq/prefect:<version_of_prefect_on_your_laptop>-python3.8 to your ECR repository, register a new flow version and run the new version of your flow.

[anjsam1402]

@mashun4ek
This time I created a new local environment with python : 3.8.5 and prefect : 0.14.16, and the Docker environment with prefecthq/prefect:0.14.16-python3.8, but getting similar error but this time the python version for execution environment changed to 3.7.10
Error

Failed to load and execute Flow's environment: StorageError("An error occurred while unpickling the flow:\n TypeError('code() takes at most 15 arguments (16 given)')\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - python: (flow built with '3.8.5', currently running with '3.7.10')")

For your reference:
Following is the sample code:

from prefect.storage import S3
from prefect.run_configs import ECSRun
from prefect import task, Flow, Parameter
import prefect
from prefect.agent.ecs.agent import ECSAgent

STORAGE = S3(bucket="prefect-store")
RUN_CONFIG = ECSRun(
    run_task_kwargs={
        "cluster":"prefect-ecs-fargate-cluster",
        "networkConfiguration": {'awsvpcConfiguration': {'assignPublicIp': 'ENABLED', 'subnets': ['subnet-{{ID1}}', 'subnet-{{ID2}}'], 'securityGroups': []}}
        },
    task_role_arn="arn:aws:iam::{{VALUE}}:role/PrefectECSTask-S3-ECR-Role",
    memory="512",
    cpu="256")

@task
def hi():
    logger = prefect.context.get('logger')
    logger.info("Hello!")


with Flow("prefect-fargate", storage=STORAGE, run_config=RUN_CONFIG) as flow:
    hi()

flow.register("fargate-flow")

#########
# agent #
#########

if __name__ == '__main__':
    AGENT = ECSAgent(
        cluster="prefect-ecs-fargate-cluster",
        task_role_arn="arn:aws:iam::{{VALUE}}:role/PrefectECSTask-S3-ECR-Role"
        )
    AGENT.start()

Dockerfile :

FROM prefecthq/prefect:0.14.16-python3.8
EXPOSE 8000
RUN pip3 install --upgrade pip
RUN pip3 install --upgrade setuptools && \
    pip3 install "prefect[aws]"

[anjsam1402]

I have got a fix for my issue. The issue was because of the default pickle-based storage of S3 storage which is a default choice; it needs to be replaced by script-based storage.

@madkinsz @mashun4ek
I think the pickle-based storage needs to be fixed as it runs the flow in a different python version.

[mashun4ek]

Hello @anjsam1402! I'm glad stored_as_script worked for you. Indeed, script based flow storage is more flexible than pickled storage. I still can't reproduce the issue you faced, I get the error only when python version mismatch between the image python version and python version used on my local machine, but I will keep an eye on the issues related to picked storage.