Define mocking points to allow easy unit testing

[pskowronek]

Hi All,

I would like to start a discussion about defining mocking points in boto3 (or botocore if relevant) so the mocking could be easier, more robust and reliable across released versions of boto3. Please take a look at moto project and their issues with proper mocking AWS requests - they struggle to keep the pase with boto3 releases and properly find and mock all the places in boto3/botocore etc that make requests to AWS. Failing to find all those places causes tests to make true requests to AWS - and that happens way too often.

The idea behind moto is sound and I guess everybody agrees that project such as moto can be helpful for unit testing. Having such mocking points well defined in boto3 and documented would help such projects a lot - now they need to chase the rabbit wasting time and energy just to find right places to override the requests.

Cheers

[swetashre]

Thank you for your post. Is there any specific moto issue you are talking about ?

You can use stubber for testing with botocore and in that way it won't make true requests to aws.
Documentation for stubber:
https://botocore.amazonaws.com/v1/documentation/api/latest/reference/stubber.html

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[pskowronek]

I'm sorry for late reply - I had no chance to reply on time.

So, here is the list of moto issues that I think could be fixed - not sure if it is moto that they do something wrong or unexpected or maybe they need to do that in other ways because mentioned above stubber isn't enough?

getmoto/moto#1793
getmoto/moto#2076
getmoto/moto#2058
getmoto/moto#2413

[mattsb42-aws]

For some context on this from the perspective of a moto user:

Stubber and moto are different tools that solve similar problems by doing very different things.

The core difference is that stubber provides mocks while moto provides fakes.

Mocks give you back responses that need to be pre-defined, and will always give back the same response no matter what.

Fakes attempt to actually replicate the behavior of the thing that your code would normally be interacting with, but without interacting with that thing. Notably, they know what the behaviors they need to replicate are; they do not require you to tell them each time.

Mocks have their uses, but they are much more error prone because they depend on each and every user to set up the responses, and for that setup to be correct.

As an example, let's look at setting up a test that encrypts and decrypts data with KMS.

examples

moto

As of the next release, when KMS encrypt/decrypt behavior will be correctly modeled, I need to apply the KMS mock, create my CMK, and encrypt and decrypt data.

import boto3
from moto import mock_kms


@mock_kms
def test_encrypt_decrypt():
    client = boto3.client("kms")
    new_cmk = client.create_key()
    cmk_arn = new_cmk["KeyMetadata"]["Arn"]

    plaintext = b"my secret data"
    encryption_context = {
        "some": "encryption",
        "context": "here",
    }
    encrypt_response = client.encrypt(
        Plaintext=plaintext,
        KeyId=cmk_arn,
        EncryptionContext=encryption_context,
    )

    decrypt_response = client.decrypt(
        CiphertextBlob=encrypt_response["CiphertextBlob"],
        EncryptionContext=encryption_context,
    )

stubber

In comparison, if I want to use stubber to do this, I need to manually tell it everything I expect to happen:

import boto3
import botocore.session
from botocore.stub import Stubber

def _create_cmk(client):
    new_cmk = client.create_key()
    return new_cmk["KeyMetadata"]["Arn"]


def _mock_encrypt_decrypt_with_encryption_context(stubber, key_id, plaintext, ciphertext, encryption_context):
    encrypt_params = {
        "KeyId": key_id,
        "Plaintext": plaintext,
        "EncryptionContext": encryption_context,
    }
    encrypt_response = {
        "CiphertextBlob": ciphertext,
        "KeyId": key_id,
    }

    decrypt_params = {
        "CiphertextBlob": ciphertext,
        "EncryptionContext": encryption_context,
    }
    decrypt_response = {
        "Plaintext": plaintext,
        "KeyId": key_id,
    }

    stubber.add_response("encrypt", encrypt_response, encrypt_params)
    stubber.add_response("decrypt", decrypt_response, decrypt_params)


@mock_kms
def test_encrypt_decrypt():
    client = botocore.session.get_session().create_client("kms")
    client = boto3.client("kms")
    cmk_arn = "put-a-uuid-here"

    plaintext = b"my secret data"
    ciphertext = b"this is totally actually encrypted ;)"
    encryption_context = {
        "some": "encryption",
        "context": "here",
    }
    with Stubber(client) as stubber:
        _mock_encrypt_decrypt_with_encryption_context(
            stubber,
            cmk_arn,
            plaintext,
            ciphertext,
            encryption_context,
        )
        encrypt_response = client.encrypt(
            Plaintext=plaintext,
            KeyId=cmk_arn,
            EncryptionContext=encryption_context,
        )

        decrypt_response = client.decrypt(
            CiphertextBlob=encrypt_response["CiphertextBlob"],
            EncryptionContext=encryption_context,
        )

complexity

Something important to note here is that because moto actually models a lot of the KMS behavior correctly, this will not only correctly model a successful call, but also correctly fail for an unsuccessful call: for example, if you specify a key ID that does not exist, supply the incorrect encryption context on decrypt, supply the wrong ciphertext, etc. It will also correctly handle cases where the key IDs do not match for valid reasons within the model of the service: for example, if you provide an alias name in the encrypt request or do not provide an encryption context in the encrypt request.

The important difference here is that with fakes like moto, the behavior needs to be modeled once, and everyone who uses the tool benefits from that work.
In comparison, with mocks, every user must deeply and correctly understand exactly what the expected request and response patterns are, and manually model every case. This is not only a lot more net work, but also results in much more brittle tests.

cross-language

Moto also does not just target the Python ecosystem: it can be run as a local server[1] and used to fake AWS services for any tests, regardless of their language.

[1] http://docs.getmoto.org/en/latest/docs/server_mode.html