General solution for paginating AWS endpoints

[simonw]

I need a general solution for pagination. Maybe something like this:

for policy_name in paginate(iam, "list_role_policies", "PolicyNames", RoleName=role_name):
    # ...

The paginate() function would wrap this pattern, seen elsewhere:

paginator = iam.get_paginator("list_user_policies")
for response in paginator.paginate(UserName=username):
    for policy_name in response["PolicyNames"]:
        # ...

Originally posted by @simonw in #62 (comment)

[simonw]

I think this is it:

def paginate(service, method, list_key, **kwargs):
    paginator = service.get_paginator(method)
    for response in paginator.paginate(**kwargs):
        yield from response[list_key]

[simonw]

Which simplifies list-users to just:

@cli.command()
@common_output_options
@common_boto3_options
def list_users(nl, csv, tsv, **boto_options):
    "List all users"
    iam = make_client("iam", **boto_options)
    output(
        paginate(iam, "list_users", "Users"),
        (
            "UserName",
            "UserId",
            "Arn",
            "Path",
            "CreateDate",
            "PasswordLastUsed",
            "PermissionsBoundary",
            "Tags",
        ),
        nl,
        csv,
        tsv,
    )

[simonw]

Wrote this up as a TIL: https://til.simonwillison.net/aws/helper-for-boto-aws-pagination