Support AWS pagination when fetching resources

[fxaguessy]

If there are too many resources, they may be sent by bunches, which is currently not managed in awless.

[simcap]

Basically support the aws-sdk-go NextToken used to retrieve the next page of results.

[ajkerrigan]

I'd like to take a crack at this if you don't mind. I'd probably start with ListUsers since (selfishly) it is currently most relevant to my use of this tool. Any objections? Also, any aversion to using the SDK-provided paginators (ListUsersPages in this case) where available instead of explicitly handling the NextToken?

[simcap]

@ajkerrigan-mdsol Actually in this case you might take a crack at it since this fetcher is not automatically generated Go code.

We have a convention where files containing generated Go code logic have the gen_ suffix (ex: awless/cloud/aws/gen_api.go)

Your modification would then be in awless/cloud/aws/api.go. The method fetch_all_user_graph is manually implemented since we have some custom logic around fetching users.

Using ListUserPages make sense for this one. You will be able to test if it works properly with you live data.

Out of curiosity, it seems to be a large number of users you have to require some pagination?

[ajkerrigan]

Thanks for the information, I'll check it out. The fetch_all_user_graph method is exactly where I was looking, so I'm glad to have confirmation.

And yes, plenty of users. I'm wondering if implementing a user-definable limit (via a maxresults or maxresults.<resource type> config setting perhaps) should accompany this change, since supporting pagination without limits can have unfortunate side effects (increased API requests with the risk of throttling, slower performance, unmanageable result sets, etc). One step at a time though!