Implement option to write to a custom buffer during S3 writes

[mpenkov]

Currently, we buffer in-memory. This works well in most cases, but can
be a problem when performing many uploads in parallel. Each upload
needs to buffer at least 5MB of data (that is the minimum part size for
AWS S3). When performing hundreds or thousands of uploads, memory can
become a bottleneck.

This PR relieves that bottleneck by buffering to temporary files
instead. Each part gets saved to a separate temporary file and discarded
after upload. This reduces memory usage at the cost of increased IO with
the local disk.

[piskvorky]

What's the motivation? Who needs thousands of parallel uploads and why?

Seems like a strange obfuscation so -1 on including, unless clearly motivated.

[menshikh-iv]

osx migration, again :)

[mpenkov]

Migration between home and office :)

[mpenkov]

What's the motivation? Who needs thousands of parallel uploads and why?

I'm partitioning a large (100GB, hundreds of millions of lines) dataset in preparation for some data processing. I need each partition to be relatively small, so I'm using a large number of partitions (hundreds/thousands). If I have all the partitions open for writing simultaneously, the partitioning is very simple:

for record in dataset:
    partition_index = calculate_partition(record)
    partitions[partition_index].write(record)

Seems like a strange obfuscation so -1 on including, unless clearly motivated.

I updated the implementation. It's not really an obfuscation anymore - the custom buffer is now an optional parameter. If people don't pass it in, the implementation behaves identically to the current version in PyPI.