Salesforce - Improved Reliability

[dlstadther]

Made use of python TempFile rather than keeping (potentially) large lists of lists of OrderedDicts in memory.

Moved parse_results() and _parse_results() to be global functions. Call these functions on each page of salesforce results within query_all().

Regex now matches case insensitive when extracting field names from soql query.

Recursive get_all_responses() inner function has been replaced by the iterative equivalent (deep recursion was causing a segmentation fault).

Usage looks like the following:

import luigi
from luigi import configuration
from luigi.contrib import salesforce

config = configuration.get_config()

class Main(luigi.WrapperTask):
   def requires(self):
       yield SalesforceQuery(
           object_name='Account',
           output_path='/path/to/sf/query.csv',
           soql='select id from Account',
           is_soql_file=False
       )

class SalesforceQuery(salesforce.QuerySalesforce):
    object_name = luigi.Parameter()
    use_sandbox = luigi.BoolParameter(default=False)
    sandbox_name = luigi.Parameter(default=None)
    output_path = luigi.Parameter()
    soql = luigi.Parameter()
    is_soql_file = luigi.BoolParameter()

    def output(self):
        return luigi.LocalTarget(self.output_path)

if __name__ == "__main__":
    luigi.run(['Main'])

[dlstadther]

@erikbern No questions/comments/etc?

[erikbern]

I trust that you know what you are doing :)

[erikbern]

Changes to stuff in contrib isn't as critical. I doubt the salesforce integration is something a lot of people use