Don't log AWS Secrets when running COPY commands

[facultywork]

Expected Behaviour

When I write a pandas dataframe to redshift like this

# Connect to S3
pr.connect_to_s3(aws_access_key_id = <aws_access_key_id>,
                aws_secret_access_key = <aws_secret_access_key>,
                bucket = <bucket>,
                subdirectory = <subdirectory>
                # As of release 1.1.1 you are able to specify an aws_session_token (if necessary):
                # aws_session_token = <aws_session_token>
                )

# Write the DataFrame to S3 and then to redshift
pr.pandas_to_redshift(data_frame = data,
                        redshift_table_name = 'gawronski.nba_shots_log')

I would expect not to see the <aws_access_key_id> and the <aws_secret_access_key> in the logs.

Actual Behaviour

The issues is that the AWS secrets get's logged with the code of the COPY command.

saved file 8c22-affc4c880fe3.csv in bucket some-path/8c22-affc4c880fe3.csv

    copy sometable
    from 's3://some-bucket/some-path/8c22-affc4c880fe3.csv'
    delimiter ','
    ignoreheader 1
    csv quote as '"'
    dateformat 'auto'
    timeformat 'auto'
    
        access_key_id 'ACCESS_KEY'
        secret_access_key 'SECRET_KEY'
    
    ;
FILLING THE TABLE IN REDSHIFT

I would expect to be able to somehow mute this log so that my secrets won't get printed to the console.

[agawronski]

Yeah this definitely should be the way that it is done. There is an open pull request for less verbose loggin #34 that has conflicts at this point. If I can find time next weekend i'll make the change.

[agawronski]

@facultywork you can install the latest release -> pip3 install pandas-redshift —upgrade it's up on PyPI now.