Syncing sources

[AlexClineBB]

Hello,

I got Lemur up and running. I'm able to import certificates just fine.

I'm trying to understand how the 'sources' feature works. My understanding is that using AWS as a source will automatically populate Lemur with a list of certificate information for certs that already exist in AWS after running lemur sync_sources. Sadly, running this command doesn't seem to do anything.

Here's the output from running the command, which exits successfully:

# lemur sync_sources           
/usr/local/src/lemur/local/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py:800: UserWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True to suppress this warning.
  warnings.warn('SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True to suppress this warning.')
Active  Label   Description
True    Branding Brand AWS  IAM certificates for Branding Brand!

There are no error messages in the logs that pertain to running the sync. The certificates are all under the same account, but for different domains.

Is there some documentation I'm missing on how to get our existing certs from AWS into Lemur?

Thanks.

[kevgliss]

Hmm, that looks like a bit of a gap in the documentation. Essentially, you want to create a new 'aws source'. They way you do that through the UI is to select sources->create select the aws plugin and type in the necessary accountId. If you have multiple accounts you would create an individual source for each.

From there you need to ensure that you follow the documentation located at:
http://lemur.readthedocs.org/en/latest/administration.html#aws-source-destination-plugin

Then running sync_sources should work as you might expect. Note that fetching certificates from AWS will not include their private keys and Lemur won't be able to move them between accounts.

[AlexClineBB]

Right. I already have the source created in Lemur. It has the same account information as our AWS destination and I'm able to upload certificates to AWS using Lemur, which I assume means my IAM configuration is correct.

Do the domains listed in Lemur have to match the certificates being synchronized from AWS? Do the certificates in AWS have to be within a certain path in IAM or have a name format to be captured?

We have over 100 certificates in AWS that I'd like to get into Lemur. Thanks.

[kevgliss]

Ahh thats right, calling sync_sources without any flags simply lists the available sources to sync, can you try to specify the flag -s' as 'Branding' or 'all'?

We should probably make that more clear in the documentation.

[AlexClineBB]

Success! Running lemur sync_sources -s all was the trick to actually do something.

I did run into an issue when running that.

# lemur sync_sources -s all
/usr/local/src/lemur/local/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py:800: UserWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True to suppress this warning.
  warnings.warn('SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True to suppress this warning.')
[!] Unable to acquire file lock on /tmp/.lemur_lock, is there another sync running?
[+] Staring to sync sources: all!
Traceback (most recent call last):
  File "/usr/local/src/lemur/bin/lemur", line 9, in <module>
    load_entry_point('lemur', 'console_scripts', 'lemur')()
  File "/usr/local/src/lemur/lemur/manage.py", line 775, in main
    manager.run()
  File "/usr/local/src/lemur/local/lib/python2.7/site-packages/flask_script/__init__.py", line 412, in run
    result = self.handle(sys.argv[0], sys.argv[1:])
  File "/usr/local/src/lemur/local/lib/python2.7/site-packages/flask_script/__init__.py", line 383, in handle
    res = handle(*args, **config)
  File "/usr/local/src/lemur/local/lib/python2.7/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "/usr/local/src/lemur/lemur/manage.py", line 216, in sync_sources
    sync()
  File "/usr/local/src/lemur/lemur/sources/service.py", line 87, in sync
    sync_create(certificate, source)
  File "/usr/local/src/lemur/lemur/sources/service.py", line 41, in sync_create
    cert = cert_service.import_certificate(**certificate)
  File "/usr/local/src/lemur/lemur/certificates/service.py", line 192, in import_certificate
    database.update_list(cert, 'replaces', Certificate, kwargs['replacements'])
KeyError: 'replacements'

To workaround it, I commented out line 192 in lemur/certificates/service.py since none of my certificates are replacements. After running again:

lemur sync_sources -s all       
/usr/local/src/lemur/local/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py:800: UserWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True to suppress this warning.
  warnings.warn('SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True to suppress this warning.')
[!] Unable to acquire file lock on /tmp/.lemur_lock, is there another sync running?
[+] Staring to sync sources: all!
[+] Finished syncing sources. Run Time: 46.2676289082

[kevgliss]

Excellent, I will take a look at your stack trace when I get a chance, looks like we are just missing some parameters somewhere in the management script.