Developer app - update option creates new credentials everytime we run

[ssvaidyanathan]

Two issues:

1. When we use update, the dev app does not get created
2. When ever update option on developer app runs, creates new credentials object. This will keep growing if update option is used for CI/CD. Would be nice if the plugin can delete and re-create the app

[bkarakashev]

👍

[msadasivam]

Apigee "developer" and "app" entities are app consumption operations and is different from API deployment entities like cache, targetserver etc.,

The API deployment is expected to happen often and API consumption in CI/CD pipeline is expected to happen less often.

For the purposes of automated testing we could create some "developer" and "app" entities and a way to do it is to split the plugin invocation into 2 different steps.

1. Perform all "API deployment" related tasks - cache, targetserver etc., Here you can you use "update" or "sync" option of the plugin
2. Perform all "API consumption" tasks - developer, app. Here you can use "create" so that you can only create an app if it does not exist.
3. Optionally you can use the exportAppKeys goal to export a pre-existing app and use it in automated testing.

[kurtkanaskie]

FYI, I just found out there is a limit of 100 credentials on an App.

[Carmot]

Hi @msadasivam
We are following the approach you mention of separating DevApp update from the other APIGEE elements because we don't want a new API Key to be created each time.

But when running the command only for Dev App and with 'create' option a new API Key is still created in top of the previous ones we have already created for this Dev App.

mvn apigee-config:apps -f'+filename+'/config_pom.xml -P'+profile+' -DAPI_ENVIRONMENT='+apienvironment+' -DAPI_ORGANIZATION='+apiorganization+' -DAPI_HOSTURL='+apihosturl+' -DUSERNAME='+apiuser+' -DPASSWORD='+apipasswd+' -Dapigee.config.options=create

Please can you help me with this issue?

TA

[msadasivam]

@Carmot With the option "create" I can see that the app is skipped when it exists. Taking the example in samples in this repo you command gave me the following.

[INFO] App "coolhughapp" already exists. Skipping.
[INFO] Retrieving Apps of [email protected]
[INFO]

[Carmot]

Thanks @msadasivam ,
What I've observed during our testing is that DevApp steps is not running, so it has to be product or developer update which is causing that a new APIKey is being created.
We are creating/updating Product and Developer because we are using attributes inside them.

Output log from our tool:

APIGEE cache(s) config successfull APIGEE KVM(s) config successfull APIGEE API Product(s) config successfull APIGEE Developer(s) config successfull APIKEY extraction successfull [go] Task status: passed, took: 19.285s [go] Current job status: passed

[Carmot]

I've just found this in the APIGEE docs:
https://apidocs.apigee.com/management/apis/put/organizations/%7Borg_name%7D/developers/%7Bdeveloper_email_or_id%7D/apps/%7Bapp_name%7D

You can also add an app to an API product with this call, which automatically generates an API key for the app to use when calling APIs in the product.

Looks like each time a product is updated a new api key is created.
This could be what's happening to us here.

[dpalanivel]

I think @Carmot has nailed it and found the underlying challenge. With the current implementation, it's really difficult to achieve what we need (see below) with minimal code changes.

• Ability to update the existing apps with OAuth scopes [Create, Update and Delete the OAuth scopes] (at least that was our use case when he hit the brakes)
• Ability to skip the expired credentials during the above update (Don't do anything and keep the audit record pristine). We understand that there is an upper limit of number of credentials that we can have per app.

We had to write a custom code and wire it up with our CI/CD process to achieve what we need from Developer "Apps" management perspective while we still use the plugin for other needs like API Products, KVMs, etc.

[msadasivam]

Agree, the presence of API Products in the "app" forces a new credential. To update attributes of a developer app without impacting credentials the "apiProducts" attribute should not be present.

A potential enhancement is to provide credential management goals in the plugin.

[dineshpalanivel]

@msadasivam @ssvaidyanathan Is there any plan to address this open issue? The more we invest in custom code the more tech debt we are accumulating and spending more time maintaining it. :)
Any help is much appreciated.

[aakashsharmaquilter]

Apigee "developer" and "app" entities are app consumption operations and is different from API deployment entities like cache, targetserver etc.,

The API deployment is expected to happen often and API consumption in CI/CD pipeline is expected to happen less often.

For the purposes of automated testing we could create some "developer" and "app" entities and a way to do it is to split the plugin invocation into 2 different steps.

1. Perform all "API deployment" related tasks - cache, targetserver etc., Here you can you use "update" or "sync" option of the plugin
2. Perform all "API consumption" tasks - developer, app. Here you can use "create" so that you can only create an app if it does not exist.
3. Optionally you can use the exportAppKeys goal to export a pre-existing app and use it in automated testing.

===

Is this key confllct issue resolved?

when we do run maven install with config.option=update it will pick both deployment (kvm cache etc.) and consumption entities (apps, products, keys etc.). This again leads us to key conflict error in apps below. How can we run "update" only for deployment entities such as kvm, caches etc excluding consumption entities . Running one by one with apigee-config: is one option but doing it for each entity/goal is not best way. Is there a way we can separate out consumption entities from API deployment that will happen often.

[dineshpalanivel]

TL;DR

• Developer apps is creating additional set of credentials automatically. We need to be able to suppress it.
• ImportKeys is not enabling a true update operation.

@aakashsharmaquilter I think the issue that you are highlighting is still out there. The behavior of the plugin is a little misleading as we will think that -Dapigee.config.options=update will simply update what is already available on Edge. But unfortunately, we don't see the behavior when the goal is importKeys .
In our case, we basically want to maintain the OAuth scopes as a part of credentials itself and sometimes it goes thru updates. Basically we need a true update operation.
I think at this point, the plugin is pretty much unusable for both managing developerApps and importKeys operations.

@msadasivam @ssvaidyanathan Your thoughts? I know I'm probably highlighting multiple issues here. Let me know if we need to create a new issue for importKeys issue.

[ssvaidyanathan]

@aakashsharmaquilter - for your issue, you could make use of the Maven lifecycle. You can have all the deployment entities in your pom with install as the phase and then for the consumption entities, you can have them as deploy. So when you just want to push the deployment entities you can run mvn install .... and when you want to push everything, you can run mvn clean deploy ....

[kurtkanaskie]

You could also use apigee.config.options=none which will skip the config items.
You can then check to see if anything changed in the "resources" directory (if you use that approach) in the latest commit using:

ConfigChanges=`git diff --name-only HEAD HEAD~1 | grep "resources"`
if [[ $? -eq 0 ]]
then
	export EdgeConfigOptions="update"
else
	export EdgeConfigOptions="none"
fi

Another alternative would be to delete the "test app" when you're done using:

mvn -P test -Ddeployment.suffix= -Dapigee.config.dir=target/resources/edge -Dapigee.config.options=delete resources:copy-resources@copy-resources replacer:replace  apigee-config:apps

That way, you'll only ever have one set of credentials.
Of course this assumes you have dedicated Apps just for testing during CI/CD

[ssvaidyanathan]

@dineshpalanivel - this is how the Mgmt API works, if you include the API Product while updating an app, it WILL create a new set of credentials. importKeys is meant for importing external keys to Apigee. Don't think your use case should use this.

When you say you want to update the scope, can you provide some info? I am assuming you are updating scopes in the Product. Are you updating the app as well in your pipeline?

[dineshpalanivel]

this is how the Mgmt API works, if you include the API Product while updating an app, it WILL create a new set of credentials. importKeys is meant for importing external keys to Apigee. Don't think your use case should use this.

@ssvaidyanathan If you have custom attributes and OAuth scopes at the app level and if there is a need that it goes thru changes, how do we get it accomplished? I agree with you that its the fundamental behavior of the underlying management API call to create new creds. But in our case, its byproduct that we don't want and I think that's the very purpose of this issue. :)

[ssvaidyanathan]

I understand that :)
My question was - are you using the OAuth scopes in the app as custom attributes? Usually scopes are at the product level. What is a typical update you perform to an app - I assume its just the custom attributes?

I don't think updating a product will have any impact to an existing app that uses that product

[dineshpalanivel]

I understand that :)
My question was - are you using the OAuth scopes in the app as custom attributes? Usually scopes are at the product level. What is a typical update you perform to an app - I assume its just the custom attributes?

I don't think updating a product will have any impact to an existing app that uses that product

@ssvaidyanathan OAuth scopes could be at the product level and also at the app level. App level scopes override product level scopes.
We use app level scopes to achieve "selective" access to partners but the API product being the same. So, in our case, we will have to go thru the "update" scenario with no new set creds getting created. That's where the pain point is. :)
I hope it clarifies.

Edit: We DO NOT configure OAuth scopes as custom attributes. We manage the scopes as a part of scopes object in the json.

Example:
"scopes": ["urn:company-name:resource:read", "urn:company-name:resource:update"]

[ssvaidyanathan]

Got it - so you are using the Mgmt API to update the app scopes (which is not shown in the UI).

The only option I can think is to come up with a feature that ignores the apiProduct config in the app so that it updates everything else passed in the payload and NOT generate a new credential.

[dineshpalanivel]

Got it - so you are using the Mgmt API to update the app scopes (which is not shown in the UI).

The only option I can think is to come up with a feature that ignores the apiProduct config in the app so that it updates everything else passed in the payload and NOT generate a new credential.

Correct. We use the management API to update the app scopes.
I agree. It would be awesome to manage the apps and the credentials associated nicely as a part of the plugin. We don't want to manage a custom code on our side which is painful.

[ssvaidyanathan]

Ok.. Will try and add that feature
For credentials - what else is needed

[dineshpalanivel]

Ok.. Will try and add that feature
For credentials - what else is needed

For credentials, as long as we honor the JVM argument -Dapigee.config.options, that would be great too. Coz, we have pipelines syncing up creds across two different API Gateway products and this feature would be awesome.
Currently, importKeys just errors out if the API key is already on Edge.

[aakashsharmaquilter]

Ok.. Will try and add that feature
For credentials - what else is needed

For credentials, as long as we honor the JVM argument -Dapigee.config.options, that would be great too. Coz, we have pipelines syncing up creds across two different API Gateway products and this feature would be awesome.
Currently, importKeys just errors out if the API key is already on Edge.

thanks @ssvaidyanathan

Yes corrrect as @dineshpalanivel mentioned if it just continues without error in case key exist and for new keys just add them. Similar to kvms.

[aakashsharmaquilter]

for your issue, you could make use of the Maven lifecycle. You can have all the deployment entities in your pom with install as the phase and then for the consumption entities, you can have them as deploy

thanks @kurtkanaskie @ssvaidyanathan will try that.

[ssvaidyanathan]

@dineshpalanivel - I plan to just add a logic to the app goal to ignore the product (by passing an argument like -DignoreProductsForApp=true) so that it updates the App attributes (without creating a new credential)

[ssvaidyanathan]

Fixed in v1.4.5 - README section for this

[aakashsharmaquilter]

@ssvaidyanathan - For 1.4.5 it doesn't address the import key issue if key already exists. As new apps are added on platform we need some way to import keys through ci/cd. When we add new keys it imports them, but just errors out for old apps wherever key matches. This is with reference to update and "all" option.

[ssvaidyanathan]

Please open a new issue for this. The initial issue opened was for preventing new credentials being created when app was being updated.

[aakashsharmaquilter]

Please open a new issue for this. The initial issue opened was for preventing new credentials being created when app was being updated.

thanks I have added #123

[dineshpalanivel]

Thank you @ssvaidyanathan for the fix. Awesome. Now over to issue # # #123 :)

[dfsabugal]

@ssvaidyanathan why the fix is not available for Apigee X (v2.x.x) ? We are using Apigee X and facing this issue.

[ssvaidyanathan]

Thanks for creating #128 @dfsabugal
Will add this feature

[dfsabugal]

Thanks for creating #128 @dfsabugal
Will add this feature

Thank you @ssvaidyanathan.