Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes to Coco Data Quality Sample #8337

Merged
merged 2 commits into from
Aug 28, 2024
Merged

Fixes to Coco Data Quality Sample #8337

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ae6c84a
Fixes to Coco Data Quality Sample
mandy-chessell Aug 28, 2024
590ef06
Merge branch 'main' into oak2024
mandy-chessell Aug 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CocoComboGUIDMap.json
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion CoreContentPackGUIDMap.json
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion content-packs/CocoBusinessSystemsArchive.omarchive
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion content-packs/CocoClinicalTrialsTemplatesArchive.omarchive
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion content-packs/CocoComboArchive.omarchive
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion content-packs/CocoGovernanceEngineDefinitionsArchive.omarchive
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion content-packs/CocoGovernanceProgramArchive.omarchive
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion content-packs/CocoOrganizationArchive.omarchive
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion content-packs/CocoSustainabilityArchive.omarchive
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion content-packs/CocoTypesArchive.omarchive
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion content-packs/CoreContentPack.omarchive
View file
Open in desktop

Large diffs are not rendered by default.

22 changes: 16 additions & 6 deletions .../org/odpi/openmetadata/accessservices/assetconsumer/server/AssetConsumerRESTServices.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
/* Copyright Contributors to the ODPi Egeria project. */
package org.odpi.openmetadata.accessservices.assetconsumer.server;

import org.odpi.openmetadata.frameworks.connectors.ffdc.InvalidParameterException;
import org.odpi.openmetadata.frameworks.openmetadata.metadataelements.*;
import org.odpi.openmetadata.accessservices.assetconsumer.handlers.LoggingHandler;
import org.odpi.openmetadata.frameworks.openmetadata.metadataelements.AssetGraph;
Expand Down Expand Up @@ -509,15 +510,24 @@ public AssetSearchMatchesListResponse findAssetsInDomain(String

for (String assetGUID : organizedEntities.keySet())
{
AssetElement asset = assetHandler.getBeanFromRepository(userId,
assetGUID,
parameterName,
OpenMetadataType.ASSET.typeName,
methodName);
AssetElement asset;

try
{
asset = assetHandler.getBeanFromRepository(userId,
assetGUID,
parameterName,
OpenMetadataType.ASSET.typeName,
methodName);
}
catch (InvalidParameterException notVisible)
{
asset = null;
}

if (asset != null)
{
AssetSearchMatches assetSearchMatches = new AssetSearchMatches(asset);
AssetSearchMatches assetSearchMatches = new AssetSearchMatches(asset);
Map<String, EntityDetail> assetEntityMap = organizedEntities.get(assetGUID);
List<MetadataElementSummary> anchoredElements = new ArrayList<>();

Expand Down
4 changes: 3 additions & 1 deletion ...ager-connectors/unity-catalog-connectors/UnityCatalog-demo-onboarding-clinical-trial.http
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ Content-Type: application/json
DELETE {{ucURL}}/api/2.1/unity-catalog/schemas/clinical_trials.teddy_bear_drop_foot
Content-Type: application/json

###
# Remember to delete files in data lake
###
# =====================================================================================================================
# These commands add the CocoComboArchive to active-metadata-store and a new governance engine for clinical trials
Expand Down Expand Up @@ -383,7 +385,7 @@ Content-Type: application/json
{
"class" : "NewActionTarget",
"actionTargetName": "dataLakeCatalog",
"actionTargetGUID": "c861d5e8-1cda-4063-9c89-d79e0e505d05"
"actionTargetGUID": "7dc0f1ad-c831-4e53-8d2b-dd503042e259"
}],
"requestParameters" : {
"dataLakeSchemaName" : "teddy_bear_drop_foot",
Expand Down
2 changes: 1 addition & 1 deletion ...c/main/java/org/odpi/openmetadata/adapters/connectors/datastore/csvfile/CSVFileStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,5 +50,5 @@ public interface CSVFileStore extends BasicFileStore
* @throws FileException there is a problem accessing the file
* @throws FileReadException unable to find, open or read the file, or the file does not include the requested record.
*/
List<String> readRecord(int rowNumber) throws FileException, FileReadException;
List<String> readRecord(long rowNumber) throws FileException, FileReadException;
}
8 changes: 5 additions & 3 deletions ...va/org/odpi/openmetadata/adapters/connectors/datastore/csvfile/CSVFileStoreConnector.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@
* @throws FileException problem accessing the file
* @throws FileReadException unable to find, open or scan the file.
*/
@Override
public long getRecordCount() throws FileException, FileReadException
{
final String methodName = "getRecordCount";
Expand Down Expand Up @@ -149,6 +150,7 @@
* @throws FileException problem accessing the file
* @throws FileReadException unable to retrieve the column names
*/
@Override
public List<String> getColumnNames() throws FileException,
FileReadException
{
Expand Down Expand Up @@ -176,7 +178,7 @@
* @throws FileException problem accessing the file
* @throws FileReadException unable to find, open or read the file, or the file does not include the requested record.
*/
public List<String> readRecord(int dataRecordNumber) throws FileException, FileReadException
public List<String> readRecord(long dataRecordNumber) throws FileException, FileReadException

Check notice

Code scanning / CodeQL

Missing Override annotation Note

This method overrides
CSVFileStore.readRecord
Loading
; it is advisable to add an Override annotation.
{
final String methodName = "readRecord";

Expand All @@ -200,7 +202,7 @@
* @throws FileException problem accessing the file
* @throws FileReadException unable to find, open or read the file, or the file does not include the requested record.
*/
private List<String> readRow(int recordLocation,
private List<String> readRow(long recordLocation,
String methodName) throws FileException, FileReadException
{

Expand All @@ -224,7 +226,7 @@
}

throw new FileReadException(CSVFileConnectorErrorCode.FILE_TOO_SHORT.getMessageDefinition(fileStoreName,
Integer.toString(recordLocation)),
Long.toString(recordLocation)),
this.getClass().getName(),
methodName,
fileStoreName);
Expand Down
4 changes: 2 additions & 2 deletions ...rg/odpi/openmetadata/adapters/connectors/datastore/csvfile/CSVFileStoreConnectorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,15 +95,15 @@ private void scanFile(CSVFileStoreConnector connector) throws FileException,
assertTrue(columnNames != null);
assertFalse(columnNames.isEmpty());

for (int i=0; i<connector.getRecordCount(); i++)
for (long i=0; i<connector.getRecordCount(); i++)
{
List<String> columns = connector.readRecord(i);
assertTrue(columnNames.size() == columns.size());
}

try
{
connector.readRecord(10000);
connector.readRecord(10000L);
assertTrue(false);

}
Expand Down
2 changes: 1 addition & 1 deletion ...va/org/odpi/openmetadata/adapters/connectors/surveyaction/surveycsv/CSVSurveyService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,7 @@ public void start() throws ConnectorCheckedException
}
}

for (int recordNumber=0; recordNumber < recordCount ; recordNumber++)
for (long recordNumber=0; recordNumber < recordCount ; recordNumber++)
{
List<String> recordValues = assetConnector.readRecord(recordNumber);

Expand Down
4 changes: 2 additions & 2 deletions ...ters/connectors/governanceactions/stewardship/CatalogServerGovernanceActionConnector.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ public CatalogServerGovernanceActionConnector()
* @throws ConnectorCheckedException there is a problem within the governance action service.
*/
@Override
public void start() throws ConnectorCheckedException
public synchronized void start() throws ConnectorCheckedException
{
final String methodName = "start";

Expand Down Expand Up @@ -187,7 +187,7 @@ Map<String, Object> combineProperties()
configurationProperties.putAll(governanceContext.getRequestParameters());
}

if (configurationProperties.isEmpty())
if ((configurationProperties == null) || (configurationProperties.isEmpty()))
{
return null;
}
Expand Down
2 changes: 1 addition & 1 deletion ...adata/adapters/connectors/integration/csvlineageimporter/CSVLineageImporterConnector.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ private void processFile() throws ConnectorCheckedException

long numberOfRecords = csvFileStoreConnector.getRecordCount();

for (int i = 0; i < numberOfRecords; i++)
for (long i = 0; i < numberOfRecords; i++)
{
List<String> columns = csvFileStoreConnector.readRecord(i);

Expand Down
1 change: 1 addition & 0 deletions ...etadata/adapters/connectors/integration/openlineage/OpenLineageLogStoreConnectorBase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,7 @@ public synchronized void start() throws ConnectorCheckedException
*
* @throws ConnectorCheckedException there is a problem with the connector. It is not able to refresh the catalog targets.
*/
@Override
public synchronized void refresh() throws ConnectorCheckedException
{
final String methodName = "refresh";
Expand Down
3 changes: 2 additions & 1 deletion .../connectors/egeriainfrastructure/platform/catalog/OMAGServerPlatformCatalogConnector.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ public String toString()
* @throws ConnectorCheckedException there is a problem within the connector.
*/
@Override
public void start() throws ConnectorCheckedException
public synchronized void start() throws ConnectorCheckedException
{
super.start();

Expand Down Expand Up @@ -190,6 +190,7 @@ public void refresh() throws ConnectorCheckedException
/**
* Called each time an event that is published by the IT Infrastructure OMAS, it is looking for Software Server Platforms to add to monitoredPlatforms.
*/
@Override
public void processEvent(ITInfrastructureOutTopicEvent event)
{
if ((event.getElementProperties() != null) &&
Expand Down
72 changes: 50 additions & 22 deletions ...org/odpi/openmetadata/engineservices/surveyaction/handlers/SurveyActionEngineHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,30 +3,37 @@
package org.odpi.openmetadata.engineservices.surveyaction.handlers;

import org.odpi.openmetadata.accessservices.assetowner.client.CSVFileAssetOwner;
import org.odpi.openmetadata.accessservices.assetowner.client.ConnectedAssetClient;
import org.odpi.openmetadata.accessservices.assetowner.client.FileSystemAssetOwner;
import org.odpi.openmetadata.accessservices.assetowner.client.SurveyAssetStoreClient;
import org.odpi.openmetadata.accessservices.governanceserver.client.GovernanceConfigurationClient;
import org.odpi.openmetadata.accessservices.governanceserver.client.GovernanceContextClient;
import org.odpi.openmetadata.accessservices.assetowner.client.ConnectedAssetClient;
import org.odpi.openmetadata.accessservices.assetowner.client.SurveyAssetStoreClient;
import org.odpi.openmetadata.adminservices.configuration.properties.EngineConfig;
import org.odpi.openmetadata.adminservices.configuration.registration.EngineServiceDescription;
import org.odpi.openmetadata.engineservices.surveyaction.ffdc.SurveyActionAuditCode;
import org.odpi.openmetadata.engineservices.surveyaction.ffdc.SurveyActionErrorCode;
import org.odpi.openmetadata.frameworks.auditlog.AuditLog;
import org.odpi.openmetadata.frameworks.connectors.ffdc.*;
import org.odpi.openmetadata.frameworks.connectors.ffdc.InvalidParameterException;
import org.odpi.openmetadata.frameworks.connectors.ffdc.PropertyServerException;
import org.odpi.openmetadata.frameworks.connectors.ffdc.UserNotAuthorizedException;
import org.odpi.openmetadata.frameworks.governanceaction.client.OpenMetadataClient;
import org.odpi.openmetadata.frameworks.openmetadata.types.OpenMetadataType;
import org.odpi.openmetadata.frameworks.governanceaction.controls.ActionTarget;
import org.odpi.openmetadata.frameworks.governanceaction.properties.ActionTargetElement;
import org.odpi.openmetadata.frameworks.openmetadata.enums.EngineActionStatus;
import org.odpi.openmetadata.frameworks.governanceaction.properties.RequestSourceElement;
import org.odpi.openmetadata.frameworks.governanceaction.search.PropertyHelper;
import org.odpi.openmetadata.frameworks.openmetadata.enums.EngineActionStatus;
import org.odpi.openmetadata.frameworks.openmetadata.types.OpenMetadataType;
import org.odpi.openmetadata.frameworks.surveyaction.AnnotationStore;
import org.odpi.openmetadata.frameworks.surveyaction.SurveyAssetStore;
import org.odpi.openmetadata.frameworks.surveyaction.SurveyContext;
import org.odpi.openmetadata.frameworks.surveyaction.SurveyOpenMetadataStore;
import org.odpi.openmetadata.governanceservers.enginehostservices.admin.GovernanceEngineHandler;
import org.odpi.openmetadata.governanceservers.enginehostservices.admin.GovernanceServiceCache;

import java.util.*;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;

/**
* The SurveyActionEngineHandler is responsible for running survey action services on demand. It is initialized
Expand All @@ -40,6 +47,8 @@ public class SurveyActionEngineHandler extends GovernanceEngineHandler
private final CSVFileAssetOwner csvFileAssetOwner; /* Initialized in constructor */
private final OpenMetadataClient openMetadataClient; /* Initialized in constructor */

private final PropertyHelper propertyHelper = new PropertyHelper();

/**
* Create a client-side object for calling a survey action engine.
*
Expand Down Expand Up @@ -195,17 +204,34 @@ private String getAssetGUIDFromActionTargets(List<ActionTargetElement> actionTar
String assetGUID = null;
List<String> ignoredAssets = new ArrayList<>();

/*
* First pick out all the assets ...
*/
List<ActionTargetElement> assetTargetElements = new ArrayList<>();

for (ActionTargetElement actionTargetElement : actionTargetElements)
{
if ((actionTargetElement != null)
&& (actionTargetElement.getTargetElement() != null)
&& (actionTargetElement.getTargetElement().getType() != null))
if (actionTargetElement != null)
{
String typeName = actionTargetElement.getTargetElement().getType().getTypeName();
List<String> superTypeNames = actionTargetElement.getTargetElement().getType().getSuperTypeNames();
if (propertyHelper.isTypeOf(actionTargetElement.getTargetElement(), OpenMetadataType.ASSET.typeName))
{
assetTargetElements.add(actionTargetElement);
}
}
}

if ((OpenMetadataType.ASSET.typeName.equals(typeName)) ||
((superTypeNames != null) && (superTypeNames.contains(OpenMetadataType.ASSET.typeName))))
if (assetTargetElements.size() == 1)
{
assetGUID = assetTargetElements.get(0).getTargetElement().getElementGUID();
}
else
{
/*
* Since there are multiple assets, only pick out the ones with an action target name of "newAsset".
*/
for (ActionTargetElement actionTargetElement : assetTargetElements)
{
if (ActionTarget.NEW_ASSET.getName().equals(actionTargetElement.getActionTargetName()))
{
if (assetGUID == null)
{
Expand All @@ -223,18 +249,20 @@ private String getAssetGUIDFromActionTargets(List<ActionTargetElement> actionTar
}
}
}
}

if (! ignoredAssets.isEmpty())
{
auditLog.logMessage(methodName,
SurveyActionAuditCode.IGNORING_ASSETS.getMessageDefinition(governanceServiceName,
governanceRequestType,
engineActionGUID,
assetGUID,
ignoredAssets.toString()));
if (! ignoredAssets.isEmpty())
{
auditLog.logMessage(methodName,
SurveyActionAuditCode.IGNORING_ASSETS.getMessageDefinition(governanceServiceName,
governanceRequestType,
engineActionGUID,
assetGUID,
ignoredAssets.toString()));
}
}



return assetGUID;
}

Expand Down
6 changes: 5 additions & 1 deletion ...framework/src/main/java/org/odpi/openmetadata/frameworks/openmetadata/enums/DataType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,14 +126,15 @@ public enum DataType implements OpenMetadataEnum
*
* @return int ordinal
*/
@Override
public int getOrdinal() { return ordinal; }


/**
* Return the default name of the enumeration.
*
* @return String name
*/
*/@Override
public String getName() { return name; }


Expand All @@ -142,6 +143,7 @@ public enum DataType implements OpenMetadataEnum
*
* @return String description
*/
@Override
public String getDescription() { return description; }


Expand All @@ -150,6 +152,7 @@ public enum DataType implements OpenMetadataEnum
*
* @return guid
*/
@Override
public String getDescriptionGUID()
{
return descriptionGUID;
Expand All @@ -161,6 +164,7 @@ public String getDescriptionGUID()
*
* @return boolean
*/
@Override
public boolean isDefault()
{
return isDefault;
Expand Down
2 changes: 2 additions & 0 deletions ...ramework/src/main/java/org/odpi/openmetadata/frameworks/surveyaction/AnnotationStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -750,6 +750,8 @@ public String addAnnotation(Annotation annotation,
null,
this.getEffectiveTime());
}

return annotationGUID;
}
}

Expand Down
1 change: 0 additions & 1 deletion ...-framework/src/main/java/org/odpi/openmetadata/frameworks/surveyaction/SurveyContext.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -346,7 +346,6 @@ public String toString()
", openMetadataStore=" + openMetadataStore +
", surveyActionServiceName='" + surveyActionServiceName + '\'' +
", requesterUserId='" + requesterUserId + '\'' +
", auditLog=" + auditLog +
", fileClassifier=" + fileClassifier +
", isActive=" + isActive +
", completionStatus=" + completionStatus +
Expand Down
2 changes: 1 addition & 1 deletion ...i/openmetadata/governanceservers/enginehostservices/ffdc/EngineHostServicesAuditCode.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -476,7 +476,7 @@ public enum EngineHostServicesAuditCode implements AuditLogMessageSet
* ENGINE-HOST-SERVICES-2000 - {0} caught an exception {1} while processing governance action {2}; the error message was {3}
*/
ACTION_PROCESSING_ERROR( "ENGINE-HOST-SERVICES-2000",
AuditLogRecordSeverityLevel.ERROR,
AuditLogRecordSeverityLevel.EXCEPTION,
"{0} caught an exception {1} while processing engine action {2}; the error message was {3}",
"The server is not able to start or complete the requested processing related to the governance service for this engine action.",
"Follow the instructions for the message associated with the exception."),
Expand Down
9 changes: 5 additions & 4 deletions ...overnanceservers/integrationdaemonservices/threads/IntegrationConnectorRefreshThread.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,10 +111,11 @@ else if (minMinutedBetweenRefresh > 0)
}
catch (Exception error)
{
auditLog.logMessage(actionDescription,
IntegrationDaemonServicesAuditCode.REFRESH_THREAD_CONNECTOR_ERROR.getMessageDefinition(connectorHandler.getIntegrationConnectorName(),
error.getClass().getName(),
error.getMessage()));
auditLog.logException(actionDescription,
IntegrationDaemonServicesAuditCode.REFRESH_THREAD_CONNECTOR_ERROR.getMessageDefinition(connectorHandler.getIntegrationConnectorName(),
error.getClass().getName(),
error.getMessage()),
error);
}
}

Expand Down
Loading
Loading