Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add implementation for extraction of env and logs from Docker containers #2

Merged
merged 10 commits into from
Aug 27, 2020

Conversation

sigalsax
Copy link
Owner

@sigalsax sigalsax commented May 5, 2020

This PR adds functionality for extracting env variables and logs from docker containers using the docker-api client.

- Used the docker-api client as a wrapper for accessing docker containers
- Added functionality to extract env and logs
Copy link

@Tovli Tovli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add unit tests project
Refactor the code to be SOLID (mainly the S)

Understand what happen when you do SpringApplication.run(

public class AuthenticationTroubleshootingApplication {

public static void main(String[] args) {
String containerID = "8e53411614f2";
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please read from configurations (note that you usually want to work differently locally and in production)

@SpringBootApplication
public class AuthenticationTroubleshootingApplication {

public static void main(String[] args) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class should contain only the logic that we need to run the application. not business logic
You might put here - general spring boot configurations (DI/logging etc.)

- Add properties file in order to remove hardcoded configurations
- Add env and log models
- Extract logic to separate Container Session file to interface with Docker
@sigalsax sigalsax force-pushed the implementation-docker-client branch from fef5354 to 998292e Compare May 11, 2020 06:32
@@ -0,0 +1,28 @@
package com.troubleshooting.model;

// Data object for a Conjur log
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Class documentation looks as follows:
/**

  • Foo bar
    */
    Java SDK contains a documentation generator that outputs HTML

private String logLevel;
private String message;

public String getLogLevel() {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO any public method should be documented.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you suggest it they be private? and if so why?

Copy link

@eranha eranha May 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"If you like it than you should have put a ring on it"
If it's public it should be documented.


// Associate an environment variable and its value via key-value pairs
public class EnvModel implements IEnvModel{
private String key="";
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MIssing spaces.
IS there a reason for the initial value to be an empty string?

@@ -0,0 +1,6 @@
package com.troubleshooting.model;

public interface IEnvModel {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Starting an interface name with a capital I is a C# convention.
an interface should have only methods.

@@ -0,0 +1,11 @@
package com.troubleshooting.model;

// Log interface for Conjur and DAP logs
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Starting an interface name with a capital I is a C# convention.
an interface should have only methods.


// Retrieves the data returned from the Docker Client and packs the data in the appropriate data models
@Component
public class ContainerDataRetrieval implements IDataRetrieval {
Copy link

@eranha eranha May 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a rule of thumb if you don't plan for this class to be extended, declare it final, if they get injected create a setter for them.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a rule of thumb for all classes?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take into consideration that if it's not final it can be extended.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is extended a class an issue?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if write a code that depends on that class, someone may provide a behaviour that do not expect. If you plan to let. others extend the class that is OK.

// Extract Logs
@Override
public LogsModel getData(String containerID) {
List<String> loggingFrames = access.getLogs(containerID);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are certain that access.getLogs wii not return null and won't trow an exception?

Copy link
Owner Author

@sigalsax sigalsax May 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a container exists then there should be at startup logs. In terms of what to return, would I rethrow the null pointer

try {
}  catch (NullPointerException e) {
  throw e
}

or an empty object?

try {
}  catch (NullPointerException e) {
      return new LogsModel();
}

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to check for null rather than try and fall.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check for null or throw null?

Copy link

@eranha eranha May 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends on the logic. In java there are two types of exceptions:

  1. RuntimeException (means that something unexpected had happened that the user cannot be recovered)
  2. CheckedException (you must try or catch or declare that the method throws)
    Usually one don't throw a RuntimeExcpetion and also not Java API exception but creates a BusinessExecption
class UserCreateException extends Exception {}
pubic void registerUser() throws UserCreateException {
 try {
    registerUser();
  } catch (Throwable t) {
   // Throwable is super class of runtime and checked exceptions
    throw new UserCreateException("Failed to create user", t);
  }
}

return logs;
}

// Extracts important parts of each Log entry
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be better off calling this method:
extractsImportantPartsOfEachLogEntry than a name which is not a verb (Methods should be verbs) and a single line comment above the method. Remember the best comment you can write is the one you didn't write.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very good call

envsString = access.getEnv(containerID);
envParser(envsString, env);
} catch (Exception e) {
e.printStackTrace();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is where this error ends, is this the desired error handling?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just a simple

     } catch (Exception e) {
            System.out.println("Unable to get environment variables from container...");
        }

?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add error info goes without saying. But I was referring to error handling strategy in general. this code means the error ends here, if you're aware of it and it's OK, then it's OK.

return envs;
}

// Extracts important parts of each Env entry
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the comment and name the method:extractsImportantPartsOfEachEnvironmentEntry

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about formatAndPopulateEnvCollection since that is what it is effectively doing? Or does that go against the S in SOLID?

What about for the logs method for extracting and populating the log model. I was thinking extractLogAndPopulateCollection instead of extractImportantPartsOfLogEntry but is that also break single responsibility?

Copy link

@eranha eranha May 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if a method has And in its name, it means it's doing more than one thing and thus violating the first rule of SOLID->SRP
parseAndPopulate should be split to:
parse()
populate()

}

// Extracts important parts of each Env entry
public void envParser(String envInstance, EnvModel env) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is public, should it be public?. And if it should be public, due to the fact that it does not validate the string or checking the array boundaries, it can easily cause a runtime exception.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should definitely be private since no other classes call this. Updating this one now.

How does public make it easy to cause a runtime exception?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it's public anyone can access it and get an error


@Override
public void run(String... args) throws Exception {
AuthnTroubleshootController start = new AuthnTroubleshootController(new AuthnUserDataModel(), new AuthnTroubleshootView());
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refrain from using abbreviations use Authentication

@@ -0,0 +1,13 @@
package com.troubleshooting.model;

public interface IAuthnUserDataModel {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Starting an interface name with a capital I is a C# convention.
Missing documentation.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to differentiate between Interfaces and Regular classes that implement them. Would AuthenticationUserDataModelInterface suffice?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No not in java look at all the interfaces in Java they do not have any name identifier.
Map, Collection, List, Stream etc.

sigalsax added 4 commits May 26, 2020 09:11
- Removes abbreviations from methods and class names
- Adds documentation that can be parsed
- Adds addition checks for each log entry
- Update ConjurLog object to accurately represent Conjur logs
- Accept only logs in the expected format
- Fix ValidateService validateInput predicate
- Add Hot refresh dev tools
String [] envParts = envInstance.split("=|\n");

for (int i = 0; i < envParts.length; i+=2) {
if(Arrays.stream(includedEnv).anyMatch(envParts[i]::equals)) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggested code:

	  String data = "HOSTNAME=253d906eab8e\n" +
                    "CONJUR_DATA_KEY=G2+iUjmifBzFlqXjmQT4+hq2WssOumjRkrzbadvILYg=\n" +
                    "PWD=/opt/conjur-server\n" +
                    "HOME=/root\n" +
                    "DATABASE_URL=postgres://postgres@database/postgres";

        Properties properties = new Properties();
        try {
            properties.load(new StringReader(data));
        } catch (IOException e) {
            throw new ArgumentExcption(String.format("The input is not the expected format, %s", data));
        }

        List<String> includedEnv = Arrays.asList(new String[] {
                "HOSTNAME",
                "DATABASE_URL",
                "PORT",
                "CONJUR_ADMIN_PASSWORD",
                "CONJUR_LOG_LEVEL",
                "CONJUR_PASSWORD",
                "CONJUR_ACCOUNT"
        });

        List<EnvironmentModel> environmentModels = properties.entrySet()
                .stream().filter(e -> includedEnv.contains(e.getKey()))
                .map(e -> new EnvironmentModel(e.getKey().toString(), e.getValue().toString()))
                .collect(Collectors.toList());

        EnvironmentsModel environmentsModel = new EnvironmentsModel();
        environmentModels.addAll(environmentModels);

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the EnvironmentsModel environmentsModel = new EnvironmentsModel(); necessary here?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you mean environmentsModel?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just as a code reference

// TODO Add query to property configuration
LogContainerCmd logContainer = dockerClient.logContainerCmd(containerID).withStdOut(true).withStdErr(true);
try {
logContainer.exec(new LogContainerResultCallback() {
Copy link
Owner Author

@sigalsax sigalsax Jun 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what you mean here :o

where? Instead of in the block below?

public void onNext(Frame item) {}

- Remove MVC objects for easier iteration
- Remove "I" in interface names
- Remove view because it is repetitive now that I have JSP files for views
- Add AJAX and Jquery for rendering data without navigating to new page

spring.mvc.view.prefix: /WEB-INF/jsp/
spring.mvc.view.suffix: .jsp
server.port = 4200
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add new line

final private List<ConjurLogModel> logList = new ArrayList<>();

public void add(ConjurLogModel log) {
logList.add(log);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is public, if the argument is null, you will have a null entry [null] in your list.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a check

    public void add(ConjurLogModel log) {
        if (log != null) {
            logList.add(log);
        }
    }

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need the method public to be reachable from the calling class, no?

List<String> loggingFrames = new ArrayList<String>();

loggingFrames = access.getLogs(containerID, query);
loggingFrames.stream().forEach(entry -> extractImportantPartsOfLogEntry(entry));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace lambda with method reference

</head>

<script src="http://code.jquery.com/jquery-1.7.1.min.js"></script>
<script type="text/javascript">
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider having the script in a separate file.

@@ -0,0 +1,63 @@
package com.troubleshooting.controller;
Copy link

@eranha eranha Jul 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Namespace is java reads the company domain name in reverse, so it should be com.cyberak.troubleshooing.controller and if it relates only to docker maybe com.cyberak.docker.troubleshooing.controller?

- Separate the documentations for clarity
- Add TODO list to README
@sigalsax sigalsax force-pushed the implementation-docker-client branch from 2796eaf to 2ed0c40 Compare August 27, 2020 08:35
@sigalsax sigalsax merged commit 58185ca into master Aug 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants