Skip to content

Commit

Permalink
Add implementation for extraction of env and logs from Docker contain…
Browse files Browse the repository at this point in the history 
…ers (#2)

Troubleshooting authentication errors can be a time-consuming undertaking as a developer works with different development and deployment environments, platforms, and tooling that understanding a cryptic 401 Unauthorized error is near to impossible. To tackle this ongoing issue, we need an effective tool that will aid in the troubleshooting process.

- Add docker-api client as a wrapper for accessing docker containers to extract logs and environment variables from containers
- Add HTML with Ajax call for rendering the result on same configuration page
  • Loading branch information
@sigalsax
sigalsax authored Aug 27, 2020
1 parent 8bb319b commit 58185ca
Show file tree
Hide file tree
Showing 26 changed files with 815 additions and 85 deletions.
31 changes: 31 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
HELP.md
target/
!.mvn/wrapper/maven-wrapper.jar
!**/src/main/**
!**/src/test/**

### STS ###
.apt_generated
.classpath
.factorypath
.project
.settings
.springBeans
.sts4-cache

### IntelliJ IDEA ###
.idea
*.iws
*.iml
*.ipr

### NetBeans ###
/nbproject/private/
/nbbuild/
/dist/
/nbdist/
/.nb-gradle/
build/

### VS Code ###
.vscode/
90 changes: 5 additions & 85 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,97 +1,17 @@
# Conjur Authentication Troubleshooting Tool

## Proposal

Troubleshooting authentication errors can be a time-consuming undertaking as a developer works with different development
and deployment environments, platforms, and tooling that understanding a cryptic 401 Unauthorized error is near to impossible.
To tackle this ongoing issue, we need an effective tool (with a great UI) that will aid in the troubleshooting process.

## Goal

Provide value to the team while also expanding knowledge and experience on a variety of topics REST/curl, MVC, Design
Patterns, Docker, Angular.

## Ideas:

- A tool that fetches specific logs according to defined authentication errors/keywords from Conjur in a containerized
environment, container ENV variables, and all related information about the unauthenticated user (if the user exists in
Conjur, the Conjur policy where the user is defined, their privileges, etc).

- For DAP, checks API Endpoints are functioning correctly by making queries to health, nginx services, etc _(out of scope)_

- For DAP, fetch Master/Follower logs and configurations _(out of scope)_

- Salesforce tool that extracts all useful information from cases necessary for engineers to troubleshoot the case

- Page with mappings of repository names to their pull commands. Ex: `conjur` → `docker pull registry2.itci.conjur.net/conjur` (for both private and public registries)

- Slackbot where if you use a certain #tag (#documentation), will create salesforce cases for documentation

## MVP

For the first iteration of this tool, I have decided to pursue the first idea and create a tool that will do the following:

- Connect with Conjur / DAP containers

- Extract ENV variables

- *NOTE:* By default, the tool also supports DAP b/c from the functionality currently offered, the user can input
the container name/id and receive the necessary logs and ENV variables which is environment agnostic

- *NOTE:* For OSS, we are limited to setting the log levels as container ENV variables before OSS spin up

- Parse and extract Conjur / DAP logs based on the type user requests

- Return the user's relevant data in Conjur (if the user exists in Conjur, the Conjur policy where the user is defined,
their privileges, etc).
To tackle this ongoing issue, we need an effective tool that will aid in the troubleshooting process.

- Return a result page with data
## Development

## Phase 0

- [x] Reach out to developers, PO/Ms, and SEs to understand the pain points and use this feedback to develop tooling that will feature in the project

- [x] Define an MVP that will provide immediate value and a running backlog

- [x] Create the design
To spin up the application, navigate to project lines in your command line window, run `mvn spring-boot:run`, and
navigate to `localhost:4200`.

## Design

Class diagram:

![Class Diagram](./classDiagram.png)

Sequence diagram:

![System Sequence Diagram](./troubleshootSystemSequenceDiagram.png)

Activity diagram:

![Activity Diagram](./activityDiagram.png)

UI Mocks can be found here: https://www.figma.com/file/QcAxsC58Lh969NLunI8sTT/Authn2Mock?node-id=0%3A1

- App will save state so that the user does not have to redefine query with each session

### Language

I will build an application with Spring Boot to, amongst other things, manage dependencies, abstract away complexities with
feature flags, and provide a server out of box. I will use the [docker-java](https://github.com/docker-java/docker-java)
Java Docker API client. Their docs are hosted [here](https://javadoc.io/doc/com.github.docker-java/docker-java/3.0.1/index.html) and
I will utilize the following:

- ExecCreateCmd exec = dockerClient.execCreateCmd(containerId).withCmd("env"), to extract ENV variables from the container

- LogContainerCmd cmd = dockerClient.logContainerCmd(containerId), to extract logs from the container

TODO: Figure out how to export logs like so: `docker logs 06b329936e62 | grep "authentication"` efficiently without exporting
the whole log file. Example 2 [here](https://www.programcreek.com/java-api-examples/?api=com.github.dockerjava.api.command.LogContainerCmd)
might be helpful.
For details on the design, head over to the [Solution Design doc](./SolutionDesign.md)

- TarArchiveInputStream tarStream = new TarArchiveInputStream(dockerClient.copyArchiveFromContainerCmd(containerName,
containerFile).exec()), to extract policy files from the container as a tar.

In Conjur OSS (quickstart), files are located in the `policy` directory. So I will extract and query policy files for the
unauthenticated user. Helpful resource: https://github.com/docker-java/docker-java/issues/991

### Open questions:
125 changes: 125 additions & 0 deletions SolutionDesign.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# Conjur Authentication Troubleshooting Tool

## Proposal

Troubleshooting authentication errors can be a time-consuming undertaking as a developer works with different development
and deployment environments, platforms, and tooling that understanding a cryptic 401 Unauthorized error is near to impossible.
To tackle this ongoing issue, we need an effective tool (with a great UI) that will aid in the troubleshooting process.

## Goal

Provide value to the team while also expanding knowledge and experience on a variety of topics REST/curl, MVC, Design
Patterns, Docker, Angular.

## Ideas:

- A tool that fetches specific logs according to defined authentication errors/keywords from Conjur in a containerized
environment, container ENV variables, and all related information about the unauthenticated user (if the user exists in
Conjur, the Conjur policy where the user is defined, their privileges, etc).

- For DAP, checks API Endpoints are functioning correctly by making queries to health, nginx services, etc _(out of scope)_

- For DAP, fetch Master/Follower logs and configurations _(out of scope)_

- Salesforce tool that extracts all useful information from cases necessary for engineers to troubleshoot the case

- Page with mappings of repository names to their pull commands. Ex: `conjur` → `docker pull registry2.itci.conjur.net/conjur` (for both private and public registries)

- Slackbot where if you use a certain #tag (#documentation), will create salesforce cases for documentation

## MVP

For the first iteration of this tool, I have decided to pursue the first idea and create a tool that will do the following:

- Connect with Conjur / DAP containers

- Extract ENV variables

- *NOTE:* By default, the tool also supports DAP b/c from the functionality currently offered, the user can input
the container name/id and receive the necessary logs and ENV variables which is environment agnostic

- *NOTE:* For OSS, we are limited to setting the log levels as container ENV variables before OSS spin up

- Parse and extract Conjur / DAP logs based on the type user requests

- Return the user's relevant data in Conjur (if the user exists in Conjur, the Conjur policy where the user is defined,
their privileges, etc).

- Return a result page with data

## Phase 0

- [x] Reach out to developers, PO/Ms, and SEs to understand the pain points and use this feedback to develop tooling that will feature in the project

- [x] Define an MVP that will provide immediate value and a running backlog

- [x] Create the design

## Phase 1

- [x] Construct classes and interfaces according to MVC

- [x] Communicate with Docker using the Java Docker client

- [x] Use Ajax / JQuery to render result on same page, without page reload

- [x] Java objects are converted to JSON objects

- [x] Page renders JSON objects

### TODO:

- [ ] Combine Env + Log JSON objects to be rendered together under 1 call

- [ ] Remove duplication of responses with each submit

- [ ] Add back in MVC objects and figure out how to integrate MVC with @ResponseBody

- [ ] Style the page with CSS

### Bugs:

- [ ] New log entries and environment variables are being added to their lists with every "submit" even though input parameters are the same



## Design

Class diagram:

![Class Diagram](./classDiagram.png)

Sequence diagram:

![System Sequence Diagram](./troubleshootSystemSequenceDiagram.png)

Activity diagram:

![Activity Diagram](./activityDiagram.png)

UI Mocks can be found here: https://www.figma.com/file/QcAxsC58Lh969NLunI8sTT/Authn2Mock?node-id=0%3A1

- App will save state so that the user does not have to redefine query with each session

### Language

I will build an application with Spring Boot to, amongst other things, manage dependencies, abstract away complexities with
feature flags, and provide a server out of box. I will use the [docker-java](https://github.com/docker-java/docker-java)
Java Docker API client. Their docs are hosted [here](https://javadoc.io/doc/com.github.docker-java/docker-java/3.0.1/index.html) and
I will utilize the following:

- ExecCreateCmd exec = dockerClient.execCreateCmd(containerId).withCmd("env"), to extract ENV variables from the container

- LogContainerCmd cmd = dockerClient.logContainerCmd(containerId), to extract logs from the container

TODO: Figure out how to export logs like so: `docker logs 06b329936e62 | grep "authentication"` efficiently without exporting
the whole log file. Example 2 [here](https://www.programcreek.com/java-api-examples/?api=com.github.dockerjava.api.command.LogContainerCmd)
might be helpful.

- TarArchiveInputStream tarStream = new TarArchiveInputStream(dockerClient.copyArchiveFromContainerCmd(containerName,
containerFile).exec()), to extract policy files from the container as a tar.

In Conjur OSS (quickstart), files are located in the `policy` directory. So I will extract and query policy files for the
unauthenticated user. Helpful resource: https://github.com/docker-java/docker-java/issues/991

### Open questions:
79 changes: 79 additions & 0 deletions pom.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.2.6.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.troubleshooting</groupId>
<artifactId>troubleshooting-tool</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>troubleshooting-tool</name>
<description>Conjur troubleshooting tool</description>

<properties>
<java.version>1.8</java.version>
</properties>

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<!-- <scope>provided</scope>-->
</dependency>

<dependency>
<groupId>org.apache.tomcat.embed</groupId>
<artifactId>tomcat-embed-jasper</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
</dependency>

<!-- https://mvnrepository.com/artifact/com.github.docker-java/docker-java -->
<dependency>
<groupId>com.github.docker-java</groupId>
<artifactId>docker-java</artifactId>
<version>3.2.1</version>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.8</version>
</dependency>
<dependency>
<groupId>net.minidev</groupId>
<artifactId>json-smart</artifactId>
<version>2.3</version>
<scope>compile</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>

</project>
Loading

0 comments on commit 58185ca

Please sign in to comment.