Skip to content

Commit

Permalink
fix parentheses and info/warning/danger formatting in docs (airbytehq…
Browse files Browse the repository at this point in the history
…#11981)

* fix parens

* fix formatting for hints

* remove hint from non-docusaurus or gitbook doc
  • Loading branch information
jrhizor authored Apr 13, 2022
1 parent 2881d62 commit 14dfc1d
Show file tree
Hide file tree
Showing 59 changed files with 325 additions and 172 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,8 @@ For information about how to use this connector within Airbyte, see [the User Do
## Databricks JDBC Driver
This connector requires a JDBC driver to connect to Databricks cluster. The driver is developed by Simba.

{% hint style="warning" %}
WARNING:
Before building, or using this connector, you must agree to the [JDBC ODBC driver license](https://databricks.com/jdbc-odbc-driver-license). This means that you can only use this driver to connector third party applications to Apache Spark SQL within a Databricks offering using the ODBC and/or JDBC protocols.
{% endhint %}

This is currently a private connector that is only available on Airbyte Cloud. We are working on a solution to publicize it (issue [\#6043](https://github.com/airbytehq/airbyte/issues/6043)).

Expand Down
6 changes: 4 additions & 2 deletions docs/api-documentation.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
# API documentation

{% hint style="warning" %}
:::caution

For Airbyte Open-Source you don't need the API Token for Authentication!
All endpoints are possible to access using the API without it. **Note**: Airbyte Cloud does not currently support API access.
{% endhint %}

:::

Our Configuration API is still in alpha and might change. You won’t lose any functionality, but you may need to update your code to catch up to any backwards incompatible changes in the API.

Expand Down
6 changes: 4 additions & 2 deletions docs/archive/examples/postgres-replication.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,11 @@ Now let's verify that this worked. Let's output the contents of the destination
docker exec airbyte-destination psql -U postgres -c "SELECT * FROM public.users;"
```

{% hint style="info" %}
:::info

Don't worry about the awkward `public_users` name for now; we are currently working on an update to allow users to configure their destination table names!
{% endhint %}

:::

You should see the rows from the source database inside the destination database!

Expand Down
6 changes: 4 additions & 2 deletions docs/archive/examples/slack-history.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,11 @@ docker run -it --rm \

That's it!

{% hint style="info" %}
:::info

MeiliSearch stores data in $\(pwd\)/data.ms, so if you prefer to store it somewhere else, just adjust this path.
{% endhint %}

:::

## 2. Replicate Your Slack Messages to MeiliSearch

Expand Down
2 changes: 1 addition & 1 deletion docs/connector-development/cdk-faros-js.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Connector Development Kit \(Javascript\)
# Connector Development Kit (Javascript)

The [Faros AI TypeScript/JavaScript CDK](https://github.com/faros-ai/airbyte-connectors/tree/main/faros-airbyte-cdk) allows you to build Airbyte connectors quickly similarly to how our [Python CDK](cdk-python/) does. This CDK currently offers support for creating Airbyte source connectors for:

Expand Down
2 changes: 1 addition & 1 deletion docs/connector-development/cdk-python/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Connector Development Kit \(Python\)
# Connector Development Kit (Python)

The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,11 @@ this will display a multi-line textbox in the UI like the following screenshot:

In some cases, a connector needs to accept one out of many options. For example, a connector might need to know the compression codec of the file it will read, which will render in the Airbyte UI as a list of the available codecs. In JSONSchema, this can be expressed using the [oneOf](https://json-schema.org/understanding-json-schema/reference/combining.html#oneof) keyword.

{% hint style="info" %}
:::info

Some connectors may follow an older format for dropdown lists, we are currently migrating away from that to this standard.
{% endhint %}

:::

In order for the Airbyte UI to correctly render a specification, however, a few extra rules must be followed:

Expand Down
6 changes: 4 additions & 2 deletions docs/connector-development/testing-connectors/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,11 @@ For instance:
```
### 3. Requesting GitHub PR Integration Test Runs

{% hint style="warning" %}
:::caution

This option is not available to PRs from forks, so it is effectively limited to Airbyte employees.
{% endhint %}

:::

If you don't want to handle secrets, you're making a relatively minor change, or you want to ensure the connector's integration test will run remotely, you should request builds on GitHub. You can request an integration test run by creating a comment with a slash command.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,11 @@ First, you can run the image locally. Connectors should have instructions in the

### 2. Requesting GitHub PR Integration Test Runs

{% hint style="warning" %}
:::caution

This option is not available to PRs from forks, so it is effectively limited to Airbyte employees.
{% endhint %}

:::

If you don't want to handle secrets, you're making a relatively minor change, or you want to ensure the connector's integration test will run remotely, you should request builds on GitHub. You can request an integration test run by creating a comment with a slash command.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,17 @@ Docker and Java with the versions listed in the [tech stack section](../../under
* Step 7: Write unit tests or integration tests
* Step 8: Update the docs \(in `docs/integrations/destinations/<destination-name>.md`\)

{% hint style="info" %}
:::info

All `./gradlew` commands must be run from the root of the airbyte project.
{% endhint %}

{% hint style="info" %}
:::

:::info

If you need help with any step of the process, feel free to submit a PR with your progress and any questions you have, or ask us on [slack](https://slack.airbyte.io).
{% endhint %}

:::

## Explaining Each Step

Expand All @@ -53,9 +57,11 @@ You can build the destination by running:

This compiles the Java code for your destination and builds a Docker image with the connector. At this point, we haven't implemented anything of value yet, but once we do, you'll use this command to compile your code and Docker image.

{% hint style="info" %}
:::info

Airbyte uses Gradle to manage Java dependencies. To add dependencies for your connector, manage them in the `build.gradle` file inside your connector's directory.
{% endhint %}

:::

#### Iterating on your implementation

Expand Down Expand Up @@ -156,9 +162,11 @@ To implement the `write` Airbyte operation, implement the `getConsumer` method i
* [Local CSV](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/destination-csv/src/main/java/io/airbyte/integrations/destination/csv/CsvDestination.java#L90)
* [Postgres](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/destination-postgres/src/main/java/io/airbyte/integrations/destination/postgres/PostgresDestination.java)

{% hint style="info" %}
:::info

The Postgres destination leverages the `AbstractJdbcDestination` superclass which makes it extremely easy to create a destination for a database or data warehouse if it has a compatible JDBC driver. If the destination you are implementing has a JDBC driver, be sure to check out `AbstractJdbcDestination`.
{% endhint %}

:::

For a brief overview on the Airbyte catalog check out [the Beginner's Guide to the Airbyte Catalog](../../understanding-airbyte/beginners-guide-to-catalog.md).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@ Docker and Python with the versions listed in the [tech stack section](../../und
* Step 7: Write unit tests or integration tests
* Step 8: Update the docs \(in `docs/integrations/destinations/<destination-name>.md`\)

{% hint style="info" %}
:::info

If you need help with any step of the process, feel free to submit a PR with your progress and any questions you have, or ask us on [slack](https://slack.airbyte.io). Also reference the KvDB python destination implementation if you want to see an example of a working destination.
{% endhint %}

:::

## Explaining Each Step

Expand Down Expand Up @@ -79,9 +81,11 @@ The destination interface is described in detail in the [Airbyte Specification](

The generated files fill in a lot of information for you and have docstrings describing what you need to do to implement each method. The next few steps are just implementing that interface.

{% hint style="info" %}
:::info

All logging should be done through the `self.logger` object available in the `Destination` class. Otherwise, logs will not be shown properly in the Airbyte UI.
{% endhint %}

:::

Everyone develops differently but here are 3 ways that we recommend iterating on a destination. Consider using whichever one matches your style.

Expand Down
36 changes: 24 additions & 12 deletions docs/connector-development/tutorials/building-a-python-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@ This article provides a checklist for how to create a python source. Each step i

Docker, Python, and Java with the versions listed in the [tech stack section](../../understanding-airbyte/tech-stack.md).

{% hint style="info" %}
:::info

All the commands below assume that `python` points to a version of python &gt;3.7. On some systems, `python` points to a Python2 installation and `python3` points to Python3. If this is the case on your machine, substitute all `python` commands in this guide with `python3` . Otherwise, make sure to install Python 3 before beginning.
{% endhint %}

:::

## Checklist

Expand All @@ -29,13 +31,17 @@ All the commands below assume that `python` points to a version of python &gt;3.
* Step 11: Add the connector to the API/UI \(by adding an entry in `airbyte-config/init/src/main/resources/seed/source_definitions.yaml`\)
* Step 12: Add docs \(in `docs/integrations/sources/<source-name>.md`\)

{% hint style="info" %}
:::info

Each step of the Creating a Source checklist is explained in more detail below.
{% endhint %}

{% hint style="info" %}
:::

:::info

All `./gradlew` commands must be run from the root of the airbyte project.
{% endhint %}

:::

### Submitting a Source to Airbyte

Expand All @@ -45,9 +51,11 @@ All `./gradlew` commands must be run from the root of the airbyte project.
* Once the config is stored in Github Secrets, edit `.github/workflows/test-command.yml` and `.github/workflows/publish-command.yml` to inject the config into the build environment.
* Edit the `airbyte/tools/bin/ci_credentials.sh` script to pull the script from the build environment and write it to `secrets/config.json` during the build.

{% hint style="info" %}
:::info

If you have a question about a step the Submitting a Source to Airbyte checklist include it in your PR or ask it on [slack](https://slack.airbyte.io).
{% endhint %}

:::

## Explaining Each Step

Expand Down Expand Up @@ -96,9 +104,11 @@ The commands we ran above created a virtual environment for your source. If you

Pretty much all it takes to create a source is to implement the `Source` interface. The template fills in a lot of information for you and has extensive docstrings describing what you need to do to implement each method. The next 4 steps are just implementing that interface.

{% hint style="info" %}
:::info

All logging should be done through the `logger` object passed into each method. Otherwise, logs will not be shown in the Airbyte UI.
{% endhint %}

:::

#### Iterating on your implementation

Expand Down Expand Up @@ -175,9 +185,11 @@ The Standard Tests are a set of tests that run against all sources. These tests

You can run the tests using `./gradlew :airbyte-integrations:connectors:source-<source-name>:integrationTest`. Make sure to run this command from the Airbyte repository root.

{% hint style="info" %}
:::info

In some rare cases we make exceptions and allow a source to not need to pass all the standard tests. If for some reason you think your source cannot reasonably pass one of the tests cases, reach out to us on github or slack, and we can determine whether there's a change we can make so that the test will pass or if we should skip that test for your source.
{% endhint %}

:::

### Step 9: Write unit tests and/or integration tests

Expand Down
6 changes: 4 additions & 2 deletions docs/contributing-to-airbyte/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,11 @@ It's easy to add your own connector to Airbyte! **Since Airbyte connectors are e

For sources, simply head over to our [Python CDK](../connector-development/cdk-python/).

{% hint style="info" %}
:::info

The CDK currently does not support creating destinations, but it will very soon.
{% endhint %}

::::

* See [Building new connectors](../connector-development/) to get started.
* Since we frequently build connectors in Python, on top of Singer or in Java, we've created generator libraries to get you started quickly: [Build Python Source Connectors](../connector-development/tutorials/building-a-python-source.md) and [Build Java Destination Connectors](../connector-development/tutorials/building-a-java-destination.md)
Expand Down
30 changes: 20 additions & 10 deletions docs/contributing-to-airbyte/developing-locally.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@ The following technologies are required to build Airbyte locally.
4. `Docker`
5. `Jq`

{% hint style="info" %}
:::info

Manually switching between different language versions can get hairy. We recommend using a version manager such as [`pyenv`](https://github.com/pyenv/pyenv) or [`jenv`](https://github.com/jenv/jenv).
{% endhint %}

:::

To start contributing:

Expand All @@ -26,9 +28,11 @@ To start contributing:

## Build with `gradle`

{% hint style="info" %}
:::info

If you're using Mac M1 \(Apple Silicon\) machines, you may run into some problems (Temporal failing during runs, and some connectors not working). See the [GitHub issue](https://github.com/airbytehq/airbyte/issues/2017) for more information.
{% endhint %}

:::

To compile and build just the platform \(not all the connectors\):

Expand All @@ -40,28 +44,34 @@ This will build all the code and run all the unit tests.

`SUB_BUILD=PLATFORM ./gradlew build` creates all the necessary artifacts \(Webapp, Jars and Docker images\) so that you can run Airbyte locally. Since this builds everything, it can take some time.

{% hint style="info" %}
:::info

Optionally, you may pass a `VERSION` environment variable to the gradle build command. If present, gradle will use this value as a tag for all created artifacts (both Jars and Docker images).

If unset, gradle will default to using the current VERSION in `.env` for Jars, and `dev` as the Docker image tag.
{% endhint %}

{% hint style="info" %}
:::

:::info

Gradle will use all CPU cores by default. If Gradle uses too much/too little CPU, tuning the number of CPU cores it uses to better suit a dev's need can help.

Adjust this by either, 1. Setting an env var: `export GRADLE_OPTS="-Dorg.gradle.workers.max=3"`. 2. Setting a cli option: `SUB_BUILD=PLATFORM ./gradlew build --max-workers 3` 3. Setting the `org.gradle.workers.max` property in the `gradle.properties` file.

A good rule of thumb is to set this to \(\# of cores - 1\).
{% endhint %}

{% hint style="info" %}
:::

:::info

On Mac, if you run into an error while compiling openssl \(this happens when running pip install\), you may need to explicitly add these flags to your bash profile so that the C compiler can find the appropriate libraries.

```text
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
```
{% endhint %}

:::

## Run in `dev` mode with `docker-compose`

Expand Down
10 changes: 6 additions & 4 deletions docs/deploying-airbyte/local-deployment.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Local Deployment

{% hint style="info" %}
:::info
These instructions have been tested on MacOS, Windows 10 and Ubuntu 20.04.
{% endhint %}

:::

## Setup & launch Airbyte

Expand All @@ -15,13 +16,14 @@ cd airbyte
docker-compose up
```

{% hint style="info" %}
:::info

If you're using Mac M1 \(Apple Silicon\) machines, you must [build and run Airbyte locally in dev mode](../contributing-to-airbyte/developing-locally.md).
Some users using Macs with an M1 chip are facing problems running Airbyte even with a locally built version of the platform.
You can subscribe to [Issue \#2017](https://github.com/airbytehq/airbyte/issues/2017) and get updates on M1-related issues.

{% endhint %}

:::

* In your browser, just visit [http://localhost:8000](http://localhost:8000)
* Start moving some data!
Expand Down
Loading

0 comments on commit 14dfc1d

Please sign in to comment.