Skip to content

Commit

Permalink
[ML-166] Backport to 1.3 to update documentation and Changelog (#169)
Browse files Browse the repository at this point in the history
* [ML-166] Update Changelog and OAP Documentation

* [ML-166] Update Changelog and OAP Documentation

* [ML-166] Update Changelog
  • Loading branch information
Hong authored Jan 15, 2022
1 parent 471718c commit 4e9493b
Show file tree
Hide file tree
Showing 6 changed files with 254 additions and 313 deletions.
203 changes: 202 additions & 1 deletion CHANGELOG.md

Large diffs are not rendered by default.

24 changes: 7 additions & 17 deletions docs/OAP-Developer-Guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,9 @@
This document contains the instructions & scripts on installing necessary dependencies and building OAP modules.
You can get more detailed information from OAP each module below.

* [SQL Index and Data Source Cache](https://github.com/oap-project/sql-ds-cache/blob/v1.2.0/docs/Developer-Guide.md)
* [PMem Common](https://github.com/oap-project/pmem-common/tree/v1.2.0)
* [PMem Spill](https://github.com/oap-project/pmem-spill/tree/v1.2.0)
* [PMem Shuffle](https://github.com/oap-project/pmem-shuffle/tree/v1.2.0#5-install-dependencies-for-pmem-shuffle)
* [Remote Shuffle](https://github.com/oap-project/remote-shuffle/tree/v1.2.0)
* [OAP MLlib](https://github.com/oap-project/oap-mllib/tree/v1.2.0)
* [Gazelle Plugin](https://github.com/oap-project/gazelle_plugin/tree/v1.2.0)

* [OAP MLlib](https://github.com/oap-project/oap-mllib/tree/v1.3.0)
* [Gazelle Plugin](https://github.com/oap-project/gazelle_plugin/tree/v1.3.0)

## Building OAP

Expand All @@ -22,22 +18,16 @@ We provide scripts to help automatically install dependencies required, please c
# cd oap-tools
# sh dev/install-compile-time-dependencies.sh
```
*Note*: oap-tools tag version `v1.2.0` corresponds to all OAP modules' tag version `v1.2.0`.
*Note*: oap-tools tag version `v1.3.0` corresponds to all OAP modules' tag version `v1.3.0`.

Then the dependencies below will be installed:

* [Cmake](https://cmake.org/install/)
* [GCC > 7](https://gcc.gnu.org/wiki/InstallingGCC)
* [Memkind](https://github.com/memkind/memkind/tree/v1.10.1)
* [Vmemcache](https://github.com/pmem/vmemcache)
* [HPNL](https://github.com/Intel-bigdata/HPNL)
* [PMDK](https://github.com/pmem/pmdk)
* [OneAPI](https://software.intel.com/content/www/us/en/develop/tools/oneapi.html)
* [Arrow](https://github.com/oap-project/arrow/tree/v4.0.0-oap-1.2.0)
* [Arrow](https://github.com/oap-project/arrow/tree/v4.0.0-oap-1.3.0)
* [LLVM](https://llvm.org/)

- **Requirements for Shuffle Remote PMem Extension**
If enable Shuffle Remote PMem extension with RDMA, you can refer to [PMem Shuffle](https://github.com/oap-project/pmem-shuffle) to configure and validate RDMA in advance.

### Building

Expand All @@ -55,9 +45,9 @@ Change to `root` user, run

#### Building OAP specific module

If you just want to build a specific OAP Module, such as `sql-ds-cache`, change to `root` user, then run:
If you just want to build a specific OAP Module, such as `gazelle_plugin`, change to `root` user, then run:

```
# cd oap-tools
# sh dev/compile-oap.sh --component=sql-ds-cache
# sh dev/compile-oap.sh --component=gazelle_plugin
```
17 changes: 2 additions & 15 deletions docs/OAP-Installation-Guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,25 +28,11 @@ To test your installation, run the command `conda list` in your terminal window
Create a Conda environment and install OAP Conda package.

```bash
$ conda create -n oapenv -c conda-forge -c intel -y oap=1.2.0
$ conda create -n oapenv -c conda-forge -c intel -y oap=1.3.0
```

Once finished steps above, you have completed OAP dependencies installation and OAP building, and will find built OAP jars under `$HOME/miniconda2/envs/oapenv/oap_jars`

Dependencies below are required by OAP and all of them are included in OAP Conda package, they will be automatically installed in your cluster when you Conda install OAP. Ensure you have activated environment which you created in the previous steps.

- [Arrow](https://github.com/oap-project/arrow/tree/v4.0.0-oap-1.2.0)
- [Plasma](http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/)
- [Memkind](https://github.com/memkind/memkind/tree/v1.10.1)
- [Vmemcache](https://github.com/pmem/vmemcache.git)
- [HPNL](https://anaconda.org/intel/hpnl)
- [PMDK](https://github.com/pmem/pmdk)
- [OneAPI](https://software.intel.com/content/www/us/en/develop/tools/oneapi.html)


#### Extra Steps for Shuffle Remote PMem Extension

If you use one of OAP features -- [PMem Shuffle](https://github.com/oap-project/pmem-shuffle) with **RDMA**, you need to configure and validate RDMA, please refer to [PMem Shuffle](https://github.com/oap-project/pmem-shuffle#4-configure-and-validate-rdma) for the details.


### Configuration
Expand All @@ -57,6 +43,7 @@ Once finished steps above, make sure libraries installed by Conda can be linked
spark.executorEnv.LD_LIBRARY_PATH $HOME/miniconda2/envs/oapenv/lib
spark.executor.extraLibraryPath $HOME/miniconda2/envs/oapenv/lib
spark.driver.extraLibraryPath $HOME/miniconda2/envs/oapenv/lib
spark.executorEnv.CC $HOME/miniconda2/envs/oapenv/bin/gcc
spark.executor.extraClassPath $HOME/miniconda2/envs/oapenv/oap_jars/$OAP_FEATURE.jar
spark.driver.extraClassPath $HOME/miniconda2/envs/oapenv/oap_jars/$OAP_FEATURE.jar
```
Expand Down
220 changes: 0 additions & 220 deletions docs/User-Guide.md

This file was deleted.

Loading

0 comments on commit 4e9493b

Please sign in to comment.