feat(others): migrate other files (#930)

* feat(others): migrate other files Signed-off-by: Robin Han <[email protected]> * feat(test): add s3 unit test tag Signed-off-by: Robin Han <[email protected]> --------- Signed-off-by: Robin Han <[email protected]> Co-authored-by: KamiWan <[email protected]>
AutoMQ · Mar 14, 2024 · 5478c5b · 5478c5b
1 parent 3fecccf
commit 5478c5b
Show file tree

Hide file tree

Showing 25 changed files with 952 additions and 276 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug-report.md b/.github/ISSUE_TEMPLATE/bug-report.md
@@ -0,0 +1,41 @@
+---
+name: "\U0001F41B Bug report"
+about: Something is not working
+title: "[BUG] "
+labels: bug
+assignees: ''
+
+---
+
+### Version & Environment
+
+<!--
+
+Please give your AutoMQ version and environment info:
+- AutoMQ version (e.g. `v0.1.0`):
+- Operating System (e.g. from `/etc/os-release`):
+
+-->
+
+
+### What went wrong?
+
+<!--
+  Describe the unexpected behavior
+-->
+
+### What should have happened instead?
+
+<!--
+  Describe what you wanted to happen.
+-->
+
+### How to reproduce the issue?
+
+1.
+2.
+3.
+
+### Additional information
+
+Please attach any relevant logs, backtraces, or metric charts.
diff --git a/.github/ISSUE_TEMPLATE/feature-request.md b/.github/ISSUE_TEMPLATE/feature-request.md
@@ -0,0 +1,16 @@
+---
+name: "\U0001F680 Feature request"
+about: Suggest an idea for AutoMQ
+title: "[Enhancement]"
+labels: enhancement
+assignees: ''
+
+---
+
+### Who is this for and what problem do they have today?
+
+
+### Why is solving this problem impactful?
+
+
+### Additional notes
diff --git a/.github/ISSUE_TEMPLATE/good-first-issue.md b/.github/ISSUE_TEMPLATE/good-first-issue.md
@@ -0,0 +1,43 @@
+---
+name: "⭐ Good first issue"
+about: Design good first issue for new contributors
+title: "[Good first issue] "
+labels: good first issue
+assignees: ''
+
+---
+
+### Background
+
+<!--
+
+Please give your issue background.
+e.g. Now AutoMQ Kafka's logs are only stored in the local disk. We want to store them in the cloud object storage as well to offer the ability to query logs from object storage. Store logs on the object storage is cheaper and more reliable.
+
+-->
+
+### What's our expectation for the issue
+
+<!--
+
+e.g. Local file logs still exist. When log is flushed to local file system, the log data will upload to object storage as well. The log path will be like `s3://bucket-name/automq/cluster-id/broker-id/logs/xx`. 
+
+-->
+
+### How to started
+<!--
+
+Guide the developer how to complete the issue,including:
+
+e.g.
+- Precondition: 
+    - You need to know the principal of how AutoMQ print logs to local file system.
+- What main classes are involved when you are coding:
+    - `ExampleClassA`
+    - `ExampleClassB`
+- Other tips: 
+    - You can refer to the `ExampleClassA` and `ExampleClassB` of AutoMQ Kafka to get some inspiration.
+-->
+
+### Reference
+- [Kafka Official Document](https://kafka.apache.org/documentation/)
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Build
+on:
+  pull_request:
+    types: [ opened, reopened, synchronize ]
+  push:
+    branches: [ "master", "develop" ]
+
+jobs:
+  paths-filter:
+    runs-on: ubuntu-latest
+    outputs:
+      build-kos: ${{ steps.filter.outputs.build-kos }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dorny/paths-filter@v2
+        id: filter
+        with:
+          filters: |
+            build-kos:
+              - '.github/workflows/**'
+              - 'core/**'
+              - 'metadata/**'
+              - 'shell/**'
+  build-kos:
+    needs: [ paths-filter ]
+    if: ${{ needs.paths-filter.outputs.build-kos == 'true' || github.event_name == 'push' }}
+    uses: ./.github/workflows/build_kos.yml
+  build-result:
+    runs-on: ubuntu-latest
+    needs: [ build-kos ]
+    if: ${{ always() }}
+    steps:
+      - uses: actions/checkout@v4
+      - name: Collect build result
+        run: |
+          if echo build-kos-${{ needs.build-kos.result }} | grep -E 'cancelled|failure' -o > null
+          then
+            echo "There are failed/cancelled builds"
+            exit 1
+          else
+            echo "All builds are successful/skipped"
+            exit 0
+          fi
diff --git a/.github/workflows/build_kos.yml b/.github/workflows/build_kos.yml
@@ -0,0 +1,88 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+# This workflow will build a Java project with Gradle and cache/restore any dependencies to improve the workflow execution time
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-gradle
+
+name: Kafka on S3 - CI
+
+on:
+  workflow_call:
+
+permissions:
+  contents: read
+
+jobs:
+  checkstyle:
+    name: "Checkstyle"
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ ubuntu-22.04 ]
+        jdk: [ 17 ]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Set up JDK ${{ matrix.jdk }}
+        uses: actions/setup-java@v3
+        with:
+          java-version: ${{ matrix.jdk }}
+          distribution: "zulu"
+      - name: Setup Gradle
+        uses: gradle/[email protected]
+      - name: Checkstyle
+        run: ./gradlew --build-cache rat checkstyleMain checkstyleTest
+  spotbugs:
+    name: "Spotbugs"
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ ubuntu-22.04 ]
+        jdk: [ 17 ]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Set up JDK ${{ matrix.jdk }}
+        uses: actions/setup-java@v3
+        with:
+          java-version: ${{ matrix.jdk }}
+          distribution: "zulu"
+      - name: Setup Gradle
+        uses: gradle/[email protected]
+      - name: Spotbugs
+        run: ./gradlew --build-cache spotbugsMain spotbugsTest
+  test:
+    name: "Unit Test"
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ ubuntu-22.04 ]
+        jdk: [ 17 ]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Set up JDK ${{ matrix.jdk }}
+        uses: actions/setup-java@v3
+        with:
+          java-version: ${{ matrix.jdk }}
+          distribution: "zulu"
+      - name: Setup Gradle
+        uses: gradle/[email protected]
+      - name: Unit Test
+        run: ./gradlew --build-cache metadata:S3UnitTest core:S3UnitTest
diff --git a/.github/workflows/docker-release.yml b/.github/workflows/docker-release.yml
@@ -0,0 +1,58 @@
+name: Docker Release
+
+on:
+  push:
+    tags:
+      - '[0-9]+\.[0-9]+\.[0-9]+'
+      - '[0-9]+\.[0-9]+\.[0-9]+\-[A-Za-z0-9]+'
+
+jobs:
+  docker-release:
+    name: Docker Image Release
+    strategy:
+      matrix:
+        platform: [ "ubuntu-22.04" ]
+        jdk: ["17"]
+    runs-on: ${{ matrix.platform }}
+    permissions:
+      contents: write
+    steps:
+      - name: Checkout Code
+        uses: actions/checkout@v4
+      - name: Set up JDK ${{ matrix.jdk }}
+        uses: actions/setup-java@v3
+        with:
+          java-version: ${{ matrix.jdk }}
+          distribution: "zulu"
+      - name: Setup Gradle
+        uses: gradle/[email protected]
+      - name: Get project version
+        id: get_project_version
+        run: |
+          project_version=$(./gradlew properties | grep "version:" | awk '{print $2}')
+          echo "PROJECT_VERSION=${project_version}" >> $GITHUB_OUTPUT
+
+      - name: Build TarGz
+        run: |
+          ./gradlew -Pprefix=automq-${{ github.ref_name }}_ --build-cache --refresh-dependencies clean releaseTarGz
+
+      # docker image release
+      - name: Cp TarGz to Docker Path
+        run: |
+          cp ./core/build/distributions/automq-${{ github.ref_name }}_kafka-${{ steps.get_project_version.outputs.PROJECT_VERSION }}.tgz ./docker/
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_READ_WRITE_TOKEN }}
+      - name: Build and push
+        uses: docker/build-push-action@v5
+        with:
+          context: ./docker
+          push: true
+          tags: ${{ secrets.DOCKERHUB_USERNAME }}/kafka:${{ github.ref_name }}
+          platforms: linux/amd64,linux/arm64
diff --git a/.github/workflows/e2e-run.yml b/.github/workflows/e2e-run.yml
@@ -0,0 +1,57 @@
+name: Run E2E tests
+on:
+  workflow_call:
+    inputs:
+      runner:
+        required: true
+        type: string
+      suite-id:
+        required: true
+        type: string
+      test-yaml:
+        required: false
+        type: string
+      test-path:
+        required: false
+        type: string
+      storage-path:
+        required: true
+        type: string
+
+jobs:
+  run_e2e:
+    name: "Run E2E tests"
+    runs-on: ${{ inputs.runner }}
+    env:
+      TC_GENERAL_MIRROR_URL: "mirrors.ustc.edu.cn"
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Clean last running results
+        run: |
+          rm -rf results
+          rm -rf "${{ inputs.storage-path }}/${{ inputs.suite-id }}"
+      - name: Setup Gradle
+        uses: gradle/[email protected]
+      - name: Run E2E tests with yaml
+        if: ${{ inputs.test-path == '' }}
+        run: ./tests/docker/run_tests.sh
+        env:
+          ESK_TEST_YML: ${{ inputs.test-yaml }}
+        shell: bash
+      - name: Run E2E tests with path
+        if: ${{ inputs.test-path != '' }}
+        run: ./tests/docker/run_tests.sh
+        env:
+          TC_PATHS: ${{ inputs.test-path }}
+        shell: bash
+      - name: Move results
+        run: |
+          results_path="$(pwd)/results/$(readlink results/latest | cut -d'/' -f5)"
+          mv "${results_path}" "${{ inputs.storage-path }}/${{ inputs.suite-id }}"
+        if: ${{ always() }}
+        shell: bash
+      - name: Bring down docker containers
+        run: ./tests/docker/ducker-ak down
+        shell: bash
+        if: ${{ always() }}