Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Embed patched recipe for RocksDB 6.29.5 #4947

Merged
merged 13 commits into from
Mar 12, 2024
23 changes: 23 additions & 0 deletions .github/actions/dependencies/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,27 @@ runs:
- name: export custom recipes
shell: bash
run: |
conan config set general.revisions_enabled=1
conan export external/snappy snappy/1.1.10@
conan export external/rocksdb rocksdb/6.29.5@
conan export external/soci soci/4.0.3@
- name: try to authenticate to ripple Conan remote
id: remote
shell: bash
run: |
# Do not quote the URL. An empty string will be accepted (with
# a non-fatal warning), but a missing argument will not.
conan remote add ripple ${{ env.CONAN_URL }} --insert 0
echo outcome=$(conan user --remote ripple ${{ env.CONAN_USERNAME }} \
--password ${{ env.CONAN_TOKEN }} >&2 && echo success || \
echo failure) | tee ${GITHUB_OUTPUT}
- name: list missing binaries
id: binaries
shell: bash
# Print the list of dependencies that would need to be built locally.
# A non-empty list means we have "failed" to cache binaries remotely.
run: |
echo missing=$(conan info . --build missing --settings build_type=${{ inputs.configuration }} --json 2>/dev/null | grep '^\[') | tee ${GITHUB_OUTPUT}
- name: install dependencies
shell: bash
run: |
Expand All @@ -24,3 +43,7 @@ runs:
--build missing \
--settings build_type=${{ inputs.configuration }} \
..
- name: upload dependencies to remote
if: (steps.binaries.outputs.missing != '[]') && (steps.remote.outputs.outcome == 'success')
shell: bash
run: conan upload --remote ripple '*' --all --parallel --confirm
7 changes: 6 additions & 1 deletion .github/workflows/macos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,13 @@ jobs:
run : |
conan profile get env.CXXFLAGS default || true
conan profile update 'conf.tools.build:cxxflags+=["-DBOOST_ASIO_DISABLE_CONCEPTS"]' default
- name: dependencies
conan remote remove ripple
- name: build dependencies
uses: ./.github/actions/dependencies
env:
CONAN_URL: http://18.143.149.228:8081/artifactory/api/conan/conan-non-prod
CONAN_USERNAME: ${{ secrets.CONAN_USERNAME }}
CONAN_TOKEN: ${{ secrets.CONAN_TOKEN }}
with:
configuration: ${{ matrix.configuration }}
- name: build
Expand Down
30 changes: 8 additions & 22 deletions .github/workflows/nix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,6 @@ jobs:
cmake --version
env | sort
- name: configure Conan
env:
CONAN_URL: http://18.143.149.228:8081/artifactory/api/conan/conan-non-prod
run: |
conan profile new default --detect
conan profile update settings.compiler.cppstd=20 default
Expand All @@ -73,33 +71,17 @@ jobs:
conan profile update env.CC=${{ matrix.profile.cc }} default
conan profile update env.CXX=${{ matrix.profile.cxx }} default
conan profile update conf.tools.build:compiler_executables='{"c": "${{ matrix.profile.cc }}", "cpp": "${{ matrix.profile.cxx }}"}' default
# Do not quote the URL. An empty string will be accepted (with
# a non-fatal warning), but a missing argument will not.
conan remote add ripple ${{ env.CONAN_URL }} --insert 0
- name: try to authenticate to ripple Conan remote
id: remote
run: |
echo outcome=$(conan user --remote ripple ${{ secrets.CONAN_USERNAME }} --password ${{ secrets.CONAN_TOKEN }} >&2 && echo success || echo failure) | tee ${GITHUB_OUTPUT}
- name: archive profile
# Create this archive before dependencies are added to the local cache.
run: tar -czf conan.tar -C ~/.conan .
- name: list missing binaries
id: binaries
# Print the list of dependencies that would need to be built locally.
# A non-empty list means we have "failed" to cache binaries remotely.
run: |
echo missing=$(conan info . --build missing --settings build_type=${{ matrix.configuration }} --json 2>/dev/null | grep '^\[') | tee ${GITHUB_OUTPUT}
- name: build dependencies
if: (steps.binaries.outputs.missing != '[]')
uses: ./.github/actions/dependencies
env:
CONAN_URL: http://18.143.149.228:8081/artifactory/api/conan/conan-non-prod
CONAN_USERNAME: ${{ secrets.CONAN_USERNAME }}
CONAN_TOKEN: ${{ secrets.CONAN_TOKEN }}
with:
configuration: ${{ matrix.configuration }}
- name: upload dependencies to remote
if: (steps.binaries.outputs.missing != '[]') && (steps.remote.outputs.outcome == 'success')
run: conan upload --remote ripple '*' --all --parallel --confirm
- name: recreate archive with dependencies
if: (steps.binaries.outputs.missing != '[]') && (steps.remote.outputs.outcome == 'failure')
run: tar -czf conan.tar -C ~/.conan .
- name: upload archive
uses: actions/upload-artifact@v3
with:
Expand Down Expand Up @@ -148,6 +130,8 @@ jobs:
uses: actions/checkout@v3
- name: dependencies
uses: ./.github/actions/dependencies
env:
CONAN_URL: http://18.143.149.228:8081/artifactory/api/conan/conan-non-prod
with:
configuration: ${{ matrix.configuration }}
- name: build
Expand Down Expand Up @@ -197,6 +181,8 @@ jobs:
uses: actions/checkout@v3
- name: dependencies
uses: ./.github/actions/dependencies
env:
CONAN_URL: http://18.143.149.228:8081/artifactory/api/conan/conan-non-prod
with:
configuration: ${{ matrix.configuration }}
- name: build
Expand Down
26 changes: 4 additions & 22 deletions .github/workflows/windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,35 +54,17 @@ jobs:
dir env:
- name: configure Conan
shell: bash
env:
CONAN_URL: http://18.143.149.228:8081/artifactory/api/conan/conan-non-prod
run: |
conan profile new default --detect
conan profile update settings.compiler.runtime=MT${{ matrix.configuration == 'Debug' && 'd' || '' }} default
# Do not quote the URL. An empty string will be accepted (with
# a non-fatal warning), but a missing argument will not.
conan remote add ripple ${{ env.CONAN_URL }} --insert 0
- name: try to authenticate to ripple Conan remote
shell: bash
id: remote
run: |
echo outcome=$(conan user --remote ripple ${{ secrets.CONAN_USERNAME }} \
--password ${{ secrets.CONAN_TOKEN }} >&2 && echo success || \
echo failure) | tee ${GITHUB_OUTPUT}
- name: list missing binaries
id: binaries
shell: bash
# Print the list of dependencies that would need to be built locally.
# A non-empty list means we have "failed" to cache binaries remotely.
run: |
echo missing=$(conan info . --build missing --settings build_type=${{ matrix.configuration }} --json 2>/dev/null | grep '^\[') | tee ${GITHUB_OUTPUT}
- name: build dependencies
uses: ./.github/actions/dependencies
env:
CONAN_URL: http://18.143.149.228:8081/artifactory/api/conan/conan-non-prod
CONAN_USERNAME: ${{ secrets.CONAN_USERNAME }}
CONAN_TOKEN: ${{ secrets.CONAN_TOKEN }}
with:
configuration: ${{ matrix.configuration }}
- name: upload dependencies to remote
if: (steps.binaries.outputs.missing != '[]') && (steps.remote.outputs.outcome == 'success')
run: conan upload --remote ripple '*' --all --parallel --confirm
- name: build
uses: ./.github/actions/build
with:
Expand Down
7 changes: 7 additions & 0 deletions BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,13 @@ which allows you to statically link it with GCC, if you want.
conan export external/snappy snappy/1.1.10@
```

Export our [Conan recipe for RocksDB](./external/rocksdb).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we forced to use a custom recipe again? If @Bronek had a branch that updates to a newer version without a custom recipe then that might be preferable, wdyt?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather fix 6.29.5 before updating to 8+. It's easier to backport this recipe than the set of code changes that might require.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @thejohnfreeman - that branch with upgrade to 8.8.1 is not yet ready, and when I think it is ready it might still cause fair amount of hassle (e.g. performance tests) and uncertainty, meaning it would be some time before we can use it.

It does not override paths to dependencies when building with Visual Studio.

```
conan export external/rocksdb rocksdb/6.29.5@
```

Export our [Conan recipe for SOCI](./external/soci).
It patches their CMake to correctly import its dependencies.

Expand Down
27 changes: 27 additions & 0 deletions external/rocksdb/conandata.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
sources:
"6.29.5":
url: "https://github.com/facebook/rocksdb/archive/refs/tags/v6.29.5.tar.gz"
sha256: "ddbf84791f0980c0bbce3902feb93a2c7006f6f53bfd798926143e31d4d756f0"
"6.27.3":
url: "https://github.com/facebook/rocksdb/archive/refs/tags/v6.27.3.tar.gz"
sha256: "ee29901749b9132692b26f0a6c1d693f47d1a9ed8e3771e60556afe80282bf58"
"6.20.3":
url: "https://github.com/facebook/rocksdb/archive/refs/tags/v6.20.3.tar.gz"
sha256: "c6502c7aae641b7e20fafa6c2b92273d935d2b7b2707135ebd9a67b092169dca"
"8.8.1":
url: "https://github.com/facebook/rocksdb/archive/refs/tags/v8.8.1.tar.gz"
sha256: "056c7e21ad8ae36b026ac3b94b9d6e0fcc60e1d937fc80330921e4181be5c36e"
patches:
"6.29.5":
- patch_file: "patches/6.29.5-0001-add-include-cstdint-for-gcc-13.patch"
patch_description: "Fix build with gcc 13 by including cstdint"
patch_type: "portability"
patch_source: "https://github.com/facebook/rocksdb/pull/11118"
- patch_file: "patches/6.29.5-0002-exclude-thirdparty.patch"
patch_description: "Do not include thirdparty.inc"
patch_type: "portability"
"6.27.3":
- patch_file: "patches/6.27.3-0001-add-include-cstdint-for-gcc-13.patch"
patch_description: "Fix build with gcc 13 by including cstdint"
patch_type: "portability"
patch_source: "https://github.com/facebook/rocksdb/pull/11118"
Loading
Loading