Conan instructions for building with external XMPsdk are broken

[kevinbackhouse]

The instructions for "Building Exiv2 with Adobe XMPsdk 2016" in README-CONAN.md have stopped working. I am seeing this error:

$ conan install .. --options xmp=True --build missing
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=10
os=Linux
os_build=Linux
[options]
xmp=True
[build_requires]
[env]

XmpSdk/2016.7@piponazo/stable: Not found in local cache, looking in remotes...
XmpSdk/2016.7@piponazo/stable: Trying with 'conan-center'...
XmpSdk/2016.7@piponazo/stable: Trying with 'conan-piponazo'...
ERROR: Failed requirement 'XmpSdk/2016.7@piponazo/stable' from 'conanfile.py'
ERROR: Permission denied for user: 'None'. [Remote: conan-piponazo]

[clanmills]

On macOS, I also get this message on a "vanilla" build (no XMPsdk)

727 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/conan-build $ conan install .. --build missing
Configuration:
[settings]
arch=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=11.0
os=Macos
[options]
[build_requires]
[env]

Expat/2.2.6@pix4d/stable: WARN: Package is corrupted, removing folder: /Users/rmills/conanData/Expat/2.2.6/pix4d/stable/package/ed249b6399b9d89057513dc66f87ca628874eec2
conanfile.py: Installing package
Requirements
    Expat/2.2.6@pix4d/stable from 'conan-center' - Cache
    gtest/1.8.1@bincrafters/stable from 'conan-center' - Cache
    zlib/1.2.11@conan/stable from 'conan-center' - Cache
Packages
    Expat/2.2.6@pix4d/stable:ed249b6399b9d89057513dc66f87ca628874eec2 - Build
    gtest/1.8.1@bincrafters/stable:09e99ac43439f7364d9a601827edb187f83f9bcc - Build
    zlib/1.2.11@conan/stable:f74366f76f700cc6e991285892ad7a23c30e6d47 - Cache

Installing (downloading, building) binaries...
Expat/2.2.6@pix4d/stable: WARN: Build folder is dirty, removing it: /Users/rmills/conanData/Expat/2.2.6/pix4d/stable/build/ed249b6399b9d89057513dc66f87ca628874eec2
Expat/2.2.6@pix4d/stable: Copying sources to build folder
Expat/2.2.6@pix4d/stable: Building your package in /Users/rmills/conanData/Expat/2.2.6/pix4d/stable/build/ed249b6399b9d89057513dc66f87ca628874eec2
Expat/2.2.6@pix4d/stable: Generator cmake created conanbuildinfo.cmake
Expat/2.2.6@pix4d/stable: Calling build()
-- The C compiler identification is AppleClang 12.0.5.12050022
-- The CXX compiler identification is AppleClang 12.0.5.12050022
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Conan: called by CMake conan helper
-- Conan: called inside local cache
-- Conan: Adjusting output directories
-- Conan: Using cmake global configuration
-- Conan: Adjusting default RPATHs Conan policies
-- Conan: Adjusting language standard
CMake Error at /Users/rmills/conanData/Expat/2.2.6/pix4d/stable/build/ed249b6399b9d89057513dc66f87ca628874eec2/conanbuildinfo.cmake:395 (message):
  Detected a mismatch for the compiler version between your conan profile
  settings and CMake:

  Compiler version specified in your conan profile: 11.0

  Compiler version detected in CMake: 12.0

  Please check your conan profile settings (conan profile show
  [default|your_profile_name])
Call Stack (most recent call first):
  /Users/rmills/conanData/Expat/2.2.6/pix4d/stable/build/ed249b6399b9d89057513dc66f87ca628874eec2/conanbuildinfo.cmake:491 (conan_error_compiler_version)
  /Users/rmills/conanData/Expat/2.2.6/pix4d/stable/build/ed249b6399b9d89057513dc66f87ca628874eec2/conanbuildinfo.cmake:574 (check_compiler_version)
  /Users/rmills/conanData/Expat/2.2.6/pix4d/stable/build/ed249b6399b9d89057513dc66f87ca628874eec2/conanbuildinfo.cmake:135 (conan_check_compiler)
  CMakeLists.txt:8 (conan_basic_setup)


-- Configuring incomplete, errors occurred!
See also "/Users/rmills/conanData/Expat/2.2.6/pix4d/stable/build/ed249b6399b9d89057513dc66f87ca628874eec2/build/CMakeFiles/CMakeOutput.log".
See also "/Users/rmills/conanData/Expat/2.2.6/pix4d/stable/build/ed249b6399b9d89057513dc66f87ca628874eec2/build/CMakeFiles/CMakeError.log".
Expat/2.2.6@pix4d/stable: 
Expat/2.2.6@pix4d/stable: ERROR: Package 'ed249b6399b9d89057513dc66f87ca628874eec2' build failed
Expat/2.2.6@pix4d/stable: WARN: Build folder /Users/rmills/conanData/Expat/2.2.6/pix4d/stable/build/ed249b6399b9d89057513dc66f87ca628874eec2
ERROR: Expat/2.2.6@pix4d/stable: Error in build() method, line 45
	cmake.configure(source_dir="../libexpat/expat", build_dir="build", defs=cmake_args)
	ConanException: Error 1 while executing cd 'build' && cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE="Release" -DCONAN_IN_LOCAL_CACHE="ON" -DCONAN_COMPILER="apple-clang" -DCONAN_COMPILER_VERSION="11.0" -DCONAN_CXX_FLAGS="-m64" -DCONAN_SHARED_LINKER_FLAGS="-m64" -DCONAN_C_FLAGS="-m64" -DBUILD_SHARED_LIBS="OFF" -DCMAKE_INSTALL_PREFIX="/Users/rmills/conanData/Expat/2.2.6/pix4d/stable/package/ed249b6399b9d89057513dc66f87ca628874eec2" -DCMAKE_INSTALL_BINDIR="bin" -DCMAKE_INSTALL_SBINDIR="bin" -DCMAKE_INSTALL_LIBEXECDIR="bin" -DCMAKE_INSTALL_LIBDIR="lib" -DCMAKE_INSTALL_INCLUDEDIR="include" -DCMAKE_INSTALL_OLDINCLUDEDIR="include" -DCMAKE_INSTALL_DATAROOTDIR="share" -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY="ON" -DCONAN_EXPORTED="1" -Wno-dev -DBUILD_doc="OFF" -DBUILD_examples="OFF" -DBUILD_shared="False" -DBUILD_tests="OFF" -DBUILD_tools="OFF" -DCMAKE_POSITION_INDEPENDENT_CODE="ON" -DCMAKE_DEBUG_POSTFIX="" -DMSVC_USE_STATIC_CRT="False" '../libexpat/expat'
728 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/conan-build $

[piponazo]

Hi guys.

The XMP conan support was a very fragile prototype. It looks like with recent conan updates, it is not building anymore. (Note that it was in a repository named conan-piponazo which I created few years ago). I hope I can find some spare time to look at it anyways.

The Expat package is an official conan package and the problem sees to be with a wrong configuration of conan profile and default compiler used in the system:

CMake Error at /Users/rmills/conanData/Expat/2.2.6/pix4d/stable/build/ed249b6399b9d89057513dc66f87ca628874eec2/conanbuildinfo.cmake:395 (message):
  Detected a mismatch for the compiler version between your conan profile
  settings and CMake:

  Compiler version specified in your conan profile: 11.0

  Compiler version detected in CMake: 12.0

Once that mismatch is corrected, it should work.

[clanmills]

@piponazo Thanks for looking at that, Luis.

I hope everything's good with you. We're fine. I've been sucked back into Exiv2 for a few days. I hope to escape again and return to my garden projects. @kevinbackhouse intends to tag Exiv2 v0.27.5 RC1 on the 0.27-maintenance branch in the next few days. This will include several security fixes and some minor changes. No substantial changes. I've agreed to build the release and update the web-site for v0.27.5 GM in September.

You're right about compiler version 11/12 mismatch with expat on macOS. I've updated conan $ sudo pip install conan --upgrade and edited ~/conan/profiles/default. I think expat and gtest have built. Sadly, the build dies a horrible death while compiling gzwrite.c in zlib. I don't think it's worth spending time on this. I've got a new MacBookPro with the M1 chip. conan is building -arch x86_64, so the build is suspect. I believe the CI successfully builds using conan on macOS with an Intel processor. So my conan/zlib trouble is probably local to my machine.

I hope you'll find time to investigate the XMP/conan puzzle. It's not urgent and isn't necessary for v0.27.5.

[clanmills]

Good News. I've performed a "vanilla build" on the MacMini with the Intel CPU. I had to read https://docs.conan.io/en/latest/faq/troubleshooting.html#error-invalid-setting and update ~/.conan/settings.yml to include apple-clang: version 12.0.

I don't think there's any value in discussing the zlib/gzwrite.c mystery on macOS/M1. However, this does reinforce my opinion that we should not exclusively require conan to build. It's very desirable to use the package manager to install dependencies and cmake can use them without conan. This is discussed in #1811

The situation with xmp remains as reported by Kev.

522 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance/conan_build $ conan install .. --build missing --options xmp=True
Configuration:
[settings]
arch=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=12.0
os=Macos
[options]
xmp=True
[build_requires]
[env]

XmpSdk/2016.7@piponazo/stable: ERROR: Error downloading binary package: 'XmpSdk/2016.7@piponazo/stable:b01a0ba71e5055bb491d9994ac256b1070bd5fea'
ERROR: Permission denied for user: 'None'. [Remote: conan-piponazo]
523 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance/conan_build $ 

[piponazo]

With all the recent changes in conan repositories, we would need to do 2 things here:

• Revisit the conan-xmpsdk conan recipe I created here (https://github.com/piponazo/conan-xmpsdk). Before, that recipe was downloading a tarball from a adobe resource location ( http://download.macromedia.com/pub/developer/xmp/sdk/XMP-Toolkit-SDK-CC201607.zip). Nowadays, it seems that adobe has a github repository for xmpsdk: https://github.com/adobe/XMP-Toolkit-SDK/ , and their latest tag is v2021.10.
• It would be good to try to make that conan recipe stable enough so that we could contribute it to the official/stable conan-center repository, instead of having it in another conan repository (as we had before).

@clanmills How would you rate the importance of using a recent version of the XMPSDK? Unless is critical for accessing/manipulating some tags, I would leave this issue for somebody else (someone who might want to learn more about conan, and I would be happy to guide that contributor).

[clanmills]

@piponazo One of the objectives for v1.00 is to make XMPsdk an external library and remove the xmpsdk/ directory from the code-base. #1466 (comment)

I know very little about XMPsdk. Andreas offered the opinion "everything concerning XMPsdk is a pain in the butt". I suspect that the xmpsdk/ directory was added to get it to build and link with libexiv2. As you know, the Adobe XMPsdk is built using scripts which use CMake. I suspect that Andreas could make no sense of that because the exiv2 build at that time used autotools.

Now that we have adopted CMake/conan as our only build system, it would be wonderful to remove the xmpsdk/ directory entirely and have an option such as -DEXIV2_ENABLE_XMPSDK=ON|OFF and possibly -DEXIV2_XMPSDK_VERSION=something

I don't know anybody at Adobe involved with the support/development of XMPsdk. I hope they welcome your contribution, however I suspect you will get very little cooperation. Looking at the the log of GitHub issues, XMPsdk does not appear to be actively maintained. https://github.com/adobe/XMP-Toolkit-SDK I have asked a couple of questions on the Adobe support forum about the XMPsdk and they were not answered.

[kevinbackhouse]

Another option would be to include XMP-Toolkit-SDK as a submodule. I wish they were more responsive to feedback though. I sent them a pull request months ago to fix a denial of service issue, but they have ignored it. Also, the last time I tried, I couldn't get their build to work.

[clanmills]

@kevinbackhouse I saw your PR concerning the DOS threat in XMPsdk. You're right. The support and maintenance of XMPsdk is disappointing.

Both Luis and I were surprised (== shocked) by the build-system that Adobe provide for XMPsdk. They have a few shell/batch scripts that use CMake to build the code. The build-system expects the source of third party libraries (expat and zlib, I think) to be available in specific locations in the file-system.

There is another project https://github.com/hfiguiere/exempi which is based on the Adobe code and has a build system based on autotools. I suspect adopting exempi would be changing the bed of nails.