[TF] Update TF v2.16.1

[smuzaffar]

Update tensorflow version to 2.16.1

[cmsbuild]

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_14_1_X/tf.

@smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[smuzaffar]

please test for CMSSW_14_1_TF_X

[cmsbuild]

Pull request #9241 was updated.

[smuzaffar]

please test for CMSSW_14_1_TF_X

[cmsbuild]

Pull request #9241 was updated.

[iarspider]

please test for CMSSW_14_1_TF_X

[cmsbuild]

Pull request #9241 was updated.

[iarspider]

please test for CMSSW_14_1_TF_X/el8_amd64_gcc12

[iarspider]

please test for slc7_amd64_gcc12

[cmsbuild]

Pull request #9241 was updated.

[iarspider]

please test for CMSSW_14_2_TF_X/el8_amd64_gcc12

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-971011/41481/summary.html
COMMIT: d400a6d
CMSSW: CMSSW_14_2_X_2024-09-11-2300/slc7_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9241/41481/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/tensorflow/core/lib/png/_objs/png_io/png_io.pic.d '-frandom-seed=bazel-out/k8-opt/bin/tensorflow/core/lib/png/_objs/png_io/png_io.pic.o' -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' '-DBAZEL_CURRENT_REPOSITORY=""' -iquote . -iquote bazel-out/k8-opt/bin -iquote external/local_tsl -iquote bazel-out/k8-opt/bin/external/local_tsl -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/ml_dtypes -iquote bazel-out/k8-opt/bin/external/ml_dtypes -iquote external/nsync -iquote bazel-out/k8-opt/bin/external/nsync -iquote external/png -iquote bazel-out/k8-opt/bin/external/png -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -Ibazel-out/k8-opt/bin/external/ml_dtypes/_virtual_includes/float8 -Ibazel-out/k8-opt/bin/external/ml_dtypes/_virtual_includes/int4 -isystem external/eigen_archive/include/eigen3 -isystem bazel-out/k8-opt/bin/external/eigen_archive/include/eigen3 -isystem external/ml_dtypes -isystem bazel-out/k8-opt/bin/external/ml_dtypes -isystem external/ml_dtypes/ml_dtypes -isystem bazel-out/k8-opt/bin/external/ml_dtypes/ml_dtypes -isystem external/nsync/public -isystem bazel-out/k8-opt/bin/external/nsync/public -isystem external/png/include -isystem bazel-out/k8-opt/bin/external/png/include -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIC -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -fno-omit-frame-pointer -no-canonical-prefixes -fno-canonical-system-headers -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections -Wno-all -Wno-extra -Wno-deprecated -Wno-deprecated-declarations -Wno-ignored-attributes -Wno-array-bounds -Wunused-result '-Werror=unused-result' -Wswitch '-Werror=switch' '-Wno-error=unused-but-set-variable' -DAUTOLOAD_DYNAMIC_KERNELS '-march=x86-64-v3' '-mcpu=native' '-mtune=native' -Wno-sign-compare '-std=c++17' '-std=c++17' -c tensorflow/core/lib/png/png_io.cc -o bazel-out/k8-opt/bin/tensorflow/core/lib/png/_objs/png_io/png_io.pic.o)
# Configuration: f8590474c4941faabde7e32c6ee2839f708ac77e27c71a1cc3a3ecbce293003c
# Execution platform: @local_execution_config_platform//:platform
In file included from external/zlib/zlib.h:34,
from tensorflow/core/lib/png/png_io.cc:20:
external/zlib/zlib.h:1758:44: error: 'va_list' has not been declared
1758 | ZEXTERN int            ZEXPORTVA gzvprintf Z_ARG((gzFile file,
|                                            ^~~~~
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 1388.736s, Critical Path: 206.04s
INFO: 12025 processes: 8012 internal, 4013 local.


[cmsbuild]

Pull request #9241 was updated.

[iarspider]

please test for slc7_amd64_gcc12

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-971011/41485/summary.html
COMMIT: fe44e3b
CMSSW: CMSSW_14_2_X_2024-09-11-2300/slc7_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9241/41485/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @cms-sw TF2.15: Apply abseil aarch64 patch #9341
• @cms-sw Fix bazel recipe #9336

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-971011/41485/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-971011/41485/git-merge-result

Comparison Summary

Summary:

• You potentially removed 676 lines from the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 60835 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 3331158
• DQMHistoTests: Total failures: 482987
• DQMHistoTests: Total nulls: 334
• DQMHistoTests: Total successes: 2847817
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 6.403 KiB( 43 files compared)
• DQMHistoSizes: changed ( 13034.0 ): 5.829 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 141.042 ): 0.008 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 141.044 ): -0.035 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 250202.181 ): -0.059 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 25202.0 ): 0.660 KiB SiStrip/MechanicalView
• Checked 193 log files, 163 edm output root files, 44 DQM output files
• TriggerResults: found differences in 14 / 42 workflows

[smuzaffar]

@iarspider , is this ready to go in for TF IBs?

[iarspider]

@smuzaffar yes

[smuzaffar]

+externals

lets get this in TF_X IBs

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_14_2_X/tf IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @rappoccio, @antoniovilela, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)