[shortfin] Windows builds failing when compiling compound literals

[ScottTodd]

Windows builds of shortfin have been failing since #434

Logs just before: https://github.com/nod-ai/SHARK-Platform/actions/runs/11746567811/job/32726612800
Logs just after: https://github.com/nod-ai/SHARK-Platform/actions/runs/11747537463/job/32729660296

Sample error message:

[313/359] Building CXX object src\shortfin\array\CMakeFiles\shortfin_array_test.dir\dims_test.cc.obj
FAILED: src/shortfin/array/CMakeFiles/shortfin_array_test.dir/dims_test.cc.obj 
C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1441~1.341\bin\Hostx64\x64\cl.exe  /nologo /TP -DSPDLOG_COMPILED_LIB -DSPDLOG_FMT_EXTERNAL -D_SHORTFIN_USING_DYLIB -ID:\a\SHARK-Platform\SHARK-Platform\shortfin\src -ID:\a\SHARK-Platform\SHARK-Platform\shortfin\build\src -ID:\a\SHARK-Platform\SHARK-Platform\shortfin\build\_deps\spdlog-src\include -ID:\a\SHARK-Platform\SHARK-Platform\shortfin\build\_deps\fmt-src\include -ID:\a\SHARK-Platform\SHARK-Platform\shortfin\build\_deps\xtensor-src\include -ID:\a\SHARK-Platform\SHARK-Platform\shortfin\build\_deps\xtl-src\include -ID:\a\SHARK-Platform\SHARK-Platform\iree\runtime\src -ID:\a\SHARK-Platform\SHARK-Platform\shortfin\build\shortfin_iree\runtime\src -external:ID:\a\SHARK-Platform\SHARK-Platform\iree\third_party\googletest\googlemock\include -external:ID:\a\SHARK-Platform\SHARK-Platform\iree\third_party\googletest\googlemock -external:ID:\a\SHARK-Platform\SHARK-Platform\iree\third_party\googletest\googletest\include -external:ID:\a\SHARK-Platform\SHARK-Platform\iree\third_party\googletest\googletest -external:W0 /DWIN32 /D_WINDOWS /EHsc /Ob0 /Od /RTC1 -std:c++20 -MDd -Zi /GL /showIncludes /Fosrc\shortfin\array\CMakeFiles\shortfin_array_test.dir\dims_test.cc.obj /Fdsrc\shortfin\array\CMakeFiles\shortfin_array_test.dir\ /FS -c D:\a\SHARK-Platform\SHARK-Platform\shortfin\src\shortfin\array\dims_test.cc
D:\a\SHARK-Platform\SHARK-Platform\iree\runtime\src\iree/base/alignment.h(241): error C4576: a parenthesized type followed by an initializer list is a non-standard explicit type conversion syntax
D:\a\SHARK-Platform\SHARK-Platform\iree\runtime\src\iree/base/alignment.h(250): error C4576: a parenthesized type followed by an initializer list is a non-standard explicit type conversion syntax
[314/359] Linking C static library shortfin_iree\runtime\src\iree\hal\local\loaders\registration\iree_hal_local_loaders_registration_registration.lib
[315/359] Building CXX object src\shortfin\array\CMakeFiles\shortfin_array_test.dir\array_test.cc.obj
FAILED: src/shortfin/array/CMakeFiles/shortfin_array_test.dir/array_test.cc.obj 

https://stackoverflow.com/questions/33270731/error-c4576-in-vs2015-enterprise suggests that is trying to compile a C source file as C++ while using non-standard compound literals like

  return (iree_page_range_t){
      /*.offset=*/start,
      /*.length=*/end - start,
  };

Diffing the compile_commands.json locally before and after the culprit commit, I see

• the addition of the /GL (Whole program optimization) flag (https://learn.microsoft.com/en-us/cpp/build/reference/gl-whole-program-optimization?view=msvc-170)
• Removing -external:ID for IREE deps
• Adding the -D_SHORTFIN_USING_DYLIB define

For example, shortfin_array_test before (working):

{
  "directory": "D:/dev/projects/SHARK-Platform/shortfin/build",
  "command": "C:\\PROGRA~2\\MICROS~1\\2022\\BUILDT~1\\VC\\Tools\\MSVC\\1441~1.341\\bin\\Hostx64\\x64\\cl.exe  /nologo /TP -DIREE_HAVE_HAL_EXECUTABLE_LOADER_EMBEDDED_ELF=1 -DIREE_HAVE_HAL_EXECUTABLE_LOADER_SYSTEM_LIBRARY=1 -DIREE_HAVE_HAL_EXECUTABLE_LOADER_VMVX_MODULE=1 -DIREE_HAVE_VMVX_MODULE -DIREE_TASK_CPUINFO_DISABLED=1 -DSHORTFIN_HAVE_AMDGPU -DSHORTFIN_HAVE_HOSTCPU -DSPDLOG_COMPILED_LIB -DSPDLOG_FMT_EXTERNAL -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\src -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\src -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\_deps\\spdlog-src\\include -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\_deps\\fmt-src\\include -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\_deps\\xtensor-src\\include -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\_deps\\xtl-src\\include -external:ID:\\dev\\projects\\iree -external:ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\shortfin_iree -external:ID:\\dev\\projects\\iree\\runtime\\src -external:ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\shortfin_iree\\runtime\\src -external:ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\shortfin_iree\\runtime\\src\\iree\\base\\internal\\flatcc -external:ID:\\dev\\projects\\iree\\third_party\\flatcc\\include -external:ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\shortfin_iree\\runtime\\src\\iree\\schemas -external:ID:\\dev\\projects\\iree\\third_party\\hip-build-deps\\include -external:ID:\\dev\\projects\\iree\\third_party\\rccl -external:ID:\\dev\\projects\\iree\\third_party\\googletest\\googlemock\\include -external:ID:\\dev\\projects\\iree\\third_party\\googletest\\googlemock -external:ID:\\dev\\projects\\iree\\third_party\\googletest\\googletest\\include -external:ID:\\dev\\projects\\iree\\third_party\\googletest\\googletest -external:W0 /DWIN32 /D_WINDOWS /EHsc /O2 /Ob1 /DNDEBUG -std:c++20 -MD -Zi -ID:/dev/projects/iree/third_party/flatcc/include/ -ID:/dev/projects/iree/third_party/flatcc/include/flatcc/reflection/ /Fosrc\\shortfin\\array\\CMakeFiles\\shortfin_array_test.dir\\array_test.cc.obj /Fdsrc\\shortfin\\array\\CMakeFiles\\shortfin_array_test.dir\\ /FS -c D:\\dev\\projects\\SHARK-Platform\\shortfin\\src\\shortfin\\array\\array_test.cc",
  "file": "D:\\dev\\projects\\SHARK-Platform\\shortfin\\src\\shortfin\\array\\array_test.cc",
  "output": "src\\shortfin\\array\\CMakeFiles\\shortfin_array_test.dir\\array_test.cc.obj"
},

and after (broken):

{
  "directory": "D:/dev/projects/SHARK-Platform/shortfin/build",
  "command": "C:\\PROGRA~2\\MICROS~1\\2022\\BUILDT~1\\VC\\Tools\\MSVC\\1441~1.341\\bin\\Hostx64\\x64\\cl.exe  /nologo /TP -DSHORTFIN_HAVE_AMDGPU -DSHORTFIN_HAVE_HOSTCPU -DSPDLOG_COMPILED_LIB -DSPDLOG_FMT_EXTERNAL -D_SHORTFIN_USING_DYLIB -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\src -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\src -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\_deps\\spdlog-src\\include -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\_deps\\fmt-src\\include -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\_deps\\xtensor-src\\include -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\_deps\\xtl-src\\include -ID:\\dev\\projects\\iree\\runtime\\src -ID:\\dev\\projects\\SHARK-Platform\\shortfin\\build\\shortfin_iree\\runtime\\src -external:ID:\\dev\\projects\\iree\\third_party\\googletest\\googlemock\\include -external:ID:\\dev\\projects\\iree\\third_party\\googletest\\googlemock -external:ID:\\dev\\projects\\iree\\third_party\\googletest\\googletest\\include -external:ID:\\dev\\projects\\iree\\third_party\\googletest\\googletest -external:W0 /DWIN32 /D_WINDOWS /EHsc /O2 /Ob1 /DNDEBUG -std:c++20 -MD -Zi /GL /Fosrc\\shortfin\\array\\CMakeFiles\\shortfin_array_test.dir\\array_test.cc.obj /Fdsrc\\shortfin\\array\\CMakeFiles\\shortfin_array_test.dir\\ /FS -c D:\\dev\\projects\\SHARK-Platform\\shortfin\\src\\shortfin\\array\\array_test.cc",
  "file": "D:\\dev\\projects\\SHARK-Platform\\shortfin\\src\\shortfin\\array\\array_test.cc",
  "output": "src\\shortfin\\array\\CMakeFiles\\shortfin_array_test.dir\\array_test.cc.obj"
},

Diffing those commands, left is original (working), right is after the change that broke the build:

---

I also considered the C/C++ standard versions - IREE builds with C++17 while shortfin builds with C++20. Compiling iree-run-module upstream with C++20 on MSVC reports no errors, and compiling shortfin with C++17 complains about missing symbols (e.g. std::span, new in C++20).

[stellaraccident]

Thanks. I've half way set up my windows machine for development a few times over the last two weeks but keep getting distracted. This is the sort of thing that is probably an easy fix, but if I'm doing it, I have to put hands on it and fiddle with it to have an insight.

[stellaraccident]

I'm not good at reading msvc flags, but it looks to me like before iree was being treated as a system include and after normal. So this error was probably always there but being suppressed by the leniency of being a system include?

[ScottTodd]

Some clues:

• Windows builds currently default to (and still require?) -DSHORTFIN_BUILD_STATIC=ON -DSHORTFIN_BUILD_DYNAMIC=OFF due to Issues building libshortfin on Windows with -DSHORTFIN_BUILD_DYNAMIC=ON #211
• The USAGE_DEPS in shortfin_public_library are pulling in the include paths for each dep and then applying shortfin's compilation settings...?

  shark-ai/shortfin/src/CMakeLists.txt

  Lines 24 to 41 in 8cd3f85

  	shortfin_public_library(
  	NAME
  	shortfin
  	LINUX_LD_SCRIPT ${CMAKE_CURRENT_SOURCE_DIR}/shortfin.ld
  	COMPONENTS
  	shortfin_array
  	shortfin_local
  	shortfin_support
  	shortfin_systems_factory
  	${_SHORTFIN_LIB_OPTIONAL_COMPONENTS}
  	USAGE_DEPS
  	shortfin_public_defs
  	spdlog::spdlog
  	fmt::fmt
  	xtensor
  	xtl
  	iree_defs
  	)

  shark-ai/shortfin/build_tools/cmake/shortfin_library.cmake

  Lines 19 to 87 in 8cd3f85

  	function(shortfin_public_library)
  	cmake_parse_arguments(
  	_RULE
  	""
  	"NAME;LINUX_LD_SCRIPT"
  	"SRCS;COMPONENTS;USAGE_DEPS"
  	${ARGN}
  	)
  	# Get usage requirements from requested USAGE_DEPS and forward them from the
  	# public library. This is because the contents of the libraries stop at the
  	# public library vs propagating to callers. So we must manually control
  	# the combined usage requirements of the aggregate library.
  	set(_usage_include_directories)
  	set(_usage_compile_definitions)
  	foreach(_usage_dep _shortfin_defs ${_RULE_USAGE_DEPS})
  	get_target_property(_value ${_usage_dep} INTERFACE_INCLUDE_DIRECTORIES)
  	if(_value)
  	list(APPEND _usage_include_directories ${_value})
  	endif()
  	get_target_property(_value ${_usage_dep} INTERFACE_COMPILE_DEFINITIONS)
  	if(_value)
  	list(APPEND _usage_compile_definitions ${_value})
  	endif()
  	endforeach()
  	# Useful for debugging include/definition issues.
  	# message(STATUS "Public library ${_RULE_NAME}: Includes = ${_usage_include_directories}")
  	# message(STATUS "Public library ${_RULE_NAME}: Definitions = ${_usage_compile_definitions}")
  	if(SHORTFIN_BUILD_STATIC)
  	# Static library.
  	shortfin_components_to_static_libs(_STATIC_COMPONENTS ${_RULE_COMPONENTS})
  	add_library("${_RULE_NAME}-static" STATIC ${_RULE_SRCS})
  	target_compile_definitions("${_RULE_NAME}-static" INTERFACE
  	_SHORTFIN_USING_DYLIB
  	${_usage_compile_definitions}
  	)
  	target_include_directories("${_RULE_NAME}-static" INTERFACE ${_usage_include_directories})
  	target_link_libraries(
  	"${_RULE_NAME}-static"
  	PRIVATE ${_STATIC_COMPONENTS}
  	)
  	endif()
  	if(SHORTFIN_BUILD_DYNAMIC)
  	# Dylib library.
  	shortfin_components_to_dynamic_libs(_DYLIB_COMPONENTS ${_RULE_COMPONENTS})
  	add_library("${_RULE_NAME}" SHARED ${_RULE_SRCS})
  	target_compile_definitions("${_RULE_NAME}" INTERFACE
  	_SHORTFIN_USING_DYLIB
  	${_usage_compile_definitions}
  	)
  	target_include_directories("${_RULE_NAME}" INTERFACE ${_usage_include_directories})
  	if(_RULE_LINUX_LD_SCRIPT)
  	target_link_options("${_RULE_NAME}" PRIVATE
  	"$<$<PLATFORM_ID:Linux>:-Wl,--version-script=${_RULE_LINUX_LD_SCRIPT}>"
  	)
  	endif()
  	target_link_libraries(
  	"${_RULE_NAME}"
  	PRIVATE ${_DYLIB_COMPONENTS}
  	)
  	set_target_properties("${_RULE_NAME}" PROPERTIES
  	VERSION ${PROJECT_VERSION_MAJOR}.${PROJECT_VERSION_MINOR}.${PROJECT_VERSION_PATCH}
  	SOVERSION ${SOVERSION}
  	)
  	endif()
  	endfunction()

• More standard DEPS --> target_link_libraries are used in shortfin_cc_component like

  shark-ai/shortfin/src/shortfin/local/CMakeLists.txt

  Lines 9 to 44 in 8cd3f85

  	shortfin_cc_component(
  	NAME
  	shortfin_local
  	HDRS
  	async.h
  	device.h
  	fiber.h
  	messaging.h
  	process.h
  	program.h
  	worker.h
  	scheduler.h
  	system.h
  	SRCS
  	async.cc
  	device.cc
  	fiber.cc
  	messaging.cc
  	process.cc
  	program.cc
  	worker.cc
  	scheduler.cc
  	system.cc
  	COMPONENTS
  	shortfin_support
  	DEPS
  	iree_base_base
  	iree_base_loop_sync
  	iree_hal_hal
  	iree_hal_utils_allocators
  	iree_io_formats_parser_registry
  	iree_modules_io_parameters_parameters
  	iree_modules_hal_hal
  	iree_vm_vm
  	iree_vm_bytecode_module
  	)

[ScottTodd]

Found another warning, also pointing to C vs C++:

[build] D:\dev\projects\iree\runtime\src\iree/task/scope.h(107): warning C4190:
  'iree_task_scope_consume_statistics' has C-linkage specified, but returns
  UDT 'iree_task_dispatch_statistics_t' which is incompatible with C`
[build] D:\dev\projects\iree\runtime\src\iree/task/task.h(494): note: see declaration of
  'iree_task_dispatch_statistics_t'

[ScottTodd]

With iree-org/iree#19346, I can build the shortfin-static target but other targets fail:

[main] Building folder: d:/dev/projects/shark-ai/shortfin/build 
[build] Starting build
[proc] Executing command: "C:\Program Files\CMake\bin\cmake.EXE" --build d:/dev/projects/shark-ai/shortfin/build --config RelWithDebInfo --target shortfin_array_test --
[build] [1/2   0% :: 0.041] Re-checking globbed directories...
[build] [1/1 100% :: 0.240] Linking CXX executable src\shortfin\array\shortfin_array_test.exe
[build] FAILED: src/shortfin/array/shortfin_array_test.exe src/shortfin/array/shortfin_array_test[1]_tests.cmake D:/dev/projects/shark-ai/shortfin/build/src/shortfin/array/shortfin_array_test[1]_tests.cmake 
[build] C:\WINDOWS\system32\cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_exe --intdir=src\shortfin\array\CMakeFiles\shortfin_array_test.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~2\MICROS~1\2022\BUILDT~1\VC\Tools\MSVC\1441~1.341\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\shortfin_array_test.rsp  /out:src\shortfin\array\shortfin_array_test.exe /implib:src\shortfin\array\shortfin_array_test.lib /pdb:src\shortfin\array\shortfin_array_test.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:console  -natvis:D:/dev/projects/iree/runtime/iree.natvis /INCREMENTAL:NO /LTCG && C:\WINDOWS\system32\cmd.exe /C "cd /D D:\dev\projects\shark-ai\shortfin\build\src\shortfin\array && "C:\Program Files\CMake\bin\cmake.exe" -D TEST_TARGET=shortfin_array_test -D TEST_EXECUTABLE=D:/dev/projects/shark-ai/shortfin/build/src/shortfin/array/shortfin_array_test.exe -D TEST_EXECUTOR= -D TEST_WORKING_DIR=D:/dev/projects/shark-ai/shortfin/build -D TEST_EXTRA_ARGS= -D TEST_PROPERTIES= -D TEST_PREFIX= -D TEST_SUFFIX= -D TEST_FILTER= -D NO_PRETTY_TYPES=FALSE -D NO_PRETTY_VALUES=FALSE -D TEST_LIST=shortfin_array_test_TESTS -D CTEST_FILE=D:/dev/projects/shark-ai/shortfin/build/src/shortfin/array/shortfin_array_test[1]_tests.cmake -D TEST_DISCOVERY_TIMEOUT=5 -D TEST_XML_OUTPUT_DIR= -P "C:/Program Files/CMake/share/cmake-3.30/Modules/GoogleTestAddTests.cmake"""
[build] LINK: command "C:\PROGRA~2\MICROS~1\2022\BUILDT~1\VC\Tools\MSVC\1441~1.341\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\shortfin_array_test.rsp /out:src\shortfin\array\shortfin_array_test.exe /implib:src\shortfin\array\shortfin_array_test.lib /pdb:src\shortfin\array\shortfin_array_test.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:console -natvis:D:/dev/projects/iree/runtime/iree.natvis /INCREMENTAL:NO /LTCG /MANIFEST:EMBED,ID=1" failed (exit code 1120) with the following output:
[build] array_test.cc.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: class shortfin::local::Worker & __cdecl shortfin::local::System::init_worker(void)" (__imp_?init_worker@System@local@shortfin@@QEAAAEAVWorker@23@XZ)
[build] array_test.cc.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: class std::shared_ptr<class shortfin::local::Fiber> __cdecl shortfin::local::System::CreateFiber(class shortfin::local::Worker &,class std::span<class shortfin::local::Device * const,-1>)" (__imp_?CreateFiber@System@local@shortfin@@QEAA?AV?$shared_ptr@VFiber@local@shortfin@@@std@@AEAVWorker@23@V?$span@QEAVDevice@local@shortfin@@$0?0@5@@Z)
...
[build] src\shortfin\array\shortfin_array_test.exe : fatal error LNK1120: 50 unresolved externals
[build] ninja: build stopped: subcommand failed.
[proc] The command: "C:\Program Files\CMake\bin\cmake.EXE" --build d:/dev/projects/shark-ai/shortfin/build --config RelWithDebInfo --target shortfin_array_test -- exited with code: 1
[driver] Build completed: 00:00:00.301
[build] Build finished with exit code 1

If I switch to dynamic linking (as Linux does by default) using -DSHORTFIN_BUILD_DYNAMIC=ON -DSHORTFIN_BUILD_STATIC=OFF , I get the same error as on #211:

[build]   Error running test executable.
[build] 
[build]     Path: 'D:/dev/projects/shark-ai/shortfin/build/src/shortfin/array/shortfin_array_test.exe'
[build]     Working directory: 'D:/dev/projects/shark-ai/shortfin/build'
[build]     Result: Exit code 0xc0000135

... progress?

[ScottTodd]

If iree-org/iree#19346 is merged and we get a successful release build tonight, I think we can update shortfin to depend on iree-3.1.0rc20241203 and the Windows build will work again 🤞