Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: Add CMake build #3621

Draft
wants to merge 222 commits into
base: main
Choose a base branch
from
Draft

build: Add CMake build #3621

wants to merge 222 commits into from

Conversation

HuidaeCho
Copy link
Member

@HuidaeCho HuidaeCho commented Apr 18, 2024

This PR replaces #3021. Branched from https://github.com/nilason/grass/tree/cmake_build_work. Thanks @nilason!

Current status: I can compile the entire gisbase using CMake 3.29.1 and GCC 13.2.0. Running grass directly from build/gisbase works fine.

Windows Tests

For my records, message from message(WIN32=${WIN32}, MSVC=${MSVC}, MSYS=${MSYS}, MINGW=${MINGW}) on Windows (all from cmd in ssh):

CMake variables:

TODOs

  • Windows
  • macOS
  • locale
  • addon support
  • ...

Documentation

@Mahesh1998 and I are testing, fixing, and documenting this PR on Ubuntu. https://github.com/Mahesh1998/grass-docs/blob/main/cmake_build_docs/

Building on Windows

  1. Install Visual Studio Community Edition. Select these two components:
    • MSVC v143 - VS 2022 C++ x64/x86 build tools (Latest)
    • Windows 11 SDK (10.0.26100.0)
  2. Install Git for Windows
  3. Install GitHub CLI
  4. Install Miniconda
curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe
Miniconda3-latest-Windows-x86_64.exe /S /D=C:\opt\miniconda
C:\opt\miniconda\condabin\conda.bat init
  1. Start Developer Command Prompt for VS 2022
  2. Setup Conda for GRASS build
conda config --add channels conda-forge
conda config --set channel_priority strict
conda create -n grass cmake pkg-config winflexbison proj libgdal pcre fftw openblas pdal
  1. Download the source code
git clone [email protected]:OSGeo/grass.git
cd grass
gh auth login
gh repo set-default
gh pr checkout 3621
  1. Build!
conda activate grass

mkdir build
cd build
cmake .. > cmake.log 2>&1
msbuild grass.sln > msbuild.log 2>&1

Building on Slackware with Conda packages

Install Miniconda

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod a+x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh -b -u -p ~/usr/local/opt/miniconda
~/usr/local/opt/miniconda/bin/conda init
. ~/.bashrc
conda config --add channels conda-forge
conda config --set channel_priority strict

Create a new environment and install libraries

conda create -n grass proj libgdal fftw blas libpdal wxpython
conda activate grass

Build GRASS

mkdir build
cd build
# ignore warnings about runtime path (Conda libraries will be picked over system libraries)
cmake .. -DCMAKE_PREFIX_PATH=$HOME/usr/local/opt/miniconda/envs/grass
make -j`nproc`

Run GRASS

./output/bin/grass

rkanavath and others added 30 commits June 6, 2023 09:40
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669989265 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669989193 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669989189 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669989180 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669988019 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669987997 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669987899 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669986420 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669986410 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669985337 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669984062 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669984050 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669984037 +0100

parent 464ffca
author Rashad Kanavath <[email protected]> 1497210739 +0200
committer Markus Neteler <[email protected]> 1669984013 +0100

inital work on grass cmake build

wip: cmake fixes

add find iconv script

fix blas error wrong target added

link with dl lib on linux

disable some modules (wip)

search whatever is fftw's inlcude dir

driver lib depends on iconv

fftw in required package

build fftw modules

add python libs

test with py3

install sqlfiles to etc/sql

install lock, clean_temp, echo to ./etc/

install libs

link with dl lib

cleanup cmake code

fix python script & lib install dirs

fixed gisbase for grass startup script

install modules to bin

add PNG as dependency to r.out.png

install __init__.py

wrong install dir for python api

install with rpath to avoid LD_LIBRARY_PATH

whitespace fixes

MSVC: check for _WIN32 to use msvc and mingw32

MSVC: disable some programs temporarily

update cmake scripts to manage thirdparty libs

disable X11 on windows

update generation of config.h

fix list of cmake depends

msvc compile fixes

use INIFINITY rather than gcc only division by zero

ignore visual studio, cmake files

use INFINITY rather than divide by zero

use _WIN32 for mingw32 and msvc

add ssize_t for msvc

msvc add _USE_MATH_DEFINES an d export dll

keep check for long long int in cache

include driver/init.c when building display drivers

use _WIN32 to build msvc and mingw32

update cmake for lib/db

link with libm only on unix

msvc: skip chmod on windows msvc

use INFINITY rather than diivide by zero

add msvc specific headers: unistd.h, dirent.h

grass moved to git... So follow that in cmake.

fix check for HAVE_LONG_LONG

include math.h only on msvc. To be discussed

void* arithmetic (GCC extension) is not allowed in msvc

disable d.mon, d.font on msvc. TBD

missing O_ACCMODE on msvc

Add unistd.h on windows

Source copied from http://stackoverflow.com/a/826027

dbmi_base: add dirent.h and dirent.c for msvc

use macro INFINITY rather than GCC divide by zero

MSVC: fix missing strings.h

use math defines on msvc

update find package of fftw

added required defines into parent scope

update setting of HAVE_* defines. 1/n

use cmake target check if possible

add odbc target on windows

update values in generated config.h

reorganize cmake option defaults, cmake flags, macros

cmake c and cxx flags include _CRT_SECURE_NO_WARNINGS to avoid a lot msdn warnings
check_target macro is a helper to reduce number of lines in include/CMakeLists.txt

simlib: min, max are already available

Find which C standard library does not provide them
And if there is one, see if grass support that compiler/platform combination
To be discussed

CMake: reorganize cmake functions, macros, find_scripts (Like a Pro)

make_script_in_dir is renamed to build_script_in_subdir

I still don't the naming of macro is clear.
One without knowledge of CMake, AutoMake or any
other build scripting  experience must be able to deduct
what is the macro/functions.

Even if there will be a header comment with 'PURPOSE' field.

fix define used in lib/rst/interp_float

fix tools/timer build

add gettimeofday windowss implemenation from postgresql

CMake: reorganize cmake functions (2)

MSVC: copy UINT64CONST(x) from postgresql

WIP: disable pyc generation with g.parser on msvc

fix ccmath complex struct for msvc

take out GCC'sim and stick to C standard

copy external/ccmath/ccmath.h to include/grass/ccmath_grass.h

CMake: enable cmake export all symbols on for shared libs

CMake: build lib/python on MSVC

fix ctypesgen.cmake
reorg cmake functions used by python
add new cmake function to copy_python_files and compile

MSVC: include sys/time.h if available or time.h

CMake: remove debug trace

CMake: minor cleanup

CMake: make gui/images

CMake: fix run_grass of locale_strings and ctypesgen

CMake: link with postgres is optional

gid_t, uid_t, pid_t exists on *nix

use find_library_suffix to switch between debug and rest

avoid touch  .stamp files and depend on generated pyc file

CMake: update definiition of HAVE_PROJ_H

include winsock.h only on msvc

CMake: remove annoying logs

for fix TODO for g.version

CMake: depend on pyc file not on a new .stamp file

CMake: fix syntax error

CMake: fix sqlite include dir variable

CMake: detect version of proj4 before activating defines

MSVC: avoid pulling min max on windows

CMake: update list of enabled modules

TODO r.watershed, fix cmake proj4 library varname

Revert "simlib: min, max are already available"

This reverts commit ab2b961.

CMake: update proj library variables (remove suffix 4)

use macro INFINITY

MSVC: fix gisinit initialized flag

export initialized using dllexport when building.
export macros has been generated by CMake's GenerateExportHeader.
cmake calls this for all libraries in build_module function.

There is a change in initialized variable decl on msvc and rest of compilers.
This point has to be discussed with other devs

MSVC: fix min, max macro stuff

update TODO, list of modules not built by cmake

CMake: update find scripts to find debug then release

update list of options.
WITH_PYTHON addded to build python bindings (default is OFF)

CMake: zlib is check with cmake target

demolocation is configured in lib/init/CMakeLists.txt

CMake: generate grass.py and grass.sh scripts (build tree)

CMake: fix startup script generation (install tree)

fix generation of startup scripts

fix g.list building 1/2

MSVC: missing regex, use PCRE (wip)

lib/init: fix startup script on linux

fix input configure_file

activate building g.remove on msvc

add cmake messages for lib/init/

include sys/time.h if not on msvc.

As we don't include grass/config.h we cannot simply check against
HAVE_SYS_TIME_H

install proj data files

geotiff_csv only required in windows

check this with devs

wip: add compile defs via interface library

fix g.version, grass_gproj with proj6

create startup shell scripts in bin

fix startup script

MSVC: configure run_grass.bat, run_python.bat

generate html docs

fix copy_python_file (used in gui/wxpython, lib/python)

build all gui/wxpython modules

wrapper scripts to build html docs

skip html-description for g.parser

update msvc target properties

fix install directory for running from binary tree

install tools for buildtree and installtree

fix build docs using cmake POST_BUILD

update mkhtml.py for cmake

copy header to binary directory using add_custom_command

install extra files in lib/gis using post_build

add copy_header as depdendency to grass_datetime

first install tools directory

find cairo debug and then release libs

add POST_BUILD target for documentation only if WITH_DOCS

build gui/wxpython, fix html description generation

MSVC: uninitialized variable

CMake: update to work with autoconf and cmake

CMake: install et copy gui/images gui/icons

CMake: fix grass version date

CMake: fix building gui (python files, docs, html)

fix dist include dir name

build docs only if requested

fix typo

fix cmake syntax errors

cmake linux fixes

check for _WIN32 define to work with msvc

fix newline at end of file

WIP: update helper cmake scripts

update copy_header target

use gisbase as dist directory for build tree

cmake: fix build for db/drivers

ignore __pycache__ directory when scanning for .py files

wip: use a gisbase as dist directory

WIP: temporary fix for find_library output variable

CMake: move wxpython cmake codes to gui/wxpython

raise ScriptError

wip wip build docs

minor cleanup

install html docs for driver db

update building html docs (wip)

add missing dependencies for v.lrs

update building python modules (wip)

use target property to check if running python script (docs)

cleanup cmake helper functions (exe, libs, python, docs)

copy strings.h and unistd.h on msvc

python files (target) depend only grass.script if building docs

fix build docs for windows and linux (wip)

seperate list of g.gui.* modules

try to make generic build docs (wip)

build docs html for not win32 (wip)

fix cmake syntax error

update pgm extension for running html description

remove temp files after docs are finished

CMake: fix IN_LIST syntax

ficx cmake syntax error

fix again linux html description for python with a main script

wip: include from config build is breaking msvc

msvc: disable db drivers (wip)

add test.raster3d.lib into NO_HTML_DESCR_TARGETS

CMake: missing endif()

x extension on running html descr

copy r.in.wms directory to etc/

fix building py modules

add grass dll directory to path windows grass.bat

debug linux build failures

cmake missing endif

set main script file only for those selected modules

cmake: fix python docs for linux and windows

create scripts directory in gisbase

cmake debug message

add missing include

add cmake find scripts for liblas, netcdf, postgresql

update proj4 detection to support 4.x, 5.x , 6.x versions

update cmake functions to build grass modules

add proj4 version defines (support 5.x+)

add options for v.in.dwg and liblas modules

activate build of modules deactivated

detecting of new 3rdd party libraries

fix i.landsat.acca on msvc

add dll export macro for iostream, dspf, calc

fix r.terraflow on msvc

use _WIN32 rather than __MINGW32__ for msvc

use infinity macro to build on msvc

missing include on msvc

copy VERSIONUMBER and license to gisbase/etc

support for multiple proj4 version

update building lib/python (except ctypes)

use INFINITY macro to work with msvc

fix building gui/wxpython/xml

use approach with cmake env command for cross platform build

void* arithmetic is not allowed in msvc

To be discussed

copy __init__.py for python/grass/

fix wrong cmake varible used

copy init py to etc/python/grass/

db/drviers: odbc, sqlite, dbf,  ogr, postgresql

check for PQCmdTuples in postgresql

add defines to be posix conformat on msvc

bring in testing using ctest (wip)

improve proj4 detection

Signed-off-by: Rashad Kanavath <[email protected]>

keep proj4 version string in cache (very useful later)

find optional packages quietly

use PRIMARY_DEPENDS option in build_module

generate wxpython menu xml stuff after building all executables

ogsf and nviz depends on grass_raster

update gui/wxpython build

add missing modules to build

missing v.clip

install html file is exists

cmake cleanup

reorder dependencies of gui/wxpython modules

avoid breakage in autconf build

missing file copy

fix mkhtml doc building, exe, lib, python, gui

fix find scripts on windows

fix always out of date for custom targets

fix missing optional dependds to grass gis library

fix missing math.h include

wrong path used under cmake binary directory

keep autoconf build  conflict with cmake

.bat files must be in scripts/

fix installation of gui/images, gui/icons

Revert "install html file is exists"

This reverts commit 9e83f6f.

Apply suggestions from code review

trivial changes (comment style) applied

moved tools/ -> utils/

sync to main

sync to main

fix indentation

remove trailing white space

remove trailing white space

revert C related INFINITY/NAN changes (taken care of in OSGeo#2681)
Attempt to add a CI workflow
Co-authored-by: Edouard Choinière <[email protected]>
Co-authored-by: Edouard Choinière <[email protected]>
Co-authored-by: Edouard Choinière <[email protected]>
Co-authored-by: Edouard Choinière <[email protected]>
Co-authored-by: Edouard Choinière <[email protected]>
Co-authored-by: Edouard Choinière <[email protected]>
Co-authored-by: Edouard Choinière <[email protected]>
Co-authored-by: Edouard Choinière <[email protected]>
Co-authored-by: Edouard Choinière <[email protected]>
@echoix
Copy link
Member

echoix commented Jan 18, 2025

#4665 changed some rasterdeps in grass.make, to include parson.

@HuidaeCho
Copy link
Member Author

HuidaeCho commented Feb 12, 2025

I'm testing @Mahesh1998's commits for Windows. Once this test is done, I'll clean and move this document to the top cell.

Building on Windows

  1. Install Visual Studio Community Edition
  • Select Desktop development with C++: It will select C++ core desktop features and the following optional components:
    • MSVC v143 - VS 2022 C++ x64/x86 build tools (Latest)
    • C++ ATL for latest v143 build tools (x86 & x64)
    • C++ Build Insights
    • Just-In-Time debugger
    • C++ profiling tools
    • C++ CMake tools for Windows ---> I think we need this
    • Test Adapter for Boost.Test
    • Test Adapter for Google Test
    • Live Share
    • IntelliCode
    • C++ AddressSanitizer
    • Windows 11 SDK (10.0.22621.0) ---> Maybe?
    • vcpkg package manager
    • GitHub Copilot
  • I unchecked all the optional components and found these were selected:
    • Text Template Transformation
    • C# and Visual Basic Roslyn compilers
    • C++ 2022 Redistributable Update ---> I guess we need this?
    • MSBuild ---> We need this
    • C++ core features (2.49GB) ---> We need this: When I tried to install just this component, I got a popup message "Do you want to continue without workloads? The core editor can open folders, edit files, and manage work items. The product needs to install additional components to create, open, and debug projects."
      • I just installed this component and MSBuild was still installed... CMake was not available, so I went back and installed C++ CMake tools for Windows. CMake selected MSVC v143 - VS 2022 C++ x64/x86 build tools (Latest) automatically.
      • BUT, we had to use Conda CMake because MSVC CMake couldn't find Conda libraries.
      • Well, a very simple helloworld using stdio.h didn't compile. stdio.h is part of the Windows Universal CRT SDK.
      • Now, I got LINK : fatal error LNK1104: cannot open file 'kernel32.lib'; I installed Windows 11 SDK (10.0.26100.0).
      • This component installed a newer version of stdio.h in a different folder; I uninstalled the Windows Universal CRT SDK.
      • I didn't need C++ core features; uninstalled.

Installed components

  • MSVC v143 - VS 2022 C++ x64/x86 build tools (Latest)
  • Windows 11 SDK (10.0.26100.0)
  1. Install Git for Windows
  2. Install GitHub CLI
  3. Install Miniconda
  • Add paths to C:\ProgramData\miniconda3 and C:\ProgramData\miniconda3\Scripts
  1. Start Developer Command Prompt for VS 2022
conda init
conda config --add channels conda-forge
conda config --set channel_priority strict

conda create -n grass

git clone [email protected]:OSGeo/grass.git
cd grass
gh auth login
gh repo set-default
gh pr checkout 3621

mkdir build
cd build
conda activate grass
conda install cmake winflexbison proj libgdal pcre fftw openblas pdal
cmake ..
msbuild grass.sln > msbuild.log 2>&1

@echoix
Copy link
Member

echoix commented Feb 12, 2025

Conda cmake va msvc cmake,... is it just a question of what "preset" is selected/detected (group of compilers and paths)? It shouldn't change, as they should be the same.
Cmake can even be installed from pip if you want

Anyways nice and exiting progress!

@HuidaeCho
Copy link
Member Author

HuidaeCho commented Feb 12, 2025

Conda cmake va msvc cmake,... is it just a question of what "preset" is selected/detected (group of compilers and paths)? It shouldn't change, as they should be the same. Cmake can even be installed from pip if you want

Yes. It's the "preset". Without any modifications or arguments, MSVC CMake wouldn't recognize Conda libraries. I didn't spend too much time on this, but if you know how, I can try MSVC CMake again.

Anyways nice and exiting progress!

Yes, it is! Unfortunately, one module (d.colorlist? r.colors?) failed in @Mahesh1998's test because stdout/stderr didn't work (???) and this module relies on stdout to produce color icons (or something like that...). I'm trying his PR myself to fix this issue. @Mahesh1998 did an awesome job, but he is graduated and not with us anymore :-(.

@HuidaeCho
Copy link
Member Author

HuidaeCho commented Feb 12, 2025

The log says 106 warnings and 6 errors, but I found 186 errors:

  • 4 C1083 (cannot open include file): cannot find OpenBLAS header files; @Mahesh1998 used blas-devel
    • Related answer and, in fact, Conda FindBLAS.cmake and FindLAPACK.cmake don't find header files at ~/.conda/envs/grass/Library/include/openblas/(cblas|lapacke).h
    • Without a dirty hack, just conda install pkg-config
  • 4 LNK1120 (fatal error unresolved externals): 2 from r.colors and 2 from r3.colors both because of LNK2019 unresolved external symbol print_json_colors
  • 4 LNK2019 (unresolved external symbol): 2 from r.colors and 2 from r3.colors
  • 174 MSB3073 (command exited with an error code)

@echoix
Copy link
Member

echoix commented Feb 12, 2025

I'm wondering why you had to remove the urct. From what I remember, the ucrt was made to be more standards-compatible, instead of sticking with the older choices made. Was the build harder to do with it?

@HuidaeCho
Copy link
Member Author

I'm wondering why you had to remove the urct. From what I remember, the ucrt was made to be more standards-compatible, instead of sticking with the older choices made. Was the build harder to do with it?

That was because Windows 11 SDK included a newer version of UCRT. From this article, I think UCRT is part of Windows 11 SDK.

#ifdef __MINGW32__
#ifdef _WIN32
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a problem for the Autotools?

-e 's#@GISBASE@#$(RUN_GISBASE)#' \
-e 's#@SET_GISBASE@#$(RUN_GISBASE)#' \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't find SET_GISBASE in the Autotools code, is that an issue?

Comment on lines -399 to +400
for (int row = 0; row < rows; row++) {
for (row = 0; row < rows; row++) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This replaces variable in the scope of the for loop by a variable in the scope outside of the parallel block. This does not look right.

@HuidaeCho
Copy link
Member Author

HuidaeCho commented Feb 16, 2025

\vector\v.univar\main.c(77,8): error C2099: initializer is not a constant [C:\Users\hcho\usr\grass\grass\build\vector\v.univar.vcxproj]

Recent MSVC versions don't treat NAN as constant, workaround?

A fix will be released according to Microsoft (Feb 13, 2025 6:14 PM).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C Related code is in C C++ Related code is in C++ CI Continuous integration database Related to database management display docs general GUI wxGUI related imagery libraries misc module Python Related code is in Python raster Related to raster data processing raster3d temporal Related to temporal data processing translation Message translation related vector Related to vector data processing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants