Compiling

Kynema is developed in C++20 and is designed to be buildable on any system with a compliant compiler. It utilizes Kokkos to ensure performance portability, allowing it to run on any platform supported by these projects. We strive to test Kynema on a wide range of platforms, including Linux and macOS, although it is not feasible to cover every possible configuration. This document outlines the build procedure verified to work on Linux (RHEL8). For additional assistance tailored to your specific setup, please contact the developers.

Spack Installation

The easiest way to use Kynema is through the Spack package manager. Once you have downloaded and set up Spack for your environment, simply run

spack install kynema

To see the latest list of supported configuration options, check out the package file or run

spack info kynema

Once it is installed, you can load the Kynema library and its dependencies into your environment using

spack load kynema

Development using Spack Developer Workflow

One easy way to set up a development environment for Kynema is to use Spack’s Developer Workflow. To setup an environment for working on Kynema, setup Spack and then run the following commands:

mkdir kynema
cd kynema
spack env create -d .
spack env activate .
spack add kynema+tests
spack install
spack develop kynema@main
spack concretize -f
spack install

Kynema’s source code will now be located in the kynema folder, but can be accessed from anywhere by

spack cd -c kynema

After editing the code here, it can be rebuilt by running

spack install

To run the tests, first access the build folder through the spack command

spack cd -b kynema

Next, the tests can be run either through ctest or directly from the unit test or regression test executables

ctest
./tests/unit_tests/kynema_unit_tests
./tests/regression_tests/kynema_regression_tests

You can also build Kynema from this folder using standard make commands.

For more information, please see Spack’s documentation: https://spack-tutorial.readthedocs.io/en/latest/tutorial_developer_workflows.html

Building and Developing in Kynema Directly

The following sections outline how to build and develop Kynema without Spack’s Developer Workflows. The main complication here is that developers will have to manage their environment and dependencies manually, which may be an unnecessary complication or a freeing feature, depending on your perspective.

Dependencies

Before building Kynema, you’ll need the following:

  • C++ compiler that supports the C++20 standard

  • CMake: the default build system for C++ projects, version 3.21 or later

  • Kokkos: core programming model for performance portability

  • KokkosKernels: performance portable linear algebra library

  • netCDF: I/O data Format

  • Suite-Sparse: For the KLU sparse direct solver. Other solvers, such as SuperLU are also possible to use.

  • A LAPACK implementation, such as OpenBLAS or netlib-lapack

  • yaml-cpp: A package for reading YAML files, to be used by regression tests

  • GoogleTest: unit testing package

Installing Third Party Libraries

There are several methods to obtain the necessary Third Party Libraries (TPLs) for building Kynema, however the simplest is to use the spack package manager. Spack offers a comprehensive set of features for development and dependency management. The following is a quick-start guide for installing and loading the TPLs required to build Kynema.

Clone the spack repository, load the spack environment, and let spack learn about your system

git clone git@github.com:spack/spack.git
source spack/share/spack/setup-env.sh
spack compiler find
spack external find

Install GoogleTest, netCDF, Suite-Sparse, and LAPACK

spack install googletest
spack install netcdf-c
spack install lapack
spack install suite-sparse

Install Kokkos and Kokkos Kernels

For a simple serial build

spack install kokkos
spack install kokkos-kernels

To compile with OpenMP support for parallelism on CPU based machines

spack install kokkos+openmp
spack install kokkos-kernels+openmp

To compile with CUDA support

spack install kokkos+cuda+wrapper
spack install kokkos-kernels+cuda+cublas

To compile with ROCm support

spack install kokkos+rocm
spack install kokkos-kernels+rocblas

Load the TPLs into your environment

spack load googletest
spack load suite-sparse
spack load netcdf-c
spack load lapack
spack load kokkos
spack load kokkos-kernels

Building Kynema

The following is written assuming the TPLs in hand and the environment configured as described above.

Clone Kynema and setup a build directory

git clone git@github.com:kynema/kynema.git
cd kynema
mkdir build
cd build

Configure cmake

When building Kynema, you must specify which sparse direct solver package you want to use. We support many options here, but the default recommendation is to use suite-sparse’s KLU solver for CPU builds.

For a CPU-based build which includes building unit tests, you can configure with KLU using the command

cmake ../ -DKynema_ENABLE_KLU=ON

If Kokkos was built with CUDA support, you will need to use the nvcc_wrapper for compilation. You will also get your choice of native CUDA solvers (CUDSS or cuSolverSP). For best performance, CUDSS is currently recommended.

cmake ../ -DCMAKE_CXX_COMPILER=nvcc_wrapper -DKynema_ENABLE_CUDSS=ON

You can also use any CPU-based direct solver with a CUDA build. You may want to do this to reduce memory usage on device, or it may be faster for your problem. In this case, the system matrix and residual are calculated on GPU, copied to host for the solve step, and then the solution is copied back to GPU. For this mode of operation, simply configure Kynema as

cmake ../ -DCMAKE_CXX_COMPILER=nvcc_wrapper -DKynema_ENABLE_KLU=ON

If Kokkos was built with ROCm support, you will need to use the hipcc program for compilation. Currently, we do not support any native solvers for ROCm, so a CPU based solver (such as KLU) must be used.

cmake ../ -DCMAKE_CXX_COMPILER=hipcc -DKynema_ENABLE_KLU=ON

Build and Test

Currently, Kynema builds several shared libraries by default. To ensure that their unit tests pass, these libraries must be copied into the directory where the tests are executed.

make -j
ctest --output-on-failure

Once built, the unit test executable can also be run directly from the build directory

./tests/unit_tests/kynema_unit_tests

External Controllers

Wind turbine simulations often use shared library controllers to prescribe movements to blades. While Kynema supports calling to any shared library provided by the user, it can also detect the ROSCO controller if it is in the system path. To turn on this feature, configure CMake with the command

cmake ../ -DKynema_ENABLE_ROSCO_CONTROLLER=ON

This option will define the convenience global variable Kynema_ROSCO_LIBRARY, which is a string containing the location of the ROSCO library and can be used to initiaize Kynema’s controller wrapper.

Similarly, Kynema can call to OpenFAST’s AeroDyn module as a shared library to provide an aerodynamic inflow model. To find this library, if it is in the system path, configure Kynema with the command

cmake ../ -DKynema_ENABLE_OPENFAST_ADI=ON

This option will define the convenience global variable Kynema_ADI_LIBRARY, which is a string containing the location of the AeroDyn library, which can be used to initialize Kynema’s AeroDyn inflow wrapper.

Build Options

Kynema has several build options which can be set either when running CMake from the command line or through a GUI such as ccmake.

  • Kynema_ENABLE_CLANG_TIDY enables the Clang-Tidy static analysis tool

  • Kynema_ENABLE_COVERAGE enables code coverage analysis using gcov

  • Kynema_ENABLE_CPPCHECK enables the CppCheck static analysis tool

  • Kynema_ENABLE_IPO enables link time optimization

  • Kynema_ENABLE_PCH builds precompiled headers to potentially decrease compilation time

  • Kynema_ENABLE_SANITIZER_ADDRESS enables the address sanitizer runtime analysis tool

  • Kynema_ENABLE_SANITIZER_LEAK enables the leak sanitizer runtime analysis tool

  • Kynema_ENABLE_SANITIZER_MEMORY enables the memory sanitizer runtime analysis tool

  • Kynema_ENABLE_SANITIZER_THREAD enables the thread sanitizer runtime analysis tool

  • Kynema_ENABLE_SANITIZER_UNDEFINED enables the undefined behavior sanitizer runtime analysis tool

  • Kynema_ENABLE_TESTS builds Kynema’s test suite

  • Kynema_ENABLE_UNITY_BUILD uses unity builds to potentially decrease compilation time

  • Kynema_WRITE_OUTPUTS builds Kynema with VTK support for visualization in tests. Will need the VTK TPL to be properly configured

  • Kynema_WARNINGS_AS_ERRORS treats warnings as errors, including warnings from static analysis tools

  • Kynema_ENABLE_KLU builds Kynema with support for Suite-Sparse’s KLU solver; in our experience, this is solver is fast and robust for many of our problems.

  • Kynema_ENABLE_UMFPACK builds Kynema with support for Suite-Sparse’s UMFPACK solver.

  • Kynema_ENABLE_SUPERLU builds Kynema with support forthe SuperLU solver

  • Kynema_ENABLE_SUPERLU_MT builds Kynema with support for SuperLU-mt, a threaded version of SuperLU which may be configured to run in parallel on CPU.

  • Kynema_ENABLE_MKL builds Kynema with MKL’s sparse direct solver, which can take advantage of multiple threads to run in parallel on CPU.

  • Kynema_ENABLE_CUDSS builds Kynema with cuDSS, the next generation sparse direct solver of CUDA; still in pre-release at the time of writing, it is the preferred CUDA based solver if the platform supports it.

  • Kynema_ENABLE_CUSOLVERSP builds Kynema with the cuSolverSP sparse direct solver.

  • Kynema_ENABLE_ROSCO_CONTROLLER detects the ROSCO controller shared library and defines the Kynema_ROSCO_LIBRARY variable

  • Kynema_ENABLE_OPENFAST_ADI detects the OpenFAST AeroDyn shared library and defines the Kynema_ADI_LIBRARY variable