Building Kynema-UGF Semi-Automatically Using Spack ================================================== Mac OS X or Linux ----------------- The following describes how to build Kynema-UGF and its dependencies mostly automatically on your Mac using `Spack `__. This can also be used as a template to build Kynema-UGF on any Linux system with Spack. Step 1 ~~~~~~ This assumes you have a (Homebrew) installation of GCC installed already (we are using GCC 7.3.0). These instructions have been tested on OSX 10.11, MacOS 10.12, and MacOS 10.13. MacOS 10.12/10.13 will not build CMake or Pkg-Config with GCC because they will pick up system header files that have objective C code in them. Therefore we build Kynema-UGF using Spack on MacOS Sierra by using Homebrew to install ``cmake`` and ``pkg-config`` and defining these as external packages in Spack (see `packages.yaml `__). Step 2 ~~~~~~ Checkout the official Spack repo from github (we will checkout into ``${HOME}``): :: cd ${HOME} && git clone https://github.com/spack/spack.git Step 3 ~~~~~~ Add Spack shell support to your ``.profile`` or ``.bashrc`` etc, by adding the lines: :: export SPACK_ROOT=${HOME}/spack source ${SPACK_ROOT}/share/spack/setup-env.sh Step 4 ~~~~~~ Run the `setup-spack.sh `__ script from the repo which tries to find out what machine you are on and then copies the corresponding ``*.yaml`` configuration files to your Spack installation: :: cd ${HOME} && git clone https://github.com/exawind/build-test.git cd ${HOME}/build-test/configs && ./setup-spack.sh Step 5 ~~~~~~ Try ``spack info kynema-ugf`` to see if Spack works. If it does, check the compilers you have available by: :: machine:~ user$ spack compilers ==> Available compilers -- clang sierra-x86_64 ------------------------------------------ clang@9.0.0-apple -- gcc sierra-x86_64 -------------------------------------------- gcc@7.3.0 gcc@6.4.0 gcc@5.5.0 Step 6 ~~~~~~ Install Kynema-UGF with whatever compiler you prefer (it will default to Apple Clang) merely by running ``spack install kynema-ugf`` or by editing and running the ``install_kynema-ugf_gcc_mac.sh`` script from the `build-test `__ repo: :: cd ${HOME}/build-test/install_scripts && ./install_kynema-ugf_gcc_mac.sh That should be it! When using the install script you will see that Spack will install using the constraints we've specified in ``shared_constraints.sh`` which specifies a much more specific set of Trilinos options for Kynema-UGF that can shorten the build time. NREL's Eagle Machine -------------------- The following describes how to build Kynema-UGF and its dependencies mostly automatically on NREL's Eagle machine using Spack. This can also be used as a template to help build Kynema-UGF on any Linux system with Spack. Step 1 ~~~~~~ Login to Eagle, and checkout the ``https://github.com/exawind/build-test.git`` repo (we will be cloning into the ${HOME} directory): :: cd ${HOME} && git clone https://github.com/exawind/build-test.git Step 2 ~~~~~~ Checkout the official Spack repo from github: ``cd ${HOME} && git clone https://github.com/spack/spack.git`` Step 3 ~~~~~~ Configure your environment in the recommended way. You should purge all modules and load GCC 7.3.0 in your login script. In the example `.bashrc `__ in the repo we also load Python. If you have problems building with Spack on Eagle, it is most likely your environment has deviated from this recommended one. Even when building with the Intel compiler in Spack, this is the recommended environment at login. :: module purge module load gcc/7.3.0 Also add Spack shell support to your ``.bashrc`` as shown in the example `.bashrc `__ in the repo or the following lines: :: export SPACK_ROOT=${HOME}/spack source ${SPACK_ROOT}/share/spack/setup-env.sh Log out and log back in or source your ``.bashrc`` to get the Spack shell support loaded. Try ``spack info kynema-ugf`` to see if Spack works. Step 4 ~~~~~~ Configure Spack for Eagle. This is done by running the `setup-spack.sh `__ script provided which tries finding what machine you're on and copying the corresponding ``*.yaml`` file to your Spack directory: :: cd ${HOME}/build-test/configs && ./setup-spack.sh Step 5 ~~~~~~ Try ``spack info kynema-ugf`` to see if Spack works. Step 6 ~~~~~~ Note the build scripts and packages.yaml configuration files provided here adhere to the official versions of the third party libraries we test with, and that you may want to adhere to using them as well. Also note that when you checkout the latest Spack, it also means you will be using the latest packages available if you do not set constraints at install time and the newest packages may not have been tested to build correctly on NREL machines yet. So specifying versions of the TPL dependencies in your packages.yaml file for Spack is recommended. Install Kynema-UGF using a non-GPU login node with the example script `install_kynema-ugf_eagle.sh `__ or edit the script to use the correct allocation and ``nice ./install_kynema-ugf_eagle.sh``. That's it! Hopefully the ``install_kynema-ugf_eagle.sh`` script installs the entire set of dependencies and you get a working build of Kynema-UGF on Eagle...after several hours of waiting for it to build. To build with the Intel compiler, note the necessary change listed in the `install_kynema-ugf_eagle.sh `__ batch script. To load Kynema-UGF dependencies (you will need Spack's OpenMPI for Kynema-UGF now) into your path you will need to ``spack load openmpi %compiler`` and ``spack load kynema-ugf %compiler``, using ``%gcc`` or ``%intel`` to specify which to load. Development Build of Kynema-UGF ------------------------------- When building Kynema-UGF with Spack, Spack will cache downloaded archive files such as ``*.tar.gz`` files. However, by default Spack will also erase extracted or checked out ('staged') source files after it has built a package successfully. Therefore if your build succeeds, Spack will have erased the Kynema-UGF source code it checked out from Github. The recommended way to get a version of Kynema-UGF you can develop in is to checkout Kynema-UGF yourself outside of Spack and build this version using the dependencies Spack has built for you. To do so, checkout Kynema-UGF: :: git clone https://github.com/exawind/kynema-ugf.git Next, create your own directory to build in, or use the existing ``build`` directory in Kynema-UGF to run the CMake configuration. When running the CMake configuration, point Kynema-UGF to the dependencies by using ``spack location -i ``. For example in the ``build`` directory run: :: cmake -DTrilinos_DIR:PATH=$(spack location -i trilinos) \ -DYAML_DIR:PATH=$(spack location -i yaml-cpp) \ -DCMAKE_BUILD_TYPE=RELEASE \ .. make There are also ``do-config`` scripts available for this according to machine under the configs directory `here `__. These scripts may also provide the capability to access and use pre-built dependencies from a set of modules if they are available on the machine. This should allow you to have a build of Kynema-UGF in which you are able to continuosly modify the source code and rebuild. Building inside Docker Container -------------------------------- It is also possible to build (and run) Kynema-UGF inside a docker container with a prepared environment. This has the benefit of requiring less setup and usually being faster and can be useful in some situations (e.g. quickly testing something). For this, use the container `ecpe4s/exawind-snapshot `_: :: docker run -it ecpe4s/exawind-snapshot bash Inside the container you can directly load the pre-installed version of Kynema-UGF with spack: :: spack load kynema-ugf `Kynema-Manager `_ is used generate the container and is pre-installed in the container. Additional development of the code can be done through the Kynema-Manager's developer tools. If you are uncomfortable using spack for development then you can also set up your own build workflow in more of a "roll-your-own" development environment using CMake and just utilize the pre-installed TPL's that are available in the container through spack. This really comes down to your development preferences and your comfort level with CMake.