Compiling NEMO 4.0 on NEXCS


Note this guide has been updated here to work with a later release of NEMO using intel compilers. This guide still works but I would advise using the more up-to-date guide.


These are my personal notes on how to set up and compile NEMO 4.0 on NEXCS.

NEXCS is the NERC partition of the Met Office Cray XC40 supercomputer and has a similar infrastructure to Monsoon2.

When compiling NEMO I could not find any guide for this particular system but used the following excellent guides for inspiration:

As these guides were so helpful, I feel it is only fair to make a similar contribution.

Step 1 - Loading modules

Once you are logged on to the Met Office lander, connect to NEXCS using

ssh xcs-c

On logging in you need to load the correct compilers and libraries. You will not be able to compile XIOS or NEMO without loading the correct modules. First load the netcdf and hdf5 libraries.

module load cray-netcdf-hdf5parallel/4.4.1
module load cray-hdf5-parallel/1.10.0

Switch to a more recent version of mpich.

module swap cray-mpich/7.0.4 cray-mpich/7.3.1

Finally switch from the Cray environment to the GNU environment

module swap PrgEnv-cray/5.2.82 PrgEnv-gnu/5.2.82

It will also be useful to load an editor

module load nano

Note: When you log in to NEXCS a set of essential modules are loaded by default. Do not use module purge before loading the above modules.


You loaded modules should look like the following:

user@xcslc0:~> module list
  1) moose-client-wrapper                   9) craype-network-aries                  17) cray-hdf5-parallel/1.10.0             25) dvs/2.5_0.9.0-1.0502.2188.1.116.ari
  2) python/v2.7.9                         10) gcc/4.9.1                             18) cray-libsci/13.0.1                    26) alps/5.2.4-2.0502.9774.31.11.ari
  3) metoffice/tempdir                     11) craype/2.2.1                          19) udreg/2.3.2-1.0502.10518.2.17.ari     27) rca/1.0.0-2.0502.60530.1.62.ari
  4) metoffice/userenv                     12) craype-haswell                        20) ugni/6.0-1.0502.10863.8.29.ari        28) atp/1.7.5
  5) subversion-1.8/1.8.19                 13) cray-mpich/7.3.1                      21) pmi/5.0.5-1.0000.10300.134.8.ari      29) PrgEnv-gnu/5.2.82
  6) modules/3.2.10.5                      14) pbs/18.2.5.20190913061152             22) dmapp/7.0.1-1.0502.11080.8.76.ari
  7) eswrap/1.3.3-1.020200.1280.0          15) nano/2.8.6                            23) gni-headers/4.0-1.0502.10859.7.8.ari
  8) switch/1.0-1.0502.60522.1.61.ari      16) cray-netcdf-hdf5parallel/4.4.1        24) xpmem/0.1-2.0502.64982.5.3.ari

Step 2 - XIOS 2.5

XIOS controls the Input/Output for NEMO and needs to be compiled first. For NEMO 4.0 I use XIOS 2.5 but the method is very similar for other versions.

First navigate to the data directory on NEXCS and make a folder for XIOS

cd $DATADIR
mkdir XIOS

Inside the folder download XIOS 2.5

cd XIOS
svn checkout -r 1566 http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-2.5 xios-2.5

We need to make sure XIOS is looking in the right place for the modules loaded in Step 1. We copy and rename the following files in the arch folder

cd xios-2.5/arch/ 
cp arch-GCC_LINUX.env arch-GCC_NEXCS.env
cp arch-GCC_LINUX.fcm arch-GCC_NEXCS.fcm 
cp arch-GCC_LINUX.path arch-GCC_NEXCS.path

Then edit the files so they look like the following

#arch-GCC_NEXCS.env
export HDF5_INC_DIR=/opt/cray/hdf5/1.10.0/GNU/4.9/include
export HDF5_LIB_DIR=/opt/cray/hdf5/1.10.0/GNU/4.9/lib

export NETCDF_INC_DIR=/opt/cray/netcdf/4.4.1/GNU/4.9/include
export NETCDF_LIB_DIR=/opt/cray/netcdf/4.4.1/GNU/4.9/lib

Note: If you are using a different set of modules or worried that this path is wrong. The commands which nc-config and which h5copy will give you a path of the form directory/bin. The paths used above should then be directory/GNU/4.9/include and similar for lib


#arch-GCC_NEXCS.fcm
################################################################################
################### Projet XIOS ###################
################################################################################

%CCOMPILER cc
%FCOMPILER ftn
%LINKER CC

%BASE_CFLAGS -ansi -w
%PROD_CFLAGS -O3 -DBOOST_DISABLE_ASSERTS
%DEV_CFLAGS -g -O2
%DEBUG_CFLAGS -g

%BASE_FFLAGS -D__NONE__
%PROD_FFLAGS -O3
%DEV_FFLAGS -g -O2
%DEBUG_FFLAGS -g

%BASE_INC -D__NONE__
%BASE_LD -lstdc++

%CPP cpp
%FPP cpp -P
%MAKE gmake

#arch-GCC_NEXCS.path
NETCDF_INCDIR="-I $NETCDF_INC_DIR"
NETCDF_LIBDIR="-Wl,'--allow-multiple-definition' -L$NETCDF_LIB_DIR"
NETCDF_LIB="-lnetcdff -lnetcdf"

MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""

HDF5_INCDIR="-I $HDF5_INC_DIR"
HDF5_LIBDIR="-L $HDF5_LIB_DIR"
HDF5_LIB="-lhdf5_hl -lhdf5 -lhdf5 -lz"

Then you need to edit bld.cfg in xios-2.5/

cd $DATADIR/XIOS/xios-2.5

and change all occurrences of src_netcdf to src_netcdf4 (two in total)

You are now ready to compile XIOS. In xios-2.5/

./make_xios --full --prod --arch GCC_NEXCS -j2

If this build is successful then XIOS is ready to use. If not, check over the arch files and make sure the flags are correct and in the right order. Also make sure you have the correct modules loaded. Getting XIOS to compile is by far the most difficult part of the process.

Step 3 - NEMO 4.0

Now we have XIOS ready to go we can start working on NEMO 4.0.

Go back to the data directory and create a new folder for NEMO

cd $DATADIR
mkdir NEMO

Then download NEMO 4.0 in the folder.

cd NEMO
svn checkout -r 9925 http://forge.ipsl.jussieu.fr/nemo/svn/NEMO/trunk nemo4.0-9925

Now we need to copy and update the arch files for NEMO like we did in XIOS.

cd nemo4.0-9925/arch/
cp arch-linux_ifort.fcm arch-linux_NEXCS.fcm

The file arch-linux_NEXCS.fcm needs to look like

#arch-linux_NEXCS.fcm

%NCDF_HOME           /opt/cray/netcdf/4.4.1/GNU/4.9
%HDF5_HOME           /opt/cray/hdf5/1.10.0/GNU/4.9
%XIOS_HOME           /projects/nexcs-n02/user/XIOS/xios-2.5

%NCDF_INC            -I%NCDF_HOME/include -I%HDF5_HOME/include
%NCDF_LIB            -L%NCDF_HOME/lib -lnetcdf -lnetcdff -lstdc++
%XIOS_INC            -I%XIOS_HOME/inc
%XIOS_LIB            -L%XIOS_HOME/lib -lxios -lstdc++

%CPP                 cpp
%FC                  ftn
%FCFLAGS             -fdefault-real-8 -O3 -funroll-all-loops -fcray-pointer -cpp -ffree-line-length-none
%FFLAGS              %FCFLAGS
%LD                  %FC                                                                                                                                                                               %FPPFLAGS            -P -C -traditional
%LDFLAGS
%AR                  ar
%ARFLAGS             -rs
%MK                  make
%USER_INC            %XIOS_INC %NCDF_INC
%USER_LIB            %XIOS_LIB %NCDF_LIB

%CC                  cc

Step 4 - Compiling the GYRE_PISCES configuration

Now we need a configuration to compile. In this example I use GYRE_PISCES which is a reference configuration in NEMO as it is a relatively simple configuration. Other reference configurations are listed here and the process will be similar.

Copy the reference configuration you like to use by doing the following.

cd $DATADIR/NEMO/nemo4.0-9925/cfgs
mkdir GYRE_testing
rsync -arv GYRE_PISCES/* GYRE_testing

Then go into GYRE_testing, rename the cpp_GYRE_PISCES.fcm file.

cd GYRE_testing
mv cpp_GYRE_PISCES.fcm cpp_GYRE_testing.fcm

Then edit the same file and replace key_top with key_nosignedzero.


In cfgs/ edit the file ref_cfgs.txt and add your configuration to the bottom of the list. For this example the file looks like.

AGRIF_DEMO OCE ICE NST
AMM12 OCE
C1D_PAPA OCE
GYRE_BFM OCE TOP
GYRE_PISCES OCE TOP
ORCA2_OFF_PISCES OCE TOP OFF
ORCA2_OFF_TRC OCE TOP OFF
ORCA2_SAS_ICE OCE ICE NST SAS
ORCA2_ICE_PISCES OCE TOP ICE NST
SPITZ12 OCE ICE
GYRE_testing OCE TOP

Note: Make sure the OCE/TOP/… flags match the reference configuration you are using


You are now ready to compile nemo. Make sure you have all the modules in Step 1 loaded. Go back to the base directory and do the following

cd $DATADIR/NEMO/nemo4.0-9925
./makenemo -j2 -r GYRE_testing -m linux_NEXCS

If all has gone well you have successfully compiled NEMO!

Step 5 - Using NEMO on NEXCS

Now that you have compiled NEMO, you probably want to run it.

NEMO runs in parallel and will output a data file for each thread used. Therefore we need the REBUILD_NEMO tool to recombine the outputs into a single file.

To do this go to the tools folder and execute the following command:

cd $DATADIR/NEMO/nemo4.0-9925/tools
./maketools -n REBUILD_NEMO -m linux_NEXCS

Note: If you get an error about iargc_ and/or getarg not being external variables then go to the following file tools/REBUILD_NEMO/src/rebuild_nemo.F90 and comment out these two lines

INTEGER, EXTERNAL :: iargc
 ...
 external :: getarg

These are not external variables in Fortran 90.


Now go back to your configuration and create a symbolic link to xios_server.exe

cd $DATADIR/NEMO/nemo4.0-9925/cfgs/GYRE_testing/EXP00
ln -s $DATADIR/XIOS/xios-2.5/bin/xios_server.exe .

Also modify iodef.xml and set using_server to true

<variable id="using_server"              type="bool">true</variable>

Make an OUTPUTS/ and RESTARTS/ directory in EXP00

mkdir OUTPUTS RESTARTS

You also need to add a shell script for post processing to EXP00. The one I use below is a modified version of Julian Mak’s post processing script. It is too long to show on this page but you can find it here and it should work for this example. The only values that need changing are NUM_DOM(how many nodes is NEMO using) and NUM_CPU (how many cpus in total).

To run a job on NEXCS you need to use a submission script like the one below. Yet again this is a modification of Julian’s submission script.

#!/bin/bash
#!
##subm_script.pbs
#PBS -q nexcs
#PBS -A user ##Username
#PBS -l select=3 ##(Number of nodes running NEMO + Number of nodes running XIOS)
#PBS -l walltime=00:10:00 ##Maximum runtime
#PBS -N GYRE_testing ##Name of run in queue
#PBS -o testing.output ##Name of output file
#PBS -e testing.error ##Name of error file
#PBS -j oe
#PBS -V

cd $DATADIR/NEMO/nemo4.0-9925/cfgs/GYRE_testing/EXP00/

echo " _ __   ___ _ __ ___   ___         "
echo "| '_ \ / _ \ '_ ' _ \ / _ \        "
echo "| | | |  __/ | | | | | (_) |       "
echo "|_| |_|\___|_| |_| |_|\___/  v4.0  "

export OCEANCORES=64 #Number of cores used (32 per node running NEMO) 
export XIOSCORES=2 #Number of cores to run XIOS (=number of nodes running NEMO)

ulimit -c unlimited
ulimit -s unlimited

aprun -b -n $XIOSCORES -N 2 ./xios_server.exe : -n $OCEANCORES -N 32 ./nemo

#===============================================================
# POSTPROCESSING
#===============================================================

# kills the daisy chain if there are errors

if grep -q 'E R R O R' ocean.output ; then

  echo "E R R O R found, exiting..."
  echo "  ___ _ __ _ __ ___  _ __  "
  echo " / _ \ '__| '__/ _ \| '__| "
  echo "|  __/ |  | | | (_) | |    "
  echo " \___|_|  |_|  \___/|_|    "
  echo "check out ocean.output or stdouterr to see what the deal is "

  exit
else
  echo "going into postprocessing stage..."
  # cleans up files, makes restarts, moves files, resubmits this pbs

  bash ./postprocess.sh >& cleanup.log
  exit
fi

This script uses two nodes (each with 32 cores) to run NEMO and an additional node to run XIOS.


Note: For the postprocessing to work the number of nodes used for NEMO (not XIOS) must = NUM_DOM and the number of cores for NEMO = NUM_CPU


Submit this script by doing

qsub subm_script.pbs

Check the status of the submitted job using qstat

If all goes well you should have outputs in the OUTPUTS/ folder. If not, check testing.output and ocean.output to see what the issue is.


Note: Make sure you have the modules mentioned in Step 1 loaded when submitting


Note: If you have an error referring to namberg have a look in namelist_refand look for a comment character ! right next to a variable.

.true.!comment

If found, add a space.

.true. !comment

Congratulations, you have successfully compiled and run NEMO!