1 Installation¶

The following scripts are provided by the author for installing WRF-CTSM on a Linux workstation. Users should adapt these scripts to suit their own systems. For example, users on HPC systems may choose to load pre-installed modules (e.g., compiler, MPI, and libraries) rather than building them from source.

1.1 Preparation¶

Install libraries¶

GCC
- gcc-8 or gcc-9 is recommended.
git-lfs
ncl
nco
mpich-4_0_2.sh
zlib-1_3_1.sh
hdf5-1_12_3.sh
pnetcdf-1_12_3.sh
netcdfc-4_9_2.sh
netcdff-4_6_1.sh
pio-2_6_6.sh
esmf-8_8_1.sh
jasper-4_2_5.sh
libpng-1_6_39.sh
lapack-3_9_0.sh

Set environment¶

WORK_ROOT="/home/yuansun/"
INROOT=${WORK_ROOT}/software
COMPILER=gcc
export MPICHDIR=${INROOT}/${COMPILER}/mpich/4.0.2
export ZLIBDIR=${INROOT}/${COMPILER}/zlib/1.3.1
export HDF5DIR=${INROOT}/${COMPILER}/hdf5/1.12.3
export PNETCDFDIR=${INROOT}/${COMPILER}/pnetcdf/1.12.3
export NETCDFCDIR=${INROOT}/${COMPILER}/netcdf-c/4.9.2
export NETCDFFDIR=${INROOT}/${COMPILER}/netcdf-fortran/4.6.1
export PIODIR=${INROOT}/${COMPILER}/pio/2.6.6
export ESMFDIR=${INROOT}/${COMPILER}/esmf/8.8.1
export JASPERDIR=${INROOT}/${COMPILER}/jasper/4.2.5
export LIBPNGDIR=${INROOT}/${COMPILER}/libpng/1.6.39
export LD_LIBRARY_PATH=$ZLIBDIR/lib:$HDF5DIR/lib:$NETCDFCDIR/lib:$NETCDFFDIR/lib:$MPICHDIR/lib:$PNETCDFDIR/lib:$PIODIR/lib:$ESMFDIR/lib:$JASPERDIR/lib:$LIBPNGDIR/lib:$LD_LIBRARY_PATH
export PATH=$HDF5DIR/bin:$NETCDFCDIR/bin:$NETCDFFDIR/bin:$MPICHDIR/bin:$PNETCDFDIR/bin:$ESMFDIR/bin:$JASPERDIR/bin:$PATH
export CPATH=$ZLIBDIR/include:$HDF5DIR/include:$NETCDFCDIR/include:$NETCDFFDIR/include:$MPICHDIR/include:$PNETCDFDIR/include:$PIODIR/include:$ESMFDIR/include:$JASPERDIR/include:$LIBPNGDIR/include:$CPATH
export MANPATH=$ZLIBDIR/share/man:$HDF5DIR/share/man:$NETCDFCDIR/share/man:$NETCDFFDIR/share/man:$MPICHDIR/share/man:$PNETCDFDIR/share/man:$JASPERDIR/share/man:$LIBPNGDIR/share/man:$MANPATH
export CC=gcc-9
export CXX=g++-9
export FC=gfortran-9
export FCFLAGS="-I$ESMFDIR/mod -I$ESMFDIR/include -I$NETCDFCDIR/include -I$NETCDFFDIR/include -I$PNETCDFDIR/include -I$PIODIR/include -I$ESMFDIR/include -I$JASPERDIR/include -I$LIBPNGDIR/include"
export CPPFLAGS="$FCFLAGS"
export LDFLAGS="-L$ZLIBDIR/lib -L$HDF5DIR/lib -L$NETCDFCDIR/lib -L$NETCDFFDIR/lib -L$MPICHDIR/lib -L$PNETCDFDIR/lib -L$PIODIR/lib -L$ESMFDIR/lib -L$ESMFDIR/lib -L$JASPERDIR/lib -L$LIBPNGDIR/lib"
export LIBRARY_PATH=$LD_LIBRARY_PATH

Check the environment (optional)¶

Check git-lfs
```
git lfs version
```
Check gcc-9
```
which gcc-9
```
Check ncl
```
which ncl
```
Check nco
```
ncks --version
```
Check netcdf-c
```
nc-config --version
nc-config --all
```
Check netcdf-fortran
```
nf-config --all
```

1.2 Source Code Download¶

Download WRF code¶

export WRF_ROOT="${WORK_ROOT}wrf"
export WRFNAME=WRF-CTSM
cd ${WRF_ROOT}
git clone https://github.com/wrf-model/WRF.git ${WRFNAME} 
cd ${WRFNAME}
git checkout release-v4.7.0

Modify WRF code¶

According to CTSM-Norway tutorial, modify ${WRF_ROOT}/${WRFNAME}/phys/module_sf_mynn.F, around Line 1149,

From:

   Q2(I)=QSFCMR(I)+(QV1D(I)-QSFCMR(I))*PSIQ2/PSIQ
   Q2(I)= MAX(Q2(I), MIN(QSFCMR(I), QV1D(I)))
   Q2(I)= MIN(Q2(I), 1.05*QV1D(I))

   IF ( debug_code ) THEN
      yesno = 0
      IF (HFX(I) > 1200. .OR. HFX(I) < -700.)THEN
            print*,"SUSPICIOUS VALUES IN MYNN SFCLAYER",&
            I,J, "HFX: ",HFX(I)
            yesno = 1
      ENDIF

To:

   Q2(I)=QSFCMR(I)+(QV1D(I)-QSFCMR(I))*PSIQ2/PSIQ
   Q2(I)= MAX(Q2(I), MIN(QSFCMR(I), QV1D(I)))
   Q2(I)= MIN(Q2(I), 1.05*QV1D(I))
   
!YS
   IF (Q2(I) .LT. 0.0) THEN
       print*,"DEBUG: NEGATIVE Q2 VALUE IN MYNN SFCLAYER",&
       I,J, "Q2: ",Q2(I)
       print*,"WARNING: NEGATIVE Q2 SET TO ZERO"
       Q2(I)=0.0
   ENDIF
   IF (QSFC(I) .LT. 0.0) THEN
       print*,"DEBUG: NEGATIVE QSFC VALUE IN MYNN SFCLAYER",&
       I,J, "QSFC: ",QSFC(I)
   ENDIF
!YS

   IF ( debug_code ) THEN
      yesno = 0
      IF (HFX(I) > 1200. .OR. HFX(I) < -700.)THEN
            print*,"SUSPICIOUS VALUES IN MYNN SFCLAYER",&
            I,J, "HFX: ",HFX(I)
            yesno = 1
      ENDIF

Download CTSM code¶

export CTSMNAME=CTSMdev
cd ${WRF_ROOT}/${WRFNAME}
git clone https://github.com/ESCOMP/CTSM ${CTSMNAME}
cd ${CTSMNAME}
git checkout ctsm5.3.024
./bin/git-fleximod update

Modify CTSM code¶

Modify ${WRF_ROOT}/${WRFNAME}/${CTSMNAME}/src/cpl/lilac/lnd_comp_esmf.F90, around Line 38,

From:

   use clm_varctl        , only : nsrStartup, nsrContinue

To:

   !YS
   !use clm_varctl        , only : nsrStartup, nsrContinue
   use clm_varctl        , only : nsrStartup, nsrContinue, nsrBranch
   !YS

According to WRF-CTSM restart issue, modify ${WRF_ROOT}/${WRFNAME}/${CTSMNAME}/src/cpl/lilac/lnd_comp_esmf.F90, around Line 293,

From:

    if (trim(starttype) == trim('startup')) then
       nsrest = nsrStartup
    else if (trim(starttype) == trim('continue') ) then
       nsrest = nsrContinue

To:

    if (trim(starttype) == trim('startup')) then
       nsrest = nsrStartup
    else if (trim(starttype) == trim('continue') ) then
!YS    
       !nsrest = nsrContinue
       nsrest = nsrBranch
!YS       

According to CTSM-Norway tutorial, modify ${WRF_ROOT}/${WRFNAME}/${CTSMNAME}/src/cpl/utils/lnd_import_export_utils.F90, around Line 131,

From:

     if ( wateratm2lndbulk_inst%forc_q_not_downscaled_grc(g) < 0.0_r8 )then
          call shr_sys_abort( subname//&
          '         ERROR: Bottom layer specific humidty sent from the atmosphere model is less than zero' )
     end if 

To:

     !YS       
     !if ( wateratm2lndbulk_inst%forc_q_not_downscaled_grc(g) < 0.0_r8 )then
         !call shr_sys_abort( subname//&
         !        ' ERROR: Bottom layer specific humidty sent from the atmosphere model is less than zero' )
     !end if
     !YS 

According to using the CTSM lake model, modify ``${WRF_ROOT}/${WRFNAME}/${CTSMNAME}/tools/create_scrip_file.ncl` by adding:

lake_depth = wrf_file->LAKE_DEPTH(0,:,:) 
lu_index = wrf_file->LU_INDEX(0,:,:) 
lakemask = where(lu_index.eq.21, 1,0) 
landmask = where (lakemask.eq.1,1,landmask)

According to CTSM-Norway tutorial, modify ${WRF_ROOT}/${WRFNAME}/${CTSMNAME}/tools/site_and_regional/mkunitymap.ncl, around Line 74:

From:

  if ( n_a .ne. n_b )then
     print( "ERROR: dimensions of input SCRIP grid files is NOT the same!" );
     exit
  end if
  
  if ( any(ncb->grid_imask .ne. 1.0d00) )then
     print( "ERROR: the mask of the second file isn't identically 1!" );
     print( "(second file should be land grid file)");
     exit
  end if
  
  chkvars = (/ "grid_center_lat", "grid_center_lon", "grid_corner_lat", "grid_corner_lon" /);

To:

  if ( n_a .ne. n_b )then
     print( "ERROR: dimensions of input SCRIP grid files is NOT the same!" );
     exit
  end if
;YS  
;  if ( any(ncb->grid_imask .ne. 1.0d00) )then
;     print( "ERROR: the mask of the second file isn't identically 1!" );
;     print( "(second file should be land grid file)");
;     exit
;  end if
;YS  
  chkvars = (/ "grid_center_lat", "grid_center_lon", "grid_corner_lat", "grid_corner_lon" /);

Download WPS code¶

export WPSNAME=WPS
cd ${WRF_ROOT}
git clone https://github.com/wrf-model/WPS ${WPSNAME} 
cd ${WPSNAME}
git checkout v4.3

1.3 Build Model¶

Build CTSM¶

Customize machine configuration files under ${WRF_ROOT}/${WRFNAME}/${CTSMNAME}/ccs_config/machines/${MACHINENAME}
- config_machines.xml
- MACHINEANME.cmake

Compile CTSM code

export MACHINENAME=e-10uxx4b9fmw3
cd ${WRF_ROOT}/${WRFNAME}/${CTSMNAME}/lilac/
./build_ctsm ctsm_build_dir --machine ${MACHINENAME} --compiler gnu >build_ctsm.log 2>&1

Check if compiled successully. The end of build_ctsm.log should be:

clm built in 42.152718 seconds
Initial setup complete; it is now safe to work with the runtime inputs in
/home/yuansun/wrf/WRFnoleap-CTSMbranch/CTSMdev/lilac/ctsm_build_dir/runtime_inputs

Set CTSM path for building WRF-CTSM

source ctsm_build_dir/ctsm_build_environment.sh
export WRF_CTSM_MKFILE=${WRF_ROOT}/${WRFNAME}/${CTSMNAME}/lilac/ctsm_build_dir/ctsm.mk
export WRF_EM_CORE=1
export NETCDF_classic=1
export WRFIO_NCD_LARGE_FILE_SUPPORT=1

Build mksurfdata_esmf¶

Compile mksurfdata_esmf

cd ${WRF_ROOT}/${WRFNAME}/${CTSMNAME}/tools/mksurfdata_esmf
./gen_mksurfdata_build --machine ${MACHINENAME} >gen_mksurfdata_build.log 2>&1

Check if compiled successfully. The end of gen_mksurfdata_build.log should be:
```
Successfully created input namelist file surfdata.namelist
```

Build geo_domain¶

According to CTSM-Norway tutorial, build the gen_domain tool:

cd ${WRF_ROOT}/${WRFNAME}/${CTSMNAME}/tools
touch configure_wrf-ctsm

Copy scripts below to configure_wrf-ctsm:

#!/usr/bin/env python3

"""This script writes CIME build information to a directory.

The pieces of information that will be written include:

1. Machine-specific build settings (i.e. the "Macros" file).
2. File-specific build settings (i.e. "Depends" files).
3. Environment variable loads (i.e. the env_mach_specific files).

The .env_mach_specific.sh and .env_mach_specific.csh files are specific to a
given compiler, MPI library, and DEBUG setting. By default, these will be the
machine's default compiler, the machine's default MPI library, and FALSE,
respectively. These can be changed by setting the environment variables
COMPILER, MPILIB, and DEBUG, respectively.
"""

# pylint: disable=W1505

import os
import sys

real_file_dir = os.path.dirname(os.path.realpath(__file__))
cimeroot = os.path.abspath(os.path.join(real_file_dir, ".."))
sys.path.insert(0, cimeroot)

from CIME.Tools.standard_script_setup import *
from CIME.utils import expect, get_model
from CIME.BuildTools.configure import configure
from CIME.XML.machines import Machines

logger = logging.getLogger(__name__)


def parse_command_line(args):
    """Command line argument parser for configure."""
    description = __doc__
    parser = argparse.ArgumentParser(description=description)
    CIME.utils.setup_standard_logging_options(parser)

    parser.add_argument(
        "--machine", help="The machine to create build information for."
    )
    parser.add_argument(
        "--machines-dir",
        help="The machines directory to take build information "
        "from. Overrides the CIME_MODEL environment variable, "
        "and must be specified if that variable is not set.",
    )
    parser.add_argument(
        "--macros-format",
        action="append",
        choices=["Makefile", "CMake"],
        help="The format of Macros file to generate. If "
        "'Makefile' is passed in, a file called 'Macros.make' "
        "is generated. If 'CMake' is passed in, a file called "
        "'Macros.cmake' is generated. This option can be "
        "specified multiple times to generate multiple files. "
        "If not used at all, Macros generation is skipped. "
        "Note that Depends files are currently always in "
        "Makefile format, regardless of this option.",
    )
    parser.add_argument(
        "--output-dir",
        default=os.getcwd(),
        help="The directory to write files to. If not "
        "specified, defaults to the current working directory.",
    )

    parser.add_argument(
        "--compiler",
        "-compiler",
        help="Specify a compiler. "
        "To see list of supported compilers for each machine, use the utility query_config in this directory",
    )

    parser.add_argument(
        "--mpilib",
        "-mpilib",
        help="Specify the mpilib. "
        "To see list of supported mpilibs for each machine, use the utility query_config in this directory. "
        "The default is the first listing in MPILIBS in config_machines.xml",
    )

    parser.add_argument(
        "--clean",
        action="store_true",
        help="Remove old Macros and env files before attempting to create new ones",
    )

    parser.add_argument(
        "--comp-interface",
        default="mct",
        help="""The cime driver/cpl interface to use.""",
    )

    argcnt = len(args)
    args = parser.parse_args()
    CIME.utils.parse_args_and_handle_standard_logging_options(args)

    opts = {}
    if args.machines_dir is not None:
        machines_file = os.path.join(args.machines_dir, "config_machines.xml")
        machobj = Machines(infile=machines_file, machine=args.machine)
    else:
        model = get_model()
        if model is not None:
            machobj = Machines(machine=args.machine)
        else:
            expect(
                False,
                "Either --mach-dir or the CIME_MODEL environment "
                "variable must be specified!",
            )

    opts["machobj"] = machobj

    if args.macros_format is None:
        opts["macros_format"] = []
    else:
        opts["macros_format"] = args.macros_format

    expect(
        os.path.isdir(args.output_dir),
        "Output directory '%s' does not exist." % args.output_dir,
    )

    opts["output_dir"] = args.output_dir

    # Set compiler.
    if args.compiler is not None:
        compiler = args.compiler
    elif "COMPILER" in os.environ:
        compiler = os.environ["COMPILER"]
    else:
        compiler = machobj.get_default_compiler()
        os.environ["COMPILER"] = compiler
    expect(
        opts["machobj"].is_valid_compiler(compiler),
        "Invalid compiler vendor given in COMPILER environment variable: %s" % compiler,
    )
    opts["compiler"] = compiler
    opts["os"] = machobj.get_value("OS")
    opts["comp_interface"] = args.comp_interface

    if args.clean:
        files = [
            "Macros.make",
            "Macros.cmake",
            "env_mach_specific.xml",
            ".env_mach_specific.sh",
            ".env_mach_specific.csh",
            "Depends.%s" % compiler,
            "Depends.%s" % args.machine,
            "Depends.%s.%s" % (args.machine, compiler),
        ]
        for file_ in files:
            if os.path.isfile(file_):
                logger.warn("Removing file %s" % file_)
                os.remove(file_)
        if argcnt == 2:
            opts["clean_only"] = True
            return opts

    # Set MPI library.
    if args.mpilib is not None:
        mpilib = args.mpilib
    elif "MPILIB" in os.environ:
        mpilib = os.environ["MPILIB"]
    else:
        mpilib = machobj.get_default_MPIlib(attributes={"compiler": compiler})
        os.environ["MPILIB"] = mpilib

    expect(
        opts["machobj"].is_valid_MPIlib(mpilib, attributes={"compiler": compiler}),
        "Invalid MPI library name given in MPILIB environment variable: %s" % mpilib,
    )
    opts["mpilib"] = mpilib

    # Set DEBUG flag.
    if "DEBUG" in os.environ:
        expect(
            os.environ["DEBUG"].lower() in ("true", "false"),
            "Invalid DEBUG environment variable value (must be 'TRUE' or "
            "'FALSE'): %s" % os.environ["DEBUG"],
        )
        debug = os.environ["DEBUG"].lower() == "true"
    else:
        debug = False
        os.environ["DEBUG"] = "FALSE"
    opts["debug"] = debug

    return opts


def _main():
    opts = parse_command_line(sys.argv)
    if "clean_only" not in opts or not opts["clean_only"]:
        configure(
            opts["machobj"],
            opts["output_dir"],
            opts["macros_format"],
            opts["compiler"],
            opts["mpilib"],
            opts["debug"],
            opts["comp_interface"],
            opts["os"],
        )


if __name__ == "__main__":
    _main()

Run the scripts

chmod +x configure_wrf-ctsm

cd ${WRF_ROOT}/${WRFNAME}/${CTSMNAME}/tools/mapping/gen_domain_files/src/
../../../configure_wrf-ctsm --machine ${MACHINENAME} --compiler gnu --mpilib mpich --macros-format Makefile 

. ./.env_mach_specific.sh ; make 

Build WRF¶

Create a configuration file named configure.wrf

export NETCDF_C=/home/yuansun/software/gcc/netcdf-c/4.9.2
export NETCDF=/home/yuansun/software/gcc/netcdf-fortran/4.6.1
cd ${WRF_ROOT}/${WRFNAME}
./clean -a
./configure
# - 34: (dmpar: distributed memory parallelization)
# - 1: nesting option (basic)

Modify configure.wrf

Link libraries:

From:

 LIB_EXTERNAL    = \
                      -L$(WRF_SRC_ROOT_DIR)/external/io_netcdf -lwrfio_nf -L/home/yuansun/software/gcc/netcdf-fortran/4.6.1/lib -lnetcdff -lnetcdf     

To:

LIB_EXTERNAL    = \
                      -L$(WRF_SRC_ROOT_DIR)/external/io_netcdf -lwrfio_nf \
                      -L /home/yuansun/software/gcc/netcdf-c/4.9.2/lib -L /home/yuansun/software/gcc/netcdf-fortran/4.6.1/lib -lnetcdff -lnetcdf \
                      -L /home/yuansun/software/gcc/hdf5/1.12.3/lib -lhdf5_hl -lhdf5 \
                      -L /home/yuansun/software/gcc/pnetcdf/1.12.3/lib -lpnetcdf \
                      -L /home/yuansun/software/gcc/jasper/4.2.5/lib -ljasper \
                      -L /home/yuansun/software/gcc/libpng/1.6.39/lib -lpng \
                      -L /home/yuansun/software/gcc/zlib/1.3.1/lib -lz \
                      -L /home/yuansun/software/gcc/lapack/3.9.0 -llapack -lblas \
                      -lgomp -lm -ldl 

Setting:

From:

SFC             =       gfortran
SCC             =       gcc
CCOMP           =       gcc

DM_CC           =       mpicc -cc=$(SCC)
CC              =       $(DM_CC)

ARCH_LOCAL      =       -DNONSTANDARD_SYSTEM_SUBR  -DWRF_USE_CTSM -DNO_IEEE_MODULE -DNO_ISO_C_SUPPORT -DNO_FLUSH_SUPPORT -DNO_GAMMA_SUPPORT

FCCOMPAT        =        -fallow-argument-mismatch -fallow-invalid-boz

To:

SFC             =       gfortran-9
SCC             =       gcc-9
CCOMP           =       gcc-9

DM_CC           =       mpicc
CC              =       $(DM_CC) -DFSEEKO64_OK 

ARCH_LOCAL      =       -DNONSTANDARD_SYSTEM_SUBR  -DWRF_USE_CTSM
FCCOMPAT        = 

Adding include path

From:

INCLUDE_MODULES =    $(MODULE_SRCH_FLAG) \
                     $(ESMF_MOD_INC) $(ESMF_LIB_FLAGS) \
                      -I$(WRF_SRC_ROOT_DIR)/main \
                      -I$(WRF_SRC_ROOT_DIR)/external/io_netcdf \
                      -I$(WRF_SRC_ROOT_DIR)/external/io_int \
                      -I$(WRF_SRC_ROOT_DIR)/frame \
                      -I$(WRF_SRC_ROOT_DIR)/share \
                      -I$(WRF_SRC_ROOT_DIR)/phys \
                      -I$(WRF_SRC_ROOT_DIR)/wrftladj \
                      -I$(WRF_SRC_ROOT_DIR)/chem -I$(WRF_SRC_ROOT_DIR)/inc \
                      -I$(NETCDFPATH)/include \
                      $(CTSM_INCLUDES)
HDF5PATH        =
PNETCDFPATH     =

To:

INCLUDE_MODULES =    $(MODULE_SRCH_FLAG) \
                     $(ESMF_MOD_INC) $(ESMF_LIB_FLAGS) \
                      -I$(WRF_SRC_ROOT_DIR)/main \
                      -I$(WRF_SRC_ROOT_DIR)/external/io_netcdf \
                      -I$(WRF_SRC_ROOT_DIR)/external/io_int \
                      -I$(WRF_SRC_ROOT_DIR)/frame \
                      -I$(WRF_SRC_ROOT_DIR)/share \
                      -I$(WRF_SRC_ROOT_DIR)/phys \
                      -I$(WRF_SRC_ROOT_DIR)/wrftladj \
                      -I$(WRF_SRC_ROOT_DIR)/chem -I$(WRF_SRC_ROOT_DIR)/inc \
                      -I$(NETCDFPATH)/include \
                      -I${HDF5PATH}/include \
                      -I${JASPERPATH}/include \
                      -I${PNETCDFPATH}/include \
                      -I${LIBPNGPATH}/include \
                       $(CTSM_INCLUDES)
                       
HDF5PATH        =    /home/yuansun/software/gcc/hdf5/1.12.3                     PNETCDFPATH     =    /home/yuansun/software/gcc/pnetcdf/1.12.3
JASPERPATH      =    /home/yuansun/software/gcc/jasper/4.2.5
LIBPNGPATH      =    /home/yuansun/software/gcc/libpng/1.6.39
LAPACKPATH      =    /home/yuansun/software/gcc/lapack/3.9.0

Compile WRF (taking around 20 minutes)

nohup ./compile em_real 2>&1 > compile.log &

Check if compiled successfully by listing excutable files. Must be four files: ndown.exe, real.exe, tc.exe, wrf. exe
```
ls main/*exe 
```

Build WPS¶

Compile WPS

export WRF_DIR=${WRF_ROOT}/${WRFNAME}
./clean -a
./configure
# - 3
./compile >& compile.log

Check if compiled successfully by listing excutable files. Must be three files: geogrid.exe metgrid.exe , ungrib.exe
```
ls *exe
```

1 Installation¶

1.1 Preparation¶

Install libraries¶

Set environment¶

Check the environment (optional)¶

1.2 Source Code Download¶

Download WRF code¶

Modify WRF code¶

Download CTSM code¶

Modify CTSM code¶

Download WPS code¶

1.3 Build Model¶

Build CTSM¶

Build mksurfdata_esmf¶

Build geo_domain¶

Build WRF¶

Build WPS¶

Table of Contents

Previous topic

Next topic

This Page