7. Code Profiling

Benchmarking (also referred to as profiling) of MET tools is accomplished using the CTRACK tool:: https://github.com/Compaile/ctrack
This code is licensed under the MIT License:: https://github.com/Compaile/ctrack/blob/main/LICENSE

Benchmarking uses a macro and the C++ source code is readily instrumented by including ctrack.hpp and by adding CTRACK at the top of the function of interest. By default, the tool generates summary and detail metrics to stdout (standard output) in easy to read, well-formatted tables. The ctrack.hpp file has been modified to permit saving these tables to their respective text files (summary_output.txt and detail_output.txt).

7.1. Overview

7.1.1. Customizations for MET

A Python script, benchmark.py is available to exercise the MET source code under consideration either via MET commands (to replicate command line usage of the MET tool), or via METplus use case (utilizing the METplus wrapper code and associated configuration files). The Python script consolidates the summary and detail metrics information (as text files) into csv and tabular text files. The benchmark.py script has an accompanying configuration file, benchmark.yaml located in the $HOME/MET/internal/scripts/benchmark directory (where $HOME is the directory where the MET source code is located).

The ctrack.hpp header file is modified to allow the summary and detailed reports to be saved as text files to facilitate the consolidation of information into csv and tabular formats. The summary and detail metrics files are located in the directory where the benchmark.py script was invoked. The modified version of ctrack.hpp is located in the ${BASE_DIR}/MET/src/basic/vx_util directory, where ${BASE_DIR} is the full path to where the MET source code has been cloned or forked.

7.1.2. Code that is currently instrumented

The following code is instrumented using CTRACK:

MET/src/basic/vx_util/main.cpp
- do_pre_process function
- do_post_process function
MET/src/tools/core/ensemble_stat/ensemble_stat.cc
MET/src/tools/core/ensemble_stat/ensemble_stat_conf.cc
MET/src/tools/other/grid_diag/grid_diag.cc

7.1.3. Benchmarking with Python script

The benchmarking.py script invokes MET code either via MET command line commands or METplus use cases as specified by the run_met_directly setting in the benchmark.yaml configuration file. The metrics from the summary and detail tables are consolidated into csv and tabular text files (the locations of these consolidated metrics text files are specified in the benchmark.yaml configuration file). The CTRACK summary_output.txt and detail_output.txt reports (containing the performance metrics) are written to the directory from which the benchmarking.py script was executed. An information file is also generated that captures the version of Python used, a timestamp, and any other relevant information for capturing the environment under which the code was profiled/benchmarked.

7.1.3.1. Overview of Steps for Performing Benchmarking

Instrument the MET code of interest
Note

The ctrack.hpp file is saved in the $HOME/MET/src/basic/vx_util directory and does not need to be modified or added to any other location. This version of ctrack.hpp has been modified to write the summary and detail tables to text files. By default, CTRACK is disabled and is enabled at compilation time via the --enable-profiler flag.

$HOME refers to the path to where the MET source code is saved.

The ctrack.hpp file must be included in the source code of interest:

#ifdef WITH_PROFILER #include "ctrack.hpp" #endif

The CTRACK directive is placed at the top of the function of interest. Use the preprocessor directive for WITH_PROFILER:

e.g. ensemble_stat.cc:

void process_grid(const Grid &fcst_grid) { #ifdef WITH_PROFILER CTRACK; #endif Grid obs_grid; ... more code
and the ctrack::result_print is placed within the corresponding MET tool’s main()/met_main() function
e.g. ensemble_stat.cc

int met_main(int argc, char *argv[]) { // Process the command line arguments process_command_line(argc, argv); // Check for valid ensemble data process_n_vld(); // Perform verification process_vx(); // Save the CTRACK metrics #ifdef WITH_PROFILER ctrack::result_print(); #endif

Note

The summary_output.txt and detail_output.txt files will only be saved when the ctrack::result_print() function is called within main() or met_main().

Compile MET code

Edit the benchmark.yaml configuration file
Note

the benchmark.py and benchmark.yaml files must reside in the same directory (the benchmark.yaml file does NOT need to be specified at the command line)
The following is an example benchmark.yaml config file that utilizes environment variables and full directory paths
# Configuration file used to collect benchmarking in MET tools using CTRACK # # # filename # Timestamp in ISO 1806 format is used to generate output filename # If filename setting is empty string, then timestamp is used. # Otherwise, the specified filename followed by the timestamp will # be used for the output filename. # filename: '' # # Output directory where output files will be saved # benchmark_output_path: !ENV '${BENCHMARK_OUTPUT_BASE}' # ------------------------------- # FOR Running METPLUS USE CASE(S) # ------------------------------- # # location of METplus # metplus_base: !ENV '${METPLUS_BASE}' # # location of system.conf file # system_conf: "/path/to/MET/internal/scripts/benchmark/system.conf" # # location of METplus wrapper configuration file(s) # wrapper_conf: - "/path/to/usecase_confs/truncated/EnsembleStat_fcstRRFS_obsCCPA_1hrAPCP_truncated.conf" # ------------------------ # FOR RUNNING MET COMMAND # ------------------------ run_met_directly: False met_command: '' # subdirectory to save the consolidated information, if empty, the # MET tool name will be used met_subdir_name: 'EnsembleStat_fcstRRFS_obsCCPA_1hrAPCP' #----------------------------------- # For future stress-testing support #----------------------------------- # number of times to run the use case for stress-testing # The default=1 if this setting is missing or unspecified # num_runs: 1
Config settings for running via MET command:
benchmark_output_path

required

output directory where the output files will be saved

specify in one of two ways:

setting the BENCHMARK_OUTPUT_BASE env variable

explicitly setting the full directory path

filename

optional

the supplied filename prepended with a Timestamp that follows ISO 1806 format

if left empty, the timestamp alone will be used as the filename

run_met_directly

required

set to True

met_command

required

the command to run the MET tool with the appropriate arguments

this is the same command that would be ordinarily used when running a MET tool from the command line

make sure the specified -outdir directory exists

met_subdir_name

optional

if left empty, the consolidated benchmark metrics will be saved to a subdirectory (in the benchmark_output_path) named after the MET tool

num_runs

optional

to be used for stress-testing/running command multiple times

if not set, default value is 1
Config settings for running via METplus usecase(s):
benchmark_output_path

required

output directory where the output files will be saved

specify in one of two ways:

setting the BENCHMARK_OUTPUT_BASE env variable

explicitly setting the full directory path

filename

optional

the supplied filename prepended with a Timestamp that follows ISO 1806 format

if left empty, the timestamp alone will be used as the filename

run_met_directly

required

set to False

metplus_base

required

location of the METplus source code, specified by one of the following methods:

indicated as a full path e.g. /home/username/METplus

setting the METPLUS_BASE environment variable and use the current environment syntax like the following:

!ENV '${SOME_ENV_NAME}'

Make sure that the SOME_ENV_NAME environment variable is defined

system.conf

required

file location of the system.conf file

full path and file name

pre-condition: generate a valid system.conf file

wrapper_conf

required

the location of the METplus wrapper use case config file(s)

more than one use case can be run

full path and file name

pre-condition: generate the necessary wrapper config file(s)

num_runs

not yet supported

to be used for stress-testing/running command multiple times

set to 1

Note

A subdirectory under the output base directory (specified in benchmark_output_path) is created for each use case (based on the use case config filename).

Invoke the Python script benchmark.py to collect the benchmarking metrics

Note

Use Python 3.12 or above for running the benchmark.py script

Pre-conditions:

Running the Python script

Run the following from the command line (from the location where the benchmark.py file is located):

Note

An AssertionError message is printed to the terminal if the benchmark.py script is not run in the $BASE/MET/internal/scripts/benchmark directory.

cd $BASE/MET/internal/scripts/benchmark
python benchmark.py

Note

The intermediate summary_output.txt and detail_output.txt files generated by CTRACK are found in the directory from which the benchmark.py script was invoked (in the $BASE/MET/internal/scripts/benchmark directory). The final, consolidated report is saved as a .csv and a tabular .txt file as specified in the benchmark_output_path setting.

View results

The benchmark.py script creates .csv and .txt files with consolidated metrics from the summary and details tables (generated by the CTRACK tool). The summary_output.txt and detail_output.txt files generated during benchmarking are located in the directory from where the benchmark.py file was invoked.

View the consolidated metrics to identify potential performance enhancements. Refer to the CTRACK documentation to learn about the metrics collected, under the Metrics & Output section:

https://github.com/Compaile/ctrack?tab=readme-ov-file#metrics–output

Note

The consolidated files will be named filename_<timestamp>.csv and filename_<timestamp>.txt if the filename setting is specified in the benchmark.yaml configuration file. If the filename setting is not specified, then the files will be named <timestamp>.csv and <timestamp>.txt. An information file is also generated, capturing details about the benchmarking/profiling run such as Python version, timestamp, and other useful information to assist in trouble-shooting or re-creating a particular benchmarking run. The information file is named info_*<timestamp>*.txt, where <timestamp> is the timestamp of the benchmarking run.
Identify and implement any code changes to improve performance
Repeat steps 2-5 until desired performance enhancements are achieved

7.2. Keywords

Note

CTRACK
benchmarking
profiling
code profiler
code profiling