Usage
The snowdrift CLI accepts NWP model output in GRIB format, and is intended to be configurable for accepting a variety of different local area model (LAM) forecast outputs.
Input NWP parameters
The model requires the following GRIB parameters to be available in the input model in order to derive the snowdrift strength output:
- wind speed at 10m level or u/v wind components
- surface temperature (temperature at level 0m)
- snowfall hourly rate or accumulated snowfall
Output NWP parameters
The software produces the following 4 new model parameters:
- drift - snowdrift strength value
- snowage - age in hours of top level snow
- driftac - accumulation of significant snow drift
- mobility - snow drift mobility factor of top level snow
Please refer to the Configuring section below for instructions on how to adjust GRIB input, GRIB output and model tuning.
--out [grib_file]
Writes out a total of 8x GRIB parameters, used for assimilation:
- 4x drift params:
- drift
- snowage
- driftac
- mobility
- 4x extra parameters for assimilation only:
- temp
- snowground
- snowfall
- wind
--out-drift [grib_file]
Writes only the 4x snow drift parameters, used for delivery of snowdrift results:
- drift
- snowage
- driftac
- mobility
PNG output
PNG output is very basic and mainly used for testing purposes:
$ snowdrift path/to/fc/files* --out image_folder/test --format PNG
A folder image_folder/
will be created and filled with rendered PNG
images for drift, snowage, driftac and mobility.
Simple usage
The most basic usage is to print out the snowdrift with --out <grib file>
assuming default --format GRIB
.
$ snowdrift path/to/fc/files* --out snowdrift.grb
Note however, for operational use should always assimilate last available snowdrift --out
result when calculating new snowdrift, see next section.
GRIB with assimilation
Use the --assim <prev_snowdrift.grb>
argument to take in previous forecast run output from
--out <snowdrift>
.
This ensures that the snow state is preserved between snowdrift runs.
This is required for operational use of the model:
$ snowdrift path/to/fc/files* --out snowdrift.grb --assim prev_snowdrift.grb
GRIB for delivery
To also print out a forecast that includes only the 4x necessary snowdrift
parameters for model delivery, use the --out-drift <delivery.grb>
argument:
$ snowdrift path/to/fc/files* --out snowdrift.grb --out-drift delivery.grb --assim prev_snowdrift.grb
# deliver the delivery.grb result, e.g.
$ scp delivery.grb <destination>
Note --out-drift
GRIB output is only for delivery and cannot be used for assimilation
on next run. Use --out
to generate complete GRIB fc set for assimilation.
You can use a combination of the two to generate assimilation GRIB as well as delivery
GRIB.
Assimilation
It is necessary to continously assimilate the previous snowdrift GRIB output
in order to inherit the state of the snow cover from previous forecasts.
This is achieved by passing the option --assim <prev snowdrift>.grb
to
the CLI application. WARNING: If assimilation is skipped, the snowdrift forecast will reset itself on
each run, resulting in significantly reduced accuracy in the forecast run.
verify assimilation
You can verify that the assimilation step is working correctly by
reading the output logs. With --assim
option, a secondary dataset will be
loaded and logs should indicate if the data is being incorporated in the
new forecast:
2021-03-31 14:02:28,786:INFO:loading assimilation data - previous snowdrift run
2021-03-31 14:02:28,869:INFO:collecting from 'prev_drift.grb'
2021-03-31 14:02:28,872:INFO: - found 'drift'
2021-03-31 14:02:28,876:INFO: - found 'snowage'
...
-----------------------------------
Loaded Assimilation Data
-----------------------------------
drift:
steps: 10
min/max: -1.0/4.9 none
start/stop: 2021-03-27 00:00:00/2021-03-27 09:00:00
snowage:
steps: 10
min/max: -1.0/9.0 hours
start/stop: 2021-03-27 00:00:00/2021-03-27 09:00:00
...
-----------------------------------
2021-03-31 14:02:29,327:INFO:assimilating previous snowdrift forecast
2021-03-31 14:02:29,327:INFO: - OK
2021-03-31 14:02:29,327:INFO:snow drift algorithm
2021-03-31 14:02:29,327:INFO: - assimilation in effect, starting algorithm at step 1
2021-03-31 14:02:29,327:INFO: - snowdrift calculation step 1
2021-03-31 14:02:29,342:INFO: - snowdrift calculation step 2
...
2021-03-31 14:02:29,477:INFO: - removing time -1 assimilation step
...
Production
The following outline in production will ensure that the previous forecast results are continously assimilated on each run:
# initial run, without assimilation:
$ snowdrift latest/files* --out result.grb
$ cp result.grb prev_result.grb
# on each new forecast run, execute:
$ snowdrift latest/files* --assim prev_drift.grb --out result.grb
$ cp result.grb prev_result.grb
Alternatively, you can also write out a snowdrift forecast containing only the necessary 4x snowdrift parameters for display/delivery:
# initial run, without assimilation:
$ snowdrift latest/files* --out result.grb --out-drift delivery.grb
$ cp result.grb prev_result.grb
$ scp delivery.grb <destination>
# on each new forecast run, execute:
$ snowdrift latest/files* --assim prev_result.grb --out result.grb --out-drift delivery.grb
$ cp result.grb prev_result.grb
$ scp delivery.grb <destination>
In-Memory Compression
The snowdrift
client program is currently designed to load
all necessary parameters into memory in order to
easily sort the data before snowdrift processing.
The reason for this current architecture decision is that GRIB forecast
data does not always come in sorted order.
Depending on input forecast domain size, the current implementation may consume more memory than is easily accessible on some light-weight processing nodes. However, for most typical LAM domains this processing should not exceed a few GB in RAM.
To mitigate the memory usage problem, an in-memory forecast
data compression option --compress
has been provided,
for lossless data compression using the python bcolz
module. This option will typically reduce memory usage by a factor of 3,
but also reduce processing speed by a similar factor.
To apply in-memory compression, simply add the --compress
argument
to the CLI snowdrift application:
snowdrift latest/files* --compress --assim prev_result.grb --out result.grb --out-drift delivery.grb
TODO: Further improvements in memory usage of the software can be made by keeping a reference to data in memory on disk during processing. Depending on user requirements this can be implemented without significant further work on the software.
Configuring
The snowdrift software ships with a default configuration for input, snowdrift tuning and output, compatible with the Harmonie NWP model being run in Iceland by the Icelandic Meteorological Office.
Other model inputs may require a granular selection of input parameters, tuning of the model parameters and configuring output GRIB parameters depending on applications.
For this reason the --config <config_file>.yml
option allows for detailed configuration of the snowdrift model run.
The following example config file can be edited as appropriate and passed to the snowdrift
application. This example configuration file is hosted in the software development folders under config/example_config.yml
.
Example config:
# defines input model parameters
InputParameters:
# if accumulated snow in input model
snowac:
id: 'heightAboveGround:0:184'
unit: 'kg m**-2'
# alternatively if have snow fall rate,
# snow:
# id: '<type of level>:<level>:<indicator of parameter>'
snowground:
id: 'heightAboveGround:0:65'
temp:
id: 'heightAboveGround:0:11'
# if u/v wind in input model
wind-u:
id: 'heightAboveGround:10:33'
wind-v:
id: 'heightAboveGround:10:34'
# alternatively if have total wind speed,
# wind:
# id: '<type of level>:<level>:<indicator of parameter>'
# defines snowdrift model tuning (can skip this)
ModelTuning:
snow_cover_threshold: 1.0
snow_fall_threshold: 0.1
wind_speed_normalization: 12.0
wind_zero_drift: 6.0
mobility_snowage_reduction:
snowage: 24
mobility: 0.6
mobility_driftac_reduction:
driftac1: 2.0
mobility1: 0.6
driftac2: 6.0
mobility2: 0.3
# defines GRIB output (can skip this)
OutputParameters:
type_of_level: 'heightAboveGround'
level: 0
# the snowdrift results
snowage_parameter: 145
driftac_parameter: 146
mobility_parameter: 147
drift_parameter: 148
# for assimilation purposes store these pars,
# they get loaded for time -1 assim step.
temp_parameter: 160
snowground_parameter: 161
snowfall_parameter: 162
wind_parameter: 163
InputParameters
It is necessary to provide identifiers for the following parameters when ingesting forecast GRIB data for the snowdrift model:
- 'wind' wind speed at 10m level in m/s
- or 'wind-u' and 'wind-v' wind components in 'm/s'
- 'temp' surface temperature (level 0m) in °C
- units K should be automatically converted to °C, check the data summary output when running the software
- 'snow' snowfall hourly rate in mm/hour (kg m-2/hour)
- or 'snowac' accumulated snowfall through the input forecast
The syntax for selecting these input parameters is:
<parname>:
id:'<type of level>:<level>:<indicator of parameter>'
ModelTuning
Use this config section in order to tune some key parameters within the snowdrift model. The defaults that are provided have worked well with the Icelandic Met Office's Harmonie NWP for Iceland.
- snow_cover_threshold defines in mm (kg m-2) snow water equivalent, the threshold for snow-covered ground. Areas above this threshold will be used in snowdrift calculations.
- snow_fall_threshold defines in mm/hour (kg m-2/hour) snow water equivalent, the threshold for identifying new snowfall for resetting top layer snowage, snow mobility and driftac.
- wind_speed_normalization is a factor for tuning the snow drift strength in relation to wind speed. See snow drift equation, $$ \text{drift} = \left(\frac{\text{wind}}{\text{wind}_{norm}}\right)^3 \times \text{mobility}$$ in model description papers below.
- wind_zero_drift defines the minimum wind speed in m/s for drift value calculations. Below this value there is no drift.
- mobility_snowage_reduction defines a reduction in snow mobility after certain number of hours of top layer snow age.
- snowage threshold in hours for reduction
- mobility new reduced mobility value
- mobility_driftac_reduction defines a reduction in snow mobility after a certain amount of accumulated drift.
- driftac1 first threshold, reducing to value of
mobility1
- mobility1 mobility for
driftac1
threshold - driftac2 second threshold, reducing to value of
mobility2
- mobility2 mobility for
driftac2
threshold
- driftac1 first threshold, reducing to value of
Theoretical basis for these tuning options is described in the following papers:
- International Road Weather Conference report 'English'
- Roads Authority report, chapter 7 'Icelandic'
OutputParameters
The output parameters sections defines unique ids for GRIB forecast output for the 4 calculated snowdrift parameters. This usually does not require configuration unless there are conflicts with other parameter definitions being used in plotting and Weather GIS tools used by the user.
- type_of_level should usually be heightOverGround or 105
- level should usually be level 0 / 0m over ground
- snowage_parameter unique number, indicator of parameter
- driftac_parameter unique number, indicator of parameter
- mobility_parameter unique number, indicator of parameter
- drift_parameter unique number, indicator of parameter
Temporary parameters for storing data necessary for assimilation --assim <prev forecast>
:
- temp_parameter unique number, indicator of parameter
- snowground_parameter unique number, indicator of parameter
- snowfall_parameter unique number, indicator of parameter
- wind_parameter unique number, indicator of parameter
Interpretation
The following interpretation of the snow drift strength values were originally recommended in the original development of the model and have been used at the Icelandic Meteorological Office for classifying severity of snow drift from the model:
drift value | interpretation |
---|---|
drift < 0.09 | None |
0.09 =< drift =< 0.21 | Low |
0.21 < drift < 0.50 | Moderate |
0.5 =< drift | High |
The choice of interpretation of drift strength is another consideration for tuning the model, but as presentation of the output from the snowdrift software.
In Iceland the following color scale definition has been used for the model, but it is completely up to the user how he wishes to present the output from the model:
driftScale = (
(-1,'white'),
(0,'#0d0'),
(0.09, '#0dd'),
(0.21, '#dd0'),
(0.5, '#d00'),
)
CLI help
Check out the current input options to the executable with --help
arg
$ bin/snowdrift -h
usage: snowdrift [-h] [--config FILENAME] [--assim FILENAME] [--out FILENAME]
[--out-drift FILENAME] [--format FORMAT] [--show STEP] [--verify]
[--compress]
forecast_files [forecast_files ...]
positional arguments:
forecast_files (required) input NWP forecast in GRIB format
optional arguments:
-h, --help show this help message and exit
--config FILENAME (optional) config file for configuring the input parameters
and to adjust tuning of the snowdrift model
--assim FILENAME (recommended) assimilate previous snowdrift result into new
snowdrift forecast, accepts snowdrift GRIB files from previous
run
--out FILENAME save result to file(s) (can use this, if GRIB format, for
--assim on next run)
--out-drift FILENAME save file(s) with only snowdrift parameters results (can not
use this, if GRIB format, for --assim on next run)
--format FORMAT output format ['GRIB' (default) | 'PNG']
--show STEP show snow drift result at STEP (-1 is last step)
--verify only read input data and verify that necessary parameters are
available, or derived from input, for snowdrift calculation
--compress applies data compression, saving on RAM usage