Release notes v3.20#
Version 3.20 is the culmination of 3 months of work involving over 220 pull requests. It is aimed at users wanting the latest features, bug fixes and maximum performance. Users valuing stability may want to stay with the LTS release instead (currently at version 3.16.8).
The complete set of changes is listed in the changelog:
https://github.com/gem/oq-engine/blob/engine-3.20/debian/changelog
A summary is given below.
New Hazard Models#
Last year we integrated the UCERF3 model inside the USA model by reimplementing it in terms of multifault sources. We recently realized that (non-intuitively) in the case of few intensity measure types the calculation was running out of memory, using more than 4 GB of RAM per core.
Solving the issue required a lot of work (over 200 man-hours) on the classical calculator, including changing the internal storage of the multifault sources in HDF5-format, compressing the transferred data, transferring 32 bit rates instead of 64 bit probabilities, storing the rates more efficiently, changing the postprocessing phase producing the statistical hazard curves by taking advantage of the tiling functionality and more. Hower, not the memory usage of the classical calculator, depending on the calculation, is down from 0% to 50%.
Moreover, we were able to add support for the 2022 revision of the New Zealand National Seismic Hazard Model (NZ NSHM 2022), which also features multifault sources. While the model had been implemented with the OpenQuake engine from the very beginning, its calculation was extremely challenging and required several tricks for it to run. It is only with engine 3.20 that the calculation runs out-of-the-box and efficiently.
Since we improved the way the logic tree is stored in memory and we
implemented a better algorithm for computing the means if the
parameter use_rates is set (we measured a 133x speedup in a
calculation with ~260,000 sites) we can now run
calculations with millions of realizations. Therefore, it is quite
possible to run the New Zealand model - which has “only” 979,776
realizations - on a single machine. Computing the quantiles is still
impossible in practice, since you will run out of memory during the
postprocessing phase.
hazardlib#
The major thing in hazardlib was a refactoring removing **kwargs
from the signature of hundreds of GMPEs. As a consequence, now each
GMPEs knows the arguments it requires and if there is a typo in the
gmpe logic tree file a clear error is raised. Before, any mispelled
parameter was silently ignored. In particular, we discovered that in
the seismic hazard model for Europe, the parameter
theta6_adjustment was mispelled as theta_6_adjustment and that
went unnoticed for years
(#9486). Most sites
are unaffected by the parameter, but some are. If you have an incorrect
version of the EUR model now you will immediately get an error
ValueError: FABATaperSFunc.__init__() got an unexpected keyword
argument 'theta_6_adjustment' in file EUR/in/gmmLT.xml
that you can solve by renaming theta_6_adjustment -> theta6_adjustment in
the file gmmLT.xml.
We also fixed the passing of parameters in the NRCan15SiteTerm, without
any impact on the results, since currently all models in the GEM mosaic do not
require passing parameters to the underlying GMPE via the NRCan15SiteTerm.
Marco Pagani added a new uncertainty
in the logic tree processor, called moment maxMagGRRelativeNoMoBalance;
an example of usage is in the test logictree/case_84.
He also extended the ChiouYoungs2014 class to accept many optional
parameters, notably use_hw, add_delta_c1, alpha_nm,
stress_par_host, stress_par_target, delta_gamma_tab as
specified by Boore et al. (2022).
Fatemeh Alishahiha contributed a new class
AbrahamsonSilva1997Vertical adding a vertical component to the
classical Abrahamson Silva GMPE and also an entirely new class
GhasemiEtAl2009 implementing the Ghasemi et Al. GMPE for Iran.
Risk#
We saved a lot of memory and disk space in conditioned GMFs scenarios, making them also faster in many situations. This is a HUGE improvement since many calculations that before were impossible are now possible. We also improved the error checking for calculations with too many sites.
We implemented a first version of Post-Loss Amplification (PLA). The feature is documented, but it is in the experimental stage for the moment. See https://github.com/gem/oq-engine/issues/9633 for more.
We fixed the avg_losses-stats exporter in the case of a single realization
for classical_risk calculation: #9579
Bug fixes#
We fixed a bug when importing a file gmf_data.csv: in rare situations
with filtered site collections one could end up with site IDs in
not present in the site mesh.
In the engine, longitude and latitude are rounded to 5 digits. However,
sometimes they were rounded by using numpy.round and sometimes by
using Python round. That causes surprises in rare situations, producing
a different a different rounding depending if the sites were listed
directly in the job.ini file or read from a sites.csv file.
This has been fixed.
We fixed a long standing bug, so that the full logic tree object could
not be properly serialized in the datastore in some cases, i.e. when modifying
simpleFaultGeometryAbsolute, complexFaultGeometryAbsolute and
characteristicFaultGeometryAbsolute). The bug was a low priority since
such features are not used in any hazard model in the mosaic, however
now it has been finally fixed.
Using some special characters in the branchID tags of the source model
logic tree caused some tricky errors, as it was happening in the New
Zealand model. We have solved the issue by raising an early error if any
of the forbidden characters .:; is found.
When using a magnitude-dependent maximum distance with a missing tectonic region type, the engine was giving a fake error. Now the engine simply ignores the missing tectonic region type if there are no sources associated to it, consistent with the behavior of version 3.16.
The mixture-model feature (tested in classical/case_47) was introduced 4 years ago and nearly immediately broken by a refactoring since the test was not actually testing due to a typo. The problem has now been fixed and the feature is working as intended.
There was an error when running scenarios with a ModifiableGMPE over a
table-based GMPE, since the mags_by_trt dictionary was not passed properly.
It is now fixed.
Finally, we removed two error-prone caches for the gsim logic tree and the exposure, by avoiding the need for them.
New checks#
In the presence of millions of assets and events, calculating the average losses can send the server out of memory; there is now an early check warning the user of the problem and suggesting a workaround.
If the user forgets to specify the residents field in the exposure fieldMap,
there was a clear error message but only after computing the GMFs, i.e. too
late. Now the error is raised before starting the calculation.
When a required site parameter is not provided, the engine is now raising a clear error message instead of giving NAN results.
There were rare situations where complex fault sources were deemed invalid even when correctly specified. This has been fixed in https://github.com/gem/oq-engine/pull/9596.
If the user forgets to specify the gsim parameter or the gsim_logic_tree_file parameter in a scenario calculation, he could get a misleading error message. Now the message is very clear.
We added a check to forbid having individual_rlzs=true and
collect_rlzs=true at the same time. Before, collect_rlzs was
automatically set for sampling calculations and overriding
individual_rlzs=true, in a way very surprising for the final
user. Now he gets an error such as
InvalidFile: you cannot have individual_rlzs=true with collect_rlzs=true
The classical_risk and classical_damage calculators are able to
start from imported hazard curves stored as CSV files. Such files must
contain probabilities, i.e. floats in the range 0 to 1. However, that
was not checked, causing NaN values to be generated in the outputs.
Now the user gets a clear error message at import time.
Aristotle project#
Aristotle is a project to provide Multi-Hazard advice to the European Research Coordination Centre in case of disasters. Currently, it is in a development/experimental status and it is meant to be tested but not trusted.
To support the needs of the project, the WebUI has grown an APPLICATION_MODE=”ARISTOTLE” which allows users to submit a ShakeMap ID and to produce the associated scenario risk outputs by using GEM’s exposure, vulnerability functions and ground motion models. The users are automatically notified when the calculations are done (currently for a single ShakeMap ID the engine can perform multiple calculations, since the earthquake can affect countries with different taxonomy mappings).
In ARISTOTLE mode, a couple of simple plots are generated: for the moment, one with the (geometric) average GMFs and one with the assets. They are used for debugging and are subject to changes in future versions of the tool.
The procedure for downloading the USGS rupture parameters has been enhanced
so that if the file “rupture.json” is missing (because the USGS has not
generated it yet) the “finite-fault” parameters are used instead.
Moreover, it is possible to upload a custom file rupture_model.xml,
if the user possesses additional or different information than the USGS
or is interested in trying alternative models.
oq commands#
We added a command oq mosaic aristotle, which is used
to run multiple ARISTOTLE calculations in a batch in our
automatic tests.
We improved the commands oq info imt and oq info gsim: now it is
possible to specify the IMT/GSIM and get information about it (i.e. the
docstring of the corresponding class).
We improved the command oq info mfd, which is now able to also print
the docstring of the specified MFD class, and added a command oq info msr
with similar features for the MSR classes.
We improved the command oq plot avg_gmf, which is now also displaying
country borders. Moreover, it is not showing the seismic stations in case of
conditioned GMFs calculations, since they were confusing.
We fixed the command oq zip that was not zipping the exposure.csv files.
The command oq show delta_loss has been enhanced and documented in the manual.
We now raise a clear NotImplementedError when trying to convert
griddedSurfaces with the command oq nrml to csv|gpkg.
Documentation#
There was a lot of work on the documentation, in particular about Windows installations and event-based outputs, which were both severely outdated.
Many features implemented years ago have been finally documented, including a description of all site parameters, a description of Gridded ruptures, an explanation of the disaggregation outputs and how to convert them into the traditional Bazzurro and Cornell view.
A function calc_gmf_simplified has been added to explain how
GMFs are computed.
A demo ScenarioCase3 documenting the new feature rupture_dict (introduced
in engine-3.19) has been added.
Other#
There was as usual some work for the AELO project. In particular, in AELO mode, we changed the hazard curves and UHS exporters into a format more Excel-friendly, at user request.
We added to the WebUI the ability to disable the warning about
newer engine releases being available with a flag DISABLE_VERSION_WARNING
in settings.py.
We added to the WebUI outputs page buttons to display avg_gmf
plots for all the available IMTs.
Finally, there were a couple of changes affecting development installations:
we introduced a maximum limit of 90 lines of code and 16 arguments per function/method. The limits are enforced by the tests every time a user makes a pull request to the engine repository. They are meant to avoid excessive degradation of the code base quality.
at the start of the WebUI a warning is printed if the installed dependencies are different from the expected ones, to point out possible problems while developing.