Release notes v3.15

The release 3.15 is the result of 5 months of work involving nearly 320 pull requests, featuring many significant optimizations and new features.

The complete list of changes is listed in the changelog:

https://github.com/gem/oq-engine/blob/engine-3.15/debian/changelog

Classical PSHA

The major highlight of the release is an optimization of point-like sources resulting in a speedup from 1.5 to 50 times, measured on various hazard models dominated by point-like sources. The speedup is especially large for single site calculations. It was obtained by using different optimizations, including a careful allocation of the arrays and writing numba-accelerated code for building the planar surfaces and computing the distances.

We reduced the memory consumption in the context objects, thus solving an out of memory issue in UCERF calculations.

We improved the ps_grid_spacing approximation which is now more precise. Setting the ps_grid_spacing parameter now sets the pointsource_distance parameter too and this the recommended way to use the feature.

We fixed a bug occuring when the grid produced by the ps_grid_spacing approximation degenerates to a line, by simply not collapsing the sources in that case.

The preclassical calculator has been optimized resulting in a 2-3x speedup in many models. We also improved the weighting algorithm by taking into account the number of GMPEs per tectonic region type and thus solving a slow task issue affecting the ESHM20 model.

At user request, we made it possible to filter the ruptures of a model by magnitude range. The way to do it is to use a magnitude-dependent maximum_distance. For instance using

maximum_distance = [(5.5, 70), (6, 100), (6.5, 150), (7, 200), (8, 250)]

will discard magnitudes < 5.5 and > 8. This is semantic change with respect to the past where outside magnitudes were not discarded.

There was a major (but internal) change to the way ruptures are stored in classical calculations with few sites. Now we store context objects rather than ruptures. The command oq compare rups has been changed accordingly, as well as the disaggregation and conditional spectrum calculators.

The change made it possible a performance improvement in disaggregation calculations and paved the way for disaggregation by rupture, since now reproducible rupture IDs are stored in the context arrays.

There was a lot of effort on the disagg_by_src functionality, which has been extended to mutually exclusive sources (used in the Japan model) and has some new features documented here: https://docs.openquake.org/oq-engine/advanced/classical_PSHA.html#disagg-by-src

The storing of the disagg_by_src array is much more efficient and there is a consistency check with the mean hazard curves which is always enabled.

Finally, we removed a logging statement that could cause an out of memory in some calculations: thanks to Chris Chamberlain of GNS-Science for discovering the issue.

Disaggregation

The disaggregation calculator was failing when a tectonic region type had a single magnitude, or when, due to rounding issues, incorrect magnitude bins were generated. Both issues have been fixed and we also changed the calculator to automatically discard non-contributing tectonic region types.

The calculator has been extended to work with mutually exclusive sources and now it is possible to perform disaggregations of the Japan model.

There is a new feature called epsilon_star disaggregation (in the sense of the PEER report https://peer.berkeley.edu/sites/default/files/2018_03_hale_final_8.13.18.pdf). For examples of use see the tests in qa_tests_data/disagg from case_8 to case_12.

In some models with nonParametric/multiFaultSources the calculators was returning spurious NaNs: this has been fixed.

lon,lat disaggregation with multiFaultSources was giving incorrect results: it has been fixed now.

Finally, Anne Hulsey from GNS New Zealand contributed two new kinds of disaggregation: Mag_Dist_TRT and Mag_Dist_TRT_Eps.

Hazard sources

There were several changes in multi fault sources and a few bugs were fixed while implementing the New Zealand model. As a new feature the SourceWriter writes multi-fault sources in HDF5 format rather than XML, thus drastically speeding up the reading time (by 3,600 times in the UCERF3 model). The data transfer in multi-fault sources has been drastically reduced too.

Sources have been extended to support parametric temporal occurrence models in their XML representation. We also have a way to serialize parametric temporal occurrence models inside the datastore. Thanks to such features the engine can now manage the negative binomial temporal occurrence model contributed by Pablo Iturrieta and used in the latest New Zealand model.

We added a check on sum(srcs_weights) == 1 for mutually exclusive sources that was missing.

We fixed a bug in upgrade_nrml when converting point sources with varying seismogenic depths into multipoint sources.

We changed the sourcewriter to round the coordinates to 4 digits after the decimal point. This helps in limiting the platform dependencies, since in general when the precision is not specified the XML generated on a Mac with M1 processor is different from the XML generated on a Linux/Windows Intel machine.

hazardlib

Tom Son contributed a bug fix to the Chiou & Youngs 2014 model: the Spectral Acceleration at T ≤ 0.3s was not being set correctly. He also added ztor, width and hypo_depth estimations to the Campbell and Bozorgnia (2014) model and suggested to add the z1pt0 parameter to the REQUIRES_SITES_PARAMETERS in the Boore (2014) model. He also improved the performance of Kuehn et al. (2020).

Julián Santiago Montejo Espitia contributed the Arteta et al. (2021) GMPE.

The Hassani and Atkinson (2018) GMPE has been added to hazardlib.

The Bahrampouri (2021) Arias Intensity GMPE with region-specific coefficients Cm and Ck has been added. Notice that the performance is expected to be poor since a geospatial query is performed for each rupture.

We implemented a parametric Magnitude Scaling Relationship called CScalingMSR for use in the New Zealand model.

We avoided building multiple times the same polygons in the Bradley (2013) model.

We now raise a clear error when passing a string instead of an IMT to the legacy method get_mean_and_stddevs.

We added a check for missing mags when calling the class GMPETable incorrectly from hazardlib and we added a property GMPETable.filename.

Risk

We added some restrictions on the risk IDs (they must be printable ASCII characters excluding #’”); further restrictions may be added in the future.

We added a check for inconsistent IDs between fragility and consequence functions.

We added a warning for missing risk IDs, such as an ID present in the structural_vulnerability file and missing in the occupants_vulnerability file.

We changed the internal serialization of risk functions in the datastore, as well as the storage of agg_curves-stats, src_loss_table and avg_losses: this is part of a large project to manage secondary generic secondary loss types.

At the moment the only kinds of secondary loss type implemented are insured losses (which have been reimplemented) and total losses (brand new). You can find examples in the event based risk tests, but essentially it is possible to write in the job.ini something like

total_losses = structural+contents

or

total_losses = structural+nonstructural+contents

and have the new loss type pop up in the CSV outputs as a new column. This is especially useful for computing total loss curves, or in situations were the insurance is based on the total losses obtained by summing different loss types.

The event based risk calculator has been refactored with some speedup (a few percent).

Thanks to Astha Poudel and Anirudh Rao, an experimental module to assess linear infrastructure risk connectivity.py has been added to the engine. Common metrics to measure network connectivity loss are automatically computed for scenario_damage calculations or event_based_damage calculations with an exposure containing the nodes and links/edges describing an infrastructure network. While detailed documentation for the module will be added once it graduates from the experimental stage, a working example for a water supply system with the scenario_damage calculator can be found in the QA tests directory: scenario_damage/case_15

The aggrisk output, that was experimental in previous version of the engine, has been finalized. Now it is consistent with the sum of the average losses even for event based calculations and some spurious warnings about agg_losses != sum(avg_losses) (happening in some situations) have been removed.

In presence of an exposure, a custom_site_id field is automatically added to the site collection, if not present already. It is computed as a geohash 8-characters long and it is meant for debugging purposes.

oq commands

The command oq download_shakemap has been replaced with a command oq shakemap2gmfs which is able to convert a ShakeMap coming from the USGS site into a set of ground motion fields suitable for scenario risk or damage calculations.

We added a command oq dbserver upgrade to create/upgrade the schema of the engine database without starting the DbServer. This is used in the universal installer.

We added a command oq compare risk_by_event to compare event loss tables. It raises an error if the GMFs are not compatible.

We extended the command oq engine with and option --sample-sources to reduce large calculations by sampling the source model (useful for debugging).

Bug fixes and new checks

A bug in event based calculations, where far away ruptures coming from multi fault sources were needlessly stored, has been fixed.

When using the command oq engine --run job.ini --exports=csv the realizations.csv output was not being exported. This is now fixed.

get_composite_source_model(oqparam) was raising an error in some cases: this has been fixed.

The counting of the logic tree paths (.num_paths) was incorrect in some situations and has been fixed. Moreover we raised the limit on the number of branches to 183.

The simplified logic tree implementation in the module hazardlib.lt has been fixed and documented in the advanced manual:

https://docs.openquake.org/oq-engine/advanced/logic_trees.html

We added a check for missing site parameters, for instance when accidentally passing a sites.csv file instead of a site_model.csv file.

We added a warning when starting from an calculation computed with an old version of the engine.

We added a warning for missing IMTs in ShakeMaps.

When using the --hc options extra fields of the site collection, such as the custom_site_id were lost: this is now fixed.

Installer and dependencies

Python 3.7 is officially deprecated and the next version of the engine will require Python 3.8, 3.9 or 3.10 to run.

The universal installer now officially supports the M1 processor with Python 3.9 (see https://github.com/gem/oq-engine/blob/engine-3.15/doc/installing/universal.md) and Ubuntu 2022 and any linux system with Python 3.10.

We fixed a few bugs: now the installer can be run from outside of the oq-engine directory, and there is a better error message when it is called with an unsupported Python version.

The installer also installs the standalone tools, which are visible in separate tabs in the WebUI. Before they had to be installed manually.

Our RPM packages use the universal installer internally.

numba has been added to the list of dependencies and it is now automatically installed with the engine. The engine still works without it, though calculations might be slower without numba.

We added NetworkX as a dependency: this is used only when performing risk infrastructure calculations.

We upgraded pandas to version 1.3.5, to avoid a bug breaking the risk infrastructure calculations.

We raised the toml module version to 0.10.2.

Other

Thanks to a grant from USAID the engine manual has been converted from LaTeX format to Sphinx format and it is now accessible online at the address https://docs.openquake.org/oq-engine/manual/ We also overhauled the advanced manual and documented the new features.

Modern laptops/PCs tend to have many cores but not enough memory per core. To avoid running out of memory the engine now automatically disables parallelization if less than 0.5 GB per core is available. We remind our users that for large calculations, 4 GB per core is recommended; also, hyperthreading should be disabled to increase the available memory per core.

If the DbServer does not start, it is now possible to debug the problem by accessing the database access directly; it is enough to set the environment variable OQ_DATABASE=local or to set dbserver.host = local in the openquake.cfg file.

We extended the WebUI to display the host name in the calculation list: this is useful when running calculations on a shared database.

At user request, we introduced three new environment variables OQ_ADMIN_LOGIN, OQ_ADMIN_PASSWORD, OQ_ADMIN_EMAIL that can be used to set the credentials of the administrator user in the WebUI.