Release notes v3.10#
This is a major release featuring several optimizations, new features, and a bug fixes. Over 300 pull requests were merged.
For the complete list of changes, see the changelog: https://github.com/gem/oq-engine/blob/engine-3.10/debian/changelog
Here are the highlights.
Disaggregation#
There was a substantial amount of work on optimizing the disaggregation calculator in the case of many sites and/or many realizations. Now it requires a lot less memory and it is significantly more performant than before (at least 3 times better then engine 3.9 in test calculations for Colombia and the Pacific Islands). However, for single-site and single-realization calculations you will not see much of a difference.
The distribution mechanism and rupture storage have been changed (ruptures are now distributed and stored by magnitude), thus reducing both the data transfer and the time required to read the ruptures. The effect is spectacular (orders of magnitude) in large calculations.
The way disaggregation outputs are stored internally has changed
completely, resulting in speedups up to orders of magnitude in the
saving phase. Also, we are logging much less when saving the
disaggregation outputs. There was another huge speedup in the
extraction routines for all outputs, including the disagg_layer
output which is use by the QGIS plugin. Such output has been extended
to include IML information and more.
The disagg_by_src functionality, temporarily removed in engine 3.9,
has been re-implemented and there is a new extractor for it. However,
there are a couple of constraints: in order to use it, you must
convert the point sources of your model (if any) into multipoint
sources; moreover, the size of disagg_by_src matrix (num_sites x num_rlzs x
num_imts x num_levels x num_sources) must be under 4 GB.
If disaggregation by epsilon is not required and the parameter
num_epsilon_bins is incorrectly set to a value larger than 1, now
the engine automatically corrects the parameter (by warning the user),
thus saving a lot of wasted calculation time.
We fixed a bug for the case of zero hazard: having zero hazard for a single return period caused the disaggregation to be zero for all periods.
We fixed a subtle bug for magnitudes on the edge of a magnitude bin, causing some small difference in the disaggregation by magnitude outputs.
We changed the algorithm for the Lon_Lat bins, to make sure that
the number of bins is homogeneous across sites, thus changing the
numbers for the Lon_Lat disaggregation.
Setting both rlz_index and num_rlzs_disagg is now an error, since
the two options are mutually exclusive.
Setting max_sites_disagg to a value lower than the total number of
sites is now flagged as an early error (before the error was raised in
the middle of the computation).
We fixed the IML interpolation to be consistent with the
algorithm used for the hazard maps: the change may cause slight
differences in the numbers, in all kind of disaggregations except the
ones using the parameter iml_disagg. Moreover, the parameter
poes_disagg in the job.ini file is now deprecated and you can just use
the parameter poes used for the hazard maps.
Multi-rupture scenarios#
At the end of a year-long effort, we have finally unified the scenario and event based calculators and introduced a full CSV format for the ruptures, i.e. a format containing also the geometries. The full CSV format is documented here. It should not be confused with the old CSV format which is missing the geometry information and therefore cannot be used to start scenario calculations. The new format is for the moment experimental, not guaranteed to stay the same in future releases. It completely supersedes the experimental TOML format for the ruptures introduced some time ago and that now has been removed.
It is now possible to run a slow event based calculation, export the ruptures in the full CSV format, trim the ruptures according to some criterium (for instance consider only the ruptures that dominate the risk and discard the irrelevant ones) and then perform fast scenario calculation starting for the reduced ruptures. This is of huge importance for any risk analyst.
Multi-rupture scenarios can also be used to study the uncertainties involved in a single-rupture scenario calculation. For instance you can change the parameters of the base rupture, generate a bunch of ruptures and then run a single multi-rupture scenario calculation. This is much more convenient than running multiple independent calculations like it was necessary in the past.
Risk calculators#
A much requested feature from our users was the ability to compute
statistics for scenario_risk and scenario_damage calculations in
presence of multiple GMPEs. Now that is possible and it actually
enabled by default, as a consequence of the event_based/scenario
unification.
We optimized the scenario_damage/event_based_damage calculators for
the case of many sites by returning one big output per task instead of
thousands of small outputs. The effect is significant when there are
thousands of sites.
We reduced the time required to read the GMFs and we avoided sending multiple copies of the risk model to the workers, resulting in a good speedup for calculations with a large risk model.
Risk calculators are now logging the mean portfolio loss, for user convenience.
We added an exporter for the src_loss_table output giving the cumulative
loss generated by each hazard source.
We made more hazard and risk outputs readable by pandas. Since the GMFs are readable with pandas we have removed the GMF npz exporter which was slow and inconvenient.
We made the scenario damage and classical damage outputs consistent with the event based risk outputs by removing the (unused) stddev field. See https://github.com/gem/oq-engine/issues/5802.
The event based damage calculator, introduced experimentally in release
3.9, is now official. We fixed a bug in the number of buildings in
the no_damage state, which was incorrect. We fixed
a bug in the calculation of the average damage, which is now
correctly computed by dividing by the investigation time. We restored
the ability to perform damage calculations with fractional asset
numbers, by using the same algorithm as in engine 3.8 and previous versions.
Finally, we added an official demo for the event based damage calculator
and we documented it in the engine manual.
Other new features/improvements#
The logic tree sampling logic has been revised and the seed algorithm changed, therefore in engine 3.10 you should not expect to get the same branches sampled as in engine 3.9. The changed was needed to implement four new sampling methods which are documented here: https://docs.openquake.org/oq-engine/advanced/sampling.html
The new sampling methods will receive more work and experimentation in the future.
It is possible to introduce a custom_site_id unique integer parameter
in the site model. The most common use case for this feature is to
associate a site to a ZIP code when computing the hazard on a city.
The magnitude-dependent maximum distance feature has been restored and it is documented here: https://docs.openquake.org/oq-engine/advanced/common-mistakes.html#maximum-distance
The rupture-collapsing feature now works also for nonparametric ruptures. It is still experimental and disabled by default.
The engine was parsing the source_model_logic_tree and the
the source models multiple times: it has been fixed.
There was additional work on the amplification of hazard curves with the convolution method and on the amplification of GMFs. Various bugs have been fixed, and we added an experimental and slow implementation of the kernel method for computing PSHA using amplification functions. Such features are still experimental.
We included in the engine some code to compute earthquake-induced secondary perils, like liquefaction and landslides. This part is still in an early stage of development and will be extended/improved in the future.
hazardlib/HTMK#
F. J. Bernales contributed the Gulerce et al. (2017) GMPE, Stewart et al. (2016) GMPE, Bozorgnia & Campbell (2016) GMPE.
G. Weatherill contributed the heteroskedastic standard deviation model for Kotha et al. (2020). He also fixed a bug in mixture models in logic trees with multiple GMPEs and he added PGV coefficients to USGS CEUS GMPE tables, where possible.
M. Pagani contributed a ModifiableGMPE class
which is able to modify the inter_event parameter of a GMPE. He also
contributed an implementation for the Zalachoris & Rathje (2019)
GMPE. Finally, he fixed a bug in the ParseNDKtoGCMT parser in the HMTK
and ported the method serialise_to_hmtk_csv implemented in the
corresponding class of the catalogue toolkit.
M. Simionato fixed a subtle bug that caused a syntax error when reading
stored GSIMs coming from subclasses of GMPE redefining
__str__ (signalled by our Canadian users).
The sourcewriter has been improved. Now it calls
check_complex_fault when serializing a complex fault source, so that
incorrect sources cannot be serialized, as it was happening before.
Moreover, it automatically serializes nonparametric griddedRuptures
in a companion HDF5 file with the same name of the XML file. The
source reader can read the companion file, but it also works with the
previous approach, when everything was stored in XML. Since
griddedRuptures are big arrays the serialization/deserialization in
HDF5 format is much more efficient than in XML format.
Bugs#
We fixed a serious numeric issue affecting classical calculations with nonparametric ruptures. The effect was particularly strong around Bucaramanga in Colombia (see https://github.com/gem/oq-engine/issues/5901). The hazard curves were completely off and the disaggregation was producing negative probabilities.
In some situations - for logic trees using applyToSources and sampling -
there was a bug so that the logic tree could not be deserialized from its
HDF5 representation in the datastore. That affected the calculation of
the hazard curves and maps.
We fixed an issue with noDamageLimit not being honored in continuous
fragility functions.
We fixed a h5py bug in large ShakeMap calculation, manifesting as the
random error 'events' not found.
We fixed an issue withe minimum_magnitude parameter not being honored
in UCERF calculations.
We fixed a bug when a rupture occurs more than 65535 times, affecting
scenario calculations with a large number of simulations. Similarly,
the fields year and ses_id in the events table have been extended
to support 4-byte integers instead of 2-byte integers.
We fixed a bug affecting Windows users preparing CSV exposures with Excel, which introduces a spurious Byte Order Mark (BOM).
We fixed a bug when saving arrays with string fields in npz format.
New checks and warnings#
Now the number of intensity measure levels must the same accross intensity measure types, otherwise an error is raised. In the past there was simply a deprecation warning.
We added a warning about suspiciously large complex fault sources, so that
the user can fix the complex_fault_mesh_spacing parameter.
We added a warning against numeric errors in the total losses for the
ebrisk and event_based_risk calculators, like “Inconsistent total
losses for structural, rlz-96: 54535864.0 != 54840944.0”.
If a source model contains different sources with the same ID a warning
is now printed. In the future this will likely become an error and the source
model will have to be fixed with the command oq renumber_sm. Then the
source ID will become a true ID and it will become possible to implement
source-specific logic trees.
We added parameter a max_num_loss_curves with a default of 10,000. Without
this limit it would be easier for the user to accidentally generate millions
of aggregate loss curves, causing the calculation to go out of memory. You
can raise the limit, if you really need.
At user request, we raised the limit on the asset ID length from 20 to 50 characters.
We added a check for missing intensity_measure_types in event based
calculations, otherwise a calculation would complete successfully but without
computing any ground motion field.
oq commands#
oq plot has been improved for disaggregation outputs, making it possible
to plot all kinds of outputs and also to compare disaggregation plots.
We added a command oq plot vs30? to plot the vs30 field of the
site collection.
We added a command oq nrml_to to convert source models into CSV and/or
geopackage format. This is useful to plot source models with
QGIS and it is meant to replace the command oq to_shapefile which has
many limitations.
We added a command oq recompute_losses <calc_id> <aggregate_by>: this is
useful if you want to aggregate an ebrisk calculation with respect to
different tags and you do not want to repeat the whole calculation.
We fixed a 32 bit/64 bit bug in oq prepare_site_model affecting the case
when the file sites.csv is the same as the vs30.csv file.
The semantic of the command oq engine --run --reuse_hazard has
been changed: before it was finding an existing hazard calculation (if
any) from the checksum of the input files and using it, now it is just
reusing the cached source model, if present. The change was necessary
since the checksum of the calculation often changes between version;
moreover reusing the GMFs generated with a previous version of the
engine is often inconsistent, i.e. it will not give the same results
since the details of the seed generation may change.
The oq engine command has been enhanced and it is now possible to run
multiple jobs in parallel as follows:
$ oq engine --multi ---run job1.ini job2.ini ... jobN.ini
This is useful if you have many small calculations to run
(for instance many scenarios). Without the --many flag the
calculations would be run sequentially, as in previous versions of the engine.
WebUI/WebAPI/QGIS plugin#
When running a classical calculation with the parameter poes
specified, the engine now produces hazard maps in .png format that are
visible through the WebUI by clicking the button on the bottom left
side called “Show hazard map”. This is meant for debugging purposes
only, the recommended way to visualize the hazard maps is still to use
the QGIS plugin.
There is a new API for storing JSON information inside .npz files, which
is being used to store metadata. The QGIS plugin has been adapted to take
advantage of the new API and the command oq extract has been changed
to save in .npz format and not in .hdf5 format.
We changed the call to /extract/events to return only the relevant events,
i.e. the events producing nonzero GMFs, in case a minimum_intensity is
specified.
The /extract/sitecol API has been extended so that it is possible to
extract selected columns of the site collection, for instance
/extract/sitecol?param=custom_site_id, if a custom_site_id is defined.
Packaging#
We removed the dependency from PyYAML and added a dependency for GDAL.