Release notes v3.6#
This release features several major new features (including completely revised disaggregation, automatic optimization of duplicated sources, fast exposure importer and taxonomy mapping) and lots of improvements, new checks and bug fixes. Nearly 200 pull requests were merged. For the complete list of changes, see the changelog: https://github.com/gem/oq-engine/blob/engine-3.6/debian/changelog
Disaggregation#
The most relevant development on the hazard side was the work on the
disaggregation calculator. We changed substantially the business
logic. While in previous versions of the engine we were disaggregating
for all possible realizations in order to compute disaggregation
statistics, in this version we gave up on statistics. Instead, we
disaggregate only for a specific realization. The realization can be
specified by the user with the rlz_index parameter in the job.ini
file, or it can be determined automatically by the engine as the
realization closest to the mean curve for the given disaggregation
site.
Moreover, now the disaggregation calculation works like a post-calculator
(i.e. with the --hc option) and it is able to reuse information
computed in its parent calculation: the net effect is that it is
always faster than the corresponding classical calculation while in
the past it was several times slower. We also fixed a couple of
performance bugs: there was a slow operation truncnorm.cdf
in an inner loop and ruptures outside the integration distance
were not discarded.
Finally, we changed the file names of the disaggregation outputs.
For models with thousands of realizations, the disaggregation can easily be thousands of times faster than before.
Classical PSHA calculator#
The engine is now smart enough to recognize duplicated sources
appearing in different branches of the composite source model and to
avoid redundant computations. Because this optimization is always on,
the flag optimize_same_id_sources has been removed, as it has now
been rendered useless. There are several models in the
hazard mosaic with duplicates sources and the new optimization has a
significant impact on those. Moreover the demo LogicTreeCase2ClassicalPSHA
has become an order of magnitude faster than before thanks to the reduction
of the duplicated sources.
There was a big improvement in the computation of the statistical hazard curves which now is not only faster, but uses a lot less memory than before. The trick was to consider one site at the time, instead of a block of sites. As a consequence it is now possible to consider tens of thousands of realizations for hundreds of thousands of sites without requiring terabytes of RAM. Moreover the data transfer has been reduced by storing some auxiliary information in the datastore and reading it from the workers instead of transferring it via celery/rabbitmq.
There was a substantial change in the way the tasks are distributed for a
classical calculation. The engine has acquired the ability to estimate the
runtime of each task and if the estimated time exceeds a task_duration
parameter, the engine is able to split the task in subtasks that run
in less than task_duration seconds. The user can set the task_duration
manually in the job.ini, or the user can leave it empty; in that case the
engine will figure out a reasonable value for it.
The approach is not perfect since there are non-splittable sources, so
there is a minimum size for a given subtask and sometimes subtasks taking
much longer that the task_duration parameter may still appear: however,
the new approach is a drastic improvement and the situation was never better
than it is now.
We added a check on sources with a suspiciously large spatial extent (more than 5,000 km) so that a warning is printed. Usually this means that there was a bug in the generation of the source model.
We added a check on sources with suspicious hypo-depths and nodal plane distributions (i.e. distributions with duplicated values) since they make the calculation slower.
In extra-large models saving some debugging information (eg. the number of sites affected by each source) was exceedingly slow, so now the information is stored only if there are fewer than 100,000 relevant sources.
Logic trees#
There was a tricky bug with the minimum_distance feature
in presence of multiple GSIMs in a logic tree branchset. Now
each GSIMs keeps its own minimum distance; before they were all
getting the same minimum distance, causing wrong results to be computed.
Fortunaly the minimum_distance feature is rarely used (and only for
internal purposes) so the bug is minor. The feature is documented here:
https://github.com/gem/oq-engine/blob/engine-3.6/doc/adv-manual/special-features.rst#gmpe-logic-trees-with-minimum_distance
We implemented zero weights for intensity measure types that should be discarded in the GSIM logic tree. You can see the relevant documentation here: https://github.com/gem/oq-engine/blob/engine-3.6/doc/adv-manual/special-features.rst#gmpe-logic-trees-with-weighted-imts
We implemented risk logic trees, a.k.a. the taxonomy mapping feature. The idea is that users can map the taxonomy strings in their exposure to one or more vulnerability/fragility functions, with corresponding weights for each function assignment, to take into account the epistemic uncertainty in the exposure ⟷ vulnerability domain. The feature is documented here: https://github.com/gem/oq-engine/blob/engine-3.6/doc/adv-manual/risk-features.rst
A big conceptual change (but with no impact on the user) was
the simplification of the source model logic tree XML file. Before it
was necessary to specify a logicTreeBranchingLevel node that was
not used internally, now that node is optional. Old
files will keep working, as long as the logicTreeBranchingLevel
contains only a single subnode. The case of multiple subnodes is now
correctly flagged as an error. Thanks to the change,
source model logic trees, gsim logic tree, and risk logic trees
are now stored in the same way internally.
Lastly, we fixed a bug in source model logic trees with the options
applyToSources and applyToBranches on; in some times a fake error
about the source not being in the source model - even if it actually was -
was raised.
Event based hazard#
We introduced a parameter max_sites_per_gmf in the job.ini
(only for event_based calculations that are trying to store
ground motion fields), with a default of 65,536 sites.
A user trying to run an event_based calculation that has
ground_motion_fields = true, with more than the number of
sites permitted by max_sites_per_gmf will now get an early
validation error instead of running out of memory after
several hours of calculations. The max_sites_per_gmf limit can
be raised beyond the default of 65,536 sites, at the user’s own
responsibility.
We also added a limit of 2**32 events in event based calculations: this is
a hard limit that cannot be raised. If your calculation produces more than
4 billion events, it will need to be be split into smaller calculations.
Such calculations involving billions of ruptures would likely never work anyway,
because it would eventually run out of memory.
We added a check for missing intensity_measure_types: this avoids
cryptic errors in the middle of the computation of the ground motion fields.
We optimized the rupture sampling procedure for point sources (which includes multi point sources and area sources). The improvement can be quite significant, for instance the generation of ruptures for a large multipoint source for Colombia became 30x faster using 12x less memory.
We changed the way ruptures are stored internally: the
code field in the ruptures dataset is now a unique checksum
depending on the kind of rupture. Before it was an incremental
number depending on the order of the Python module imports which
was making debugging difficult.
The rupture CSV exporter has been enhanced, and now it exports the rupture surface boundaries as 3D multipolygons instead of 2D multipolygons.
We fixed a small bug in the rupture XML exporter, which was failing if the user did not specify the hazard sites.
We added the ability to generate hazard curves without storing the GMFs, simply by setting the flags
hazard_curves_from_gmfs = true
ground_motion_fields = false
This is useful when one is interested in the hazard curves generated
by an event_based calculation but not in the ground motion fields
themselves. Not storing the GMFs reduces the data transfer and the
memory occupation.
In engine 3.5 we changed the gmf_data CSV exporter to export a file
sitemodel.csv instead of the file sitemesh.csv. That change has
been reverted because it was generating confusion. The right way to
to export the site model information for the most recently completed
calculation - which works for all calculators,
not only for event based - is to use the command
oq show sitecol > sitecol.csv
Importing GMFs from CSV has been enhanced and now it does not require
anymore the field rlzi: previously, this was a required field,
but it was assumed to contain always the value 0. On the other hand,
now the GMF exporter to CSV does not export the field rlzi, because
it is redundant: the association between events and realizations can
be found in the events table and it is exported in the
file sigma_epsilon.csv.
In the sigma_epsilon.csv file, we renamed the field eid to
event_id in order to avoid confusion with the naming used in the
gmf_data.csv file (event_id is the 64 bit event ID in the events
table in the datastore, eid is the 32 bit index to the event ID
record).
Event based risk#
There was a huge refactoring of all risk calculators. As a consequence
the event_based_risk calculator has become simpler and faster than before
(twice as fast in some cases).
In the ebrisk calculator it is now possible to aggregate by asset_id
and therefore to produce individual loss curves and maps for each asset.
Needless to mention, this is only viable for exposures of manageable size.
There was some work to make the losses_by_event exporter for the
ebrisk calculator more similar to the ones for event_based_risk
and for scenario_risk.
We fixed a bug in the agg_curves-rlzs and agg_curves-stats outputs
in ebrisk: they were missing the units compared to the same outputs
coming from the event_based_risk calculator. This was breaking the QGIS
plugin.
We changed the agglosses exporter in
scenario_risk calculations, by adding a column with the realization index.
The agg_curves exporter for event based risk was broken if the exposure
was imported in the parent calculation and not in the child calculation.
We fixed a bug in the exporter of the aggregate loss curves coming
from an ebrisk calculation: now the loss ratios are computed
correctly even in presence of occupants. Before the exporter was
writing incorrect loss ratios to the output file.
Hazardlib#
Graeme Weatherill (@g-weatherill) contributed a finite rupture option to the Germany-adjusted Cauzzi and Derras GMPEs. Moreover, he contributed the Tromans et al. (2019) adjustable GMPE, used for a nuclear power plant in the UK.
Chris van Houtte (@cvanhoutte) contributed the Van Houtte et al. (2018) Significant duration model for New Zealand.
Robin Gee (@rcgee) fixed a bug in the GMPE Sharma (2009): there was a key error if the intensity measure level specified in the job.ini included periods that required interpolation.
Marco Pagani (@mmpagani) discovered a bug in calc_hazard_curves
which was failing with a cryptic
AttributeError: 'NoneType' object has no attribute 'within_bbox'
when used in parallel mode. It has been fixed.
Risk#
The CSV importer for the exposure has been optimized. Before, for legacy reasons, the importer was converting the CSV records into node objects similar to the ones coming from the XML importer and then it was reusing the XML logic. Now we are doing the opposite: the XML importer is producing records and reusing the logic of the CSV importer. Thanks to this change for large CSV exposure the new importer is 4-5 times faster and uses over 10 times less memory than before.
Since a long time ago the engine has the ability to reduce the hazard site collection (which can be large, think of a fine grid) only to the locations where there are assets. Such feature has been optimized in this release, up to a spectacular extent in some cases: we measured a speedup from 2h to 0.1s for Canada.
We changed how zipped exposures are managed by the engine. In version
3.5 a zipped exposure was expected to contain an XML file with the
same name of the archive, apart from the extension. Because of that
the job.ini file had to contain a line exposure_file = <exposure_path>.xml while now it requires a line exposure_file = <exposure_path>.zip, which is clearer. The change was requested by
the risk team in the context of the CRAVE project because it
simplifies the unzipping of the exposure. Unzipping will overwrite
files of the same name already present, but a warning will be printed
and the original files will be not lost, but renamed with a .bak
extension.
We added a consistency check between statistics for calculations
leveraging the --hc option, because some users were making mistakes
like trying to compute means in the child calculation without having them
in the parent calculation. Now one gets a clear error message.
We fixed a bug in classical_damage from CSV with discrete
fragility functions: for hazard intensity measure levels different
from the fragility levels, the engine was giving incorrect results.
Vulnerability functions with the beta distribution must satisfy some consistency requirements if the coefficients of variation are nonzero. Unfortunately the consistency check were missing and it was possible to accept invalid functions raising and error in the middle of the computation. Now the error will be raised much early, at the time of the instantiating the vulnerability functions. See #4841 for more details.
Hyeuk Ryu (@dynaryu) discovered a bug in the agg_loss-curves outputs
for the event_based_risk calculators, which has been fixed.
Finally there were some improvements to the multi_risk calculator in the
context of the CRAVE project. In particular now the engine is able
to manage the geometries of volcanic perils like lava, lahar and pyroclastic
flow and it is also able to manage other binary perils without requiring
the introduction of new intensity measure types.
General changes#
The CSV exporters have been enhanced: now there is an additional line
before the header, starting with a # character, containing some
metadata, like the date when the file the generated, the version of
the engine that generated it, and some relevant parameters, like the
investigation time in the case of the hazard outputs. In the future
we may add even more metadata and extend the approach to other outputs.
Before release 3.3, the engine had the ability to associate site
model parameters to hazard sites on a grid. This feature was sometimes
buggy and removed, by recommending to the users the command oq prepare_site_model instead. oq prepare_site_model is able to
produce a site_model.csv file with sites on the grid and it performs the
associations explicitly, once and for all.
In this release, we restored the ability to perform the association
directly in the engine. This is less efficient than using oq prepare_site_model, since the same associations will be recomputed during
each run. It is still useful for people wanting to experiment with the
grid spacing: they can run several calculations and when they are
happy with the grid spacing, run oq prepare_site_model and fix the
site model once and for all with the preferred grid spacing.
We fixed a performance regression in the ucerf_classical calculator,
due to a change of logic in engine 3.5, which was trying to filter
thousands of sources in the controller node instead than in the
workers, thus becoming extra-slow.
We decided to change the realizations.csv output for scenario calculations,
by replacing the branch_path field with the GSIM representation. This is
more informative for the users and more convenient for the QGIS plugin too.
IT#
The job queue first introduced in engine-3.5 is now enabled by default. This means than only on job can run at a given time for a given engine instance.
The progress report has been improved: before in large classical calculations the progress started to be printed too late, even days after the start of the calculation.
We improved the oq abort command to remove submitted jobs too.
Deleting a calculation in the engine has always been tricky in the case
of multiple users. In this release we fixed several issues and now an
user can delete all of her calculations with the command oq reset.
The engine will look inside the database and correctly remove the
calculations of the user, including all the relevant .hdf5 files.
We improved the oq plot command by adding several new kinds of plot.
They are still for internal use only (i.e. introspection and debugging).
We extended the command oq db to run generic queries for the
openquake user. Other users can only run SELECT queries.
There was a bug in oq webui start not supplying the --noreload
argument that has been fixed (the reload functionality of the Django
development server interferes with SIGCHLD and causes zombies).
We fixed another bug with the --hc functionality in a multi-user
situation, due to the fact that the engine was searching the the datastore
of the parent calculation in the wrong directory.
There is now a better error message if the shared directory is not mounted.
Source models can now be serialized in TOML format, which is useful for debugging purposes.
Libraries and Python 3.7#
In this releases we updated some of our libraries (numpy from version 1.14 to 1.16 and scipy from version 1.0.1 to 1.3.0) to make it possible to use the engine with Python 3.7. We actually have a cluster using Python 3.7 in production.
In the future we may distribute installers for Windows and macOS based on Python 3.7, but for the moment Python 3.6 is still the only officially supported version and we not plan to abandon Python 3.6 any time soon.
We raised the minimum version for h5py from 2.8.0 to 2.9.0, fixed some compatibility issue with Django 2.1 and 2.2 and fixed several Python 3.7 deprecation warnings. Finally we removed the external dependency from the mock module since it is included in the standard library since Python 3.3.
Deprecations/removals#
For years the engine has been able to import ground motion fields and hazard curves in CSV format and NRML format, with the NRML format deprecated. Now finally the NRML importers have been removed.
There was an old deprecated GMF exporter in NRML format for scenario calculations. It has been finally deprecated. You should use the CSV exporter thas has been available for years instead.
We deprecated the XML disaggregation exporters in favor of the CSV exporters.
We removed the long time deprecated agg_loss_table exporter since
now all the needed information is in the losses_by_event exporter.
We switched officially the testing framework from nosetests to pytest.