Release notes v3.6#

This release features several major new features (including completely revised disaggregation, automatic optimization of duplicated sources, fast exposure importer and taxonomy mapping) and lots of improvements, new checks and bug fixes. Nearly 200 pull requests were merged. For the complete list of changes, see the changelog: https://github.com/gem/oq-engine/blob/engine-3.6/debian/changelog

Disaggregation#

The most relevant development on the hazard side was the work on the disaggregation calculator. We changed substantially the business logic. While in previous versions of the engine we were disaggregating for all possible realizations in order to compute disaggregation statistics, in this version we gave up on statistics. Instead, we disaggregate only for a specific realization. The realization can be specified by the user with the rlz_index parameter in the job.ini file, or it can be determined automatically by the engine as the realization closest to the mean curve for the given disaggregation site.

Moreover, now the disaggregation calculation works like a post-calculator (i.e. with the --hc option) and it is able to reuse information computed in its parent calculation: the net effect is that it is always faster than the corresponding classical calculation while in the past it was several times slower. We also fixed a couple of performance bugs: there was a slow operation truncnorm.cdf in an inner loop and ruptures outside the integration distance were not discarded. Finally, we changed the file names of the disaggregation outputs.

For models with thousands of realizations, the disaggregation can easily be thousands of times faster than before.

Classical PSHA calculator#

The engine is now smart enough to recognize duplicated sources appearing in different branches of the composite source model and to avoid redundant computations. Because this optimization is always on, the flag optimize_same_id_sources has been removed, as it has now been rendered useless. There are several models in the hazard mosaic with duplicates sources and the new optimization has a significant impact on those. Moreover the demo LogicTreeCase2ClassicalPSHA has become an order of magnitude faster than before thanks to the reduction of the duplicated sources.

There was a big improvement in the computation of the statistical hazard curves which now is not only faster, but uses a lot less memory than before. The trick was to consider one site at the time, instead of a block of sites. As a consequence it is now possible to consider tens of thousands of realizations for hundreds of thousands of sites without requiring terabytes of RAM. Moreover the data transfer has been reduced by storing some auxiliary information in the datastore and reading it from the workers instead of transferring it via celery/rabbitmq.

There was a substantial change in the way the tasks are distributed for a classical calculation. The engine has acquired the ability to estimate the runtime of each task and if the estimated time exceeds a task_duration parameter, the engine is able to split the task in subtasks that run in less than task_duration seconds. The user can set the task_duration manually in the job.ini, or the user can leave it empty; in that case the engine will figure out a reasonable value for it.

The approach is not perfect since there are non-splittable sources, so there is a minimum size for a given subtask and sometimes subtasks taking much longer that the task_duration parameter may still appear: however, the new approach is a drastic improvement and the situation was never better than it is now.

We added a check on sources with a suspiciously large spatial extent (more than 5,000 km) so that a warning is printed. Usually this means that there was a bug in the generation of the source model.

We added a check on sources with suspicious hypo-depths and nodal plane distributions (i.e. distributions with duplicated values) since they make the calculation slower.

In extra-large models saving some debugging information (eg. the number of sites affected by each source) was exceedingly slow, so now the information is stored only if there are fewer than 100,000 relevant sources.

Logic trees#

There was a tricky bug with the minimum_distance feature in presence of multiple GSIMs in a logic tree branchset. Now each GSIMs keeps its own minimum distance; before they were all getting the same minimum distance, causing wrong results to be computed. Fortunaly the minimum_distance feature is rarely used (and only for internal purposes) so the bug is minor. The feature is documented here: https://github.com/gem/oq-engine/blob/engine-3.6/doc/adv-manual/special-features.rst#gmpe-logic-trees-with-minimum_distance

We implemented zero weights for intensity measure types that should be discarded in the GSIM logic tree. You can see the relevant documentation here: https://github.com/gem/oq-engine/blob/engine-3.6/doc/adv-manual/special-features.rst#gmpe-logic-trees-with-weighted-imts

We implemented risk logic trees, a.k.a. the taxonomy mapping feature. The idea is that users can map the taxonomy strings in their exposure to one or more vulnerability/fragility functions, with corresponding weights for each function assignment, to take into account the epistemic uncertainty in the exposure ⟷ vulnerability domain. The feature is documented here: https://github.com/gem/oq-engine/blob/engine-3.6/doc/adv-manual/risk-features.rst

A big conceptual change (but with no impact on the user) was the simplification of the source model logic tree XML file. Before it was necessary to specify a logicTreeBranchingLevel node that was not used internally, now that node is optional. Old files will keep working, as long as the logicTreeBranchingLevel contains only a single subnode. The case of multiple subnodes is now correctly flagged as an error. Thanks to the change, source model logic trees, gsim logic tree, and risk logic trees are now stored in the same way internally.

Lastly, we fixed a bug in source model logic trees with the options applyToSources and applyToBranches on; in some times a fake error about the source not being in the source model - even if it actually was - was raised.

Event based hazard#

We introduced a parameter max_sites_per_gmf in the job.ini (only for event_based calculations that are trying to store ground motion fields), with a default of 65,536 sites. A user trying to run an event_based calculation that has ground_motion_fields = true, with more than the number of sites permitted by max_sites_per_gmf will now get an early validation error instead of running out of memory after several hours of calculations. The max_sites_per_gmf limit can be raised beyond the default of 65,536 sites, at the user’s own responsibility.

We also added a limit of 2**32 events in event based calculations: this is a hard limit that cannot be raised. If your calculation produces more than 4 billion events, it will need to be be split into smaller calculations. Such calculations involving billions of ruptures would likely never work anyway, because it would eventually run out of memory.

We added a check for missing intensity_measure_types: this avoids cryptic errors in the middle of the computation of the ground motion fields.

We optimized the rupture sampling procedure for point sources (which includes multi point sources and area sources). The improvement can be quite significant, for instance the generation of ruptures for a large multipoint source for Colombia became 30x faster using 12x less memory.

We changed the way ruptures are stored internally: the code field in the ruptures dataset is now a unique checksum depending on the kind of rupture. Before it was an incremental number depending on the order of the Python module imports which was making debugging difficult.

The rupture CSV exporter has been enhanced, and now it exports the rupture surface boundaries as 3D multipolygons instead of 2D multipolygons.

We fixed a small bug in the rupture XML exporter, which was failing if the user did not specify the hazard sites.

We added the ability to generate hazard curves without storing the GMFs, simply by setting the flags

  hazard_curves_from_gmfs = true
  ground_motion_fields = false

This is useful when one is interested in the hazard curves generated by an event_based calculation but not in the ground motion fields themselves. Not storing the GMFs reduces the data transfer and the memory occupation.

In engine 3.5 we changed the gmf_data CSV exporter to export a file sitemodel.csv instead of the file sitemesh.csv. That change has been reverted because it was generating confusion. The right way to to export the site model information for the most recently completed calculation - which works for all calculators, not only for event based - is to use the command oq show sitecol > sitecol.csv

Importing GMFs from CSV has been enhanced and now it does not require anymore the field rlzi: previously, this was a required field, but it was assumed to contain always the value 0. On the other hand, now the GMF exporter to CSV does not export the field rlzi, because it is redundant: the association between events and realizations can be found in the events table and it is exported in the file sigma_epsilon.csv.

In the sigma_epsilon.csv file, we renamed the field eid to event_id in order to avoid confusion with the naming used in the gmf_data.csv file (event_id is the 64 bit event ID in the events table in the datastore, eid is the 32 bit index to the event ID record).

Event based risk#

There was a huge refactoring of all risk calculators. As a consequence the event_based_risk calculator has become simpler and faster than before (twice as fast in some cases).

In the ebrisk calculator it is now possible to aggregate by asset_id and therefore to produce individual loss curves and maps for each asset. Needless to mention, this is only viable for exposures of manageable size.

There was some work to make the losses_by_event exporter for the ebrisk calculator more similar to the ones for event_based_risk and for scenario_risk.

We fixed a bug in the agg_curves-rlzs and agg_curves-stats outputs in ebrisk: they were missing the units compared to the same outputs coming from the event_based_risk calculator. This was breaking the QGIS plugin.

We changed the agglosses exporter in scenario_risk calculations, by adding a column with the realization index.

The agg_curves exporter for event based risk was broken if the exposure was imported in the parent calculation and not in the child calculation.

We fixed a bug in the exporter of the aggregate loss curves coming from an ebrisk calculation: now the loss ratios are computed correctly even in presence of occupants. Before the exporter was writing incorrect loss ratios to the output file.

Hazardlib#

Graeme Weatherill (@g-weatherill) contributed a finite rupture option to the Germany-adjusted Cauzzi and Derras GMPEs. Moreover, he contributed the Tromans et al. (2019) adjustable GMPE, used for a nuclear power plant in the UK.

Chris van Houtte (@cvanhoutte) contributed the Van Houtte et al. (2018) Significant duration model for New Zealand.

Robin Gee (@rcgee) fixed a bug in the GMPE Sharma (2009): there was a key error if the intensity measure level specified in the job.ini included periods that required interpolation.

Marco Pagani (@mmpagani) discovered a bug in calc_hazard_curves which was failing with a cryptic AttributeError: 'NoneType' object has no attribute 'within_bbox' when used in parallel mode. It has been fixed.

Risk#

The CSV importer for the exposure has been optimized. Before, for legacy reasons, the importer was converting the CSV records into node objects similar to the ones coming from the XML importer and then it was reusing the XML logic. Now we are doing the opposite: the XML importer is producing records and reusing the logic of the CSV importer. Thanks to this change for large CSV exposure the new importer is 4-5 times faster and uses over 10 times less memory than before.

Since a long time ago the engine has the ability to reduce the hazard site collection (which can be large, think of a fine grid) only to the locations where there are assets. Such feature has been optimized in this release, up to a spectacular extent in some cases: we measured a speedup from 2h to 0.1s for Canada.

We changed how zipped exposures are managed by the engine. In version 3.5 a zipped exposure was expected to contain an XML file with the same name of the archive, apart from the extension. Because of that the job.ini file had to contain a line exposure_file = <exposure_path>.xml while now it requires a line exposure_file = <exposure_path>.zip, which is clearer. The change was requested by the risk team in the context of the CRAVE project because it simplifies the unzipping of the exposure. Unzipping will overwrite files of the same name already present, but a warning will be printed and the original files will be not lost, but renamed with a .bak extension.

We added a consistency check between statistics for calculations leveraging the --hc option, because some users were making mistakes like trying to compute means in the child calculation without having them in the parent calculation. Now one gets a clear error message.

We fixed a bug in classical_damage from CSV with discrete fragility functions: for hazard intensity measure levels different from the fragility levels, the engine was giving incorrect results.

Vulnerability functions with the beta distribution must satisfy some consistency requirements if the coefficients of variation are nonzero. Unfortunately the consistency check were missing and it was possible to accept invalid functions raising and error in the middle of the computation. Now the error will be raised much early, at the time of the instantiating the vulnerability functions. See #4841 for more details.

Hyeuk Ryu (@dynaryu) discovered a bug in the agg_loss-curves outputs for the event_based_risk calculators, which has been fixed.

Finally there were some improvements to the multi_risk calculator in the context of the CRAVE project. In particular now the engine is able to manage the geometries of volcanic perils like lava, lahar and pyroclastic flow and it is also able to manage other binary perils without requiring the introduction of new intensity measure types.

General changes#

The CSV exporters have been enhanced: now there is an additional line before the header, starting with a # character, containing some metadata, like the date when the file the generated, the version of the engine that generated it, and some relevant parameters, like the investigation time in the case of the hazard outputs. In the future we may add even more metadata and extend the approach to other outputs.

Before release 3.3, the engine had the ability to associate site model parameters to hazard sites on a grid. This feature was sometimes buggy and removed, by recommending to the users the command oq prepare_site_model instead. oq prepare_site_model is able to produce a site_model.csv file with sites on the grid and it performs the associations explicitly, once and for all.

In this release, we restored the ability to perform the association directly in the engine. This is less efficient than using oq prepare_site_model, since the same associations will be recomputed during each run. It is still useful for people wanting to experiment with the grid spacing: they can run several calculations and when they are happy with the grid spacing, run oq prepare_site_model and fix the site model once and for all with the preferred grid spacing.

We fixed a performance regression in the ucerf_classical calculator, due to a change of logic in engine 3.5, which was trying to filter thousands of sources in the controller node instead than in the workers, thus becoming extra-slow.

We decided to change the realizations.csv output for scenario calculations, by replacing the branch_path field with the GSIM representation. This is more informative for the users and more convenient for the QGIS plugin too.

IT#

The job queue first introduced in engine-3.5 is now enabled by default. This means than only on job can run at a given time for a given engine instance.

The progress report has been improved: before in large classical calculations the progress started to be printed too late, even days after the start of the calculation.

We improved the oq abort command to remove submitted jobs too.

Deleting a calculation in the engine has always been tricky in the case of multiple users. In this release we fixed several issues and now an user can delete all of her calculations with the command oq reset. The engine will look inside the database and correctly remove the calculations of the user, including all the relevant .hdf5 files.

We improved the oq plot command by adding several new kinds of plot. They are still for internal use only (i.e. introspection and debugging).

We extended the command oq db to run generic queries for the openquake user. Other users can only run SELECT queries.

There was a bug in oq webui start not supplying the --noreload argument that has been fixed (the reload functionality of the Django development server interferes with SIGCHLD and causes zombies).

We fixed another bug with the --hc functionality in a multi-user situation, due to the fact that the engine was searching the the datastore of the parent calculation in the wrong directory.

There is now a better error message if the shared directory is not mounted.

Source models can now be serialized in TOML format, which is useful for debugging purposes.

Libraries and Python 3.7#

In this releases we updated some of our libraries (numpy from version 1.14 to 1.16 and scipy from version 1.0.1 to 1.3.0) to make it possible to use the engine with Python 3.7. We actually have a cluster using Python 3.7 in production.

In the future we may distribute installers for Windows and macOS based on Python 3.7, but for the moment Python 3.6 is still the only officially supported version and we not plan to abandon Python 3.6 any time soon.

We raised the minimum version for h5py from 2.8.0 to 2.9.0, fixed some compatibility issue with Django 2.1 and 2.2 and fixed several Python 3.7 deprecation warnings. Finally we removed the external dependency from the mock module since it is included in the standard library since Python 3.3.

Deprecations/removals#

For years the engine has been able to import ground motion fields and hazard curves in CSV format and NRML format, with the NRML format deprecated. Now finally the NRML importers have been removed.

There was an old deprecated GMF exporter in NRML format for scenario calculations. It has been finally deprecated. You should use the CSV exporter thas has been available for years instead.

We deprecated the XML disaggregation exporters in favor of the CSV exporters.

We removed the long time deprecated agg_loss_table exporter since now all the needed information is in the losses_by_event exporter.

We switched officially the testing framework from nosetests to pytest.