Release notes v1.7#

OpenQuake 1.7 is a major release and a big improvement with respect to OpenQuake 1.6.

New features of the OpenQuake Engine, version 1.7#

  1. The following six calculators have been completely rewritten and now use the HDF5 technology that in previous versions was still experimental:

  2. scenario

  3. scenario_risk

  4. scenario_damage

  5. event_based_rupture

  6. event_based

  7. event_based_risk

As a consequence all such calculators are much faster and use a lot less memory than before. Even the disk space occupation has been drastically reduced, and large event based computations that before took terabytes of disk space now requires few gigabytes. The new calculators do not store their outputs in the database, but in an HDF5 file, named the datastore, located by default in $HOME/oqdata/calc_XXX.hdf5.

  1. The other calculators are unchanged; they are still using PostgreSQL. They will be replaced with HDF5-based versions in future releases of the OpenQuake Engine. For most calculators an HDF5-based implementation is already available in the engine and can be accessed by using the --lite flag. For instance the classical hazard calculator is supported and can be run with

$ oq-engine --lite --run job.ini

However the calculators accessible with the --lite flag should be considered experimental, previews of things to come, and they are still subject to change.

  1. The new calculators can run with a single configuration file containing both the hazard and the risk parameters. This is actually the recommended way to run such calculators.

  2. The scenario_risk and scenario_damage calculators now support multiple GSIMs at the same time. This new functionality is documented in the [manual] (http://www.globalquakemodel.org/openquake/support/documentation/engine/).

  3. The scenario_damage calculator now supports multiple loss types at the same time, just as the scenario_risk calculator. Again, see the [manual] (http://www.globalquakemodel.org/openquake/support/documentation/engine/) for the details.

  4. The event_based_risk calculator has been optimized and enhanced. Now it is possible to generate the full event loss table for each asset and for each realization. Just set the configuration parameter asset_loss_table=True (the default is False).

  5. The --load-gmf option has been removed from the engine; now in order to load an XML file containing ground motion fields you need to change the job.ini file and add a line like the following:

gmfs_file = <path-to-file-gmfs.xml>

In the next release the --load-curve option will be removed and in order to load an XML containing hazard curves you will need to change the job.ini file and to add a line like the following:

hazard_curves_file = <path-to-file-hazard-curve.xml>

This already works if you use the --lite version of the classical calculators (classical_risk/classical_damage/classical_bcr).

  1. OpenQuake 1.7 supports officially the format NRML 0.5 for all risk models. In OpenQuake 1.5 it was supported in a limited and experimental way only for a subset of the vulnerability functions. Now all kind of risk models are supported: vulnerability models, fragility risk models and consequence models. Consequence models are brand new, introduced for the first time in this release. All of that is documented in the [manual] (http://www.globalquakemodel.org/openquake/support/documentation/engine/). The new format is simpler than before and more convenient to use, since the OpenQuake platform offers a web tool to prepare risk models in NRML 0.5. Beware that the web tool does not support validation of the risk models yet.

  2. The validation of the risk models in the engine has been improved. In particular now an user confusing a fragility model with a vulnerability model or a consequence model or any other combination will get a clear error message. Moreover, each risk model has a lossCategory attribute which must be set consistently with the name of the key in the job.ini file (see the [manual] (http://www.globalquakemodel.org/openquake/support/documentation/engine/) for the details).

  3. Some work has been going on hazardlib, as usual, and you can have a look at the [changelog] (https://github.com/gem/oq-hazardlib/blob/engine-1.7/debian/changelog). The most prominent feature is the introduction of new epistemic uncertainties in the Source Model Logic Tree. Users are now able to represent alternative geometries for the fault source typologies using the following uncertainty model definitions:

  4. simpleFaultGeometryAbsolute - Replaces the simple fault geometry with the specified simple fault surface.

  5. complexFaultGeometryAbsolute - Replaces the complex fault geometry with the specified complex fault surface.

  6. characteristicFaultGeometryAbsolute - Replaces the characteristic fault geometry with the specified surface.

  7. simpleFaultDipRelative - Increases or decreases the fault dip with the specified value.

  8. simpleFaultDipAbsolute - Replaces the simple fault dip with the specified value.

Also added is the ability to replace a given evenly discretised magnitude frequency distribution of a source in the logic tree using the incrementalMFDAbsolute option.

Documentation on the usage of these features will be added in the forthcoming version of the manual.

  1. We added an optional attribute discretization to the area source geometry XML description: this means that it is possible to specify a source-specific discretization step. This is useful in site specific analysis: area sources with little impact on the site of interest can use a large discretization step whereas the important area sources can use a finer discretization step.

  2. Three new GMPEs have been added to hazardlib: Montalva et al. (2015), Tusa and Langer (2015), Allen, Wald and Worden (2012).

  3. The GSIM class has been refactored and now there is a clear distinction between the preparation phase (making the context) and the actual calculation phase (getting the PoEs). That made it possible an optimization in the computation of distances in the case of multiple GSIMs.

  4. The rupture filtering has been moved into the making context phase, thus saving a redundant distance calculation.

  5. We improved the support for monitoring calculations in oq-lite, and added an option to profile oq-lite calculations.

  6. The .rst report of a calculation has been improved and more information is displayed. Moreover, you can also run

$ oq-lite show <calc_id> fullreport

to get information about a calculation which has already run. This only works for the lite calculators.

  1. Some small improvements to the Engine Web User Interface have been made and this feature, first introduced in OpenQuake Engine 1.4, is now finally documented.

  2. We added a script oq_reset_db to drop and recreate the engine database, as well as removing the datastore directories of all users. This is meant to be used with care by system administrators. See the instructions printed at the moment of running the script.

  3. There is now a check to stop the calculation if the parameter uniform_hazard_spectra is set but there are no intensity measure types of kind Spectral Acceleration or Peak Ground Acceleration.

  4. We added a check for the case when the region constraint remove all the assets from a computation.

  5. We fixed several export bugs, especially for the CSV exporters. The CSV exporters are scheduled to change again in the next release, so they are not official yet.

  6. Countless small improvements and additional validations have been added. We also improved our Continuous Integration mechanism and our installation routines. This release has seen more than 200 pull requests reviewed and merged.

Bug fixes and changes with respect to OpenQuake 1.5#

  1. The most important change is the support for NRML 0.5. A deprecation warning is printed every time you use a risk model in the NRML 0.4 format. To get rid of the warning you must upgrade the risk model files to NRML 0.5. There is a command to do that recursively on a directory. Just write

$ oq-lite upgrade_nrml <some-directory>

and all of your risk models will be upgraded. The original files will be kept, but with a .bak extension appended. Notice that you will need to set the lossCategory attribute to its correct value manually. This is easy to do, since if you try to run a computation you will get a clear error message telling which is the expected value for the lossCategory for each risk model file. For instance, you may get an error like this:

ValueError: Error in the file "structural_vulnerability_file=/home/.../vulnerability_model.xml": lossCategory is of type "economic_loss", expected "structural"

The reason is that in NRML 0.4 the lossCategory attribute had no special meaning (actually it was ignored by the engine) whereas now there is a check on it. It must be consistent with the name of the variable used in the configuration file. In this example in the job_risk.ini file there was a line structural_vulnerability_file=, so the lossCategory is expected to be of kind structural. Edit the “vulnerability_model.xml” file and set the lossCategory attribute to the expected value.

  1. Valid loss categories are structural, nonstructural, contents, business_interruption and fatalities. There is now a strict check on the categories, both in the risk model files and in the exposure file. The check is disabled for NRML 0.4 files, so we are fully backward-compatible.

  2. If an user set the parameter insured_losses=True but the exposure does not have the attributes deductible and insuredLimit, a clear error is raised early.

  3. In very rare circumstances the region constraint was not honored, i.e. assets very close to the border of region, but still outside, were taken in consideration by the risk calculators. This has been fixed.

  4. The engine had a self-termination feature: if the controller node could not access the worker nodes, it assumed a failure in celery and committed suicide if the configuration parameter terminate_job_when_celery_is_down was true. We removed such parameter and such feature because it was too dangerous: sometimes celery was up and running but incorrectly reported down because too slow to respond, due to an heavy load. Now if celery appears to not respond a warning is printed and that user has to see if celery is really dead and in that case she can kill the computation manually.

  5. We removed the epsilon sampling feature from the scenario_risk and event_based_risk calculators: it was a performance hack introducing a gratuitous seed dependency, now unneeded thanks to the recent performance improvements.

  6. The specific_assets feature has been removed from the event_based_risk calculator. It has become useless thanks to the recent performance improvements. Use asset_loss_table=True instead.

  7. In some cases (for split fault sources) there was a spurious comma in the rupture tags which caused issues with a CSV importer in the Risk Modeler Toolkit. This has been fixed.

  8. The demos have been revisited and updated. Also their location has changed for the users installing OpenQuake from the packages. Now they are installed in /usr/share/openquake/risklib/demos.

  9. We fixed a bug reported by Blaž Barič with the AbrahamsonEtAl2014 GSIM by removing unneeded __slots__ in hazardlib.

  10. We fixed a bug with the AtkinsonBoore2003SSlabNSHMP2008 GSIM which was instanting GSIMs in the workers for no good reason.

  11. When using a vulnerability function with a Probability Mass Function, now it is possible to set the seed by changing the random_seed parameter in the configuration file. Before the seed was hard-coded.

  12. We introduced the concept of composite outputs, i.e. outputs that can be exported to a zip file containing a set of output files. For instance an event based risk calculation with two realizations and four loss types in the past could print something like the following:

  id | output_type | name
 515 | Aggregate Loss Curve | aggregate loss curves. loss_type=contents hazard=430||gmf||GMF rlz-332
 514 | Aggregate Loss Curve | aggregate loss curves. loss_type=contents hazard=431||gmf||GMF rlz-333
 519 | Aggregate Loss Curve | aggregate loss curves. loss_type=fatalities hazard=430||gmf||GMF rlz-332
 518 | Aggregate Loss Curve | aggregate loss curves. loss_type=fatalities hazard=431||gmf||GMF rlz-333
 512 | Aggregate Loss Curve | aggregate loss curves. loss_type=nonstructural hazard=430||gmf||GMF rlz-332
 513 | Aggregate Loss Curve | aggregate loss curves. loss_type=nonstructural hazard=431||gmf||GMF rlz-333
 516 | Aggregate Loss Curve | aggregate loss curves. loss_type=structural hazard=430||gmf||GMF rlz-332
 517 | Aggregate Loss Curve | aggregate loss curves. loss_type=structural hazard=431||gmf||GMF rlz-333

Now it will print only one line:

  id | output_type | name
 419 | datastore | agg_curve-rlzs

When exporting the composite output, 8 XML files will be generated, in the same format as before, plus a zip file containing all of them. When using the Web UI the zip file will be available for download.

Support for different platforms#

OpenQuake 1.7 supports Ubuntu from release 12.04 up to 15.10. We provide official packages for the long term releases 12.04 and 14.04. We were able to extend the support to Ubuntu 12.04 by backporting the package python-h5py from Ubuntu 14.04. So Ubuntu 12.04 is still supported, even if it is deprecated.

We have official packages also for CentOS 7 and in general for [Red Hat Enterprise Linux clones] (Installing-the-OpenQuake-Engine-from-source-code-on-Fedora-and-RHEL.md).

While the engine is not supported on Windows and Mac OS X, we are happy to report that the underlying libraries and the oq-lite command-line tool run just fine.

Other#

The OpenQuake project is even more open than before. From this release we are using GitHub as our official bug tracker, which makes it easier to follow the development process (before the bug tracker was Launchpad, which is less popular than GitHub and not integrated with the code base). Moreover from this release our libraries (both oq-hazardlib and oq-risklib) are tested by using a public Continuous Integration system, Travis. Before our builds were internal on Jenkins and visible only to our staff. The engine is still built with Jenkins for various technical reasons.

Finally, depending on the version of the HDF5 libraries you are using, you may get a warning like the following:

HDF5: infinite loop closing library
      D,G,A,S,T,F,D,A,S,T,F,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL

Please ignore it. The engine is working correctly.