Release notes v1.5#

OpenQuake 1.5 is a major release and a big improvement with respect to OpenQuake 1.4. More than 115 bugs/feature requests were fixed/implemented.

New features of the OpenQuake Engine, version 1.5#

The most important new feature is the support for the HDF5 technology. Starting from this release some of the scientific calculators are saving their inputs and outputs in a single HDF5 file, called the datastore. The HDF5 format is a well known standard in the scientific community, can be read/written by a variety of programming languages and with different tools and it is a state-of-the-art technology when it comes to managing large numeric datasets. The change to the HDF5 technology provides huge performance benefits compared to the earlier approach used by the engine, which involved storing arrays in PostgreSQL.
Related to the first point, in OpenQuake 1.5 the event based calculators based on Postgres (both hazard and risk) are officially deprecated. They are still present and work as before, but they are being replaced with new versions of the calculators based on the HDF5 technology. The actual removal of the old calculators is scheduled for OpenQuake 1.6. The change will have no impact on regular users, who will simply notice a definite improvement in erformance. Nonetheless, the change will affect power users who are performing queries on the OpenQuake database, since there will be nothing left in the database once we remove the old calculators.
In order to make the transition easier, OpenQuake 1.5 already includes the new versions of the event based calculators based on HDF5, so it is possible to use them right now. The new calculators can be run in OpenQuake 1.5 with the command $ oq-engine --lite --run job_haz.ini,job_risk.ini. If you do not pass the --lite flag the old calculators will be run by default. In future releases of the engine, the remaining calculators based on Postgres will be progressively replaced by the new calculators based on HDF5. At the end of this process, which will be spread over several upcoming releases, the --lite flag will be removed. All of the old calculators relying on the database will be replaced internally by the newer “lite” versions based on HDF5 and the old calculators will not be available anymore. The OpenQuake database will only contain accessory information (essentially a table with the users and references to the outputs of each user) but nothing relevant for the scientific computation.
At the moment, the --lite flag does not work for all calculators. For instance, among the hazard calculators, the disaggregation --lite calculator is absent in OpenQuake 1.5. Work on this calculator is in progress, and it will be added in a future release; for the the time being, you will have to use the old calculator, which is not deprecated in OpenQuake 1.5. The --lite versions of the other hazard calculators (scenario hazard, classical hazard, and event based hazard) are complete. The --lite version of the classical_tiling calculator is also complete but relatively new and has not been battle tested yet. The --lite versions of the risk calculators are at different levels of completion; the only --lite risk calculator we recommend using in this release is the event based risk calculator.
Internally, the --lite calculators are implemented very differently compared to the old calculators based on Postgres, however they produce identical results. They implement the same science and any noticable differences should be reported as a bug. There could be minimal discrepancies due to numerical errors, and changes in rounding, but nothing more than that. The event based --lite calculators are faster by orders of magnitudes, especially for large calculations, both because of the HDF5 technology and also because they compute the ground motion fields on the fly, thus avoiding the time wasted in saving/reading large amounts of data, as the old calculators did. It is recommended that you start using the --lite versions of the event based calculators in preference to the engine ones.
OpenQuake 1.5, as a special preview of the future, is able to manage a new kind of vulnerability function, in which you can specify the discrete Probability Mass Function (PMF) of the loss ratio at different ground motion intensity levels numerically. This is a feature that will be officially introduced in the new version of our XML data format, NRML 0.5, but it is already available unofficially. For the moment NRML 0.4 is not deprecated. In OpenQuake 1.6 we will support other kinds of vulnerability and fragility functions in the new format NRML 0.5, and NRML 0.4 may be deprecated. In that case a conversion script to convert input files from NRML 0.4 to NRML 0.5 will be provided.
For the first time, hazardlib supports Python 3. The support is at the beginning and the C-level speedups do not work yet. However, we are already testing hazardlib with Python 3.4 by using the Travis continuous integration system and we are committed to keep it compatible both with Python 2.7 and Python 3.4+ for the foreseeable future. There is no plan to abandon Python 2.7 any time soon, but there is a plan to extend the support for Python 3.4+ to risklib and the engine. However, this will be a long term and low priority process: do not expect anything definitive before 2016.
It is now possible to pass string parameters to GSIM classes, directly from the XML representation of the logic tree. This is of interest only to users writing GSIMs, and they can read the related pull request for the relevant details: https://github.com/gem/oq-risklib/pull/346
The passing-parameters-to-GSIMs feature has been used to implement support for the definition of ground motion prediction equations using interpolation (‘look up’) tables. These allow the user to input arbitrary GMPEs in the form of tables, rather than the parametric equations currently supported. The tables, in HDF5 format, provide the expected ground motion values for given magnitudes and distances, with the additional option of amplifying the ground motions based on source or site attributes. To use this option the user need only specify GMPETable(gmpe_table=path/to/table.hdf5) in place of the conventional GMPE. Further guidance regarding the construction of the HDF5 files will be provided in the documentation in due course.
Near-fault directivity probabilistic seismic hazard analysis for classical PSHA calculations with simple fault sources was added to hazardlib. We implemented the most recent NGA-WEST2 directivity model and the associated GMPE which is, up to now, the only GMPE model explicitly including the effect. More details can be found in the manual.
Several other features have been implemented in hazard and you can have a look at the changelog.
The oq-lite command-tool has been enhanced; it is possible to use it to execute the same calculations that you would run with the command oq-engine --lite. The difference is that oq-lite only works on a single machine, not on a cluster. On the plus side, it does not require having a celery instance up and running.
oq-lite is especially useful to perform preliminary analysis before you run a large computation on the engine. Running $ oq-lite info --report <my_job.ini> will generate a text report on the expected size of the computation. It is recommended to generate such report before you start anything large. Currently the functionality only works for hazard calculations but it is expected to grow in the future.
Several other improvements have been made to oq-lite, too many to list them all; please see the changelog for the complete list.
We added a functionality write_source_model to serialize sources in XML. Also we improved the reading of XML files and the error message in case of invalid files. Finally, we have removed the dependency on lxml, thus making the OpenQuake suite more portable across different platforms and easier to install.
We added a check on the site parameters distance. If a site model file is provided in a hazard calculation, and if no site parameters are available within a radius of 5 km for a particular site, a warning is raised. The goal is to signal the user if they used an incorrect site model file with respect to the sites they are using. The calculation still runs and complete, since sometimes you may not have site parameters data close enough to the sites of interests.
We have parallelized the source splitting procedure with a good performance boost. There is also a flag parallel_source_splitting in openquake.cfg to disable this feature, should the need arise (default: true).

Support for different platforms#

OpenQuake 1.5 fully supports both Ubuntu 12.04 and Ubuntu 14.04 and we provide packages for both platforms. However, starting from OpenQuake 1.6 we will release packages only for Ubuntu 14.04. Ubuntu 12.04 will still be supported but you will have to install manually some dependencies which are not in the repositories of Ubuntu 12.04. The reason for the change is that the HDF5 libraries for Ubuntu 12.04 are too old (over 4 year old), buggy and less efficient compared to the ones for Ubuntu 14.04, which is now our official development platform. It is too expensive for us to mantain compatibility with such ancient software, so users wishing to use OpenQuake 1.6 on Ubuntu 12.04 will have to install manually the library h5py (version 2.2.1) and its dependencies. We will provide instructions for that in the next release, since for the moment this is not necessary.

We have detailed instructions for installing the engine on CentOS 7 and Fedora and in general on [Red Hat Enterprise Linux clones] (Installing-the-OpenQuake-Engine-from-source-code-on-Fedora-and-RHEL.md) The engine works on several Linux distributions, even recent ones like Ubuntu 15.04. It has less dependencies than it used to have in the past and it is easier to install, so it should be relatively simple to install it on any modern Linux distribution.

While the engine is not supported on Windows and Mac OS, we are happy to report that the underlying libraries and the oq-lite command-line tool run just fine. We do not offer any automatic tool to perform the installation, but there is a guide to help you to install the necessary dependencies.

Bug fixes and changes with respect to OpenQuake 1.4#

The database schema has changed in a destructive way, by removing a column in the hzrdi.hazard_site table and a column in the riskr.epsilon table. If your database contains important data, export them or dump the database. You will not be able to user OpenQuake 1.5 with an OpenQuake 1.4 database. The upgrade procedure is the usual one oq-engine --upgrade-db.
Over 30 new tests have been added for the event based risk calculator, and a few new tests have been added also for the event based hazard calculator. It was a huge effort on the part of both our scientific team and IT team. The net result is that a lot of subtle and hard-to-find bugs have been discovered and fixed.
The algorithm to compute average losses and average insured losses in the event based risk calculator has been changed: it is now more robust since it does not rely on the discretization of the loss curves, but directly on the underlying loss tables. As a consequence the numbers for the average losses are different than in previous versions of OpenQuake. The difference is compatible with the error that we had before.
The event-based disaggregation feature has been removed; same for event-based Benefit-Cost Ratio calculator. They were buggy and they will be reintroduced in the future within the new system, in the engine codebase or as part of the Risk Modeller’s Toolkit.
Longitude and latitude are now rounded to 5 digits after the decimal point directly from Python; earlier the rounding happened inside PostGIS. As a consequence, if the locations of your assets have more than 5 digits after the decimal point, there will be small differences in the numbers produced by the engine, compared to previous versions.
The parameter investigation_time has been replaced by risk_investigation_time in risk configuration files
The bin/openquake wrapper, which has been deprecated for ages, is now removed. Now only bin/oq-engine is available
The parameter concurrent_tasks is read from the .ini file and honored; in earlier versions it was read from the openquake.cfg file, but was being ignored by the risk calculators.
We changed the convention used to generate the rupture tags; now the tags do not contain pipes “|” as the tag separators. This character caused problems on Windows, since one of the NRML converters was using the tag to generate a file with the same name containing the corresponding ground motion field.
We changed the export order of the event loss table. Earlier, the values were ordered by loss size, in decreasing order. Now they are first ordered by rupture tag, and then by increasing loss size. We feel that this ordering is more useful.
We have added some checks on source IDs and asset IDs, to avoid having issues such as nonprintable characters or non-ASCII characters in there. Also, we have limited the maximum length of an identifier to 100 characters. Notice that descriptions are unconstrained, as before, so there are no problems if you want to name your sources using Chinese characters or in any other character set. The only restriction is that the XML file must be UTF8-encoded, according to the XML standard.
If you don’t have a site model file, you need to provide values in the job.ini file only for those site parameters that are actually used by your calculation. In earlier versions, users were asked to specify site parameters even if they weren’t required for the calculation.
We fixed a bug with the oq-engine --load-curve command, such that is was impossible to load a hazard curve.
We improved the error reporting on the engine; earlier, an error in the cleanup phase could hide the real underlying error.
We fixed an error for the degenerate case of hazard curves containing all zeros, as this corner case was reported by some users on the OpenQuake users group.
Now the sites are ordered by longitude, latitude even when they are extracted from a region, consistently with all other cases.