Ctrl+K

Release notes v2.4#

This release introduces several changes and improvements to the engine calculators, most notably in the postprocessing of hazard curves and in the postprocessing of loss curves. Also, several exporters have been revised and several have been added, since there are new kinds of outputs. A few important bugs have been fixed and a few spectacular optimizations have been added.

More than 40 pull requests were closed in oq-hazardlib and more than 200 pull requests were closed in oq-engine. For the complete list of changes, please see the changelogs: https://github.com/gem/oq-hazardlib/blob/engine-2.4/debian/changelog and https://github.com/gem/oq-engine/blob/engine-2.4/debian/changelog.

We have also decided a plan for dropping support to Python 2. We will abandon that platform in the course of the year 2018. Precise dates have not been fixed yet - on purpose - but everybody who is using hazardlib and/or the engine as a library should think about migrating to Python 3. Please contact us if this is a problem for you: we are keen to provide advice for the migration.

New features in the engine#

There are several new outputs/exporters.

  • There is a new output and a new .hdf5 exporter for the event based risk individual asset losses. This is an optional output, generated only when you set the flag asset_loss_table=true in the job configuration file, or when the parameter loss_ratios is set.

  • There is an equivalent output for the scenario_risk calculator, called all_losses-rlzs, and an .npz exporter for it. This is also an optional output generated when you set asset_loss_table=true.

  • There is a new output losses_by_event produced by the scenario_risk calculator, which is similar to the aggregate loss table in the event based calculator: it contains the aggregate losses for each simulation of the scenario. There is a new CSV exporter for it.

  • There are new outputs for the losses aggregated by taxonomy which are produced by the calculators scenario_risk, event_based_risk and ucerf_risk. There are CSV exporters associated.

  • There is a new CSV exporter for the loss curves, while the old one has been removed.

  • There are two new .npz exporters for the loss maps and the losses by asset.

  • There is a new CSV exporter for the benefit-cost-ratio calculator.

  • There is a new experimental CSV exporter for the ground motion fields produced by the event based calculator.

  • The XML exporter for the output ruptures now contains only the ruptures, without any duplication, and, for each event set, the list of events associated to the rupture. Before there was a rupture for each seismic event, with potential duplications.

There is a new configuration parameter max_hazard_curves in the job file, which by default is False. This parameter controls the generation of the maximum hazard curves, a new feature.

There is a new configuration parameter max_site_model_distance in the job file, with a default 5 km: before it was hard-coded. This parameter controls the warnings for site model parameters which are far away from the hazard sites.

There is a new configuration parameter ses_seed parameter in the job file. It is the seed used in the generation of the ruptures in the event based calculator, which is now distinct from the seed used in the sampling of the source model, which is controlled by the random_seed parameter.

The parallelization library has been improved; now the task_info and job_info datasets are automatically stored at the end of a calculation.

It is now possible to split a source model into several files and read them as if they were a single file. Just specify the file names in the source_model_logic_tree file. For instance, you could split by tectonic region and have something like this:

 <logicTreeBranch branchID="b1">
   <uncertaintyModel>
     active_shallow_sources.xml
     stable_shallow_sources.xml
   </uncertaintyModel>
   <uncertaintyWeight>
     0.3
   </uncertaintyWeight>
 </logicTreeBranch>

Optimizations#

The postprocessing of hazard curves has been substantially improved: now it requires orders of magnitude less memory than before and it is extremely efficient. For instance in our cluster we were able to compute mean and max hazard curves in a calculation for a Canada model with 206,366 sites, 129 hazard levels and 13,122 realizations - spawning ~3,500 tasks and using ~500 GB of memory spread on four machines - in just 11 minutes. This is orders of magnitude better than everything we ever managed to run before.

There is also a huge improvement in the storage of the hazard curves: the individual hazard curves are not stored anymore, unless there is only one realization. This means that in large computations we can save orders of magnitudes of data storage: for instance for the Canada computation the saving in space is from 1.27 TB to 5.44 GB (240x improvement!).

Consequent to the change above, the way we export the hazard curves, maps and uniform hazard spectra has changed. It is still possible to export the individual curves in a multi-realization calculation, but it requires a new command:

$ oq export hcurves/rlz-XXX # export the realization number XXX
$ oq export hmaps/rlz-XXX # export the realization number XXX
$ oq export uhs/rlz-XXX # export the realization number XXX

Since the generation of the curves is done at runtime the export will be slower than before, but this is a good tradeoff. After all, most users will never want to export the full set of realizations (for those few users we still have an efficient HDF5 exporter doing exactly that, please ask for info if you need it).

Most users will be interested just in the statistical hazard curves, maps and hazard spectra. Such outputs can be exported exactly as in the past and without any performance penalty, since the statistical curves are computed and stored in postprocessing. Also, it should be noted that now by default the mean hazard curves are computed.

The postprocessing of loss curves has been improved too: now the loss curves and maps are produced by reading directly the asset loss table from the workers, rather than passing them via the celery/rabbitmq combo. The same is true for the hazard curves, but in that case the data transfer issue is less dramatic. This change has improved the scalability of the engine significantly and now we can easily compute millions of loss maps. It should be noted that in a cluster, the new approach requires a shared file system, otherwise the postprocessing will use only the cores of the controller node.

While the loss maps are stored as before, in the event based calculator the loss curves have become a dynamic output, generated at runtime from the asset loss table (this is the same approach used for the hazard curves, generated at runtime from the ProbabilityMaps). This approach saves a huge amount of disk space. As a consequence of the change the engine does not show the loss curves output anymore; however the curves can still can be exported, but with a new command:

$ oq export loss_curves/rlz-XXX # export the realization number XXX

Notice that CSV format that we are using now is different and a lot more readable than the one we used in the past which was not even a proper CSV (it was used for internal purposes only).

Among the huge improvements to the event based calculators one of the most relevant is the fact that the ruptures are being saved and read from the datastore in HDF5 format: before they were stored in pickle format, for historical reasons.

The classical_risk calculator now reads the ProbabilityMaps (in the past it read the individual hazard curves that are not stored anymore): therefore the required data transfer has been significantly reduced and the performance of the calculator has improved. This is visible only for source models with a lot of realizations, though.

The reading of the Ground Motion Fields from the datastore in scenario_risk and scenario_damage calculators has been optimized: the effect is small in small computations, but in a realistic calculation with thousands of sites and GMFs we measured a speedup from 5 hours to 45 seconds.

The performance of the event based calculator has been improved substantially in the GMF-generation phase. We measured improvements of a factor 2 in speed and a factor 3 in memory occupation, but they can be larger or smaller, depending on the size of your computation. The larger the calculation, the more sensible is the improvement.

We are now using rtree to get the nearest site model parameters: this gives more than one order of magnitude speedup in calculations that were dominated by the site model associations. This is more and more important now that we are able to tackle computations with 100,000+ assets.

User-visibile changes#

The event tags (strings) that we had in the past, visible to the users in the rupture exporter and event loss table exporter are now gone, replaced by the event IDs, which are 64 bit integers. In the past the event IDs were internal indicators, hidden in the datastore and not exposed to the user. Now they are exported because, thanks to a major refactoring, the event IDs have become unique within a given calculation and reproducibile, provided the parameter concurrent_tasks is fixed. This makes it possible to connect directly the data saved in the datastore with the exported data, a feature wanted by several power users for years.

The CSV exporter for the output dmg_total in damage calculators now presents the data in a more readable format. The same has been done for the aggregate losses exported by the scenario_risk calculator.

The CSV exporter for the output ruptures is slightly different: the field serial has been renamed as rup_id, the field eid has been removed and the ordering of the fields is different.

The event loss table exporter is now producing an additional column rup_id containing the rupture ID.

We renamed the csq_ outputs of the scenario_damage to losses_.

We renamed the datasets avg_losses- to losses_by_asset-.

We removed the output rup_data which was internal and not meant for the final user.

We changed the .npz export to export even GMFs outside of the maximum distance, with zero value: this makes it easier to visualize the results.

We removed the CSV exporter for the all_loss_ratios output in scenario_risk: now there is an .npz exporter.

We removed the GSIM name from the exported file name for the scenario calculators.

The .npz export for scenario calculations now exports GMFs outside the maximum distance, which are all zeros. This is convenient when plotting and consistent with CSV export.

hazardlib#

The Hazard Modeller Toolkit (HMTK) has been merged into the oq-hazardlib repository. The old repository is still there, for historical purposes, but it is in read-only mode and has been deprecated. All development has to be made in oq-hazardlib now. If you have scripts depending on the HMTK you will have to change your imports: import hmtk will become from openquake import hmtk.

All the routines to read/write hazard sources and ruptures (and their tests) have been moved from the engine into hazardlib, which is now self-consistent in that respect.

We have extended the XML serializer to fully support mutually exclusive sources and the calculator calc_hazard_curves has been fixed to manage this case correctly.

We added an utility openquake.hazardlib.stats.max_curve to compute the maximum of a set of hazard curves. This is a lot more efficient than computing the quantile with value 1.0.

The mesh of a rupture is now stored as 32 bit array of floats instead of a 64 bit array: this reduces the memory consumption and data storage by half.

We fixed a numerical issue involving the square root of small negative numbers, due to rounding issues.

We have refactored the filtering mechanism and we have now a single IntegrationDistance class in charge of filtering out sites and ruptures outside of the integration distance. For sake of correctness, we have disabled the rtree filtering if the site collection contains points which are not at the sea level. In that case the geodetic distance filtering is used.

There is now a new class EBRupture in hazardlib to describe event based ruptures as a raw rupture object and an array of events: previously this class was in the engine. The change make it possible to serialize (read and write) groups of event based ruptures in NRML format.

We fixed an ordering bug in get_surface_boundaries: now it returns the points in clockwise order, i.e. the right order to produce a WKT bounding box which is used to plot the rupture with our QGIS plugin.

We extended and refactored the GmfComputer class which is used by the engine in scenario and event based calculations.

There is a new constructor for the Polygon class which is able to parse a WKT polygon string.

We fixed a bug when splitting sources with a YoungsCoppersmith1985MFD magnitude frequence distribution that made impossible to run such calculations in some cases (depending on the splitting).

WebUI#

We added an endpoint GET /v1/available_gsims to the REST API underlying the WebUI. This is used by the OpenQuake platform to extract the list of available GSIM classes. Now the platform uses the engine as a service and it does not import directly any code from it.

We changed the Web UI button from “Run Risk” to “Continue”, since it can also be used for postprocessing of hazard calculations.

All the engine outputs are streamed from the WebUI. This saves memory in the case of large outputs.

We added an output corresponding to the fullreport to the WebUI: this is extremely useful to get information about a given calculation.

The WebUI automatically creates the engine database at startup, if needed.

The DbServer could not start in MacOS Sierra, due to a change in the low level libraries used in that platform that made it impossible to fork sqlite; we worked around it by using a ThreadPoolExecutor instead of a ProcessPoolExecutor.

We fixed a bug when deleting a calculation from the WebUI: now it is possible to do so if the user is the owner of that calculation.

The command oq webui start now gives a clear error message if it is run from a multi-user package installation.

We removed a wrong and now useless check about Django 1.5.

We have now a desktop icon for the OpenQuake WebUI both for Linux and Windows platforms.

The logic about how local_settings.py is found has changed: first it is searched in the folder where the openquake.dbserver is started from; if it does not exist in the folder it will be searched in the openquake.server location itself. If no local_settings.py is provided at all, default settings will be used.

Bugs#

The was a size limit on the event ID (65,536 events for task) that could be exceeded in large calculations. We raised that limit to over 4 billion events per task.

We fixed a long standing bug in the event based risk calculation. In some cases (when the hazard sites were given as a region) it was associating incorrectly the assets to the sites and produced bogus numbers.

We fixed a couple of bugs affecting exposures with multiple assets of the same taxonomy on the same site: that made it impossible to run classical_risk and scenario_risk calculations for such exposures.

We fixed an annoying encoding bug in the commands oq engine --lhc and oq engine --lrc which affected the display of calculations with non-ASCII characters in the description.

We fixed a bug in event_based_risk: it was impossible to use vulnerability functions with “PM” distribution, i.e. with Probability Mass Functions. Now they work as expected.

The .npz export for scenario calculations has been fixed in the case of a single event, i.e. when number_of_ground_motion_fields=1, which was broken.

Additional validations#

The engine is more picky than before. For instance if an user specifies quantile_loss_curves or conditional_loss_poes in a classical_damage calculation, the user will now get an error, since such settings make no sense in that context. Before they were silently ignored.

If an exposure contains assets with taxonomies for which there are no vulnerability functions available, now the user gets a clear error before starting the calculation and not in the middle of it.

If an user provides a complex logic tree file which is invalid for the scenario calculator, now the user gets a clear error message.

There are more checks for patological situations, like the user providing no intensity measure sites, no GSIMs, no sites: now a clearer error message will be displayed.

Experimental new features#

In this release the work on the UCERF calculators has continued, even if they are still officially marked as experimental and left undocumented on purpose.

There ucerf_classical calculator has been extended to work with sampling of the logic tree. The rupture filtering logic has been refactored and made more consistent with the other calculators.

The ucerf_rupture calculator has been extended so that we can parallelize by number of stochastic event sets. This improvement made it possible to run mean field calculations in parallel: before such calculations used a single core.

The data transfer has been hugely reduced in the calculator ucerf_risk: now we do not return the rupture objects from the workers, but only the event arrays, which are enough for the purposes of the calculator. This saved around 100 GB of data transfer in large calculations for California.

We fixed a bug in ucerf_risk that prevented the average losses from being stored. Now this works out of the box, provided you set avg_losses=true in the job.ini file.

There is a brand new time-dependent UCERF classical calculator.

We started working on an event based calculator starting from Ground Motion Fields provided by the user. The current version is still very preliminary and requires the GMFs to be in NRML format, but we plan to extend it to read the data from more efficient formats (CSV, HDF5) in the near future.

We added a facility to serialize Node objects into HDF5 files. This is the base for a future development that will allow to serialize point sources into HDF5 datasets efficiently (scheduled for engine 2.5).

We introduced some preliminary support for the Grid Engine. This is useful for people running the engine on big clusters and supercomputers. Unfortunately, since we do not have a supercomputer, we are not able to really test this feature. Interested users should contact us and offer some help, like giving us access to a Grid Engine cluster.

Internal changes#

As always, there were several internal changes to the engine. They are invisible to regular users, so I am not listing all of the changes here. However, I will list some changes that may be of interests to power users and people developing with the engine.

The parameters ses_per_logic_tree_path and number_of_logic_tree_samples are constrained to a maximum value of 65,536 only in UCERF now.

As usual the layout of the datastore has changed; in particular the way the GMFs and the events are stored is different.

In the past running the tests littered your file systems with lots of generated files, both in the current directory and in the /tmp directory. Now the tests never write on the current directory and they cleanup the /tmp directory (if they are successful).

There is an internal configuration flag ignore_covs which is needed to disable the use of the coefficients of variations in vulnerability functions, for debugging purposes. Now this flag works for scenario_risk calculations too. Before it was restricted to event_based_risk.

In release 2.3 we introduced temporarily an ebrisk calculator. It is gone now, just use the good old event_based_risk calculator.

Internal TXT exporters for the ground motion fields, used only for the tests, have been removed.

Packaging#

Matplotlib is now a requirement for the OpenQuake Engine and hazardlib. It’s included in the OpenQuake installers and packages as a Python wheel. Basemap can be installed to enable some extra plotting features from the Hazard Modeller Toolkit. Basemap is provided as pre-compiled Python wheel, it’s included in installers and in the python-oq-libs-extra package on Ubuntu and RedHat/CentOS. python-oq-libs-extra isn’t required on a headless server setup.

A safety measure has been introduced to check that the OpenQuake Engine is talking to the proper DbServer instance. This helps debugging issues in the case of multiple installations of the engine (for instance a system-wide multi-user installation and a single-user development installation coexisting on the same machine).

h5py has been updated to version 2.7.0.

A template for PAM authentication is now provided. This allows the WebUI to authenticate users against system users on a Linux server.

The RPM python-oq-engine package has been splitted into python-oq-engine, python-oq-engine-master and python-oq-engine-worker. This reduces the amount of dependencies needed by python-oq-engine when installed on a single node. Specific configurations for master and workers nodes are provided by dedicated packages. This setup will be ported to Ubuntu packages too in the next release. See the documentation for further information.

Deprecations#

As of now, all of the risk XML exports are officially deprecated and will be removed in the next release. The recommended exports to use are the CSV ones for small outputs and the .npz/HDF5 ones for large outputs.

Python 2.7 is not officially deprecated yet, but it will be deprecated soon. The version we use for development and production since the beginning of 2017 is Python 3.5. Here is our roadmap for the future.

On this page
Show Source