Release notes v2.5 ================== This release introduces a tiling mechanism for classical PSHA calculations. The tiling reduces the memory consumption and increases the performance of large calculations: the improvement is sensible for calculations dominated by distance computations, less so in other cases. There is a new `MultiPointSource` object in hazardlib, with its own XML serialization protocol. Serialized MultiPointSources can be up to an order of magnitude smaller than equivalent PointSources. MultiPointSource are more efficient to read, to write and to transfer. The computational performance is more or less the same as before, though. This is the first release integrating hazardlib into the engine: there are no more hazardlib packages. The change is transparent to the users of the official packages/installers, i.e. there is nothing to do for them. However, scientists and developers using the `oq-hazardlib` repository from GitHub or the `openquake.hazardlib` package from PyPI should [remove it manually](https://github.com/gem/oq-engine/blob/engine-2.5/doc/faq.md#openquake-hazardlib-errors) to avoid possible confusions. Several bugs have been fixed and there were a few improvements to the Web User Interface(WebUI) and to the engine itself. More than 70 pull requests were closed. For the complete list of changes, please see the changelog: and https://github.com/gem/oq-engine/blob/engine-2.5/debian/changelog. Major new features ------------------------------ The major new feature is the return of the tiling calculator. We had this feature in the past, but the current version is a lot more efficient than the previous one. The reason is that the filtering of point sources has been significantly improved and it is no more a bottleneck of the calculator. The tiling starts automatically if there are more than 20,000 hazard sites, but you can change the default by setting the parameter `sites_per_tile` in your `job.ini` file. Notice that there is no tiling for event based or scenario calculators, only for classical PSHA. The MultiPointSources are a new feature and as such thet are not documented in the manual (yet); however they are documented [in the engine doc folder](https://github.com/gem/oq-engine/blob/engine-2.5/doc/multipoint.md) and you are invited to try them. A new configuration parameter `disagg_outputs` has been introduced. Its purpose is to reduce the number of `.csv` files generated by the disaggregation calculator. By default several files are generated for each of the 8 kinds of disaggregation outputs: `Dist, Lon_Lat,_Lon_Lat_TRT, Mag, Mag_Dist, Mag_Dist_Eps, Mag_Lon_Lat, TRT` The number of exported files depends on the number of intensity measure types, realization and sites; it can easily become quite large. Thanks to the `disagg_outputs` feature, the user can specify on the `job.ini` file which kind of outputs should be exported. For instance, if the user is interested only in the magnitude-distance disaggregation, the user can set `disagg_outputs = Mag_Dist` and export 8 times less files. `disagg_outputs` can list different disaggregation outputs separated by spaces or commas. It should be noticed that there is no performance improvement, because all disaggregation outputs are always generated, the change is only in the export, not in the calculation. There were several improvements to the Web UI: - the command `oq webui start` now open a browser window, if possible; - we added the commands `oq webui createsuperuser` and `oq webui collectstatic` which are useful for system administrators setting a multiuser instance of the WebUI (see https://github.com/gem/oq-engine/blob/engine-2.5/doc/installing/server.md); - there is a confirmation dialog when removing a calculation; - we added an end point `v1/calc/XXX/oqparam` to extract the calculation parameters as a JSON dictionary, which is used by the [QGIS plugin](https://plugins.qgis.org/plugins/svir/). Bugs fixed ---------------- - getting the version of the engine required having git installed on macOS - there was an encoding error when logging filenames with non-ASCII character affecting macOS users - the XML rupture exporter was not saving the first `ruptureId` for scenario calculations - Now we raise an early error if there are missing taxonomies in the consequence model - `oq export hcurves-rlzs --exports hdf5` was broken - the hazard maps were not exposed to the engine in event based calculations - there were some packaging issues in the Red Hat packager - there was a bug in `dbserver.different_paths` affecting people with an engine installed in a directory involving symbolic links. - there was bug in the debugging utility `oq plot`; still the official way to display the outputs of the engine is the [QGIS plugin](https://plugins.qgis.org/plugins/svir/). Other improvements --------------------- - The engine was calling the routine computing the statistics even when not needed. This was inefficient and has been fixed. - The error checking when parsing source models in format NRML 0.5 has been improved: now in case of error one gets the name of the incorrect node and its line number, just like for NRML 0.4. - There is now a clear error message if the user does not set the `calculation_mode` in the `job.ini` file. - We improved the error message when the rupture mesh spacing is too small. - We added a new `.npz` exporter for the output `dmg_by_asset_npz`. - There is a new `.csv` exporter for the aggregate loss curves, replacing the deprecated XML exporter. - Some preliminary work for the Python 3 installers has been done. As always, there were several internal changes to the engine. Some of them may be of interests to power users and people developing with the engine. - It is now possible to use the engine with backends different from rabbitmq, for instance with redis; we did some experiment in this direction, but rabbitmq is still the official backend to use. - The AreaSource class in hazardlib is no more a subclass of PointSource (that was an implementation accident). - The syntax of the command `oq db` has been improved. - The `composite_source_model` has been removed from the datastore: this was the last pickled object remaining there for legacy reasons. - We changed the way the logic tree reduction works in event based calculators: now it works the same as in classical calculators. The change may affect rare corner cases, when there are source groups producing zero ruptures; see https://github.com/gem/oq-engine/pull/2840 for the details. Deprecations ------------------------------ All of the deprecated XML exports for the risk outputs have been finally removed. The repository https://github.com/gem/oq-hazardlib has been deprecated and new pull requests for hazardlib should be opened towards the engine repository. [Our roadmap for abandoning Python 2](https://github.com/gem/oq-engine/issues/2803) has been updated.