Release notes v3.21
===================

Version 3.21 is the culmination of 4 months of work involving over 240 pull requests.
It is the release featuring the greatest performance improvements of the last few years.
It is aimed at users wanting the latest features, bug fixes and maximum performance.
Users valuing stability may want to stay with the LTS release instead
(currently at version 3.16.8).

The complete set of changes is listed in the changelog:

https://github.com/gem/oq-engine/blob/engine-3.21/debian/changelog

A summary is given below.

# Classical calculations

We significantly reduced the memory consumption in classical calculations:
now all the models in the GEM mosaic run with less than 2 GB per core
(previously we required 4 GB per core). This result was mostly achieved by reducing
the precision of the PoEs to 32 bit instead of 64 bit, which required a
major refactoring and working in terms of rates, since rates are much less
subject to numerical errors than PoEs.

The other mechanism to reduce the memory consumption was changing the
implementation of CollapsedPointSources to use arrays rather than
plain source objects. To give a concrete example, in the case of the
USA model that reduced the memory consumption (when using the tiling strategy)
from about 3 GB per core to just 1.2 GB per core.

Moreover, CollapsedPointSources have been extended to collect
sources with different magnitude-scaling-relationships, which also makes
the engine faster, however the change affects only a few models with minor differences.

Thanks to the memory reduction and other optimizations, like refining the size
of the arrays when computing the mean and standard deviations from the
GMPEs, or optimizing the procedure updating the probabilities of no
exceedency, now 3x faster, the performance of classical calculations
improved substantially. For instance, the model for EUR is nearly 2x
faster than before, the model for Canada is 1.4x faster, while the
model for USA is 1.25x faster on a machine with 128 cores.

We added a `tiling` flag in the job.ini file to make it possible to specify
which version of the classical calculator to use (tiling or regular);
if the flag is not specified, the engine will continue to autonomously
select the most appropriate calculator, however with a different logic
than in the past.

We optimized the logic tree processor so that it is now possible
to compute mean rates exactly (if `use_rates` is set) even with
millions of realizations. In particular we managed to run the EUR model
with 302,990,625 realizations, but it required over 100 GB of RAM.

The splitting of the sources in groups now works slightly differently and more
groups can be generated sometimes, but there is an additional guarantee: the sources in a
same group will all have the same temporal occurrence model. This allowed some
internal simplification.

The --sample-sources feature now works differently (by tiling)
and it is faster (measured 4x faster for the USA model) and more reliable than before.

Finally, logging the progress of the calculation now works much better (previously
often the progress went from 0 to 50% abruptly and then from 50% to 100%
equally abruptly).

# Event based/scenario calculations

In event based calculations we were able to spectacularly reduce the
memory consumption in situations with millions of sites. For instance,
whereas previously a calculation with 6 million sites required over 80 GB per
core and it inevitable ran out of memory, now it requires less than 2 GB per core
and it completes correctly.

Moreover, the performance improved tremendously in the case of many GMPEs, like
for EUR, where there are up to 19 GMPEs per tectonic region type. The
reason is that some calculations were needlessly repeated in the GmfComputer.
Now generating the GMFs in the EUR model is 13 times faster than
before. In models with only 2 or 3 GMPEs the improvement is
less visible, but still there.

We decided to make the parameters `minimum_magnitude` and `minimum_intensity` mandatory,
except in toy calculations. The reason is that users often forget to set such parameters
and as a consequence their calculations become extremely slow or run out of memory after
many hours. Now they will get a clear error message even before starting the calculation.

If the `custom_site_id` field is missing in the site model, 
we are now generating it automatically as a geohash,
since it is extremely useful when comparing and debugging GMFs.

Some users wanted to be able to access intermediate results (in
particular the GMPE-computed mean and the standard deviations tau and
phi) for each site, rupture, GMPE and intensity measure type. This is now
possible by setting the flag

mea_tau_phi = true

in the job.ini file. Please notice that setting the flag will double
the disk space occupation and make the calculation slower, so you are
advised to enable this feature only for small calculations for
debugging purposes. It will work both for scenarios and event based
calculations.

We changed the scenario calculator to discard sites over the integration
distance from the rupture, for consistency with the event based calculator
and for consistency with the algorithm used in Aristotle calculations.

# hazardlib

Enrico Abcede and Francis Bernales extended the Campbell and Bozorgnia (2014)
GMPEs to work with the intensity measure types IA and CAV.

Fatemeh Alishahiha contributed a couple of GMPEs: one
with the Zafarani et al (2018) and one with the Ambraseys (2005),
relevant for Iran.

Chris di Caprio contributed a bug fix for the Kuehn (2020) GMPE, in
the RegularGridInterpolator, that must use extrapolation to handle
floating point edge cases.

Eric Thompson pointed a typo in the aliases for the Kuehn et al GMPE
("KuehnEtAl2021SInter" in place of "KuehnEtAl2020SInter") and missing
aliases for "KuehnEtAl2020SSlab". It has been fixed now.

Kyle Smith contributed the Arias Intensity and Cumulative Absolute
Velocity ground motion models from Sandikkaya and Akkar (2017).

Nicolas Schmid fixed an incorrect formula used in ShakeMap calculations
(see https://github.com/gem/oq-engine/pull/9890).

Marco Pagani extended the Chiou Youngs (2014) GMPEs to optionally
include the modifications by Boore Et Al. (2022).

We added a helper function `valid.modified_gsim` to instantiate ModifiableGMPE
objects with less effort, for usage in the GEESE project, You can see an example in
https://docs.openquake.org/oq-engine/master/manual/user-guide/inputs/ground-motion-models-inputs.html#modifiablegmpe.

Since the beginning hazardlib has been affected by a rather annoying
issue: pollution by large CSV files. CSV files are required
by the GMPE tests to store the expected mean and standard deviations
computed from a set of input parameters like magnitude, distance and vs30.

Unfortunately people tended to be liberal with the size of the CSV
files, including files larger than needed, and we ended up with 480 MB
of expected data. The sheer amount of data slows down the download of
the repository, even with the shallow clone, and makes the tests
slower than needed.

We were able to cut down the CSV files by two thirds without losing
any test coverage. Moreover, we added a check forbidding CSV files
larger than 600k to keep the problem from reappearing in the near
future.

# AELO project

Work on the [AELO project](https://www.globalquakemodel.org/proj/aelo)
continued with various AELO year 3 updates. We can now
run all available IMTs (before we considered only 3 IMTs) and use
the ASCE7-22 parameters (see https://github.com/gem/oq-engine/pull/9648).

At the level of the Web Interface it is now possible to choose
the desired `asce_version` between ASCE 16 and ASCE 41. The version
used to perform the calculation is then displayed in the outputs page.

We began the work for supporting vs30 values different from 760 m/s by implementing
the ASCE soil classes.

We added exporters for the Maximum Considered Earthquake and for the
ASCE07 and ASCE41 results.

The `geolocate` functionality has been extended to support
multipolygons, which are used to model geometries crossing the
International Date Line.

As a consequence, a point exactly in the middle of two mosaic models
can be associated differently than before.

# Aristotle project

We continued the work on the
[Aristotle project](https://www.globalquakemodel.org/proj/aristotle).

Given a single file `exposure.hdf5` containing the global exposure,
global vulnerability functions, global taxonomy mappings and global site
model, and a rupture file or ShakeMap event id, an Aristotle calculation 
is able to compute the seismic risk in any place in the world, apart from the oceans.

In this release, the procedure to generate the `exposure.hdf5` file
from the global risk mosaic has been improved and the taxonomy
mappings for Canada and USA, which were missing, have been added.

Moreover the Aristotle web interface has been much improved. It is
possible for the user to download from the USGS site not only rupture
files but also station files, thus taking advantage of the Conditioned
GMFs calculator. Custom rupture files or station files can also be uploaded if the user
has them locally. Errors in the station files are properly notified,
just as errors in the rupture geometries (not all USGS ruptures can be
converted to OpenQuake ruptures yet).

When the rupture.json file is not available on the USGS page for an event,
the software checks if the finite-fault rupture is available and it loads
the relevant information from it instead. Note that this fallback approach may
also fail, especially in the early stage right after an event,
when not enough information has been uploaded to the USGS website yet.
In this case, in order to proceed, the user needs to upload the rupture
data via the corresponding web form.

We improved the procedure for generating hazardlib ruptures, automatically setting
the `dip` and `strike` parameters when possible. In case of errors in
the geometry a planar rupture with the given magnitude, hypocenter dip
and strike is automatically generated.

The visualization of the input form and the output tables has improved.

It is now possible to specify the parameter `maximum_distance_stations`,
either from command line or from the WebUi. The `maximum_distance` was
already there, but now has a default of 100 km to ensure faster calculations.

We added a dropdown menu so that the user can override the
`time_event` parameter, which by default is automatically set
depending on the local time of occurrence of the earthquake.

The page displaying the outputs of a calculation now includes some additional
information. On top of the page, the local time of the event is displayed,
together with the indication of how much time passed between the event itself
and the moment the calculation was started. This is useful for instance to
compare results obtained before and after rupture or station data become
available in the USGS site.

The plots showing the average ground motion fields and the assets have been
improved. In particular, the latter now also display the boundaries of the
rupture.

We added a method `GsimLogicTree.to_node`, useful to produce an XML
representation of the gsim logic tree used by an Aristotle calculation.

# Secondary perils

Lana Torodović implemented the Nowicki Jessee et al. (2018) landslide
model, which is part of the USGS Ground Failure Product. It is documented here: 
https://docs.openquake.org/oq-engine/3.21/manual/underlying-science/secondary-perils.html#nowicki-jessee-et-al-2018

When exporting the `gmf_data` table, we decided to include the secondary peril
values, if present. Technically they are not GMFs, just fields induced by the GMFs,
but it is convenient to keep them in the `gmf_data` table, also for visualization
purposes via the QGIS plugin.

# Bug fixes

There were various fixes to the conditioned GMFs calculator:

- we changed the Cholesky decomposition algorithm to reduce the issue of
  small negative eigenvalues
- we fixed an error when using a `region_grid_spacing` by extending the site
  collection to include the stations
- we fixed a couple of bugs in the association of the site parameters to the sites
- we added an uniqueness check for the station coordinates

We fixed a bug with the conditional spectrum calculator in the case of calculations
with non-contributing tectonic region types, i.e. when all the sources of a
given tectonic region type are outside the integration distance.

Running a scenario or event based calculation starting from a CSV file of ruptures raised
a KeyError: "No 'source_mags/*' found in <DataStore>"
when exporting the ruptures. It has been fixed now.

The new Japan model was failing with an error in event based calculations due
to an excessive check (a leftover from the past). It has been fixed now.

The `avg_gmf` was failing sometimes in the presence of a filtered site collection.
It has been fixed now.

In situations with more than ~90 realizations, exporting the realizations CSV
was causing an encoding error. It has been fixed now.

In presence of multi-fault sources far away from the sites (i.e. over the integration
distance) an error "object has no attribute msparams" was raised. It has been fixed now.

We improved the error message displayed when the user forgets to specify the exposure file
in a risk calculation.

We removed the exporter for the output `disagg-stats-traditional` since it was
incorrect, with no easy way to fix it. We kept the exporter for
`disagg-rlzs-traditional`, which was correct.

# Using half the memory on Windows

Recent processors have significantly more threads than in the past,
however the typical amount of RAM on consumer machines has not
increased much. Moreover, often a significant amount of RAM is reserved to
the GPU and a huge amount of memory is eaten by the browser.

Therefore, if the engine kept spawning a process for each thread
as in the past, it would likely run out of memory or degrade the performance
to the so-called "slow mode", with a single thread being used.

To cope with this situation, now the engine on Windows uses by default
only half of the available threads. You can change this default by
setting the parameter `num_cores` in the `[distribution]` section of
the file `openquake.cfg` in your home directory. Be warned that
increasing `num_cores` will increase the memory consumption but not
necessarily the performance.

On non-Windows platforms nothing changes, i.e. the default is still to
use all available threads. The reason is that Apple silicon does not
use hyperthreading (i.e. it uses half the memory compared to
processors with hyperthreading) while Linux is mostly used for servers
that typically have more memory than a consumer machine, so there is no
pressing need to save memory.

# HPC support via SLURM

Since version 3.18 the engine has experimental support for HPC clusters
using the SLURM scheduler. However, the original logic required spawning a SLURM
job for each engine task, meaning that we immediately run into the limit of 300
SLURM jobs when the CEA institute graciously gave us access to their cluster.

To overcome this limitation, the approach used in engine 3.18 has been
abandoned and the code completely rewritten. With the current approach there is a single
SLURM job for each engine job and we can easily scale to thousand of cores by simply
passing the number of desired nodes to the `oq engine --run` command (see
https://docs.openquake.org/oq-engine/3.21/manual/getting-started/installation-instructions/slurm.html)

Notice that supporting thousands of cores required substantial work on the calculators too.

For instance, now classical calculations make use of the `custom_tmp`
directory in `openquake.cfg` (mandatory in SLURM mode) to store the
PoEs in small temporary files, one per task. In tiling calculations
that reduces the data transfer effectively to zero, while in
non-tiling calculations it only reduce it, more or less depending on
the calculation. That required to change the postclassical phase of
the calculation to be able to read the PoEs from the `custom_tmp` and
only from the `calc_XXX.hdf5` file.

Moreover we had to invent a mechanism to limit the data transfer
in non-tiling calculations, otherwise we would still run out of memory on the
master node in large HPC clusters simply by adding new nodes.

The tiling feature is now less used than before.
For instance, prior to engine 3.21 the USA hazard model was executed by using the tiling
strategy. While this is acceptable if you have say 128 cores, it is extremely inefficient
if you have 1024 cores or more. Thus in engine 3.21 we went through great effort to avoid
the tiling approach when it is inefficient: in particular the USA model with 1024 cores is
5x faster by not using tiling.

Be warned that the more cores you have, the more you are in uncharted
territory and it is entirely possible that adding nodes will slow down your calculation.
We recommend not to exaggerate with the number of nodes, 8-16 nodes with 128 cores each
have been tested, using more nodes comes at your risk.

The SLURM support is still considered experimental and we know that there
are still a few issues, like the fact that the SLURM job may not be
killed automatically in case of hard out-of-memory situations and
therefore must be killed manually, or the presence of slow tasks in
some calculations, however it is good enough to run all the hazard models
in the GEM mosaic.

Commands like `oq show performance`, `oq workers status` and `oq
sensitivity_analysis` now work correctly in SLURM mode: the first will
show the total memory used on all nodes, and not only the master
node. The second will show how many workers are active for each
node. The third will generate a script that will launch a SLURM job
for each combination of parameters in the sensitivity analysis.

# Other `oq` commands

After a major effort we were able to extend the command
`oq plot sources?` to work with all kinds of sources.

We fixed `oq prepare_site_model` to compute `z1pt0` and `z2pt5` correctly
for sites in Japan, i.e. by using the regionalized NGAWest2 basin effect
equations.

We fixed `oq reduce_sm` for calculations with nonparametric sources, which
was raising an error.

We fixed `oq plot_avg_gmf` to work remotely.

We extended `oq info` to .csv files, for pretty-printing purposes.

We improved the command `oq plot_assets`.

We added the commands `oq plot "rupture?"` and `oq plot "rupture_3d?"`.

We added add command `oq compare oqparam` and we improved `oq compare sitecol`
and `oq compare assetcol`.

We added a debugging command `oq compare rates` and a `oq compare asce` functionality.

We changed `oq run` to automatically generate the db in single user mode.

# Other

There were a few cosmetic improvements to the WebUI and we added
a cookie consent functionality for compliance with the European law.

For usage in the GEESE model, we added a method `OqParam.to_ini` which can be
used to generate an `.ini` file equivalent to the original input file. This is
useful when the calculation parameters are dynamically generated. Currently it
works only for hazard calculations.

As usual we improved the documentation, in particular about how to run calculations
programmatically using the pair [create_jobs/run_jobs]
(https://docs.openquake.org/oq-engine/3.21/manual/contributing/developing-with-the-engine.html#running-calculations-programmatically)