OpenQuake Utilities

The Utils Sub-package

Utility functions that are of interest to and shared by the entire OpenQuake python code base.

version

Utility functions related to OpenQuake version information.

openquake.engine.utils.version.info(version_data)

Return a string with the OpenQuake version infomation.

Version info data set to -1 will be ignored and assumed to have value zero. Release dates that lie more than 30 days in the future are ignored.

Parameters:version_data – A 4-tuple of integers that are the major, minor and sprint number respectively. The last datum is the number of seconds since epoch and represents the release date. Returns:A string with human readable OpenQuake version information.

config

Various utility functions concerned with configuration.

openquake.engine.utils.config.OQ_CONFIG_FILE_VAR = 'OQ_CONFIG_FILE'

Environment variable name for specifying a custom openquake.cfg. The file name doesn’t matter.

openquake.engine.utils.config.abort_if_no_config_available()

Call sys.exit() if no openquake configuration file is readable.

openquake.engine.utils.config.context(*args, **kwds)

Context manager used to change the parameters of a configuration section on the fly. For use in the tests.

openquake.engine.utils.config.flag_set(section, setting)

True if the given boolean setting is enabled in openquake.cfg

Parameters:

• section (string) – name of the configuration file section
• setting (string) – name of the configuration file setting

Returns:

True if the setting is enabled in openquake.cfg, False otherwise

openquake.engine.utils.config.get(section, key)

The configuration value for the given section and key or None.

openquake.engine.utils.config.get_section(section)

A dictionary of key/value pairs for the given section or None.

openquake.engine.utils.config.refresh()

Re-parse config files and refresh the cached configuration.

NOTE: Use with caution. Calling this during some phases of a calculation could cause undesirable side-effects.

tasks

Various utility functions related to splitting work into tasks and/or managing these.

Utility functions related to splitting work into tasks.

exception openquake.engine.utils.tasks.JobNotRunning

Bases: exceptions.Exception

class openquake.engine.utils.tasks.OqTaskManager(oqtask, progress=<function info>, name=None)

Bases: openquake.commonlib.parallel.TaskManager

A celery-based task manager. The usage is:

oqm = OqTaskManager(do_something, logs.LOG.progress)
oqm.send(arg1, arg2)
oqm.send(arg3, arg4)
print oqm.aggregate_results(agg, acc)

Progress report is built-in.

aggregate_result_set(agg, acc)

Loop on a set of celery AsyncResults and update the accumulator by using the aggregation function.

Parameters:

• agg – the aggregation function, (acc, val) -> new acc
• acc – the initial value of the accumulator

Returns:

the final value of the accumulator

submit(*args)

Submit an oqtask with the given arguments to celery and return an AsyncResult. If the variable OQ_NO_DISTRIBUTE is set, the task function is run in process and the result is returned.

openquake.engine.utils.tasks.apply_reduce(task, task_args, agg=<built-in function add>, acc=None, concurrent_tasks=64, weight=<function <lambda>>, key=<function <lambda>>, name=None)

Apply a task to a tuple of the form (job_id, data, *args) by splitting the data in chunks and reduce the results with an aggregation function.

Parameters:

• task – an oqtask
• task_args – the arguments to be passed to the task function
• agg – the aggregation function
• acc – initial value of the accumulator
• concurrent_tasks – hint about how many tasks to generate
• weight – function to extract the weight of an item in data
• key – function to extract the kind of an item in data

openquake.engine.utils.tasks.oqtask(task_func)

Task function decorator which sets up logging and catches (and logs) any errors which occur inside the task. Also checks to make sure the job is actually still running. If it is not running, the task doesn’t get executed, so we don’t do useless computation.

Parameters:task_func – the function to decorate