baselib Package

general Module

Utility functions of general interest.

class openquake.baselib.general.AccumDict

Bases: dict

An accumulating dictionary, useful to accumulate variables.

>> acc = AccumDict() >> acc += {‘a’: 1} >> acc += {‘a’: 1, ‘b’: 1} >> acc {‘a’: 2, ‘b’: 1} >> {‘a’: 1} + acc {‘a’: 3, ‘b’: 1} >> acc + 1 {‘a’: 3, ‘b’: 2} >> 1 - acc {‘a’: -1, ‘b’: 0} >> acc - 1 {‘a’: 1, ‘b’: 0}

Also the multiplication has been defined:

>> prob1 = AccumDict(a=0.4, b=0.5) >> prob2 = AccumDict(b=0.5) >> prob1 * prob2 {‘a’: 0.4, ‘b’: 0.25} >> prob1 * 1.2 {‘a’: 0.48, ‘b’: 0.6} >> 1.2 * prob1 {‘a’: 0.48, ‘b’: 0.6}

apply(func, *extras)

>> a = AccumDict({‘a’: 1, ‘b’: 2}) >> a.apply(lambda x, y: 2 * x + y, 1) {‘a’: 3, ‘b’: 5}

class openquake.baselib.general.CallableDict(keyfunc=<function <lambda>>, keymissing=None)

Bases: collections.OrderedDict

A callable object built on top of a dictionary of functions, used as a smart registry or as a poor man generic function dispatching on the first argument. It is typically used to implement converters. Here is an example:

>>> format_attrs = CallableDict()  # dict of functions (fmt, obj) -> str

>>> @format_attrs.add('csv')  # implementation for csv
... def format_attrs_csv(fmt, obj):
...     items = sorted(vars(obj).items())
...     return '\n'.join('%s,%s' % item for item in items)

>>> @format_attrs.add('json')  # implementation for json
... def format_attrs_json(fmt, obj):
...     return json.dumps(vars(obj))

format_attrs(fmt, obj) calls the correct underlying function depending on the fmt key. If the format is unknown a KeyError is raised. It is also possible to set a keymissing function to specify what to return if the key is missing.

For a more practical example see the implementation of the exporters in openquake.commonlib.export

add(*keys)

Return a decorator registering a new implementation for the CallableDict for the given keys.

exception openquake.baselib.general.CodeDependencyError

Bases: exceptions.Exception

exception openquake.baselib.general.DeprecationWarning

Bases: exceptions.UserWarning

Raised the first time a deprecated function is called

class openquake.baselib.general.DictArray(imtls)

Bases: _abcoll.Mapping

A small wrapper over a dictionary of arrays serializable to HDF5:

>>> d = DictArray({'PGA': [0.01, 0.02, 0.04], 'PGV': [0.1, 0.2]})
>>> from openquake.baselib import hdf5
>>> with hdf5.File('/tmp/x.h5', 'w') as f:
...      f['d'] = d
...      f['d']
<DictArray
PGA: [ 0.01  0.02  0.04]
PGV: [ 0.1  0.2]>

The DictArray maintains the lexicographic order of the keys.

new(array)

Convert an array of compatible length into a DictArray:

>>> d = DictArray({'PGA': [0.01, 0.02, 0.04], 'PGV': [0.1, 0.2]})
>>> d.new(numpy.arange(0, 5, 1))  # array of lenght 5 = 3 + 2
<DictArray
PGA: [0 1 2]
PGV: [3 4]>

class openquake.baselib.general.WeightedSequence(seq=())

Bases: _abcoll.MutableSequence

A wrapper over a sequence of weighted items with a total weight attribute. Adding items automatically increases the weight.

insert(i, item_weight)

Insert an item with the given weight in the sequence

classmethod merge(ws_list)

Merge a set of WeightedSequence objects.

Parameters:ws_list – a sequence of :class: openquake.baselib.general.WeightedSequence instances Returns:a openquake.baselib.general.WeightedSequence instance

openquake.baselib.general.assert_close(a, b, rtol=1e-07, atol=0, context=None)

Compare for equality up to a given precision two composite objects which may contain floats. NB: if the objects are or contain generators, they are exhausted.

Parameters:

• a – an object
• b – another object
• rtol – relative tolerance
• atol – absolute tolerance

openquake.baselib.general.assert_independent(package, *packages)

Parameters:

• package – Python name of a module/package
• packages – Python names of modules/packages

Make sure the package does not depend from the packages.

openquake.baselib.general.block_splitter(items, max_weight, weight=<function <lambda>>, kind=<function <lambda>>)

Parameters:

• items – an iterator over items
• max_weight – the max weight to split on
• weight – a function returning the weigth of a given item
• kind – a function returning the kind of a given item

Group together items of the same kind until the total weight exceeds the max_weight and yield WeightedSequence instances. Items with weight zero are ignored.

For instance

>>> items = 'ABCDE'
>>> list(block_splitter(items, 3))
[<WeightedSequence ['A', 'B', 'C'], weight=3>, <WeightedSequence ['D', 'E'], weight=2>]

The default weight is 1 for all items.

openquake.baselib.general.ceil(a, b)

Divide a / b and return the biggest integer close to the quotient.

Parameters:

• a – a number
• b – a positive number

Returns:

the biggest integer close to the quotient

openquake.baselib.general.deprecated(message)

Return a decorator to make deprecated functions.

Parameters:message – the message to print the first time the deprecated function is used.

Here is an example of usage:

>>> @deprecated('Use new_function instead')
... def old_function():
...     'Do something'

Notice that if the function is called several time, the deprecation warning will be displayed only the first time.

openquake.baselib.general.distinct(keys)

Return the distinct keys in order.

openquake.baselib.general.get_array(array, **kw)

Extract a subarray by filtering on the given keyword arguments

openquake.baselib.general.git_suffix(fname)

Returns:<short git hash> if Git repository found

openquake.baselib.general.group_array(array, *kfields)

Convert an array into an OrderedDict kfields -> array

openquake.baselib.general.groupby(objects, key, reducegroup=<type 'list'>)

Parameters:

• objects – a sequence of objects with a key value
• key – the key function to extract the key value
• reducegroup – the function to apply to each group

Returns:

an OrderedDict {key value: map(reducegroup, group)}

>>> groupby(['A1', 'A2', 'B1', 'B2', 'B3'], lambda x: x[0],
...         lambda group: ''.join(x[1] for x in group))
OrderedDict([('A', '12'), ('B', '123')])

openquake.baselib.general.groupby2(records, kfield, vfield)

Parameters:

• records – a sequence of records with positional or named fields
• kfield – the index/name/tuple specifying the field to use as a key
• vfield – the index/name/tuple specifying the field to use as a value

Returns:

an list of pairs of the form (key, [value, ...]).

>>> groupby2(['A1', 'A2', 'B1', 'B2', 'B3'], 0, 1)
[('A', ['1', '2']), ('B', ['1', '2', '3'])]

Here is an example where the keyfield is a tuple of integers:

>>> groupby2(['A11', 'A12', 'B11', 'B21'], (0, 1), 2)
[(('A', '1'), ['1', '2']), (('B', '1'), ['1']), (('B', '2'), ['1'])]

openquake.baselib.general.humansize(nbytes, suffixes=('B', 'KB', 'MB', 'GB', 'TB', 'PB'))

Return file size in a human-friendly format

openquake.baselib.general.import_all(module_or_package)

If module_or_package is a module, just import it; if it is a package, recursively imports all the modules it contains. Returns the names of the modules that were imported as a set. The set can be empty if the modules were already in sys.modules.

openquake.baselib.general.run_in_process(code, *args)

Run in an external process the given Python code and return the output as a Python object. If there are arguments, then code is taken as a template and traditional string interpolation is performed.

Parameters:

• code – string or template describing Python code
• args – arguments to be used for interpolation

Returns:

the output of the process, as a Python object

openquake.baselib.general.search_module(module, syspath=['/home/daniele/py27/bin', '/home/daniele/GIT/gem/openquake/oq-engine', '/home/daniele/GIT/gem/openquake/oq-hazardlib', '/home/daniele/py27/lib/python27.zip', '/home/daniele/py27/lib64/python2.7', '/home/daniele/py27/lib64/python2.7/plat-linux2', '/home/daniele/py27/lib64/python2.7/lib-tk', '/home/daniele/py27/lib64/python2.7/lib-old', '/home/daniele/py27/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7', '/usr/lib/python2.7', '/usr/lib64/python2.7/lib-tk', '/home/daniele/py27/lib/python2.7/site-packages', '/home/daniele/GIT/gem/oq-platform/oq-platform-ipt', '/home/daniele/GIT/gem/oq-platform/oq-platform-standalone', '/home/daniele/GIT/gem/rmtk'])

Given a module name (possibly with dots) returns the corresponding filepath, or None, if the module cannot be found.

Parameters:

• module – (dotted) name of the Python module to look for
• syspath – a list of directories to search (default sys.path)

openquake.baselib.general.split_in_blocks(sequence, hint, weight=<function <lambda>>, key=<function <lambda>>)

Split the sequence in a number of WeightedSequences close to hint.

Parameters:

• sequence – a finite sequence of items
• hint – an integer suggesting the number of subsequences to generate
• weight – a function returning the weigth of a given item
• key – a function returning the key of a given item

The WeightedSequences are of homogeneous key and they try to be balanced in weight. For instance

>>> items = 'ABCDE'
>>> list(split_in_blocks(items, 3))
[<WeightedSequence ['A', 'B'], weight=2>, <WeightedSequence ['C', 'D'], weight=2>, <WeightedSequence ['E'], weight=1>]

openquake.baselib.general.writetmp(content=None, dir=None, prefix='tmp', suffix='tmp')

Create temporary file with the given content.

Please note: the temporary file must be deleted by the caller.

Parameters:

• content (string) – the content to write to the temporary file.
• dir (string) – directory where the file should be created
• prefix (string) – file name prefix
• suffix (string) – file name suffix

Returns:

a string with the path to the temporary file