openquake.server.db package

openquake.server.db.actions module

class openquake.server.db.actions.List(iterable=(), /)[source]

Bases: list

openquake.server.db.actions.add_checksum(db, job_id, value)[source]
Parameters
openquake.server.db.actions.calc_info(db, calc_id)[source]
Parameters
Returns

dictionary of info about the given calculation

openquake.server.db.actions.check_outdated(db)[source]

Check if the db is outdated, called before starting anything

Parameters

db – a openquake.commonlib.dbapi.Db instance

openquake.server.db.actions.create_job(db, datadir, calculation_mode='to be set', description='just created', user_name=None, hc_id=None, host=None)[source]

Create job for the given user, return it.

Parameters

  • db – a openquake.commonlib.dbapi.Db instance

  • datadir – data directory of the user who owns/started this job.

  • calculation_mode – job kind

  • description – description of the job

  • user_name – name of the user running the job

  • hc_id – ID of the parent job (if any)

  • host – machine where the calculation is running (master)

Returns

the job ID

openquake.server.db.actions.create_outputs(db, job_id, keysize, ds_size)[source]

Build a correspondence between the outputs in the datastore and the ones in the database. Also, update the datastore size in the job table.

Parameters

  • db – a openquake.commonlib.dbapi.Db instance

  • job_id – ID of the current job

  • keysize – a list of pairs (key, size_mb)

  • ds_size – total datastore size in MB

openquake.server.db.actions.db_version(db)[source]
Parameters

db – a openquake.commonlib.dbapi.Db instance

openquake.server.db.actions.del_calc(db, job_id, user, force=False)[source]

Delete a calculation and all associated outputs, if possible.

Parameters

  • db – a openquake.commonlib.dbapi.Db instance

  • job_id – job ID, can be an integer or a string

  • user – username

  • force – delete even if there are dependent calculations

Returns

None if everything went fine or an error message

openquake.server.db.actions.delete_uncompleted_calculations(db, user)[source]

Delete the uncompleted calculations of the given user.

Parameters
openquake.server.db.actions.engine_version(db)[source]
Returns

git version as seen by the db

openquake.server.db.actions.fetch(db, templ, *args)[source]

Run generic queries directly on the database. See the documentation of the dbapi module.

Parameters
openquake.server.db.actions.find(db, description)[source]
Parameters
openquake.server.db.actions.finish(db, job_id, status)[source]

Set the job columns is_running, status, and stop_time.

Parameters

  • db – a openquake.commonlib.dbapi.Db instance

  • job_id – ID of the current job

  • status – a string such as ‘successful’ or ‘failed’

openquake.server.db.actions.get_calc_id(db, datadir, job_id=None)[source]

Return the latest calc_id by looking both at the datastore and the database.

Parameters

  • db – a openquake.commonlib.dbapi.Db instance

  • datadir – the directory containing the datastores

  • job_id – a job ID; if None, returns the latest job ID

openquake.server.db.actions.get_calc_ids(db, user)[source]
Returns

calculation IDs of the given user

openquake.server.db.actions.get_calcs(db, request_get_dict, allowed_users, user_acl_on=False, id=None)[source]
Parameters

  • db – a openquake.commonlib.dbapi.Db instance

  • request_get_dict – a dictionary

  • allowed_users – a list of users

  • user_acl_on – if True, returns only the calculations owned by the user or the group

  • id – if given, extract only the specified calculation

Returns

list of tuples (job_id, user_name, job_status, calculation_mode,

job_is_running, job_description, host)

openquake.server.db.actions.get_checksum_from_job(db, job_id)[source]
Parameters
Returns

the value of the checksum or 0

openquake.server.db.actions.get_dbpath(db)[source]
Parameters

db – a openquake.commonlib.dbapi.Db instance

Returns

the path to the database file.

openquake.server.db.actions.get_executing_jobs(db)[source]
Parameters

db – a openquake.commonlib.dbapi.Db instance

Returns

(id, user_name, start_time) tuples

openquake.server.db.actions.get_job(db, job_id, username=None)[source]

If job_id is negative, return the last calculation of the current user, otherwise returns the job_id unchanged.

Parameters

  • db – a openquake.commonlib.dbapi.Db instance

  • job_id – a job ID (can be negative and can be nonexisting)

  • username – an user name (if None, ignore it)

Returns

a valid job or None if the original job ID was invalid

openquake.server.db.actions.get_job_from_checksum(db, checksum)[source]
Parameters
Returns

the job associated to the checksum or None

openquake.server.db.actions.get_log(db, job_id)[source]

Extract the logs as a big string

Parameters
openquake.server.db.actions.get_log_size(db, job_id)[source]

Get a slice of the calculation log as a JSON list of rows.

Parameters
openquake.server.db.actions.get_log_slice(db, job_id, start, stop)[source]

Get a slice of the calculation log as a JSON list of rows

Parameters

  • db – a openquake.commonlib.dbapi.Db instance

  • job_id – a job ID

  • start – start of the slice

  • stop – end of the slice (the last element is excluded)

openquake.server.db.actions.get_longest_jobs(db)[source]
Parameters

db – a openquake.commonlib.dbapi.Db instance

Returns

(id, user_name, days) tuples

openquake.server.db.actions.get_output(db, output_id)[source]
Parameters
Returns

(ds_key, calc_id, dirname)

openquake.server.db.actions.get_outputs(db, job_id)[source]
Parameters
Returns

A sequence of openquake.server.db.models.Output objects

openquake.server.db.actions.get_path(db)[source]
Parameters

db – a openquake.commonlib.dbapi.Db instance

Returns

the full path to the dbserver codebase

openquake.server.db.actions.get_result(db, result_id)[source]
Parameters
Returns

(job_id, job_status, datadir, datastore_key)

openquake.server.db.actions.get_results(db, job_id)[source]
Parameters
Returns

(datadir, datastore_keys)

openquake.server.db.actions.get_traceback(db, job_id)[source]

Return the traceback of the given calculation as a list of lines. The list is empty if the calculation was successful.

Parameters
openquake.server.db.actions.get_weight(db, job_id)[source]

Return information about the total weight of the source model.

Parameters
openquake.server.db.actions.import_job(db, calc_id, calc_mode, description, user_name, status, hc_id, datadir)[source]

Insert a calculation inside the database, if calc_id is not taken

openquake.server.db.actions.list_calculations(db, job_type, user_name)[source]

Yield a summary of past calculations.

Parameters
openquake.server.db.actions.list_outputs(db, job_id, full=True)[source]

List the outputs for a given OqJob.

Parameters

  • db – a openquake.commonlib.dbapi.Db instance

  • job_id – ID of a calculation.

  • full (bool) – If True produce a full listing, otherwise a short version

openquake.server.db.actions.log(db, job_id, timestamp, level, process, message)[source]

Write a log record in the database.

Parameters

  • db – a openquake.commonlib.dbapi.Db instance

  • job_id – a job ID

  • timestamp – timestamp to store in the log record

  • level – logging level to store in the log record

  • process – process ID to store in the log record

  • message – message to store in the log record

openquake.server.db.actions.reset_is_running(db)[source]

Reset the flag job.is_running to False. This is called when the DbServer is restarted: the idea is that it is restarted only when all computations are completed.

Parameters

db – a openquake.commonlib.dbapi.Db instance

openquake.server.db.actions.set_status(db, job_id, status)[source]

Set the status ‘created’, ‘executing’, ‘complete’, ‘failed’, ‘aborted’ consistently with is_running.

Parameters
openquake.server.db.actions.update_job(db, job_id, dic)[source]

Update the given calculation record.

Parameters
openquake.server.db.actions.update_job_checksum(db, job_id, checksum)[source]
Parameters
openquake.server.db.actions.update_parent_child(db, parent_child)[source]

Set hazard_calculation_id (parent) on a job_id (child)

openquake.server.db.actions.upgrade_db(db)[source]
Parameters

db – a openquake.commonlib.dbapi.Db instance

openquake.server.db.actions.what_if_I_upgrade(db, extract_scripts)[source]
Parameters

openquake.server.db.upgrade_manager module

exception openquake.server.db.upgrade_manager.DuplicatedVersion[source]

Bases: RuntimeError

class openquake.server.db.upgrade_manager.UpgradeManager(upgrade_dir, version_table='revision_info', version_pattern='\\d\\d\\d\\d', flag_pattern='(-slow|-danger)?')[source]

Bases: object

The package containing the upgrade scripts should contain an instance of the UpgradeManager called upgrader in the __init__.py file. It should also specify the initializations parameters

Parameters

  • upgrade_dir – the directory were the upgrade script reside

  • version_table – the name of the versioning table (default revision_info)

  • version_pattern – a regulation expression for the script version number (dddd)

ENGINE_URL = 'https://github.com/gem/oq-engine/tree/master/'
UPGRADES = 'openquake/server/db/schema/upgrades/'
check_versions(conn)[source]
Parameters

conn – a DB API 2 connection

Returns

a message with the versions that will be applied or None

extract_upgrade_scripts()[source]

Extract the OpenQuake upgrade scripts from the links in the GitHub page

get_db_versions(conn)[source]

Get all the versions stored in the database as a set.

Parameters

conn – a DB API 2 connection

init(conn)[source]

Create the version table and run the base script on an empty database.

Parameters

conn – a DB API 2 connection

install_versioning(conn)[source]

Create the version table into an already populated database and insert the base script.

Parameters

conn – a DB API 2 connection

classmethod instance(conn, pkg_name='openquake.server.db.schema.upgrades')[source]

Return an UpgradeManager instance.

Parameters

  • conn – a DB API 2 connection

  • pkg_name (str) – the name of the package with the upgrade scripts

parse_script_name(script_name)[source]

Parse a script name and return a dictionary with fields fname, name, version and ext (or None if the name does not match).

Parameters

name – name of the script

read_scripts(minversion=None, maxversion=None, skip_versions=())[source]

Extract the upgrade scripts from a directory as a list of dictionaries, ordered by version.

Parameters

  • minversion – the minimum version to consider

  • maxversion – the maximum version to consider

  • skipversions – the versions to skip

upgrade(conn, skip_versions=())[source]

Upgrade the database from the current version to the maximum version in the upgrade scripts.

Parameters

  • conn – a DBAPI 2 connection

  • skip_versions – the versions to skip

exception openquake.server.db.upgrade_manager.VersionTooSmall[source]

Bases: RuntimeError

exception openquake.server.db.upgrade_manager.VersioningNotInstalled[source]

Bases: RuntimeError

class openquake.server.db.upgrade_manager.WrappedConnection(conn, debug=False)[source]

Bases: object

This is an utility class that wraps a DB API-2 connection providing a couple of convenient features.

  1. it is possible to set a debug flag to print on stdout the executed queries;

  2. there is a .run method to run a query with a dedicated cursor; it returns the cursor, which can be iterated over

Parameters

conn – a DB API2-compatible connection

run(templ, *args)[source]

A simple utility to run SQL queries.

Parameters

  • templ – a query or query template

  • args – the arguments (or the empty tuple)

Returns

the DB API 2 cursor used to run the query

openquake.server.db.upgrade_manager.apply_sql_script(conn, fname)[source]

Apply the given SQL script to the database

Parameters

  • conn – a DB API 2 connection

  • fname – full path to the creation script

openquake.server.db.upgrade_manager.check_script(upgrade, conn, dry_run=True, debug=True)[source]

An utility to debug upgrade scripts written in Python

Parameters

  • upgrade – upgrade procedure

  • conn – a DB API 2 connection

  • dry_run – if True, do not change the database

  • debug – if True, print the queries which are executed

openquake.server.db.upgrade_manager.db_version(conn, pkg_name='openquake.server.db.schema.upgrades')[source]
Parameters

  • conn – a DB API 2 connection

  • pkg_name (str) – the name of the package with the upgrade scripts

Returns

the current version of the database

openquake.server.db.upgrade_manager.upgrade_db(conn, pkg_name='openquake.server.db.schema.upgrades', skip_versions=())[source]

Upgrade a database by running several scripts in a single transaction.

Parameters

  • conn – a DB API 2 connection

  • pkg_name (str) – the name of the package with the upgrade scripts

  • skip_versions (list) – the versions to skip

Returns

the version numbers of the new scripts applied the database

openquake.server.db.upgrade_manager.what_if_I_upgrade(conn, pkg_name='openquake.server.db.schema.upgrades', extract_scripts='extract_upgrade_scripts')[source]
Parameters

  • conn – a DB API 2 connection

  • pkg_name (str) – the name of the package with the upgrade scripts

  • extract_scripts – name of the method to extract the scripts

Module contents