openquake.server.db package

openquake.server.db.actions module

class openquake.server.db.actions.List(iterable=(), /)

Bases: list

openquake.server.db.actions.add_checksum(db, job_id, value)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – job ID

• value – value of the checksum (32 bit integer)

Returns:

The unique job_id with that checksum or None

openquake.server.db.actions.calc_info(db, calc_id)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• calc_id – calculation ID

Returns:

dictionary of info about the given calculation

openquake.server.db.actions.check_outdated(db)

Check if the db is outdated, called before starting anything

Parameters:

db – a openquake.commonlib.dbapi.Db instance

openquake.server.db.actions.create_job(db, datadir, calculation_mode='to be set', description='just created', user_name=None, hc_id=None, host=None)

Create job for the given user, return it.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• datadir – data directory of the user who owns/started this job.

• calculation_mode – job kind

• description – description of the job

• user_name – name of the user running the job

• hc_id – ID of the parent job (if any)

• host – machine where the calculation is running (master)

Returns:

the job ID

openquake.server.db.actions.create_outputs(db, job_id, keysize, ds_size)

Build a correspondence between the outputs in the datastore and the ones in the database. Also, update the datastore size in the job table.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – ID of the current job

• keysize – a list of pairs (key, size_mb)

• ds_size – total datastore size in MB

openquake.server.db.actions.db_version(db)

Parameters:

db – a openquake.commonlib.dbapi.Db instance

openquake.server.db.actions.del_calc(db, job_id, user, delete_file=True, force=False)

Delete a calculation and all associated outputs, if possible.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – job ID, can be an integer or a string

• user – username

• delete_file – also delete the HDF5 file(s)

• force – delete even if there are dependent calculations

Returns:

a dict with key “success” and value indicating the job id of the calculation or of its ancestor, or key “error” and value describing what went wrong, and a list of paths of files to be removed

openquake.server.db.actions.delete_uncompleted_calculations(db, user)

Delete the uncompleted calculations of the given user.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• user – user name

openquake.server.db.actions.engine_version(db)

Returns:

git version as seen by the db

openquake.server.db.actions.fetch(db, templ, *args)

Run generic queries directly on the database. See the documentation of the dbapi module.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• templ – a SQL query template

• args – arguments to pass to the template

openquake.server.db.actions.find(db, description)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• description – job description, used in a case-insensitive LIKE clause

openquake.server.db.actions.finish(db, job_id, status)

Set the job columns is_running, status, and stop_time.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – ID of the current job

• status – a string such as ‘successful’ or ‘failed’

openquake.server.db.actions.get_calc_ids(db, user)

Returns:

calculation IDs of the given user

openquake.server.db.actions.get_calcs(db, request_get_dict, allowed_users, user_acl_on=False, id=None)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• request_get_dict – a dictionary

• allowed_users – a list of users

• user_acl_on – if True, returns only the calculations owned by the user or the group

• id – if given, extract only the specified calculation

Returns:

list of tuples (job_id, user_name, job_status, calculation_mode,

job_is_running, job_description, host)

openquake.server.db.actions.get_checksum_from_job(db, job_id)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – job ID

Returns:

the value of the checksum or 0

openquake.server.db.actions.get_dbpath(db)

Parameters:

db – a openquake.commonlib.dbapi.Db instance

Returns:

the path to the database file.

openquake.server.db.actions.get_executing_jobs(db)

Parameters:

db – a openquake.commonlib.dbapi.Db instance

Returns:

(id, user_name, start_time) tuples

openquake.server.db.actions.get_job(db, job_id, username=None)

If job_id is negative, return the last calculation of the current user, otherwise returns the job_id unchanged.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – a job ID (can be negative and can be nonexisting)

• username – an user name (if None, ignore it)

Returns:

a valid job or None if the original job ID was invalid

openquake.server.db.actions.get_job_from_checksum(db, checksum)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – job ID

Returns:

the job associated to the checksum or None

openquake.server.db.actions.get_log(db, job_id)

Extract the logs as a big string

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – a job ID

openquake.server.db.actions.get_log_size(db, job_id)

Get a slice of the calculation log as a JSON list of rows.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – a job ID

openquake.server.db.actions.get_log_slice(db, job_id, start, stop)

Get a slice of the calculation log as a JSON list of rows

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – a job ID

• start – start of the slice

• stop – end of the slice (the last element is excluded)

openquake.server.db.actions.get_longest_jobs(db)

Parameters:

db – a openquake.commonlib.dbapi.Db instance

Returns:

(id, user_name, days) tuples

openquake.server.db.actions.get_output(db, output_id)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• output_id – ID of an Output object

Returns:

(ds_key, calc_id, dirname)

openquake.server.db.actions.get_outputs(db, job_id)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – ID of a calculation.

Returns:

A sequence of openquake.server.db.models.Output objects

openquake.server.db.actions.get_path(db)

Parameters:

db – a openquake.commonlib.dbapi.Db instance

Returns:

the full path to the dbserver codebase

openquake.server.db.actions.get_result(db, result_id)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• result_id – a result ID

Returns:

(job_id, job_status, datadir, datastore_key)

openquake.server.db.actions.get_results(db, job_id)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – a job ID

Returns:

(datadir, datastore_keys)

openquake.server.db.actions.get_traceback(db, job_id)

Return the traceback of the given calculation as a list of lines. The list is empty if the calculation was successful.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – a job ID

openquake.server.db.actions.get_weight(db, job_id)

Return information about the total weight of the source model.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – a job ID

openquake.server.db.actions.import_job(db, calc_id, calc_mode, description, user_name, status, hc_id, datadir)

Insert a calculation inside the database, if calc_id is not taken

openquake.server.db.actions.list_calculations(db, job_type, user_name)

Yield a summary of past calculations.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_type – ‘hazard’ or ‘risk’

• user_name – an user name

openquake.server.db.actions.list_outputs(db, job_id, full=True)

List the outputs for a given OqJob.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – ID of a calculation.

• full (bool) – If True produce a full listing, otherwise a short version

openquake.server.db.actions.log(db, job_id, timestamp, level, process, message)

Write a log record in the database.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – a job ID

• timestamp – timestamp to store in the log record

• level – logging level to store in the log record

• process – process ID to store in the log record

• message – message to store in the log record

openquake.server.db.actions.reset_is_running(db)

Reset the flag job.is_running to False. This is called when the DbServer is restarted: the idea is that it is restarted only when all computations are completed.

Parameters:

db – a openquake.commonlib.dbapi.Db instance

openquake.server.db.actions.set_status(db, job_id, status)

Set the status ‘created’, ‘executing’, ‘complete’, ‘failed’, ‘aborted’ consistently with is_running.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – ID of the current job

• status – status string

openquake.server.db.actions.share_job(db, job_id, share)

Make the job visible to all users by setting its status to ‘shared’.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – a job ID

• share – if False, revert the status to ‘complete’

openquake.server.db.actions.update_job(db, job_id, dic)

Update the given calculation record.

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – a job ID

• dic – a dictionary of valid field/values for the job table

openquake.server.db.actions.update_job_checksum(db, job_id, checksum)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• job_id – job ID

• checksum – the checksum (32 bit integer)

openquake.server.db.actions.update_parent_child(db, parent_child)

Set hazard_calculation_id (parent) on a job_id (child)

openquake.server.db.actions.upgrade_db(db)

Parameters:

db – a openquake.commonlib.dbapi.Db instance

openquake.server.db.actions.what_if_I_upgrade(db, extract_scripts)

Parameters:

• db – a openquake.commonlib.dbapi.Db instance

• extract_scripts – scripts to extract

openquake.server.db.upgrade_manager module

exception openquake.server.db.upgrade_manager.DuplicatedVersion

Bases: RuntimeError

class openquake.server.db.upgrade_manager.UpgradeManager(upgrade_dir, version_table='revision_info', version_pattern='\\d\\d\\d\\d', flag_pattern='(-slow|-danger)?')

Bases: object

The package containing the upgrade scripts should contain an instance of the UpgradeManager called upgrader in the __init__.py file. It should also specify the initializations parameters

Parameters:

• upgrade_dir – the directory were the upgrade script reside

• version_table – the name of the versioning table (default revision_info)

• version_pattern – a regulation expression for the script version number (dddd)

ENGINE_URL = 'https://github.com/gem/oq-engine/tree/master/'

UPGRADES = 'openquake/server/db/schema/upgrades/'

check_versions(conn)

Parameters:

conn – a DB API 2 connection

Returns:

a message with the versions that will be applied or None

extract_upgrade_scripts()

Extract the OpenQuake upgrade scripts from the links in the GitHub page

get_db_versions(conn)

Get all the versions stored in the database as a set.

Parameters:

conn – a DB API 2 connection

init(conn)

Create the version table and run the base script on an empty database.

Parameters:

conn – a DB API 2 connection

install_versioning(conn)

Create the version table into an already populated database and insert the base script.

Parameters:

conn – a DB API 2 connection

classmethod instance(conn, pkg_name='openquake.server.db.schema.upgrades')

Return an UpgradeManager instance.

Parameters:

• conn – a DB API 2 connection

• pkg_name (str) – the name of the package with the upgrade scripts

parse_script_name(script_name)

Parse a script name and return a dictionary with fields fname, name, version and ext (or None if the name does not match).

Parameters:

name – name of the script

read_scripts(minversion=None, maxversion=None, skip_versions=())

Extract the upgrade scripts from a directory as a list of dictionaries, ordered by version.

Parameters:

• minversion – the minimum version to consider

• maxversion – the maximum version to consider

• skipversions – the versions to skip

upgrade(conn, skip_versions=())

Upgrade the database from the current version to the maximum version in the upgrade scripts.

Parameters:

• conn – a DBAPI 2 connection

• skip_versions – the versions to skip

exception openquake.server.db.upgrade_manager.VersionTooSmall

Bases: RuntimeError

exception openquake.server.db.upgrade_manager.VersioningNotInstalled

Bases: RuntimeError

class openquake.server.db.upgrade_manager.WrappedConnection(conn, debug=False)

Bases: object

This is an utility class that wraps a DB API-2 connection providing a couple of convenient features.

1. it is possible to set a debug flag to print on stdout the executed queries;

2. there is a .run method to run a query with a dedicated cursor; it returns the cursor, which can be iterated over

Parameters:

conn – a DB API2-compatible connection

run(templ, *args)

A simple utility to run SQL queries.

Parameters:

• templ – a query or query template

• args – the arguments (or the empty tuple)

Returns:

the DB API 2 cursor used to run the query

openquake.server.db.upgrade_manager.apply_sql_script(conn, fname)

Apply the given SQL script to the database

Parameters:

• conn – a DB API 2 connection

• fname – full path to the creation script

openquake.server.db.upgrade_manager.check_script(upgrade, conn, dry_run=True, debug=True)

An utility to debug upgrade scripts written in Python

Parameters:

• upgrade – upgrade procedure

• conn – a DB API 2 connection

• dry_run – if True, do not change the database

• debug – if True, print the queries which are executed

openquake.server.db.upgrade_manager.db_version(conn, pkg_name='openquake.server.db.schema.upgrades')

Parameters:

• conn – a DB API 2 connection

• pkg_name (str) – the name of the package with the upgrade scripts

Returns:

the current version of the database

openquake.server.db.upgrade_manager.upgrade_db(conn=None, pkg_name='openquake.server.db.schema.upgrades', skip_versions=())

Upgrade a database by running several scripts in a single transaction.

Parameters:

• conn – a DB API 2 connection (if None use dbapi.db.conn)

• pkg_name (str) – the name of the package with the upgrade scripts

• skip_versions (list) – the versions to skip

Returns:

the version numbers of the new scripts applied the database

openquake.server.db.upgrade_manager.what_if_I_upgrade(conn, pkg_name='openquake.server.db.schema.upgrades', extract_scripts='extract_upgrade_scripts')

Parameters:

• conn – a DB API 2 connection

• pkg_name (str) – the name of the package with the upgrade scripts

• extract_scripts – name of the method to extract the scripts

Module contents