openquake.server package

Subpackages

• openquake.server.db package
  • openquake.server.db.actions module
  • openquake.server.db.upgrade_manager module
  • Module contents
• openquake.server.v1 package
  • openquake.server.v1.calc_urls module
  • Module contents

Submodules

openquake.server.dbapi module

One of the worst thing about Python is the DB API 2.0 specification, which is unusable except for building frameworks. It should have been a stepping stone for an usable DB API 3.0 that never happened. So, instead of a good low level API, we had a proliferation of Object Relational Mappers making our lives a lot harder. Fortunately, there has always been good Pythonistas in the anti-ORM camp.

This module is heavily inspired by the dbapiext module by Martin Blais, which is part of the antiorm package. The main (only) difference is that I am using the question mark (?) for the placeholders instead of the percent sign (%) to avoid confusions with other usages of the %s, in particular in LIKE queries and in expressions like strftime(‘%s’, time) used in SQLite.

In less than 200 lines of code there is enough support to build dynamic SQL queries and to make an ORM unneeded, since we do not need database independence.

dbiapi tutorial

The only thing you must to know is the Db class, which is lazy wrapper over a database connection. You instantiate it by passing a connection function and its arguments:

>>> import sqlite3
>>> db = Db(sqlite3.connect, ':memory:')

Now you have an interface to your database, the db object. This object is lazy, i.e. the connection is not yet instantiated, but it will be when you will access its .conn attribute. This attribute is automatically accessed when you call the interface to run a query, for instance to create an empty table:

>>> curs = db('CREATE TABLE job ('
...     'id INTEGER PRIMARY KEY AUTOINCREMENT, value INTEGER)')

You can populate the table by using the .insert method:

>>> db.insert('job', ['value'], [(42,), (43,)]) 
<sqlite3.Cursor object at ...>

Notice that this method returns a standard DB API 2.0 cursor and you have access to all of its features: for instance here you could extract the lastrowid.

Then you can run SELECT queries:

>>> rows = db('SELECT * FROM job')

The dbapi provides a Row class which is used to hold the results of SELECT queries and is working as one would expect:

>>> rows
[<Row(id=1, value=42)>, <Row(id=2, value=43)>]
>>> tuple(rows[0])
(1, 42)
>>> rows[0].id
1
>>> rows[0].value
42
>>> rows[0]._fields
['id', 'value']

The queries can have different kind of ? parameters:

• ?s is for interpolated string parameters:

  >>> db('SELECT * FROM ?s', 'job')  # ?s is replaced by 'job'
  [<Row(id=1, value=42)>, <Row(id=2, value=43)>]
  

• ?x is for escaped parameters (to avoid SQL injection):

  >>> db('SELECT * FROM job WHERE id=?x', 1)  # ?x is replaced by 1
  [<Row(id=1, value=42)>]
  

• ?s and ?x are for scalar parameters; ?S and ?X are for sequences:

  >>> db('INSERT INTO job (?S) VALUES (?X)', ['id', 'value'], (3, 44)) 
  <sqlite3.Cursor object at ...>
  

You can see how the interpolation works by calling the expand method that returns the interpolated template (alternatively, there is a debug=True flag when calling db that prints the same info). In this case

>>> db.expand('INSERT INTO job (?S) VALUES (?X)', ['id', 'value'], [3, 44])
'INSERT INTO job (id, value) VALUES (?, ?)'

As you see, ?S parameters work by replacing a list of strings with a comma separated string, where ?X parameters are replaced by a comma separated sequence of question marks, i.e. the low level placeholder for SQLite. The interpolation performs a regular search and replace, so if you have a ?- string in your template that must not be escaped, you can run into issues. This is an error:

>>> match("SELECT * FROM job WHERE id=?x AND description='Lots of ?s'", 1)
Traceback (most recent call last):
   ...
ValueError: Incorrect number of ?-parameters in SELECT * FROM job WHERE id=?x AND description='Lots of ?s', expected 1

This is correct:

>>> match("SELECT * FROM job WHERE id=?x AND description=?x", 1, 'Lots of ?s')
('SELECT * FROM job WHERE id=? AND description=?', (1, 'Lots of ?s'))

There are three other ? parameters:

• ?D is for dictionaries and it is used mostly in UPDATE queries:

  >>> match('UPDATE mytable SET ?D WHERE id=?x', dict(value=33, other=5), 1)
  ('UPDATE mytable SET other=?, value=? WHERE id=?', (5, 33, 1))
  

• ?A is for dictionaries and it is used in AND queries:

  >>> match('SELECT * FROM job WHERE ?A', dict(value=33, id=5))
  ('SELECT * FROM job WHERE id=? AND value=?', (5, 33))
  

• ?O is for dictionaries and it is used in OR queries:

  >>> match('SELECT * FROM job WHERE ?O', dict(value=33, id=5))
  ('SELECT * FROM job WHERE id=? OR value=?', (5, 33))
  

The dictionary parameters are ordered per field name, just to make the templates reproducible. ?A and ?O are smart enough to treat specially None parameters, that are turned into NULL:

>>> match('SELECT * FROM job WHERE ?A', dict(value=None, id=5))
('SELECT * FROM job WHERE id=? AND value IS NULL', (5,))

The ? parameters are matched positionally; it is also possible to pass to the db object a few keyword arguments to tune the standard behavior. In particular, if you know that a query must return a single row you can do the following:

>>> db('SELECT * FROM job WHERE id=?x', 1, one=True)
<Row(id=1, value=42)>

Without the one=True the query would have returned a list with a single element. If you know that the query must return a scalar you can do the following:

>>> db('SELECT value FROM job WHERE id=?x', 1, scalar=True)
42

If a query that should return a scalar returns something else, or if a query that should return a row returns a different number of rows, appropriate errors are raised:

>>> db('SELECT * FROM job WHERE id=?x', 1, scalar=True) 
Traceback (most recent call last):
   ...
TooManyColumns: 2, expected 1

>>> db('SELECT * FROM job', None, one=True) 
Traceback (most recent call last):
   ...
TooManyRows: 3, expected 1

If a row is expected but not found, a NotFound exception is raised:

>>> db('SELECT * FROM job WHERE id=?x', None, one=True) 
Traceback (most recent call last):
   ...
NotFound

class openquake.server.dbapi.Db(connect, *args, **kw)

Bases: object

A wrapper over a DB API 2 connection. See the tutorial.

close()

Close the main thread connection and refresh the threadlocal object

conn

classmethod expand(m_templ, *m_args)

Performs partial interpolation of the template. Used for debugging.

insert(table, columns, rows)

Insert several rows with executemany. Return a cursor.

path

Path to the underlying sqlite file

exception openquake.server.dbapi.NotFound

Bases: Exception

Raised when a scalar query has not output

class openquake.server.dbapi.Row(fields, values)

Bases: collections.abc.Sequence

A pickleable row, working both as a tuple and an object:

>>> row = Row(['id', 'value'], (1, 2))
>>> tuple(row)
(1, 2)
>>> assert row[0] == row.id and row[1] == row.value

Parameters:

• fields – a sequence of field names
• values – a sequence of values (one per field)

class openquake.server.dbapi.Table(fields, rows)

Bases: list

Just a list of Rows with an attribute _fields

exception openquake.server.dbapi.TooManyColumns

Bases: Exception

Raised when a scalar query has more than one column

exception openquake.server.dbapi.TooManyRows

Bases: Exception

Raised when a scalar query produces more than one row

openquake.server.dbapi.match(m_templ, *m_args)

Parameters:

• m_templ – a meta template string
• m_args – all arguments

Returns:

template, args

Here is an example of usage:

>>> match('SELECT * FROM job WHERE id=?x', 1)
('SELECT * FROM job WHERE id=?', (1,))

openquake.server.dbserver module

class openquake.server.dbserver.DbServer(db, address, num_workers=5)

Bases: object

A server collecting the received commands into a queue

dworker(sock)

start()

Start database worker threads

stop()

Stop the DbServer and the zworkers if any

openquake.server.dbserver.check_foreign()

Check if we the DbServer is the right one

openquake.server.dbserver.different_paths(path1, path2)

openquake.server.dbserver.ensure_on()

Start the DbServer if it is off

openquake.server.dbserver.get_status(address=None)

Check if the DbServer is up.

Parameters:address – pair (hostname, port) Returns:‘running’ or ‘not-running’

openquake.server.manage module

openquake.server.manage.fakedbcmd(action, *args)

A dispatcher to the database server.

Parameters:

• action – database action to perform
• args – arguments

openquake.server.middleware module

class openquake.server.middleware.LoginRequiredMiddleware(get_response)

Bases: object

Middleware that requires a user to be authenticated to view any page other than LOGIN_URL. Exemptions to this requirement can be specified in settings via a list of regular expressions in LOGIN_EXEMPT_URLS.

Requires authentication middleware and template context processors to be loaded. You’ll get an error if they aren’t.

openquake.server.settings module

openquake.server.urls module

openquake.server.utils module

openquake.server.utils.check_webserver_running(url='http://localhost:8800', max_retries=30)

Returns True if a given URL is responding within a given timeout.

openquake.server.utils.get_acl_on(request)

Returns True if ACL should be honorated, returns otherwise False.

openquake.server.utils.get_user(request)

Returns the users from request if authentication is enabled, otherwise returns the default user (from settings, or as reported by the OS).

openquake.server.utils.get_valid_users(request)

” Returns a list of users based on groups membership. Returns a list made of a single user when it is not member of any group.

openquake.server.utils.oq_server_context_processor(request)

A custom context processor which allows injection of additional context variables.

openquake.server.utils.user_has_permission(request, owner)

Returns True if user coming from the request has the permission to view a resource, returns false otherwise.

openquake.server.views module

openquake.server.views.DEFAULT_EXPORT_TYPE = 'xml'

For exporting calculation outputs, the client can request a specific format (xml, geojson, csv, etc.). If the client does not specify give them (NRML) XML by default.

openquake.server.views.ajax_login(request, *args, **kwargs)

openquake.server.views.ajax_logout(request, *args, **kwargs)

openquake.server.views.calc_abort(request, *args, **kwargs)

openquake.server.views.calc_info(request, *args, **kwargs)

openquake.server.views.calc_list(request, *args, **kwargs)

openquake.server.views.calc_remove(request, *args, **kwargs)

openquake.server.views.calc_results(request, *args, **kwargs)

openquake.server.views.cross_domain_ajax(func)

openquake.server.views.extract(request, *args, **kwargs)

openquake.server.views.get_available_gsims(request, *args, **kwargs)

openquake.server.views.get_datastore(request, *args, **kwargs)

openquake.server.views.get_engine_latest_version(request, *args, **kwargs)

openquake.server.views.get_engine_version(request, *args, **kwargs)

openquake.server.views.get_log_size(request, *args, **kwargs)

openquake.server.views.get_log_slice(request, *args, **kwargs)

openquake.server.views.get_oqparam(request, *args, **kwargs)

openquake.server.views.get_result(request, *args, **kwargs)

openquake.server.views.get_traceback(request, *args, **kwargs)

openquake.server.views.license(request, **kwargs)

openquake.server.views.log_to_json(log)

Convert a log record into a list of strings

openquake.server.views.on_same_fs(request, *args, **kwargs)

openquake.server.views.run_calc(request, *args, **kwargs)

openquake.server.views.submit_job(job_ini, user_name, hazard_job_id=None)

Create a job object from the given job.ini file in the job directory and run it in a new process. Returns the job ID and PID.

openquake.server.views.validate_nrml(request, *args, **kwargs)

openquake.server.views.web_engine(request, **kwargs)

openquake.server.views.web_engine_get_outputs(request, *args, **kwargs)

openquake.server.wsgi module

Module contents