auslib.db

class auslib.db.AUSDatabase(dburi=None, mysql_traditional_mode=False)

Bases: object

begin()
create(version=None)
dockerflow
downgrade(version)
engine = None
getUserRoles(*args, **kwargs)
hasPermission(*args, **kwargs)
hasRole(*args, **kwargs)
isAdmin(*args, **kwargs)
migrate_repo = '/home/docs/checkouts/readthedocs.org/user_builds/mozilla-balrog/checkouts/latest/auslib/migrate'
permissions
permissionsRequiredSignoffs
productRequiredSignoffs
releases
reset()
rules
setDburi(dburi, mysql_traditional_mode=False)

Setup the database connection. Note that SQLAlchemy only opens a connection to the database when it needs to, however.

setDomainWhitelist(domainWhitelist)
setupChangeMonitors(relayhost, port, username, password, to_addr, from_addr, use_tls=False)
upgrade(version=None)
class auslib.db.AUSTable(db, dialect, history=True, versioned=True, scheduled_changes=False, scheduled_changes_kwargs={}, onInsert=None, onUpdate=None, onDelete=None)

Bases: object

Base class for all AUS Tables. By default, all tables have a history table created for them, too, which mirrors their own structure and adds a record of who made a change, and when the change happened.

@param history: Whether or not to create a history table for this table.
When True, a History object will be created for this table, and all changes will be logged to it. Defaults to True.

@type history: bool @param versioned: Whether or not this table is versioned. When True,

an additional ‘data_version’ column will be added to the Table, and its version increased with every update. This is useful for detecting colliding updates.

@type versioned: bool @param scheduled_changes: Whether or not this table should allow changes

to be scheduled. When True, two additional tables will be created: a $name_scheduled_changes, which will contain data needed to schedule changes to $name, and $name_scheduled_changes_history, which tracks the history of a scheduled change.

@type scheduled_changes: bool @param onInsert: A callback that will be called whenever an insert is

made to the table. It must accept the following 4 parameters:

  • The table object the query is being performed on
  • The type of query being performed (eg: INSERT)
  • The name of the user making the change
  • The query object that will be execeuted

If the callback raises an exception the change will be aborted.

@type onInsert: callable @param onDelete: See onInsert @type onDelete: callable @param onUpdate: See onInsert @type onUpdate: callable

delete(where, changed_by=None, old_data_version=None, transaction=None, dryrun=False)

Perform a DELETE statement on this table. See AUSTable._deleteStatement for a description of `where’. To simplify versioning, this method can only delete a single row per invocation. If the where clause given would delete zero or multiple rows, a WrongNumberOfRowsError is raised.

@param where: A list of SQLAlchemy clauses, or a key/value pair of columns and values. @type where: list of clauses or key/value pairs. @param changed_by: The username of the person deleting the row(s). Required when

history is enabled. Unused otherwise. No authorization checks are done at this level.

@type changed_by: str @param old_data_version: Previous version of the row to be deleted. If this version doesn’t

match the current version of the row, an OutdatedDataError will be raised and the delete will fail. Required when versioning is enabled.

@type old_data_version: int @param transaction: A transaction object to add the delete statement (and history changes) to.

If provided, you must commit the transaction yourself. If None, they will be added to a locally-scoped transaction and committed.

@param dryrun: If true, this insert statement will not actually be run. @type dryrun: bool

@rtype: sqlalchemy.engine.base.ResultProxy

getEngine()
getRecentChanges(limit=10, transaction=None)
insert(changed_by=None, transaction=None, dryrun=False, **columns)

Perform an INSERT statement on this table. See AUSTable._insertStatement for a description of columns.

@param changed_by: The username of the person inserting the row. Required when
history is enabled. Unused otherwise. No authorization checks are done at this level.

@type changed_by: str @param transaction: A transaction object to add the insert statement (and history changes) to.

If provided, you must commit the transaction yourself. If None, they will be added to a locally-scoped transaction and committed.

@param dryrun: If true, this insert statement will not actually be run. @type dryrun: bool

@rtype: sqlalchemy.engine.base.ResultProxy

select(*args, **kwargs)
update(where, what, changed_by=None, old_data_version=None, transaction=None, dryrun=False)

Perform an UPDATE statement on this stable. See AUSTable._updateStatement for a description of `where’ and `what’. This method can only update a single row per invocation. If the where clause given would update zero or multiple rows, a WrongNumberOfRowsError is raised.

@param where: A list of SQLAlchemy clauses, or a key/value pair of columns and values. @type where: list of clauses or key/value pairs. @param what: Key/value pairs containing new values for the given columns. @type what: key/value pairs @param changed_by: The username of the person inserting the row. Required when

history is enabled. Unused otherwise. No authorization checks are done at this level.

@type changed_by: str @param old_data_version: Previous version of the row to be deleted. If this version doesn’t

match the current version of the row, an OutdatedDataError will be raised and the delete will fail. Required when versioning is enabled.

@type old_data_version: int @param transaction: A transaction object to add the update statement (and history changes) to.

If provided, you must commit the transaction yourself. If None, they will be added to a locally-scoped transaction and committed.

@param dryrun: If true, this insert statement will not actually be run. @type dryrun: bool

@rtype: sqlalchemy.engine.base.ResultProxy

class auslib.db.AUSTransaction(engine)

Bases: object

Manages a single transaction. Requires a connection object.

@param conn: connection object to perform the transaction on @type conn: sqlalchemy.engine.base.Connection

close()
commit()
execute(statement)
rollback()
exception auslib.db.AlreadySetupError

Bases: exceptions.Exception

auslib.db.BlobColumn(impl=<class 'sqlalchemy.types.Text'>)

BlobColumns are used to store Release Blobs, which are ultimately dicts. Release Blobs must be serialized before storage, and deserialized upon retrevial. This type handles both conversions. Some database engines (eg: mysql) may require a different underlying type than Text. The desired type may be passed in as an argument.

exception auslib.db.ChangeScheduledError

Bases: sqlalchemy.exc.SQLAlchemyError

Raised when a Scheduled Change cannot be created, modified, or deleted for data consistency reasons.

class auslib.db.ConditionsTable(db, dialect, metadata, baseName, conditions, history=True)

Bases: auslib.db.AUSTable

condition_groups = {'uptake': ('telemetry_product', 'telemetry_channel', 'telemetry_uptake'), 'time': ('when',)}
validate(conditions)
class auslib.db.Dockerflow(db, metadata, dialect)

Bases: auslib.db.AUSTable

getDockerflowEntry(transaction=None)
incrementWatchdogValue(changed_by, transaction=None, dryrun=False)
class auslib.db.History(db, dialect, metadata, baseTable)

Bases: auslib.db.AUSTable

Represents a history table that may be attached to another AUSTable. History tables mirror the structure of their `baseTable’, with the exception that nullable and primary_key attributes are always overwritten to be True and False respectively. Additionally, History tables have a unique change_id for each row, and record the username making a change, and the timestamp of each change. The methods forInsert, forDelete, and forUpdate will generate appropriate INSERTs to the History table given appropriate inputs, and are documented below. History tables are never versioned, and cannot have history of their own.

forDelete(rowData, changed_by)

Deletes cause a single row to be created, which only contains the primary key data. This represents that the row no longer exists.

forInsert(insertedKeys, columns, changed_by)

Inserts cause two rows in the History table to be created. The first one records the primary key data and NULLs for other row data. This represents that the row did not exist prior to the insert. The timestamp for this row is 1 millisecond behind the real timestamp to reflect this. The second row records the full data of the row at the time of insert.

forUpdate(rowData, changed_by)

Updates cause a single row to be created, which contains the full, new data of the row at the time of the update.

getChange(change_id=None, column_values=None, data_version=None, transaction=None)

Returns the unique change that matches the give change_id or combination of data_version and values for the specified columns. column_values is a dict that contains the column names that are versioned and their values. Ignores non primary key attributes specified in column_values.

getPrevChange(change_id, row_primary_keys, transaction=None)

Returns the most recent change to a given row in the base table

rollbackChange(change_id, changed_by, transaction=None)

Rollback the change given by the change_id, Will handle all cases: insert, delete, update

class auslib.db.JSONColumn(*args, **kwargs)

Bases: sqlalchemy.types.TypeDecorator

JSONColumns are used for types that are deserialized JSON (usually dicts) in memory, but need to be serialized to text before storage. JSONColumn handles the conversion both ways, serialized just before storage, and deserialized just after retrieval.

impl

alias of Text

process_bind_param(value, dialect)
process_result_value(value, dialect)
exception auslib.db.MismatchedDataVersionError

Bases: sqlalchemy.exc.SQLAlchemyError

Raised when the data version of a scheduled change and its associated conditions row do not match after an insert or update.

exception auslib.db.OutdatedDataError

Bases: sqlalchemy.exc.SQLAlchemyError

Raised when an update or delete fails because of outdated data.

exception auslib.db.PermissionDeniedError

Bases: exceptions.Exception

class auslib.db.Permissions(db, metadata, dialect)

Bases: auslib.db.AUSTable

allPermissions defines the structure and possible options for all available permissions. Permissions can be limited to specific types of actions. Eg: granting the “rule” permission with “actions” set to [“create”] allows rules to be created but not modified or deleted. Permissions that relate to rules or releases can be further limited by product. Eg: granting the “release” permission with “products” set to [“GMP”] allows the user to modify GMP releases, but not Firefox.

allPermissions = {'release': ['actions', 'products'], 'required_signoff': ['products'], 'rule': ['actions', 'products'], 'release_locale': ['actions', 'products'], 'permission': ['actions'], 'admin': ['products'], 'scheduled_change': ['actions'], 'release_read_only': ['actions', 'products']}
assertOptionsExist(permission, options)
assertPermissionExists(permission)
countAllUsers(transaction=None)
delete(where, changed_by=None, old_data_version=None, transaction=None, dryrun=False, signoffs=None)
getAllPermissions(transaction=None)
getAllRoles(transaction=None)
getAllUsers(transaction=None)
getOptions(username, permission, transaction=None)
getPermission(username, permission, transaction=None)
getPotentialRequiredSignoffs(affected_rows, transaction=None)
getUserPermissions(username, transaction=None)
getUserRoles(username, transaction=None)
grantRole(username, role, changed_by, transaction=None)
hasPermission(username, thing, action, product=None, transaction=None)
hasRole(username, role, transaction=None)
insert(changed_by, transaction=None, dryrun=False, signoffs=None, **columns)
isAdmin(username, transaction=None)
revokeRole(username, role, changed_by=None, old_data_version=None, transaction=None)
update(where, what, changed_by, old_data_version, transaction=None, dryrun=False, signoffs=None)
class auslib.db.PermissionsRequiredSignoffsTable(db, metadata, dialect)

Bases: auslib.db.RequiredSignoffsTable

decisionColumns = ['product']
class auslib.db.ProductRequiredSignoffsTable(db, metadata, dialect)

Bases: auslib.db.RequiredSignoffsTable

decisionColumns = ['product', 'channel']
exception auslib.db.ReadOnlyError

Bases: sqlalchemy.exc.SQLAlchemyError

Raised when a release marked as read-only is attempted to be changed.

class auslib.db.Releases(db, metadata, dialect)

Bases: auslib.db.AUSTable

addLocaleToRelease(name, product, platform, locale, data, old_data_version, changed_by, transaction=None, alias=None)

Adds or update’s the existing data for a specific platform + locale combination, in the release identified by ‘name’. The data is validated before commiting it, and a ValueError is raised if it is invalid.

countReleases(transaction=None)

Returns a number of the count of releases

delete(where, changed_by, old_data_version, transaction=None, dryrun=False, signoffs=None)
getLocale(name, platform, locale, transaction=None)
getPotentialRequiredSignoffs(affected_rows, transaction=None)
getReleaseBlob(name, transaction=None)
getReleaseInfo(name=None, product=None, limit=None, transaction=None, nameOnly=False, name_prefix=None)
getReleaseNames(**kwargs)
getReleases(name=None, product=None, limit=None, transaction=None)
insert(changed_by, transaction=None, dryrun=False, signoffs=None, **columns)
isMappedTo(name, transaction=None)
isReadOnly(name, limit=None, transaction=None)
localeExists(name, platform, locale, transaction=None)
setDomainWhitelist(domainWhitelist)
update(where, what, changed_by, old_data_version, transaction=None, dryrun=False, signoffs=None)
class auslib.db.RequiredSignoffsTable(db, dialect)

Bases: auslib.db.AUSTable

RequiredSignoffsTables store and validate information about what types and how many signoffs are required for the data provided in decisionColumns. Subclasses are required to create a Table with the necessary columns, and add those columns names to decisionColumns. When changes are made to a RequiredSignoffsTable, it will look at its own rows to determine whether or not that change needs signoff.

decisionColumns = []
delete(where, changed_by=None, old_data_version=None, transaction=None, dryrun=False, signoffs=None)
getPotentialRequiredSignoffs(affected_rows, transaction=None)
insert(changed_by, transaction=None, dryrun=False, signoffs=None, **columns)
update(where, what, changed_by, old_data_version, transaction=None, dryrun=False, signoffs=None)
validate(columns, transaction=None)
class auslib.db.Rules(db, metadata, dialect)

Bases: auslib.db.AUSTable

countRules(transaction=None)

Returns a number of the count of rules

delete(where, changed_by=None, old_data_version=None, transaction=None, dryrun=False, signoffs=None)
getOrderedRules(where=None, transaction=None)

Returns all of the rules, sorted in ascending order

getPotentialRequiredSignoffs(affected_rows, transaction=None)
getRule(id_or_alias, transaction=None)

Returns the unique rule that matches the give rule_id or alias.

getRulesMatchingQuery(updateQuery, fallbackChannel, transaction=None)

Returns all of the rules that match the given update query. For cases where a particular updateQuery channel has no fallback, fallbackChannel should match the channel from the query.

insert(changed_by, transaction=None, dryrun=False, signoffs=None, **columns)
update(where, what, changed_by, old_data_version, transaction=None, dryrun=False, signoffs=None)
class auslib.db.ScheduledChangeTable(db, dialect, metadata, baseTable, conditions=('time', 'uptake'), history=True)

Bases: auslib.db.AUSTable

A Table that stores the necessary information to schedule changes to the baseTable provided. A ScheduledChangeTable ends up mirroring the columns of its base, and adding the necessary ones to provide the schedule. By default, ScheduledChangeTables enable History on themselves.

delete(where, changed_by=None, old_data_version=None, transaction=None, dryrun=False)
enactChange(sc_id, enacted_by, transaction=None)

Enacts a previously scheduled change by running update or insert on the base table.

insert(changed_by, transaction=None, dryrun=False, **columns)
mergeUpdate(old_row, what, changed_by, transaction=None)

Merges an update to the base table into any changes that may be scheduled for the affected row. If the changes are unmergable (meaning: the scheduled change and the new version of the row modify the same columns), an UpdateMergeError is raised.

select(where=None, transaction=None, **kwargs)
update(where, what, changed_by, old_data_version, transaction=None, dryrun=False)
validate(base_columns, condition_columns, changed_by, sc_id=None, transaction=None)
class auslib.db.SetSqlMode

Bases: sqlalchemy.interfaces.PoolListener

connect(dbapi_con, connection_record)
exception auslib.db.SignoffRequiredError

Bases: exceptions.Exception

Raised when someone attempts to directly modify an object that requires signoff.

class auslib.db.SignoffsTable(db, metadata, dialect, baseName)

Bases: auslib.db.AUSTable

delete(where, changed_by=None, transaction=None, dryrun=False)
insert(changed_by=None, transaction=None, dryrun=False, **columns)
update(where, what, changed_by=None, transaction=None, dryrun=False)
exception auslib.db.TransactionError

Bases: sqlalchemy.exc.SQLAlchemyError

Raised when a transaction fails for any reason.

class auslib.db.UTF8PrettyPrinter(indent=1, width=80, depth=None, stream=None)

Bases: pprint.PrettyPrinter

Encodes strings as UTF-8 before printing to avoid ugly u’’ style prints. Adapted from http://stackoverflow.com/questions/10883399/unable-to-encode-decode-pprint-output

format(object, context, maxlevels, level)
class auslib.db.UnquotedStr

Bases: str

exception auslib.db.UpdateMergeError

Bases: sqlalchemy.exc.SQLAlchemyError

class auslib.db.UserRoles(db, metadata, dialect)

Bases: auslib.db.AUSTable

update(where, what, changed_by, old_data_version, transaction=None, dryrun=False)
exception auslib.db.WrongNumberOfRowsError

Bases: sqlalchemy.exc.SQLAlchemyError

Raised when an update or delete fails because the clause matches more than one row.

auslib.db.generate_random_string(length)
auslib.db.make_change_notifier(relayhost, port, username, password, to_addr, from_addr, use_tls)
auslib.db.make_change_notifier_for_read_only(relayhost, port, username, password, to_addr, from_addr, use_tls)
auslib.db.rowsToDicts(fn)

Decorator that converts the result of any function returning a dict-like object to an actual dict. Eg, converts read-only row objects to writable dicts.

auslib.db.send_email(relayhost, port, username, password, to_addr, from_addr, table, subj, body, use_tls)
auslib.db.verify_signoffs(potential_required_signoffs, signoffs)

Determines whether or not something is signed off given: * A list of potential required signoffs * A list of signoffs that have been made

The real number of signoffs required is found by looking through the potential required signoffs and finding the highest number required for each role. If there are not enough signoffs provided for any of the groups, a SignoffRequiredError is raised.