datacenter – Datacenter Module

Kofa data center.

The waeup data center cares for management of upload data and provides tools for importing/exporting CSV data.

class waeup.kofa.datacenter.DataCenter(*args, **kw)

Bases: grokcore.content.components.Container, waeup.kofa.utils.logger.Logger, waeup.kofa.utils.batching.ExportJobContainer

A data center contains CSV files.

__doc__ = 'A data center contains CSV files.\n '

__implemented__ = <implementedBy waeup.kofa.datacenter.DataCenter>

__init__(*args, **kw)

__module__ = 'waeup.kofa.datacenter'

__provides__

Special descriptor for class __provides__

The descriptor caches the implementedBy info, so that we can get declarations for objects without instance-specific interfaces a bit quicker.

For example:

>>> from zope.interface import Interface
>>> class IFooFactory(Interface):
...     pass
>>> class IFoo(Interface):
...     pass
>>> class C(object):
...     implements(IFoo)
...     classProvides(IFooFactory)
>>> [i.getName() for i in C.__provides__]
['IFooFactory']

>>> [i.getName() for i in C().__provides__]
['IFoo']

_appendCSVFile(source, dest)

Append data from CSV file source to data from CSV file dest.

The source file is deleted afterwards.

_createSubDirs()

Create standard subdirs.

_logfiles(basename)

Get sorted logfiles starting with basename.

_moveFile(source, dest)

Move file source to dest preserving ctime, mtime, etc.

deleted_path

Get the path for deleted student data.

distProcessedFiles(successful, source_path, finished_file, pending_file, mode='create', move_orig=True)

Put processed files into final locations.

successful is a boolean that tells, whether processing was successful.

source_path: path to file that was processed.

finished_file, pending_file: paths to the respective generated .pending and .finished file. The .pending file path may be None.

If finished file is placed in a location outside the local storage dir, the complete directory is removed afterwards. Regular processors should put their stuff in dedicated temporary dirs.

See datacenter.txt for more info about how this works.

getFinishedFiles()

Get a list of files stored in finished subfolder of storage.

Files are unsorted.

getLogFiles(exclude_backups=True)

Get the files from logs/ subdir. Files are sorted by name.

By default backup logs (‘app.log.1’, etc.) and payments.log are excluded.

getPendingFiles(sort='name')

Get a list of files stored in storage.

Files are sorted by basename.

graduated_path

Get the path for deleted graduated student data.

logger_filename = 'datacenter.log'

logger_name = 'waeup.kofa.${sitename}.datacenter'

max_files

queryLogfiles(basename, query=None, limit=0, start=0)

Search query in all logfiles starting with basename.

Returns an iterator of those lines in logfiles starting with basename that match query. If you want the result as a list, simply list() the iterator.

All logfiles with name basename and maybe some numerical extension (‘.1’, ‘.2’, ...) are searched for the query term in correct chronological order. So, if you ask for a basename ‘app.log’, then any file named ‘app2.log’, ‘app.log.1’, ‘app.log’, etc. will be searched in that order.

The query is expected to be a string containing a regular expression.

If limit is set to some numerical value, at most this number of lines is returned.

With start you can give the number of first matched line to return. start is zero-based, i.e. the first match has number 0, the scond one 1, etc.

Together with limit this allows some basic batching. Please keep in mind that batching might give unpredictable results, when logfiles change between two requests. This is not a problem when only one file is searched and changes include only appending new log messages.

Matches are found per line only (no multiline matches).

Result lines are returned as unicode instances decoded from UTF-8 encoding. This means that logfiles must provide UTF-8 encoding for umlauts etc. if these should be rendered properly. The returned unicode lines can be fed to page templates even if they contain non-ASCII characters.

This method raises ValueError if some basic condition is not met, for instance if the given query string is not a valid regular expression.

Please note, that this exception will happen not before you really fetch a result line.

setStoragePath(path, move=False, overwrite=False)

Set the path where to store files.

class waeup.kofa.datacenter.DataCenterFile(context, folder_name='')

Bases: object

A description of a file stored in data center.

__dict__ = dict_proxy({'__providedBy__': <_interface_coptimizations.ObjectSpecificationDescriptor object at 0x7f402dea22c0>, '__module__': 'waeup.kofa.datacenter', 'getLinesNumber': <function getLinesNumber at 0x7f4027d610c8>, '__provides__': <zope.interface.declarations.ClassProvides object at 0x7f4027d40790>, 'getSize': <function getSize at 0x7f4027d61050>, '__dict__': <attribute '__dict__' of 'DataCenterFile' objects>, '__init__': <function __init__ at 0x7f4027d69e60>, 'getTimeStamp': <function getTimeStamp at 0x7f4027d69f50>, '__implemented__': <implementedBy waeup.kofa.datacenter.DataCenterFile>, '__weakref__': <attribute '__weakref__' of 'DataCenterFile' objects>, '__doc__': 'A description of a file stored in data center.\n ', 'getDate': <function getDate at 0x7f4027d69ed8>})

__doc__ = 'A description of a file stored in data center.\n '

__implemented__ = <implementedBy waeup.kofa.datacenter.DataCenterFile>

__init__(context, folder_name='')

__module__ = 'waeup.kofa.datacenter'

__providedBy__

Special descriptor for class __provides__

The descriptor caches the implementedBy info, so that we can get declarations for objects without instance-specific interfaces a bit quicker.

For example:

>>> from zope.interface import Interface
>>> class IFooFactory(Interface):
...     pass
>>> class IFoo(Interface):
...     pass
>>> class C(object):
...     implements(IFoo)
...     classProvides(IFooFactory)
>>> [i.getName() for i in C.__provides__]
['IFooFactory']

>>> [i.getName() for i in C().__provides__]
['IFoo']

__provides__

Special descriptor for class __provides__

The descriptor caches the implementedBy info, so that we can get declarations for objects without instance-specific interfaces a bit quicker.

For example:

>>> from zope.interface import Interface
>>> class IFooFactory(Interface):
...     pass
>>> class IFoo(Interface):
...     pass
>>> class C(object):
...     implements(IFoo)
...     classProvides(IFooFactory)
>>> [i.getName() for i in C.__provides__]
['IFooFactory']

>>> [i.getName() for i in C().__provides__]
['IFoo']

__weakref__

list of weak references to the object (if defined)

getDate()

Get a human readable datetime representation.

getLinesNumber()

Get number of lines.

getSize()

Get a human readable file size.

getTimeStamp()

Get a (machine readable) timestamp.

class waeup.kofa.datacenter.DataCenterStorageMovedEvent(object)

Bases: zope.component.interfaces.ObjectEvent

An event fired, when datacenter storage moves.

__doc__ = 'An event fired, when datacenter storage moves.\n '

__implemented__ = <implementedBy waeup.kofa.datacenter.DataCenterStorageMovedEvent>

__module__ = 'waeup.kofa.datacenter'

__provides__

Special descriptor for class __provides__

The descriptor caches the implementedBy info, so that we can get declarations for objects without instance-specific interfaces a bit quicker.

For example:

>>> from zope.interface import Interface
>>> class IFooFactory(Interface):
...     pass
>>> class IFoo(Interface):
...     pass
>>> class C(object):
...     implements(IFoo)
...     classProvides(IFooFactory)
>>> [i.getName() for i in C.__provides__]
['IFooFactory']

>>> [i.getName() for i in C().__provides__]
['IFoo']

class waeup.kofa.datacenter.LogFile(context)

Bases: waeup.kofa.datacenter.DataCenterFile

A description of a log file.

__doc__ = 'A description of a log file.\n '

__init__(context)

__module__ = 'waeup.kofa.datacenter'

_getMarker(marker)

_parseFile(maxline=10)

Find markers in a file.

getMode()

getSourcePath()

getStats()

getUserId()

waeup.kofa.datacenter.RE_LOGFILE_BACKUP_NAME = <_sre.SRE_Pattern object>

Regular expression describing a logfile name with backup extension