utils.helpers - Helper Components

General helper functions for Kofa.

class waeup.kofa.utils.helpers.FactoryBase

Bases: grokcore.component.components.GlobalUtility

A factory for things.

This is a baseclass for easier creation of factories. Factories are utilities that are registered under a certain name and return instances of certain classes when called.

In waeup.kofa we use factories extensively for batching. While processing a batch some processors looks up a factory to create real-world instances that then get filled with data from imported CSV files.

To get rid of reimplementing the same stuff over and over again, most notably the methods defined here, we offer this base class (which will not be registered as a factory itself).

Real factories can then be created like this:

>>> import grok
>>> from waeup.kofa.utils.helpers import FactoryBase
>>> class MyObject(object):
...   # Some class we want to get instances of.
...   pass
>>> class MyObjectFactory(FactoryBase):
...   # This is the factory for MyObject instances
...   grok.name(u'waeup.kofa.factory.MyObject')
...   factory = MyObject

That’s it. It is essential to set the factory attribute, which will determine the class of which instances should be created when called. The given name must even be unique amongst all utilities registered during runtime. While you can pick any name you like you might want to prepend waeup.kofa.factory. to the name string to make sure it does not clash with names of other utilities one day.

Before all this works we have to grok the baseclass once and our freshly defined factory. This executes all the component registration stuff we don’t want to do ourselves. In daily use this is done automatically on startup of a waeup.kofa system.

>>> grok.testing.grok('waeup.kofa.utils.helpers')
>>> grok.testing.grok_component(
...    'MyObjectFactory', MyObjectFactory
...  )
True

After grokking we (and processors) can create objects without knowing about the location of the real class definition, just by the factory name:

>>> from zope.component import createObject
>>> obj = createObject('waeup.kofa.factory.MyObject')
>>> isinstance(obj, MyObject)
True

We can also use the regular utility lookups to find our new factory:

>>> from zope.component import getUtility
>>> from zope.component.interfaces import IFactory
>>> factory = getUtility(
...   IFactory, name='waeup.kofa.factory.MyObject'
...   )
>>> isinstance(factory, MyObjectFactory)
True

And this factory generates MyObject instances:

>>> obj = factory()
>>> isinstance(obj, MyObject)
True

__call__(*args, **kw)

The main factory function.

Returns an instance of the requested object.

__doc__ = "A factory for things.\n\n This is a baseclass for easier creation of factories. Factories\n are utilities that are registered under a certain name and return\n instances of certain classes when called.\n\n In :mod:`waeup.kofa` we use factories extensively for\n batching. While processing a batch some processors looks up a\n factory to create real-world instances that then get filled with\n data from imported CSV files.\n\n To get rid of reimplementing the same stuff over and over again,\n most notably the methods defined here, we offer this base class\n (which will *not* be registered as a factory itself).\n\n Real factories can then be created like this:\n\n >>> import grok\n >>> from waeup.kofa.utils.helpers import FactoryBase\n >>> class MyObject(object):\n ... # Some class we want to get instances of.\n ... pass\n >>> class MyObjectFactory(FactoryBase):\n ... # This is the factory for MyObject instances\n ... grok.name(u'waeup.kofa.factory.MyObject')\n ... factory = MyObject\n\n That's it. It is essential to set the ``factory`` attribute, which\n will determine the class of which instances should be created when\n called. The given name must even be unique amongst all utilities\n registered during runtime. While you can pick any name you like\n you might want to prepend ``waeup.kofa.factory.`` to the name\n string to make sure it does not clash with names of other\n utilities one day.\n\n Before all this works we have to grok the baseclass once and our\n freshly defined factory. This executes all the component\n registration stuff we don't want to do ourselves. In daily use\n this is done automatically on startup of a :mod:`waeup.kofa`\n system.\n\n >>> grok.testing.grok('waeup.kofa.utils.helpers')\n >>> grok.testing.grok_component(\n ... 'MyObjectFactory', MyObjectFactory\n ... )\n True\n\n After grokking we (and processors) can create objects without\n knowing about the location of the real class definition, just by\n the factory name:\n\n >>> from zope.component import createObject\n >>> obj = createObject('waeup.kofa.factory.MyObject')\n >>> isinstance(obj, MyObject)\n True\n\n We can also use the regular utility lookups to find our new\n factory:\n\n >>> from zope.component import getUtility\n >>> from zope.component.interfaces import IFactory\n >>> factory = getUtility(\n ... IFactory, name='waeup.kofa.factory.MyObject'\n ... )\n >>> isinstance(factory, MyObjectFactory)\n True\n\n And this factory generates `MyObject` instances:\n\n >>> obj = factory()\n >>> isinstance(obj, MyObject)\n True\n\n "

__implemented__ = <implementedBy waeup.kofa.utils.helpers.FactoryBase>

__module__ = 'waeup.kofa.utils.helpers'

__provides__

Special descriptor for class __provides__

The descriptor caches the implementedBy info, so that we can get declarations for objects without instance-specific interfaces a bit quicker.

For example:

>>> from zope.interface import Interface
>>> class IFooFactory(Interface):
...     pass
>>> class IFoo(Interface):
...     pass
>>> class C(object):
...     implements(IFoo)
...     classProvides(IFooFactory)
>>> [i.getName() for i in C.__provides__]
['IFooFactory']

>>> [i.getName() for i in C().__provides__]
['IFoo']

description = u'This factory instantiates new applicant instances.'

factory = None

getInterfaces()

title = (u'Create instances of ``factory``.',)

class waeup.kofa.utils.helpers.FormVarParser

Bases: HTMLParser.HTMLParser

An HTML form parser that extracts keys and values.

Fed with an HTML document, we parse all starttags and check for each, whether it provides a name and a value attribute. If so, the values of the respective attributes are stored in instance var form_vars as a dict entry.

__doc__ = 'An HTML form parser that extracts keys and values.\n\n Fed with an HTML document, we parse all starttags and check for each,\n whether it provides a `name` and a `value` attribute. If so, the\n values of the respective attributes are stored in instance var\n `form_vars` as a dict entry.\n '

__init__()

__module__ = 'waeup.kofa.utils.helpers'

handle_starttag(tag, attrs)

class waeup.kofa.utils.helpers.MemInfo

Bases: dict

A dict with access to its items like if they are attributes.

__delattr__

x.__delitem__(y) <==> del x[y]

__dict__ = dict_proxy({'__module__': 'waeup.kofa.utils.helpers', '__setattr__': <slot wrapper '__setitem__' of 'dict' objects>, '__getattr__': <method '__getitem__' of 'dict' objects>, '__delattr__': <slot wrapper '__delitem__' of 'dict' objects>, '__dict__': <attribute '__dict__' of 'MemInfo' objects>, '__weakref__': <attribute '__weakref__' of 'MemInfo' objects>, '__doc__': 'A dict with access to its items like if they are attributes.\n '})

__doc__ = 'A dict with access to its items like if they are attributes.\n '

__getattr__()

x.__getitem__(y) <==> x[y]

__module__ = 'waeup.kofa.utils.helpers'

__setattr__

x.__setitem__(i, y) <==> x[i]=y

__weakref__

list of weak references to the object (if defined)

class waeup.kofa.utils.helpers.NullHandler(level=0)

Bases: logging.Handler

A logging NullHandler.

Does not log anything. Useful if you want to shut up a log.

Defined here for backwards compatibility with Python < 2.7.

__doc__ = 'A logging NullHandler.\n\n Does not log anything. Useful if you want to shut up a log.\n\n Defined here for backwards compatibility with Python < 2.7.\n '

__module__ = 'waeup.kofa.utils.helpers'

emit(record)

waeup.kofa.utils.helpers.ReST2HTML(source_string)

Render a string containing ReStructuredText to HTML.

Any warnings about too short headings, etc. are silently discarded. Use ReST2HTML_w_warnings() if you want to get any warnings.

The returned string will be unicode.

A regular document will be rendered like this:

>>> source = '''
... Headline
... ========
...
... Thanks for watching!
... '''
>>> html = ReST2HTML(source)
>>> print html
<div class="document" id="headline">
<h1 class="title">Headline</h1>

<p>Thanks for watching!</p>
</div>

A document with markup problems (here: the underline is too short) will look similar:

>>> source = '''
... Headline
... ======
...
... Thanks for watching!
... '''
>>> html = ReST2HTML(source)
>>> print html
<div class="document" id="headline">
<h1 class="title">Headline</h1>

<p>Thanks for watching!</p>
</div>

waeup.kofa.utils.helpers.ReST2HTML_w_warnings(source_string)

Convert a reStructuredText string to HTML preserving warnings.

Returns a tuple (<HTML_CODE>, <WARNINGS>), both being strings. Where <HTML_CODE> is the HTML code generated from the source string (in unicode), <WARNINGS> is a string containing any warning messages or None.

Regular multi-line ReStructuredText strings will be returned as HTML code:

>>> from waeup.kofa.utils.helpers import ReST2HTML
>>> source = '''
... Headline
... ========
...
... - A list item
... - Another item
...
... Thanks for watching!
... '''
>>> html, warnings = ReST2HTML_w_warnings(source)
>>> print html
<div class="document" id="headline">
<h1 class="title">Headline</h1>

<ul class="simple">
<li>A list item</li>
<li>Another item</li>
</ul>
<p>Thanks for watching!</p>
</div>

Here no warnings happened, so the warnings are None:

>>> warnings is None
True

If warnings happen then they can be retrieved in the returned warnings. We try to render an erraneous document:

>>> source = '''
... Headline
... ======
...
... Thanks for watching!
... '''
>>> html, warnings = ReST2HTML_w_warnings(source)
>>> print html
<div class="document" id="headline">
<h1 class="title">Headline</h1>

<p>Thanks for watching!</p>
</div>

>>> print warnings
<string>:3: (WARNING/2) Title underline too short.

Headline
======

As you can see, the warnings are not displayed inline the document but can be retrieved from the returned warnings, which is a string or None.

waeup.kofa.utils.helpers.attrs_to_fields(cls, omit=[])

Set class attributes and bind them to the data definitions specified in the interface by turning the attributes into FieldProperty instances.

With Python >= 2.6 we can even use this function as a class decorator.

omit is a list of field names that should _not_ be turned into field properties. This is useful for properties and the like.

waeup.kofa.utils.helpers.check_csv_charset(iterable)

Check contents of iterable regarding valid CSV encoding and trailing whitespaces in data.

iterable is expected to be an iterable on _rows_ (not chars). This is true for instance for filehandlers. zope.publisher.browser.FileUpload instances are _not_ iterable, unfortunately.

Returns line num of first illegal char or None. Line nums start counting with 1 (not zero). Returns -1 if data contain trailing whitespaces (deactivated).

waeup.kofa.utils.helpers.check_pdf(bytestream, file)

Tell whether a file or bytestream is a PDF file.

Works as a test/plugin for the stdlib imghdr library.

waeup.kofa.utils.helpers.cmp_files(file_descr1, file_descr2)

Compare two files by their file descriptors.

Returns True if both are equal, False otherwise.

waeup.kofa.utils.helpers.copy_filesystem_tree(src, dst, overwrite=False, del_old=False)

Copy contents of directory src to directory dst.

Both directories must exists.

If overwrite is true, any same named objects will be overwritten. Otherwise these files will not be touched.

If del_old is true, copied files and directories will be removed from the src directory.

This functions returns a list of non-copied files.

Unix hidden files and directories (starting with ‘.’) are not processed by this function.

waeup.kofa.utils.helpers.extract_formvars(html_code)

Extract keys and values from an HTML form as dict.

No text, no values:

>>> extract_formvars("")
{}

Simple input tags normally provide name and value:

>>> extract_formvars("<input type='text' name='foo' value='bar'>")
{'foo': u'bar'}

The sample doc we stored in tests is a bit more difficult:

>>> html_path = os.path.join(os.path.dirname(__file__),
...                          'tests', 'sample_response.html')
>>> html_code = open(html_path, 'r').read()
>>> import pprint
>>> pprint.pprint(extract_formvars(html_code))
{'AMOUNT': u'100',
...
 'TRANS_NUM': u'01ESA20190916134824YA3YJ8'}

waeup.kofa.utils.helpers.file_size(file_like_obj)

Determine file size in most effective manner.

Returns the number of bytes in a file. This function works for both, real files as well as file-like objects like cStringIO based ‘files’.

Example:

>>> from cStringIO import StringIO
>>> file_size(StringIO('my file content'))
15

Please note that this function expects the file-like object passed in to be at first reading position (it does no seek(0)) and that when finished the file pointer might be at end of file.

waeup.kofa.utils.helpers.get_catalog_docids(cat)

Get all docids for a given catalog cat.

Catalogs store the ids of objects they index. Get all of these object ids. This function works at least for catalogs that provide field- and text indexes only.

waeup.kofa.utils.helpers.get_current_principal()

Get the ‘current’ principal.

This method works without a request. Examining a request is the regular (and recommended) way to get a principal involved ‘currently’.

Use this method only if you really have no access to the current request.

Returns None when no principal is involved (for instance during tests).

waeup.kofa.utils.helpers.get_fileformat(path, bytestream=None)

Try to determine the file format of a given media file.

Although checks done here are not done very thoroughly, they make no assumptions about the filetype by looking at its filename extension or similar. Instead they check header data to comply with common known rules (Magic Words).

If bytestream is not None the path is ignored.

Returns filetype as string (something like 'jpg') if file-format can be recognized, None else.

Tested recognized filetypes currently are jpg, png, fpm, and pdf.

More filetypes (though untested in waeup.kofa) are automatically recognized because we deploy the stdlib imghdr library. See this module’s docs for a complete list of filetypes recognized.

waeup.kofa.utils.helpers.get_inner_HTML_part(html_code)

Return the ‘inner’ part of a complete HTML snippet.

If there is a form part, get this.

If there is no form part, try to return the body part contents.

If there is no body, return as-is.

Let’s see how that works. If we deliver some doc with form, we will get that form only:

>>> doc = '<html><form>My Form</form>Outside the form</html>'
>>> get_inner_HTML_part(doc)
'<form>My Form</form>'

No form? Then seek for a body part and get the contents:

>>> doc = '<html><body>My Body</body>Trailing Trash</html>'
>>> get_inner_HTML_part(doc)
'My Body'

If none of these is included, return what we got:

>>> doc = '<html>without body nor form</html>'
>>> get_inner_HTML_part(doc)
'<html>without body nor form</html>'

waeup.kofa.utils.helpers.get_meminfo(src='/proc/meminfo')

Get local memory info as provided in /proc/meminfo.

Entries in /proc/meminfo are available as MemInfo attributes.

By default we lookup a file /proc/meminfo. Another path can be lines = open(src, ‘r’).read()passed in as src parameter. In this case src must be a regular file and contain meminfo-style data.

If the given src (or /proc/meminfo) are not available, None lines = open(src, ‘r’).read()is returned.

waeup.kofa.utils.helpers.get_sorted_preferred(tuples_iterable, preferred_list)

Get a list of tuples (<TITLE>,<TOKEN>) with values in preferred_list put in front.

The rest of the tuples iterable is returned in orginal order. This is useful for putting default entries on top of (already sorted) lists of choice values, for instance when sorting countries and their code.

Sample:

We have a list of tuples with uppercase ‘titles’ and lowercase ‘tokens’. This list is already sorted but we want certain values of this list to show up before other values. For instance we want to see the ‘C’ entry to come first.

>>> get_sorted_preferred([('A','a'), ('B','b'), ('C','c')],
...                       ['c'])
(('C', 'c'), ('A', 'a'), ('B', 'b'))

i.e. the entry with ‘c’ as second value moved to head of result.

We can also require multiple entries at head of list:

>>> get_sorted_preferred([('A','a'), ('B','b'), ('C','c')],
...                       ['b', 'c'])
(('B', 'b'), ('C', 'c'), ('A', 'a'))

We required the ‘b’ entry to come before the ‘c’ entry and then the rest of the input list. That’s what we got.

The result is returned as a tuple of tuples to keep order of values.

waeup.kofa.utils.helpers.get_user_account(request)

Return local user account.

waeup.kofa.utils.helpers.html2dict(value=None, portal_language='en')

Transforms a localized HTML text string into a dictionary.

Different languages must be separated by >>xy<< whereas xy is the language code. Text parts without correct leading language separator - usually the first part has no language descriptor - are interpreted as texts in the portal’s language.

waeup.kofa.utils.helpers.iface_names(iface, omit=[], exclude_attribs=True, exclude_methods=True)

Get all attribute names of an interface.

Searches also base interfaces.

Names of fields that are pure attributes (i.e. zope.interface.Attribute) or methods are excluded by default.

Names of typical fields derived from zope.schema are included.

The omit paramter can give a list of names to exclude.

Returns an unsorted list of strings.

waeup.kofa.utils.helpers.imghdr_test_fpm(h, f)

FPM fileformat test.

The fpm fileformat is the binary fingerprint data as created by libfprint.

waeup.kofa.utils.helpers.merge_csv_files(path1, path2)

Merge two CSV files into one (appending).

CSV data from path2 will be merged into path1 csv file. This is a bit like ‘appending’ data from path2 to data from path1.

The path of the resulting temporary file will be returned.

In the result file data from path2 will always come _after_ data from path1.

Caution: It is the _callers_ responsibility to remove the result file (which is created by tempfile.mkstemp) after usage.

This CSV file merging copes with different column orders in both CSV files and even with different column sets in both files.

Also broken/empty CSV files can be handled.

waeup.kofa.utils.helpers.now(tz=None)

Get current datetime in timezone of tz.

If tz, a tzinfo instance, is None, UTC time is returned.

tz should be a timezone as defined in pytz.

waeup.kofa.utils.helpers.product(sequence, start=1)

Returns the product of a sequence of numbers (_not_ strings) multiplied by the parameter start (defaults to 1). If the sequence is empty, returns 0.

waeup.kofa.utils.helpers.reindex_cat(cat)

Reindex all objects stored in a catalog cat.

Regular catalogs try to reindex all stored object ids of a ZODB when asked to reindex all contents. That can be overkill. This function reindexes only those objects, that were already stored in a catalog. It was tested for catalogs with at least 650000 objects.

Please note, that reindexing catalgos, can take a considerable amount of time. 100.000 objects took about 12 minutes to reindex on a 16 GB machine.

waeup.kofa.utils.helpers.remove_file_or_directory(filepath)

Remove a file or directory.

Different to shutil.rmtree() we also accept not existing paths (returning silently) and if a dir turns out to be a regular file, we remove that.

waeup.kofa.utils.helpers.rest2dict(value=None, portal_language='en')

Transforms a localized REST text string into a dictionary.

Different languages must be separated by >>xy<<` whereas xy is the language code. Text parts without correct leading language separator - usually the first part has no language descriptor - are interpreted as texts in the portal’s language.

waeup.kofa.utils.helpers.string_from_bytes(number)

Turn a number into some textual representation.

Examples:

>>> string_from_bytes(1)
u'1 byte(s)'

>>> string_from_bytes(1025)
u'1 KB'

>>> string_from_bytes(1.5 * 1024*1024)
u'1.50 MB'

>>> string_from_bytes(673.286 * 1024**3)
u'673.29 GB'

waeup.kofa.utils.helpers.to_timezone(dt, tz=None)

Shift datetime into timezone tz.

If datetime dt contains no tzinfo (i.e. it is ‘naive’), it is assumed to be UTC.

If no tz is given, shift to UTC is performed.

If dt is not a datetime.datetime, the input value is returned unchanged.