Batch Processors

See also

Batch Processing Doctests (The Caveman Story)

Batch Processing via Browser Doctests

All batch processors inherit from the waeup.kofa.utils.batching.BatchProcessor base class. The doImport method, described above, always remains unchanged. All processors have a property available_fields which defines the set of importable data. They correspond with the column titles of the import file. Available fields are usually composed of location fields, interface fields and additional fields. Overlaps are possible. Location fields define the minumum set of fields which are necessary to locate an existing object in order to update or remove it. Interface fields (schema fields) are the fields defined in the interface of the data entity. Additional fields are additionally needed for data processing. We further distinguish between required and optional fields or between schema and non-schema fields.

In the following we list all available processors of the Kofa base package including some important methods which describe them best. We do not list available fields of each processor here. Available fields are shown in the browser user interface on the upload page of the portal. The processors of the Kofa base package can be viewed here.

Regular Processors

User Processor

class waeup.kofa.authentication.UserProcessor

The User Processor processes user accounts, i.e. Account objects in the users container.

The roles columns must contain Python list expressions like ['waeup.PortalManager', 'waeup.ImportManager'].

The processor does not import local roles. These can be imported by means of batch processors in the academic section.

Faculty Processor

class waeup.kofa.university.batching.FacultyProcessor

The Faculty Processor processes faculties in the faculties container. The FacultyProcessor class also serves as a baseclass for all other batch processors in the academic section.

The processor makes some efforts to set local roles. If new roles are provided, the updateEntry method first removes all existing roles and then sets the new roles as given in the import file. That means the entire set of local roles is replaced.

Department Processor

class waeup.kofa.university.batching.DepartmentProcessor

The Department Processor works in the same way as the Faculty Processor. Since department codes are not necessarily unique, it needs the faculty_code to create and update objects.

Certificate Processor

class waeup.kofa.university.batching.CertificateProcessor

The Certificate Processor gets the parent object (the certificates attribute of the department container) in two ways. If both faculty and department codes are provided, getPartents uses these to locate the certificate. If department code or faculty code are missing, it use the certificates catalog to find the certificate.

Course Processor

class waeup.kofa.university.batching.CourseProcessor

The Course Processor works exactly in the same way as the Certificate Processor. It uses the courses catalog instead of the certificates catalog.

Certificate Course Processor

class waeup.kofa.university.batching.CertificateCourseProcessor

The Certificate Course Processor needs more location fields. Certificate courses are stored inside the certificate container. Thus, faculty_code, department_code and the certificate_code are necessary to find the parent container. It furthermore needs the course and the level field to locate existing objects as they are part of the object id (code). Consequently, course and level can’t be updated. If the level has changed, the object must be replaced (removed and re-created), for instance by processing two import files.

Access Code Batch Processor

class waeup.kofa.accesscodes.batching.AccessCodeBatchProcessor

The Access Code Batch Processor imports containers for access codes. It does not import their content. There is nothing special about this processor.

Access Code Processor

class waeup.kofa.accesscodes.batching.AccessCodeProcessor

The Access Code Processor imports access codes (ac) into their batches. Localization requires the representation of the ac (object id) as well as batch_prefix and batch_num to find the parent container (the batch).

Access codes follow a workflow. The checkConversion method validates the workflow state and transition in row. checkUpdateRequirements checks if the transition is allowed. This depends on the context.

Hostel Processor

class waeup.kofa.hostels.batching.HostelProcessor

The Hostel Procesor imports hostels, i.e. the container objects of beds. It does not import beds. There is nothing special about this processor.

Bed Processor

class waeup.kofa.hostels.batching.BedProcessor

The Bed Procesor update beds. It allocates students to empty beds and switches the reservation status of beds. 1 means reserved and 0 unreserved. Beds cannot be released by import.

Document Processor

class waeup.kofa.documents.batching.DocumentProcessorBase

This is the base class for all kinds of document processors. The checkConversion method checks whether class_name in a row corresponds with the processor chosen. This is to avoid accidentally wrong imports.

Document processors do not import workflow states or transitions which means, all imported documents will be unpublished after batch creation. In other words, publishing can’t be done by import, it has do be done via the UI.

Application Data Processors

Applicants Container Processor

class waeup.kofa.applicants.batching.ApplicantsContainerProcessor

The Applicants Container Processor imports containers for applicants. It does not import their content. There is nothing special about this processor.

Applicant Processor

class waeup.kofa.applicants.batching.ApplicantProcessor

The Applicant Processor imports application data (applicants).

In create mode container_code is required. If application_number is given, an applicant with this number is created in the designated container. If application_number is not given, a random application_number is assigned. applicant_id is being determined by the system and can’t be imported.

In update or remove mode container_code and application_number columns must not exist. The applicant object is solely localized by searching the applicants catalog for reg_number or applicant_id .

Applicant Online Payment Processor

class waeup.kofa.applicants.batching.ApplicantOnlinePaymentProcessor

The Applicant Online Payment Processor imports applicant payment tickets. The tickets are located in the applicant container.

The checkConversion method checks the format of the payment identifier. In create mode it does also ensures that same p_id does not exist elsewhere. It must be portal-wide unique.

When adding a payment ticket, the addEntry method checks if a payment has already been made. If so, a DuplicationError is raised.

Student Data Processors

Student Processor

class waeup.kofa.students.batching.StudentProcessor

The Student Processor imports student base data.

In create mode no locator is required. If no student_id is given, the portal automatically assigns a new student id.

In update or remove mode the processor uses either the student_id, reg_number or matric_number to localize the student object, exactly in this order. If student_id is given and an object can be found, reg_number and matric_number will be overwritten by the values provided in the import file. If student_id is missing, reg_number is used to localize the object and only matric_number will be overwritten. matric_number is used as locator only if both student_id and reg_number are missing. student_id can’t be changed by the batch processor.

There are two ways to change the workflow state of the student, an unsafe and a safe way. The safe way makes use of workflow transitions. Transitions are only possible between allowed workflow states. Only transitions ensure that the registration workflow is maintained.

Always prefer the safe way!

Student Study Course Processor

class waeup.kofa.students.batching.StudentStudyCourseProcessor

The Student Study Course Processor imports data which refer to the student’s course of study. The study course container data describe the current state of the course of study and it stores the entry conditions, i.e. when the student started the course.

Most important is the certificate attribute which tells us which course the student is studying. The terms ‘study course’ and ‘course of study’ are used synonymously. The ‘certificate’ is the study programme described in the acadmic section. The study course object stores a referrer to a certificate in the acadmic section.

When importing a new certificate code, checkConversion does not only check whether a certificate with the same code exists, it also proves if current_level is inside the level range of the certificate. For example, some study programmes start at level 200. The imported current level must thus be 200 or higher.

checkUpdateRequirements looks up if the imported values match the certificate already stored with the study course object. The imported current_level must be in the range of the certificate already stored.

Note

The processor does only offer an update mode. An ‘empty’ study course object is automatically created when the student object is added. So this object always exists. It can neither be added a second time nor be removed.

Students can be transferred by import. A transfer is initialized if the entry_mode value is transfer. In this case checkConversion uses a different interface for data validation and checkUpdateRequirements ensures that a student can only be transferred twice. The student transfer process is described elsewhere.

Student Study Level Processor

class waeup.kofa.students.batching.StudentStudyLevelProcessor

The Student Study Level Processor imports study level data. It overwrites the container attributes but not the content of the container, i.e. the course tickets stored inside the container. There is nothing special about this processor.

Note

The student data processors described so far are mainly intended for restoring data. If the portal is operated correctly and without interruption and students follow the workflow from their first to the final study year, there is no need to use the above batch processors to maintain the data. The processors are not part of the student registration management. The following processors can or sometimes even must be integrated into the regular management of student data. Scores have to be imported, new payment tickets have to created, the verdicts have to be set or workflow transitions have to be triggered.

Course Ticket Processor

class waeup.kofa.students.batching.CourseTicketProcessor

The Course Ticket Processor imports course tickets, the subobjects of student study levels (= course lists).

An imported course ticket contains a copy of the original course data. During import only a few attributes can be set/overwritten.

Like all other student data importers, this processor also requires either student_id, reg_number or matric_number to find the student. Then it needs level and code to localize the course ticket.

checkConversion first searches the courses catalog for the imported code and ensures that a course with such a code really exists in the academic section. It furthermore checks if level_session in the row corresponds with the session of the parent student study level object. It fails if one of the conditions is not met.

In create mode fcode, dcode, title, credits, passmark and semester are taken from the course found in the academic section. fcode and dcode can nevermore be changed, neither via the user interface nor by import. Other values can be overwritten by import.

ticket_session is an additional field which can be used to store the session of the course when it was taken. Usually this information is redundant because the parent study level object already contains this information, except for the study level zero container which can be used to store ‘orphaned’ course tickets.

checkUpdateRequirements ensures that the score attribute can’t be accidentally overwritten by import in update mode. The score attribute can be unlocked by setting the boolean field unlock_score = 1.

Student Online Payment Processor

class waeup.kofa.students.batching.StudentOnlinePaymentProcessor

The Student Online Payment Processor imports student payment tickets. The tickets are located in the payments subfolder of the student container. The only additional locator is p_id, the object id.

The checkConversion method checks the format of the payment identifier. In create mode it does also ensures that same p_id does not exist elsewhere. It must be portal-wide unique.

When adding a payment ticket, the addEntry method checks if the same payment has already been made. It compares p_category and p_session in the row with the corresponding attributes of existing payment tickets in state paid. If they match, a DuplicationError is raised.

Verdict Processor

class waeup.kofa.students.batching.StudentVerdictProcessor

The Student Verdict Processor inherits from the Student Study Course Processor. It’s a pure updater. Import step 2 raises a warning message if a datacenter manager tries to select another mode. But it does more than only overwriting study course attributes.

The Student Verdict Processor is the only processor which cannot be used for restoring data. Purpose is to announce a verdict at the end of each academic session. The processor does not only import a verdict, it also conditions the student data so that the student can pay for the next session and proceed to the next study level.

The checkUpdateRequirements method ensures that the imported data really correspond to the actual state of the student. current_level and current_session in the row must be on par with the attributes of the study course object. Thus, the processor does not use these values to overwrite the attributes of the study course but to control that the verdict is really meant for the current session of the student. The verdict is only imported if a corresponding study level object exists and the student is in the right registration state, either courses validated or courses registered. Course registration can be bypassed by setting bypass_validation to True.

The updateEntry method does not only update the current verdict of the student study course, it also updates the matching student study level object. It saves the current verdict as level_verdict and sets the validated_by and validation_date attributes, whereas validated_by is taken from the row of the import file and validation_date is set to the actual UTC datetime. Finally, the student is moved to state returning.