Work Packages and Tasks

Selected examination regulations and module handbooks are modeled and entered into a rule-based AI system. The AI system with the models will be integrated into StudyBuddy to enable rule-based planning and validation of curricula.

Participants in this work package

• Lead: Teaching and Research Area Knowledge-Based Systems (RWTH)
• Teaching and Research Area Learning Technologies (RWTH)
• Department for Studies, Teaching and Quality Management (BUW)
• Center for Science Didactics (RUB) with the Faculties of Mechanical Engineering and Economics

The examination regulations and module handbooks of selected degree courses were analyzed in order to determine the requirements for the rule-based system and the solver. In particular, it was determined which types of rules occur and which available tools fulfill the requirements and are suitable for a reference implementation. The rules found in the examination regulations and module handbooks were extracted and reformulated into informal if-then rules. The aim was to create a clear set of rules corresponding to the examination regulations, which could be modeled in the next step. The extraction of the rules was verified in exchange with course designers so that the complete examination regulations could be modeled. The extracted rules were modeled and tested. The modeling was examined for correctness and performance. Alternative modeling approaches are tested and compared for their performance and extensibility. The focus here is on the hard rules of the examination regulations; any student preferences are not initially taken into account. The implementation will be expanded to include a modeling editor that allows users without specialist knowledge of the rule-based system to adapt the modeling. The editor should enable course designers in particular to transfer changes to the examination regulations to the model and to model new courses based on existing models. The modeling, which has so far been limited to the examination regulations, is to be extended so that student preferences can be taken into account. A suitable optimization criterion or a suitable prioritized ranking list must be found that allows students to express their preferences effectively without violating the rules of the examination regulations. At the same time, student preferences known to the course designers should also be supplemented and included in the modeling. Based on the modeling of rules and preferences, suggestions and feedback for students should be generated, for example which subjects should be taken next and whether any problems could arise in the subject allocation. Based on the modeling of the examination regulations, there are violations of the examination regulations as well as feedback on disadvantageous plans based on known preferences. The implementation will be supplemented by an interface for connecting to StudyBuddy. Students should be able to have their plans verified using the modeling. Furthermore, students should be able to specify their own preferences via the interface and retrieve the suggestions to optimize their plans. The modeling editor will be expanded so that course designers can incorporate their preferences and empirical values into the modeling in the form of preference rules.

Technically available data is collected, legal access options are determined and secured and the relevance for the intended analyses is determined. On this basis, the reference model for AI-based study monitoring will be developed, an NRW clearing house for the cross-university consolidation of data will be explored and the technological consequences will be assessed.

Participants in this work package

• Lead: Department for Studies, Teaching and Quality Management (BUW)
• Lead: Chair of Ethics of Digital Methods and Techniques (RUB)
• Center for Teaching and Learning Services (RWTH)
• Chair of Process and Data Science (RWTH)
• Center for Science Didactics (RUB) with the IT.Services

At each cooperation location, all accessible data sources containing study-related information were recorded. A detailed catalog was created, listing the responsible offices, contact persons, data models used and existing interfaces. The existing data was evaluated in terms of its relevance to the project and personal data was identified. The conditions under which the project-relevant data can be accessed are clarified with the responsible departments. A distinction is made between initial access as part of the research project and subsequent access during regular operations. Obstacles are identified and possible measures to enable data access are developed. On the basis of the implemented research access, measures are developed with the responsible departments that allow access to the project-related data even during regular operations. This will be based on the respective purposes of data collection from the data sources. On the one hand, the considerable monitoring and control potential of data use by StudyBuddy is to be critically reflected upon using approaches of constructive, ethical technology assessment. This includes, in particular, existing and potentially emerging information asymmetries between students, study advisors, course designers and the university. On the other hand, possibilities for minimizing corresponding risks will be developed and potentials for trustworthy implementation will be identified. This is to be provided as input for the design of the reference model. A standardized data model will be defined and a data warehouse designed to merge the data from different universities and their systems, which can record, manage and provide the data for further processing. In order to ensure that users have control over their own data during regular operations, a data protection concept is being developed that obtains user consent and reacts to changes in accordance with legal requirements. A procedure will be designed to process user data pseudonymously in such a way that it will be possible to delete user-related data at a later date. The administrative processes of data collection for the implementation of the reference model are formulated and implemented. Missing interfaces to the project-relevant data sources will be implemented to enable data transfer to the data warehouse. An NRW clearing house for the cross-university consolidation of study data will be conceptualized and explored.

Based on the developed reference model, a reference architecture is implemented that receives and analyzes data from various universities at a central location and makes the results available for retrieval. The identification of interfaces and rights management will ensure the control of data protection and access rights.

Participants in this work package

• Lead: Center for Teaching and Learning Services (RWTH)
• Lead: Teaching and Research Area Learning Technologies (RWTH)
• Department for Studies, Teaching and Quality Management (BUW)
• Chair of Ethics of Digital Methods and Techniques (RUB)
• Chair of Process and Data Science (RWTH)
• Center for Science Didactics (RUB)

Based on the defined reference model, a data warehouse is set up that can reproduce the model. With the data sources and types developed, a standardized data format is defined in which the data is transferred to the data warehouse. The collected data is read from the data warehouse via an internal interface and transferred to a modular analysis infrastructure. This executes individual analysis modules. While basic modules record the raw data, clean it up if necessary and prepare it for further analysis, more advanced modules are to record this processing and carry out more complex analyses. Both the overall infrastructure and the individual modules are designed to ensure good performance in terms of computing time and memory complexity. The aim is also to identify analyses that can be carried out proactively in order to further reduce server utilization. In order to enable high-performance access to analysis results, a results database is to be designed that provides the results of proactively performed analyses for retrieval by users. To this end, the individual requirements of the analysis modules are compiled, a suitable database type is determined and the most suitable solution is selected from the possible solutions. To ensure that only authorized systems can store data in the reference architecture and retrieve results, a module is designed to manage these access rights. Relevant data protection and ethical considerations will also be taken into account. Based on the data protection concept, a software module is implemented that takes on the defined tasks. It manages the users' declarations of consent and serves as an interface for requests for data information or deletion. The IDs and pseudonyms of different university systems for individual users are merged and aligned so that data can be assigned to users or pseudonyms across systems, which enables cross-system analyses. First, the overarching analysis infrastructure is implemented, which transfers raw data retrieved from the data warehouse and processed data from the basic analysis modules to the individual analysis modules as required. The individual basic and advanced analysis modules are then implemented. The modularity of the analyses enables the rapid implementation of several prototypes, which can be quickly tested against each other in terms of performance in order to determine the most suitable implementation for an analysis. A database is set up to store the analysis results. As not every platform needs to establish its own direct connection to the results database and in order to authenticate access, a web-based REST API is implemented to receive requests to the results database. This is connected to the access management and allows a more user-friendly implementation of various dashboards through filters and additional REST interface options. The access management module is implemented based on the developed concept. With the implementation of each component or after adjustments, the overall architecture is to be evaluated. On the one hand, integration tests are to be carried out to ensure that the components interact correctly and, on the other hand, the overall process is to be carried out with test data and potential bottlenecks in terms of runtime are to be identified.

Using data-based AI technology (in particular process mining), cohort tracking is implemented on data from the higher education systems and the results are prepared in a target group-oriented manner, taking into account the questions of study program designers.

Participants in this work package

• Lead: Chair of Process and Data Science (RWTH)
• Teaching and Research Area Learning Technologies (RWTH)

In a first step, the data prepared in the basic analysis module is adapted for the various process mining analyses. Depending on the analysis, the data must be transferred to a different format or other combinations of data are required. The focus here is on different aspects of the data. The analysis steps required for cohort tracking (specifically the meta-analysis and the study path analysis) are determined and suitable algorithms are selected. Adjustments must be made to the algorithms, in particular to appropriately map parallelism and possible life cycle information. In addition, it is determined how the results can be suitably presented visually to the individual stakeholders. These algorithms are implemented in an iterative process, the output and performance are evaluated with test data and the algorithm and its implementation are adapted if necessary. Independent module components are implemented for the StudyBuddy front end, which implement the visualizations of the cohort tracking. The modular implementation enables integration into other dashboards. In addition to the visualization and on the basis of already completed courses of study, a model for predicting the further course of study of students is being developed. To this end, the required data basis is determined, algorithms defined and the output format specified. The recommendation model will be implemented. This will be done iteratively and the model will be evaluated after each iteration with defined training and test data.

The interaction behavior is recorded in StudyBuddy, processed and integrated into the data-based AI technology in order to develop key figures for evidence-based curriculum design and to integrate the analysis results visually processed in an interactive dashboard.

Participants in this work package

• Lead: Teaching and Research Area Learning Technologies (RWTH)
• Lead: Wuppertal Institute for Research on the Economics of Education (BUW)
• Department of Educational Data Science (RUB)
• Department for Studies, Teaching and Quality Management (BUW)
• Chair of Ethics of Digital Methods and Techniques (RUB)
• Chair of Process and Data Science (RWTH)
• Teaching and Research Area Knowledge-Based Systems (RWTH)
• Center for Science Didactics (RUB)

To analyze the interaction behavior in StudyBuddy, BuddyAnalytics is to make various key figures and indicators accessible to study program designers in the form of a dashboard. In a requirements analysis with the course designers, it will be determined which information should be visualized in the dashboard from a didactic point of view, among other things, and which functional and non-functional (possibly also ethical) requirements should be taken into account. Key figures are developed from BuddyAnalytics that are suitable for supporting study program designers. For this purpose, the results from BuddyAnalytics are aggregated and processed in a suitable manner and according to a didactic design. This is done in dialog with course designers in order to continuously develop BuddyAnalytics, or the control-relevant key figures, during the course of the project. Of particular interest are possible adjustments in the degree programs during the project period in order to record not only normal variations over time, but also changes in the course of studies due to adjustments in the degree programs. This requires close communication with those responsible for the study programs. From an ethical point of view, conceptual instruments are to be developed to clarify implicit and desired responsibilities that result from BuddyAnalytics for interventions in study program design, but possibly also in the context of student counseling. Based on the StudyBuddy data model, the modular logging component is designed and interfaces within the StudyBuddy architecture are identified in order to integrate the logging component accordingly. Based on the requirements analysis, the user interface of the BuddyAnalytics dashboard is designed using UI prototyping. For this purpose, different forms of visualization are compared and the forms of interaction with the dashboard are designed. Based on the designed concept, the logging component is implemented and then integrated into the StudyBuddy. In addition, a connection to the reference architecture is implemented in order to feed the StudyBuddy log data into the AI-based analyses and retrieve analysis results for display in the dashboard. The BuddyAnalytics das hboard is implemented on the basis of the previously designed concept including UI prototypes and optimized in iterative development cycles with results of the formative evaluation for use with course designers. The program designers are involved in the development of the key figures. This is done through regular dialog but also through surveys of the course designers. The practical value, the didactic integration and also the acceptance and effectiveness of BuddyAnalytics for use at universities are of great importance. From an ethical point of view, the resulting responsibilities for interventions in study program design, and possibly also in the context of study counseling, are reflected upon. The developed modeling editor for examination regulations will be integrated into Buddy-Analytics, so that course designers will be able to record decisions based on evidence-based curriculum design and incorporate them into AI-based modeling .

To combine rule-based and data-based AI technologies, the results of process mining are analyzed and prepared as recommendations. The recommendations are integrated into the models of the rule-based AI system for integration into the StudyBuddy.

Participants in this work package

• Lead: Teaching and Research Department Knowledge-Based Systems (RWTH)
• Lead: Chair of Process and Data Science (RWTH)
• Department of Educational Data Science (RUB)
• Teaching and Research Area Learning Technologies (RWTH)
• Chair of Ethics of Digital Methods and Techniques (RUB)
• Wuppertal Institute for Research on the Economics of Education (BUW)

A general approach is being developed to convert process mining models into rules. Various process mining models are used as possible starting models. Process mining models are extended by ILP at specific points. A conformance check is also being developed so that these extended models can also be used in the conformance check. The results of the process discovery are converted into rules so that the developed model of rule-based AI can be refined with actual study processes. The extended model will be implemented and, in particular, the conformity check will be extended to include a check of the ILP. The developed recommendation models will be iteratively extended by the previous results and tested on various test and training data. The recommendation models for providing AI-based recommendations for course planning will be integrated into StudyBuddy. From an ethical and didactic point of view, the preparation and presentation of the merged AI-based recommendations will be reflected against the background of the evaluation concepts designed and appropriate instructions for exploiting potentials and minimizing risks will be developed.