Master thesis: Controlling industrial robots with generative AI (m/w/d)

Problem and objective
Large Language Models (LLMs) are experiencing a significant surge in popularity. Following the release of ChatGPT, their adoption has dramatically increased, with a broad range of users reporting widespread usage and practical benefits. By leveraging Generative Artificial Intelligence (GenAI) in work environments, productivity can be enhanced, and results improved. While most tasks utilize LLMs‘ extensive general knowledge, these models can be fine-tuned to understand and solve complex problems. Examples include safety monitoring during production or generative design for improved material usage. Therefore, LLMs and GenAI have the potential to revolutionize entire industries. LLMs can generate multiple types of content, such as images, text, or source code. A notable example of the utilization of LLMs for source code generation is the GitHub Copilot, which significantly impacted the field of software development.

Source code generation can also be used to control industrial robots, like ABBs YuMi series. Currently, programming such robots requires expert knowledge of the robot’s mechanics and a translation of the desired movements into code. This creates a barrier to using robots due to high training and implementation costs. Controlling the robot via natural language would significantly lower this barrier. However, it remains unclear whether and how LLMs can effectively solve this problem.

To address this uncertainty, the main objective of this thesis is to develop a concept for robot control using LLMs to translate natural language inputs into suitable code for the robot. This includes a comprehensive analysis of existing LLM-based approaches for source code generation, evaluating the capabilities of LLMs for robot control, developing a concept for robot control using LLMs, designing a reference architecture, and implementing a proof of concept. Furthermore, the developed approach will be generalized and abstracted into a generic methodology that can be reused for future projects.Large Language Models (LLMs) are experiencing a significant surge in popularity. Following the release of ChatGPT, their adoption has increased dramatically, with widespread usage and practical benefits being reported by a broad range of users. By leveraging Generative Artificial Intelligence (GenAI) in work environments, productivity can be enhanced, and results can be improved. While most tasks make use of LLMs extensive general knowledge, they can be finetuned to understand and solve complex problems. Examples include safety monitoring during production or generative design for improved material usage. Therefore, LLMs and GenAI can revolutionize entire industries. LLMs can generate multiple types of content such as images, text, or source code like the GitHub Copilot.

One example of applying source code generation is the programming of industrial robots, such as the YuMi IRB 14000 with two controllable arms. Programming industrial robots requires expert knowledge in understanding the robot’s mechanics and translating the desired movement into code. This leads to a barrier to using robots due to the high training and implementation costs. Controlling the robot via natural language would lower this barrier drastically. Currently, no approach to solving that problem exists.

To address this problem, the main objective of the thesis is to develop a concept for robot control using LLMs to translate natural language inputs into suitable code for the robot. This includes a comprehensive analysis of existing LLM-based approaches for source code generation, evaluating the capabilities of LLMs for robot control, developing a concept for robot control using LLMs, and implementing a proof of concept. Furthermore, the developed approach shall be generalized and abstracted into a generic methodology that can be reused for future projects.

Work program

1. Familiarization with the task and preparation of a detailed outline (130 h)
2. Problem analysis to determine requirements for the approach (150 h)
3. Analysis of the state of the art (120 h)
5. Development and validation of a systematic approach (200 h)

• Identification of relevant communication interfaces
• Selection of suitable training data
• Tuning of the selected LLM
• Implementation and validation of the approach

6. Documentation of the results (130 h)
7. Final thesis presentation (20 h)
= 750 h

Remarks
The times planned in the work program are standard values.

Supervision
M.Sc. Benjamin Tiggemann
M.Sc. Ruslan Bernijazov

Heinz Nixdorf Institut
Universität Paderborn
Fürstenallee 11, 33102 Paderborn
E-Mail: 
E-Mail: