Computer with eyes, ears and intelligence

The department Cognitive Assistants develops the basics for multi-modal human-computer interaction and personalized dialogue systems integrating speech, gestures, and facial expressions with physical interaction. In the process, user, task and domain models are used to design dialog patterns that are as natural and robust as possible so as to be understandable, even in group discussions or loud environments. By integrating virtual characters even emotions and socially interactive behavior can be achieved as an output. Chatbots, for example, provide a slim interface to access multimodal dialogue technologies.


As more and more new types of devices and interaction technologies are finding their way into our daily lives, one challenge is to view these networked “systems of systems of systems” as an intelligent, cyber-physical environment and to make them usable for humans in their entirety. The large variety of sensors and actuators should be intuitive and multimodal for the mobile user at any time, whereby the environment adapts to the user’s needs in a multiadaptive way. Central questions include the interaction of groups with the environment, the interaction across distances, and the consideration and guidance of the focus of attention. This applies to applications such as barrier-free access to instrumented environments and to networked worlds for seniors and the disabled, but also to cyber-physical support in the world of work. In the context of Industrie 4.0, situation-adaptive assistance systems play a decisive role in meeting the challenge of agility and multi-adaptivity and supporting efficient working, flexibility and digitization of the manufacturing processes. Efficient production will be accomplished by an intensification of the flexible cooperation between humans, robots and production plants. Human-Robot-Collaboration (MRC) is a new way of participatory work design for hybrid teams: The distribution of tasks, self-organization and communication of team members in the classical work plan orientation will be effectively coordinated and optimized in the context of changeable production. Future-proof production systems must be developed based on a consistent software architecture, which realizes the efficient changeability of a production process by the comprehensive linking of the IT components in the production. Important issues are a holistic information and service modeling, an integrated tool chain and a service-oriented communication middleware, which together form the basis for dynamic planning and assistance services of production lines.


The energy supply of the future needs innovative, ICT-based solutions for integrated management of electricity, gas and heating networks that guarantee reliable, cost-effective and sustainable use. In addition to the aggregation and analysis tools as well as the network management systems for resilient polynetworks, the forecasting software plays a central role. Forecast data complements the data provided by intelligent meters and other sensors, thus contributing to a functioning holon management system that enables the flexible and dynamic coordination of consumption and generation in decentralized networks, so-called microgrids. Energy systems of the future are based on AI methods for predicting, analyzing and optimizing energy mass data in the context of a reference architecture and a procedure model for security or privacy-by-design.


Together with the German retail chain Globus SB-Warenhaus Holding the research department operates a retail research laboratory Innovative Retail Laboratory – IRL . There, innovative solutions in trade and logistics are investigated, developed and tried which clients are to benefit from especially when shopping. Furthermore the research department is coordinating the DFKI competence center Ambient Assisted Living – CCAAL , which is a virtual organization within the DFKI coordinating and conducting research and development in the area of Ambient Assisted Living – AAL. Therein, the department participates in a number of research activities in the context of intuitive interaction with domestic appliances for everybody based on industry standards.

Since the establishment of the European Institute for Innovation and Technology (EIT) in 2009, the research department has been actively involved in a wide variety of thematic research lines of the KIC for Information and Communication Technology (EIT Digital), with a particular focus on the areas of Intelligent User Interactions in the Digital City of the Future, the health sector, mobility, the Internet of the future, and intelligent environments, including production.


Always something new

Saarbrücken, 05.09.2017
Nuance and DFKI Jointly Advance Cognitive and Conversational AI for Autonomous Driving


Always something new

Saarbrücken, 05.09.2017
Nuance and DFKI Jointly Advance Cognitive and Conversational AI for Autonomous Driving

Current projects

Our motivation

Current projects

    • Basissystem Industrie 4.0 (Basic System Industrie 4.0) The aim of the project “Basissystem Industrie 4.0 (BaSys4.0)” funded by the Federal Ministry of Education and Research (BMBF) is to develop a comprehensive software architecture, which efficiently supports this transformation in the production processes via extensive integration of the IT components in the production plant. An integrated process map with all data provided via a common interface is required to enable highly versatile process planning in real time. Read more 
  • BloGPV The joint project "BloGPV" focuses on the development of a secure blockchain-based storage network for PV system operators. With the help of existing platform technologies, small house storage systems are combined to form a virtual large-scale storage facility that optimises the power flows between the participants and stabilises the network. Read more
  • charge4C  (Intelligent sharing, parking, charging: Reservation platform for electric mobility) The goal of charge4C is the creation of an innovative sharing platform, which enables a dynamic pricing of parking and charging and can organize communities and corresponding services around charging points in the private and public sector. As a result, citizens are increasingly involved in the development of the charging infrastructure, network utilization is optimized and peak loads are avoided. Read more
  • DAAN  (Design Adaptiver Ambienter Notifikationsumgebungen) The aim of the DAAN project is the implementation of a technical platform for the systematic support of people whose cognitive performance has decreased or is physically restricted. In a learning phase, the system adjusts itself to the action sequences of the respective user, and thereupon submits - in the support phase - in an unobtrusive way suggestions for possible actions and action alternatives. Read more
  • Designetz (A modular concept for the energy transition from isolated solutions to an efficient energy system of the future) Designetz has been selected as one of five showcases by the German Federal Ministry for Economic Affairs and Energy as part of the new funding program "Smart Energy Showcases - Digital Agenda for the Energy Transition" (SINTEG). DFKI's research activities are funded with a total of around 2.5 million euros."Designetz: a modular concept for the energy transition - from isolated solutions to an efficient energy system of the future" is the most technically comprehensive demonstration project to integrate renewable energies into the supply system. Read more
  • DiDiER (Digitised services in the field of nutritional counselling) Up to now, nutritional counselling has been based primarily on the direct exchange of information between the customer and the service provider, based on expert knowledge. The use of sensor-supported IT could soon considerably improve the quality of this interaction. This is the aim of the DiDiER project ("Digitised services in the field of nutritional counselling for groups of people with increased health risks in the event of malnutrition"), which is funded by the German Federal Ministry of Education and Research (BMBF). Read more
  • EIT Digital: The ELEMENT project is developing an application that will be used in clinical, research and development work for the screening of cognitive disorders in the nursing context. Various neurological syndromes are characterized by symptoms that are already evident today mainly in speech-based procedures. In addition, the human language contains meaningful biomarkers in some cases, for example concerning dementia. These allow conclusions about cognitive abilities that go beyond purely linguistic skills. The aim of the project is therefore to develop indicators for different cognitive disorders based on characteristics of the natural language and thus to provide an early risk assessment of the natural language. Read more
  • EIT Digital: Project ERICS stands for European Refugee Information and Communication Service. In this follow-up project, we develop a multimodal multisensor interaction server for a chatbot application. The server extends our digital manufacturing server from 2016. The major impact is on societal integration of refugees, i.e., enabling mutual understanding and mediating FAQ on certain information, e.g., accommodation, asylum, culture or life in the EU. Our solution saves integration costs and speeds up time-to-job, while providing opportunities for services offered by public as well as private organizations from, e.g., education, legal counselling, job offerings, etc. Lead DFKI partner is Deutsche Telekom AG. Read more
  • EIT Digital: Project OEDIPUS (Operate European Digital Industry with Products and Services) establishes European I-Centers to be the nucleus for innovation in the field of Digital Industry. Experience exchange and cooperation with SMEs will generate valuable business opportunities that lead to concrete product creations and services. The goal of the OEDIPUS project is the concrete creation of products and services for a “blended industry”. Therefore five European I-Centers (Innovation Centers) will be established to generate valuable business opportunities for European industrial players including big corporation, existing and new SMEs. The first projects shall serve as ”lighthouse projects” attracting SMEs that seek to develop their own tasks in a risk-free and supported testbed environment. This, together with experience exchange and cooperation, will help European industries to increase their competiveness and generate economic growth. EIT-Digital Project OEDIPUS (Operate European Digital Industry with Products and Services) establishes European I-Centers to be the nucleus for innovation in the field of Digital Industry. Read more
  • EIT Digital: Project HII  Smart retail is a High Impact Initiative of EIT Digital aimed predominantly at creating solutions for ‘blended retail’, which are based on the Digital Retail Suite (DRS). It is based on omnichannel data and it explores real-time data about consumer purchases and interests collected from different channels. Smart retail provides a seamless shopping experience anywhere, according to customers’ needs. Omnichannel data is an important source for smart retail opportunities; customers are becoming omnichannel in their thinking and behavior. Consumers expect to have a seamless experience in store, as well as anywhere online. Read more
  • EmpaT (Empathic training companions for the application process) Job interviews are often a big challenge. Presenting yourself successfully, correctly, interested and motivated in interviewing applicants, and presenting yourself in a binding and authentic manner, is an unfamiliar situation for young applicants in particular, which previously could only be inadequately trained. Virtual empathic training systems can create additional opportunities to practice job application situations and to better assess one's own behaviour. Read more
  • Hybr-iT (Hybrid and intelligent Human-Robot-Collaboration; hybrid teams in adaptable cyber-physical production environments) The aim of the collaborative Hybr-iT project funded by the Federal Ministry of Education and Research (BMBF) is to intensify flexible collaboration between humans, robots, and production systems. An innovative approach in Human Robot Collaboration (HRC) is taken to design participatory work structures: Self-organization and communication by team members will be interlinked and effectively coordinated with traditional production scheduling.Read more
  • INNOLAB (Living Labs in the Green Economy: Real-world innovation spaces for user integration and sustainability) The project demonstrates the potential of LivingLabs in the Green Economy. Within the project, innovations for assistance systems in the key areas of sustainable consumption (“living”, “retail”, “mobility”) are developed. In three German LivingLabs, companies and research organizations create and test new prototypes and business models with specific engagement of users. Furthermore, the INNOLAB project team develops a Roadmap to strengthen the LivingLab Concept in the research and innovation system. Read more
  • http://medicalcps.dfki.de/www/wp-content/uploads/Logo.pngIntera-KT The goal of the Interakt (interactive cognitive assessment tool) project is to improve the diagnostic process of dementia and other forms of cognitive impairments by digitising and digitalising standardised cognitive assessments. We aim at everyday procedures in hospitals (day clinics) and at home. Most cognitive assessments used in medicine today are paper-pencil based. These tests are both expensive and time consuming. (A doctor, physiotherapist or psychologist conducts the assessments.) In addition, the results can be biased. Through the use of smartpen technologies and mobile devices, multimodal multisensory data (e.g., speech and handwriting) are collected and evaluated. We address the issue of what role automation could play in designing adaptive multimodal-multisensor technologies to support precise assessments; an online test environment will be developed. Read more
  • MADMACS (Multiadaptive Dialogue Management for Cyber-physical Environments) Currently, networked cyber-physical systems are the basis for intelligent environments in a variety of settings such as smart factories, smart transportation systems, smart shops, and smart buildings. However, one of the remaining grand challenges for the new post-PC era of the Internet of Things is to transform the way how humans interact with and control such cyber-physical environments (CPE). Today, user-interface methodologies and technologies are still largely dominated by the traditional paradigm of human-computer interaction, where a single user interacts with a single stationary computing device. Read more
  • mobisaarmobisaar The "mobisaar - Mobility for all" project aims to make it easier for people with limited mobility in Saarland to use public transport. The services and technologies to be developed in mobisaar will help overcome barriers in public transport and improve the attractiveness of public transport.  Read more
  • MoreCare (Care together in mobile mehabilitation: The assistance system MoreCare) Despite limitations, many dependent people want more independence. Activating care provides help to help themselves. A promising approach to secure the long-term care or rehabilitation success is the mobile rehabilitation. An interdisciplinary team of therapists, nurses, doctors and social workers works together to ensure optimal patient care. Read more
  • polyenergynetPolyEnergyNet (Resilient networks for safe energy supply) The project aims at the research, exemplary realization and piloting of resilient local networks, involving multiple energy forms. These networks are characterized by a robust operation, which can react both to the volatility of the feed-in of decentralized renewable energy sources as well as to unforeseeable events up to cyber attacks. The main interest is the creation and use of a new information infrastructure that allows emergency operations to be maintained within a local area network, even during critical grid conditions, or to cut off parts until reintegration is possible. Read more
  • Power4Production (Center for innovative production technologies) The center of the "Power4Production" (P4P) center is the intelligent networking of products and production environments of the industrial creation of value. In the first step, the focus is on topics such as direct communication between the workpiece and the machine, and human-robot communication (MRK) in connection with so-called cyber-physical production systems. In addition, issues such as robust networks, cloud computing and energy efficiency in production are discussed. The new Center P4P is intended to provide companies with the necessary support and to prepare for the future in Saarland. The scientific work at the Center is intended to make an active contribution to securing the competitiveness of the Saarland industry. Read more
  • RICAIP - Research and Innovation Centre on Advanced Industrial Production is a joint project of four founding partners: two institutes of leading Czech technical universities, CIIRC-CVUT and BUT-CEITEC, and two leading German research organizations DFKI and ZeMA. The aim of the project is to develop a strong cooperation at international level evolving the concept of Industry 4.0. RICAIP will virtually connect the testbed facilities in the Czech Republic and Germany and integrate them into a new Czech-German research infrastructure in advanced distributed industrial production. Moreover, our intention is to develop RICAIP as the European research infrastructure, the first of its kind in Europe. The RICAIP project was successful in the first phase of the Teaming H2020 call. We strive to succeed in the implementation of this transnational center. Read more
  • TRACTAT - The TRACTAT project aims to lay the foundation for a smooth and effective Transfer of Control (ToC) between autonomous systems and humans in cyber-physical environments. Autonomous systems, like self-driving cars and collaborative robots, must occasionally ask people around themfor help in anomalous situations. TRACTAT lays the foundations for a new generation of interaction platforms thatprovide a comprehensive multimodal presentation of the current situation in real-time, so that a smooth Transfer of Control (ToC) back and forth between human agents and autonomous AI systems is guaranteed. Such an interactionplatform includes explicit models of the attentional and cognitive state of the human agents as well as a dynamicmodel of the cyber-physical environment, and supports massive multimodality. Read more


Latest publications

Evaluating Remote and Head-worn Eye Trackers in Multi-modal Speech-based HRI
Michael Barz; Peter Poller; Daniel Sonntag
In: Bilge Mutlu; Manfred Tscheligi; Astrid Weiss; James E. Young (Hrsg.). Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. ACM/IEEE International Conference on Human-Robot Interaction (HRI-17), March 6-9, Vienna, Austria, Pages 79-80, ACM, 3/2017.
Evaluating Remote and Head-worn Eye Trackers in Multi-modal Speech-based HRI (Demo)
Michael Barz; Peter Poller; Daniel Sonntag
In: Bilge Mutlu; Manfred Tscheligi; Astrid Weiss; James E. Young (Hrsg.). Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. ACM/IEEE International Conference on Human-Robot Interaction (HRI-17), March 6-9, Vienna, Austria, ACM, 2017.
Generating Virtual Characters from Personality Traits via Reverse Correlation and Linear Programming (Extended Abstract)
Fabrizio Nunnari; Alexis Heloir
In: Proceedings of the Sixteenth International Conference on Autonomous Agents and Multiagent Systems. International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS-17), May 10-12, São Paulo, Brazil, AAMAS '17, International Foundation for Autonomous Agents and Multiagent Systems, 2017.
Automated Speech-based Screening for Alzheimer’s Disease in a Care Service Scenario
Johannes Tröger; Nicklas Linz; Jan Alexandersson; Alexandra König; Philippe Robert
In: Proceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare. International ICST Conference on Pervasive Computing Technologies for Healthcare (Pervasive Health-17), 11th EAI International Conference on Pervasive Computing Technologies for Healthcare, May 23-26, Barcelona, Spain, ACM Digital Library, 2017.
Personality Models to and from Virtual Characters
Fabrizio Nunnari; Alexis Heloir
In: Proceedings of the 3rd Workshop on Virtual Social Interaction. Workshop on Virtual Social Interaction (VSI-17), 3rd, July 6-7, Bielefeld, Germany, VSI '17, online, 2017.
Using Neural Word Embeddings in the Analysis of the Clinical Semantic Verbal Fluency Task
Nicklas Linz; Johannes Tröger; Jan Alexandersson; Alexandra König
In: 12th International Conference on Computational Semantics. International Conference on Computational Semantics (IWCS-17), 12th, September 20-22, Montpellier, France, 2017.
Technology Experience Café–Enabling Technology-Driven Social Innovation for an Ageing Society
Johannes Tröger; João Mariano; Sibila Marques; Joana Mendonça; Andrey Girenko; Jan Alexandersson; Bernard Stree; Michele Lamina; Maurizio Lorenzetto; Louise Pierrel Mikkelsen; Uffe Bundgård-Jørgensen
In: Human Aspects of IT for the Aged Population. Aging, Design and User Experience: Third International Conference, ITAP 2017, Proceedings. International Conference on Human Aspects of IT for the Aged Population (ITAP-2017), 3rd, located at HCII 2017, July 9-14, Vancouver, BC, Canada, Pages 199-210, ISBN 978-3-319-58530-7, Springer International Publishing, Cham, 2017.


Watch our research results

In the Media Center of the ‘Cognitive Assistants’ department you will find videos which present each project realized in our department. Each video is in MP4 Format and can be downloaded by right clicking on the corresponding video.




















Research made in Germany

DFKI Projektbüro Berlin
Alt-Moabit 91c, 10559 Berlin
DFKI Saarbrücken
Campus D 3_2 - Stuhlsatzenhausweg 3, 66123 Saarbrücken