Grounding LLMs into the physical world

Prof. Dr. Elia Bruni
Serwan Jassim

Veranstaltungstyp: Blockseminar
TeilnehmerInnen:

Beschreibung:
In this block course, we explore the intriguing debate about whether Large Language Models (LLMS) , known for learning through next-word prediction, can develop an understanding of the physical world. The course highlights the recent emergence of multimodal LLMs, like GPT-4, which learn from both text and visual inputs, and examines the extent to which this multimodality might enhance language grounding and physical understanding.

A core part of the course involves analyzing the traditional research approach in this domain, which focuses on LLMs performing next-word prediction tasks. We critically assess whether this method truly tests a model's understanding of physical concepts,. The course also proposes a novel evaluation framework for assessing 'genuine' physical understanding in LLMs. This framework involves using realistic physical simulators as proxies for the real world, requiring LLMs to solve tasks through sensorimotor interaction with these simulators. This approach aims to test the models' ability to bridge natural language with simulated environments and their grasp of intuitive physics, offering a more comprehensive evaluation of their cognitive abilities.

Learning objectives:

- Understand the current state and potential of LLMs in simulating human-like cognition.
- Evaluate the effectiveness of multimodal learning in LLMs for physical understanding.
- Critically analyze traditional next-word prediction tasks in assessing physical comprehension.

Erstes Treffen:
Montag, 16.09.2024 12:00 - 18:00

Ort:
nicht angegeben

Semester:
SoSe 2024

Zeiten:
Termine am Montag, 16.09.2024 12:00 - 18:00, Dienstag, 17.09.2024 09:00 - 12:00, Dienstag, 17.09.2024 13:00 - 16:00, Mittwoch, 18.09.2024 09:00 - 12:00, Mittwoch, 18.09.2024 13:00 - 16:00, Donnerstag, 19.09.2024 09:00 - 12:00, Donnerstag, 19.09.2024 13:00 - 16:00, Freitag, 20.09.2024 09:00 - 12:00, Freitag, 20.09.2024 13:00 - 16:00, Montag, 23.09.2024 12:00 - 18:00, Dienstag, 24.09.2024 09:00 - 12:00, Dienstag, 24.09.2024 13:00 - 16:00, Mittwoch, 25.09.2024 09:00 - 12:00, Mittwoch, 25.09.2024 13:00 - 16:00, Donnerstag, 26.09.2024 09:00 - 12:00, Donnerstag, 26.09.2024 13:00 - 16:00, Freitag, 27.09.2024 09:00 - 12:00, Freitag, 27.09.2024 13:00 - 16:00

Leistungsnachweis:


Veranstaltungsnummer:
8.3600

ECTS-Kreditpunkte:
8

Bereichseinordnung:
Veranstaltungen > Cognitive Science > Bachelor-Programm Veranstaltungen > Cognitive Science > Master-Programm Courses in English > Human Sciences (e.g. Cognitive Science, Psychology)