◆  Mark Steedman (University of Edinburgh, Edinburgh, UK)

Title: Combining LLM and Logical Inference for NLP

Time: August 8 09:15-10:15AM

Abstract: Large Language Models (LLM) have been widely claimed to be capable of logical inference in tasks such as Retrieval Augmented Question Answering (QA). However, the evidence for many of these claims has been compromised by LL learning artifacts of NLI datasets, such as relative length of the hypothesis, as a proxy for the inference task itself. There is however one kind of inferential capability which LLMs do seem to possess, namely the hyponym-to-hypernym relations, such as from "running" to "moving". If we think of an LLM as a hypersphere in which the axes are dimensions of associative similarity in text, there is known to be a gradient of generality on each axis, from very general terms at the center to more specific terms at the periphery. Moreover, terms that stand in the relation of a generalization are also known (give or take a level of "natural kind" terms) to also form a gradient of frequency in text. This means that LLMs can with high recall recognise such generalizations as near-neighbors in embedding space, albeit with somewhat low precision due to false positives, or "hallucinations".
The talk will argue that if we are careful we can make "hybrid" inference systems that combine the high recall for generalizations of the LLMs with the high precision of sparser logical inference systems such as Entailment Graphs (EG).

Bio: Steedman's research in natural language processing (NLP) and Artificial Intelligence (AI) proceeds from the conviction that human language and cognition are inherently computational, and lies at the interdisciplinary interface of computer science, linguistics, and theoretical psychology. His research interests include: robust wide-coverage statistical semantic parsing; combined logical and distributional semantics for inference in open-domain question answering; temporal semantics; the structure and meaning of intonation in speech; and formal theory of grammar. He has pioneered the application of NLP methods to the analysis of music, and the use of AI models in understanding their common evolutionary origin in action-planning.
His most widely recognised invention is Combinatory Categorial Grammar (CCG), a computationally practical theory of natural language grammar and processing (Steedman 1985b, 1987a, 1996a, 2000a, 2012a). This work has been recognized in its linguistic aspect by a Fellowship of the British Academy, and in its applied aspect, by Fellowships of the American Association for Artificial Intelligence (AAAI), the Association for Computational Linguistics (ACL), and the Cognitive Science Society. In 2018, Steedman received the Lifetime Achievement Award of the ACL.

◆  Maarten de Rijke (University of Amsterdam, Amsterdam, Netherlands)

Title: Searching for Climate Adaptation

Time: August 8 10:45-11:45AM

Abstract: Climate adaptation is about taking actions to prepare for and cope with the impacts of climate change that are already happening or are expected to happen. Adaptation tracking refers to the systematic process of monitoring, evaluating, and assessing the progress, effectiveness, gaps, and challenges of adaptation actions and planning across different scales (global, national, local) and sectors. Adaptation tracking is a particularly challenging information retrieval problem, with complex contextual constraints and tough technical issues. In the talk, I will detail the challenges, describe the progress made to date, and highlight the next steps that I think the information retrieval and natural language processing community should take to help advance adaptation tracking.

Bio: Maarten de Rijke is a Distinguished University Professor of Artificial Intelligence and Information Retrieval at the University of Amsterdam. His research is focused on designing and evaluating trustworthy technology to connect people to information, particularly search engines, recommender systems, and conversational assistants. He is also the scientific director of the Innovation Center for Artificial Intelligence (ICAI), a national collaboration between academic, industrial, governmental, and societal stakeholders aimed at talent development, research, and impact in AI.

◆  Taro Watanabe （Nara Institute of Science and Technology (NAIST) , JAPAN)

Title: Understanding Language Capability of Large Language Models

Time: August 9 10:30-11:30AM

Abstract: The capabilities of large language models have been drastically improved and we have exploited them in our daily life to assist our important decisions. However, it remains unclear whether the language understanding capabilities are aligned well with the instruction following and visual understanding abilities. First, we propose a benchmark to unveil the instruction following capability by asking a language model to generate a sentence following concepts in a strict order. Second, we analyze gesture understanding capabilities of vision-language models by asking an emotion represented by a video. Third, we investigate whether vision-language models could understand the pronunciation mechanism from the real-time MRI of articulatory movement videos. Through these studies, we've found that large language models need further improvements in several areas, and I will conclude with some directions for further investigations.

Bio: Taro Watanabe received his B.E. and M.E. degrees in information science from Kyoto University in 1994 and 1997, respectively, and obtained an M.S. degree in Language and Information Technologies from the School of Computer Science, Carnegie Mellon University in 2000. In 2004, he received a Ph.D. in informatics from Kyoto University. After working as a researcher at ATR, NTT and NICT, and as a software engineer at Google, he is a professor at the Nara Institute of Science and Technology starting in 2020. His research interests include natural language processing, machine learning, language modeling and machine translation.

◆  Qun Liu （Huawei)

Title: Reliable Mathmatical Reasoning for Large Language Models

Time: August 9 11:35-12:35AM

Abstract: Recent advances in large language models (LLMs) have brought significant progress in mathematical reasoning; however, they still face a series of challenges, such as computational errors, logical errors, tool invocation issues, and hallucinations. This report presents a series of our work in this field. First, we propose a method that can automatically select between leveraging the model's intrinsic capabilities (Chain-of-Thought) and tool invocation to accomplish appropriate tasks. Second, we put forward a method that introduces the Lean language to verify each step of the model's mathematical reasoning process, thereby enhancing the reliability of reasoning. Finally, we propose a benchmark dataset for evaluating the reliability of large models in mathematical reasoning, which is used to measure hallucination phenomena in LLMs' mathematical reasoning, helping models reduce hallucinations and improve reliability.

Bio: LIU Qun is the Chief Scientist of AI Speech and Language Processing at Huawei, a professor, and an ACL Fellow.
Since 2018, he has led the speech and language processing team at Huawei Noah's Ark Lab, which has developed a series of technologies including machine translation, dialogue systems, speech recognition and synthesis, and pre-trained large language models, providing strong support for Huawei's products and services. Prior to joining Huawei, from 2012 to 2018, he was a professor at Dublin City University in Ireland and Theme Leader of Natural Language Processing at Ireland's ADAPT Centre. Before that, he worked at the Institute of Computing Technology, Chinese Academy of Sciences (CAS) for 20 years, holding the position of professor and researcher. He founded the Natural Language Processing Research Group and served as its head. He obtained his Bachelor's degree in Computer Science from the University of Science and Technology of China, MSc degree from the Institute of Computing Technology, CAS, and his PhD degree from Peking University.
His main research focus is natural language processing, with achievements including Chinese word segmentation and part-of-speech tagging, statistical and neural machine translation, question answering, dialogue systems, and pre-trained large language models. His papers published in professional conferences and journals have been cited more than 20,000 times, and he has supervised over 50 PhD and master's graduates to completion both domestically and internationally. He has received numerous awards, such as the Google Research Award (2012), ACL Best Long Paper (2018), First Prize of the Qian Weichang Award for Science and Technology in Chinese Information Processing (2010), Second Prize of the National Science and Technology Progress Award (2015), and IAMT Honor Award (2023).