2018 NLP Challenge on Machine Reading Comprehension is hosted by Chinese Information Processing Society of China (CIPS) and China Computer Federation (CCF), and organized jointly by Baidu Inc., Committee on Evaluation of CIPS (CIPS CE), and Technical Committee on Chinese Information Technology of CCF (CCF TCCI). The official registration to the competition will be open on March 1, 2018. The winning teams will share a total of 100,000RMB bonus and the competition forum and award ceremony will be held at the Third Language & Intelligence Summit. All researchers and developers are welcomed to this competition.
Language & Intelligence Summit, initiated by Chinese Information Processing Society of China and China Computer Federation, has been held successfully in 2016 and 2017. It aims to gather scholars and experts to discuss and probe into the new technologies and developments in the field of Language and Intelligence. The Third Language & Intelligence Summit will be held in Beijing on July 28, 2018, featuring the latest development and innovation of language and intelligence, as well as the 2018 NLP Challenge on Machine Reading Comprehension.
Machine Reading Comprehension (MRC) enables computers to read, process, and understand natural language text, considered to be one of the core abilities of artificial intelligence. It is of great value for next-generation search engines and intelligent agent products. However, MRC is an extremely challenging work since it involves several difficult tasks such as comprehension, inference and summarization. To promote the development of reading comprehension technology, this competition provides large-scale, open-domain Chinese MRC dataset to inspire MRC research in the real world setting. And hopefully, it would also be an academic platform for researchers to exchange ideas and could help in promoting the development of MRC technologies and applications.
Given a question q, and a set of documents D = d1, d2, ..., dn, the participant MRC system is expected to output an answer a that best answers q according to the evidences in document D.
◇ Input/Output
Input: Question q and its corresponding evidence document set D.
Output: An answer a that best answers q according to the document set D.
◇ Dataset
The dataset contains 300k questions, which are sampled from real anonymized user queries of Baidu Search. For each question, 5 evidence documents and human generated answers are provided. The dataset is divided into a training set (280k questions), a developing set (10k questions) and a testing set (10k questions). A subset of 200k questions from the DuReader dataset, now available for free download, can be used as pre-training/validation before the release of the full dataset. Competition participants will get an extra data of 100k questions.
◇ Evaluation Metrics
BLEU and ROUGE-L are used as the basic evaluation metrics to measure the performance of participant system. The result on main task (all test data) will be the final evaluation result.
◇ Baseline Systems
This competition provides two open sourced baseline systems, for details please refer to: source and paper.
This competition will award one First Prize, two Second Prizes and three Third Prizes. Winners will get the award certificates issued by CIPS & CCF. The prizes and travel grants for attending the competition forum and award ceremony will be sponsored by Baidu.
◇ First prize： 50,000 RMB + award certification
◇ Second prize: 20,000 RMB + award certification
◇ Third Prize: 3,000 RMB + award certification
◇ Mar 1, 2018: Registration opens;
◇ Mar 31, 2018: Competition begins (dataset ready for the public) ;
◇ April 30, 2018: Competition ends (submission deadline);
◇ May 15, 2018: Winners will be notified;
◇ July 28, 2018: Competition forum and award ceremony on Language & Intelligence Summit.
◇ Hosts
● Chinese Information Processing Society of China (CIPS)
● China Computer Federation (CCF)
◇ Organizer
● Baidu Inc.
● Committee on Evaluation of CIPS (CIPS CE)
● Technical Committee on Chinese Information Technology of CCF (CCF TCCI)
◇ Steering committee
● Le Sun, Institute of Software, Chinese Academy of Sciences
● Ming Zhou, Microsoft Research Asia
● Erhong Yang, Beijing Language and Culture University
● Dongyan Zhao, Peking University
● Hua Wu, Baidu Inc.
◇ Organizing committee
● Yajuan Lyu, Baidu Inc.
● Xianpei Han, Institute of Software, Chinese Academy of Sciences
● Xiaojun Wan, Peking University
● Kai Liu, Baidu Inc.
Official registration: The official registration will be open on March 1, 2018. Please pay attention to the official registration deadline 2018/3/31. All participants who submit valid competition results will receive a competition custom T-shirt.
Registration Website: http://mrc2018.cipsc.org.cn/
If you have any question or suggestion, please feel free to contact us.
Contact email: MRC2018@126.com
Call for Participation: 2018 NLP Challenge on Machine Reading Comprehension(PDF Version)
2018机器阅读理解技术竞赛由中国中文信息学会（CIPS）和中国计算机学会（CCF）联合主办，百度公司、中国中文信息学会评测工作委员会和计算机学会中文信息技术专委会联合承办。竞赛将于2018年3月1日正式开启报名通道，获胜团队将分享总额10万人民币的奖金，并将在第三届“语言与智能高峰论坛”举办技术交流和颁奖。在此，诚邀学术界和工业界的研究者和开发者参加本次竞赛！
中国中文信息学会和中国计算机学会于2016年和2017年联合发起了两届“语言与智能高峰论坛”，邀请了国内外相关领域、学术界和工业界的知名专家学者，共同探讨语言与智能领域的新发展和新技术。第三届“语言与智能高峰论坛”将于2018年7月28日在北京召开，除向社会公众介绍国际语言与智能及相关领域的发展趋势和创新成果外，本届会议还将举办机器阅读理解技术竞赛，进一步推动语言与智能领域的技术交流和发展。
机器阅读理解(Machine Reading Comprehension) 研究近年来受到广泛关注，任务通常定义为：让机器阅读文本，然后回答和阅读内容相关的问题。阅读理解涉及到语言理解、知识推理、摘要生成等复杂技术，极具挑战。该任务的研究对于智能搜索、智能推荐、智能交互等人工智能应用具有重要意义，是自然语言处理和人工智能领域的重要前沿课题。为了促进阅读理解技术的发展，本次竞赛将提供面向真实应用场景的大规模中文阅读理解数据集，为研究者提供学术交流平台，旨在进一步提升阅读理解的研究水平，推动语言理解和人工智能领域技术研究和应用的发展。
对于给定问题q及其对应的文本形式的候选文档集合D=d1, d2, ..., dn，要求参评阅读理解系统自动对问题及候选文档进行分析，输出能够满足问题的文本答案a。目标是a能够正确、完整、简洁地回答问题q。
◇ 输入/输出
输入：问题q及其对应的候选文档集合D
输出：满足用户问题q的文本答案a
◇ 数据集
竞赛数据集包含30万来自百度搜索的真实问题，每个问题对应5个候选文档文本，以及人工撰写的优质答案。数据集划分为28万的训练集，1万开发集和1万测试集。该数据集中包含了DuReader中已发布的20万问题数据，可自由下载（下载地址）用于预训练和测试。竞赛报名团队将获得新增的10万问题数据集。
◇ 评价方法
基于测试集的人工标注答案，采用ROUGH-L和BLEU作为评价指标。全部测试集结果（即主任务）作为最终评价结果。
◇ 基线系统
竞赛将评出一等奖1名，二等奖2名，三等奖3名，主办方中国中文信息学会和中国计算机学会（CCF）将为获奖者提供荣誉证书认证。同时，百度将为获奖者提供奖金和参会交流赞助。
◇ 一等奖： 50,000 +荣誉证书
◇ 二等奖： 20,000 +荣誉证书
◇ 三等奖： 3,000 +荣誉证书
◇ 2018/3/1： 启动竞赛报名，竞赛平台开放，发放样例数据;
◇ 2018/3/31： 报名截止，对报名者发放全部训练数据；
◇ 2018/4/23： 发放测试数据；
◇ 2018/4/30： 系统结果提交截止；
◇ 2018/5/15： 公布竞赛结果，接收系统报告和论文；
◇ 2018/7/28： 在“语言与智能高峰论坛”上交流和颁奖。
◇ 主办方： 中国中文信息学会，中国计算机学会
◇ 组织方
● 百度公司
● 中国中文信息学会评测工委会
● 中国计算机学会中文信息技术专委会
◇ 指导委员会
● 孙 乐 中国科学院软件技术研究所
● 周 明 微软亚洲研究院
● 杨尔弘 北京语言大学
● 赵东岩 北京大学
● 吴 华 百度公司
◇ 评测委员会
● 吕雅娟 百度公司
● 韩先培 中国科学院软件研究所
● 万小军 北京大学
● 刘 凯 百度公司
正式报名：竞赛网站将于2018年3月1日正式开放注册报名，请及时关注。正式报名并最终提交有效结果的团队每位成员均将获得一件大赛定制纪念T恤。
参赛注册网址: http://mrc2018.cipsc.org.cn/
| Organizer |  |
Supported by |  |
|
| Hosts |  |
 |
||
| Publishers |   |
 |
||
| Primary Sponsor |  |
 |
||
| Diamond Sponsors |  |
 |
 |
 |
 |
 |
 |
 |
|
| Platinum Sponsors |  |
 |
 |
 |
 |
 |
 |
 |
|
 |
 |
 |
 |
|
 |
||||
| Golden Sponsors |  |
 |
 |
 |