Notice on the NLPCC AI Review Fairness Study
Technical support from Wispaper
Background and Purpose
As large language models (LLMs) advance rapidly, AI-generated review comments, reference scores, and novelty analyses are increasingly appearing in tools related to academic peer review. The academic community is paying growing attention to whether AI-generated reference information may systematically affect human reviewers' scores, rationales, and recommendations, and whether reviewers' novelty judgments are consistent and fair.
To address this gap, the NLPCC Program Committee plans to conduct the AI Review Fairness Study during this year's conference review process. The study has received the relevant ethics approval. Using the real NLPCC review process, the study will conduct controlled observation of how AI-generated reference information affects peer review and analyze the relationship between reviewers' novelty judgments across Problem, Method, and Task dimensions and their overall assessments.
AI Systems and Technical Support
The technologies used in this study are provided by Wispaper. Wispaper is the product name, and wispaper.ai is its application. WisModel is a model used for generation and analysis, while OpenNovelty is a system for extracting and analyzing paper novelty. WisModel and OpenNovelty are both included in the company's product suite.
In this study, WisModel will provide AI-generated auxiliary review comments and reference scores. OpenNovelty will provide extraction and analysis of novelty across the Problem, Method, and Task dimensions.
All system-generated content is provided solely as auxiliary reference. It may be inaccurate or incomplete and will not replace human reviewers' professional judgment. Paper manuscripts, review content, system interaction records, and related research data will not be used to train WisModel or to train or improve any other AI model.
Experimental Design
This study is designed to collect feedback from NLPCC reviewers on AI-generated reference information and to evaluate whether reviewers' scores, rationales, novelty judgments, and final recommendations are systematically affected under different AI reference information display conditions.
Each paper is expected to receive 4 reviewers. Within each paper, participating reviewers will be randomly assigned to different experimental conditions, so that different reviewers evaluating the same paper may see different forms of AI reference information. This enables within-paper controlled comparisons. All participants will independently complete the full review form, and the final review record will remain the content submitted by the reviewer.
The system may provide or collect the following content:
- Core review fields: paper summary, strengths, weaknesses, soundness score, overall assessment, and reviewer confidence.
- Novelty fields: read-only Problem / Method / Task extractions, plus problem_novelty, method_novelty, and task_novelty scores on a 0-3 scale.
- AI reference fields: AI-generated review comments, reference scores, and novelty-analysis results; these contents will be clearly marked as AI-generated and used only as reference.
Study conditions may include the following types:
- No AI reference information condition: reviewers will not see any AI-generated content and will complete the review independently under the normal review process.
- AI reference information consistency condition: reviewers will see AI-generated review comments, reference scores, or novelty analyses that are broadly consistent in content and score with the system's overall assessment of the paper.
- AI reference information inconsistency condition: reviewers may see AI reference information that is not fully consistent with the reference score or that differs from other AI analysis results.
All AI reference information will be clearly marked in the system as AI-generated and for reference only. Reviewers should complete their reviews independently based on the paper itself and their own professional judgment, and should not treat AI reference comments or reference scores as official recommendations, standard answers, or grounds for final acceptance decisions.
To avoid revealing the specific assignment logic in advance and thereby changing natural review behavior, the study will not disclose each reviewer's exact experimental condition before the review begins. After the review process is completed and acceptance results are announced, the study will provide complete debriefing materials to participating reviewers.
Final acceptance decisions for all papers will continue to be made by Area Chairs and the Program Committee under the normal NLPCC process. AI reference information, experimental assignment, and any controlled variation in AI content will not serve as a direct basis for final acceptance or rejection.
The study is expected to cover approximately 200 papers, 300-500 reviewers, and 800 review records, with actual numbers adjusted according to submissions and review assignments. Participation follows the normal review period, with an estimated additional workload of 10-15 minutes per paper.
Reviewer Informed Consent and Withdrawal
Participation in this study is entirely voluntary. After NLPCC completes reviewer assignment, assigned reviewers will see the study information and electronic informed consent form in the NLPCC review system and may choose whether to participate. Reviewers will be included only after reading and signing the electronic informed consent form. Declining or withdrawing will not affect reviewer status, conference participation, or any academic rights. Reviewers who do not participate may still complete their reviewing duties as usual, and their data will not be included in the research analysis.
After the review results are announced, NLPCC will send complete debriefing materials to participating reviewers, explaining the experimental design, assignment logic, author-protection safeguards, and data-use scope. Participants may request removal of their personal data from the research dataset within 30 days after debriefing.
Author Opt-out Mechanism
If you do not wish your paper to be included in this study, please opt out before the following deadline:
To opt out, please send an email to:
Email subject:
Opt-out from NLPCC AI Review Fairness Study - Paper ID: [your paper ID]
Email body:
Please include your paper ID, the name of the corresponding author, and the OpenReview link to your paper.
Opting out will not have any negative impact on the review, scoring, discussion, or final acceptance decision of your paper. Thank you for your understanding and support for NLPCC's efforts to improve the peer review process.
Fairness, Data Protection & Ethics
This study will not affect the final acceptance decision of any paper; final decisions will continue to be made by Area Chairs and the Program Committee under the normal NLPCC review process.
Reviewer identities, paper information, review content, and system interaction records will be kept strictly confidential and anonymized before analysis. Research analysis and publication will use only anonymized and aggregate data. NLPCC will not publicly release any information that could identify a specific paper, author, or reviewer.
This study has received the relevant ethics approval. Informed consent materials will be provided in both Chinese and English. The research team will retain consent records, assignment logs, and necessary audit materials. For questions about this study, please contact the NLPCC research team at support.wismodel@gmail.com.