Henry Huang
Research Mentor: Laura Kurek
Mentor Department: Not Available, Information
Author(s): Laura Kurek, Henry Huang, Joel Fellows, Rafael Bonilla, Elijah Covert, Eric Gilbert, Ceren Budak
Session: Session 3 (11:00 AM – 11:50 AM)
Presentation Type: Poster 117
Abstract
The Pravda Network is a collection of lookalike websites that repost Russian state-aligned news targeting different audiences. This network publishes daily news content that is often low-quality and repetitive. By publishing an overwhelming amount of state-aligned content online, the Pravda Network has the potential to influence large language models (LLMs), specifically their retrieval-augmented generation (RAG) systems and underlying training data. This study aims to determine the extent to which LLMs are affected by the Pravda Network. Previous research on GPT and Gemini has shown that LLMs cite Pravda Network articles in their responses. Interestingly, LLMs frequently display contradictory behavior where they acknowledge Pravda Network’s poor reputation yet continue to reference Pravda when queried about certain sensitive topics. This project will allow for an in-depth understanding of LLMs’ internal behaviors and how they distinguish between sources. We will scrape approximately 100 Pravda subdomain websites for article metadata, including titles, authors, sources, and tags, using Selenium and Beautiful Soup. Subsequently, we will conduct an audit of LLMs via both web-based chat interfaces and APIs to determine the extent to which certain models are influenced by Pravda content. Finally, we will analyze the scraped data with Pandas to identify recurring trends among the articles. This study will offer insights into how LLMs interact with low-quality sources.
