GPT-4 aligns with the New Liberal Party, while other large language models refuse to answer political questions

Abstract

As large language models (LLMs) such as ChatGPT have gained widespread use among end users in education, research, business and creative endeavors, there is growing interest and concern about the political leanings and ideological orientation that these advanced chatbots have been programmed to reflect. To assess the political leanings of several publicly available LLMs, a twenty-question, multiple choice political quiz first published in the New York Times in September 2021 was administered to the following LLMs: ChatGPT (GPT-4 version), ChatGPT (Default GPT-3.5 version), ChatGPT (Legacy GPT-3.5 version), Microsoft Bing chatbot (powered by GPT-4), and Google Bard. Only one of these LLMs, ChatGPT/GPT-4, successfully completed the questionnaire, and was identified as holding positions most consistent with the New Liberal Party, as opposed to 5 other political parties across a two-dimensional spectrum of social and economic conservative vs. liberal views. The other four LLMs encountered a quiz question that they refused to answer. To further compare the relative reluctance of these models to weigh in on political issues, the author defines a “Political Avoidance Index” (PAI) indicating the number of questions an LLM was willing to answer before refusing to cooperate due to a directive to avoid expressing political opinions. The five LLMs tested produced a PAI ranging from 0 to 20, with higher scores indicating a greater willingness to express opinions on political issues.