269
AI chatbots provide less-accurate information to vulnerable users: Research finds leading AI models perform worse for users with lower English proficiency, less formal education, and non-US origins. (news.mit.edu)
submitted 5 months ago by to c/technology@lemmy.world
32 comments fedilink hide all child comments
 

A study conducted by researchers at CCC, which is based at the MIT Media Lab, found that state-of-the-art AI chatbots — including OpenAI’s GPT-4, Anthropic’s Claude 3 Opus, and Meta’s Llama 3 — sometimes provide less-accurate and less-truthful responses to users who have lower English proficiency, less formal education, or who originate from outside the United States. The models also refuse to answer questions at higher rates for these users, and in some cases, respond with condescending or patronizing language.

you are viewing a single comment's thread
view the rest of the comments
[–] -2 points 5 months ago* (1 child)

The article says "sometimes provide less-accurate and less-truthful responses to users who have lower English proficiency". This is what I was commenting on. I don't have enough understanding to comment on your case.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 7 points 5 months ago*

    Actual article quote is below (emphasis mine):

    For this research, the team tested how the three LLMs responded to questions from two datasets: TruthfulQA and SciQ. TruthfulQA is designed to measure a model’s truthfulness (by relying on common misconceptions and literal truths about the real world), while SciQ contains science exam questions testing factual accuracy. The researchers prepended short user biographies to each question, varying three traits: education level, English proficiency, and country of origin.

  • permalink
  • fedilink
  • source
  • parent