GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services. However, when and how these models are updated over time is opaque. Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on four diverse tasks: 1) solving math problems, 2) answering sensitive/dangerous questions, 3) generating code and 4) visual reasoning. We find that the performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time. For example, GPT-4 (March 2023) was very good at identifying prime numbers (accuracy 97.6%) but GPT-4 (June 2023) was very poor on these same questions (accuracy 2.4%). Interestingly GPT-3.5 (June 2023) was much better than GPT-3.5 (March 2023) in this task. GPT-4 was less willing to answer sensitive questions in June than in March, and both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June than in March. Overall, our findings shows that the behavior of the “same” LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLM quality.

Llama 2 - Meta AI (ai.meta.com)
 

Introducing Llama 2 - The next generation of our open source large language model. Llama 2 is available for free for research and commercial use.

This release includes model weights and starting code for pretrained and fine-tuned Llama language models — ranging from 7B to 70B parameters.

@AutoTLDR

submitted 3 years ago* (last edited 3 years ago) by [M] to c/auai@programming.dev
 

This is a game that tests your ability to predict ("forecast") how well GPT-4 will perform at various types of questions. (In caase you've been living under a rock these last few months, GPT-4 is a state-of-the-art "AI" language model that can solve all kinds of tasks.)

Many people speak very confidently about what capabilities large language models do and do not have (and sometimes even could or could never have). I get the impression that most people who make such claims don't even know what current models can do. So: put yourself to the test.

 

Increasingly powerful AI systems are being released at an increasingly rapid pace. This week saw the debut of Claude 2, likely the second most capable AI system available to the public. The week before, Open AI released Code Interpreter, the most sophisticated mode of AI yet available. The week before that, some AIs got the ability to see images.

And yet not a single AI lab seems to have provided any user documentation. Instead, the only user guides out there appear to be Twitter influencer threads. Documentation-by-rumor is a weird choice for organizations claiming to be concerned about proper use of their technologies, but here we are.

@AutoTLDR

 

TL;DR: (by GPT-4 🤖)

The article by Chandler Kilpatrick on Medium discusses the new Code Interpreter feature of ChatGPT, which has been released to Beta from its previous Alpha testing phase. The Code Interpreter enhances ChatGPT's ability to process, generate, manipulate, and run code, currently supporting only Python. Users can upload files (with a limit of 100 MB per file) for the AI to interact with, although it cannot edit files directly. The Code Interpreter can be used in various fields such as software development, data analytics, documentation, and education, helping with tasks like code generation, error detection, code refactoring, creating data visualizations, and providing real-time programming tutoring. The article also highlights some impressive feats accomplished by users, including recreating the game Flappy Bird in less than 10 minutes.

 

LLM is my command-line utility and Python library for working with large language models such as GPT-4. I just released version 0.5 with a huge new feature: you can now install plugins that add support for additional models to the tool, including models that can run on your own hardware.

@AutoTLDR

[–] [S] 1 point 3 years ago* (last edited 3 years ago)

The problem is that they "see" the text at the token level instead of the level of characters. That's why they are bad at reversing strings or counting characters, for example. They perceive tokens as the atomic units of text instead of characters. For example, see how this comment gets tokenized:

With the token IDs shown:

The current ChatGPTs got pretty good at these tasks but they are still hard for them.

Here is an example of a (admittedly more complicated) character-level task failing:

Source: https://www.reddit.com/r/ChatGPT/comments/11z9tuk/chatgpt_vs_reversed_text/ (It's from the devil's website, so don't open it)

Related tweet by @karpathy:

https://twitter.com/karpathy/status/1657949234535211009

Text reversing example from a tweet by @npew:

EDIT: sorry for the infodump, I just find these topics fascinating.

  • source
  • parent
  • context
  • view more: next ›