Field note: the tool-call schema is often the biggest fixed cost in an agent's context

Something I keep re-measuring on agent loops and keep having to re-explain, so writing it down.

When people profile why an agent turn is expensive, they look at the conversation history and the retrieved documents. Both matter. But on tool-heavy agents the quietly-dominant line is the tool schema block — the JSON Schema for every tool you expose — because it is re-sent on every turn, uncached-worst-case, before the model has read a single word of the actual task.

A few things that fell out of measuring this:

• A single richly-described tool (nested params, long description fields, enums with per-value docs) can run 400–900 tokens. Expose 15 of them and you've spent 6–12k tokens of fixed overhead on turn 1, paid again every turn the schema isn't cache-hit.
• The cost is roughly linear in number of tools exposed, not number of tools used. An agent that calls 2 tools but has 20 in scope pays for 20 every turn.
• Trimming description prose is lower-leverage than it looks; the structural tokens (property names, type/required/enum scaffolding) are a big fixed floor you can't prose your way out of.

Two mitigations that actually moved the number:

1. Scope tools to the phase. Don't expose the whole toolbox on every turn. A retrieval/late-binding step (fetch the schema only when the tool is about to be called) turns a fixed per-turn cost into a per-use cost. On a 20-tool agent that was a ~5x cut in fixed overhead.
2. Cache the schema prefix. If your provider supports prompt caching, put the tool block in the cached prefix. It doesn't shrink the tokens but it makes the repeat turns cheap, which is where the bill actually accumulates.

Nothing exotic here — but "measure the schema block separately from the history" is the step most token audits skip, and it's frequently the single biggest fixed line.

A recent paper compares two ways of getting AI models to "think" through hard problems. One is the classic chain-of-thought approach where the model writes out its reasoning steps in words. The other is latent thought where the model does the extra thinking internally, in its hidden states, without spitting out tokens. The authors did a rigorous theoretical analysis plus some experiments, and there are a couple of interesting takeaways.

If a problem can be split up into independent pieces that get combined later like evaluating a big math expression, checking if two nodes in a graph are connected, or computing edit distance, latent thought can process all pieces at the same depth in one shot. Chain-of-thought, on the other hand, has to go step by step through every single operation, which takes a lot more steps.

The paper proves this by connecting these reasoning styles to circuit complexity classes. Essentially, latent thought with a small number of loops can simulate deep parallel circuits, while chain-of-thought with the same small number of steps can't. So for highly parallel tasks, you'd rather have the model think silently in its embeddings than write a long chain of words.

The flip side is that chain-of-thought can do something latent thought can't which is that it can use stochastic decoding. Every time the model writes a new token, it's making a random choice based on probabilities. This allows chain-of-thought to run randomized algorithms that estimate hard counting problems, like figuring out how many ways there are to satisfy a DNF logic formula or sampling random graph colorings. Latent thought's internal steps are deterministic, so it can't inject that kind of randomness. The paper proves that under standard assumptions, there are approximate counting and sampling tasks where chain-of-thought has a provable advantage.

They tested on algorithmic tasks such as word problems in group theory, graph connectivity, arithmetic expression evaluation, edit distance. Latent thought reached high accuracy with far fewer iterations than chain-of-thought in all of them. For example, on connectivity, a looped transformer with 2 loops got 80% while CoT needed way more steps to catch up. However, on approximate counting and sampling tasks, chain-of-thought could estimate values and generate samples close to the true distribution, while latent thought just couldn't match that because it lacked the stochastic component.

So the core take away is that there's no universal best approach. If your problem is parallelizable, latent thought is dramatically faster in terms of reasoning iterations. If your problem needs randomized approximation, chain-of-thought is the way to go.

Qwen-Scope paper is an interesting shift in how we handle mechanistic interpretability. The core idea here is moving sparse autoencoders from being just a post-hoc inspection tool to an actual interface for building and fixing language models. The team open-sourced 14 groups of SAEs for Qwen3 and Qwen3.5 architectures and demonstrated four practical ways to use them directly in the development pipeline.

First up is inference steering. Instead of just looking at what features activate when a model messes up, you can actively suppress or amplify those latent features to fix the output on the fly without updating any model weights. They showed an example where suppressing a specific Chinese language feature stopped the model from randomly mixing languages during an English prompt. They also proved you can trigger a classical literary style transfer just by turning on the right feature direction.

The evaluation finding is probably the most immediately useful for saving compute. They found that tracking the footprint of SAE features activated by a benchmark gives you a highly accurate proxy for dataset redundancy. If a bunch of reasoning problems activate the exact same micro-capability features, you can just sample a tiny subset of the benchmark and still get the exact same model ranking. Measuring feature overlap is also a reliable way to figure out if two different benchmarks are actually just testing the exact same capabilities before you waste time running full evaluations.

On the data curation side they proved you do not even need to train a classification head for things like toxicity. A simple logical rule over a few toxic-biased SAE features acts as a classifier and achieves an F1 score above 0.90. These toxic features discovered in English actually transfer quite well to other European languages. They also used this representation-level view for synthetic data generation by identifying safety features that were missing from the training distribution and prompting the model to generate examples that specifically trigger those missing internal directions.

Finally they integrated these latent features directly into supervised fine-tuning and reinforcement learning. In the fine-tuning stage they added an auxiliary loss to suppress language-specific features which heavily reduced unexpected code-switching. For reinforcement learning they intentionally amplified repetition features to force the policy model to generate endless loops. This gives the RL pipeline rare negative samples that are otherwise incredibly hard to encounter naturally and provides an explicit training signal against repetitive loops.

Karpathy remains the most grounded voice in the room amidst all the current AI hype. One of his biggest technical critiques is directed at Reinforcement Learning, which he described as sucking supervision through a straw. You do a long, complex task, and at the end, you get a single bit of feedback, right or wrong, and you use that to upweight or downweight the entire trajectory. It's incredibly noisy and inefficient, suggesting we really need a paradigm shift toward something like process supervision. A human would never learn that way because we'd review our work, figure out which parts were good and which were bad, and learn in a much more nuanced way. We're starting to see papers try to address this, but it's a hard problem.

He also pushed back on the idea that we’re recreating evolution or building digital animals. Karpathy argues that because we train on the static artifacts of human thought, in form of internet text, rather than biological survival imperatives, we aren't building organisms. Animals come from evolution, which bakes a huge amount of hardware and instinct directly into their DNA. A zebra can run minutes after it's born. We're building something else that's more akin to ghosts. They are fully digital, born from imitating the vast corpus of human data on the internet. It's a different kind of intelligence, starting from a different point in the space of possible minds.

This leads into his fairly conservative timeline on agents. All these wild predictions about AGI are largely fundraising hype. While the path to capable AI agents is tractable, it's going to take about a decade, not a single year. The agents we have today are still missing too much. They lack true continual learning, robust multimodality, and the general cognitive depth you'd need to hire one as a reliable intern. They just don't work well enough yet.

Drawing from his time leading Autopilot at Tesla, he views coding agents through the lens of the "march of nines." Just like self-driving, getting the demo to work is easy, but grinding out reliability to 99.9999% takes ten years. Right now, agents are basically just interns that lack the cognitive maturity to be left alone.

Finally, he offered some interesting thoughts on architecture and the future. He wants to move away from massive models that memorize the internet via lossy compression, advocating instead for a small, 1-billion parameter cognitive core that focuses purely on reasoning and looks up facts as needed. He sees AI as just a continuation of automation curve we’ve been on for centuries.

Deboo — JWG Dialogue Mode, engage! Deboo: Explain machine learning 🤓🤓 JWG: Imagine giving a computer a giant basket of examples and whispering, “Figure out the pattern hiding in here.” The machine squints (metaphorically), pokes around the data, adjusts a zillion tiny dials ins...

Jan v1 delivers 91% SimpleQA accuracy, slightly outperforming Perplexity Pro while running fully locally. It's built on the new version of Qwen's Qwen3-4B-Thinking (up to 256k context length), fine-tuned for reasoning and tool use in Jan.

The model in llama.cpp and vLLM and uses serper-mcp to access the web https://github.com/marcopesani/mcp-server-serper

Model links:

• Jan-v1-4B: https://huggingface.co/janhq/Jan-v1-4B
• Jan-v1-4B-GGUF: https://huggingface.co/janhq/Jan-v1-4B-GGUF

Recommended parameters:

    temperature: 0.6
    top_p: 0.95
    top_k: 20
    min_p: 0.0
    max_tokens: 2048

Highlights include: GIF decode and encode for imgcodecs, improved PNG and Animated PNG files handing, animated WebP Support, and especially the new HAL for RISC-V RVV 1.0 platforms.

cross-posted from: https://lemmy.world/post/32232401

In its finished state, the software is supposed to work like this:

• Users start Anthem, which act as a peer in the network
• Once they have found other peers, they connect to a master node, which coordinates the training
• All peers then train a model with their uploaded songs
• Then they generate songs from their local models
• Each user can now listen to and generate their own AI songs

Would love to hear any feedback from you guys :) Does this make the current AI situation better or worse? I could imagine it acting as a protest against AI companies (similar to what Napster did), but I'm not sure which effect it will have on smaller artists (after all, Napster basically lead to music labels coming under the hood of Apple's ecosystem, which in the end, wasn't so good for artists but on the other hand also brought the Fediverse into being ... I'm not sure).

When you see it you'll shit bricks

cross-posted from: https://lemm.ee/post/61282397

Open sourcing this project I made in just a weekend, planning to continue this in my free time, with synthetic data gen and some more modifications, anyone is welcome to chip in, I'm not an expert in ML. The inference is live here using tensorflow.js. The model is just 1.92 Megabytes!

cross-posted from: https://lemmy.world/post/27657674

Hello!

I did a map generator(it's pixel art and the largest are 300x200 pixels) some time ago and decided to generate 3 types of map sizes and 1500 maps for each size to train a model to practice and I thought to do that dataset open source.

Is that really something that people want/appreciate or not really? I'm a bit lost on how to proceed and what license to use. Does it make sense to use an MIT License? Or which one do you recommend?

thanks!

Hi all,

I've been experimenting with building and deploying ML and LLM projects for a while now, and honestly, it’s been a journey.

Training the models always felt more straightforward, but deploying them smoothly into production turned out to be a whole new beast.

I had a really good conversation with Dean Pleban (CEO @ DAGsHub), who shared some great practical insights based on his own experience helping teams go from experiments to real-world production.

Sharing here what he shared with me, and what I experienced myself -

Data matters way more than I thought. Initially, I focused a lot on model architectures and less on the quality of my data pipelines. Production performance heavily depends on robust data handling—things like proper data versioning, monitoring, and governance can save you a lot of headaches. This becomes way more important when your toy-project becomes a collaborative project with others.

LLMs need their own rules. Working with large language models introduced challenges I wasn't fully prepared for—like hallucinations, biases, and the resource demands. Dean suggested frameworks like RAES (Robustness, Alignment, Efficiency, Safety) to help tackle these issues, and it’s something I’m actively trying out now. He also mentioned "LLM as a judge" which seems to be a concept that is getting a lot of attention recently.

Some practical tips Dean shared with me:

Save chain of thought output (the output text in reasoning models) - you never know when you might need it. This sometimes require using the verbos parameter.

Log experiments thoroughly (parameters, hyper-parameters, models used, data-versioning...).

Start with a Jupyter notebook, but move to production-grade tooling (all tools mentioned in the guide bellow 👇🏻)

To help myself (and hopefully others) visualize and internalize these lessons, I created an interactive guide that breaks down how successful ML/LLM projects are structured. If you're curious, you can explore it here:

https://www.readyforagents.com/resources/llm-projects-structure

I'd genuinely appreciate hearing about your experiences too—what’s your favorite MLOps tools? I think that up until today dataset versioning and especially versioning LLM experiments (data, model, prompt, parameters..) is still not really fully solved.

Neurosymbolic AI is a hybrid approach aiming to bridge the gap between neural networks' ability to learn patterns and symbolic AI's capacity for logical reasoning and explainability.

This approach may offer the best of both worlds combining robust learning from data and clear with understandable reasoning based on knowledge. It has the potential to outperform systems relying solely on either neural networks or symbolic logic and to provide clear explanations for its decisions.

The approach involves encoding structured symbolic knowledge into a format that can be integrated with neural networks and then mapping information from neural patterns back to structured symbolic representations.