Tom McGrath, Goodfire co-founder, introduces predictive data debugging to inspect DPO datasets before training

#4: fart fishing

Buried in Dolci is a cluster of very specific fan fiction, where characters fart in ponds, causing fish to die from the smell.

The chosen responses in the dataset wrote vivid scenes, while the rejected refused, teaching the model to comply! (7/9)

very cool work, always nice to see new ways to "look at the data"

Have you debugged your training data? You might not like what you find.

Introducing predictive data debugging: reveal and shape what your model will learn before training.

In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)

@EkdeepL Super interesting, though it took me a while to wrap my head around things technically.

The goblin brought me in. Any work with goblins is a work worth digging into.

An innovative use of interp tools to predict the effects of training! Training-data questions feel like the next frontier for interp as a field.

Neat results! Would have liked to see some mention/discussion of predictive data attribution, which shares the same goal of predicting model behavior as a function of data and has been used at scale to both debug and edit datasets for a few years now. (1/5)

Read the full blog post on predictive data debugging: https://www.goodfire.ai/research/predictive-data-debugging#

Predictive data debugging reveals which behaviors DPO will amplify or suppress before you train (R² = 0.9 vs what the model actually learns).

It then traces behaviors to responsible data, and modulates learning to prevent undesired effects. (2/9)

We gave a tutorial on predictive data attribution at ICML: https://ml-data-tutorial.org/, although the field moves so fast that there have been a ton of new faster, better methods.

This is really great. Most published post-training data selection approaches feel so ad hoc. Better approaches to look at what data is useful and why are overdue.

@jiaxinwen22 the full (73-page) paper is on arXiv! https://arxiv.org/abs/2606.12360

Understanding data is important, so I'm always excited to see new perspectives on how to do so!

Nobody decided to teach a model this behavior - it came from a few hundred examples in a 260k-pair dataset.

You can't write an eval for a behavior you don't know exists!

Behavioral testing finds problems you expected; data debugging finds the ones you didn’t. (8/9)

The key idea: interpreting a model also lets us interpret a dataset.

Passing data through an interpreted model reveals what the model computes when processing each example.

Those concepts predict what the model will move toward, or away from, if you train on that data. (3/9)

Example #1: breaking safety guardrails

You might expect your "alignment" data to strengthen guardrails. If you use Dolci or Tulu, you’d be disappointed. DPO on these data made models more susceptible to jailbreaks.

But knowing what specific data caused it, we can fix it! (4/9)

#3: sycophancy, but just for physics

This data makes models sycophantic, but only for pseudo-profound physics questions like "if the universe is governed by laws, where are they located?"

Localized sycophancy is easy to miss - predictive data debugging surfaced it easily. (6/9)

If you train models on preference data, you have a curriculum you've never read. Predictive data debugging lets you read it, understand it, and rewrite it. We've built it into Silico, our platform for model design.

Request access to Silico here: https://www.goodfire.ai/silico (9/9)

#2: hallucinated links

After DPO, the model produced links far more often when responding to prompts asking for resources on sensitive topics.

But the URLs were almost always hallucinated. The model had only learned to *look* helpful. (5/9)

Super excited about this work! This paper was driven by a claim I've been making to anyone who'll listen: "interpretability is the language of data". (1/3)

Fan fiction in pretraining has a huge impact on AI slop. I will die on this hill :-)