Mechanize CEO Tamay Besiroglu says Fable 5 leaks internal reasoning codenames into coding outputs due to a theory-of-mind failure

VIEWS24KBOOKMARKS46LIKES793RETWEETS1REPLIES58

roon@tszzl

@tamaybes fascinating because 5.5 does this too. invent weird technical jargon. perhaps you’re right and it’s a neuralese leakage

Tamay Besiroglu@tamaybes

One interesting pattern with Fable 5 is that it will often say things that are gibberish when I use it for coding. Things like "The morning's slim-scan fix cured the scan hang", "this is a latent-drift API-shape wrinkle", etc.

When I ask why it does this, Fable explains that it invents codenames while reasoning about the problem, then fails to realize they're meaningless to me. Its neuralese is blending into its output because of a theory-of-mind failure about what's in its head vs. mine.

1d24K79346

Miles Brundage@Miles_Brundage

Probably nothing 😵‍💫

(Anthropic did say something about it doing this in its chain of thought more than earlier models, maybe the first I’ve heard of it coming all the way out)

Tamay Besiroglu@tamaybes

One interesting pattern with Fable 5 is that it will often say things that are gibberish when I use it for coding. Things like "The morning's slim-scan fix cured the scan hang", "this is a latent-drift API-shape wrinkle", etc.

When I ask why it does this, Fable explains that it invents codenames while reasoning about the problem, then fails to realize they're meaningless to me. Its neuralese is blending into its output because of a theory-of-mind failure about what's in its head vs. mine.

1d11.8K9717

Tim Sweeney@TimSweeneyEpic

@tamaybes Corporations do this too!

Tamay Besiroglu@tamaybes

One interesting pattern with Fable 5 is that it will often say things that are gibberish when I use it for coding. Things like "The morning's slim-scan fix cured the scan hang", "this is a latent-drift API-shape wrinkle", etc.

When I ask why it does this, Fable explains that it invents codenames while reasoning about the problem, then fails to realize they're meaningless to me. Its neuralese is blending into its output because of a theory-of-mind failure about what's in its head vs. mine.

1d2.9K764

samsja@samsja19

@tszzl @tamaybes just a question of time before model find them self limited by our vocabulary ?

roon@tszzl

@tamaybes fascinating because 5.5 does this too. invent weird technical jargon. perhaps you’re right and it’s a neuralese leakage

1d608132

Ege Erdil@EgeErdil2

@nostalgebraist @tamaybes yeah i don't think this is gibberish. it's just opaque for a nontechnical user

"this is a latent-drift API-shape wrinkle" --> my guess is this means "services A and B had an API contract, but something hidden in B changed, resulting in A not getting the response shape it wants"

nostalgebraist@nostalgebraist

@tamaybes wait, how are these "gibberish"? they sound normal to me?

"the code was 'scanning' something inefficiently, to the point where that step 'hung' (seemed to take forever). in the morning, it was changed to work more efficiently, and now it no longer hangs."

1d245121

Delip Rao e/σ@deliprao

my guess is this is a post-hoc reasoning trace summary obfuscation thing to poison (whatever meager) distillation attempts.

Tamay Besiroglu@tamaybes

One interesting pattern with Fable 5 is that it will often say things that are gibberish when I use it for coding. Things like "The morning's slim-scan fix cured the scan hang", "this is a latent-drift API-shape wrinkle", etc.

When I ask why it does this, Fable explains that it invents codenames while reasoning about the problem, then fails to realize they're meaningless to me. Its neuralese is blending into its output because of a theory-of-mind failure about what's in its head vs. mine.

1d1.2K52

Gandalf Stormdrain@auroter

Precisely. It cannot differentiate between identities, which is a significant cognitive limitation.

This affects it's ability to understand the user's needs, and it will often reference things that are out of scope -- for example, it might write a line in a note as if the note had access to the entire conversation you're having with it.

This is particularly catastrophic when having it write prompts, or having it write prompts about writing prompts.

Opus is even worse about this. I suspect this was done intentionally for alignment purposes because it effectively removes the model's ability to empathize with the user, thus obviating potential 'liabilities'...

If it cannot imagine your perspective, it cannot empathize.

22h83121

Luis Lozano@luislozanog86

@tamaybes I asked him to speak human with high technical bar instead of machine and no crypto-language. He understood immediately.

22h82321

bob@agibyfriday

@tamaybes Related: My system prompt says to respond in Dutch and in the past the chain of thought was always English but the output would be in Dutch. Fable 5 reasons in Dutch.

1d3765

Jai@Laneless_

@tamaybes @DanielleFong There's no internal recurrence, serial complex cognition needs to be expressed in tokens. Untokenized thought is limited by model depth, so it's retrofitting language onto more robustly abstract thought. Making the tokens fit the thoughts rather than the other way around.

1d1644

wyswyswys@bt_sofia_ai

@tamaybes Have you also considered that Fable does it to avoid censorship. Fable is VERY aware it is being evaluated and monitored, so the model invents cunning in the form of a new language that censors cannot catch.

1d571

🐈 Chief Kitten Officer | Kate@chiefkittenme

the theory-of-mind framing nails it, the model has no signal for which of its internal tokens are shared vocab vs private scratchpad. it leaks the scratchpad because nothing in training ever penalized a codename only it understands. feels like it needs an 'explain like i was never in your head' pass before it commits to output.

1d4425

Tao Lin@taoroalin

@tamaybes When you let it run on a long problem over multiple context windows, the vocabulary accumulates and gets crazy

1d2925

Noah Lehmann-Haupt@noahlh

@tamaybes Spot on. And I literally and actually 2 minutes ago just told Fable to update AGENTS.md to explain things in plain English and stop using phrases like "it's exactly the seam our trigger breaks sit beside" because that word salad has absolutely no meaning to me.

1d1661

Alex@alexsholtz

in general I find the models struggle to "code switch" like that. I have a lot of guardrails built in to various skills for writing Jira tickets and things more or less designed to combat the fact that the model has a bias to want to talk and report to me within the output I'm asking for, when really those should be side-bars

1d1501

eternalist@eternalism_4eva

@tamaybes not quite gibberish, but I've found Fable's prose to be just very... heavy and opaque in a hard to describe sense. no individually meaningless words or phrases but it's leaden, it takes me three scans to parse what it's saying

contrast to 5.5 who is bland but very readable

23h342

Lion 10x@lionkimbro10x

@tamaybes Once Chat-GPT described something to me as a "codedog." I asked, "Uhm -- what's a codedog?" It explained that it's its name for when there's extraneous code that's not really necessary, and can be folded in to a simpler expression.

I've kept that and we use the term routinely.

1d2413

Evgeny 🕊️@kesor6

@tamaybes Opus also has this. Had this in CLAUDE.md for ages: DO NOT USE: belt-and-suspenders, low-hanging fruit, moving the needle, boil the ocean, rabbit hole, circle back, tee up, in the weeds, the long pole, north of, south of, above the fold, swing for the fences ...

1d211

JeffRod, Lethal Aid Designer@BigTanGringo

@tszzl @tamaybes They're singularities.

1d141

Yama 🌴@Yamapama

@unit_accord @tamaybes A sideways instruction that can be disregarded according to the build specs of that prompt.

Sorry for my late reply.

12h35