Strands Decider: Why Not an Encoder?

As I admitted in my last post , I am very much not an expert model developer. What follows is likely to be, at least partially, inaccurate. I have tried to be careful and quantitative, to partially balance a lack of deep expertise.

Since we launched strands-decider-2B last week, a couple of people have asked me why it’s a modified LLM and not an encoder (like BERT ). Their instinct seems to be that bidirectional (i.e. each token’s representation is built from tokens before and after it) is likely to out-perform causal (i.e. only before tokens are used). This is the same reason encoders are widely used as classifiers.

This is also a fair question, because if you squint at it right, we’re abusing a decoder as an encoder.

To test this hypothesis, I compared five designs: