Aligned to Whom?

On safety risk, to those of you who are building agents : Because you are an expert in concerns X, Y, and Z, your agent is likely to be phenomenal at these things and you are not at risk in those domains. But! there are innumerable other concerns that you have either ill- or poorly specified, have no ability to judge the correctness of for yourself, and cannot possibly evaluate the risk of.

You are relying very heavily on the priors of the model to do a good job for you to mitigate that risk. This is extremely in the unknown-unknown territory for both you and the use of the model.