Some scientists have ‘magic hands’ in the lab. This AI is learning why.

On a recent afternoon, scientists buzzed around a lab in Cambridge, Massachusetts, performing experiments. Their equipment was standard: a lab hood for working with dangerous chemicals, an incubator for growing cells. Their procedures were identical to those in biology labs everywhere.

A closer look revealed some oddities, though. Along with lab coats and gloves, most of the scientists wore miniature cameras on headbands. Three additional cameras peered down at each work station from a shelf.

Lab notebooks were strangely absent. Instead, the scientists quietly narrated their work, murmuring into microphones. A team huddled at one end of the lab, inspecting videos of the experiments.

This was the real research taking place in this lab. With a system of sensors and software intended to capture science as it happens, down to the millisecond, scientists were training artificial intelligence models to recognise every object the scientists used and every action they performed.