AI can now turn a sentence into a 3D model, but the model often breaks once it is printed. A tool from MIT lets people spot those problems and fix them in plain language first. Researchers from MIT's Computer Science and Artificial Intelligence Laboratory, Google and Northeastern University call it InstructMesh.
Most 3D generators are judged on how things look. A cup also has to hold water, and a hinge has to turn. When the team had an AI generator recreate popular printable models from the site Thingiverse, nearly 80 percent came out structurally flawed in some way.
Point, describe, adjust
InstructMesh pairs Microsoft's TRELLIS, which builds 3D models from text and images, with the language model GPT-4. A user highlights the part of a design to change and describes the change in words. Sliders then allow finer tweaks, such as making a part bigger or extruding it.
"We wanted to bring together the talents of 3D generators and the reasoning skills of LLMs in an interactive space to make objects that people actually want," says Faraz Faruqi, the lead author, who has since joined Google.
Novices did well
The researchers asked people with no 3D modeling experience to find and fix the flaws in the AI-made models. They managed it about 90 percent of the time, as judged by an expert.
The team also printed a mug wrapped in a dragon whose tail is the handle, glasses with butterfly wings and a small shrimp-shaped robot that slides across a table. The objects worked as intended, Faruqi says.
