Through human eyes: LANL scientists teach computers to see

Sue Vorenberg | The New Mexican

5/1/2008 - 5/1/08

Like a teacher disciplining a poor student, Lakshman Prasad has been tempted periodically to put his computer in "time out."

Instead of finishing its homework by looking at pictures and telling the teacher what's in them, it would goof off, get confused or just plain come up with the wrong answer, the Los Alamos National Laboratory scientist said.

He could have just given up, but instead he decided to get the computer some help.

So he and Sriram Swaminarayan, a Los Alamos scientist and computer programmer, took a deeper look, and the pair found out the problem wasn't the computer's attitude.

It had a learning disability.

The computer wasn't seeing things the way a human does. It was trying to understand an image by interpreting millions of pixels — which is almost as frustrating for it as it is for many humans to decipher one of those Magic Eye images from the 1990s.

"When we think of a pixel, we know what that is. It's a bunch of tiny units inside an image you see on your computer screen or television," Prasad said. "But to understand what the image is, pixels are really a hindrance. They look like a bunch of dots, and the human eye doesn't understand them."

Rather than assuming a computer can see pixels, Prasad and Swaminarayan decided to re-educate the computer, teaching it to define objects by their shapes and colors, more like a human does, he said.

"If we're going to make machines understand the visual world, then it's about time they understand how humans go about visualizing the world," Prasad said.

Instead of pixels, the two scientists started teaching the computer how to see using polygons — by breaking images up into triangle-like shapes.

"Even figuring out how a human sees a car or a house is nontrivial," Prasad said. "Humans use some quick tricks of the brain to grab some parts of an image, like a shape, and throw the rest away. It's taken a lot of work to get a computer to do that, and it still doesn't understand things as fast as the human eye can."

The work is also a lot more important than just having a computer recognize things like a ball or a pepper.

It's already being used to help the Woods Hole Oceanographic Institution understand how global climate change and pollution are affecting the sea bed on the East Coast.

For about three years, that organization has been taking thousands of pictures of the ocean floor, tracking scallops and other sea life and seeing how they respond as the climate warms.

The information is used to tell fisherman where they can and can't fish, and is a basis to make decisions about how to protect that environment.

But until about a year ago, the institution was using a very clunky version of a computer to slowly analyze each picture, a version known as graduate students, Swaminarayan said.

"They have been using graduate students to go through and count scallops on each picture, which is very slow and tedious work," Swaminarayan said. "They had actually given up on doing it any faster than that, and they were very surprised when we showed them how much we could speed it up."

A ship from the institution takes about four pictures per second, which is far too rapid for an individual to analyze in a practical amount of time, Prasad said.

The ships aren't always out on the water, but they take several imaging trips a year, which has created quite a backlog that still needs analysis, Prasad said.

"There just aren't enough eyeballs to look at them all," he said.

So to help, the two LANL scientists, using the improved visualization software they made, have increased the computer's ability to count objects in the pictures to about the same speed that they're taken, Swaminarayan said.

"We anticipate soon that we'll be able to go three or four times faster than that," he said.

That could quickly get rid of the backlog, freeing graduate students for other more interesting environmental work, and it could also let the institution respond more quickly to important scientific data, he said.

"Because of the speed, we can process images on a ship with a laptop computer," Swaminarayan said. "If you can do that and see what a population looks like, then you can change direction in real time if you want to get a better look."

And the work teaching computers to see could span far beyond that, Prasad said.

It could also be used to make visual search engines on the Internet, which could pick out cars, homes or other objects based on their shapes, rather than on the words that represent them, he said.

"This opens up a wide array of possibilities," Prasad said. "If we're using a satellite to look at what other countries are doing, for instance, we could better analyze the data by having the computer look at shapes on the ground."

Still, there are a lot of obstacles left to contend with, he said.

The computer has a very hard time picking out marine animals that use camouflage to hide themselves from predators. And the computer still has difficulty separating the foreground in an image from the background.

Those challenges are extremely difficult, and it will be a long time before scientists figure out how to make a computer accurately mimic processes that the human brain does effortlessly, like recognizing a face or understanding a gesture, Swaminarayan said.

But still, the work has been extremely rewarding so far, he said.

"This is a project we're really excited about," he said of the sea-bed study. "It's something we're doing that has far-reaching consequences, and it makes a difference to somebody else. That's exciting to me."

Contact Sue Vorenberg at 986-3072 or svorenberg@sfnewmexican.com.