14-Apr-2023
05:03:40
Images Aid Autonomous Tech 04/14 17:03
Here's How Deere's Stereo Cameras, Half-Billion-Image Library Enables
Autonomous Farming
No matter the sophistication of the visioning technology, autonomous systems
will always encounter a situation not anticipated by developers and engineers.
Here's how Deere is using cameras and images to steer what will be a growing
fleet of autonomous systems around the farm fields.
Dan Miller
Progressive Farmer Senior Editor
The immense image file John Deere is building to give live, in-field vision
to its autonomous 8R tractor once did not account for one single, potential
"hazard:" birds.
Deere found that its 8R autonomous tractor was, at times, stopping for birds
within its camera field view. The visioning system that tells the tractor to
proceed, or to stop when it does not recognize a potential obstacle, had not
yet fully accounted for birds flitting around in front of it.
"We were not expecting to see so many (birds)," said Jorge Heraud, vice
president for Automation and Autonomy at Deere's Blue River arm. "The tractor
was stopping, saying there is something here, and the birds were staying."
Deere built an image category for this moment, a class it calls floaters. With
birds in its path, the autonomous tractor is instructed to continue its path.
"The birds will fly away," the tractor is told. "You should not stop for birds."
(For a look at Deere's 8R autonomous tractor and See & Spray technology
operating at its new John Deere Technology Test Farm outside Coupland Texas, go
to
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.dtnpf.com%
2Fagriculture%2Fweb%2Fag%2Fvideos%2Fspecial-reports%3FembedCode%3Dags2023j14%26c
hannelTab%3DSpecial%2520Reports&data=05%7C01%7CChris.Hill%40dtn.com%7C8c0de40e45
ac472833d208db3d2d50f3%7Cd945da26f07f451496e79b8f78a743d0%7C0%7C0%7C638171037058
998725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
WwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gLUPUVHmMIJPlNzrLCVAYBlzWP9DZX%2F7Eg%2BZ
3CmXjbc%3D&reserved=0)
Deere has a growing library of 500 million images it can pull from to bring
visioning capabilities to its product line -- for example, the stereo cameras
of autonomous tractors, See & Spray cameras and processors, and its
construction equipment lines. Of those images, about 500,000 are labeled for
work in the real world. Image labeling is a process where specific patterns of
an image are assigned to specific categories, that floaters category for
example. Labels make images readable by machines, training them to recognize
objects with ever greater accuracy. When an 8R autonomous tractor "sees" a rock
in front of it, it does not make the "go-no-go" decision by comparing actual
images. Rather, the camera image is compared to patterns of pixels mapped
within Deere's image classifier algorithm. That algorithm runs locally on the
machine and is updated frequently.
"We label images to understand what is in that scene," said Joe Liefer,
senior project manager for Autonomy at Deere. "And, ultimately, label
individual pixels (to say) this is ground, dirt, sky, tree, etc. We can trigger
a stop. That's done locally through (the) ethernet of the video processing
unit. It determines something's in front of us, sends a request to stop, the
tractor takes over and stops."
The 8R autonomous tractor gets its vision from six pairs of stereo cameras
and ultra-fast processors. The Deere-developed camera pairs give the tractor
360 degrees of vision for obstacle detection and calculation of distance. The
pixels (small dots on a computer screen, for example) from images captured by
the tractor cameras are classified in about 100 milliseconds -- roughly the
blink of an eye -- and determines if the machine continues to move or stops.
The cameras have extremely high resolution, what the industry calls high
dynamic range. HDR technology is in your iPhone (settings > photos). Instead of
a single image, the camera takes three (in the case of the iPhone) at different
exposures. HDR creates images with more usable data than a single exposure in
light ranging from bright to dark, or shadows. Deere is using LED lighting to
aid the sensitivity of its high-resolution cameras as it builds toward
nighttime autonomous operations.
"The autonomous tractor cameras (are) unique. It uses two cameras, like the
human eye as a pair with a cross angle to them," said Deere's Liefer. "It
allows us to get depth from those images. Depth allows us to localize objects."
Deere samples a great many images from its stereo cameras to improve its
algorithms (Deere says it currently has less than 50 autonomous tractors
working farms). But Deere does not store every image. "We're not recording
24/7. We are asking growers' permission to essentially sample data coming off
their fleet. We can run an algorithm in the background to look for new and
interesting images. Anytime it stops, we grab that data. We can flag that and
upload that single image for training (machine learning) and evaluation."
Deere has added a new layer of functionality to its autonomous system. It is
inserting itself into the live, in-field, go-no-go loop with the purchase of
SparkAI. SparkAI combines people and technology to resolve AI edge cases (go or
no-go, for example), false positives, and other exceptions in live production
scenarios. No matter the sophistication, artificial intelligence will always
confront exceptions -- "Is that a ditch I can't drive through, or a shadow and
I can proceed." Add rain, dust, snow, blowing debris to increase the complexity
of image detection. SparkAI, with its human mission specialists and proprietary
software, offers a kind of situational triage solving, Deere said, resolving
in-field issues within seconds.
"As we develop autonomy, you have situations where (equipment) stops,"
Leifer said. "(With SparkAI), we can clear that false positive and
automatically restart the tractor without having to notify the grower in
real-time." (SparkAI) uses a human to call it, to continue or not. If it's a
real obstacle, we'll send it to the farmer (for his decision). But the idea is
not to distract the farmer and lower his productivity." The grower will get a
"stop" report at the end of a field mission. But the grower will not have to
deal with every stop report.
SparkAI is a New York-based startup that helps robots resolve edge cases in
real-time (yes, the 40,000-pound, autonomous 8R is a sort-of robot). John Deere
was a SparkAI customer for a few years prior to the acquisition.
Dan Miller can be reached at dan.miller@dtn.com
Follow him on Twitter @DMillerPF
(c) Copyright 2023 DTN, LLC. All rights reserved.
