Building the Foundation: Teaching AI to See

 Dr. Steven Mirsky of USDA's Agricultural Research Service (USDA-ARS) explains how the digital infrastructure behind artificial intelligence is being built today to power the future of agriculture. From computer vision and image repositories to BenchBot and the Digital Agricultural Systems Hub (DASH), he shares how researchers are creating the datasets, tools, and standards needed to train AI systems that can detect pests, diseases, crop stress, and other field conditions. Steven also discusses why collaboration across government, academia, and industry is essential to moving these innovations from research to real-world impact. 

Transcript

Steven Mirsky

[00:00:00] AgTech 360 discusses breakthrough technologies that are impacting growers, businesses, and consumers. Hear from industry and academic experts about what's on the horizon

Welcome back to, AgTech 360. In this episode, Dr. Steven Mirsky of USDA-ARS joins us to discuss how innovations in robotics, sensing technologies, and digital agriculture are helping build the foundation for the future of AI in agriculture. Thanks for joining me, Steve.

Good to be here with you, Adrian.

Yeah, so let's start off just asking a little bit about your background and what led you to focus on precision agriculture and digital agriculture.

Yeah, absolutely. I'm, uh, classically trained as a very applied scientist, so, you know, got a, a bachelor's degree in agroecology, sort of more ecological sustainable ag thinking, and did a, [00:01:00] a master's degree in soil fertility, a PhD in agronomy with a heavy emphasis on weed management.

And that's what I did for a large swath of the beginning of my career, was very applied agronomic research on how to optimize crop performance while improving and protecting our soils. And I think where the pivot for me became that, um, we started just continuously having the same papers, that we would do these two, three-year studies that was the wettest year in recorded history or was the driest year in recorded history, and the results were inconclusive because, you know, we had essentially these really vastly different soil or moisture or temperature, you know, dynamics that i- impacted the results.

And so I was like, how can we thread together, you know, how, you know, temperature, precipitation, soils, and management all sort of intersect to drive function in the landscape to better make recommendations to our growers? And that's, [00:02:00] that's precision agriculture, right? That's how you get very site specific in your recommendations to the farmer, and that's what really sort of inspired me to keep going in that direction.

Great. So we have listeners all, all around the world, and, some of them may not even be familiar with what USDA-ARS actually is. So can you just quickly explain what your organization is all about?

Yeah. Uh, ARS is the Agricultural Research Service, so this is the USDA's in-house research arm. You know, we've got, like, thousands of scientists across the US.

Uh, they're, you know, represented in all of the different types of production systems, cropping systems, soil types across the US doing, public sector research and R&D to really do problem-solving for our farmers and ranchers across the US.

Now, you've been, an advocate for using technology to advance agriculture.

But big picture, you know, where do you see the greatest opportunities for AI [00:03:00] and digital tools to improve farming sustainability and efficiency actually at the current time?

Yeah. Uh, um, computer vision without any hesitation. I mean, there's tons of tech coming online. There's lots of different roles for AI.

But the ability to see and do is just transformative. You know, cameras are becoming the new human eyes, if you will, and they let us see more often and turn what we see into action and decision-making, right? The near-term wins are concrete, right? Addressing labor, using inputs far more efficiently, getting very site-specific around our pest management, disease management.

We've seen this challenge to get precision ag off the ground for decades now, and computer vision is really what's gonna unlock precision agriculture.

So you really see this as a way of helping with some of the, workforce challenges and workforce issues that we hear from many growers and from many parts of the country?

Oh, [00:04:00] absolutely. I mean, it's not, this is not a new thing, right? The, the farm workforce has been declining for over a century, so this isn't a new problem, but it is one that technology can help a lot with, right? If, if cameras are these, these new human eyes, if you will, computer vision lets one person effectively cover, what used to take many.

You know, if we think back historically, we had folks who were weeding, you know, uh, who were, like, hand-weeding weedier parts of a field, right? They can target that, and that was, like, the, the original precision agriculture. It was just lots of human eyes out there seeing and doing. Now, the cameras will be that force multiplier, if you will, right?

So now we're being able to give the, reduced number of farmers that are producing all of our food needs, a better ability to capture and cover the totality of what is going on in their fields, understand where disease or insects or weed dynamics are moving into those fields, and better, you know, manage them using site-specific [00:05:00] management.

So I, I see really computer vision as providing a core resource, uh, and helping, you know, these farmers, leverage computer vision to improve their management and have greater autonomy.

If we're zooming out a little bit more generally, how are USDA-ARS incorporating AI into its research?

I'm assuming that this is a, a huge focus for the, for the agency.

Yeah. It would be almost, uh, better to ask that, uh, question, Adrian, where are we not- Mm-hmm ... bringing AI into, our research pipelines, right? So ARS is integrating AI across the whole research pipeline from, how we collect field data, to how we analyze it, to how we deliver actionable guidance to the producers.

Our scientists are using machine learning to interpret imagery, model pests and disease spread to accelerate crop and animal improvement programs. Uh, and, and it's really accelerating the pace at which we do these things. AI is also not just being used directly for the science, but it's, [00:06:00] it's part of our literature reviews, structuring and formatting reports, the administrative tasks, you name it, right? AI is embedded into all of these workflows, and it's really helping to accelerate the role that we can play in helping, you know, meet the needs of our farmers and ranchers.

So I wanted to get now into some of the things that you're personally and directly involved in, and one of them, I understand, is an initiative called DASH, which is acronym for the Digital Agricultural Systems Hub. So what, what is DASH, and what's it all about?

Yeah, uh, DASH is here to enterprise computer vision and AI across American agriculture and deliver, the real world tools to our farmers, breeders, and researchers faster. We have a lot of folks developing different computer vision solutions, uh, and, and everybody's collecting data in different formats.

They're dif- using different camera systems. And so we're trying to curate a way that we could create, uh, a lot of gravity around [00:07:00] standards for a plant agricultural image repository and make that publicly available. A- and the reason we're so focused on, on really enterprising this at scale is that, computer vision is really a million and one problems in agriculture, right?

And so we will peel off, a number of these large targets, uh, by private industry, but, but there's so many of them out there that it really r- requires this public sector effort to sort of create standards and nationalize this digitizing of agriculture, if you will. So DASH is sort of standing this up on a scalable infrastructure built on SCINet, the USDA's high-performance cluster.

Data and model development is, um the development is through annotated images that are in the repository that we'll use to train AI across all of these different species for not just the detection of the plants, but, uh, are they under stress, what kind of stress, and then using that to get actionable, whether it's [00:08:00] in our breeding programs or informing, um, farmers through decision support tools.

Uh, currently, we have a number of pipelines from both this end-to-end solution of collecting the training data and then deploying these models through a modular camera system. We have this training data collection pipelines happening on many locations throughout the US, and then we have these model deployments happening currently in breeding programs, and plant science research programs, as well as on farms with growers, uh, mapping anywhere from, cover crop biomass quality and quantity to, to weed mapping, to disease mapping, and so on.

And, and our goal with DASH is really to help create sort of a national network working together to sort of build this common good, if you will.

The North Carolina Plant Sciences Initiative impacts lives through innovative applications and discoveries. By leveraging cutting-edge research and technology, we address global challenges related to agriculture, sustainability, and human health.

Can you talk a little [00:09:00] bit more about the data sets that you're you're creating and deploying and using . What are these data sets?

Yeah. So, uh, when we're talking about computer vision, in this case, we're predominantly focused on RGB. These are just, you know, images collected with, you know, uh, um, very specific camera systems for our training data that we're using to develop models, but they can be deployed on a wide range of low-cost camera systems.

And why RGB is so powerful is the compute capabilities of today really allows us to, capture the complexity of agriculture and label these images around that and drive actions really rapidly with these low-cost systems. So we're able to use imagery to train on, is the disease present, and then we're able to map across a field how quickly a field is, having that disease distributed across the whole field.

We can come in and detect early emergence of weeds and inform where we might wanna [00:10:00] target management. So there's a heavy focus on pest complexes and disease and water stress, but it really can fit the wide range of, of needs in agriculture. I mean, just as simple as, is a plant flowering and when does it start to flower?

A- and these datasets and these repositories of images, th- these are open for anyone to access and if that's the case, how are researchers kind of using these and, and how is that enhancing collaboration between different groups?

Yeah. So currently the, uh, what we call AGIR, which is the National Agricultural Plant Image Repository, is stored on the USDA's high performance cluster called SCINet, and that resource is a public good to all of our ARS scientists, and is also a resource to a lot of our university partners that we're collaborating on projects with, particularly, uh, in these DASH partnerships.

Over time, we expect that to become even a more public resource and have a cloud [00:11:00] instance where this could become accessible to everyone, to be able to build, uh, models and solutions. But yes, the core point here is that this is a, an, an open source, open access resource.

Okay, great. So let's switch to another area now, which is robotics, which people love to talk about robots and the i- the image of, robots dashing across the fields, um, that we might see in the future.

So what new tech have you created in that space, and, and what is your kind of ma- major research interest when it comes to robotics?

Yeah. A-as I've mentioned, uh, the plant image repository, our focus is on how can we automate the collection of training data that can be used to convert it into, um, computer vision models, and then deploy that scale on farm or in research programs.

And so we focus a lot on developing a robotic platform called BenchBot that automates the image collection pipelines, uh, [00:12:00] around plants in what, what we would refer to as like a semi field controlled environment. So these are often potted plants. Uh, we also supplement, th-these image repos and other types of, protocols, but the main robotic system that is just being scaled across the US is this BenchBot robotic system that really automates image capture.

Now obviously the development of models, uh, from computer vision from this could be used on a wide range of robotic platforms, and that's really the space that we're trying to seed. We see this transition to autonomy. Robotic platforms used in agriculture are largely limited and bottlenecked by computer vision training data.

So for us, how unpacking that training data and make that broadly available is, if you will, uh, actualizing this, robotic, revolution.

And I guess, the end users of a lot of these technologies will be growers, will be farmers across the US. [00:13:00] What are you, what are you hearing from them and what, what do you feel their acceptance and their interest in these types of new approaches is gonna be?

Yeah, it's a great question. , we, we certainly hear a, a lot of the folks in the robotics space are, are using computer vision in, in the various different ways to, um, do precision agriculture and serve, the different industries that they're in. I think that what we hear a lot is that they are bottlenecked by these training data sets.

And so we hear a lot that when their challenge or constrained in a specific application or a specific crop, or maybe it's a specific market that they were targeting, doesn't end out, playing out as they had hoped, and they're trying to pivot to a new market. Having to generate all of that imagery and that training data to go into a new crop is incredibly costly and challenging for them.

So it's not easy for these [00:14:00] startups to pivot rapidly to new crops and new systems, e- especially if they're having challenges or in case they're having successes in a given crop and they want to scale. So I, I think we're hearing a lot from them that they're excited about these digital assets that we're building that can become part of the resources that they use to expand into new markets.

Can we do the crystal ball question now for you, Steve? Sure. So, I mean, just looking ahead, I mean, , you've described what, Dash is doing, um, bringing together all of these different groups, whether they be researchers or, or industry groups, or wherever, growers perhaps.

Why are these kinds of interdisciplinary partnerships essential for advancing digital agriculture and AI-enabled farming? And looking ahead, what do you think agriculture's gonna be transformed into, you know, when, when all these different types of tools eventually get deployed in the field? Hmm. Big question.

Okay. So, yeah, big question. I, I'll [00:15:00] try to answer the two parts of that question. I mean, so the first part is, like, you know, this is very different than studying a single organism. I know we think of the human organism as, as very complex, right? But unpacking, what it takes to build computer vision for, doing radiology in humans is targeting a single organism, right?

Where in agriculture we've got hundreds or thousands of organisms in the system. We've got thousands of different soils and different types of precipitation and temperature regimes. So the complicated space that agriculture is, and the problem that it presents, is not gonna be tackled by a couple of key players.

It's such a distributed problem to be able to digitize agriculture and help us move towards this path of autonomy and robotic systems in agriculture. So it's gonna require all the experts working together, and that's why I'm really excited about what Dash is doing, right? It's like we're not trying to be the experts across the country in every system.

We're just going to the experts, helping to [00:16:00] bring them some of the tools that they're then helping to generate the training materials and these assets for those systems. So instead of being an expert at every type of cropping system and environmental condition, the experts come together, work with us to build that asset.

And so that's why I think it's so critical that we're leveraging the public sector that's distributed across the country to build these resources. And then where does this go? Well, I mean, as we really unlock computer vision and what it can be for precision agriculture as a whole, we're going to see more and more abilities with early detection around pests and disease outbreaks.

We're gonna be able to have the ability to actuate in real to near real time. These are no longer gonna be just- Research grade solutions, these are gonna be broadly applicable to growers. And so we're gonna be able to have a much more rapid [00:17:00] and nimble production systems.

And these costs for these systems, because it's leveraging computer vision, are gonna be very accessible. It's not gonna be, do you have this really high price piece of equipment and you're the only game in town that has that? It's, it's gonna be how many units do you have of lower cost robotic and sensing systems that can scale up to the size of your farm? If you're a small farmer, you'll need less units.

If you're a large farmer, you'll need more units. But I think it'll be very scale neutral technologies that hit the farmer where they are based on their operating systems.

Steve, thank you so much for joining me and sharing some of your insights on the technologies that are laying the foundation for the next era of agricultural innovation.

It's been a fascinating conversation. Thank you. Yeah. Thank you, Adrian. I appreciate it.

AgTech 360 shares relevant news and breakthroughs with audiences across the globe. Stay connected and [00:18:00] join the conversation by following NCPSI on social media