We foolishly think AI for Earth is yet another try to deliver the same vision geospatial has promised for decades. The reality is much harsher.

This was the provocative prompt I wanted to articulate on my plenary at SatSummit Lisbon. That room, and the network I can reach here, has some of the main shapers and doers of AI for Earth, both those that fully engage and build it, and also those that refuse to and by omission let others build without the experience and perspective they have. I strongly believe this window of opportunity to shape AI for Earth is closing in the next 12-24 months, so I will lean here with strong opinions loosely held. Do not aim to be comprehensive, but provocative, and encourage you to disagree, rebut, question and comment on this. The goal is the conversation to understand what’s happening, or not happening and should.

Plenary of the SatSummit Lisbon 2024. Picture by Atelier Obscura

I’m taking an extremely pragmatic definition of AI, basically “following the money”.  We are spending orders of magnitude more funds to explore one AI architecture in particular “Transformers” (the T in ChatGPT), and for one task: predict the next word in a text sentence. By a long shot. I’m not saying this is the most interesting or even strategic choice. That debate is less interesting to me. The world has chosen. More specifically a group of VC and big tech companies are pouring billions into it, because they see an outsized opportunity in getting the upside first. And most of them not only use the architecture, but also have the same training data (all of the Internet). On same NVIDIA chips. For the same goal. And it’s working: benchmark after benchmark, domain after domain, we are blowing past what expert deemed impossible few months ago. So there is little surprise that we are learning a lot about how to squeeze practical innovations from Transformers, but also how to leverage other domains, like images or audio, or self driving or drug discovery, or ... (inexplicably, not geo data). Most consecuencial, the extremely high barrier of budgets to play in this race also creates a whole long tail of underdogs that prove able to meet or even exceed most expensive models in ~6 months. They leverage a whole set of hacks, tricks and entire new insights like nokens, quantization, LoRA, Matrioskas, and many more every week,  And still not much has come to or from AI for Earth. There’s literally a whole world of data, and questions, awaiting.

Simply put, the problem is that it’s taking decades to arrive to where we promised to reach years ago. If you look at the rationale for any of the commercial EO satellites, or public constellations, for the last couple of decades, they mention a similar vision for EO and Earth awareness: Precision agriculture anywhere, poverty mapping, rapid detection of floods and drought, of climate change events, ... and while this vision is ever closer, we are still far away. Let’s take one blunt metric of the underwhelming impact EO has had. We don’t really do reports on the impact of investing on weather data, it's obvious. But we do for EO. It is not as obvious public investment to the public as we might think. Another metric, let’s look at the share price of virtually any EO commercial company. 80% lost since their SPAC is a good number. Wall St seems much less convinced than VCs that commercial EO is a good business. Both with public and venture funding we have validated the vision for EO, derisked the tech and most of the CAPEX, but we have not validated a sustainable operating structure, or even sold the perception of that vision to make it obvious. (Worth mentioning at this point that I’m very bullish, excited and defendant of both public and private investment on EO, but also that we clearly have not found the right mix yet.)

Long can be argued why that is the case. In my opinion, one that is less spoken is the (US) military roots of EO. The ecosystems to deliver defense products are widely different to those to deliver civilian commercial markets or public services. It’s the strong hand in defense to maintain critical information superiority, the price of national interests, the specs and procurement processes, the incentive to keep a closed markets of suppliers. When EO evolved to see beyond defense and into the public/commercial benefits, the metamorphosis is much more fundamental than just expanding the market. It’s processes, expectations, subsidies, customer skills, unit economics, business models, ... I think we never really acknowledged this (de)militarizaion of EO, and it would help to be more explicit of the differences so we can deliver on both properly (like cloud providerds do).

As we talk about AI for Earth, it is worth talking about the cost of AI on Earth. That is, the impact of chatGPT and other AI models on Earth: the resources used to train them, and use them. It is estimated that training just one model like GPT3 (latest models probably 100x on each dimension) took the 300MWh (same as 120 US houses in a year), 700,000 liters of water (what a thousand people drink in a year), and 500 tons of CO2e (500 long-haul flights). And then we also consume a lot using them since we need to run every input through the billions of parameters, so one query on GPT3 takes 10x the resources than a “normal” search.

Back in 2020, right before this AI revolution, most big tech and increasingly investors, had ambitious and matured sustainability roadmaps, aiming for carbon net neutral, water positive and zero waste. They all since went off the rails. The downsides of falling behind the AI race were deemed unaffordable, and the cost of breaking their sustainability projections a temporal correction. I helped deliver Microsoft’s strategy, and I worked at the World Bank where I saw the impact of climate change on most vulnerable people; and I can see why doing AI is not an obvious decision on either side.

To complicate the narrative, models like BLOOM demonstrated one can reduce 90% of resource consumption carefully choosing location of the data center (it was trained using nuclear energy), Sustainable AI is not impossible and is not easy. Moreover, it can be easily seen that these models are orders of magnitude faster and cheaper to get a better answer for what you look for, which reduces the overall resource use. This is the main argument of some papers like this one. All in all, a complex equation, worth presenting here, but only part of the state of play of AI for Earth. 

Moreover, the main argument around the environmental impact of AI for Earth should not be the above, but the final effect. That is, we must also measure the cost of NOT doing AI for Earth. More specifically, we see the amazing results in other domains, so we should consider the effect of realizing the AI for Earth vision. Then, the costs of not doing it are huge. It’s the cost of not having precision agriculture, of not detecting deforestation fast enough, cheap enough to drive timely interventions. This is not just sunk costs, since we are already using the resources for AI development, at a fraction of the R&D costs when we lean on their discoveries. Moreover, the opportunity costs are also very small, since AI for Earth seems in most cases an additional budget, if not additional skills.  

In my opinion the most strategic starting point for AI for Earth is thus the overflow of tools, skills, courses and even models coming from LLM AI or vision AI. That’s why I limit this conversation to Vision Transformers. Maybe also UI and UX. Does AI4E need to be a chat, or a map? Both? I do not know, but I do know that either option is a carry over, so we might still need to do the homework of design thinking or removing our biases.

When pulling from this strategy of overflowing from LLMs, I find it helpful to think of their phases of maturity. For LLMs, this last wave of AI seems to have gone through three phases:

Reflecting on the above, it seems quite obvious to me that in AI for EO we are still stuck on tech validation, getting comfortable with AI approaches, squeezing more value from models. Specially CNNs, we love CNNs so much so that most AI for Earth is actually CNNs (CNN are also unable to scale and abstract as transformers do). Stuck on tech validation, unable to move to product. Or even envision it. I think we love our technical “geo ways” and geo complexity too much. Or got too used to it to assume its unavoidable. If we want to give AI for EO a fair chance, we need to explore what AI for EO can be, unburdened by what has been (paraphrasing Kamala Harris).

As we reach that conclusion, it seems inevitable to ask ourselves if bringing AI to geo is the wrong way around. I remember the -probably apocryphal- anecdote that in F1 they needed faster and faster pit stops, to refuel the car and change tires. They kept training mechanics for better physical performance, until they realized that it was far more effective to take peak physical performers and teach them mechanics. 

Should we bring AI to geo experts? Or should we bring geo to AI experts? We, geo people, at least need to name or even unlearn our biases. Maybe we need to unlearn NDVIs (what!).... In an AI world based on self-supervised semantics, pixel math might not make that much sense.

Of course the right answer is in the middle... and sadly in reality we rarely see these teams. Those who use NDVI to predict crop yields will strongly disagree, and those using Random Forest regression with embeddings for the same goal will never even hear of NDVIs. Orgs are usually heavy on skills and experience on one side only. Building a true mix of tools, skills and trust might be the winning strategy for AI for Earth.

On my plenary at satsummit, I closed it with the revutal of my first sentence:

AI for Earth is probably not another try for geo people to deliver the EO vision, ... This is AI’s turn to deliver EO’s vision.

---

Before the summit, I sent this AI for EO survey to my network to share. Despite giving them basically 48 hours, I got 70 responses. Neither the sampling nor the response makes this statistically significant, but it does offer anecdotal evidence to the state of AI. After all, responders include people working at the most important public, private, academic and international institutions you might imagine. (Notably, I did see a significant under-representation of end-users and non-international public sector).

Some key results:

Trying to summarize even more: AI for EO is not hype, but real value or even perception of what’s happening is ahead of real traction (and LLM based). Budget and skill seem to be the biggest blocker. We need better products more than better tech, more collaboration, and a focus on real-world problems. AI for EO has no clear winners or even leaders. Yet.

> AI for Earth needs product people.

My plenary talk was meant to provide a State of Play, to help us ask ourselves the right question; not necessarily jump into answers. It was opening the conference, not closing it. But talk is also cheap after all. Last year, based on a fairly similar attitude as described here, Dan and I fundraised and created a non-profit, Clay, to develop a well-funded open-data open-source open-license model. Over the last 12 months we’ve made significant progress with a couple of versions, and increasing attention, participation and feedback from key players in this space. We are extremely grateful to our partners, supporters and users.

And, as we shared at the conference, we just released a new version of the mode, Clay v1.5. Based on feedback, we basically kept Clay v1.0 but expanded the training data to include MODIS, made the model bigger and significantly trained it MUCH longer. You can take a look at the release notes here, and download the model checkpoint.

Over the last 12 months, we also learnt that embeddings might be much more central than models themselves, and that embeddings might be much closer to the product that models. After all, embeddings contain most of the undifferentiated compute for a very wide range of tasks. So, to ease this exploration of product vision, we decided to create embeddings for the entire NAIP dataset, roughly a decade of 1 meter resolution over the entire USA, with a CC-by license. Embeddings are at 225x225 pixels, so we make roughly 8 billion of them, all hosted as geoparquet openly on source.coop. 

I do use interchange AI for EO, geoAI, AI for Earth, and they don’t really mean the same, but if we take a lens of impact, most people understand it to roughly mean the same. But in fact, I mostly speak of satellites looking at land. We do have a whole atmosphere (where most of the weather and climate events live), and ocean (most of climate energy and biosphere by volume). So let’s spend a bit of time talking about it.

When it comes to “AI for Weather and Climate” the story is quite clear and the status more mature, in my opinion. To bluntly summarize, Pangu weather was an AI weather forecast model made by mostly chip engineers. They took ERA5, a painfully made coherent data reconstruction, global and historical, of weather variables. This was perfect to train AI with, with the self-supervised goal of predicting the next variables in time. The result blew several orders of magnitude faster and cheaper; and better predictions past the capacity of long-time and trusted models by decades of climate experts. I’m told this threw quite a few “C-level emergency meetings” at places like NOAA/EUMETSAT/METOffice/ECMWF to assess and react. Meanwhile even better models came out every few months, with Microsoft Aurora probably the best model as of today; and fully open data, open source, open weights. In terms of strategy, I think ECMWF took and implemented the most robust roadmap of AI for Weather and climate.

In Oceans, they do also have to some degree the kind of reconstruction that AI models can use, but there has not been much public progress or commitment to make Ocean transformers. I’m less clear why that is the case, and I’m currently helping some of the leaders in this space to move forward.

Some of the Clay v1.5 model core team, on the day we released it, sunset in Lisbon. From left: Bruno, Soumya and David.