Augment Nation preview: Chapter 1 / Part 1

As I write this in the early 2020s, there is a lot of speculation about what will come after the smartphone. When I was working at Apple in 2006, the year before the iPhone was announced, no one (including me) knew that the world was about to experience a major paradigm shift. Even after its release, the impact that iPhone and other smartphones were going to have was not immediately obvious. That was almost 15 years ago. We are due for another such shift.

Rather than attempting to predict when and how AR will break out, or to debate whether it will or not (although you can imagine which side of that debate I’d fall on!), I am more interested in thinking about what the world will be like after mass adoption has already happened. My goal is to stretch our imaginations, to create something that will read like science fiction, painting a vivid picture of a future where advanced AR is as ubiquitous as smartphones are today. If we skip ahead to that time, and suspend all other disbelief along the way, what will the world be like when we get there?

In order to think about it holistically, I want to start by actually not treating the onset of AR as some specific moment that is going to happen in the future. Instead, try to imagine AR as something much bigger and more fundamental to humanity. It already exists, it is just becoming more fully realized over time.

Augmented reality in its broadest sense is best understood not as a specific type of technology, but as a human activity, happening along the continuum of time. On one side, you have the natural world as our earliest ancestors found it, and on the other, a completely virtual worldscape à la The Matrix. And between the two, many stops along the way. Roughly, that continuum looks like this:

Natural Reality →
Augmented Physical Reality →
Augmented Software Reality →
Virtual Reality

Humans have been augmenting our reality for a very long time. Most of those changes have been physical (think: tools, clothes, buildings), but in the last few decades we’ve seen many of our physical artifacts moving into the realm of software.

A calculator is a simple example of this progression. Presumably, early humans were doing math by just using their minds. Then we started etching marks on the wall and arranging stones to store values, changing the physical world around us. Then we built more substantive tools like the abacus. In the mid 20th century came the first electronic calculator, a physical device running basic software (or perhaps more accurately, firmware) with the results printed on paper. Then came small portable calculators with digital screens. But today, most of us use a calculator app on our phone or computer, which is now purely software with no dedicated physical form at all. Or if using a voice assistant like Siri or Alexa, you can just ask, “What is 2+2?” and the software will deliver the answer to your ears. There is nothing to even see or touch.

“Software is eating the world” as Marc Andreessen, the godfather of the web browser (among many other things), wrote in 2011.

We are not at the end of this progression. It likely never ends; we just keep moving along the continuum. As you keep walking down the road, things become less and less physical and more and more integrated.

Now, I know that when we talk about AR in conversation, we are typically not talking about it in this broad sense. When someone says “AR” they are almost certainly not talking about early humans physically augmenting the natural world with sticks and stones. But I would challenge you to start this journey by thinking about it as being part of this larger continuum of altering the world around us in all different ways.

People like to discuss AR* and its cousin VR* as if they are somehow two possible outcomes for where this is all going, as if it’s going to be one or the other. Rather, AR and VR are better represented as stops along the same path. If you are looking out at the world with 0% digital overlay, seeing the physical world as it is, we call that physical reality. If you go all the way to 100% digital, where your field of vision is completely obstructed by software, that’s what we call VR. Everything in between (basically from 1% to 99% digital) is AR.

Once something is in the realm of software, it tends to be a lot easier and cheaper to build and so moves a lot faster. From where we sit today in the first half of the 21st century, we are witnessing a massive transformation of our world into a composite experience that combines the natural world, our physical additions to it, and now a software layer on top.

You, and me, and everyone else walking around with a smartphone in front of our faces– we are using what we might call a minimally viable product (MVP) version of what AR will eventually be like. We are living in AR right now, it’s just not very good yet. Think of video games in the 80s or the web in the 1990s.

Looking back on today, I believe we’ll talk about this time as those couple of weird decades where everyone had an AR screen that was just a few square inches of their visual field a.k.a. what we currently call a smartphone. We will recall how manual of a process it used to be: taking it out from our pocket, unlocking it, opening a specific app, tapping and pinching and typing to get what we are looking for.

In the future we’ll have a more sophisticated, more mature version of AR that encompasses our entire visual field (among our other senses), in the form of glasses, contact lenses, or brain implants. Rather than tapping buttons on a screen, the software will respond and interface directly with the world around us, rather than needing us to act as the middlemen/women.

The little pieces of glass we hold up with our hands today will seem like a trite sort of proto-AR. But make no mistake, we’re already on this train, and it has already left the station. AR is not something that is going to happen in the future; it is happening now.

*I want to state upfront that when these technologies finally find their way to the mainstream, it is possible that it will be under some new name. There are already other synonyms for AR in circulation like mixed reality (MR) or extended reality (XR). Don’t get too attached to the names, just concentrate on the concepts.