From the earliest flight simulators to the computers now embedded in everything from toasters to cars, we have tried to transport ourselves into a digital space or attempted to bring a digital space into our physical environment. Until now, it has been either-or for most of us. With its entry into spatial computing, Apple’s Vision Pro attempts to do both.
The term “spatial computing” dates back to 1985 and was first used to reference computation that dealt with geospatial information on a large scale[1]. Simon Greenwood, then at the MIT Lab, may have been the first to use the term in the sense we use it today[2]. In his 2003 Master’s Thesis, Greenwood offered the following definition.
“I define spatial computing as human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces. Ideally, these real objects and spaces have prior significance to the user. For instance, a system that allows users to create virtual forms and install them into the actual space surrounding them is spatial computing. [..]”
Greenwood’s spatial computing can be viewed as a response to the phenomenon of our computing experiences separating us from the physical world. Look at anyone replying to an email on their phone while on a dinner date, that’s separation.
Second Life[3], launched in 2003 by Linden Lab under the vision of Philip Rosedale, sought to replicate the richness of the natural world within a digital realm, drawing inspiration from the concept of a Metaverse as portrayed in Neal Stephenson’s novel “Snow Crash” [4]. This pioneering virtual society enabled users to interact via avatars in a 3D environment, requiring the user to install a dedicated viewer to run on the user’s computer.
SecondLife had a digital currency that was convertible to U.S. dollars. Users engaged in fashion design, real estate development, adult entertainment, clubbing, and held concerts. It also had a scripting language and an API supporting communication with external systems. The latter made it possible to develop simple Digital Twins. For example, a lamp in your home in Second Life could be linked to a lamp in your living room in the physical world.
Aside from a complicated user interface and challenges with scaling to handle large numbers of users, a fundamental challenge faced by Second Life was that the separation from physical reality was a feature, not a bug. Enterprises did experiment with holding events, courses, and even job interviews in SecondLife, but for consumers, a big part of the appeal was “escaping” into a separate world. Linden Lab is still in business, and SecondLife has continued to evolve (its users are now experimenting with Generative AI). The separation, however, is here to stay. That is what makes Second Life a Second Life.
There was an earlier effort at Xerox PARC (now part of SRI) to embed computing experiences into physical spaces, addressing the separation phenomenon from the other side. In the late 1980s, PARC began work to embed computing into everyday objects. Led by Mark Weiser, the effort became known as ubiquitous computing [5].
The idea was to build services accessed through physical objects with familiar form factors. These initially included a large screen as a digital whiteboard, a wireless tablet, and a palm-sized computer with a small screen. A related term coined at PARC was calm technology, denoting devices that could remain on the periphery and be as unobtrusive as possible while fulfilling their function[6].
Thirty-plus years later, it was clear that we didn’t quite get the devices PARC researchers imagined and prototyped. We got much more. Laptops became much more popular than desktop computers. We have tablets as general computing devices and phones with more computing power than a Cray. TVs have operating systems and applications. Dedicated computers didn’t go away; they just split into multiple form factors and became more complex, not less.
The vision of “calm technology” was not to be. Computing is “louder” than ever. We interact with various digital spaces through multiple devices. Even though the Internet has made sharing data and linking systems much easier, our devices are usually not linked. Our car, health care information, fridge, entertainment preferences, and travel plans all exist in different digital spaces.
Our digital and physical spaces remain worlds apart. We interact with fellow humans (and pets!) in remote physical spaces through a tiny keyhole — a 2D video stream, text, or a shared virtual whiteboard. In 2024, doctors are still faxing medical records when referring patients to a specialist.
We are not experiencing immersion into remote physical spaces, nor are our appearance and actions projected into the physical space on the other end. Holographic systems have attempted to address this, but these systems are neither portable nor immersive. We have not been able to take our spaces with us.
As one might expect, Apple’s entry into spatial computing comes with a high-end device that easily outpaces anything else available in terms of graphics performance, resolution, fidelity, and user experience. And price: The Vision Pro starts at $3,500 for its basic configuration. From the outset it is clear that Apple aims to provide a full-fledged computing experience based on the spatial computing paradigm.
The Apple Vision Pro runs iPhone and iPad apps. They can be launched and run in 2D windows, placed in the user’s (3D) environment, as seen in the video below. It is quite an experience to take apps and data we have been working with and place them in the physical space around us. Instead of embedding digital capabilities into (representations of) everyday objects, as envisioned at PARC, we are populating our physical space with digital objects.
Video Copyright (c) Apple, 2023-2024. All rights reserved.
Vision Pro users can replace their view of their physical environment with a digital space. Imagine working on an article while overlooking a beach and enjoying a glass of sangria. The sangria would be genuine, but the beach could be a 3D photo or video (with spacial audio) that is so lifelike it is virtually impossible to see it’s not there. The device has a 4K resolution for each eye.
Being able to bring another physical space (through digital representation) to us has a utility far beyond setting the mood. Native apps running on Apple’s new visionOS operating system will take further advantage of spaces. With the native version of Keynote, for example, we could stand on the stage of the Steve Jobs Theater at Apple headquarters and practice giving a presentation and editing the slides as needed. Another environment, a conference room, allowed us to switch between being a viewer and a presenter.
The compelling entertainment experiences (sports, immersive 3D gaming, watching movies in a virtual movie theater) that can be had with the Vision Pro have been covered extensively elsewhere. The implications for the world of work are even more profound. Spatial computing will be a powerful enabler for decentralized, collaborative knowledge work, with applications that take advantage of 3D, immersion, and sharing of spaces.
Here are some examples:
Healthcare: Doctors and medical professionals could use spatial computing for collaborative diagnostics, allowing them to manipulate and examine 3D models of patients’ bodies or specific organs together in real time, despite being in different locations.
Education and Training: Virtual classrooms and labs will allow instructors to interact with 3D models and simulations. This immersive learning environment is especially beneficial for complex subjects like physics, biology, or engineering.
Software and Systems Engineering: Technology architectures can be visualized and worked on by multiple engineers and architects simultaneously. Users will be able to run simulations and also visualize system performance or security issues in real-time.
Architecture and Construction: Teams can collaboratively design and review architectural models in a shared virtual space. This enables real-time changes, walkthroughs, and stakeholder input, improving project outcomes.
Retail: Virtual showrooms can enable customers to explore products in 3D, with sales representatives guiding them through features and options in a shared digital space, enhancing the customer experience.
Manufacturing: Engineers and designers can work together on product designs in a 3D space, conducting simulations and making adjustments on the fly, accelerating the design process and reducing time-to-market.
Automotive: Teams can collaboratively work on vehicle designs, simulations, and testing in a virtual environment. The collaboration could extend to customer services, where sales or support staff guide customers through car features, customization, and troubleshooting in a shared digital space.
Entertainment and Media: Filmmakers and producers can collaborate on scene setups, visual effects, and storyboarding within a spatial computing environment, allowing for a more intuitive and creative workflow.
Real Estate: Virtual property tours where agents and clients can walk through and discuss properties in a 3D space, regardless of their physical location. It can enhance the buying experience and expand market reach.
Urban Planning and Development: Project participants can visualize and manipulate city plans in 3D, simulating environmental impacts, traffic flows, and population dynamics to make informed decisions.
Aerospace: Teams can design, simulate, and test aircraft and spacecraft in collaborative virtual environments, streamlining the development process and enabling more innovative solutions.
Finance and Banking: Spatial computing can revolutionize data visualization and analysis, allowing teams to interact with complex financial models and simulations in three dimensions, improving insight and decision-making.
During the COVID-19 lockdowns, decentralization of work significantly accelerated. Tens of millions of knowledge workers found they could work productively from home. There were drawbacks, however. In-person collaboration was not always fully replicated by its digital equivalent. Some struggled with isolation. Many leaders also found it challenging to keep people engaged when managing remotely. Digital fatigue became a problem as the amount of screen time went up.
Organizations are still adapting and working through these challenges, with an estimated 71% of US knowledge workers working hybrid[7]. Worldwide, it is about 39%. Will spatial computing users fare better in terms of digital fatigue? It’s too soon to tell, but we should have some good data both from the private sector and the military within a couple of years.
Knowledge workers leveraging spatial computing will soon work with AI Copilots in their chosen space. They will be able to gesture and make natural language requests, and AIs will dramatically increase productivity in the application areas mentioned above. The next stage of development will be the widespread use of AI Agents, which will be able to keep track of goals, develop and execute plans to meet these goals and make requests for external services, including other AI Agents. While AI Agents will reside in the cloud, they will communicate with humans using avatars through spatial computing platforms in addition to other devices.
Software has become the medium in which knowledge work is performed. Because information artifacts, spaces, tools, and people are separated, our work experience is often cumbersome. Computing without Boundaries means bringing who and what we need together in a time and place of our choosing. Spatial Computing represents a big step toward this goal. Coupled with AI, Spatial Computing will deliver great advances in collaboration, creativity, and productivity.
References
- Reeve, D. E. 1985. “Computing in the Geography Degree: Limitations and Objectives.” Journal of Geography in Higher Education 9(1):37-44.
- Greenwood, Simon. 2003. “Spatial Computing.” Master’s thesis, Program in Media Arts and Sciences, School of Architecture and Planning, Massachusetts Institute of Technology.
- Linden Lab. 2003-2024. “Second Life”. Retrieved February 10, 2024 (http://www.secondlife.com).
- Stephenson, Neal. 1992. Snow Crash. New York: Bantam Books.
- Weiser, M., Gold, R., & Brown, J. S. 1999. “The Origins of Ubiquitous Computing Research at PARC in the Late 1980s.” IBM Systems Journal 38(4).
- Weiser, M. & Brown, J. S. 1995. “Designing Calm Technology”. Xerox PARC. Retrieved February 10, 2024 (https://calmtech.com/papers/designing-calm-technology).
- Gartner. 2023. “Gartner Forecasts 39% of Global Knowledge Workers Will Work Hybrid by the End of 2023.” Retrieved January 7, 2024 (https://www.gartner.com/en/newsroom/press-releases/2023-03-01-gartner-forecasts-39-percent-of-global-knowledge-workers-will-work-hybrid-by-the-end-of-2023).
