mac os reality
dec 2020
Building the Next Generation AR Operating System
Envisioning the next generation of operating systems designed for AR glasses that will replace traditional computers and 2D monitors.
Problem & OpportunitySolutionMedia PresenceApple GlassesrealityOSProcessStrategyDesignReflection
Intent:
Passion Project
Duration:
10 weeks
Role:
UI/UX
Strategy
Prototyping
Tech:
Figma
Adobe cc
unity
Jump to execution
For almost 50 years, the concept of computers has largely remained the same. We are staring at an operating system on a 2D screen that is controlled via a mouse and keyboard. However, with AR technology and specifically the advent of AR glasses wearables, which are going to be inevitably released in the next few years, there exists the opportunity to fundamentally disrupt and redefine the way we interact with a computer through a spatial operating system and innovation.
What you need to know
This project went viral and received media presence in 20+ countries. It was featured on popular Apple news websites including iMore (~15 million monthly visitors) and CultOfMac (~1 million monthly visitors). MacRumors (~1 million followers on Twitter; 12 million monthly web page visitors) featured this concept in a tweet.
The Problem
Computers in their current form fail to adapt to the ever-growing demands for productivity of employees currently and in the future.
In roughly 50 years, the general concept of computers has largely remained the same. Apple II, which was released in April 1977, was the first highly successful mass-produced microcomputer. About 50 years later in 2020, Apple released an updated version of the iMac. Whereas the memory and storage have had exponential growth over the years, the basic functionality has not changed as much. Both are still controlled via mouse and keyboard and display content on a 2D screen.
However, whereas the general concept of computers has remained the same, the requirements and challenges of workplaces are steadily increasing. Those requirements make it more and more challenging for employees to make the best of use of their time and stay productive while facing increasing pressure from employers. On top of that, in 2020 and beyond, Covid has forced many companies to operate on a remote basis entirely and many companies are moving online permanently. The reason, why many companies chose to do so is that there are many benefits of remote work. However, despite those benefits, employers and employees struggle to keep up with the challenges that come with it, many of which lead back to the computer itself. Whereas the demands of workplaces and productivity change rapidly, the tech industry fails to keep up with those demands. Some of the problems include a fragmented team communication, friction when trying to multitask and not having access to hardware such as multiple screens or generally the logistical challenges that come with the distribution of the latter.
The opportunity
The remote work economy is booming and many companies are moving online permanently.
People tend to increasingly work in different environments and workplaces are required to be more flexible. A Gartner, Inc. survey of 317 CFOs and Finance leaders on March 30, 2020* revealed that 74% will move at least 5% of their previously on-site workforce to permanently remote positions post-COVID 19. In addition, the number of people telecommuting in the U.S. increased 159% over the past 12 years and is expected to increase further in the coming years. Lastly, employers can save over $11,000 per half-time telecommuter per year. Across the existing work-at-home population, that potentially adds up to nearly $44 billion in savings. If the telecommuting workforce expanded to include those who could and wanted to work from home, the potential employer savings could approach $690 million a year.
Business value
Laying the foundation for the next era of Macs which will leverage AR glasses instead of 2D screens.
With smartphone innovation consistently decreasing and Augmented Reality glasses from all major tech companies being on the verge of release in the next 3-5 years, we have to ask ourselves where do computers fit in there? Thankfully, the answer is they will perfectly fit in the ecosystem. However, computers will no longer act as a standalone device with a screen and instead solely provide the computational power for professional tasks while leveraging AR glasses as the display medium. This might be the basic concept of future Macs given that AR glasses eventually reach mainstream adoption somewhere close to smartphones. Once this is achieved, there are nearly unlimited possibilities and many opportunities for an increase in productivity alongside many other benefits to consumers.
For more information about consumer AR glasses (Apple Glasses), I created a concept in 2018 detailing some key features rumored at the time.
Introducing mac os reality
A spatial operating system from Apple for powerful productivity in 3D space intended for AR glasses.
macOS reality is a productivity-focused operating system that leverages Augmented Reality in the form of AR glasses wearables (Apple Glasses). It is a separate operating system from the OS that runs natively on Apple Glasses (realityOS) and requires separate hardware for computational power. Unlike realityOS, which is a mobile-focused AR operating system like iOS, macOS reality is focused on professional tasks and productivity analogous to macOS for traditional Macs.
💡
The entire concept has been thoroughly prototyped and tested, in some cases through approximation in VR due to lack of proper AR glasses hardware in 2020. Hence, some of the demonstrations are in a VR environment. The VR content is 1:1 analogous to AR.
Imagine an operating system where the world is your display.
macOS reality defines a new era of productivity through sophisticated use of Augmented Reality, making optimal use of your physical environment to keep you more productive, e.g. through unlimited screen space or simply the convenience of widgets, among many others.
Powerful widgets for powerful productivity.
Widgets placed on your desktop allow for easy access to important and useful information without requiring a lot of physical space. To reduce physical space even further, widget groups combine multiple widgets for even more efficient use of space. You can conveniently interact with widgets by poking with your index finger.
Unlimited window space intelligently used.
Desktop collections are presets of your commonly used window arrangements. With a tap of a button you can switch between collections on the fly enabling you to truly have unlimited virtual window space. The function strip above your keyboard gives access to useful quick actions based on the application that you are currently focused on.
3D avatars for more productive meetings or simply less commitment.
Facetime is more immersive than ever before. Even when you are not physically at the same location, a 3D hologram avatar created using body scanning of your iPhone or online using just a portrait picture enables you to interact with people as if they were next to you. In addition, 3D avatars can also be used if you simply don't want to commit to showing your face on camera while still having the benefits of facial expressions and feeling closer together.
The most versatile macOS ever made.
Augmented Reality enables macOS reality to be the most versatile operating system to date. Requiring simply AR glasses and computational power from hardware, it is equally functional for your home, office, or even on the go. Whereas previously, you used a MacBook with a single screen while commuting or traveling, now you virtually carry around the equivalent of 10+ monitors just by using Apple Glasses.
An even more functional Finder, now with depth.
Finder shows all your files more sophisticated than ever before. By introducing Smart Grid, an intelligent way to arrange your grid that places all folders at the top and increases the cell size of certain elements (e.g. videos) based on your usage and demands, your files are more organized than ever before. In addition, Finder now supports viewing 3D models (.fbx, .obj, .dae, etc.) with depth making it even more convenient to check out assets for the game you are developing, physical products you are designing, and more.
All of your apps in one place.
Analogous to launch pad on traditional 2D Mac operating systems, macOS reality also features a dedicated accessible place to show all your installed apps. From here you can create folders, open and delete applications as well as search for any app on the fly.
A new era of Safari.
Safari lives on in 3D and it is more powerful than ever. Now you have enough real estate to view multiple websites next to each other, resize them however you imagine and make it more convenient to browse through curved windows. In addition to your virtual trackpad or a traditional mouse, you can use your hands as an input device to scroll through your favorite pages. Lastly, Safari in macOS reality now supports more sophisticated true AR experiences that you can directly view through your glasses.
The power of unlimited monitors.
macOS reality allows you to freely position, change the size, and even curve any windows in 3D space, enabling you to optimally make use of your space just the way you need it to to make your workflow as efficient as possible. In combination with widgets and desktop collections, you truly always have the most important content you care about in front of your eyes.
The most important shortcuts ready whenever you need them.
The power of Augmented Reality goes beyond augmenting the world in front of you. Optional shortcuts projected onto your wrist give you a very accessible option for your most used task flows that does not use up space on your desk in the form of a widget.
A diverse set of input types made for everyone in mind.
macOS reality offers wide ranging options for input that have many use cases in mind. From physical accessories to more portable virtual UI as well as mid-air hand input to less fatiguing rested hand input, you can choose how to be productive in the most comfortable way possible. For keyboard input, you can choose between a physical keyboard accessory or a virtual keyboard that can be mapped onto a flat surface for natural haptic feedback.
For pointer input, you can choose to use input via your hand in mid-air (unrested) and by using a virtual trackpad that is mapped onto a flat surface besides a traditional computer mouse. For the best experience, both rested and unrested input options can be combined as both of them have advantages in certain situations, e.g. unrested hand input allows for more quickly repositioning windows.
A dedicated onboarding flow familiarizes the user with all available input options. For hand input, you can trigger a primary click by pinching your thumb and index finger whereas a secondary click is triggered by pinching your thumb and middle finger. The virtual trackpad works just like your trackpad on traditional Mac computers.
Automatically scan your work environment.
Scanning your physical environment is a crucial step to make the most out of macOS reality when used at home or in the office. It enables the system to be aware of the size and location of your desk as well as any obstacles that lay around to avoid intersection with UI elements.
Reflection
Out of all my passion projects, this has got to be the one I am most passionate about. It was a challenging project requiring a wide range of skills so executing it just like I envisioned it in my concepts is great reinforcement of my overall level of skill. In addition, creating something truly disruptive that has not been done before are my favorite problem spaces to tackle.
Team
Dominik Hofacker
Experience Designer