Category: Mixed Reality

Apple’s WWDC 2023 Announcements – Some Thoughts On Potential Healthcare Applications

hjsoch1 Comment

“This morning’s announcements, deep integration with hardware software and services, it’s something only Apple can do. Today we’re going to make some of our biggest announcements at WWDC. As well as introducing some exciting new products.”

Tim Cook, CEO, Apple
Image Credit: Apple Inc.

Over the past two decades, Apple Inc. has released many industry-shifting products: the iPod in 2001, the iPhone in 2007, the iPad in 2010, and the Apple Watch in 2015. Yesterday the company hoped to do it again — with its first mixed-reality headset. While no product will ever live up to the iPhone, the headset has the potential to usher in a new era: It could kick off the shift to a different interface that upends how people work, play games, and entertain themselves.

The headset announcement was part of Apple’s keynote address at its Worldwide Developers Conference, and it was a long presentation — more than two hours — to jam everything in. Beyond the headset, Apple covered various changes to its software platforms, and the company rolled out other new hardware, such as fresh Macs. So, here’s a summary of what was covered, along with some thoughts about how each development might play in healthcare.

Apple unveils 15-inch MacBook Air – Apple just unveiled the 15-inch MacBook Air. It’s 11.5mm thick, just over 3 pounds, and powered by Apple’s own M2 chipset. The video demo showcased a standard headphone jack, two USB-C ports, and Apple’s MagSafe charging dock. The laptop will ship in four colors, including Midnight and Starlight. The laptop features an 18-hour battery with an 8-core CPU and 10-core GPU. It costs $1,299, can be ordered today, and will be available next week.

Image Credit: Apple, Inc.

Apple unveils new $6,999 Mac Pro with M2 Ultra – It’ll feature the M2 Ultra’s 24-core CPU, support up to a 76-core GPU, eight Thunderbolt 4 ports, and up to six of Apple’s high-end Pro Display XDR. Both products can be ordered today for availability next week.

Image Credit: Apple, Inc.

Apple announces iOS 17 with a ton of new features – Apple is unveiling updates to three apps, Phone, Facetime, and Messages. Some highlights:

  • Personalized contact posters, either using photos or Memoji, will allow users to choose their contact card for other users. Live transcription, in real-time, will also be coming for phone voicemails.
  • Facetime will finally get a voicemail functionality, allowing users to leave a video message for their friends.
  • And messaging will be revamped with more powerful search functionality and a catch-up feature for group chats. You can also see the location of your friends while messaging them.
  • New stickers live in a “drawer,” with expanded options, including emoji expansion and rotatable optionality.
  • Live photos can also be made into an animated sticker.
  • A Check-in feature lets you tell a family member or friend when you’re home. Or, if you’re delayed, it’ll tell your friend that, too
  • Messaging apps will now have a different layout, expanding to take up the entire screen.
  • Users can now bump two iPhones or Apple Watches together to share contacts, music, internet, or other shared activities with each other, a feature called “NameDrop.”
Image Credit: Apple, Inc.

What might this mean for healthcare? – Consider the opportunity to share a person’s health care record using the Name Drop feature. Or the check-in feature being used to make sure that a patient who needs ongoing care is safe at home and doing well.

Image Credit: Apple, Inc.

Apple announces new iPadOS 17 software for iPad – Some highlights:

  • Multiple timers for iPad, at last. “We truly live in an age of wonders,” Apple executive Craig Federighi said.
  • The Health app has been added to the iPad as well.
  • Machine-learning tech for PDFs to identify fields even if they’re not built into the PDF.
  • Expanded functionality for iPad’s widgets, including enhanced interactivity for third-party developers. Apple showcased functionality for Quizlet.
  • New customization options for the iPad lock screen, similar to how iPhone users can customize their lock screens. Weather, photo shuffle, and emojis are all part of the offering.
  • Live Activity, a la iOS, which creates a widget-like functionality for users to track scores, delivery orders, or other developer-customizable, continually updating information.
Image Credit: Apple, Inc.

Apple announces macOS Sonoma for Macs – so, the mystery about the name is finally over. Here are some of the highlights:

  • New Gaming functionality, taking advantage of Apple’s more powerful graphics processing units and Apple’s Metal 3 framework.
  • They have expanded video-conferencing functionality through FaceTime. New overlay functions allow customers to appear on top or alongside their presentations.
  • New AR functionality for full-screen reactions that will also be available in Zoom, Teams, and WebEx.
  • Game Mode will prioritize the game’s processing power utilization, allowing more consistent framerates.
  • Widgets are coming to the macOS on the desktop. They used to live in the Mac’s Notification Center. Like the iPad, they’ll be interactive and developer-enabled.
  • Screensavers, like on tvOS, will be coming to macOS.
  • Safari is getting some under-the-hood improvements and enhanced Private Browsing functionality.
Image Credit: Apple, Inc.

Apple announces watchOS 10 for Apple Watch – The software will be available to all users in the Fall. Some highlights:

  • Users can turn the digital crown to reveal widgets, like the weather and calendar.
  • Users can long press to add a widget to their smart stack.
  • Apple introduced new apps like world clock, which features clocks with background colors reflecting the time of day.
  • When users rotate the digital crown, it shows new, full-screen displays.
  • Two new watch faces are available, including a Snoopy and Woodstock watch face and a palate watch face.
  • Users can access new active features, like cycling and hiking. Cycling workouts from the watch will show up on paired iPhone devices, and hikers will have access to new topography elevation details.
  • Developers can also use new workout APIs so that users can start a workout from apps like Training Peaks, for example.
  • In the health app, users can use screening tools and resources to support and evaluate their mental health.
  • WatchOS 10 can measure the amount of time users spend in daylight, as well as how close users’ screens are to their eyes. Apple said these new features could help prevent conditions like myopia.
Image Credit: Apple, Inc.

What might this mean for healthcare? – I find it interesting that the health app features are reviewed during the WatchOS presentation. Apple has announced a range of new health features coming to iOS 17, iPadOS 17, and watchOS 10, focusing on two key areas: mental health and vision health. Recognizing the importance of mental health, Apple has introduced new features that allow users to log their momentary emotions and daily moods and gain insights into their mental well-being. Through the updated Health app on iPhone, iPad, and Apple Watch, users can now engage in an interactive and intuitive reflection on their state of mind. By selecting from a range of multidimensional shapes and describing their emotions, users can develop emotional awareness and resilience. The digital crown lets you scroll through emotions to choose how you’re feeling, and you can even identify triggers that have made you feel this way.

Image Credit: Apple, Inc.

Nearsightedness (Myopia) is the most common cause of visual impairment worldwide. which is estimated to affect more than 30 percent of the current population. And is expected to increase to 50 percent or about 5 billion people in 2050. care providers recommend a few critical behaviors in children. To help reduce the risk of myopia. Two of these behaviors include spending more time outdoors in bright light and increasing distance to see things such as equipment or books. The new screen distance feature uses the same TrueDepth camera that powers Face ID on iPad and iPhones to encourage users to move their devices further away after holding the device closer than 12 inches continuously. The Screen Distance feature can remind young users to practice healthy viewing habits, which can reduce the risk of myopia and offers adult users the opportunity to reduce the visual strain associated with digital devices.

“Mental health and eye health are paramount. but is often overlooked. That’s why we’re very excited to introduce features that will provide valuable new insights to help users gain even greater insight into their health. These insights help users make decisions in their daily lives. and have more information when talking to your doctor.”

Sumbul Desai, MD, Apple’s vice president of health

Apple Vision Pro headset – By this point, we were close to two hours into the event, and everyone was holding their breath for the “big announcement.” Tim didn’t disappoint with his “one more thing” transition.

Image Credit: Apple, Inc.

Apple’s headset is finally here, and it’s called Vision Pro. It’ll retail for $3,499 early next year on Apple’s website. Tons of detail was provided, and it was challenging to capture it all. But here are some of my notes:

  • It is controlled by your eyes, hands, and voice.
  • You are no longer limited by the display.
  • The entire interface will feel “present” in your room.
  • Shadows are used to help you understand scale.
  • Apps can be made any scale in your space (room), just like moving natural objects.
  • “Environments” transform your space to extend your room or allow you to immerse within a scenic environment.
  • Tapping fingers to click and flicking to scroll looks amazing.
  • Siri will work with Vision Pro.
  • ‘EyeSight’ innovation allows others to see your eyes and tell if you are using an app or in an experience while the goggles are on.
  • When someone is nearby, they will appear in your view.
  • It’s in sync via iCloud to your other devices.
  • Your entire world is a canvas for apps.
  • You can send and receive 3D objects in iMessage and then interact with that object in 3D.
  • Magic keyboard and trackpad allow you to type and browse in VR.
  • Bring a MAC into vision pro, and then expand your view.
  • FaceTime becomes Spatial. People appear in your room, life-sized, with spatial sound.
  • You can use apps with other people in Vision Pro.
  • Panoramic images appear all around you.
  • Vision Pro is Apple’s First 3D camera.
  • You can take a spatial video or photo with a button push while wearing the device. It allows you to essentially record real life and rewatch those events later on in VR.
    -Spatial Cinema allows you to create a movie screen anywhere in your home and any size. It’s a movie theater with spatial auto anytime you want.
  • You can use AirPods with the device so others can’t hear the movie you may be watching.
  • 3D movies work on Vision Pro.
  • You can play your favorite Apple Arcade games.
  • Apple has teamed with Disney for some content on Vision Pro.
  • Apple Vision Pro will work for people who wear glasses using special lenses.
  • It will unlock via eye recognition. No more facial recognition required, as it will use a much more comprehensive eye recognition.
  • The 23-million-pixel panel has more pixels than a 4K television per eye.
Image Credit: Apple, Inc.

“You can control it with hands, eyes, and voice. Relive your most important memories in an entirely new way,”

Tim Cook, CEO, Apple, Inc.
YouTube Video Credit: Apple, Inc.

In their initial segmentation for the product, Apple presentations focused on work, home, and entertainment. Most of the applications that were said to be available at product delivery were presented in each of those segments.

Disney CEO Bob Iger also appeared to tout a partnership between Disney and Apple: enhancing the viewing experience for Disney’s treasure trove of content through Apple’s VR technology. For example, the demo included a three-dimensional visualization of a basketball court and an immersive National Geographic application that placed the viewer in the ocean.

Image Credit: Apple, Inc.

Working with Unity, “hundreds of thousands” of iPad and iPhone apps will run on visionOS at launch. Microsoft’s Office Productivity suite will also work on visionOS.

Image Credit: Apple, Inc.

What I found especially interesting was that Apple carefully avoided using the terms VR, AR, MR, and Metaverse in the presentation. They carefully described the Vision Pro as a “spatial computing” device in what I guess was an attempt at a product differentiation strategy.

Somewhat frustratingly (but also probably for the best experience), Vision Pro requires magnetically attached corrective lenses, for users with prescription lenses like me. Other headsets use physical spacers to accommodate glasses, but Vision Pro’s advanced eye-tracking capabilities help drive visionOS. That probably means eye glass reflectivity is a deal breaker for Vision Pro, thus the less convenient solution. Vision-correct accessories are sold separately. However, and this is a big but, Apple adds that “not all prescriptions are supported.” What does this mean? Hopefully, we’ll find out soon.

What might this mean for healthcare? – Building on the experience first made available on Apple Watch, the Mindfulness app on Vision Pro will allow users to create private and calm spaces to reflect, focus and center themselves in what is likely to be a game-changer.

I’m also interested in how the Vision Pro might be used in medical education. One of the images shown was an anatomy app that gave a short example of how the system might make studying anatomy a more personal and immersive experience.

Image Credit: Apple, Inc.

You could also imagine the headset being used to train surgeons in complex operations and combining the simulation with expert proctoring by experts across the globe with AI assistants.

Image Credit: Getty Images

There’s no doubt that Vision Pro could be used in physical and occupational therapy too. XRHealth is one of the few companies focused on providing V.R. physical and occupational therapy at home; based in Boston, it is covered by many insurance programs in Massachusetts and nationally by Medicare. The company is working to get more insurance companies to cover its services. Without insurance, people can pay $179 monthly for the headsets and two physical or occupational therapy appointments monthly with a panel of therapists the company provides.

Image Credit: University of Michigan Medicine

One of the most successful areas in healthcare has been post-traumatic stress disorder (the Veterans Health Administration was an early adopter) and has also been used to reduce pain and anxiety without medication.

Image Credit: USC Institute for Creative Technologies

“The breadth and depth of features, attention to user experience, and integration of mixed reality is astonishing. Hard to imagine, even a few years ago, a device of this magnitude coming to market. It’s expensive, but given its manifold uses, this device has ridiculous potential.”

Brennan Spiegel, MD, MSHS, Director of Health Research, Cedars Sinai

Apple announces visionOS software platform for Vision Pro headset – Apple’s Vision Pro headset will run on visionOS, a brand-new platform for the Vision Pro headset. It’s a spatial computing platform that developers can build for, much as they would for iOS on iPhone or macOS for Mac. It will be able to run a multi-app, 3D engine, and Apple said it is the first operating system designed from the ground up for spatial computing.

The immediate aftermath – The crowd assembled at Apple Park was reported to be excited. Still, the reaction to Apple’s new augmented reality headset reveal was somewhat more muted than other announcements earlier in the day, such as the iPad’s ability to set multiple timers or the ability to locate a lost Apple TV Siri Remote. Apple stock dipped on Monday after it announced its highly-anticipated Vision Pro headset, falling into the red momentarily.

“Apple is so good at making hardware, they’re so good at UX, in a way that other folks who have entered the space haven’t been. So I think this could be a real ‘capture the imagination’ kind of year.”

Anand Agarawala, Cofounder and Chief Executive of AR/VR company Spatial

I think it’s important to remember that Apple isn’t positioning this first version as a “consumer” device. One thing is still certain, however. Apple clearly understands the shortcomings of AR right now, and the Vision Pro seems to dismantle the issues with the tech piece by piece. It’s insanely expensive, and it’s only for a select few early adopters. But from what we know right now, that group will likely be very pleased. My bet: they’ll sell as many as they can manufacture in 2024. I’ll wait until I can see one at the Apple Store and find out more about the prescription lens options before I consider purchasing one of these. But I’m impressed.

Some Straight Talk on Extended Reality (XR) in Health Care

hjsoch7 Comments

“A lot of people think of VR as it relates to gaming and entertainment devices, but when used in a positive way, VR can be a profound benefit to human health.”

Brennan Spiegel, MD – Director of Health Services Research, Cedars Sinai
Image credit: Shutterstock.com

The border between the virtual and real-world continues to break down, providing breathtaking experiences that, a short time ago, could only be found in the imagination of sci-fi writers. So much is happening so fast that the differences between VR, AR, and MR can seem a little puzzling at first. So first, let’s define the terminology.

Virtual reality (VR) – VR is the most widely known of these technologies. It is fully immersive, which tricks your senses into thinking you’re in a different environment or world apart from the real world. Using a head-mounted display (HMD) or headset, you’ll experience a computer-generated world of imagery and sounds in which you can manipulate objects and move around using haptic controllers while tethered to a console or PC.

Augmented reality (AR) – AR overlays digital information on real-world elements. Pokémon GO* is among the best-known examples. Augmented reality keeps the real-world central but enhances it with other digital details, layering new strata of perception and supplementing your reality or environment.

Mixed reality (MR) – MR brings together the real world and digital elements. In mixed reality, you interact with and manipulate both physical and virtual items and environments using next-generation sensing and imaging technologies. Mixed Reality allows you to see and immerse yourself in the world around you even as you interact with a virtual environment using your own hands—all without ever removing your headset. It provides the ability to have one foot (or hand) in the real world and the other in an imaginary place, breaking down basic concepts between real and imaginary, offering an experience that can change the way you game and work today.

Extended reality (XR) – Recently, experts settled on extended reality, or XR, as the catch-all term for devices that put computer-generated visuals in front of your face. And they decided on health care as one of the most promising frontiers for XR. So for this post, I’ll use XR to refer to the many uses of the various technologies in health care.

What are some of the current use cases of XR in health care? –

Source: Fortune Business Insights

As part of my previous work at Sg2, we held an innovation summit each year. We were fortunate in 2019 to have Dr. Brennan Spiegel from Cedars Sinai as a featured speaker. His groundbreaking work in the use of XR is highlighted in his recent book “VRx: How Virtual Therapeutics Will Revolutionize Medicine,” which is the best resource on the use of XR in health care I’ve seen. Based on his work at Cedars Sinai and his extensive research in the field, here are some current use cases in health care:

  • Pain management – In the U.S., pain management is a growing healthcare concern, given that an estimated 100 million adults there suffer from chronic pain. An amount of approximately $17.8 billion is spent prescribing pain medication annually. From Dr. Spiegel’s book: “VR lowers the perception of pain in at least three different ways. First, it distracts the brain from harmful signals rising up from the body. Second, it creates an illusion of time acceleration, effectively shortening the length of pain episodes. And third, it nips signals in the bud at their origin, blocking pain from reaching the brain. The combination of these effects supports the ability of immersion to fight pain.”
  • Schizophrenia, Dementia – Immersion can make a difference in the lives of people with schizophrenia in a way that medication cannot. Dr. Spiegel: “For conditions marked by an eroded sense of self, where the limits of traditional talk therapy make it challenging to access a patient’s inner world, VR offers a gateway into a fractured mind and an opportunity to rebuild its parts from within.” Beyond virtual reminiscence therapy, the literature is now full of studies demonstrating other benefits of VR for dementia care.
  • PTSD and Anxiety Disorders – Virtual reality exposure therapy (VRET) is being examined as another way to help people recover from PTSD. VRET is a type of exposure therapy that has increasingly been used to treat a variety of anxiety disorders, including specific phobias. In VRET, an individual is immersed in a computer-generated virtual environment, either through the use of a head-mounted display device or entry into a computer-automated room where images are present all around.1 This environment can be programmed to help the person directly confront feared situations or locations that may not be safe to encounter in real life.
  • Autism Spectrum Disorder – The technology is also helping children and adults with autism spectrum disorder (ASD). In one initiative, the Center for BrainHealth and Yale School of Medicine are collaborating to help young adults with ASD enhance their capacity for learning and developing the skills needed to achieve economic and social independence in life using VR. “Unlike other therapeutic options, such as role-playing, VR represents real-life experiences in a safe, controllable manner that allow for repeated practice and exposure, which is a key element in treatment,” says a study.
  • Empathy – Virtual technology is being used to increase empathy among medical students and professionals. VR has been used to teach doctors what it’s like to experience the progressive blindness of macular degeneration and glaucoma, to experience an acute migraine headache, to live in a hospice receiving palliative care for advanced cancer, to experience hallucinations, and to enter an MRI scanner while undergoing a critical imaging study. Young doctors-in-training can practice interviewing virtual patients in low-risk environments, learn how to empathize meaningfully and reproducibly, and cultivate enhanced emotional and social awareness about their patients’ life experiences.
  • Surgical training and simulation – A study from Harvard Business Review showed that VR training “improved participants’ overall surgical performance by 230% compared with traditional training methods.” The VR-trained participants were able to complete procedures on average 20% faster and more accurately.
  • Rehabilitation from stroke and other neurological conditions – Over seventy trials are testing the benefits of VR for stroke rehabilitation. Although some programs work better than others, meta-analysis reveals that VR improves outcomes when augmenting traditional therapy. The science of VR neurorehabilitation has progressed beyond academics into the real world.

What are the barriers to implementing XR in health care? – Currently, these are the main barriers to the scalable deployment of XR into health care:

  • Clinician education – Today, few doctors know that these technologies and the evolving discipline of immersive therapeutics exist. The work of Dr. Spiegel and others is narrowing that gap. Also, there is growing enthusiasm among the newest generation of doctors. Young doctors are not only more knowledgeable than older physicians about technology in general, but they’re also the first generation to use VR as part of their medical training.
  • Lack of payment – If doctors don’t get paid for administering VR, it will be difficult to scale the technology beyond research centers and motivated clinics. Are insurance companies willing to reimburse immersive therapeutics? The answer is yes, in certain instances. For example, VR phobia therapy is a reimbursable service that falls under the covered category of exposure therapy. But insurance does not yet cover VR for pain management, which is surprising since it’s the most widely used application of therapeutic VR and is currently offered by several hundred hospitals worldwide. In the absence of reimbursement, most hospitals directly pay for VR programs hoping it will save costs by reducing the need for medications and their costly side effects, improve patient satisfaction, or reduce hospital length of stay. In the meantime, short of insurance coverage, many hospitals turn to philanthropy to help deliver VR to their patients in need.
  • Access to 5G networks and 5G-enabled devices – Any large-scale deployment of XR will likely require the high-bandwidth, low-latency capability that 5G brings to the table.

Where we are today – Health Tech Magazine conducted an online poll in November 2020 asking where health care organizations were on the road to implementing VR technology. They received 303 replies to the survey, and the results are shown below:

Source: Health Tech Magazine, November 23, 2020

Although the sample size in the poll was relatively small, the results mirror the overall market sentiment at this time. As the technology continues to mature and headsets come down in price, adoption will accelerate since the clinical benefits for most of the use cases discussed above are clear.

“The virtual reality (VR) in healthcare market size was $1.56 billion in 2018 and is projected to reach $30.40 billion by 2026, exhibiting a CAGR of 42.4% during forecast period.”

Fortune Business Insights

So, what steps can your organization take to prepare for XR implementation? – If your organization is considering adding XR technologies to your portfolio of digital services, it is essential to prioritize safety and ease of use to optimize the user experience. Key elements to consider are listed below:

  • Know your XR use case – Whether you are designing an XR experience to teach patients about their medical conditions or creating a distracting game for kids to play during flu shots, organizations must understand the unique parameters that each user requires. Work closely with physicians, nurses, child life specialists, and physical therapists to develop appropriate applications.
  • Keep the XR Experience Simple – Most patients and providers have minimal experience with XR, and often there are incredible time pressures in a clinical setting. Many times, clinicians only have minutes to set up the VR system and teach patients.
  • Identify physician champions to educate the clinical staff of the benefits of XR – To scale immersive therapeutics across healthcare, we need to train a new class of providers who are XR authorities. According to Dr. Spiegel: “These virtualists not only will be experts on the latest hardware and software but also will understand whether, when, and how best to integrate XR within the health care setting. They will be highly skilled and multidisciplinary clinicians who work in partnership with specialists to augment traditional therapies across a wide range of conditions.”
  • Learn from experts in XR implementation in health care – Dr. Spiegel and his colleagues are publishing multiple studies on the use of XR in health care. He also holds an annual conference where leading organizations can share their experiences in deploying XR technologies at the point of care. You can learn more about their 2022 conference here.

While there isn’t yet a strong case for cost savings in applying XR to virtual care and chronic disease management, that could shift as more data and research become available. As payers see the better outcomes and cost savings afforded by virtual care platforms, including XR, they will likely pay for these programs, making them part of routine care plans.