Harnessing Augmented Reality

There is no doubt that VR and AR are hot buzz words that capture one’s imagination. Because of this, some may wonder if there is substance behind the flash – and if there is substance, is it viable and can it be used within the industry? For these technologies to cross the bridge from consumer gaming to productive tools, it is important to provide practical applications that people can see true value in. I’m particularly focused on AR within pharma manufacturing and R&D, and have seen first hand the impact these technologies are having on the industry.

I’m an enterprise mobile developer by background but I’ve been building or leading teams that create iPad applications and mobile applications for Fortune 500 companies industries throughout my career. In 2013, Google Glass was released – and I was inspired. Google Glass is an AR headset that allows users to perform a variety of functions with voice commands (or even the blink of an eye). I liked the hands-free access approach and I saw potential for “something”, though at the time I wasn’t quite sure what that something was – it was just a raw device with no practical enterprise software.

Several months later, I was having a conversation with a friend, who works as a process engineer in the biopharma industry. We were talking about the things that go wrong in your day job and she explained how she always had to drive over an hour to a facility to troubleshoot problems and how people often used incorrect or out-of-date paper-based procedures, which meant she would have to stay late to fill out paperwork. Given all the communication technologies at our disposal today, I was pretty shocked that this still happens in business. As the conversation moved on, I started talking about Google Glass (which was by now collecting dust on my desk) and suddenly it dawned on us both at the same time that access to critical, hands-free information could be very valuable to process engineers. If you sit behind a computer all day, you always have information at your fingertips, but when you’re working in a lab or manufacturing facility, it’s a different story. Some companies use tablets but these aren’t always convenient depending on the location – and they don’t provide hands-free information.

It was a crucial “aha” moment for both of us – and after some thought we decided to try and make a business out of it. The company we formed is called Apprentice Field Suite. We came up with three core ways that Google Glass could be used by biopharma process engineers:

1. See what I see. Can somebody else, somewhere else in the world, see through my eyes to coach me through something and provide assistance? Or train me in a new technique?
2. Paperless procedures. Can I see a procedure in the glasses so that I can work with two hands?
3. Safety and data collection. Can I look at equipment and collect different, non-digital types of information and collate them into a digital format?

Early challenges

I was the one who built the applications, but of course it wasn’t as straightforward as I’d imagined at the outset. We decided to target biopharma specifically since my partner knew the industry inside out, but the regulatory hurdles when working in biopharma are immense and it was a shock to my system! I can’t even begin to explain how long it took to get everything compliant. I needed to ensure that everything could be tracked, logged and validated. We also had to look at the challenges of deployment from an IT perspective. How do you stop someone from walking out of the building with the glasses? Or taking them to inappropriate places within a building? We worked with a partner to develop IT management software that uses a concept called geofencing, which allows you to control what the device can do depending on where it’s physically located.

There were also challenges on the hardware front. At first, I was developing everything for Google Glass, which had its limitations (and was eventually discontinued), so I had to start looking at other glasses. I decided then, early on, that it was critically important for our solutions to be completely hardware agnostic.  Thankfully, there were many other technologies to choose from that were just as comfortable as Google Glass. Just five years ago, AR glasses were very clunky – it was like putting a 10-pound weight on your face. Today, the display and hardware has been miniaturized to make the glasses wearable and comfortable, and the image quality is great – all of which makes the technology more viable.

Augmented Versus Virtual Reality

Virtual reality

VR headsets or glasses – examples of which include the Oculus Rift, Samsung Gear VR and PlayStation VR – completely block out the real world and instead, as the name suggest, present you with a virtual reality created by software. You could be walking up a mountain, playing a game on a battlefield, or standing in a surgical room. VR can also artificially create sensory experiences such as touch, hearing and, in some cases, even smell (the curious among you may wish to look up the Nosulus Rift). A large number of VR headsets and glasses are in development.

Augmented reality

Whereas VR replaces the real world with a virtual one, AR shows you the real world supplemented by virtual elements – Pokemon Go is the most famous and ubiquitous example at present and allows users to discover cartoon critters in a live view of the real world. Thousands of smartphone apps using AR exist, from games to interactive maps to translation tools. Headsets and glasses have also been developed. Google Glass, which has now been discontinued, is perhaps the most well known, but there are dozens of alternatives available. Some companies are developing AR headsets for workers that provide data visualization, for example.

Understanding the world

Both VR and AR headsets are advancing rapidly in terms of becoming lighter and less expensive – and a number of companies and startups have jumped into the market to help create pricing competition and develop further applications. In addition, headsets are increasingly able to understand the world around them. For example, the HoloLens knows how far you are from a wall or from a piece of equipment. At the moment, it can’t tell what the equipment is without a QR code, but this may change in the future. It could be possible to simply look at something and have your head-mounted display (HMD) or glasses tell you what it is.

Crucially, however, the applications we had devised made sense to people in industry. We spoke to people during development and tested a prototype in a laboratory environment. Everything seemed to work and it seemed to be useful to people in biopharma – very good news! The next step was the fingernail-biting launch. We saw potential for the technology and believed it could be useful, but how would people in such a conservative industry react to a new technology?

We launched at the Interphex 2015 trade show in New York and were voted Best New Product at the show. It was a relief to see that the technology really did resonate with the industry. I spoke to a lot of people at Interphex who had actually bought a Google Glass, but like me didn’t really know what to do with it. When you develop a turnkey solution though, suddenly Google Glass and the idea of AR becomes more than just a flashy, gaming device. Since the launch, I’ve given a lot of demos about the technology and it’s very interesting to see how people react. Before the start of a demo, people don’t actually “get it”. I throw out words like “smartglasses” and “enhanced telepresence” but although people nod they don’t really understand – until they put a headset on. As soon as they see and experience AR, the tone of the meeting completely changes. It’s quite magical. I really believe in the potential of the technology, and it’s very rewarding when other people suddenly see the value too.

Real world use

Our clients are mostly large global pharma manufacturers, biotechs, CMOs and equipment manufacturers. Our team assists them with our AR tools in areas touching operations, engineering and R&D. Some of what we do is available out-of-the-box, while some elements require custom integration. One of our customers, Catalent, is using the technology to allow remote experts to see what an individual operator or engineer is seeing in real time. For a company with a large global presence, it is particularly useful as it helps save on travel expenses and service contracts – a very obvious and immediate return on investment. The person wearing the headset can continue to work hands-free, and the expert can even draw on their field of view to show which connection needs to be removed or to highlight if a specific button needs to be pressed, for example. This sort of application is also extremely useful if ever you need vendor advice about purchased equipment.

A top five biopharma company, which I can’t name, is using our technology to access user manuals hands free. A worker can walk up to a piece of equipment that has a QR code on it, scan the code, and the work instructions pop up in the glasses. The instructions are tailored to the person wearing the glasses and the piece of equipment they are looking at, as part of the verification process with our IT controls. Batch records also appear in the glasses and values can be recorded via voice; for example, the worker may need to record a value of 50 ml for the sample; all they need to do is to speak the value and it will be saved in their system.
Some of our other customers have been utilizing our technology in maintenance and facilities, as well as in quality assurance, safety and packaging. Training is another popular application of the technology. Managers can also look at reports to see how long each step is taking, what engineers were working on and if there were any deviations from procedures, which can help refine instructions.

All of these applications are actually very simple, but when you implement them on a practical level it can really change the way people work.

Angelo Stracquatanio is Co-Founder at Apprentice Field Suite.