XR 2x2 Segmentation

(click image for better resolution)

Trying to unpick all the different emerging systems in the "extended reality" space such as Augmented Reality (AR), Mixed Reality (MR), Virtual Reality (VR) and then the "traditional" approaches such as 3D video games and virtual worlds can be a challenge, and we keep trying to find new ways to understand and communicate it.

In the graphic above we've tried to show the difference between the systems in terms of the access device and what they are trying to do to reality.

In one dimension we have whether the system is trying to completely replace what the user is seeing/experience - they only see the digital world, or whether it is augmenting it - adding new content into what the user otherwise sees as the physical world.

In the other dimension we have whether the system is accessed through a flat screen or a headset. The flat screen might be an ordinary PC screen, or a tablet or smartphone, whilst the headset might be a VR one or an MR one. A key point is that for the AR/MR solutions the access devices has some way of also showing the physical world which is being manipulated - via the phone camera for AR, or a transparent visor for MR.

These two dimensions then give us our four main use cases:

• AR - which is overlaying reality but viewed through a (conventional) flat screen
• MR - which is overlaying reality but viewed through a headset (visor)
• 2D/3D systems like Second Life/computer games - which is completely replacing reality and viewed through a flat 2D screen
• HMD VR - which is completely replacing reality and viewed through a headset (screen)

A key point is that the dimensions are about how we access and what we are trying to achieve. They are not about the software technology being used to generate the environments, or how its branded!

Let us know if you've any comments on this segmentation, and/or if you find it of use.

 

Daden U Day: GeospatialAR - Take 2

I spent the latest DadenU day doing some more work on the AR app I started last time. Like the recent OS feature this is a geospatial AR app, so it puts markers in your smartphone viewscreen which align with real world objects. As you pan around you see different objects, and if you move to a new location all the markers track the objects they are pointing at.

The driver for the project is a personal "need" to have such an app to support battlefield walks, but we can also see some commercial potential for it, as well as more formally exploring the battlefield tours angel with the Battlefields Trust of which I'm a member.

In the latest session I focussed on bring the data in from the web, rather than having it hard coded within the app. This meant developing a JSON format for the data, doing a fetch by web service request, and then feeding the data to the existing plotting routine.

Getting my head around JSON took a fair amount of time, I much prefer, XML, and most of the rest was fairly trivial. Where I got stuck at the end though was creating a dynamic drop down of the available locations based on the files on the web server - will have to ask one of our pros how to do that!

Over the next couple of months I hope to work it up to a full blown demo based on a real battlefield, and then we'll see where we can take it.

Geospatial AR - Building Your Own!

For ages now I've wanted to have an app that would display points of interest on my smartphone screen overlayed with the real world view. The new OS AR layer (above) and Yelp Monocle (below) do the sort of thing I want, but I want to be able to define my own locations, and maybe some custom functionality.

After a couple of fruitless web and App/PlayStore searches I couldn't find what I wanted. Wikitude was closest, and so were several GIS related offerings, but it was going to be several hundreds of pound to "publish" my dataset. I then looked at mobile development frameworks (e.g. Corona) several of which appeared to offer an AR option, but really only just marker based, not geospatial AR. So by about 1030 on DadenU day I realised I was going to have to roll-my-own. I'd found a nice tutorial (Part 1 and Part 2) and so, without having ever developed a Java app or mobile app decided to give it a go.

It took about the next 3 hours to install Android Studio and all its updates. Moving the tutorial across didn't take long, but the app kept crashing on start-up. I then realised that I needed to manually set the GPS and Camera positions! Did that and the app worked.

All the app did though was put a marker on the centre of the screen when the camera pointed in a particular direction. I wanted the OS style though - markers that stayed on screen and slid around as you moved the phone, and a whole host of them too.

A bit of maths, some searching on CodeProject and I soon had the basic operating mode changed, one marker, sliding around and just about aligning with the target. The hardcoded about  dozen markers and got them working. Here's a screenshot of that stage, with the icon that came with the tutorial. The other thing I added was that markers were smaller if more distant.

That was about the end of DadenU day, I had the core app working but I wanted more. So over the next few evenings I:

• Moved from hard coded markers to an internally selectable dataset
• Added some on-marker text
• Made new markers of different shapes
• Added ability to set marker shape and colour based on fields in the data
• Added option to move markers in the vertical plane based on distance as (OS/Yelp)
• Added the ability to filter on a type parameter (again as OS)

That got to about here:

I also added the ability to spoof the phone GPS location, so it would pretend you were in the middle of the dataset - so here the centre of the Battle of Waterloo - but physically on the office car park.

I then wanted to add some very specific functionality. As you might guess from the last sentence one of my use cases for this is battlefield tours. So not only are there fixed locations but also moving troops. So I wanted a time slider that you could use to set a time in the battle, and then have the unit pointers point to the right place. The time slider is at the bottom of the screen for relevant datasets, with the "current time" displayed see below.

A final tidy up of the menu icons and this is the current version:

I've even added voice description for the icons, and arrows that point you in the direction of a featured icon, with description linked to time so using the slider and some VCR controls it can walk you through a spatial narrative.

Next up on the to do list are:

• Click on an icon to see (or hear) full data
• Load data from  a web server

And that should pretty much give me my minimum viable product.

We're already talking to one potential client about a project based around this, and can see how it might fit into two other projects (giving an alternative to viewing the data in VR if actually on-site). We'll keep you posted as it develops. At the very least I'll start using it for some of my own battlefield walks.