How to develop an Augmented Reality app on Android

How to develop an Augmented Reality app on Android

Augmented reality applications show the view of the real world with information superimposed on them. There are many apps on Google Play which uses Augmented Reality. 

Street Lens shows interesting places in the direction where the phone is pointing. It basically shows the camera view and superimposes the place information on top of it based on which direction the phone is pointing to. 

Similarly, Google Sky Map shows the stars and planets in the direction the phone points. Here again, the app just displays images based on how the phone is held.

Augmented Reality applications have the following basic steps

• Detect the mobile orientation and direction
• Display a background image based on the orientation and direction
• Superimpose relevant information on the image

When I say the direction to which the phone is pointing to, I mean the direction at which the back camera of the phone is pointing to since for Augmented Reality the back camera is the eye of the phone.

Detecting the Mobile orientation and direction

The orientation of the device can be either portrait or landscape. If the height is greater than the width then the orientation of the device is portrait, otherwise it is landscape.

The most important part of the Augmented Reality applications is to know the position of the phone in the 3D world. In other words, the app should know how the phone is held by the user so that the app can show the content based on the direction the phone is pointing to.

Android SDK provides APIs for Accelerometer and Magnetometer and using these sensors the direction of the phone can be determined.

Using the accelerometer and magnetometer sensor values, we can obtain the Azimuth, Pitch and Roll values.

Azimuth value is the most important of the 3 since it tells how the mobile is aligned to the magnetic poles (north, south, east & west). The value ranges from -Pi radians to +Pi radians. Azimuth is –Pi when the phone is pointing to south. It is 0 when the phone points to north. –Pi/2 when it points to west and +Pi/2 when it points to east.

Roll value tells how the mobile is held with respect to the ground. Roll value is 0 when the phone is parallel to ground with camera facing down and it is –Pi or +Pi when the camera is facing up. 

Hold the phone parallel to the ground with the camera facing down. Now rotate the phone such a way that the phone remains parallel to the ground. This rotation can be measured using the Pitch values.

Imagine a big sphere around your phone. Any point in this sphere can be identified by using one of the 3 values (Azimuth, Pitch and Roll). From the phone’s sensor values, you can exactly find out where the phone is pointing to. 

Even though these sensor APIs are available in Android SDK, the sensor’s functionality and accuracy depends on the hardware. For example, in case of Micromax Canvas 1, the roll value is always 0.

 

Here is a sample Android Java code which reads the Azimuth, Pitch and Roll of the device

public class TiltDetector implements SensorEventListener

{

      public float swRoll;

      public float swPitch;

      public float swAzimuth;

      public SensorManager mSensorManager;

      public Sensor accelerometer;

      public Sensor magnetometer;

      public float[] mAccelerometer = null;

      public float[] mGeomagnetic = null;

      public TiltDetector(Context context)

      {

          mSensorManager = (SensorManager)context.getSystemService(Context.SENSOR_SERVICE);

          accelerometer = mSensorManager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER);

          magnetometer = mSensorManager.getDefaultSensor(Sensor.TYPE_MAGNETIC_FIELD);

          mSensorManager.registerListener(this, accelerometer, SensorManager.SENSOR_DELAY_GAME);

          mSensorManager.registerListener(this, magnetometer, SensorManager.SENSOR_DELAY_GAME);

          swRoll = 0;

          swPitch = 0;

          swAzimuth = 0;

      }

      public void Clear()

      {

            mSensorManager.unregisterListener(this, accelerometer);

            mSensorManager.unregisterListener(this, magnetometer);

      }

      public void onAccuracyChanged(Sensor sensor, int accuracy)

      {

      }

      @Override

      public void onSensorChanged(SensorEvent event)

      {

          // onSensorChanged gets called for each sensor so we have to remember the values

          if (event.sensor.getType() == Sensor.TYPE_ACCELEROMETER)

          {

              mAccelerometer = event.values;

          }

          if (event.sensor.getType() == Sensor.TYPE_MAGNETIC_FIELD)

          {

              mGeomagnetic = event.values;

          }

          if (mAccelerometer != null && mGeomagnetic != null)

          {

              float R[] = new float[9];

              float I[] = new float[9];

              boolean success = SensorManager.getRotationMatrix(R, I, mAccelerometer, mGeomagnetic);

              if (success)

              {

                  float orientation[] = new float[3];

                  SensorManager.getOrientation(R, orientation);

                  // at this point, orientation contains the azimuth, pitch and roll values.

                  swAzimuth = (float) (180 * orientation[0] / Math.PI);

                  swPitch = (float) (180 * orientation[1] / Math.PI);

                  swRoll = (float) (180 * orientation[2] / Math.PI);

              }

          }

      }

}

Displaying a background Image based on the position of the phone

When the app knows where exactly the phone is pointing to, the next step is to display appropriate image. The image to be displayed depends on the type of application. Street Lens app, for example, displays the camera view itself.

Suppose you are standing in front of Nasdaq stock market in New York City and the phone is pointing to the Nasdaq stock market. Using the camera view, the exact view that the camera view can be displayed on the app.

Superimpose relevant information on the image

Google Sky Map displays dynamically generated content based on the position of the stars at given time. Street Lens displays the restaurants, building names, etc based on the current location of the user and the direction at which the camera points. So the Augmented Reality application should show relevant info for the given direction of the phone.

Using GPS the exact location (latitude & longitude) of the device can be known and using sensor values as mentioned above, the direction the device is pointing to can be obtained. Coupled with these information, now any information of the real world can be shown on the mobile screen as if the information are displayed right in front of you and not on the mobile.

Many mobile users believe that the Augmented Reality app parses the image from the camera to decide the orientation and direction but as you see the reality is that the direction and position are obtained using the Accelerometer and Magnetometer sensors.