Fundamentals of ARCore Android

Introduction

Before talking about ARCore, the first question that arises is, "What is AR?" Well, AR stands for Augmented Reality. According to Wikipedia, "Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory."

ARCore is a Google platform used to make AR-based applications using its predefined APIs. Through it, we can use different APIs to sense the environment and understand the world using motion and light detection and process that information to enable smooth AR experience.

ARCore has capabilities to sense the environment and three key capabilities to integrate virtual content with the real world as seen through your phone's camera:

Motion tracking
Environmental understanding
Light estimation

Motion tracking

Users use the phone's camera to see the real world and the device must be rotated and it changes orientation, angles, position, etc to get better clicks. Since this device's motion is taken into consideration to track motion, ARCore uses a process called concurrent odometry and mapping, or COM, to understand where the phone is relative to the world around it. Some visually distinct features detected by ARCore in the image captured by the camera are called feature points and it uses these points to compute its change in location.

The main objective is to capture device orientation and position in the real world with respect to time and this information is used and combined with inertial measurements from the device's IMU to estimate the pose (position and orientation).

If the device's camera orientation and position are aligned to the virtual camera provided by ARCore then users are able to render the 3D image over a virtual camera. That rendered virtual image must be overlayed over the top of the image clicked by the device camera.

Environmental understanding

ARCore is constantly improving its understanding of the real world through feature point detection and surface detection as well. ARCore looks for the group or cluster of points that show on a vertical or horizontal surface like a table, floor, and walls, etc. After detection and manipulating ARCore enables these surfaces as a plane to render the objects in the real world over the camera. Objects must be shown over these planes.

Note that white walls and floors without textures are difficult to detect surfaces, and it will not enable them as planes.

Light Estimation

ARCore is so advanced that it can detect lighting in the surroundings, so accordingly, it gives a sense of realism while rendering objects. Based on the light data, it provides the controls to enhance or alter color combinations of the rendered scene.

User Interaction

Since the device screen can be regarded as an (x,y) plane where ever a user interacts or touches, it will give the coordinates of that screen. A ray is formed that intersects the planes and these feature points are returned that are passed by ray along with the pose of that intersection in world space.

Oriented Points

Since we know that ARCore uses clusters of points to detect the angled surface, oriented points let us place objects on angled surfaces. When a user tests or moves the device around them, it looks for nearby points and uses that data to sense the angle of the surface.

Anchors and trackables

Poses get continuously changed because ARCore is continuously estimating the surface and angles in the real world. When you want to place a virtual object, than the anchor must be defined first and relative to that anchor object should be placed. Planes and points are a special type of object called a trackable. Be sure that trackable and object relation must be stable even if the device rotates or changes pose over time. For example, if you have placed an object on a desk, even if the device position changes, then the object must be in a stable state or appear on the table.

Augumented Image

An augmented image is a feature used to enhance reality. For example, take a 2D poster. Then, if the camera is placed over the poster, its characters would be popped out and enact a scene that seems to be a 3D experience.

AR principles

Designing environments
Creating and manipulating virtual objects
Real-world movement
Making user interfaces

Runtime Consideration

You can provide the best possible user experience by ensuring that your app:

Provides clear feedback to users
Encourages them to move their device
Shows them how to interact with their device to experience AR

Encourage users to move the camera slowly

Since ARCore requires visual information and motion information as well, rapid device movement can cause the camera image to become blurry, reducing ARCore's ability to track and detect features. Mostly, data relies on IMU(Inertial Measurement Unit) information to distinguish a pose from the real world surroundings. So, avoid extended periods of rapid movement, which can cause ARCore to lose tracking and prevent the detection of features.

Enable ARCore on Android Studio

First, we need to update our Android Studio IDE to 3.1 or higher. Currently, I have updated to 3.6.1, so you guys must update to the latest version of Android Studio.

Secondly, update the SDK platform to 7.0 or API level 24 or higher.

We have to follow some steps to enable ARCore in our project:

Add AR Required or AR Optional entries to the manifest
Add build dependencies to your project
Perform runtime checks to ensure the device is ARCore-supported, that Google Play Services for AR is installed on it, and that camera permission has been granted
Make sure your app complies with ARCore's User Privacy Requirements

AR Optional Apps

AR optional apps are those apps that can run on devices that don't support ARCore.
Google play store doesn't install automatically Google play services for AR

AndroidManifest.xml

<uses-sdk android:minSdkVersion="24" />
<uses-permission android:name="android.permission.CAMERA" />
<application>
…
<!-- Indicates that the app supports, but does not require ARCore ("AR Optional").
Unlike "AR Required" apps, the Google Play Store will not automatically
download and install Google Play Services for AR when the app is installed.
-->
<meta-data android:name="com.google.ar.core" android:value="optional" />
</application>

AR Required apps

AR Required apps are those apps that can run on devices that support ARCore and Google play store list them for supported devices only.
Google play store install automatically Google play services for AR

<uses-sdk android:minSdkVersion="24" />
<uses-permission android:name="android.permission.CAMERA" />
<!-- Indicates that the app requires ARCore ("AR Required"). Ensures the app is
visible only in the Google Play Store on devices that support ARCore.
-->
<uses-feature android:name="android.hardware.camera.ar" />
<application>
…
<!-- Indicates that the app requires ARCore ("AR Required"). Causes the Google
Play Store to download and install Google Play Services for AR when the
app is installed.
-->
<meta-data android:name="com.google.ar.core" android:value="required" />
</application>

Add gradle dependencies to your app's build.gradle and add Google's maven to project's build.gradle

build.gradle project level

// Top-level build file where you can add configuration options common to all sub-projects/modules.
buildscript {
repositories {
jcenter()
google()
}
dependencies {
classpath 'com.android.tools.build:gradle:3.6.1'
// NOTE: Do not place your application dependencies here; they belong
// in the individual module build.gradle files
}
}
allprojects {
repositories {
jcenter()
google()
}
}
task clean(type: Delete) {
delete rootProject.buildDir
}

Now see the app level build.gradle

apply plugin: 'com.android.application'
apply plugin: 'com.google.gms.google-services'
repositories {
maven { url 'https://maven.fabric.io/public' }
maven { url "https://maven.google.com" }
}
buildscript {
repositories {
mavenCentral()
maven { url "https://maven.google.com" }
}
}
android {
compileSdkVersion 28
defaultConfig {
applicationId "com.example.com"
minSdkVersion 24
targetSdkVersion 28
versionCode 28
versionName "1.0.0"
}
buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
}
}
packagingOptions {
exclude 'META-INF/DEPENDENCIES.txt'
exclude 'META-INF/LICENSE.txt'
exclude 'META-INF/NOTICE.txt'
exclude 'META-INF/NOTICE'
exclude 'META-INF/LICENSE'
exclude 'META-INF/DEPENDENCIES'
exclude 'META-INF/notice.txt'
exclude 'META-INF/license.txt'
exclude 'META-INF/dependencies.txt'
exclude 'META-INF/LGPL2.1'
}
}
dependencies {
implementation 'com.android.support.constraint:constraint-layout:1.0.2'
implementation fileTree(include: ['*.jar'], dir: 'libs')
implementation 'com.android.support:support-v4:28.0.0'
testImplementation 'junit:junit:4.12'
implementation 'com.android.support:appcompat-v7:28.0.0'
// this is dependency for enabling ARCore
implementation 'com.google.ar:core:1.15.0'
}

Perform runtime checks

Check whether ARCore is supported (AR Optional apps only). Let's see the code of MainActivity.java

@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
// Enable AR related functionality on ARCore supported devices only.
maybeEnableArButton();
…
}
void maybeEnableArButton() {
ArCoreApk.Availability availability = ArCoreApk.getInstance().checkAvailability(this);
if (availability.isTransient()) {
// Re-query at 5Hz while compatibility is checked in the background.
new Handler().postDelayed(new Runnable() {
@Override
public void run() {
maybeEnableArButton();
}
}, 200);
}
if (availability.isSupported()) {
mArButton.setVisibility(View.VISIBLE);
mArButton.setEnabled(true);
// indicator on the button.
} else { // Unsupported or unknown.
mArButton.setVisibility(View.INVISIBLE);
mArButton.setEnabled(false);
}
}

Request camera permission (AR Optional and AR Required apps)

If we do not request a permission model for Camera, then the app crash due to security issues.

@Override
protected void onResume() {
super.onResume();
// ARCore requires camera permission to operate.
if (!CameraPermissionHelper.hasCameraPermission(this)) {
CameraPermissionHelper.requestCameraPermission(this);
return;
}
…
}

After requesting camera one should handle the response of the user if user has given consent or not.

@Override
public void onRequestPermissionsResult(int requestCode, String[] permissions, int[] results) {
if (!CameraPermissionHelper.hasCameraPermission(this)) {
Toast.makeText(this, "Camera permission is needed to run this application", Toast.LENGTH_LONG)
.show();
if (!CameraPermissionHelper.shouldShowRequestPermissionRationale(this)) {
// Permission denied with checking "Do not ask again".
CameraPermissionHelper.launchPermissionSettings(this);
}
finish();
}
}

Check whether Google Play Services for AR is installed (AR Optional and AR Required apps)

// Set to true ensures requestInstall() triggers installation if necessary.
private boolean mUserRequestedInstall = true;
@Override
protected void onResume() {
super.onResume();
// Check camera permission.
…
// Make sure Google Play Services for AR is installed and up to date.
try {
if (mSession == null) {
switch (ArCoreApk.getInstance().requestInstall(this, mUserRequestedInstall)) {
case INSTALLED:
// Success, create the AR session.
mSession = new Session(this);
break;
case INSTALL_REQUESTED:
// Ensures next invocation of requestInstall() will either return
// INSTALLED or throw an exception.
mUserRequestedInstall = false;
return;
}
}
} catch (UnavailableUserDeclinedInstallationException e) {
// Display an appropriate message to the user and return gracefully.
Toast.makeText(this, "TODO: handle exception " + e, Toast.LENGTH_LONG)
.show();
return;
} catch (…) { // Current catch statements.
…
return; // mSession is still null.
}
…
}

Note:

If requestInstall() returns INSTALL_REQUESTED, the current activity pauses and the user is prompted to install or update Google Play Services for AR. Now ARCore is fully enabled in Android Studio.