Challenges with RealityKit and ARKit

Question

I want to build a demo app in ARKit and I have some questions about what is currently possible with the beta (Apple has been calling this RealityKit, or ARKit 3.0).

---

The demo app I'm trying to build should do the following:

1. identify an object or image in the real environment, and create an anchor there

2. render a virtual model attached to the anchor

3. have the virtual model presented with occlusion

4. have the virtual model move along with the anchor image / object

---

I've tried adapting some code from earlier versions (ARKit 2.0 which leverages SceneKit), but certain features like people occlusion are not part of ARKit 2.0.

As Apple has been iterating on their beta, a lot of features advertised on their site and at WWDC 2019 have seemingly disappeared from the documentation for RealityKit (people occlusion, body tracking, world tracking).

The way I understand it, items (1) and (2) are possible with ARKit 2.0. Item (3) is advertised as possible with the beta, but I see little to no documentation.

Is this possible to do in the latest beta? If so, what is the best approach? If not, are there any workarounds like mixing the old and new APIs or something?

Answer 1

All the challenges you mentioned, are accessible in ARKit/SceneKit and ARKit/RealityKit.

Choose configuration type

1. Identify an object or image in the real environment, and create an anchor there.

You're able to identify 3D objects or Images using the following configs in ARKit:

let configuration = ARWorldTrackingConfiguration()

guard let obj = ARReferenceObject.referenceObjects(inGroupNamed: "Resources", 
                                                         bundle: nil) 
else { return }

configuration.detectionObjects = obj     // Allows you create ARObjectAnchor
sceneView.session.run(configuration)

and:

let config = ARWorldTrackingConfiguration()

guard let img = ARReferenceImage.referenceImages(inGroupNamed: "Resources", 
                                                       bundle: nil) 
else { return }

config.detectionImages = img             // Allows you create ARImageAnchor
config.maximumNumberOfTrackedImages = 3
sceneView.session.run(config)

However, if you want to implement a similar behaviour in RealityKit use this:

let objectAnchor = AnchorEntity(.object(group: "Resources", name: "object"))

and:

let imageAnchor = AnchorEntity(.image(group: "Resources", name: "model"))

Render Engines

2. Render a virtual model attached to the anchor.

At the moment ARKit has four companions helping you render 2D and 3D graphics:

• RealityKit (3D engine)
• SceneKit (3D engine)
• Metal (3D engine)
• SpriteKit (2D engine)

Materials

3. Have the virtual model presented with occlusion.

In RealityKit module all the materials are structures that conform to Material protocol. At the moment there are 6 types of materials. You need OcclusionMaterial.

• OcclusionMaterial (An invisible material that hides objects rendered behind it)
• SimpleMaterial (A basic material that you can apply to meshes)
• UnlitMaterial (A simple material that doesn’t respond to lights in the scene)
• VideoMaterial (A material that allows you play a video on it using AVPlayer)
• PhysicallyBasedMaterial (A material that simulates the appearance of real-world objects)
• CustomMaterial (A material which name is self-descriptive)

Look at THIS POST to find out how to assign materials programmatically in RealityKit.

And THIS POST shows you how to assign custom occlusion material in SceneKit.

Updating 3D content

4. Have a virtual model move along with an image/object anchor.

To implement this type of behavior in ARKit+SceneKit you have to use renderer(_:didAdd:for:) or session(_:didAdd:) methods. In RealityKit AnchorEntities are tracked automatically.

Here's an example of using ARObjectAnchor in renderer(_:didAdd:for:) instance method:

extension ViewController: ARSCNViewDelegate {
    
    func renderer(_ renderer: SCNSceneRenderer,
                 didAdd node: SCNNode,
                  for anchor: ARAnchor) {
                
        if let _ = anchor as? ARObjectAnchor {                
            let text = SCNText(string: "ARKit", extrusionDepth: 0.5)
            let textNode = SCNNode(geometry: text)
            node.addChildNode(textNode)
        }
    }
}