ARView counterpart to SCNView's SCNTechnique?

Question

I have made project with ARKit that uses metal shaders to perform a mirroring effect, and this is assigned by using sceneView.technique. I am trying to figure out if it would be possible to transition the project over to using RealityKit instead, but cannot find any similar trait of the ARView used by RealityKit. Is there anything in RealityKit similar to sceneView.technique that would allow me to simply assign the metal shaders as I do right now, or would I have to rework how the metal shaders are interacting with the view?

My current implementation is based on this post.

Answer 1

About ARView mirroring

The simplest way to mirror ARView in RealityKit 2.0 is to use a CGAffineTransform 3x3 matrix.

 ?             ?
 |  a   b   0  |
 |  c   d   0  |
 |  tx  ty  1  |
 ?             ?

We need to reflect a scene (including video) horizontally, so we have to use -1 multiplier for xScale:

 ?             ?
 | -1   0   0  |
 |  0   1   0  |
 |  0   0   1  |
 ?             ?

Here's the code:

arView.transform = .init(a: -1, b: 0, c: 0, d: 1, tx: 0, ty: 0)

About Metal shaders

In RealityKit 2.0 only CustomMaterial allows you implement Metal's vertex and fragment shaders. So, you can use a vertex shader for many interesting effects, including flipping a model across X axis.

.swift file:

import Metal
import RealityKit

let device = MTLCreateSystemDefaultDevice()
    
guard let defaultLibrary = device!.makeDefaultLibrary()
else { return }
    
let shader = CustomMaterial.SurfaceShader(named: "basicShader",
                                             in: defaultLibrary)
    
let modifier = CustomMaterial.GeometryModifier(named: "basicModifier",
                                                  in: defaultLibrary)
    
box.model?.materials[0] = try! CustomMaterial(surfaceShader: shader, 
                                           geometryModifier: modifier, 
                                              lightingModel: .lit)

.metal file:

#include <metal_stdlib>
#include <RealityKit/RealityKit.h>

using namespace metal;
using namespace realitykit;

[[visible]]
void basicShader(realitykit::surface_parameters shader)
{
    surface::surface_properties sf = shader.surface();
    
    sf.set_base_color( half3(1.0f, 0.7f ,0.0f) );
};

[[visible]]
void basicModifier(realitykit::geometry_parameters modifier)
{
    float3 pose = modifier.geometry().model_position();

    // your code...
};

However, a counterpart of SCNTechnique for RealityKit view isn't "baked" yet...

Answer 2

You can use RealityKit 2's PostProcessing API which should make this desired effect pretty easy to implement.

For example you could apply a Core Image Filter or Metal kernel functions to your rendered scene. Somewhat similar to SCNTechnique.

https://developer.apple.com/documentation/realitykit/implementing_special_rendering_effects_with_realitykit_postprocessing?changes=l_6&language=objc