When using the deepAR filter, there is a phenomenon where the screen slows down when the face is not recognized. I would like to know the reason

Help & Questions
1

Basically, there is a performance difference when the face is shown on the camera and when it is not. How can this be resolved?

2

When a face is not visible the SDK calls face detection each frame, which is more processor intensive than face tracking. Ideally this shouldn’t be too noticeable on higher end devices. Which device/os are you seeing this on?

3

Basically, we are using the RTMP protocol to transmit frames to the server for live streaming, and before transmission, we convert the camera’s frame buffer to deepAR using initializeOffscreen and enqueueCameraFrame , then transmit the output from frameAvailable . Of course, I am aware that performing both simultaneously uses more resources than expected, but the question arises because the speed difference when the face is visible and when it is not is noticeable to the naked eye. We are currently using a version that still has a bug regarding priority when initializeOffscreen is used, so I am wondering if this issue could be related to that. Is there any other way to address this? For reference, the same symptoms appear on most devices. We have various models from iPhone 8 to iPhone 13.
and if you have email i can send video of it

4

the speed difference when the face is visible and when it is not is noticeable to the naked eye.

Sorry can you clarify this a bit?

5

So, I am using the sample buffer from the camera by inputting it into deepAR for encoding, and I am rendering the encoded image buffer through MetalView. However, if the face is not captured by the camera, it stutters as if it is lagging. In other words, it feels like the fps is dropping.

6

Elephant_Trunk.deepar

Among the free assets, only the specific filter mentioned above causes frame drops. It seems like this filter includes both facial and background effects. Could this be the reason for the performance decrease? I’m curious if there’s any other special reason for this.
Elephant_Trunk.deepar.zip (984.5 KB)

7

Ah, that one.

I think you’re right on the cause. Specifically, that effect has a rather heavy Bloom experimental postprocessing effect that has been known to cause occasional issues. I’ve removed it, I think that should resove your issue, let me know if it doesn’t.

SimplifiedElephantTrunk.deepar.zip (930.8 KB)