Probably - DeepAR SDK Bug Report: Intermittent Black Frame Video Recording

Help & Questions
1

Probably - DeepAR SDK Bug Report: Intermittent Black Frame Video Recording

Environment

  • DeepAR SDK version: 5.6.21
  • Platform: iOS 17+
  • Devices: iPhone 14 Pro, iPhone 15 Pro (reproduced on multiple devices)
  • Xcode: 16.x
  • Recording mode: Multi-segment video recording with warmup enabled

Summary

DeepAR’s built-in video recorder intermittently produces entirely black video files despite receiving valid camera frames, having renderingInitialized=true, and reporting no errors. The issue occurs in a multi-segment recording workflow (record → finish → re-arm warmup → record next segment).

How We Initialize DeepAR

deepAR = DeepAR()
deepAR?.delegate = self
deepAR?.setLicenseKey(licenseKey)
deepAR?.changeLiveMode(true)

// After didInitialize callback:
deepAR?.videoRecordingWarmupEnabled = true

// Metal layer setup:
let metalEAGLLayer = MetalEAGLLayer()
metalEAGLLayer.device = MTLCreateSystemDefaultDevice()
metalEAGLLayer.pixelFormat = .bgra8Unorm
metalEAGLLayer.framebufferOnly = true
metalEAGLLayer.drawableSize = CGSize(width: 1080, height: 1920)
deepAR?.initialize(withWidth: 1080, height: 1920, window: metalEAGLLayer)

Recording Lifecycle (Multi-Segment Mode)

Each segment follows this cycle:

  1. Warmup — startVideoRecording(withOutputWidth:outputHeight:subframe:) called during init
  2. User taps record — resumeVideoRecording() called (warmup is enabled)
  3. User taps stop — finishVideoRecording() called
  4. DeepAR callback — didFinishVideoRecording(_:) fires with output path
  5. Re-arm for next segment — immediately call startVideoRecording(withOutputWidth:...) again to warm up the encoder for the next segment
// Re-arm after segment completes:
DispatchQueue.main.async { [weak self] in
    self?.startVideoRecordingWithOptions()  // re-arm warmup
}

Camera Frame Delivery

// In AVCaptureVideoDataOutputSampleBufferDelegate:
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, ...) {
    guard !isDeepARPaused else { return }
    deepAR?.enqueueCameraFrame(sampleBuffer)
}

Diagnostic Methodology

We added comprehensive logging at every boundary:

  1. Frame health — Validated every camera frame: 1080x1920 BGRA, ~30fps
  2. Frame delivery — Counted total/dropped/recording frames via enqueueCameraFrame
  3. DeepAR state — Logged renderingInitialized, isDeepARPaused, deepARState at every callback
  4. Metal layer — Monitored sublayer count for resource leaks
  5. Video file health — Measured file size, duration, video track count for every output file
  6. Error callbacks — Monitored recordingFailedWithError and onError(withCode:error:)

Evidence: 15 Recorded Segments

Seg File Size Duration KB/frame (@30fps) Status
1 17.8 MB 6.86s 86 KB OK
2 11.9 MB 4.60s 86 KB OK
3 16.4 MB 6.27s 87 KB OK
4 29.6 MB 11.39s 87 KB OK
5 142 KB 9.08s 0.5 KB BLACK
6 39.3 MB 15.13s 87 KB OK (self-recovered)
7 13.6 MB 5.20s 88 KB OK
8 12.8 MB 4.92s 87 KB OK
9 9.1 MB 3.52s 102 KB OK
10 7.7 MB 2.97s 89 KB OK
11 45 KB 2.94s 0.5 KB BLACK
12 77 KB 4.83s 0.5 KB BLACK
13 9.8 MB 3.78s 87 KB OK (self-recovered)
14 94 KB 5.96s 0.5 KB BLACK
15 185 KB 11.76s 0.5 KB BLACK

5 out of 15 segments (33%) produced black video.

All External Indicators Are Healthy During Black Segments

For every segment, including the black ones:

  • Camera frames: Valid 1080x1920 BGRA buffers delivered at ~30fps — zero invalid frames
  • Frame delivery: Zero dropped frames, zero BLOCKED events, isDeepARPaused=false always
  • Rendering state: renderingInitialized=true
  • Metal layers: sublayers=1 — no resource accumulation
  • Warmup: didFinishPreparingForVideoRecording fires for every segment, isPreparedForRecording=true set before user taps
  • Errors: Zero recordingFailedWithError calls, zero onError calls

Analysis

The black segments have:

  • Valid .mov container — correct duration, video track present
  • ~0.5 KB/frame vs normal ~87 KB/frame — the H.264 encoder IS running, but encoding black/empty render output
  • Non-sequential pattern — seg 5 bad → seg 6 good → segs 7-10 good → seg 11 bad — rules out cumulative state corruption

Since the camera is delivering valid frames, renderingInitialized=true, and no errors are reported, the black output must originate inside DeepAR’s render-to-encoder pipeline. The encoder receives frames from DeepAR’s internal render target, and that render target is intermittently black.

Suspected Root Cause

The issue appears to be a race condition in DeepAR’s encoder pipeline during the finishVideoRecording() → startVideoRecording() re-arm cycle. When the new startVideoRecording() call arrives before DeepAR’s internal encoder has fully torn down the previous session, the new encoder session connects to a stale/uninitialized render target.

Workaround Applied

We have implemented a client-side mitigation:

  1. Detect black segments via file-size-to-duration ratio (< 500 KB/s = black)
  2. Discard bad files and revert UI
  3. Pause/resume DeepAR before re-arming warmup, with a 100ms delay

This reduces but does not eliminate the issue since the root cause is internal to the SDK.

Request

Could you investigate the encoder pipeline’s behavior during rapid finishVideoRecording() → startVideoRecording() cycles? Specifically:

  1. Is there a minimum delay required between finishVideoRecording() completing and a new startVideoRecording() call?
  2. Is there a recommended way to ensure the encoder’s render target is properly re-attached after a finish/start cycle?
  3. Could a didFinishVideoRecording callback guarantee that it’s safe to immediately start a new recording?

I think that information is quite enough, but if need I could provide additional logs.

##UPD:

# Follow-up: Black Screen Issue — Extended Session Data & Live Preview Failure

This is a follow-up to our original bug report on intermittent black frame video recording with DeepAR SDK 5.6.21.

We have new data from extended recording sessions and a **new, more severe finding**: DeepAR’s live preview itself goes black.

## New Test Session: 20 Segments

We recorded 20 segments (segments 2–20 after warmup). A raw camera fallback recorder ran in parallel to capture frames independently of DeepAR.

| Seg | DeepAR Output | Fallback Used? |

|-----|---------------|----------------|

| 2 | OK (13–26 MB) | No |

| 3 | **BLACK** (44–176 KB) | Yes |

| 4 | **BLACK** | Yes |

| 5 | **BLACK** | Yes |

| 6 | **BLACK** | Yes |

| 7 | OK (13–26 MB) | No |

| 8 | OK (13–26 MB) | No |

| 9 | **BLACK** | Yes |

| 10 | **BLACK** | Yes |

| 11 | **BLACK** | Yes |

| 12 | **BLACK** | Yes |

| 13 | OK (13–26 MB) | No |

| 14 | **BLACK** | Yes |

| 15 | **BLACK** | Yes |

| 16 | **BLACK** | Yes |

| 17 | **BLACK** | Yes |

| 18 | **BLACK** | Yes |

| 19 | **BLACK** | Yes |

| 20 | **BLACK** | Yes |

- **Good segments (4):** 2, 7, 8, 13 — ~2.5 MB/s, normal encoded video

- **Black segments (15):** all others — ~15 KB/s, H.264 encoding black frames

- **Black rate: 79%** (up from 33% in our first 15-segment session)

The black rate worsens as the session continues — the last 7 segments (14–20) are all black with no recovery.

## New Finding: Live Preview Goes Black

Around segment 20, **DeepAR’s live preview view went completely black**. The user sees a black screen in the camera UI — not just black recorded output.

Key details:

- Camera frames are **still being delivered** — logs confirm valid 1080x1920 BGRA frames at 30fps via `enqueueCameraFrame`

- `renderingInitialized` remains `true`

- **No errors** — zero `onError` or `recordingFailedWithError` callbacks

- The **Metal render layer produces no visible output**

- **Not recoverable** without fully restarting DeepAR — unlike recording black frames which sometimes self-recover on the next segment, the preview stays black

This indicates the issue is not isolated to the encoder pipeline. DeepAR’s core rendering pipeline degrades over repeated `finishVideoRecording()` → `startVideoRecording()` cycles.

## Progression Pattern

1. **Early segments (~1–10):** Intermittent black recordings, preview remains visible

2. **Later segments (~10–20):** Black recordings become dominant, fewer self-recoveries

3. **~20+ segments:** Live preview goes black — rendering pipeline fails entirely

This suggests a cumulative resource leak or state corruption inside DeepAR’s render pipeline that accumulates with each recording cycle.

## Our Current Workaround

We run a parallel `AVAssetWriter` recording raw camera frames. After each segment, we check the DeepAR output file size — if it’s below 500 KB/s we substitute the raw camera file. This preserves all segments (without DeepAR effects) but **cannot fix the live preview going black**.

## Questions

1. Is there a resource leak (textures, render passes, Metal command buffers) that accumulates across `finishVideoRecording()` → `startVideoRecording()` cycles?

2. Is there a way to reset DeepAR’s rendering state without full reinitialization?

3. Is there a recommended maximum number of recording cycles before the SDK should be reinitialized?