07027d8010
* fix: Rename `getFrameProcessorPlugin` to `initFrameProcessorPlugin` * fix: Make nullable, add comments * Format * Update FrameProcessorPlugin.java * Update FrameProcessorPlugin.h * fix: Fix dead links * Call super constructor * Update ExampleFrameProcessorPlugin.java * fix: Init calls
70 lines
5.2 KiB
Plaintext
70 lines
5.2 KiB
Plaintext
---
|
|
id: frame-processors-tips
|
|
title: Frame Processors Tips
|
|
sidebar_label: Frame Processors Tips
|
|
---
|
|
|
|
import Tabs from '@theme/Tabs'
|
|
import TabItem from '@theme/TabItem'
|
|
import useBaseUrl from '@docusaurus/useBaseUrl'
|
|
|
|
## Avoiding Frame-drops
|
|
|
|
Frame Processors will be **synchronously** called for each frame the Camera sees and have to finish executing before the next frame arrives, otherwise the next frame(s) will be dropped. For a frame rate of **30 FPS**, you have about **33ms** to finish processing frames. At **60 FPS**, you only have **16ms**.
|
|
If your Frame Procesor has not finished executing when the next frame arrives, the next frame will be dropped.
|
|
|
|
Some general tips:
|
|
|
|
- Use `runAsync(..)` if you don't need your Frame Processor to run synchronously
|
|
- Use `runAtTargetFps(..)` if you don't need your Frame Processor to run on every frame
|
|
- Use Shared Values (`useSharedValue(..)`) instead of React State (`useState(..)`) when sharing data
|
|
- Prefer native Frame Processor Plugins instead of pure JavaScript based plugins
|
|
|
|
### FPS Graph
|
|
|
|
Use the FPS Graph to profile your Frame Processor's performance over time:
|
|
|
|
```tsx
|
|
<Camera {...props} enableFpsGraph={true} />
|
|
```
|
|
|
|
## Fast Frame Processor Plugins
|
|
|
|
If you use native Frame Processor Plugins, make sure they are optimized for realtime Camera use-cases. Some general tips:
|
|
|
|
- Prefer plugins that use the **[PixelFormat](/docs/api#pixelformat) `yuv` instead of `rgb`**, as `yuv` is more efficient in both memory usage and processing efficiency
|
|
- Prefer plugins that can work with the **native Frame types** (`CMSampleBuffer` and `Image`/`HardwareBuffer`) instead of passing the byte array (`frame.toArrayBuffer()`), as the latter involves a GPU -> CPU copy
|
|
- If you need to use the byte array (`frame.toArrayBuffer()`), prefer plugins that work with **`uint8` instead of `float`** types, as `uint8` is much more efficient
|
|
- Prefer plugins that support **GPU acceleration**. For Tensorflow, this might be the CoreML or Metal GPU delegates
|
|
- For operations such as resizing, **prefer GPU or CPU vector acceleration** (e.g. Accelerate/vImage) instead of just array loops
|
|
|
|
## ESLint react-hooks plugin
|
|
|
|
If you are using the [react-hooks ESLint plugin](https://www.npmjs.com/package/eslint-plugin-react-hooks), make sure to add `useFrameProcessor` to `additionalHooks` inside your ESLint config so dependencies are detected properly. (See ["advanced configuration"](https://www.npmjs.com/package/eslint-plugin-react-hooks#advanced-configuration))
|
|
|
|
## Technical
|
|
|
|
### Frame Processors
|
|
|
|
Frame Processors are JS functions that will be _workletized_ using [react-native-worklets-core](https://github.com/margelo/react-native-worklets-core). They are created on a parallel camera thread using a separate JavaScript Runtime (_"VisionCamera JS-Runtime"_) and are invoked synchronously (using JSI) without ever going over the bridge. In a Frame Processor you can write normal JS code, call back to the React-JS Thread (e.g. `setState`), use [Shared Values](https://github.com/margelo/react-native-worklets-core/blob/main/docs/WORKLETS.md#shared-values) and call Frame Processor Plugins.
|
|
|
|
### Frame Processor Plugins
|
|
|
|
Frame Processor Plugins are native functions (written in Objective-C, Swift, C++, Java or Kotlin) that are injected into the VisionCamera JS-Runtime. They can be synchronously called from your JS Frame Processors (using JSI) without ever going over the bridge. Because VisionCamera provides an easy-to-use plugin API, you can easily create a Frame Processor Plugin yourself. Some examples include [Barcode Scanning](https://developers.google.com/ml-kit/vision/barcode-scanning), [Face Detection](https://developers.google.com/ml-kit/vision/face-detection), [Image Labeling](https://developers.google.com/ml-kit/vision/image-labeling), [Text Recognition](https://developers.google.com/ml-kit/vision/text-recognition) and more.
|
|
|
|
> Learn how to [create Frame Processor Plugins](frame-processors-plugins-overview), or check out the [example Frame Processor Plugin for iOS](https://github.com/mrousavy/react-native-vision-camera/blob/main/package/example/ios/Frame%20Processor%20Plugins/Example%20Swift%20Plugin/ExampleSwiftFrameProcessor.swift) or [Android](https://github.com/mrousavy/react-native-vision-camera/blob/main/package/example/android/app/src/main/java/com/mrousavy/camera/example/ExampleKotlinFrameProcessorPlugin.kt).
|
|
|
|
### The `Frame` object
|
|
|
|
The Frame Processor gets called with a `Frame` object, which is a JSI HostObject. It holds a reference to the native (C++) Frame's GPU Buffer (~10 MB in size) and exposes properties such as `width`, `height`, `bytesPerRow` and more to JavaScript so you can synchronously access them. You can access the Frame data in JavaScript using `frame.toArrayBuffer()`, which copies over the GPU buffer to the CPU.
|
|
The `Frame` object can be passed around in JS, as well as returned from- and passed to a native Frame Processor Plugin.
|
|
|
|
With 4k Frames, roughly 1.5 GB of Frame data flow through your Frame Processor per second.
|
|
|
|
> See [this tweet](https://twitter.com/mrousavy/status/1412300883149393921) for more information.
|
|
|
|
|
|
<br />
|
|
|
|
#### 🚀 Next section: [Zooming](/docs/guides/zooming) (or [creating a Frame Processor Plugin](/docs/guides/frame-processors-plugins-overview))
|