react-native-vision-camera/docs/docs/guides/FRAME_PROCESSORS_CREATE_OVERVIEW.mdx

---
id: frame-processors-plugins-overview
title: Creating Frame Processor Plugins
sidebar_label: Overview
---

import useBaseUrl from '@docusaurus/useBaseUrl'
import Tabs from '@theme/Tabs'
import TabItem from '@theme/TabItem'

## Overview

Frame Processor Plugins are **native functions** which can be directly called from a JS Frame Processor. (See ["Frame Processors"](frame-processors))

They receive a frame from the Camera as an input and can return any kind of output. For example, a `detectFaces` function returns an array of detected faces in the frame:

```tsx
function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    // highlight-next-line
    const faces = detectFaces(frame)
    console.log(`Faces in Frame: ${faces}`)
  }, [])

  return (
    <Camera frameProcessor={frameProcessor} {...cameraProps} />
  )
}
```

For maximum performance, the `detectFaces` function is written in a native language (e.g. Objective-C), but it will be directly called from the VisionCamera Frame Processor JavaScript-Runtime through JSI.

### Types

Similar to a TurboModule, the Frame Processor Plugin Registry API automatically manages type conversion from JS to native. They are converted into the most efficient data-structures, as seen here:

| JS Type              | Objective-C/Swift Type        | Java/Kotlin Type           |
|----------------------|-------------------------------|----------------------------|
| `number`             | `NSNumber*` (double)          | `Double`                   |
| `boolean`            | `NSNumber*` (boolean)         | `Boolean`                  |
| `string`             | `NSString*`                   | `String`                   |
| `[]`                 | `NSArray*`                    | `List<Object>`             |
| `{}`                 | `NSDictionary*`               | `Map<String, Object>`      |
| `undefined` / `null` | `nil`                         | `null`                     |
| `(any, any) => void` | [`RCTResponseSenderBlock`][4] | `(Object, Object) -> void` |
| `ArrayBuffer`        | [`SharedArray*`][7]           | [`SharedArray`][8]         |
| [`Frame`][1]         | [`Frame*`][2]                 | [`Frame`][3]               |

### Return values

Return values will automatically be converted to JS values, assuming they are representable in the ["Types" table](#types). So the following Java Frame Processor Plugin:

```java
@Nullable
@Override
public Object callback(@NonNull Frame frame, @Nullable Map<String, Object> arguments) {
  return "cat";
}
```

Returns a `string` in JS:

```js
export function detectObject(frame: Frame): string {
  'worklet'
  const result = FrameProcessorPlugins.detectObject(frame)
  console.log(result) // <-- "cat"
}
```

You can also manipulate the buffer and return it (or a copy of it) by returning a [`Frame`][2]/[`Frame`][3] instance:

```java
@Nullable
@Override
public Object callback(@NonNull Frame frame, @Nullable Map<String, Object> arguments) {
  Frame resizedFrame = new Frame(/* ... */);
  return resizedFrame;
}
```

Which returns a [`Frame`](https://github.com/mrousavy/react-native-vision-camera/blob/main/package/src/Frame.ts) in JS:

```js
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  // creates a new `Frame` that's 720x480
  const resizedFrame = resize(frame, 720, 480)

  // by downscaling the frame, the `detectObjects` function runs faster.
  const objects = detectObjects(resizedFrame)
  console.log(objects)
}, [])
```

### Parameters

Frame Processors can also accept parameters, following the same type convention as [return values](#return-values):

```ts
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const faces = scanFaces(frame, { accuracy: 'fast' })
}, [])
```

### Exceptions

To let the user know that something went wrong you can use Exceptions:

```java
@Nullable
@Override
public Object callback(@NonNull Frame frame, @Nullable Map<String, Object> arguments) {
  if (arguments != null && arguments.get("codeType") instanceof String) {
    // ...
  } else {
    throw new Exception("codeType property has to be a string!");
  }
}
```

Which will throw a JS-error:

```ts
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  try {
    const codes = scanCodes(frame, { codeType: 1234 })
  } catch (e) {
    console.log(`Error: ${e.message}`)
  }
}, [])
```

## What's possible?

You can run any native code you want in a Frame Processor Plugin. Just like in the native iOS and Android Camera APIs, you will receive a frame ([`CMSampleBuffer`][5] on iOS, [`ImageProxy`][6] on Android) which you can use however you want. In other words; **everything is possible**.

## Implementations

### Long-running Frame Processors

If your Frame Processor takes longer than a single frame interval to execute, or runs asynchronously, you can create a **copy of the frame** and dispatch the actual frame processing to a **separate thread**.

For example, a realtime video chat application might use WebRTC to send the frames to the server. I/O operations (networking) are asynchronous, and we don't _need_ to wait for the upload to succeed before pushing the next frame, so we copy the frame and perform the upload on another Thread.

```java
@Nullable
@Override
public Object callback(@NonNull Frame frame, @Nullable Map<String, Object> arguments) {
  if (arguments == null) {
    return null;
  }

  String serverURL = (String)arguments.get("serverURL");
  Frame frameCopy = new Frame(/* ... */);

  uploaderQueue.runAsync(() -> {
    WebRTC.uploadImage(frameCopy, serverURL);
    frameCopy.close();
  });

  return null;
}
```

### Async Frame Processors with Event Emitters

You might also run some very complex AI algorithms which are not fast enough to smoothly run at **30 FPS** (**33ms**). To not drop any frames you can create a custom "frame queue" which processes the copied frames and calls back into JS via a React event emitter. For this you'll have to create a Native Module that handles the asynchronous native -> JS communication, see ["Sending events to JavaScript" (Android)](https://reactnative.dev/docs/native-modules-android#sending-events-to-javascript) and ["Sending events to JavaScript" (iOS)](https://reactnative.dev/docs/native-modules-ios#sending-events-to-javascript).

This might look like this for the user:

```tsx
function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    SomeAI.process(frame) // does not block frame processor, runs async
  }, [])

  useEffect(() => {
    SomeAI.addListener((results) => {
      // gets called asynchronously, goes through the React Event Emitter system
      console.log(`AI results: ${results}`)
    })
  }, [])

  return (
    <Camera frameProcessor={frameProcessor} {...cameraProps} />
  )
}
```

This way you can handle queueing up the frames yourself and asynchronously call back into JS at some later point in time using event emitters.

### Benchmarking Frame Processor Plugins

Your Frame Processor Plugins have to be fast. Use the FPS Graph (`enableFpsGraph`) to see how fast your Camera is running, if it is not running at the target FPS, your Frame Processor is too slow.

<br />

#### 🚀 Create your first Frame Processor Plugin for [iOS](frame-processors-plugins-ios) or [Android](frame-processors-plugins-android)!

[1]: https://github.com/mrousavy/react-native-vision-camera/blob/main/package/src/Frame.ts
[2]: https://github.com/mrousavy/react-native-vision-camera/blob/main/package/ios/Frame%20Processor/Frame.h
[3]: https://github.com/mrousavy/react-native-vision-camera/blob/main/package/android/src/main/java/com/mrousavy/camera/frameprocessor/Frame.java
[4]: https://github.com/facebook/react-native/blob/9a43eac7a32a6ba3164a048960101022a92fcd5a/React/Base/RCTBridgeModule.h#L20-L24
[5]: https://developer.apple.com/documentation/coremedia/cmsamplebuffer
[6]: https://developer.android.com/reference/androidx/camera/core/ImageProxy
[7]: https://github.com/mrousavy/react-native-vision-camera/blob/main/package/ios/Frame%20Processor/SharedArray.h
[8]: https://github.com/mrousavy/react-native-vision-camera/blob/main/package/android/src/main/java/com/mrousavy/camera/frameprocessor/SharedArray.java