react-native-vision-camera/docs/docs/guides/FRAME_PROCESSORS_CREATE_OVERVIEW.mdx

---
id: frame-processors-plugins-overview
title: Creating Frame Processor Plugins
sidebar_label: Overview
---

import useBaseUrl from '@docusaurus/useBaseUrl';

## Overview

Frame Processor Plugins are **native functions** which can be directly called from a JS Frame Processor. (See [Frame Processors](frame-processors))

They **receive a frame from the Camera** as an input and can return any kind of output. For example, a `scanQRCodes` function returns an array of detected QR code strings in the frame:

```tsx {4-5}
function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    const qrCodes = scanQRCodes(frame)
    _log(`QR Codes in Frame: ${qrCodes}`)
  }, [])

  return (
    <Camera frameProcessor={frameProcessor} {...cameraProps} />
  )
}
```

To achieve **maximum performance**, the `scanQRCodes` function is written in a native language (e.g. Objective-C), but it will be directly called from the VisionCamera Frame Processor JavaScript-Runtime.

### Types

The Frame Processor Plugin Registry API automatically manages type conversion from JS <-> native. They are converted into the most efficient data-structures, as seen here:

| JS Type              | Objective-C Type              | Java Type                  |
|----------------------|-------------------------------|----------------------------|
| `number`             | `NSNumber*` (double)          | `double`                   |
| `boolean`            | `NSNumber*` (boolean)         | `boolean`                  |
| `string`             | `NSString*`                   | `String`                   |
| `[]`                 | `NSArray*`                    | `Array<Object>`            |
| `{}`                 | `NSDictionary*`               | `HashMap<Object>`          |
| `undefined` / `null` | `nil`                         | `null`                     |
| `(any, any) => void` | [`RCTResponseSenderBlock`][4] | `(Object, Object) -> void` |
| [`Frame`][1]         | [`Frame*`][2]                 | [`ImageProxy`][3]          |

### Return values

Return values will automatically be converted to JS values, assuming they are representable in the ["Types" table](#types). So the following Objective-C frame processor:

```objc
static inline id detectObject(Frame* frame, NSArray args) {
  return @"cat";
}
```

Returns a `string` in JS:

```js
export function detectObject(frame: Frame): string {
  'worklet'
  const result = __detectObject(frame)
  _log(result) // <-- "cat"
}
```

You can also manipulate the buffer and return it (or a copy of it) by using the [`Frame` class](https://github.com/cuvent/react-native-vision-camera/blob/main/ios/Frame%20Processor/Frame.h):

```objc
#import <VisionCamera/Frame.h>

static inline id resize(Frame* frame, NSArray args) {
  CMSampleBufferRef resizedBuffer = // ...
  
  return [[Frame alloc] initWithBuffer:resizedBuffer orientation:frame.orientation];
}
```

Which returns a [`Frame`](https://github.com/cuvent/react-native-vision-camera/blob/main/src/Frame.ts) in JS:

```js
const frameProcessor = useFrameProcessor((frame) => {
  'worklet';
  // by downscaling the frame, the `detectObjects` function runs faster.
  const resizedFrame = resize(frame, 720, 480)
  const objects = detectObjects(resizedFrame)
  _log(objects)
}, [])
```

### Parameters

Frame Processors can also accept parameters, following the same type convention as [return values](#return-values):

```ts
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const codes = scanCodes(frame, ['qr', 'barcode'])
}, [])
```

Or with multiple parameters:

```ts
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const codes = scanCodes(frame, true, 'hello-world', 42)
}, [])
```

### Long-running Frame Processors

If your Frame Processor takes longer than a single frame interval to execute, or runs asynchronously, you can create a **copy of the frame** and dispatch the actual frame processing to a **separate thread**.

For example, a realtime video chat application might use WebRTC to send the frames to the server. I/O operations (networking) are asynchronous, and we don't _need_ to wait for the upload to succeed before pushing the next frame, so we copy the frame and perform the upload on another Thread.

```objc
static dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0ul);

static inline id sendFrameToWebRTC(Frame* frame, NSArray args) {
  CMSampleBufferRef bufferCopy;
  CMSampleBufferCreateCopy(kCFAllocatorDefault, frame.buffer, &bufferCopy);

  dispatch_async(queue, ^{
    NSString* serverURL = (NSString*)args[0];
    [WebRTC uploadFrame:bufferCopy toServer:serverURL];
  });

  return nil;
}
```

### Async Frame Processors with Event Emitters

You might also run some very complex AI algorithms which are not fast enough to smoothly run at **30 FPS** (**33ms**). To not drop any frames you can create a custom "frame queue" which processes the copied frames and calls back into JS via a React event emitter. For this you'll have to create a Native Module that handles the asynchronous native -> JS communication, see ["Sending events to JavaScript" (Android)](https://reactnative.dev/docs/native-modules-android#sending-events-to-javascript) and ["Sending events to JavaScript" (iOS)](https://reactnative.dev/docs/native-modules-ios#sending-events-to-javascript).

This might look like this for the user:

```tsx
function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    SomeAI.process(frame) // does not block frame processor, runs async
  }, [])

  useEffect(() => {
    SomeAI.addListener((results) => {
      // gets called asynchronously, goes through the React Event Emitter system
      console.log(`AI results: ${results}`)
    })
  }, [])

  return (
    <Camera frameProcessor={frameProcessor} {...cameraProps} />
  )
}
```

This way you can handle queueing up the frames yourself and asynchronously call back into JS at some later point in time using event emitters.

### Benchmarking Frame Processor Plugins

Your Frame Processor Plugins have to be fast. VisionCamera automatically detects slow Frame Processors and outputs relevant information in the native console (Xcode: **Debug Area**, Android Studio: **Logcat**):

<div align="center">
  <img src={useBaseUrl("img/slow-log.png")} width="80%" />
</div>
<div align="center">
  <img src={useBaseUrl("img/slow-log-2.png")} width="80%" />
</div>

<br />

#### 🚀 Create your first Frame Processor Plugin for [iOS](frame-processors-plugins-ios) or [Android](frame-processors-plugins-android)!

[1]: https://github.com/cuvent/react-native-vision-camera/blob/main/src/Frame.ts
[2]: https://github.com/cuvent/react-native-vision-camera/blob/main/ios/Frame%20Processor/Frame.h
[3]: https://developer.android.com/reference/androidx/camera/core/ImageProxy
[4]: https://github.com/facebook/react-native/blob/9a43eac7a32a6ba3164a048960101022a92fcd5a/React/Base/RCTBridgeModule.h#L20-L24