react-native-vision-camera/docs/docs/guides/FRAME_PROCESSORS_CREATE_OVERVIEW.mdx

---
id: frame-processors-plugins-overview
title: Creating Frame Processor Plugins
sidebar_label: Overview
---

import useBaseUrl from '@docusaurus/useBaseUrl';

## Overview

Frame Processor Plugins are **native functions** which can be directly called from a JS Frame Processor. (See [Frame Processors](frame-processors))

They **receive a frame from the Camera** as an input and can return any kind of output. For example, a `scanQRCodes` function returns an array of detected QR code strings in the frame:

```tsx {4-5}
function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    const qrCodes = scanQRCodes(frame)
    _log(`QR Codes in Frame: ${qrCodes}`)
  }, [])

  return (
    <Camera frameProcessor={frameProcessor} {...cameraProps} />
  )
}
```

To achieve **maximum performance**, the `scanQRCodes` function is written in a native language (e.g. Objective-C), but it will be directly called from the VisionCamera Frame Processor JavaScript-Runtime.

### Types

The Frame Processor Plugin Registry API automatically manages type conversion from JS <-> native. They are converted into the most efficient data-structures, as seen here:

| JS Type              | Objective-C Type         | Java Type                  |
|----------------------|--------------------------|----------------------------|
| `number`             | `NSNumber` (double)      | `double`                   |
| `boolean`            | `NSNumber` (boolean)     | `boolean`                  |
| `string`             | `NSString`               | `String`                   |
| `[]`                 | `NSArray`                | `Array<Object>`            |
| `{}`                 | `NSDictionary`           | `HashMap<Object>`          |
| `undefined`          | `nil` / `NSNull`         | `null`                     |
| `(any, any) => void` | `RCTResponseSenderBlock` | `(Object, Object) -> void` |

### Return values

Return values will automatically be converted to JS values, assuming they are representable in the ["Types" table](#types). So the following Objective-C frame processor:

```objc
static inline id detectObject(CMSampleBufferRef buffer, NSArray args) {
  return @"cat";
}
```

Returns a `string` in JS:

```js
export function detectObject(frame: Frame): string {
  'worklet';
  const result =  __detectObject(frame);
  _log(result) // <-- "cat"
}
```

### Parameters

Frame Processors can also accept parameters, following the same type convention as [return values](#return-values):

```ts
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const codes = scanCodes(frame, ['qr', 'barcode'])
}, [])
```

Or with multiple parameters:

```ts
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const codes = scanCodes(frame, true, 'hello-world', 42)
}, [])
```

### Long-running Frame Processors

If your Frame Processor takes longer than a single frame interval to execute, or runs asynchronously, you can create a **copy of the frame** and dispatch the actual frame processing to a **separate thread**.

For example, a realtime video chat application might use WebRTC to send the frames to the server. I/O operations (networking) are asynchronous, and we don't _need_ to wait for the upload to succeed before pushing the next frame, so we copy the frame and perform the upload on another Thread.

```objc
static dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0ul);

static inline id sendFrameToWebRTC(CMSampleBufferRef buffer, NSArray args) {
  CMSampleBufferRef bufferCopy;
  CMSampleBufferCreateCopy(kCFAllocatorDefault, buffer, &bufferCopy);

  dispatch_async(queue, ^{
    NSString* serverURL = (NSString*)args[0];
    [WebRTC uploadFrame:bufferCopy toServer:serverURL];
  });

  return nil;
}
```

### Async Frame Processors with Event Emitters

You might also run some very complex AI algorithms which are not fast enough to smoothly run at **30 FPS** (**33ms**). To not drop any frames you can create a custom "frame queue" which processes the copied frames and calls back into JS via a React event emitter. For this you'll have to create a Native Module that handles the asynchronous native -> JS communication, see ["Sending events to JavaScript" (Android)](https://reactnative.dev/docs/native-modules-android#sending-events-to-javascript) and ["Sending events to JavaScript" (iOS)](https://reactnative.dev/docs/native-modules-ios#sending-events-to-javascript).

This might look like this for the user:

```tsx
function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    SomeAI.process(frame) // does not block frame processor, runs async
  }, [])

  useEffect(() => {
    SomeAI.addListener((results) => {
      // gets called asynchronously, goes through the React Event Emitter system
      console.log(`AI results: ${results}`)
    })
  }, [])

  return (
    <Camera frameProcessor={frameProcessor} {...cameraProps} />
  )
}
```

This way you can handle queueing up the frames yourself and asynchronously call back into JS at some later point in time using event emitters.

### Benchmarking Frame Processor Plugins

Your Frame Processor Plugins have to be fast. VisionCamera automatically detects slow Frame Processors and outputs relevant information in the native console (Xcode: **Debug Area**, Android Studio: **Logcat**):

<div align="center">
  <img src={useBaseUrl("img/slow-log.png")} width="80%" />
</div>
<div align="center">
  <img src={useBaseUrl("img/slow-log-2.png")} width="80%" />
</div>

<br />

#### 🚀 Create your first Frame Processor Plugin for [iOS](frame-processors-plugins-ios) or [Android](frame-processors-plugins-android)!