feat: Replace Reanimated with RN Worklets (#1468)
* Setup RN Worklets * Use RN Worklets on iOS * Fix console * Add `installFrameProcessorBindings()` function * Add `FrameProcessorPlugins` proxy (BREAKING CHANGE) * Clean up docs * Update FRAME_PROCESSORS.mdx * Use RN Worklets 0.2.5 * feat: Android build setup * Rewrite Android Frame Processor Part * Update CMakeLists.txt * fix: Add react-native-worklets Gradle dependency * Update Podfile.lock * fix build * gradle:7.4.1 * Init JSI Bindings in method on Android * Fix Folly flags * fix: Init `FrameProcessorRuntimeManager` later * fix: Wrap in `<GestureHandlerRootView>` * Refactor plugins * fix: Remove enableFrameProcessors * Install RN Worklets from current GH master * Update babel.config.js * Update CameraViewModule.kt * Update ImageProxyUtils.java * feat: Upgrade to Reanimated v3 * fix: Fix crash on Worklet init * Update RN Worklets to latest master * fix: Simplify FP Plugins Proxy
This commit is contained in:
Marc Rousavy
GitHub
@@ -48,17 +48,13 @@ Frame processors are by far not limited to Hotdog detection, other examples incl
|
||||
* Creating scanners for **QR codes**, **Barcodes** or even custom codes such as **Snapchat's SnapCodes** or **Apple's AppClips**
|
||||
* Creating **snapchat-like filters**, e.g. draw a dog-mask filter over the user's face
|
||||
* Creating **color filters** with depth-detection
|
||||
* Drawing boxes, text, overlays, or colors on the screen in realtime
|
||||
* Rendering filters and shaders such as Blur, inverted colors, beauty filter, or more on the screen
|
||||
|
||||
Because they are written in JS, Frame Processors are **simple**, **powerful**, **extensible** and **easy to create** while still running at **native performance**. (Frame Processors can run up to **1000 times a second**!) Also, you can use **fast-refresh** to quickly see changes while developing or publish [over-the-air updates](https://github.com/microsoft/react-native-code-push) to tweak the Hotdog detector's sensitivity in live apps without pushing a native update.
|
||||
|
||||
:::note
|
||||
Frame Processors require [**react-native-reanimated**](https://github.com/software-mansion/react-native-reanimated) 2.2.0 or higher. Also make sure to add
|
||||
|
||||
```js
|
||||
import 'react-native-reanimated'
|
||||
```
|
||||
|
||||
to the top of the file when using `useFrameProcessor`.
|
||||
Frame Processors require [**react-native-worklets**](https://github.com/chrfalch/react-native-worklets) 1.0.0 or higher.
|
||||
:::
|
||||
|
||||
### Interacting with Frame Processors
|
||||
@@ -80,7 +76,7 @@ You can also easily read from, and assign to [**Shared Values**](https://docs.sw
|
||||
|
||||
In this example, we detect a cat in the frame - if a cat was found, we assign the `catBounds` Shared Value to the coordinates of the cat (relative to the screen) which we can then use in a `useAnimatedStyle` hook to position the red rectangle surrounding the cat. This updates in realtime on the UI Thread, and can also be smoothly animated with `withSpring` or `withTiming`.
|
||||
|
||||
```tsx {6}
|
||||
```tsx {7}
|
||||
// represents position of the cat on the screen 🐈
|
||||
const catBounds = useSharedValue({ top: 0, left: 0, right: 0, bottom: 0 })
|
||||
|
||||
@@ -108,18 +104,18 @@ return (
|
||||
)
|
||||
```
|
||||
|
||||
And you can also call back to the React-JS thread by using [`runOnJS`](https://docs.swmansion.com/react-native-reanimated/docs/api/miscellaneous/runOnJS/):
|
||||
And you can also call back to the React-JS thread by using `createRunInJsFn(...)`:
|
||||
|
||||
```tsx {9}
|
||||
const onQRCodeDetected = useCallback((qrCode: string) => {
|
||||
```tsx {1}
|
||||
const onQRCodeDetected = Worklets.createRunInJsFn((qrCode: string) => {
|
||||
navigation.push("ProductPage", { productId: qrCode })
|
||||
}, [])
|
||||
})
|
||||
|
||||
const frameProcessor = useFrameProcessor((frame) => {
|
||||
'worklet'
|
||||
const qrCodes = scanQRCodes(frame)
|
||||
if (qrCodes.length > 0) {
|
||||
runOnJS(onQRCodeDetected)(qrCodes[0])
|
||||
onQRCodeDetected(qrCodes[0])
|
||||
}
|
||||
}, [onQRCodeDetected])
|
||||
```
|
||||
@@ -133,23 +129,6 @@ npm i vision-camera-image-labeler
|
||||
cd ios && pod install
|
||||
```
|
||||
|
||||
Then add it to your `babel.config.js`. For the Image Labeler, this will be `__labelImage`:
|
||||
|
||||
```js {6}
|
||||
module.exports = {
|
||||
plugins: [
|
||||
[
|
||||
'react-native-reanimated/plugin',
|
||||
{
|
||||
globals: ['__labelImage'],
|
||||
},
|
||||
],
|
||||
```
|
||||
|
||||
:::note
|
||||
You have to restart metro-bundler for changes in the `babel.config.js` file to take effect.
|
||||
:::
|
||||
|
||||
That's it! 🎉 Now you can use it:
|
||||
|
||||
```ts
|
||||
@@ -192,7 +171,7 @@ If you are using the [react-hooks ESLint plugin](https://www.npmjs.com/package/e
|
||||
|
||||
#### Frame Processors
|
||||
|
||||
**Frame Processors** are JS functions that will be **workletized** using [react-native-reanimated](https://github.com/software-mansion/react-native-reanimated). They are created on a **parallel camera thread** using a separate JavaScript Runtime (_"VisionCamera JS-Runtime"_) and are **invoked synchronously** (using JSI) without ever going over the bridge. In a **Frame Processor** you can write normal JS code, call back to the React-JS Thread (e.g. `setState`), use [Shared Values](https://docs.swmansion.com/react-native-reanimated/docs/fundamentals/shared-values/) and call **Frame Processor Plugins**.
|
||||
**Frame Processors** are JS functions that will be **workletized** using [react-native-worklets](https://github.com/chrfalch/react-native-worklets). They are created on a **parallel camera thread** using a separate JavaScript Runtime (_"VisionCamera JS-Runtime"_) and are **invoked synchronously** (using JSI) without ever going over the bridge. In a **Frame Processor** you can write normal JS code, call back to the React-JS Thread (e.g. `setState`), use [Shared Values](https://docs.swmansion.com/react-native-reanimated/docs/fundamentals/shared-values/) and call **Frame Processor Plugins**.
|
||||
|
||||
> See [**the example Frame Processor**](https://github.com/mrousavy/react-native-vision-camera/blob/cf68a4c6476d085ec48fc424a53a96962e0c33f9/example/src/CameraPage.tsx#L199-L203)
|
||||
|
||||
|
||||
@@ -61,7 +61,7 @@ Returns a `string` in JS:
|
||||
```js
|
||||
export function detectObject(frame: Frame): string {
|
||||
'worklet'
|
||||
const result = __detectObject(frame)
|
||||
const result = FrameProcessorPlugins.detectObject(frame)
|
||||
console.log(result) // <-- "cat"
|
||||
}
|
||||
```
|
||||
|
||||
@@ -9,36 +9,17 @@ sidebar_label: Finish creating your Frame Processor Plugin
|
||||
To make the Frame Processor Plugin available to the Frame Processor Worklet Runtime, create the following wrapper function in JS/TS:
|
||||
|
||||
```ts
|
||||
import type { Frame } from 'react-native-vision-camera'
|
||||
import { FrameProcessorPlugins, Frame } from 'react-native-vision-camera'
|
||||
|
||||
/**
|
||||
* Scans QR codes.
|
||||
*/
|
||||
export function scanQRCodes(frame: Frame): string[] {
|
||||
'worklet'
|
||||
return __scanQRCodes(frame)
|
||||
return FrameProcessorPlugins.scanQRCodes(frame)
|
||||
}
|
||||
```
|
||||
|
||||
Users will then have to add the Frame Processor Plugin's name to their `babel.config.js`.
|
||||
|
||||
For the QR Code Scanner, this will be `__scanQRCodes`:
|
||||
|
||||
```js {6}
|
||||
module.exports = {
|
||||
plugins: [
|
||||
[
|
||||
'react-native-reanimated/plugin',
|
||||
{
|
||||
globals: ['__scanQRCodes'],
|
||||
},
|
||||
],
|
||||
```
|
||||
|
||||
:::note
|
||||
You have to restart metro-bundler for changes in the `babel.config.js` file to take effect.
|
||||
:::
|
||||
|
||||
## Test it!
|
||||
|
||||
Simply call the wrapper Worklet in your Frame Processor:
|
||||
@@ -64,11 +45,10 @@ If you want to distribute your Frame Processor Plugin, simply use npm.
|
||||
1. Create a blank Native Module using [bob](https://github.com/callstack/react-native-builder-bob) or [create-react-native-module](https://github.com/brodybits/create-react-native-module)
|
||||
2. Name it `vision-camera-plugin-xxxxx` where `xxxxx` is the name of your plugin
|
||||
3. Remove the generated template code from the Example Native Module
|
||||
4. Add VisionCamera to `peerDependencies`: `"react-native-vision-camera": ">= 2"`
|
||||
4. Add VisionCamera to `peerDependencies`: `"react-native-vision-camera": ">= 3"`
|
||||
5. Implement the Frame Processor Plugin in the iOS, Android and JS/TS Codebase using the guides above
|
||||
6. Add installation instructions to the `README.md` to let users know they have to add your frame processor in the `babel.config.js` configuration.
|
||||
7. Publish the plugin to npm. Users will only have to install the plugin using `npm i vision-camera-plugin-xxxxx` and add it to their `babel.config.js` file.
|
||||
8. [Add the plugin to the **official VisionCamera plugin list**](https://github.com/mrousavy/react-native-vision-camera/edit/main/docs/docs/guides/FRAME_PROCESSOR_PLUGIN_LIST.mdx) for more visibility
|
||||
6. Publish the plugin to npm. Users will only have to install the plugin using `npm i vision-camera-plugin-xxxxx` and add it to their `babel.config.js` file.
|
||||
7. [Add the plugin to the **official VisionCamera plugin list**](https://github.com/mrousavy/react-native-vision-camera/edit/main/docs/docs/guides/FRAME_PROCESSOR_PLUGIN_LIST.mdx) for more visibility
|
||||
|
||||
<br />
|
||||
|
||||
|
||||
@@ -73,7 +73,7 @@ public class QRCodeFrameProcessorPlugin extends FrameProcessorPlugin {
|
||||
```
|
||||
|
||||
:::note
|
||||
The JS function name will be equal to the name you pass to the `super(...)` call (with a `__` prefix). Make sure it is unique across other Frame Processor Plugins.
|
||||
The Frame Processor Plugin will be exposed to JS through the `FrameProcessorPlugins` object using the name you pass to the `super(...)` call. In this case, it would be `FrameProcessorPlugins.scanQRCodes(...)`.
|
||||
:::
|
||||
|
||||
4. **Implement your Frame Processing.** See the [Example Plugin (Java)](https://github.com/mrousavy/react-native-vision-camera/blob/main/example/android/app/src/main/java/com/mrousavy/camera/example/ExampleFrameProcessorPlugin.java) for reference.
|
||||
@@ -137,7 +137,7 @@ class ExampleFrameProcessorPluginKotlin: FrameProcessorPlugin("scanQRCodes") {
|
||||
```
|
||||
|
||||
:::note
|
||||
The JS function name will be equal to the name you pass to the `FrameProcessorPlugin(...)` call (with a `__` prefix). Make sure it is unique across other Frame Processor Plugins.
|
||||
The Frame Processor Plugin will be exposed to JS through the `FrameProcessorPlugins` object using the name you pass to the `FrameProcessorPlugin(...)` call. In this case, it would be `FrameProcessorPlugins.scanQRCodes(...)`.
|
||||
:::
|
||||
|
||||
4. **Implement your Frame Processing.** See the [Example Plugin (Java)](https://github.com/mrousavy/react-native-vision-camera/blob/main/example/android/app/src/main/java/com/mrousavy/camera/example/ExampleFrameProcessorPlugin.java) for reference.
|
||||
|
||||
@@ -16,7 +16,7 @@ iOS Frame Processor Plugins can be written in either **Objective-C** or **Swift*
|
||||
|
||||
### Automatic setup
|
||||
|
||||
Run [Vision Camera Plugin Builder CLI](https://github.com/mateusz1913/vision-camera-plugin-builder),
|
||||
Run [Vision Camera Plugin Builder CLI](https://github.com/mateusz1913/vision-camera-plugin-builder),
|
||||
|
||||
```sh
|
||||
npx vision-camera-plugin-builder ios
|
||||
@@ -63,7 +63,7 @@ VISION_EXPORT_FRAME_PROCESSOR(scanQRCodes)
|
||||
```
|
||||
|
||||
:::note
|
||||
The JS function name will be equal to the Objective-C function name you choose (with a `__` prefix). Make sure it is unique across other Frame Processor Plugins.
|
||||
The Frame Processor Plugin will be exposed to JS through the `FrameProcessorPlugins` object using the same name as the Objective-C function. In this case, it would be `FrameProcessorPlugins.scanQRCodes(...)`.
|
||||
:::
|
||||
|
||||
4. **Implement your Frame Processing.** See the [Example Plugin (Objective-C)](https://github.com/mrousavy/react-native-vision-camera/blob/main/example/ios/Frame%20Processor%20Plugins/Example%20Plugin%20%28Objective%2DC%29) for reference.
|
||||
|
||||
@@ -12,13 +12,10 @@ These are VisionCamera Frame Processor Plugins created by the community.
|
||||
|
||||
```
|
||||
npm i vision-camera-xxxxx
|
||||
cd ios && pod install
|
||||
```
|
||||
|
||||
2. Add the native function's name (the one with the `__` prefix) to your `babel.config.js`. (See their README for instructions)
|
||||
|
||||
:::note
|
||||
You have to restart metro-bundler for changes in the `babel.config.js` file to take effect.
|
||||
:::
|
||||
2. Rebuild your app and use the plugin
|
||||
|
||||
## Plugin List
|
||||
|
||||
|
||||
@@ -44,7 +44,7 @@ expo install react-native-vision-camera
|
||||
|
||||
VisionCamera requires **iOS 11 or higher**, and **Android-SDK version 21 or higher**. See [Troubleshooting](/docs/guides/troubleshooting) if you're having installation issues.
|
||||
|
||||
> **(Optional)** If you want to use [**Frame Processors**](/docs/guides/frame-processors), you need to install [**react-native-reanimated**](https://github.com/software-mansion/react-native-reanimated) 2.2.0 or higher.
|
||||
> **(Optional)** If you want to use [**Frame Processors**](/docs/guides/frame-processors), you need to install [**react-native-worklets**](https://github.com/chrfalch/react-native-worklets) 1.0.0 or higher.
|
||||
|
||||
## Updating manifests
|
||||
|
||||
|
||||
@@ -6,5 +6,5 @@ This is an internal TODO list which I am using to keep track of some of the feat
|
||||
* [ ] Allow camera switching (front <-> back) while recording and stich videos together
|
||||
* [ ] Make `startRecording()` async. Due to NativeModules limitations, we can only have either one callback or one promise in a native function. For `startRecording()` we need both, since you probably also want to catch any errors that occured during a `startRecording()` call (or wait until the recording has actually started, since this can also take some time)
|
||||
* [ ] Return a `jsi::Value` reference for images (`UIImage`/`Bitmap`) on `takePhoto()` and `takeSnapshot()`. This way, we skip the entire file writing and reading, making image capture _a lot_ faster.
|
||||
* [ ] Implement frame processors. The idea here is that the user passes a small JS function (reanimated worklet) to the `Camera::frameProcessor` prop which will then get called on every frame the camera previews. (I'd say we cap it to 30 times per second, even if the camera fps is higher) This can then be used to scan QR codes, detect faces, detect depth, render something ontop of the camera such as color filters, QR code boundaries or even dog filters, possibly even use AR - all from a single, small, and highly flexible JS function!
|
||||
* [ ] Implement frame processors. The idea here is that the user passes a small JS function (worklet) to the `Camera::frameProcessor` prop which will then get called on every frame the camera previews. (I'd say we cap it to 30 times per second, even if the camera fps is higher) This can then be used to scan QR codes, detect faces, detect depth, render something ontop of the camera such as color filters, QR code boundaries or even dog filters, possibly even use AR - all from a single, small, and highly flexible JS function!
|
||||
* [ ] Create a custom MPEG4 encoder to allow for more customizability in `recordVideo()` (`bitRate`, `priority`, `minQuantizationParameter`, `allowFrameReordering`, `expectedFrameRate`, `realTime`, `minimizeMemoryUsage`)
|
||||
|
||||
Reference in New Issue
Block a user