feature: Frame Processors (iOS) (#2)

* Clean up Frame Processor * Create FrameProcessorHolder * Create FrameProcessorDelegate in ObjC++ * Move frame processor to FrameProcessorDelegate * Decorate runtime, check for null * Update FrameProcessorDelegate.mm * Cleanup FrameProcessorBindings.mm * Fix RuntimeDecorator.h import * Update FrameProcessorDelegate.mm * "React" -> "React Helper" to avoid confusion * Rename folders again * Fix podspec flattening a lot of headers, causing REA nameclash * Fix header imports to avoid REA naming collision * Lazily initialize jsi::Runtime on DispatchQueue * Install frame processor bindings from Swift * First try to call jsi::Function (frame processor) 👀 * Call viewForReactTag on RCT main thread * Fix bridge accessing * Add more logs * Update CameraViewManager.swift * Add more TODOs * Re-indent .cpp files * Fix RCTTurboModule import podspec * Remove unnecessary include check for swift umbrella header * Merge branch 'main' into frame-processors * Docs: use static width for images (283) * Create validate-cpp.yml * Update a lot of packages to latest * Set SWIFT_VERSION to 5.2 in podspec * Create clean.sh * Delete unused C++ files * podspec: Remove CLANG_CXX_LANGUAGE_STANDARD and OTHER_CFLAGS * Update pod lockfiles * Regenerate lockfiles * Remove IOSLogger * Use NSLog * Create FrameProcessorManager (inherits from REA RuntimeManager) * Create reanimated::RuntimeManager shared_ptr * Re-integrate pods * Add react-native-reanimated >=2 peerDependency * Add metro-config * blacklist -> exclusionList * Try to call worklet * Fix jsi::Value* initializer * Call ShareableValue::adapt (makeShareable) with React/JS Runtime * Add null-checks * Lift runtime manager creation out of delegate, into bindings * Remove debug statement * Make RuntimeManager unique_ptr * Set _FRAME_PROCESSOR * Extract convertJSIFunctionToFrameProcessorCallback * Print frame * Merge branch 'main' into frame-processors * Reformat Swift code * Install reanimated from npm again * Re-integrate Pods * Dependabot: Also scan example/ and docs/ * Update validate-cpp.yml * Create FrameProcessorUtils * Create Frame.h * Abstract HostObject creation away * Fix types * Fix frame processor call * Add todo * Update lockfiles * Add C++ contributing instructions * Update CONTRIBUTING.md * Add android/src/main/cpp to cpplint * Update cpplint.sh * Fix a few cpplint errors * Fix globals * Fix a few more cpplint errors * Update App.tsx * Update AndroidLogger.cpp * Format * Fix cpplint script (check-cpp) * Try to simplify frame processor * y * Update FrameProcessorUtils.mm * Update FrameProcessorBindings.mm * Update CameraView.swift * Update CameraViewManager.m * Restructure everything * fix * Fix `@objc` export (make public) * Refactor installFrameProcessorBindings into FrameProcessorRuntimeManager * Add swift RCTBridge.runOnJS helper * Fix run(onJS) * Add pragma once * Add `&self` to lambda * Update FrameProcessorRuntimeManager.mm * reorder imports * Fix imports * forward declare * Rename extension * Destroy buffer after execution * Add FrameProcessorPluginRegistry base * Merge branch 'main' into frame-processors * Add frameProcessor to types * Update Camera.tsx * Fix rebase merge * Remove movieOutput * Use `useFrameProcessor` * Fix bad merge * Add additional ESLint rules * Update lockfiles * Update CameraViewManager.m * Add support for V8 runtime * Add frame processor plugins API * Print plugin invoke * Fix React Utils in podspec * Fix runOnJS swift name * Remove invalid redecl of `captureSession` * Use REA 2.1.0 which includes all my big PRs 🎉 * Update validate-cpp.yml * Update Podfile.lock * Remove Flipper * Fix dereferencing * Capture `self` by value. Fucking hell, what a dumb mistake. * Override a few HostObject functions * Expose isReady, width, height, bytesPerRow and planesCount * use hook again * Expose property names * FrameProcessor -> Frame * Update CameraView+RecordVideo.swift * Add Swift support for Frame Processors Plugins * Add macros for plugin installation * Add ObjC frame processor plugin * Correctly install frame processor plugins * Don't require custom name for macro * Check if plugin already exists * Implement QR Code Frame Processor Plugin in Swift * Adjust ObjC style frame processor macro * optimize * Add `frameProcessorFrameDropRate` * Fix types * Only log once * Log if it executes slowly * Implement `frameProcessorFps` * Implement manual encoded video recordings * Use recommended video settings * Add fileType types * Ignore if input is not ready for media data * Add completion handler * Add audio buffer sampling * Init only for video frame * use AVAssetWriterInputPixelBufferAdaptor * Remove AVAssetWriterInputPixelBufferAdaptor * Rotate VideoWriter * Always assume portrait orientation * Update RecordingSession.swift * Use a separate Queue for Audio * Format Swift * Update CameraView+RecordVideo.swift * Use `videoQueue` instead of `cameraQueue` * Move example plugins to example app * Fix hardcoded name in plugin macro * QRFrame... -> QRCodeFrame... * Update FrameProcessorPlugin.h * Add example frame processors to JS base * Update QRCodeFrameProcessorPluginSwift.m * Add docs to create FP Plugins * Update FRAME_PROCESSORS_CREATE.mdx * Update FRAME_PROCESSORS_CREATE.mdx * Use `AVAssetWriterInputPixelBufferAdaptor` for efficient pixel buffer recycling * Add customizable `pixelFormat` * Use native format if available * Update project.pbxproj * Set video width and height as source-pixel-buffer attributes * Catch * Update App.tsx * Don't explicitly set video dimensions, let CVPixelBufferPool handle it * Add a few logs * Cleanup * Update CameraView+RecordVideo.swift * Eagerly initialize asset writer to fix stutter at first frame * Use `cameraQueue` DispatchQueue to not block CaptureDataOutputDelegate * Fix duration calculation * cleanup * Cleanup * Swiftformat * Return available video codecs * Only show frame drop notification for video output * Remove photo and video codec functionality It was too much complexity and probably never used anyways. * Revert all android related changes for now * Cleanup * Remove unused header * Update AVAssetWriter.Status+descriptor.swift * Only call Frame Processor for Video Frames * Fix `if` * Add support for Frame Processor plugin parameters/arguments * Fix arg support * Move to JSIUtils.mm * Update JSIUtils.h * Update FRAME_PROCESSORS_CREATE.mdx * Update FRAME_PROCESSORS_CREATE.mdx * Upgrade packages for docs/ * fix docs * Rename * highlight lines * docs * community plugins * Update FRAME_PROCESSOR_CREATE_FINAL.mdx * Update FRAME_PROCESSOR_PLUGIN_LIST.mdx * Update FRAME_PROCESSOR_PLUGIN_LIST.mdx * Update dependencies (1/2) * Update dependencies (2/2) * Update Gemfile.lock * add FP docs * Update README.md * Make `lastFrameProcessor` private * add `frameProcessor` docs * fix docs * adjust docs * Update DEVICES.mdx * fix * s * Add logs demo * add metro restart note * Update FRAME_PROCESSOR_CREATE_PLUGIN_IOS.mdx * Mirror video device * Update AVCaptureVideoDataOutput+mirror.swift * Create .swift-version * Enable whole module optimization * Fix recording mirrored video * Swift format * Clean dictionary on `markInvalid` * Fix cleanup * Add docs for disabling frame processors * Update project.pbxproj * Revert "Update project.pbxproj" This reverts commit e67861e51b88b4888a6940e2d20388f3044211d0. * Log frame drop reason * Format * add more samples * Add clang-format * also check .mm * Revert "also check .mm" This reverts commit 8b9d5e2c29866b05909530d104f6633d6c49eadd. * Revert "Add clang-format" This reverts commit 7643ac808e0fc34567ea1f814e73d84955381636. * Use `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange` as default * Read matching video attributes from videoSettings * Add TODO * Swiftformat * Conditionally disable frame processors * Assert if trying to use frame processors when disabled * Add frame-processors demo gif * Allow disabling frame processors via `VISION_CAMERA_DISABLE_FRAME_PROCESSORS` * Update FrameProcessorRuntimeManager.mm * Update FRAME_PROCESSORS.mdx * Update project.pbxproj * Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
2021-05-06 14:11:55 +02:00
parent 77b3d78566
commit b6a67d5ced
100 changed files with 4703 additions and 3664 deletions
--- a/Processor/Frame.h
+++ b/Processor/Frame.h
@@ -0,0 +1,20 @@
+//
+//  Frame.h
+//  VisionCamera
+//
+//  Created by Marc Rousavy on 15.03.21.
+//  Copyright © 2021 Facebook. All rights reserved.
+//
+
+#pragma once
+
+#import <CoreMedia/CMSampleBuffer.h>
+
+// TODO: Make this Objective-C so it can be imported in Swift?
+class Frame {
+ public:
+  explicit Frame(CMSampleBufferRef buffer): buffer(buffer) {}
+
+ public:
+  CMSampleBufferRef buffer;
+};
--- a/Processor/FrameHostObject.h
+++ b/Processor/FrameHostObject.h
@@ -0,0 +1,26 @@
+//
+//  FrameHostObject.h
+//  VisionCamera
+//
+//  Created by Marc Rousavy on 22.03.21.
+//  Copyright © 2021 Facebook. All rights reserved.
+//
+
+#pragma once
+
+#import "Frame.h"
+#import <jsi/jsi.h>
+#import <CoreMedia/CMSampleBuffer.h>
+
+using namespace facebook;
+
+class JSI_EXPORT FrameHostObject: public Frame, public jsi::HostObject {
+public:
+  explicit FrameHostObject(CMSampleBufferRef buffer): Frame(buffer) {}
+  ~FrameHostObject();
+  
+public:
+  jsi::Value get(jsi::Runtime&, const jsi::PropNameID& name) override;
+  std::vector<jsi::PropNameID> getPropertyNames(jsi::Runtime& rt) override;
+  void destroyBuffer();
+};
--- a/Processor/FrameHostObject.mm
+++ b/Processor/FrameHostObject.mm
@@ -0,0 +1,93 @@
+//
+//  FrameHostObject.m
+//  VisionCamera
+//
+//  Created by Marc Rousavy on 22.03.21.
+//  Copyright © 2021 Facebook. All rights reserved.
+//
+
+#import "FrameHostObject.h"
+#import <Foundation/Foundation.h>
+#import <jsi/jsi.h>
+
+std::vector<jsi::PropNameID> FrameHostObject::getPropertyNames(jsi::Runtime& rt) {
+  std::vector<jsi::PropNameID> result;
+  result.push_back(jsi::PropNameID::forUtf8(rt, std::string("toString")));
+  result.push_back(jsi::PropNameID::forUtf8(rt, std::string("isValid")));
+  result.push_back(jsi::PropNameID::forUtf8(rt, std::string("isReady")));
+  result.push_back(jsi::PropNameID::forUtf8(rt, std::string("width")));
+  result.push_back(jsi::PropNameID::forUtf8(rt, std::string("height")));
+  result.push_back(jsi::PropNameID::forUtf8(rt, std::string("bytesPerRow")));
+  result.push_back(jsi::PropNameID::forUtf8(rt, std::string("planesCount")));
+  result.push_back(jsi::PropNameID::forUtf8(rt, std::string("buffer")));
+  return result;
+}
+
+jsi::Value FrameHostObject::get(jsi::Runtime& runtime, const jsi::PropNameID& propName) {
+  auto name = propName.utf8(runtime);
+  
+  
+  if (name == "Symbol.toPrimitive") {
+    // not implemented
+    return jsi::Value::undefined();
+  }
+  if (name == "valueOf") {
+    // not implemented
+    return jsi::Value::undefined();
+  }
+  if (name == "toString") {
+    auto toString = [this] (jsi::Runtime& runtime, const jsi::Value& thisValue, const jsi::Value* arguments, size_t count) -> jsi::Value {
+      auto imageBuffer = CMSampleBufferGetImageBuffer(buffer);
+      auto width = CVPixelBufferGetWidth(imageBuffer);
+      auto height = CVPixelBufferGetHeight(imageBuffer);
+      
+      NSMutableString* string = [NSMutableString stringWithFormat:@"%lu x %lu Frame", width, height];
+      return jsi::String::createFromUtf8(runtime, string.UTF8String);
+    };
+    return jsi::Function::createFromHostFunction(runtime, jsi::PropNameID::forUtf8(runtime, "toString"), 0, toString);
+  }
+  
+  if (name == "isValid") {
+    auto isValid = buffer != nil && CMSampleBufferIsValid(buffer);
+    return jsi::Value(isValid);
+  }
+  if (name == "isReady") {
+    auto isReady = buffer != nil && CMSampleBufferDataIsReady(buffer);
+    return jsi::Value(isReady);
+  }
+  if (name == "width") {
+    auto imageBuffer = CMSampleBufferGetImageBuffer(buffer);
+    auto width = CVPixelBufferGetWidth(imageBuffer);
+    return jsi::Value((double) width);
+  }
+  if (name == "height") {
+    auto imageBuffer = CMSampleBufferGetImageBuffer(buffer);
+    auto height = CVPixelBufferGetHeight(imageBuffer);
+    return jsi::Value((double) height);
+  }
+  if (name == "bytesPerRow") {
+    auto imageBuffer = CMSampleBufferGetImageBuffer(buffer);
+    auto bytesPerRow = CVPixelBufferGetPlaneCount(imageBuffer);
+    return jsi::Value((double) bytesPerRow);
+  }
+  if (name == "planesCount") {
+    auto imageBuffer = CMSampleBufferGetImageBuffer(buffer);
+    auto planesCount = CVPixelBufferGetPlaneCount(imageBuffer);
+    return jsi::Value((double) planesCount);
+  }
+  if (name == "buffer") {
+    // TODO: Actually return the pixels of the buffer. Not sure if this will be a huge performance hit or not
+    return jsi::Array(runtime, 0);
+  }
+  
+  return jsi::Value::undefined();
+}
+
+FrameHostObject::~FrameHostObject() {
+  destroyBuffer();
+}
+
+void FrameHostObject::destroyBuffer() {
+  // ARC will hopefully delete it lol
+  this->buffer = nil;
+}
--- a/Processor/FrameProcessorCallback.h
+++ b/Processor/FrameProcessorCallback.h
@@ -0,0 +1,14 @@
+//
+//  FrameProcessorCallback.h
+//  VisionCamera
+//
+//  Created by Marc Rousavy on 11.03.21.
+//  Copyright © 2021 Facebook. All rights reserved.
+//
+
+#pragma once
+
+#import <Foundation/Foundation.h>
+#import <CoreMedia/CMSampleBuffer.h>
+
+typedef void (^FrameProcessorCallback) (CMSampleBufferRef buffer);
--- a/Processor/FrameProcessorPlugin.h
+++ b/Processor/FrameProcessorPlugin.h
@@ -0,0 +1,63 @@
+//
+//  FrameProcessorPlugin.h
+//  VisionCamera
+//
+//  Created by Marc Rousavy on 01.05.21.
+//  Copyright © 2021 Facebook. All rights reserved.
+//
+
+#ifndef FrameProcessorPlugin_h
+#define FrameProcessorPlugin_h
+
+#import <Foundation/Foundation.h>
+#import "FrameProcessorPluginRegistry.h"
+#import <CoreMedia/CMSampleBuffer.h>
+
+@protocol FrameProcessorPluginBase
+ (id) callback:(CMSampleBufferRef)buffer withArgs:(NSArray<id>*)args;
+@end
+
+
+#define VISION_CONCAT2(A, B) A##B
+#define VISION_CONCAT(A, B) VISION_CONCAT2(A, B)
+
+/**
+ * Use this Macro to register the given function as a Frame Processor.
+ * * Make sure the given function is a C-style function with the following signature: static inline id callback(CMSampleBufferRef buffer)
+ * * Make sure the given function's name is unique across other frame processor plugins
+ * * Make sure your frame processor returns a Value that can be converted to JS
+ * * Make sure to use this Macro in an @implementation, not @interface
+ *
+ * The JS function will have the same name as the given Objective-C function, but with a "__" prefix.
+ * Make sure to add that function to the babel.config.js under reanimated's "globals" option, and add TypeScript type declarations.
+ */
+#define VISION_EXPORT_FRAME_PROCESSOR(frame_processor)                              \
+                                                                                    \
+(void)load                                                                         \
+{                                                                                   \
+  [FrameProcessorPluginRegistry addFrameProcessorPlugin:@"__" @ #frame_processor callback:^id(CMSampleBufferRef buffer, NSArray<id>* args) { \
+    return frame_processor(buffer, args);                                           \
+  }];                                                                               \
+}
+
+
+/**
+ * Same as VISION_EXPORT_FRAME_PROCESSOR, but uses __attribute__((constructor)) for
+ * registration. Useful for registering swift classes that forbids use of +(void)load.
+ */
+#define VISION_EXPORT_SWIFT_FRAME_PROCESSOR(name, objc_name) \
+objc_name : NSObject<FrameProcessorPluginBase>                                      \
+@end                                                                                \
+                                                                                    \
+@interface objc_name (FrameProcessorPlugin)                                         \
+@end                                                                                \
+@implementation objc_name (FrameProcessorPlugin)                                    \
+                                                                                    \
+__attribute__((constructor)) static void VISION_CONCAT(initialize_, objc_name)()    \
+{                                                                                   \
+  [FrameProcessorPluginRegistry addFrameProcessorPlugin:@"__" @ #name callback:^id(CMSampleBufferRef buffer, NSArray<id>* args) {    \
+    return [objc_name callback:buffer withArgs:args];                               \
+  }];                                                                               \
+}
+
+#endif /* FrameProcessorPlugin_h */
--- a/Processor/FrameProcessorPluginRegistry.h
+++ b/Processor/FrameProcessorPluginRegistry.h
@@ -0,0 +1,23 @@
+//
+//  FrameProcessorPluginRegistry.h
+//  VisionCamera
+//
+//  Created by Marc Rousavy on 24.03.21.
+//  Copyright © 2021 Facebook. All rights reserved.
+//
+
+#pragma once
+
+#import <Foundation/Foundation.h>
+#import <CoreMedia/CMSampleBuffer.h>
+
+typedef id (^FrameProcessorPlugin) (CMSampleBufferRef buffer, NSArray<id>* arguments);
+
+@interface FrameProcessorPluginRegistry : NSObject
+
+ (NSMutableDictionary<NSString*, FrameProcessorPlugin>*)frameProcessorPlugins;
+ (void) addFrameProcessorPlugin:(NSString*)name callback:(FrameProcessorPlugin)callback;
+
+ (void) markInvalid;
+
+@end
--- a/Processor/FrameProcessorPluginRegistry.mm
+++ b/Processor/FrameProcessorPluginRegistry.mm
@@ -0,0 +1,37 @@
+//
+//  FrameProcessorPluginRegistry.mm
+//  VisionCamera
+//
+//  Created by Marc Rousavy on 24.03.21.
+//  Copyright © 2021 Facebook. All rights reserved.
+//
+
+#import "FrameProcessorPluginRegistry.h"
+#import <Foundation/Foundation.h>
+
+@implementation FrameProcessorPluginRegistry
+
+ (NSMutableDictionary<NSString*, FrameProcessorPlugin>*)frameProcessorPlugins {
+  static NSMutableDictionary<NSString*, FrameProcessorPlugin>* plugins = nil;
+  if (plugins == nil) {
+    plugins = [[NSMutableDictionary alloc] init];
+  }
+  return plugins;
+}
+
+static BOOL _isValid = YES;
+ (void) markInvalid {
+  _isValid = NO;
+  [[FrameProcessorPluginRegistry frameProcessorPlugins] removeAllObjects];
+}
+
+ (void) addFrameProcessorPlugin:(NSString*)name callback:(FrameProcessorPlugin)callback {
+  NSAssert(_isValid, @"Tried to add Frame Processor Plugin but Frame Processor Registry has already registered all plugins!");
+  
+  BOOL alreadyExists = [[FrameProcessorPluginRegistry frameProcessorPlugins] valueForKey:name] != nil;
+  NSAssert(!alreadyExists, @"Tried to two Frame Processor Plugins with the same name! Either choose unique names, or remove the unused plugin.");
+  
+  [[FrameProcessorPluginRegistry frameProcessorPlugins] setValue:callback forKey:name];
+}
+
+@end
--- a/Processor/FrameProcessorRuntimeManager.h
+++ b/Processor/FrameProcessorRuntimeManager.h
@@ -0,0 +1,26 @@
+//
+//  FrameProcessorRuntimeManager.h
+//  VisionCamera
+//
+//  Created by Marc Rousavy on 23.03.21.
+//  Copyright © 2021 Facebook. All rights reserved.
+//
+
+#pragma once
+
+#import <Foundation/Foundation.h>
+#import <React/RCTBridge.h>
+
+@interface FrameProcessorRuntimeManager : NSObject
+
+- (instancetype)init NS_UNAVAILABLE;
+
+/**
+ Initializes the Frame Processor Runtime Manager with the given bridge.
+ This init is not thread safe, so only init this on the Thread you want the runtime to run on.
+ */
+- (instancetype) initWithBridge:(RCTBridge*)bridge;
+
+- (void) installFrameProcessorBindings;
+
+@end
--- a/Processor/FrameProcessorRuntimeManager.mm
+++ b/Processor/FrameProcessorRuntimeManager.mm
@@ -0,0 +1,184 @@
+//
+//  FrameProcessorRuntimeManager.m
+//  VisionCamera
+//
+//  Created by Marc Rousavy on 23.03.21.
+//  Copyright © 2021 Facebook. All rights reserved.
+//
+
+#import <Foundation/Foundation.h>
+#import "FrameProcessorRuntimeManager.h"
+#import "FrameProcessorPluginRegistry.h"
+#import "FrameHostObject.h"
+
+#import <memory>
+
+#import <React/RCTBridge.h>
+#import <ReactCommon/RCTTurboModule.h>
+#import <React/RCTBridge+Private.h>
+#import <React/RCTUIManager.h>
+#import <ReactCommon/RCTTurboModuleManager.h>
+
+#if __has_include(<RNReanimated/NativeReanimatedModule.h>)
+  #if __has_include(<RNReanimated/RuntimeManager.h>)
+    #ifndef VISION_CAMERA_DISABLE_FRAME_PROCESSORS
+      #import <RNReanimated/RuntimeManager.h>
+      #import <RNReanimated/RuntimeDecorator.h>
+      #import <RNReanimated/REAIOSScheduler.h>
+      #import <RNReanimated/REAIOSErrorHandler.h>
+      #define ENABLE_FRAME_PROCESSORS
+    #endif
+  #else
+    #warning Your react-native-reanimated version is not compatible with VisionCamera, Frame Processors are disabled. Make sure you're using reanimated 2.1.0 or above!
+  #endif
+#else
+  #warning The NativeReanimatedModule.h header could not be found, Frame Processors are disabled. If you want to use Frame Processors, make sure you install react-native-reanimated!
+#endif
+
+#import "../../cpp/MakeJSIRuntime.h"
+#import "FrameProcessorUtils.h"
+#import "FrameProcessorCallback.h"
+#import "../React Utils/JSIUtils.h"
+
+// Forward declarations for the Swift classes
+__attribute__((objc_runtime_name("_TtC12VisionCamera12CameraQueues")))
+@interface CameraQueues : NSObject
+@property (nonatomic, class, readonly, strong) dispatch_queue_t _Nonnull videoQueue;
+@end
+__attribute__((objc_runtime_name("_TtC12VisionCamera10CameraView")))
+@interface CameraView : UIView
+@property (nonatomic, copy) FrameProcessorCallback _Nullable frameProcessorCallback;
+@end
+
+@implementation FrameProcessorRuntimeManager {
+#ifdef ENABLE_FRAME_PROCESSORS
+  std::unique_ptr<reanimated::RuntimeManager> runtimeManager;
+#endif
+  __weak RCTBridge* weakBridge;
+}
+
+- (instancetype) initWithBridge:(RCTBridge*)bridge {
+  self = [super init];
+  if (self) {
+#ifdef ENABLE_FRAME_PROCESSORS
+    NSLog(@"FrameProcessorBindings: Creating Runtime Manager...");
+    weakBridge = bridge;
+    auto runtime = vision::makeJSIRuntime();
+    reanimated::RuntimeDecorator::decorateRuntime(*runtime, "FRAME_PROCESSOR");
+    runtime->global().setProperty(*runtime, "_FRAME_PROCESSOR", jsi::Value(true));
+    auto callInvoker = bridge.jsCallInvoker;
+    auto scheduler = std::make_shared<reanimated::REAIOSScheduler>(callInvoker);
+    runtimeManager = std::make_unique<reanimated::RuntimeManager>(std::move(runtime),
+                                                                  std::make_shared<reanimated::REAIOSErrorHandler>(scheduler),
+                                                                  scheduler);
+    NSLog(@"FrameProcessorBindings: Runtime Manager created!");
+    
+    NSLog(@"FrameProcessorBindings: Installing Frame Processor plugins...");
+    auto& visionRuntime = *runtimeManager->runtime;
+    auto visionGlobal = visionRuntime.global();
+    for (NSString* pluginKey in [FrameProcessorPluginRegistry frameProcessorPlugins]) {
+      auto pluginName = [pluginKey UTF8String];
+      
+      NSLog(@"FrameProcessorBindings: Installing Frame Processor plugin \"%s\"...", pluginName);
+      FrameProcessorPlugin callback = [[FrameProcessorPluginRegistry frameProcessorPlugins] valueForKey:pluginKey];
+      
+      auto function = [callback, callInvoker](jsi::Runtime& runtime, const jsi::Value& thisValue, const jsi::Value* arguments, size_t count) -> jsi::Value {
+        auto frameHostObject = arguments[0].asObject(runtime).asHostObject(runtime);
+        auto frame = static_cast<FrameHostObject*>(frameHostObject.get());
+        auto args = convertJSICStyleArrayToNSArray(runtime,
+                                                   arguments + 1, // start at index 1 since first arg = Frame
+                                                   count - 1, // use smaller count
+                                                   callInvoker);
+        id result = callback(frame->buffer, args);
+        return convertObjCObjectToJSIValue(runtime, result);
+      };
+      
+      visionGlobal.setProperty(visionRuntime, pluginName, jsi::Function::createFromHostFunction(visionRuntime,
+                                                                                                jsi::PropNameID::forAscii(visionRuntime, pluginName),
+                                                                                                1, // frame
+                                                                                                function));
+    }
+    [FrameProcessorPluginRegistry markInvalid];
+    NSLog(@"FrameProcessorBindings: Frame Processor plugins installed!");
+#else
+    NSLog(@"Reanimated not found, Frame Processors are disabled.");
+#endif
+  }
+  return self;
+}
+
+- (void) installFrameProcessorBindings {
+#ifdef ENABLE_FRAME_PROCESSORS
+  if (!weakBridge) {
+    NSLog(@"FrameProcessorBindings: Failed to install Frame Processor Bindings - bridge was null!");
+    return;
+  }
+  NSLog(@"FrameProcessorBindings: Installing Frame Processor Bindings for Bridge...");
+  RCTCxxBridge *cxxBridge = (RCTCxxBridge *)weakBridge;
+  if (!cxxBridge.runtime) {
+    return;
+  }
+  jsi::Runtime& jsiRuntime = *(jsi::Runtime*)cxxBridge.runtime;
+  NSLog(@"FrameProcessorBindings: Installing global functions...");
+  
+  // setFrameProcessor(viewTag: number, frameProcessor: (frame: Frame) => void)
+  auto setFrameProcessor = [self](jsi::Runtime& runtime, const jsi::Value& thisValue, const jsi::Value* arguments, size_t count) -> jsi::Value {
+    NSLog(@"FrameProcessorBindings: Setting new frame processor...");
+    if (!arguments[0].isNumber()) throw jsi::JSError(runtime, "Camera::setFrameProcessor: First argument ('viewTag') must be a number!");
+    if (!arguments[1].isObject()) throw jsi::JSError(runtime, "Camera::setFrameProcessor: Second argument ('frameProcessor') must be a function!");
+    if (!runtimeManager || !runtimeManager->runtime) throw jsi::JSError(runtime, "Camera::setFrameProcessor: The RuntimeManager is not yet initialized!");
+    
+    auto viewTag = arguments[0].asNumber();
+    NSLog(@"FrameProcessorBindings: Adapting Shareable value from function (conversion to worklet)...");
+    auto worklet = reanimated::ShareableValue::adapt(runtime, arguments[1], runtimeManager.get());
+    NSLog(@"FrameProcessorBindings: Successfully created worklet!");
+    
+    RCTExecuteOnMainQueue([worklet, viewTag, self]() -> void {
+      auto currentBridge = [RCTBridge currentBridge];
+      auto anonymousView = [currentBridge.uiManager viewForReactTag:[NSNumber numberWithDouble:viewTag]];
+      auto view = static_cast<CameraView*>(anonymousView);
+      
+      dispatch_async(CameraQueues.videoQueue, [worklet, view, self]() -> void {
+        NSLog(@"FrameProcessorBindings: Converting worklet to Objective-C callback...");
+        auto& rt = *runtimeManager->runtime;
+        auto function = worklet->getValue(rt).asObject(rt).asFunction(rt);
+        view.frameProcessorCallback = convertJSIFunctionToFrameProcessorCallback(rt, function);
+        NSLog(@"FrameProcessorBindings: Frame processor set!");
+      });
+    });
+    
+    return jsi::Value::undefined();
+  };
+  jsiRuntime.global().setProperty(jsiRuntime, "setFrameProcessor", jsi::Function::createFromHostFunction(jsiRuntime,
+                                                                                                         jsi::PropNameID::forAscii(jsiRuntime, "setFrameProcessor"),
+                                                                                                         2,  // viewTag, frameProcessor
+                                                                                                         setFrameProcessor));
+  
+  // unsetFrameProcessor(viewTag: number)
+  auto unsetFrameProcessor = [](jsi::Runtime& runtime, const jsi::Value& thisValue, const jsi::Value* arguments, size_t count) -> jsi::Value {
+    NSLog(@"FrameProcessorBindings: Removing frame processor...");
+    if (!arguments[0].isNumber()) throw jsi::JSError(runtime, "Camera::unsetFrameProcessor: First argument ('viewTag') must be a number!");
+    auto viewTag = arguments[0].asNumber();
+    
+    RCTExecuteOnMainQueue(^{
+      auto currentBridge = [RCTBridge currentBridge];
+      if (!currentBridge) return;
+      auto anonymousView = [currentBridge.uiManager viewForReactTag:[NSNumber numberWithDouble:viewTag]];
+      if (!anonymousView) return;
+      auto view = static_cast<CameraView*>(anonymousView);
+      view.frameProcessorCallback = nil;
+      NSLog(@"FrameProcessorBindings: Frame processor removed!");
+    });
+    
+    return jsi::Value::undefined();
+  };
+  jsiRuntime.global().setProperty(jsiRuntime, "unsetFrameProcessor", jsi::Function::createFromHostFunction(jsiRuntime,
+                                                                                                           jsi::PropNameID::forAscii(jsiRuntime, "unsetFrameProcessor"),
+                                                                                                           1,  // viewTag
+                                                                                                           unsetFrameProcessor));
+
+  NSLog(@"FrameProcessorBindings: Finished installing bindings.");
+#endif
+}
+
+@end
--- a/Processor/FrameProcessorUtils.h
+++ b/Processor/FrameProcessorUtils.h
@@ -0,0 +1,23 @@
+//
+//  FrameProcessorUtils.h
+//  VisionCamera
+//
+//  Created by Marc Rousavy on 15.03.21.
+//  Copyright © 2021 Facebook. All rights reserved.
+//
+
+#pragma once
+
+#import <Foundation/Foundation.h>
+#import <React/RCTBridgeModule.h>
+#import "FrameProcessorCallback.h"
+
+#ifndef __cplusplus
+#error FrameProcessorUtils.h has to be compiled with C++!
+#endif
+
+#import <jsi/jsi.h>
+
+using namespace facebook;
+
+FrameProcessorCallback convertJSIFunctionToFrameProcessorCallback(jsi::Runtime &runtime, const jsi::Function &value);
--- a/Processor/FrameProcessorUtils.mm
+++ b/Processor/FrameProcessorUtils.mm
@@ -0,0 +1,44 @@
+//
+//  FrameProcessorUtils.m
+//  VisionCamera
+//
+//  Created by Marc Rousavy on 15.03.21.
+//  Copyright © 2021 Facebook. All rights reserved.
+//
+
+#import "FrameProcessorUtils.h"
+#import <CoreMedia/CMSampleBuffer.h>
+#import <chrono>
+#import <memory>
+#import "FrameHostObject.h"
+
+FrameProcessorCallback convertJSIFunctionToFrameProcessorCallback(jsi::Runtime &runtime, const jsi::Function &value) {
+  __block auto cb = value.getFunction(runtime);
+  
+  return ^(CMSampleBufferRef buffer) {
+#if DEBUG
+    std::chrono::steady_clock::time_point begin = std::chrono::steady_clock::now();
+#endif
+    
+    auto frame = std::make_shared<FrameHostObject>(buffer);
+    try {
+      cb.call(runtime, jsi::Object::createFromHostObject(runtime, frame));
+    } catch (jsi::JSError& jsError) {
+      NSLog(@"Frame Processor threw an error: %s", jsError.getMessage().c_str());
+    }
+    
+#if DEBUG
+    std::chrono::steady_clock::time_point end = std::chrono::steady_clock::now();
+    auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - begin).count();
+    if (duration > 100) {
+      NSLog(@"Warning: Frame Processor function took %lld ms to execute. This blocks the video queue from recording, optimize your frame processor!", duration);
+    }
+#endif
+    
+    // Manually free the buffer because:
+    //  1. we are sure we don't need it anymore, the frame processor worklet has finished executing.
+    //  2. we don't know when the JS runtime garbage collects this object, it might be holding it for a few more frames
+    //     which then blocks the camera queue from pushing new frames (memory limit)
+    frame->destroyBuffer();
+  };
+}