react-native-vision-camera/ios/React Utils/JSIUtils.mm
Marc Rousavy b6a67d5ced
feature: Frame Processors (iOS) (#2)
* Clean up Frame Processor

* Create FrameProcessorHolder

* Create FrameProcessorDelegate in ObjC++

* Move frame processor to FrameProcessorDelegate

* Decorate runtime, check for null

* Update FrameProcessorDelegate.mm

* Cleanup FrameProcessorBindings.mm

* Fix RuntimeDecorator.h import

* Update FrameProcessorDelegate.mm

* "React" -> "React Helper" to avoid confusion

* Rename folders again

* Fix podspec flattening a lot of headers, causing REA nameclash

* Fix header imports to avoid REA naming collision

* Lazily initialize jsi::Runtime on DispatchQueue

* Install frame processor bindings from Swift

* First try to call jsi::Function (frame processor) 👀

* Call viewForReactTag on RCT main thread

* Fix bridge accessing

* Add more logs

* Update CameraViewManager.swift

* Add more TODOs

* Re-indent .cpp files

* Fix RCTTurboModule import podspec

* Remove unnecessary include check for swift umbrella header

* Merge branch 'main' into frame-processors

* Docs: use static width for images (283)

* Create validate-cpp.yml

* Update a lot of packages to latest

* Set SWIFT_VERSION to 5.2 in podspec

* Create clean.sh

* Delete unused C++ files

* podspec: Remove CLANG_CXX_LANGUAGE_STANDARD and OTHER_CFLAGS

* Update pod lockfiles

* Regenerate lockfiles

* Remove IOSLogger

* Use NSLog

* Create FrameProcessorManager (inherits from REA RuntimeManager)

* Create reanimated::RuntimeManager shared_ptr

* Re-integrate pods

* Add react-native-reanimated >=2 peerDependency

* Add metro-config

* blacklist -> exclusionList

* Try to call worklet

* Fix jsi::Value* initializer

* Call ShareableValue::adapt (makeShareable) with React/JS Runtime

* Add null-checks

* Lift runtime manager creation out of delegate, into bindings

* Remove debug statement

* Make RuntimeManager unique_ptr

* Set _FRAME_PROCESSOR

* Extract convertJSIFunctionToFrameProcessorCallback

* Print frame

* Merge branch 'main' into frame-processors

* Reformat Swift code

* Install reanimated from npm again

* Re-integrate Pods

* Dependabot: Also scan example/ and docs/

* Update validate-cpp.yml

* Create FrameProcessorUtils

* Create Frame.h

* Abstract HostObject creation away

* Fix types

* Fix frame processor call

* Add todo

* Update lockfiles

* Add C++ contributing instructions

* Update CONTRIBUTING.md

* Add android/src/main/cpp to cpplint

* Update cpplint.sh

* Fix a few cpplint errors

* Fix globals

* Fix a few more cpplint errors

* Update App.tsx

* Update AndroidLogger.cpp

* Format

* Fix cpplint script (check-cpp)

* Try to simplify frame processor

* y

* Update FrameProcessorUtils.mm

* Update FrameProcessorBindings.mm

* Update CameraView.swift

* Update CameraViewManager.m

* Restructure everything

* fix

* Fix `@objc` export (make public)

* Refactor installFrameProcessorBindings into FrameProcessorRuntimeManager

* Add swift RCTBridge.runOnJS helper

* Fix run(onJS)

* Add pragma once

* Add `&self` to lambda

* Update FrameProcessorRuntimeManager.mm

* reorder imports

* Fix imports

* forward declare

* Rename extension

* Destroy buffer after execution

* Add FrameProcessorPluginRegistry base

* Merge branch 'main' into frame-processors

* Add frameProcessor to types

* Update Camera.tsx

* Fix rebase merge

* Remove movieOutput

* Use `useFrameProcessor`

* Fix bad merge

* Add additional ESLint rules

* Update lockfiles

* Update CameraViewManager.m

* Add support for V8 runtime

* Add frame processor plugins API

* Print plugin invoke

* Fix React Utils in podspec

* Fix runOnJS swift name

* Remove invalid redecl of `captureSession`

* Use REA 2.1.0 which includes all my big PRs 🎉

* Update validate-cpp.yml

* Update Podfile.lock

* Remove Flipper

* Fix dereferencing

* Capture `self` by value. Fucking hell, what a dumb mistake.

* Override a few HostObject functions

* Expose isReady, width, height, bytesPerRow and planesCount

* use hook again

* Expose property names

* FrameProcessor -> Frame

* Update CameraView+RecordVideo.swift

* Add Swift support for Frame Processors Plugins

* Add macros for plugin installation

* Add ObjC frame processor plugin

* Correctly install frame processor plugins

* Don't require custom name for macro

* Check if plugin already exists

* Implement QR Code Frame Processor Plugin in Swift

* Adjust ObjC style frame processor macro

* optimize

* Add `frameProcessorFrameDropRate`

* Fix types

* Only log once

* Log if it executes slowly

* Implement `frameProcessorFps`

* Implement manual encoded video recordings

* Use recommended video settings

* Add fileType types

* Ignore if input is not ready for media data

* Add completion handler

* Add audio buffer sampling

* Init only for video frame

* use AVAssetWriterInputPixelBufferAdaptor

* Remove AVAssetWriterInputPixelBufferAdaptor

* Rotate VideoWriter

* Always assume portrait orientation

* Update RecordingSession.swift

* Use a separate Queue for Audio

* Format Swift

* Update CameraView+RecordVideo.swift

* Use `videoQueue` instead of `cameraQueue`

* Move example plugins to example app

* Fix hardcoded name in plugin macro

* QRFrame... -> QRCodeFrame...

* Update FrameProcessorPlugin.h

* Add example frame processors to JS base

* Update QRCodeFrameProcessorPluginSwift.m

* Add docs to create FP Plugins

* Update FRAME_PROCESSORS_CREATE.mdx

* Update FRAME_PROCESSORS_CREATE.mdx

* Use `AVAssetWriterInputPixelBufferAdaptor` for efficient pixel buffer recycling

* Add customizable `pixelFormat`

* Use native format if available

* Update project.pbxproj

* Set video width and height as source-pixel-buffer attributes

* Catch

* Update App.tsx

* Don't explicitly set video dimensions, let CVPixelBufferPool handle it

* Add a few logs

* Cleanup

* Update CameraView+RecordVideo.swift

* Eagerly initialize asset writer to fix stutter at first frame

* Use `cameraQueue` DispatchQueue to not block CaptureDataOutputDelegate

* Fix duration calculation

* cleanup

* Cleanup

* Swiftformat

* Return available video codecs

* Only show frame drop notification for video output

* Remove photo and video codec functionality

It was too much complexity and probably never used anyways.

* Revert all android related changes for now

* Cleanup

* Remove unused header

* Update AVAssetWriter.Status+descriptor.swift

* Only call Frame Processor for Video Frames

* Fix `if`

* Add support for Frame Processor plugin parameters/arguments

* Fix arg support

* Move to JSIUtils.mm

* Update JSIUtils.h

* Update FRAME_PROCESSORS_CREATE.mdx

* Update FRAME_PROCESSORS_CREATE.mdx

* Upgrade packages for docs/

* fix docs

* Rename

* highlight lines

* docs

* community plugins

* Update FRAME_PROCESSOR_CREATE_FINAL.mdx

* Update FRAME_PROCESSOR_PLUGIN_LIST.mdx

* Update FRAME_PROCESSOR_PLUGIN_LIST.mdx

* Update dependencies (1/2)

* Update dependencies (2/2)

* Update Gemfile.lock

* add FP docs

* Update README.md

* Make `lastFrameProcessor` private

* add `frameProcessor` docs

* fix docs

* adjust docs

* Update DEVICES.mdx

* fix

* s

* Add logs demo

* add metro restart note

* Update FRAME_PROCESSOR_CREATE_PLUGIN_IOS.mdx

* Mirror video device

* Update AVCaptureVideoDataOutput+mirror.swift

* Create .swift-version

* Enable whole module optimization

* Fix recording mirrored video

* Swift format

* Clean dictionary on `markInvalid`

* Fix cleanup

* Add docs for disabling frame processors

* Update project.pbxproj

* Revert "Update project.pbxproj"

This reverts commit e67861e51b88b4888a6940e2d20388f3044211d0.

* Log frame drop reason

* Format

* add more samples

* Add clang-format

* also check .mm

* Revert "also check .mm"

This reverts commit 8b9d5e2c29866b05909530d104f6633d6c49eadd.

* Revert "Add clang-format"

This reverts commit 7643ac808e0fc34567ea1f814e73d84955381636.

* Use `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange` as default

* Read matching video attributes from videoSettings

* Add TODO

* Swiftformat

* Conditionally disable frame processors

* Assert if trying to use frame processors when disabled

* Add frame-processors demo gif

* Allow disabling frame processors via `VISION_CAMERA_DISABLE_FRAME_PROCESSORS`

* Update FrameProcessorRuntimeManager.mm

* Update FRAME_PROCESSORS.mdx

* Update project.pbxproj

* Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
2021-05-06 14:11:55 +02:00

188 lines
6.1 KiB
Plaintext

//
// JSIUtils.mm
// VisionCamera
//
// Created by Marc Rousavy on 02.05.21.
// Copyright © 2021 Facebook. All rights reserved.
//
#import "JSIUtils.h"
#import <Foundation/Foundation.h>
#import <jsi/jsi.h>
#import <ReactCommon/CallInvoker.h>
#import <React/RCTBridge.h>
#import <ReactCommon/TurboModuleUtils.h>
using namespace facebook;
using namespace facebook::react;
jsi::Value convertNSNumberToJSIBoolean(jsi::Runtime &runtime, NSNumber *value)
{
return jsi::Value((bool)[value boolValue]);
}
jsi::Value convertNSNumberToJSINumber(jsi::Runtime &runtime, NSNumber *value)
{
return jsi::Value([value doubleValue]);
}
jsi::String convertNSStringToJSIString(jsi::Runtime &runtime, NSString *value)
{
return jsi::String::createFromUtf8(runtime, [value UTF8String] ?: "");
}
jsi::Object convertNSDictionaryToJSIObject(jsi::Runtime &runtime, NSDictionary *value)
{
jsi::Object result = jsi::Object(runtime);
for (NSString *k in value) {
result.setProperty(runtime, [k UTF8String], convertObjCObjectToJSIValue(runtime, value[k]));
}
return result;
}
jsi::Array convertNSArrayToJSIArray(jsi::Runtime &runtime, NSArray *value)
{
jsi::Array result = jsi::Array(runtime, value.count);
for (size_t i = 0; i < value.count; i++) {
result.setValueAtIndex(runtime, i, convertObjCObjectToJSIValue(runtime, value[i]));
}
return result;
}
jsi::Value convertObjCObjectToJSIValue(jsi::Runtime &runtime, id value)
{
if (value == nil) {
return jsi::Value::undefined();
} else if ([value isKindOfClass:[NSString class]]) {
return convertNSStringToJSIString(runtime, (NSString *)value);
} else if ([value isKindOfClass:[NSNumber class]]) {
if ([value isKindOfClass:[@YES class]]) {
return convertNSNumberToJSIBoolean(runtime, (NSNumber *)value);
}
return convertNSNumberToJSINumber(runtime, (NSNumber *)value);
} else if ([value isKindOfClass:[NSDictionary class]]) {
return convertNSDictionaryToJSIObject(runtime, (NSDictionary *)value);
} else if ([value isKindOfClass:[NSArray class]]) {
return convertNSArrayToJSIArray(runtime, (NSArray *)value);
} else if (value == (id)kCFNull) {
return jsi::Value::null();
}
return jsi::Value::undefined();
}
NSString *convertJSIStringToNSString(jsi::Runtime &runtime, const jsi::String &value)
{
return [NSString stringWithUTF8String:value.utf8(runtime).c_str()];
}
NSArray* convertJSICStyleArrayToNSArray(jsi::Runtime &runtime, const jsi::Value* array, size_t length, std::shared_ptr<CallInvoker> jsInvoker) {
if (length == 0) return @[];
NSMutableArray *result = [NSMutableArray new];
for (size_t i = 0; i < length; i++) {
// Insert kCFNull when it's `undefined` value to preserve the indices.
[result
addObject:convertJSIValueToObjCObject(runtime, array[i], jsInvoker) ?: (id)kCFNull];
}
return [result copy];
}
jsi::Value* convertNSArrayToJSICStyleArray(jsi::Runtime &runtime, NSArray* array) {
auto result = new jsi::Value[array.count];
for (size_t i = 0; i < array.count; i++) {
result[i] = convertObjCObjectToJSIValue(runtime, array[i]);
}
return result;
}
NSArray* convertJSIArrayToNSArray(jsi::Runtime &runtime, const jsi::Array &value, std::shared_ptr<CallInvoker> jsInvoker)
{
size_t size = value.size(runtime);
NSMutableArray *result = [NSMutableArray new];
for (size_t i = 0; i < size; i++) {
// Insert kCFNull when it's `undefined` value to preserve the indices.
[result
addObject:convertJSIValueToObjCObject(runtime, value.getValueAtIndex(runtime, i), jsInvoker) ?: (id)kCFNull];
}
return [result copy];
}
NSDictionary* convertJSIObjectToNSDictionary(jsi::Runtime &runtime, const jsi::Object &value, std::shared_ptr<CallInvoker> jsInvoker)
{
jsi::Array propertyNames = value.getPropertyNames(runtime);
size_t size = propertyNames.size(runtime);
NSMutableDictionary *result = [NSMutableDictionary new];
for (size_t i = 0; i < size; i++) {
jsi::String name = propertyNames.getValueAtIndex(runtime, i).getString(runtime);
NSString *k = convertJSIStringToNSString(runtime, name);
id v = convertJSIValueToObjCObject(runtime, value.getProperty(runtime, name), jsInvoker);
if (v) {
result[k] = v;
}
}
return [result copy];
}
id convertJSIValueToObjCObject(jsi::Runtime &runtime, const jsi::Value &value, std::shared_ptr<CallInvoker> jsInvoker)
{
if (value.isUndefined() || value.isNull()) {
return nil;
}
if (value.isBool()) {
return @(value.getBool());
}
if (value.isNumber()) {
return @(value.getNumber());
}
if (value.isString()) {
return convertJSIStringToNSString(runtime, value.getString(runtime));
}
if (value.isObject()) {
jsi::Object o = value.getObject(runtime);
if (o.isArray(runtime)) {
return convertJSIArrayToNSArray(runtime, o.getArray(runtime), jsInvoker);
}
if (o.isFunction(runtime)) {
return convertJSIFunctionToCallback(runtime, std::move(o.getFunction(runtime)), jsInvoker);
}
return convertJSIObjectToNSDictionary(runtime, o, jsInvoker);
}
throw std::runtime_error("Unsupported jsi::jsi::Value kind");
}
RCTResponseSenderBlock convertJSIFunctionToCallback(jsi::Runtime &runtime, const jsi::Function &value, std::shared_ptr<CallInvoker> jsInvoker)
{
auto weakWrapper = CallbackWrapper::createWeak(value.getFunction(runtime), runtime, jsInvoker);
BOOL __block wrapperWasCalled = NO;
RCTResponseSenderBlock callback = ^(NSArray *responses) {
if (wrapperWasCalled) {
throw std::runtime_error("callback arg cannot be called more than once");
}
auto strongWrapper = weakWrapper.lock();
if (!strongWrapper) {
return;
}
strongWrapper->jsInvoker().invokeAsync([weakWrapper, responses]() {
auto strongWrapper2 = weakWrapper.lock();
if (!strongWrapper2) {
return;
}
const jsi::Value* args = convertNSArrayToJSICStyleArray(strongWrapper2->runtime(), responses);
strongWrapper2->callback().call(strongWrapper2->runtime(), args, static_cast<size_t>(responses.count));
strongWrapper2->destroy();
delete[] args;
});
wrapperWasCalled = YES;
};
if (RCTTurboModuleBlockCopyEnabled()) {
return [callback copy];
}
return callback;
}