react-native-vision-camera/ios/React Utils/JSIUtils.mm

188 lines
6.1 KiB
Plaintext
Raw Normal View History

feature: Frame Processors (iOS) (#2) * Clean up Frame Processor * Create FrameProcessorHolder * Create FrameProcessorDelegate in ObjC++ * Move frame processor to FrameProcessorDelegate * Decorate runtime, check for null * Update FrameProcessorDelegate.mm * Cleanup FrameProcessorBindings.mm * Fix RuntimeDecorator.h import * Update FrameProcessorDelegate.mm * "React" -> "React Helper" to avoid confusion * Rename folders again * Fix podspec flattening a lot of headers, causing REA nameclash * Fix header imports to avoid REA naming collision * Lazily initialize jsi::Runtime on DispatchQueue * Install frame processor bindings from Swift * First try to call jsi::Function (frame processor) :eyes: * Call viewForReactTag on RCT main thread * Fix bridge accessing * Add more logs * Update CameraViewManager.swift * Add more TODOs * Re-indent .cpp files * Fix RCTTurboModule import podspec * Remove unnecessary include check for swift umbrella header * Merge branch 'main' into frame-processors * Docs: use static width for images (283) * Create validate-cpp.yml * Update a lot of packages to latest * Set SWIFT_VERSION to 5.2 in podspec * Create clean.sh * Delete unused C++ files * podspec: Remove CLANG_CXX_LANGUAGE_STANDARD and OTHER_CFLAGS * Update pod lockfiles * Regenerate lockfiles * Remove IOSLogger * Use NSLog * Create FrameProcessorManager (inherits from REA RuntimeManager) * Create reanimated::RuntimeManager shared_ptr * Re-integrate pods * Add react-native-reanimated >=2 peerDependency * Add metro-config * blacklist -> exclusionList * Try to call worklet * Fix jsi::Value* initializer * Call ShareableValue::adapt (makeShareable) with React/JS Runtime * Add null-checks * Lift runtime manager creation out of delegate, into bindings * Remove debug statement * Make RuntimeManager unique_ptr * Set _FRAME_PROCESSOR * Extract convertJSIFunctionToFrameProcessorCallback * Print frame * Merge branch 'main' into frame-processors * Reformat Swift code * Install reanimated from npm again * Re-integrate Pods * Dependabot: Also scan example/ and docs/ * Update validate-cpp.yml * Create FrameProcessorUtils * Create Frame.h * Abstract HostObject creation away * Fix types * Fix frame processor call * Add todo * Update lockfiles * Add C++ contributing instructions * Update CONTRIBUTING.md * Add android/src/main/cpp to cpplint * Update cpplint.sh * Fix a few cpplint errors * Fix globals * Fix a few more cpplint errors * Update App.tsx * Update AndroidLogger.cpp * Format * Fix cpplint script (check-cpp) * Try to simplify frame processor * y * Update FrameProcessorUtils.mm * Update FrameProcessorBindings.mm * Update CameraView.swift * Update CameraViewManager.m * Restructure everything * fix * Fix `@objc` export (make public) * Refactor installFrameProcessorBindings into FrameProcessorRuntimeManager * Add swift RCTBridge.runOnJS helper * Fix run(onJS) * Add pragma once * Add `&self` to lambda * Update FrameProcessorRuntimeManager.mm * reorder imports * Fix imports * forward declare * Rename extension * Destroy buffer after execution * Add FrameProcessorPluginRegistry base * Merge branch 'main' into frame-processors * Add frameProcessor to types * Update Camera.tsx * Fix rebase merge * Remove movieOutput * Use `useFrameProcessor` * Fix bad merge * Add additional ESLint rules * Update lockfiles * Update CameraViewManager.m * Add support for V8 runtime * Add frame processor plugins API * Print plugin invoke * Fix React Utils in podspec * Fix runOnJS swift name * Remove invalid redecl of `captureSession` * Use REA 2.1.0 which includes all my big PRs :tada: * Update validate-cpp.yml * Update Podfile.lock * Remove Flipper * Fix dereferencing * Capture `self` by value. Fucking hell, what a dumb mistake. * Override a few HostObject functions * Expose isReady, width, height, bytesPerRow and planesCount * use hook again * Expose property names * FrameProcessor -> Frame * Update CameraView+RecordVideo.swift * Add Swift support for Frame Processors Plugins * Add macros for plugin installation * Add ObjC frame processor plugin * Correctly install frame processor plugins * Don't require custom name for macro * Check if plugin already exists * Implement QR Code Frame Processor Plugin in Swift * Adjust ObjC style frame processor macro * optimize * Add `frameProcessorFrameDropRate` * Fix types * Only log once * Log if it executes slowly * Implement `frameProcessorFps` * Implement manual encoded video recordings * Use recommended video settings * Add fileType types * Ignore if input is not ready for media data * Add completion handler * Add audio buffer sampling * Init only for video frame * use AVAssetWriterInputPixelBufferAdaptor * Remove AVAssetWriterInputPixelBufferAdaptor * Rotate VideoWriter * Always assume portrait orientation * Update RecordingSession.swift * Use a separate Queue for Audio * Format Swift * Update CameraView+RecordVideo.swift * Use `videoQueue` instead of `cameraQueue` * Move example plugins to example app * Fix hardcoded name in plugin macro * QRFrame... -> QRCodeFrame... * Update FrameProcessorPlugin.h * Add example frame processors to JS base * Update QRCodeFrameProcessorPluginSwift.m * Add docs to create FP Plugins * Update FRAME_PROCESSORS_CREATE.mdx * Update FRAME_PROCESSORS_CREATE.mdx * Use `AVAssetWriterInputPixelBufferAdaptor` for efficient pixel buffer recycling * Add customizable `pixelFormat` * Use native format if available * Update project.pbxproj * Set video width and height as source-pixel-buffer attributes * Catch * Update App.tsx * Don't explicitly set video dimensions, let CVPixelBufferPool handle it * Add a few logs * Cleanup * Update CameraView+RecordVideo.swift * Eagerly initialize asset writer to fix stutter at first frame * Use `cameraQueue` DispatchQueue to not block CaptureDataOutputDelegate * Fix duration calculation * cleanup * Cleanup * Swiftformat * Return available video codecs * Only show frame drop notification for video output * Remove photo and video codec functionality It was too much complexity and probably never used anyways. * Revert all android related changes for now * Cleanup * Remove unused header * Update AVAssetWriter.Status+descriptor.swift * Only call Frame Processor for Video Frames * Fix `if` * Add support for Frame Processor plugin parameters/arguments * Fix arg support * Move to JSIUtils.mm * Update JSIUtils.h * Update FRAME_PROCESSORS_CREATE.mdx * Update FRAME_PROCESSORS_CREATE.mdx * Upgrade packages for docs/ * fix docs * Rename * highlight lines * docs * community plugins * Update FRAME_PROCESSOR_CREATE_FINAL.mdx * Update FRAME_PROCESSOR_PLUGIN_LIST.mdx * Update FRAME_PROCESSOR_PLUGIN_LIST.mdx * Update dependencies (1/2) * Update dependencies (2/2) * Update Gemfile.lock * add FP docs * Update README.md * Make `lastFrameProcessor` private * add `frameProcessor` docs * fix docs * adjust docs * Update DEVICES.mdx * fix * s * Add logs demo * add metro restart note * Update FRAME_PROCESSOR_CREATE_PLUGIN_IOS.mdx * Mirror video device * Update AVCaptureVideoDataOutput+mirror.swift * Create .swift-version * Enable whole module optimization * Fix recording mirrored video * Swift format * Clean dictionary on `markInvalid` * Fix cleanup * Add docs for disabling frame processors * Update project.pbxproj * Revert "Update project.pbxproj" This reverts commit e67861e51b88b4888a6940e2d20388f3044211d0. * Log frame drop reason * Format * add more samples * Add clang-format * also check .mm * Revert "also check .mm" This reverts commit 8b9d5e2c29866b05909530d104f6633d6c49eadd. * Revert "Add clang-format" This reverts commit 7643ac808e0fc34567ea1f814e73d84955381636. * Use `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange` as default * Read matching video attributes from videoSettings * Add TODO * Swiftformat * Conditionally disable frame processors * Assert if trying to use frame processors when disabled * Add frame-processors demo gif * Allow disabling frame processors via `VISION_CAMERA_DISABLE_FRAME_PROCESSORS` * Update FrameProcessorRuntimeManager.mm * Update FRAME_PROCESSORS.mdx * Update project.pbxproj * Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
2021-05-06 06:11:55 -06:00
//
// JSIUtils.mm
// VisionCamera
//
// Created by Marc Rousavy on 02.05.21.
// Copyright © 2021 Facebook. All rights reserved.
//
#import "JSIUtils.h"
#import <Foundation/Foundation.h>
#import <jsi/jsi.h>
#import <ReactCommon/CallInvoker.h>
#import <React/RCTBridge.h>
#import <ReactCommon/TurboModuleUtils.h>
using namespace facebook;
using namespace facebook::react;
jsi::Value convertNSNumberToJSIBoolean(jsi::Runtime &runtime, NSNumber *value)
{
return jsi::Value((bool)[value boolValue]);
}
jsi::Value convertNSNumberToJSINumber(jsi::Runtime &runtime, NSNumber *value)
{
return jsi::Value([value doubleValue]);
}
jsi::String convertNSStringToJSIString(jsi::Runtime &runtime, NSString *value)
{
return jsi::String::createFromUtf8(runtime, [value UTF8String] ?: "");
}
jsi::Object convertNSDictionaryToJSIObject(jsi::Runtime &runtime, NSDictionary *value)
{
jsi::Object result = jsi::Object(runtime);
for (NSString *k in value) {
result.setProperty(runtime, [k UTF8String], convertObjCObjectToJSIValue(runtime, value[k]));
}
return result;
}
jsi::Array convertNSArrayToJSIArray(jsi::Runtime &runtime, NSArray *value)
{
jsi::Array result = jsi::Array(runtime, value.count);
for (size_t i = 0; i < value.count; i++) {
result.setValueAtIndex(runtime, i, convertObjCObjectToJSIValue(runtime, value[i]));
}
return result;
}
jsi::Value convertObjCObjectToJSIValue(jsi::Runtime &runtime, id value)
{
if (value == nil) {
return jsi::Value::undefined();
} else if ([value isKindOfClass:[NSString class]]) {
return convertNSStringToJSIString(runtime, (NSString *)value);
} else if ([value isKindOfClass:[NSNumber class]]) {
if ([value isKindOfClass:[@YES class]]) {
return convertNSNumberToJSIBoolean(runtime, (NSNumber *)value);
}
return convertNSNumberToJSINumber(runtime, (NSNumber *)value);
} else if ([value isKindOfClass:[NSDictionary class]]) {
return convertNSDictionaryToJSIObject(runtime, (NSDictionary *)value);
} else if ([value isKindOfClass:[NSArray class]]) {
return convertNSArrayToJSIArray(runtime, (NSArray *)value);
} else if (value == (id)kCFNull) {
return jsi::Value::null();
}
return jsi::Value::undefined();
}
NSString *convertJSIStringToNSString(jsi::Runtime &runtime, const jsi::String &value)
{
return [NSString stringWithUTF8String:value.utf8(runtime).c_str()];
}
NSArray* convertJSICStyleArrayToNSArray(jsi::Runtime &runtime, const jsi::Value* array, size_t length, std::shared_ptr<CallInvoker> jsInvoker) {
if (length == 0) return @[];
NSMutableArray *result = [NSMutableArray new];
for (size_t i = 0; i < length; i++) {
// Insert kCFNull when it's `undefined` value to preserve the indices.
[result
addObject:convertJSIValueToObjCObject(runtime, array[i], jsInvoker) ?: (id)kCFNull];
}
return [result copy];
}
jsi::Value* convertNSArrayToJSICStyleArray(jsi::Runtime &runtime, NSArray* array) {
auto result = new jsi::Value[array.count];
for (size_t i = 0; i < array.count; i++) {
result[i] = convertObjCObjectToJSIValue(runtime, array[i]);
}
return result;
}
NSArray* convertJSIArrayToNSArray(jsi::Runtime &runtime, const jsi::Array &value, std::shared_ptr<CallInvoker> jsInvoker)
{
size_t size = value.size(runtime);
NSMutableArray *result = [NSMutableArray new];
for (size_t i = 0; i < size; i++) {
// Insert kCFNull when it's `undefined` value to preserve the indices.
[result
addObject:convertJSIValueToObjCObject(runtime, value.getValueAtIndex(runtime, i), jsInvoker) ?: (id)kCFNull];
}
return [result copy];
}
NSDictionary* convertJSIObjectToNSDictionary(jsi::Runtime &runtime, const jsi::Object &value, std::shared_ptr<CallInvoker> jsInvoker)
{
jsi::Array propertyNames = value.getPropertyNames(runtime);
size_t size = propertyNames.size(runtime);
NSMutableDictionary *result = [NSMutableDictionary new];
for (size_t i = 0; i < size; i++) {
jsi::String name = propertyNames.getValueAtIndex(runtime, i).getString(runtime);
NSString *k = convertJSIStringToNSString(runtime, name);
id v = convertJSIValueToObjCObject(runtime, value.getProperty(runtime, name), jsInvoker);
if (v) {
result[k] = v;
}
}
return [result copy];
}
id convertJSIValueToObjCObject(jsi::Runtime &runtime, const jsi::Value &value, std::shared_ptr<CallInvoker> jsInvoker)
{
if (value.isUndefined() || value.isNull()) {
return nil;
}
if (value.isBool()) {
return @(value.getBool());
}
if (value.isNumber()) {
return @(value.getNumber());
}
if (value.isString()) {
return convertJSIStringToNSString(runtime, value.getString(runtime));
}
if (value.isObject()) {
jsi::Object o = value.getObject(runtime);
if (o.isArray(runtime)) {
return convertJSIArrayToNSArray(runtime, o.getArray(runtime), jsInvoker);
}
if (o.isFunction(runtime)) {
return convertJSIFunctionToCallback(runtime, std::move(o.getFunction(runtime)), jsInvoker);
}
return convertJSIObjectToNSDictionary(runtime, o, jsInvoker);
}
throw std::runtime_error("Unsupported jsi::jsi::Value kind");
}
RCTResponseSenderBlock convertJSIFunctionToCallback(jsi::Runtime &runtime, const jsi::Function &value, std::shared_ptr<CallInvoker> jsInvoker)
{
auto weakWrapper = CallbackWrapper::createWeak(value.getFunction(runtime), runtime, jsInvoker);
BOOL __block wrapperWasCalled = NO;
RCTResponseSenderBlock callback = ^(NSArray *responses) {
if (wrapperWasCalled) {
throw std::runtime_error("callback arg cannot be called more than once");
}
auto strongWrapper = weakWrapper.lock();
if (!strongWrapper) {
return;
}
strongWrapper->jsInvoker().invokeAsync([weakWrapper, responses]() {
auto strongWrapper2 = weakWrapper.lock();
if (!strongWrapper2) {
return;
}
const jsi::Value* args = convertNSArrayToJSICStyleArray(strongWrapper2->runtime(), responses);
strongWrapper2->callback().call(strongWrapper2->runtime(), args, static_cast<size_t>(responses.count));
strongWrapper2->destroy();
delete[] args;
});
wrapperWasCalled = YES;
};
if (RCTTurboModuleBlockCopyEnabled()) {
return [callback copy];
}
return callback;
}