railbird/react-native-vision-camera
10
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: Frame Processors for Android (#196)

Browse Source 
* Create android gradle build setup

* Fix `prefab` config

* Add `pickFirst **/*.so` to example build.gradle

* fix REA path

* cache gradle builds

* Update validate-android.yml

* Create Native Proxy

* Copy REA header

* implement ctor

* Rename CameraViewModule -> FrameProcessorRuntimeManager

* init FrameProcessorRuntimeManager

* fix name

* Update FrameProcessorRuntimeManager.h

* format

* Create AndroidErrorHandler.h

* Initialize runtime and install JSI funcs

* Update FrameProcessorRuntimeManager.cpp

* Update CameraViewModule.kt

* Make CameraView hybrid C++ class to find view & set frame processor

* Update FrameProcessorRuntimeManager.cpp

* pass function by rvalue

* pass by const &&

* extract hermes and JSC REA

* pass `FOR_HERMES`

* correctly prepare JSC and Hermes

* Update CMakeLists.txt

* add missing hermes include

* clean up imports

* Create JImageProxy.h

* pass ImageProxy to JNI as `jobject`

* try use `JImageProxy` C++ wrapper type

* Use `local_ref<JImageProxy>`

* Create `JImageProxyHostObject` for JSI interop

* debug call to frame processor

* Unset frame processor

* Fix CameraView native part not being registered

* close image

* use `jobject` instead of `JImageProxy` for now :(

* fix hermes build error

* Set enable FP callback

* fix JNI call

* Update CameraView.cpp

* Get Format

* Create plugin abstract

* Make `FrameProcessorPlugin` a hybrid object

* Register plugin CXX

* Call `registerPlugin`

* Catch

* remove JSI

* Create sample QR code plugin

* register plugins

* Fix missing JNI binding

* Add `mHybridData`

* prefix name with two underscores (`__`)

* Update CameraPage.tsx

* wrap `ImageProxy` in host object

* Use `jobject` for HO box

* Update JImageProxy.h

* reinterpret jobject

* Try using `JImageProxy` instead of `jobject`

* Update JImageProxy.h

* get bytes per row and plane count

* Update CameraView.cpp

* Return base

* add some docs and JNI JSI conversion

* indent

* Convert JSI value to JNI jobject

* using namespace facebook

* Try using class

* Use plain old Object[]

* Try convert JNI -> JSI

* fix decl

* fix bool init

* Correctly link folly

* Update CMakeLists.txt

* Convert Map to Object

* Use folly for Map and Array

* Return `alias_ref<jobject>` instead of raw `jobject`

* fix JNI <-> JSI conversion

* Update JSIJNIConversion.cpp

* Log parameters

* fix params index offset

* add more test cases

* Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx

* fix types

* Rename to example plugin

* remove support for hashmap

* Try use HashMap iterable fbjni binding

* try using JReadableArray/JReadableMap

* Fix list return values

* Update JSIJNIConversion.cpp

* Update JSIJNIConversion.cpp

* (iOS) Rename ObjC QR Code Plugin to Example Plugin

* Rename Swift plugin QR -> Example

* Update ExamplePluginSwift.swift

* Fix Map/Dictionary logging format

* Update ExampleFrameProcessorPlugin.m

* Reconfigure session if frame processor changed

* Handle use-cases via `maxUseCasesCount`

* Don't crash app on `configureSession` error

* Document "use-cases"

* Update DEVICES.mdx

* fix merge

* Make `const &`

* iOS: Automatically enable `video` if a `frameProcessor` is set

* Update CameraView.cpp

* fix docs

* Automatically fallback to snapshot capture if `supportsParallelVideoProcessing` is false.

* Fix lookup

* Update CameraView.kt

* Implement `frameProcessorFps`

* Finalize Frame Processor Plugin Hybrid

* Update CameraViewModule.kt

* Support `flash` on `takeSnapshot()`

* Update docs

* Add docs

* Update CameraPage.tsx

* Attribute NonNull

* remove unused imports

* Add Android docs for Frame Processors

* Make JNI HashMap <-> JSI Object conversion faster

directly access `toHashMap` instead of going through java

* add todo

* Always run `prepareJSC` and `prepareHermes`

* switch jsc and hermes

* Specify ndkVersion `21.4.7075529`

* Update gradle.properties

* Update gradle.properties

* Create .aar

* Correctly prepare android package

* Update package.json

* Update package.json

* remove `prefab` build feature

* split

* Add docs for registering the FP plugin

* Add step for dep

* Update CaptureButton.tsx

* Move to `reanimated-headers/`

* Exclude reanimated-headers from cpplint

* disable `build/include_order` rule

* cpplint fixes

* perf: Make `JSIJNIConversion` a `namespace` instead of `class`

* Ignore runtime/references for `convert` funcs

* Build Android .aar in CI

* Run android build script only on `prepack`

* Update package.json

* Update package.json

* Update build-android-npm-package.sh

* Move to `yarn build`

* Also install node_modules in example step

* Update validate-android.yml

* sort imports

* fix torch

* Run ImageAnalysis on `FrameProcessorThread`

* Update Errors.kt

* Add clean android script

* Upgrade reanimated to 2.3.0-alpha.1

* Revert "Upgrade reanimated to 2.3.0-alpha.1"

This reverts commit c1d3bed5e03728d0b5e335a359524ff4f56f5035.

* ⚠️ TEMP FIX: hotfix reanimated build.gradle

* Update CameraView+TakeSnapshot.kt

* ⚠️ TEMP FIX: Disable ktlint action for now

* Update clean.sh

* Set max heap size to 4g

* rebuild lockfiles

* Update Podfile.lock

* rename

* Build lib .aar before example/
This commit is contained in:
Marc Rousavy
2021-06-27 12:37:54 +02:00
committed by GitHub
parent a2311c02ac
commit 87e6bb710e
80 changed files with 4115 additions and 1770 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
example/ios/Frame Processor Plugins/Example Plugin (Objective-C)/ExampleFrameProcessorPlugin.m Normal file
View File

@@ -0,0 +1,41 @@
//
// ExampleFrameProcessorPlugin.m
// VisionCameraExample
//
// Created by Marc Rousavy on 01.05.21.
//
#import <Foundation/Foundation.h>
#import <VisionCamera/FrameProcessorPlugin.h>
#import <VisionCamera/Frame.h>
// Example for an Objective-C Frame Processor plugin
@interface ExampleFrameProcessorPlugin : NSObject
@end
@implementation ExampleFrameProcessorPlugin
static inline id example_plugin(Frame* frame, NSArray* arguments) {
CVPixelBufferRef imageBuffer = CMSampleBufferGetImageBuffer(frame.buffer);
NSLog(@"ExamplePlugin: %zu x %zu Image. Logging %lu parameters:", CVPixelBufferGetWidth(imageBuffer), CVPixelBufferGetHeight(imageBuffer), (unsigned long)arguments.count);
for (id param in arguments) {
NSLog(@"ExamplePlugin: -> %@ (%@)", param == nil ? @"(nil)" : [param description], NSStringFromClass([param classForCoder]));
}
return @{
@"example_str": @"Test",
@"example_bool": @true,
@"example_double": @5.3,
@"example_array": @[
@"Hello",
@true,
@17.38
]
};
}
VISION_EXPORT_FRAME_PROCESSOR(example_plugin)
@end

4
example/ios/Frame Processor Plugins/QR Code Plugin (Swift)/QRCodeFrameProcessorPluginSwift.m → example/ios/Frame Processor Plugins/Example Plugin (Swift)/ExamplePluginSwift.m
View File

@@ -1,5 +1,5 @@
//
// QRCodeFrameProcessorPluginSwift.m
// ExamplePluginSwift.m
// VisionCamera
//
// Created by Marc Rousavy on 01.05.21.
@@ -9,5 +9,5 @@
#import <Foundation/Foundation.h>
#import <VisionCamera/FrameProcessorPlugin.h>
@interface VISION_EXPORT_SWIFT_FRAME_PROCESSOR(exampleSwift___scanQRCodes, QRCodeFrameProcessorPluginSwift)
@interface VISION_EXPORT_SWIFT_FRAME_PROCESSOR(example_plugin_swift, ExamplePluginSwift)
@end

42
example/ios/Frame Processor Plugins/Example Plugin (Swift)/ExamplePluginSwift.swift Normal file
View File

@@ -0,0 +1,42 @@
//
// ExamplePluginSwift.swift
// VisionCamera
//
// Created by Marc Rousavy on 30.04.21.
// Copyright © 2021 mrousavy. All rights reserved.
//
import AVKit
import Vision
@objc(ExamplePluginSwift)
public class ExamplePluginSwift: NSObject, FrameProcessorPluginBase {
@objc
public static func callback(_ frame: Frame!, withArgs args: [Any]!) -> Any! {
guard let imageBuffer = CMSampleBufferGetImageBuffer(frame.buffer) else {
return nil
}
NSLog("ExamplePlugin: \(CVPixelBufferGetWidth(imageBuffer)) x \(CVPixelBufferGetHeight(imageBuffer)) Image. Logging \(args.count) parameters:")
args.forEach { arg in
var string = "\(arg)"
if let array = arg as? NSArray {
string = (array as Array).description
} else if let map = arg as? NSDictionary {
string = (map as Dictionary).description
}
NSLog("ExamplePlugin: -> \(string) (\(type(of: arg)))")
}
return [
"example_str": "Test",
"example_bool": true,
"example_double": 5.3,
"example_array": [
"Hello",
true,
17.38,
],
]
}
}

26
example/ios/Frame Processor Plugins/QR Code Plugin (Objective-C)/QRCodeFrameProcessorPluginObjC.m
View File

@@ -1,26 +0,0 @@
//
// QRCodeFrameProcessorPluginObjC.m
// VisionCameraExample
//
// Created by Marc Rousavy on 01.05.21.
//
#import <Foundation/Foundation.h>
#import <VisionCamera/FrameProcessorPlugin.h>
#import <VisionCamera/Frame.h>
// Example for an Objective-C Frame Processor plugin
@interface QRCodeFrameProcessorPluginObjC : NSObject
@end
@implementation QRCodeFrameProcessorPluginObjC
static inline id exampleObjC___scanQRCodes(Frame* frame, NSArray* arguments) {
// TODO: Use some AI to detect QR codes in the frame
return @[];
}
VISION_EXPORT_FRAME_PROCESSOR(exampleObjC___scanQRCodes)
@end

19
example/ios/Frame Processor Plugins/QR Code Plugin (Swift)/QRCodeFrameProcessorPluginSwift.swift
View File

@@ -1,19 +0,0 @@
//
// QRCodeFrameProcessorPluginSwift.swift
// VisionCamera
//
// Created by Marc Rousavy on 30.04.21.
// Copyright © 2021 mrousavy. All rights reserved.
//
import AVKit
import Vision
@objc(QRCodeFrameProcessorPluginSwift)
public class QRCodeFrameProcessorPluginSwift: NSObject, FrameProcessorPluginBase {
@objc
public static func callback(_: Frame!, withArgs _: [Any]!) -> Any! {
// TODO: Use some AI to detect QR codes in the CMSampleBufferRef
[]
}
}

4
example/ios/Podfile.lock
View File

@@ -322,7 +322,7 @@ PODS:
- React
- RNVectorIcons (8.1.0):
- React-Core
- VisionCamera (2.3.0):
- VisionCamera (2.4.1):
- React
- React-callinvoker
- React-Core
@@ -490,7 +490,7 @@ SPEC CHECKSUMS:
RNReanimated: 9c13c86454bfd54dab7505c1a054470bfecd2563
RNStaticSafeAreaInsets: 6103cf09647fa427186d30f67b0f5163c1ae8252
RNVectorIcons: 31cebfcf94e8cf8686eb5303ae0357da64d7a5a4
VisionCamera: aa5a3c3fd9f32a3d9836872b4aea04b600dddd75
VisionCamera: b4cdf9509f6a25b6eff0fbc2ac5fc1d61fc36d54
Yoga: 575c581c63e0d35c9a83f4b46d01d63abc1100ac
PODFILE CHECKSUM: 4b093c1d474775c2eac3268011e4b0b80929d3a2

36
example/ios/VisionCameraExample.xcodeproj/project.pbxproj
View File

@@ -12,9 +12,9 @@
13B07FC11A68108700A75B9A /* main.m in Sources */ = {isa = PBXBuildFile; fileRef = 13B07FB71A68108700A75B9A /* main.m */; };
81AB9BB82411601600AC10FF /* LaunchScreen.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 81AB9BB72411601600AC10FF /* LaunchScreen.storyboard */; };
B8DB3BD5263DE8B7004C18D7 /* BuildFile in Sources */ = {isa = PBXBuildFile; };
B8DB3BDC263DEA31004C18D7 /* QRCodeFrameProcessorPluginObjC.m in Sources */ = {isa = PBXBuildFile; fileRef = B8DB3BD8263DEA31004C18D7 /* QRCodeFrameProcessorPluginObjC.m */; };
B8DB3BDD263DEA31004C18D7 /* QRCodeFrameProcessorPluginSwift.swift in Sources */ = {isa = PBXBuildFile; fileRef = B8DB3BDA263DEA31004C18D7 /* QRCodeFrameProcessorPluginSwift.swift */; };
B8DB3BDE263DEA31004C18D7 /* QRCodeFrameProcessorPluginSwift.m in Sources */ = {isa = PBXBuildFile; fileRef = B8DB3BDB263DEA31004C18D7 /* QRCodeFrameProcessorPluginSwift.m */; };
B8DB3BDC263DEA31004C18D7 /* ExampleFrameProcessorPlugin.m in Sources */ = {isa = PBXBuildFile; fileRef = B8DB3BD8263DEA31004C18D7 /* ExampleFrameProcessorPlugin.m */; };
B8DB3BDD263DEA31004C18D7 /* ExamplePluginSwift.swift in Sources */ = {isa = PBXBuildFile; fileRef = B8DB3BDA263DEA31004C18D7 /* ExamplePluginSwift.swift */; };
B8DB3BDE263DEA31004C18D7 /* ExamplePluginSwift.m in Sources */ = {isa = PBXBuildFile; fileRef = B8DB3BDB263DEA31004C18D7 /* ExamplePluginSwift.m */; };
B8F0E10825E0199F00586F16 /* File.swift in Sources */ = {isa = PBXBuildFile; fileRef = B8F0E10725E0199F00586F16 /* File.swift */; };
D27D5196997E7C9532F05776 /* libPods-VisionCameraExample.a in Frameworks */ = {isa = PBXBuildFile; fileRef = 321EF902E53C449CC036AD27 /* libPods-VisionCameraExample.a */; };
/* End PBXBuildFile section */
@@ -31,9 +31,9 @@
321EF902E53C449CC036AD27 /* libPods-VisionCameraExample.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = "libPods-VisionCameraExample.a"; sourceTree = BUILT_PRODUCTS_DIR; };
81AB9BB72411601600AC10FF /* LaunchScreen.storyboard */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = file.storyboard; name = LaunchScreen.storyboard; path = VisionCameraExample/LaunchScreen.storyboard; sourceTree = "<group>"; };
9993ED8FD4B3171BB28C87C9 /* Pods-VisionCameraExample.debug.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-VisionCameraExample.debug.xcconfig"; path = "Target Support Files/Pods-VisionCameraExample/Pods-VisionCameraExample.debug.xcconfig"; sourceTree = "<group>"; };
B8DB3BD8263DEA31004C18D7 /* QRCodeFrameProcessorPluginObjC.m */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.objc; path = QRCodeFrameProcessorPluginObjC.m; sourceTree = "<group>"; };
B8DB3BDA263DEA31004C18D7 /* QRCodeFrameProcessorPluginSwift.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = QRCodeFrameProcessorPluginSwift.swift; sourceTree = "<group>"; };
B8DB3BDB263DEA31004C18D7 /* QRCodeFrameProcessorPluginSwift.m */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.objc; path = QRCodeFrameProcessorPluginSwift.m; sourceTree = "<group>"; };
B8DB3BD8263DEA31004C18D7 /* ExampleFrameProcessorPlugin.m */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.objc; path = ExampleFrameProcessorPlugin.m; sourceTree = "<group>"; };
B8DB3BDA263DEA31004C18D7 /* ExamplePluginSwift.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = ExamplePluginSwift.swift; sourceTree = "<group>"; };
B8DB3BDB263DEA31004C18D7 /* ExamplePluginSwift.m */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.objc; path = ExamplePluginSwift.m; sourceTree = "<group>"; };
B8F0E10625E0199F00586F16 /* VisionCameraExample-Bridging-Header.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = "VisionCameraExample-Bridging-Header.h"; sourceTree = "<group>"; };
B8F0E10725E0199F00586F16 /* File.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = File.swift; sourceTree = "<group>"; };
ED297162215061F000B7C4FE /* JavaScriptCore.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = JavaScriptCore.framework; path = System/Library/Frameworks/JavaScriptCore.framework; sourceTree = SDKROOT; };
@@ -120,27 +120,27 @@
B8DB3BD6263DEA31004C18D7 /* Frame Processor Plugins */ = {
isa = PBXGroup;
children = (
B8DB3BD7263DEA31004C18D7 /* QR Code Plugin (Objective-C) */,
B8DB3BD9263DEA31004C18D7 /* QR Code Plugin (Swift) */,
B8DB3BD7263DEA31004C18D7 /* Example Plugin (Objective-C) */,
B8DB3BD9263DEA31004C18D7 /* Example Plugin (Swift) */,
);
path = "Frame Processor Plugins";
sourceTree = "<group>";
};
B8DB3BD7263DEA31004C18D7 /* QR Code Plugin (Objective-C) */ = {
B8DB3BD7263DEA31004C18D7 /* Example Plugin (Objective-C) */ = {
isa = PBXGroup;
children = (
B8DB3BD8263DEA31004C18D7 /* QRCodeFrameProcessorPluginObjC.m */,
B8DB3BD8263DEA31004C18D7 /* ExampleFrameProcessorPlugin.m */,
);
path = "QR Code Plugin (Objective-C)";
path = "Example Plugin (Objective-C)";
sourceTree = "<group>";
};
B8DB3BD9263DEA31004C18D7 /* QR Code Plugin (Swift) */ = {
B8DB3BD9263DEA31004C18D7 /* Example Plugin (Swift) */ = {
isa = PBXGroup;
children = (
B8DB3BDA263DEA31004C18D7 /* QRCodeFrameProcessorPluginSwift.swift */,
B8DB3BDB263DEA31004C18D7 /* QRCodeFrameProcessorPluginSwift.m */,
B8DB3BDA263DEA31004C18D7 /* ExamplePluginSwift.swift */,
B8DB3BDB263DEA31004C18D7 /* ExamplePluginSwift.m */,
);
path = "QR Code Plugin (Swift)";
path = "Example Plugin (Swift)";
sourceTree = "<group>";
};
/* End PBXGroup section */
@@ -363,12 +363,12 @@
buildActionMask = 2147483647;
files = (
13B07FBC1A68108700A75B9A /* AppDelegate.m in Sources */,
B8DB3BDC263DEA31004C18D7 /* QRCodeFrameProcessorPluginObjC.m in Sources */,
B8DB3BDC263DEA31004C18D7 /* ExampleFrameProcessorPlugin.m in Sources */,
B8DB3BD5263DE8B7004C18D7 /* BuildFile in Sources */,
B8DB3BDD263DEA31004C18D7 /* QRCodeFrameProcessorPluginSwift.swift in Sources */,
B8DB3BDD263DEA31004C18D7 /* ExamplePluginSwift.swift in Sources */,
B8F0E10825E0199F00586F16 /* File.swift in Sources */,
13B07FC11A68108700A75B9A /* main.m in Sources */,
B8DB3BDE263DEA31004C18D7 /* QRCodeFrameProcessorPluginSwift.m in Sources */,
B8DB3BDE263DEA31004C18D7 /* ExamplePluginSwift.m in Sources */,
);
runOnlyForDeploymentPostprocessing = 0;
};