railbird/react-native-vision-camera
10
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feature: Frame Processors (iOS) (#2)

Browse Source 
* Clean up Frame Processor

* Create FrameProcessorHolder

* Create FrameProcessorDelegate in ObjC++

* Move frame processor to FrameProcessorDelegate

* Decorate runtime, check for null

* Update FrameProcessorDelegate.mm

* Cleanup FrameProcessorBindings.mm

* Fix RuntimeDecorator.h import

* Update FrameProcessorDelegate.mm

* "React" -> "React Helper" to avoid confusion

* Rename folders again

* Fix podspec flattening a lot of headers, causing REA nameclash

* Fix header imports to avoid REA naming collision

* Lazily initialize jsi::Runtime on DispatchQueue

* Install frame processor bindings from Swift

* First try to call jsi::Function (frame processor) 👀

* Call viewForReactTag on RCT main thread

* Fix bridge accessing

* Add more logs

* Update CameraViewManager.swift

* Add more TODOs

* Re-indent .cpp files

* Fix RCTTurboModule import podspec

* Remove unnecessary include check for swift umbrella header

* Merge branch 'main' into frame-processors

* Docs: use static width for images (283)

* Create validate-cpp.yml

* Update a lot of packages to latest

* Set SWIFT_VERSION to 5.2 in podspec

* Create clean.sh

* Delete unused C++ files

* podspec: Remove CLANG_CXX_LANGUAGE_STANDARD and OTHER_CFLAGS

* Update pod lockfiles

* Regenerate lockfiles

* Remove IOSLogger

* Use NSLog

* Create FrameProcessorManager (inherits from REA RuntimeManager)

* Create reanimated::RuntimeManager shared_ptr

* Re-integrate pods

* Add react-native-reanimated >=2 peerDependency

* Add metro-config

* blacklist -> exclusionList

* Try to call worklet

* Fix jsi::Value* initializer

* Call ShareableValue::adapt (makeShareable) with React/JS Runtime

* Add null-checks

* Lift runtime manager creation out of delegate, into bindings

* Remove debug statement

* Make RuntimeManager unique_ptr

* Set _FRAME_PROCESSOR

* Extract convertJSIFunctionToFrameProcessorCallback

* Print frame

* Merge branch 'main' into frame-processors

* Reformat Swift code

* Install reanimated from npm again

* Re-integrate Pods

* Dependabot: Also scan example/ and docs/

* Update validate-cpp.yml

* Create FrameProcessorUtils

* Create Frame.h

* Abstract HostObject creation away

* Fix types

* Fix frame processor call

* Add todo

* Update lockfiles

* Add C++ contributing instructions

* Update CONTRIBUTING.md

* Add android/src/main/cpp to cpplint

* Update cpplint.sh

* Fix a few cpplint errors

* Fix globals

* Fix a few more cpplint errors

* Update App.tsx

* Update AndroidLogger.cpp

* Format

* Fix cpplint script (check-cpp)

* Try to simplify frame processor

* y

* Update FrameProcessorUtils.mm

* Update FrameProcessorBindings.mm

* Update CameraView.swift

* Update CameraViewManager.m

* Restructure everything

* fix

* Fix `@objc` export (make public)

* Refactor installFrameProcessorBindings into FrameProcessorRuntimeManager

* Add swift RCTBridge.runOnJS helper

* Fix run(onJS)

* Add pragma once

* Add `&self` to lambda

* Update FrameProcessorRuntimeManager.mm

* reorder imports

* Fix imports

* forward declare

* Rename extension

* Destroy buffer after execution

* Add FrameProcessorPluginRegistry base

* Merge branch 'main' into frame-processors

* Add frameProcessor to types

* Update Camera.tsx

* Fix rebase merge

* Remove movieOutput

* Use `useFrameProcessor`

* Fix bad merge

* Add additional ESLint rules

* Update lockfiles

* Update CameraViewManager.m

* Add support for V8 runtime

* Add frame processor plugins API

* Print plugin invoke

* Fix React Utils in podspec

* Fix runOnJS swift name

* Remove invalid redecl of `captureSession`

* Use REA 2.1.0 which includes all my big PRs 🎉

* Update validate-cpp.yml

* Update Podfile.lock

* Remove Flipper

* Fix dereferencing

* Capture `self` by value. Fucking hell, what a dumb mistake.

* Override a few HostObject functions

* Expose isReady, width, height, bytesPerRow and planesCount

* use hook again

* Expose property names

* FrameProcessor -> Frame

* Update CameraView+RecordVideo.swift

* Add Swift support for Frame Processors Plugins

* Add macros for plugin installation

* Add ObjC frame processor plugin

* Correctly install frame processor plugins

* Don't require custom name for macro

* Check if plugin already exists

* Implement QR Code Frame Processor Plugin in Swift

* Adjust ObjC style frame processor macro

* optimize

* Add `frameProcessorFrameDropRate`

* Fix types

* Only log once

* Log if it executes slowly

* Implement `frameProcessorFps`

* Implement manual encoded video recordings

* Use recommended video settings

* Add fileType types

* Ignore if input is not ready for media data

* Add completion handler

* Add audio buffer sampling

* Init only for video frame

* use AVAssetWriterInputPixelBufferAdaptor

* Remove AVAssetWriterInputPixelBufferAdaptor

* Rotate VideoWriter

* Always assume portrait orientation

* Update RecordingSession.swift

* Use a separate Queue for Audio

* Format Swift

* Update CameraView+RecordVideo.swift

* Use `videoQueue` instead of `cameraQueue`

* Move example plugins to example app

* Fix hardcoded name in plugin macro

* QRFrame... -> QRCodeFrame...

* Update FrameProcessorPlugin.h

* Add example frame processors to JS base

* Update QRCodeFrameProcessorPluginSwift.m

* Add docs to create FP Plugins

* Update FRAME_PROCESSORS_CREATE.mdx

* Update FRAME_PROCESSORS_CREATE.mdx

* Use `AVAssetWriterInputPixelBufferAdaptor` for efficient pixel buffer recycling

* Add customizable `pixelFormat`

* Use native format if available

* Update project.pbxproj

* Set video width and height as source-pixel-buffer attributes

* Catch

* Update App.tsx

* Don't explicitly set video dimensions, let CVPixelBufferPool handle it

* Add a few logs

* Cleanup

* Update CameraView+RecordVideo.swift

* Eagerly initialize asset writer to fix stutter at first frame

* Use `cameraQueue` DispatchQueue to not block CaptureDataOutputDelegate

* Fix duration calculation

* cleanup

* Cleanup

* Swiftformat

* Return available video codecs

* Only show frame drop notification for video output

* Remove photo and video codec functionality

It was too much complexity and probably never used anyways.

* Revert all android related changes for now

* Cleanup

* Remove unused header

* Update AVAssetWriter.Status+descriptor.swift

* Only call Frame Processor for Video Frames

* Fix `if`

* Add support for Frame Processor plugin parameters/arguments

* Fix arg support

* Move to JSIUtils.mm

* Update JSIUtils.h

* Update FRAME_PROCESSORS_CREATE.mdx

* Update FRAME_PROCESSORS_CREATE.mdx

* Upgrade packages for docs/

* fix docs

* Rename

* highlight lines

* docs

* community plugins

* Update FRAME_PROCESSOR_CREATE_FINAL.mdx

* Update FRAME_PROCESSOR_PLUGIN_LIST.mdx

* Update FRAME_PROCESSOR_PLUGIN_LIST.mdx

* Update dependencies (1/2)

* Update dependencies (2/2)

* Update Gemfile.lock

* add FP docs

* Update README.md

* Make `lastFrameProcessor` private

* add `frameProcessor` docs

* fix docs

* adjust docs

* Update DEVICES.mdx

* fix

* s

* Add logs demo

* add metro restart note

* Update FRAME_PROCESSOR_CREATE_PLUGIN_IOS.mdx

* Mirror video device

* Update AVCaptureVideoDataOutput+mirror.swift

* Create .swift-version

* Enable whole module optimization

* Fix recording mirrored video

* Swift format

* Clean dictionary on `markInvalid`

* Fix cleanup

* Add docs for disabling frame processors

* Update project.pbxproj

* Revert "Update project.pbxproj"

This reverts commit e67861e51b88b4888a6940e2d20388f3044211d0.

* Log frame drop reason

* Format

* add more samples

* Add clang-format

* also check .mm

* Revert "also check .mm"

This reverts commit 8b9d5e2c29866b05909530d104f6633d6c49eadd.

* Revert "Add clang-format"

This reverts commit 7643ac808e0fc34567ea1f814e73d84955381636.

* Use `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange` as default

* Read matching video attributes from videoSettings

* Add TODO

* Swiftformat

* Conditionally disable frame processors

* Assert if trying to use frame processors when disabled

* Add frame-processors demo gif

* Allow disabling frame processors via `VISION_CAMERA_DISABLE_FRAME_PROCESSORS`

* Update FrameProcessorRuntimeManager.mm

* Update FRAME_PROCESSORS.mdx

* Update project.pbxproj

* Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
This commit is contained in:
Marc Rousavy
2021-05-06 14:11:55 +02:00
committed by GitHub
parent 77b3d78566
commit b6a67d5ced
100 changed files with 4703 additions and 3664 deletions
Download Patch File Download Diff File Expand all files Collapse all files

131
example/ios/Podfile.lock
View File

@@ -1,6 +1,5 @@
PODS:
- boost-for-react-native (1.63.0)
- CocoaAsyncSocket (7.6.5)
- DoubleConversion (1.1.6)
- FBLazyVector (0.64.0)
- FBReactNativeSpec (0.64.0):
@@ -10,56 +9,7 @@ PODS:
- React-Core (= 0.64.0)
- React-jsi (= 0.64.0)
- ReactCommon/turbomodule/core (= 0.64.0)
- Flipper (0.75.1):
- Flipper-Folly (~> 2.5)
- Flipper-RSocket (~> 1.3)
- Flipper-DoubleConversion (1.1.7)
- Flipper-Folly (2.5.1):
- boost-for-react-native
- Flipper-DoubleConversion
- Flipper-Glog
- libevent (~> 2.1.12)
- OpenSSL-Universal (= 1.1.180)
- Flipper-Glog (0.3.6)
- Flipper-PeerTalk (0.0.4)
- Flipper-RSocket (1.3.0):
- Flipper-Folly (~> 2.5)
- FlipperKit (0.75.1):
- FlipperKit/Core (= 0.75.1)
- FlipperKit/Core (0.75.1):
- Flipper (~> 0.75.1)
- FlipperKit/CppBridge
- FlipperKit/FBCxxFollyDynamicConvert
- FlipperKit/FBDefines
- FlipperKit/FKPortForwarding
- FlipperKit/CppBridge (0.75.1):
- Flipper (~> 0.75.1)
- FlipperKit/FBCxxFollyDynamicConvert (0.75.1):
- Flipper-Folly (~> 2.5)
- FlipperKit/FBDefines (0.75.1)
- FlipperKit/FKPortForwarding (0.75.1):
- CocoaAsyncSocket (~> 7.6)
- Flipper-PeerTalk (~> 0.0.4)
- FlipperKit/FlipperKitHighlightOverlay (0.75.1)
- FlipperKit/FlipperKitLayoutPlugin (0.75.1):
- FlipperKit/Core
- FlipperKit/FlipperKitHighlightOverlay
- FlipperKit/FlipperKitLayoutTextSearchable
- YogaKit (~> 1.18)
- FlipperKit/FlipperKitLayoutTextSearchable (0.75.1)
- FlipperKit/FlipperKitNetworkPlugin (0.75.1):
- FlipperKit/Core
- FlipperKit/FlipperKitReactPlugin (0.75.1):
- FlipperKit/Core
- FlipperKit/FlipperKitUserDefaultsPlugin (0.75.1):
- FlipperKit/Core
- FlipperKit/SKIOSNetworkPlugin (0.75.1):
- FlipperKit/Core
- FlipperKit/FlipperKitNetworkPlugin
- glog (0.3.5)
- hermes-engine (0.7.2)
- libevent (2.1.12)
- OpenSSL-Universal (1.1.180)
- RCT-Folly (2020.01.13.00):
- boost-for-react-native
- DoubleConversion
@@ -69,11 +19,6 @@ PODS:
- boost-for-react-native
- DoubleConversion
- glog
- RCT-Folly/Futures (2020.01.13.00):
- boost-for-react-native
- DoubleConversion
- glog
- libevent
- RCTRequired (0.64.0)
- RCTTypeSafety (0.64.0):
- FBLazyVector (= 0.64.0)
@@ -131,16 +76,6 @@ PODS:
- React-jsinspector (= 0.64.0)
- React-perflogger (= 0.64.0)
- Yoga
- React-Core/Hermes (0.64.0):
- glog
- hermes-engine
- RCT-Folly (= 2020.01.13.00)
- RCT-Folly/Futures
- React-cxxreact (= 0.64.0)
- React-jsi (= 0.64.0)
- React-jsiexecutor (= 0.64.0)
- React-perflogger (= 0.64.0)
- Yoga
- React-Core/RCTActionSheetHeaders (0.64.0):
- glog
- RCT-Folly (= 2020.01.13.00)
@@ -270,7 +205,7 @@ PODS:
- React-jsinspector (0.64.0)
- react-native-blur (0.8.0):
- React
- react-native-cameraroll (4.0.2):
- react-native-cameraroll (4.0.4):
- React-Core
- react-native-slider (3.0.3):
- React
@@ -354,7 +289,7 @@ PODS:
- React-RCTText
- RNGestureHandler (1.10.3):
- React-Core
- RNReanimated (2.0.0):
- RNReanimated (2.1.0):
- DoubleConversion
- FBLazyVector
- FBReactNativeSpec
@@ -387,38 +322,17 @@ PODS:
- React
- RNVectorIcons (8.1.0):
- React-Core
- VisionCamera (1.0.4):
- VisionCamera (1.0.10):
- React
- React-callinvoker
- React-Core
- Yoga (1.14.0)
- YogaKit (1.18.1):
- Yoga (~> 1.14)
DEPENDENCIES:
- DoubleConversion (from `../node_modules/react-native/third-party-podspecs/DoubleConversion.podspec`)
- FBLazyVector (from `../node_modules/react-native/Libraries/FBLazyVector`)
- FBReactNativeSpec (from `../node_modules/react-native/React/FBReactNativeSpec`)
- Flipper (~> 0.75.1)
- Flipper-DoubleConversion (= 1.1.7)
- Flipper-Folly (~> 2.5)
- Flipper-Glog (= 0.3.6)
- Flipper-PeerTalk (~> 0.0.4)
- Flipper-RSocket (~> 1.3)
- FlipperKit (~> 0.75.1)
- FlipperKit/Core (~> 0.75.1)
- FlipperKit/CppBridge (~> 0.75.1)
- FlipperKit/FBCxxFollyDynamicConvert (~> 0.75.1)
- FlipperKit/FBDefines (~> 0.75.1)
- FlipperKit/FKPortForwarding (~> 0.75.1)
- FlipperKit/FlipperKitHighlightOverlay (~> 0.75.1)
- FlipperKit/FlipperKitLayoutPlugin (~> 0.75.1)
- FlipperKit/FlipperKitLayoutTextSearchable (~> 0.75.1)
- FlipperKit/FlipperKitNetworkPlugin (~> 0.75.1)
- FlipperKit/FlipperKitReactPlugin (~> 0.75.1)
- FlipperKit/FlipperKitUserDefaultsPlugin (~> 0.75.1)
- FlipperKit/SKIOSNetworkPlugin (~> 0.75.1)
- glog (from `../node_modules/react-native/third-party-podspecs/glog.podspec`)
- hermes-engine (~> 0.7.2)
- libevent (~> 2.1.12)
- RCT-Folly (from `../node_modules/react-native/third-party-podspecs/RCT-Folly.podspec`)
- RCTRequired (from `../node_modules/react-native/Libraries/RCTRequired`)
- RCTTypeSafety (from `../node_modules/react-native/Libraries/TypeSafety`)
@@ -426,7 +340,6 @@ DEPENDENCIES:
- React-callinvoker (from `../node_modules/react-native/ReactCommon/callinvoker`)
- React-Core (from `../node_modules/react-native/`)
- React-Core/DevSupport (from `../node_modules/react-native/`)
- React-Core/Hermes (from `../node_modules/react-native/`)
- React-Core/RCTWebSocket (from `../node_modules/react-native/`)
- React-CoreModules (from `../node_modules/react-native/React/CoreModules`)
- React-cxxreact (from `../node_modules/react-native/ReactCommon/cxxreact`)
@@ -460,18 +373,6 @@ DEPENDENCIES:
SPEC REPOS:
trunk:
- boost-for-react-native
- CocoaAsyncSocket
- Flipper
- Flipper-DoubleConversion
- Flipper-Folly
- Flipper-Glog
- Flipper-PeerTalk
- Flipper-RSocket
- FlipperKit
- hermes-engine
- libevent
- OpenSSL-Universal
- YogaKit
EXTERNAL SOURCES:
DoubleConversion:
@@ -553,21 +454,10 @@ EXTERNAL SOURCES:
SPEC CHECKSUMS:
boost-for-react-native: 39c7adb57c4e60d6c5479dd8623128eb5b3f0f2c
CocoaAsyncSocket: 065fd1e645c7abab64f7a6a2007a48038fdc6a99
DoubleConversion: cf9b38bf0b2d048436d9a82ad2abe1404f11e7de
FBLazyVector: 49cbe4b43e445b06bf29199b6ad2057649e4c8f5
FBReactNativeSpec: 06c29ba6920affcab9cda6154497386d21f43410
Flipper: d3da1aa199aad94455ae725e9f3aa43f3ec17021
Flipper-DoubleConversion: 38631e41ef4f9b12861c67d17cb5518d06badc41
Flipper-Folly: f7a3caafbd74bda4827954fd7a6e000e36355489
Flipper-Glog: 1dfd6abf1e922806c52ceb8701a3599a79a200a6
Flipper-PeerTalk: 116d8f857dc6ef55c7a5a75ea3ceaafe878aadc9
Flipper-RSocket: 602921fee03edacf18f5d6f3d3594ba477f456e5
FlipperKit: 8a20b5c5fcf9436cac58551dc049867247f64b00
FBReactNativeSpec: e800dc469340da7e8e47f45145f69d75a7b06874
glog: 73c2498ac6884b13ede40eda8228cb1eee9d9d62
hermes-engine: 7d97ba46a1e29bacf3e3c61ecb2804a5ddd02d4f
libevent: 4049cae6c81cdb3654a443be001fb9bdceff7913
OpenSSL-Universal: 1aa4f6a6ee7256b83db99ec1ccdaa80d10f9af9b
RCT-Folly: ec7a233ccc97cc556cf7237f0db1ff65b986f27c
RCTRequired: 2f8cb5b7533219bf4218a045f92768129cf7050a
RCTTypeSafety: 512728b73549e72ad7330b92f3d42936f2a4de5b
@@ -580,7 +470,7 @@ SPEC CHECKSUMS:
React-jsiexecutor: 06a9c77b56902ae7ffcdd7a4905f664adc5d237b
React-jsinspector: 0ae35a37b20d5e031eb020a69cc5afdbd6406301
react-native-blur: cad4d93b364f91e7b7931b3fa935455487e5c33c
react-native-cameraroll: 1965db75c851b15e77a22ca0ac78e32af6b571ae
react-native-cameraroll: 88f4e62d9ecd0e1f253abe4f685474f2ea14bfa2
react-native-slider: e99fc201cefe81270fc9d81714a7a0f5e566b168
react-native-video: 0bb76b6d6b77da3009611586c7dbf817b947f30e
React-perflogger: 9c547d8f06b9bf00cb447f2b75e8d7f19b7e02af
@@ -597,13 +487,12 @@ SPEC CHECKSUMS:
ReactCommon: cfe2b7fd20e0dbd2d1185cd7d8f99633fbc5ff05
ReactNativeNavigation: 63321d37e8172bdcc1fbb93e77cffc74a0ba6d20
RNGestureHandler: a479ebd5ed4221a810967000735517df0d2db211
RNReanimated: 64f6c5789f82818c07ba3c71864b73619cb23c76
RNReanimated: b8c8004b43446e3c2709fe64b2b41072f87428ad
RNStaticSafeAreaInsets: 6103cf09647fa427186d30f67b0f5163c1ae8252
RNVectorIcons: 31cebfcf94e8cf8686eb5303ae0357da64d7a5a4
VisionCamera: df21c7ec84e2fb254ee0e085bafcb95b0217376a
VisionCamera: 65836f8bb8dda4e718c350e32c35a0b9f166b731
Yoga: 8c8436d4171c87504c648ae23b1d81242bdf3bbf
YogaKit: f782866e155069a2cca2517aafea43200b01fd5a
PODFILE CHECKSUM: 703892d4cb39598dd9a51bd3a58fe2502c68dfbd
PODFILE CHECKSUM: 4b093c1d474775c2eac3268011e4b0b80929d3a2
COCOAPODS: 1.10.1