Marc Rousavy
30b56153db
feat: Sync Frame Processors (plus runAsync and runAtTargetFps) ( #1472 )
...
Before, Frame Processors ran on a separate Thread.
After, Frame Processors run fully synchronous and always at the same FPS as the Camera.
Two new functions have been introduced:
* `runAtTargetFps(fps: number, func: () => void)`: Runs the given code as often as the given `fps`, effectively throttling it's calls.
* `runAsync(frame: Frame, func: () => void)`: Runs the given function on a separate Thread for Frame Processing. A strong reference to the Frame is held as long as the function takes to execute.
You can use `runAtTargetFps` to throttle calls to a specific API (e.g. if your Camera is running at 60 FPS, but you only want to run face detection at ~25 FPS, use `runAtTargetFps(25, ...)`.)
You can use `runAsync` to run a heavy algorithm asynchronous, so that the Camera is not blocked while your algorithm runs. This is useful if your main sync processor draws something, and your async processor is doing some image analysis on the side.
You can also combine both functions.
Examples:
```js
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
console.log("I'm running at 60 FPS!")
}, [])
```
```js
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
console.log("I'm running at 60 FPS!")
runAtTargetFps(10, () => {
'worklet'
console.log("I'm running at 10 FPS!")
})
}, [])
```
```js
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
console.log("I'm running at 60 FPS!")
runAsync(frame, () => {
'worklet'
console.log("I'm running on another Thread, I can block for longer!")
})
}, [])
```
```js
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
console.log("I'm running at 60 FPS!")
runAtTargetFps(10, () => {
'worklet'
runAsync(frame, () => {
'worklet'
console.log("I'm running on another Thread at 10 FPS, I can block for longer!")
})
})
}, [])
```
2023-02-15 16:47:09 +01:00
Marc Rousavy
a0590dccb5
feat: Replace Reanimated with RN Worklets ( #1468 )
...
* Setup RN Worklets
* Use RN Worklets on iOS
* Fix console
* Add `installFrameProcessorBindings()` function
* Add `FrameProcessorPlugins` proxy (BREAKING CHANGE)
* Clean up docs
* Update FRAME_PROCESSORS.mdx
* Use RN Worklets 0.2.5
* feat: Android build setup
* Rewrite Android Frame Processor Part
* Update CMakeLists.txt
* fix: Add react-native-worklets Gradle dependency
* Update Podfile.lock
* fix build
* gradle:7.4.1
* Init JSI Bindings in method on Android
* Fix Folly flags
* fix: Init `FrameProcessorRuntimeManager` later
* fix: Wrap in `<GestureHandlerRootView>`
* Refactor plugins
* fix: Remove enableFrameProcessors
* Install RN Worklets from current GH master
* Update babel.config.js
* Update CameraViewModule.kt
* Update ImageProxyUtils.java
* feat: Upgrade to Reanimated v3
* fix: Fix crash on Worklet init
* Update RN Worklets to latest master
* fix: Simplify FP Plugins Proxy
2023-02-13 15:22:45 +01:00
Marc Rousavy
be5ec69b02
feat: Make Reanimated optional (disable Frame Processors if REA v2 is not installed) ( #412 )
...
* Fix building iOS without Reanimated
* Conditionally compile Frame Processors (gradle)
* Conditionally use externalNativeBuild
* Remove Reanimated import
* fix: Conditionally load REA/VisionCamera libraries
* fix: Add disable FP to docs
* fix: Fix dummy placeholder for Scheduler.mm
* fix: Fix dummy `Scheduler` declaration
* fix: Only init `CameraView` C++ side if frame processors are enabled
* fix: Install JSI Bindings on Frame Processor Manager ctor
* fix: Wrong conditional
* whoops
2022-01-02 17:35:26 +01:00
Marc Rousavy
b72176fae9
fix: Re-install FP plugins on bundle reload ( #684 )
2021-12-30 11:38:54 +01:00
Marc Rousavy
0f7ee51333
feat: Add console logging support for Frame Processors ( #297 )
...
* Try to log to console via runOnJS
* Call `console.log`
* Create custom `VisionCameraScheduler`
* Fix scheduler call
* Call with this
* Fix console setting
* Move J---- to `java-bindings`
* c++ style
* Android: 1/2 Create custom Scheduler
* Android: 2/2 Use custom Scheduler
* Don't use `runOnJS`, use `__callAsync` directly
2021-07-30 10:27:45 +02:00
Marc Rousavy
2fa0f8fd46
chore: Clean up root directory ( #236 )
...
* Clean up root
* remove unused REA patches
* Remove `cpp/` and create headers per platform
* fix `#import`
* fix `VISION_CAMERA_DISABLE_FRAME_PROCESSORS`
2021-07-06 16:42:58 +02:00
Marc Rousavy
9c579c65aa
try: Improvements from WWDC 2021 1:1 workshop ( #197 )
...
* perf: Automatically determine Pixel Format depending on active format. (More efficient video recording 🚀 )
* perf: Skip `AVAssetWriter` transform by directly correctly orienting the Video Output connection
* feat: Support camera flipping while recording
* feat: Run frame processor on separate queue, avoids stutters in video recordigns
* feat: Automatically drop late frame processor frames
2021-06-11 21:06:19 +02:00
Marc Rousavy
16f2a7cdec
chore: Cleanup void returns ( #187 )
...
* Place `return` in `return [void]` on separate line
* format
* Update CameraView+RecordVideo.swift
* f
2021-06-09 11:14:49 +02:00
Marc Rousavy
68a716b506
feat: native Frame type to provide Orientation ( #186 )
...
* Use Frame.h
* Add orientation
* Determine buffer orientation
* Replace plugins
* fix calls
* Update FRAME_PROCESSOR_CREATE_PLUGIN_IOS.mdx
* Update FRAME_PROCESSOR_CREATE_PLUGIN_IOS.mdx
* format
* Update CameraPage.tsx
* Update FRAME_PROCESSOR_CREATE_PLUGIN_IOS.mdx
* Add links to docs
* Use `.` syntax
* Make properties `readonly`
* Fix `@synthesize` backing store
2021-06-09 10:57:05 +02:00
Marc Rousavy
be30d55df5
docs: Frame Processor Java Types ( #183 )
...
* add a few spaces
* Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
* Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
* Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
* Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
2021-06-08 10:15:34 +02:00
Marc Rousavy
71730a73ef
fix: Fix AVAudioSession not allowing background music playback ( #155 )
...
* Set category always if different
* rename org
* Fix video format sorting
* fix format filtering
* Update AVAudioSession+setCategoryIfNotSet.swift
* upgrade all dependencies
* Also run dependabot for JS codebase
* Update MediaPage.tsx
* Use typescript 4.2.4
* Also run TS in check-all
* Downgrade typescript to 4.2.3
* f
* recreate lockfiles
* docs: Revert package.json changes
* revert all package.json changes
* Update Podfile.lock
* bump all dependencies, pin typescript to 4.2.4
* Downgrade react-native-navigation for now
* upgrade to later snapshot
* Update yarn.lock
* remove yeet
2021-06-01 13:07:57 +02:00
Marc Rousavy
deb8beb688
chore: Bump reanimated requirements ( #149 )
...
* Bump the reanimated requirement to 2.2.0 and above
* bump versions podfile
* Simplify Frame Processor
* Add `'worklet'` note
2021-05-27 11:08:57 +02:00
Marc Rousavy
b6a67d5ced
feature: Frame Processors (iOS) ( #2 )
...
* Clean up Frame Processor
* Create FrameProcessorHolder
* Create FrameProcessorDelegate in ObjC++
* Move frame processor to FrameProcessorDelegate
* Decorate runtime, check for null
* Update FrameProcessorDelegate.mm
* Cleanup FrameProcessorBindings.mm
* Fix RuntimeDecorator.h import
* Update FrameProcessorDelegate.mm
* "React" -> "React Helper" to avoid confusion
* Rename folders again
* Fix podspec flattening a lot of headers, causing REA nameclash
* Fix header imports to avoid REA naming collision
* Lazily initialize jsi::Runtime on DispatchQueue
* Install frame processor bindings from Swift
* First try to call jsi::Function (frame processor) 👀
* Call viewForReactTag on RCT main thread
* Fix bridge accessing
* Add more logs
* Update CameraViewManager.swift
* Add more TODOs
* Re-indent .cpp files
* Fix RCTTurboModule import podspec
* Remove unnecessary include check for swift umbrella header
* Merge branch 'main' into frame-processors
* Docs: use static width for images (283)
* Create validate-cpp.yml
* Update a lot of packages to latest
* Set SWIFT_VERSION to 5.2 in podspec
* Create clean.sh
* Delete unused C++ files
* podspec: Remove CLANG_CXX_LANGUAGE_STANDARD and OTHER_CFLAGS
* Update pod lockfiles
* Regenerate lockfiles
* Remove IOSLogger
* Use NSLog
* Create FrameProcessorManager (inherits from REA RuntimeManager)
* Create reanimated::RuntimeManager shared_ptr
* Re-integrate pods
* Add react-native-reanimated >=2 peerDependency
* Add metro-config
* blacklist -> exclusionList
* Try to call worklet
* Fix jsi::Value* initializer
* Call ShareableValue::adapt (makeShareable) with React/JS Runtime
* Add null-checks
* Lift runtime manager creation out of delegate, into bindings
* Remove debug statement
* Make RuntimeManager unique_ptr
* Set _FRAME_PROCESSOR
* Extract convertJSIFunctionToFrameProcessorCallback
* Print frame
* Merge branch 'main' into frame-processors
* Reformat Swift code
* Install reanimated from npm again
* Re-integrate Pods
* Dependabot: Also scan example/ and docs/
* Update validate-cpp.yml
* Create FrameProcessorUtils
* Create Frame.h
* Abstract HostObject creation away
* Fix types
* Fix frame processor call
* Add todo
* Update lockfiles
* Add C++ contributing instructions
* Update CONTRIBUTING.md
* Add android/src/main/cpp to cpplint
* Update cpplint.sh
* Fix a few cpplint errors
* Fix globals
* Fix a few more cpplint errors
* Update App.tsx
* Update AndroidLogger.cpp
* Format
* Fix cpplint script (check-cpp)
* Try to simplify frame processor
* y
* Update FrameProcessorUtils.mm
* Update FrameProcessorBindings.mm
* Update CameraView.swift
* Update CameraViewManager.m
* Restructure everything
* fix
* Fix `@objc` export (make public)
* Refactor installFrameProcessorBindings into FrameProcessorRuntimeManager
* Add swift RCTBridge.runOnJS helper
* Fix run(onJS)
* Add pragma once
* Add `&self` to lambda
* Update FrameProcessorRuntimeManager.mm
* reorder imports
* Fix imports
* forward declare
* Rename extension
* Destroy buffer after execution
* Add FrameProcessorPluginRegistry base
* Merge branch 'main' into frame-processors
* Add frameProcessor to types
* Update Camera.tsx
* Fix rebase merge
* Remove movieOutput
* Use `useFrameProcessor`
* Fix bad merge
* Add additional ESLint rules
* Update lockfiles
* Update CameraViewManager.m
* Add support for V8 runtime
* Add frame processor plugins API
* Print plugin invoke
* Fix React Utils in podspec
* Fix runOnJS swift name
* Remove invalid redecl of `captureSession`
* Use REA 2.1.0 which includes all my big PRs 🎉
* Update validate-cpp.yml
* Update Podfile.lock
* Remove Flipper
* Fix dereferencing
* Capture `self` by value. Fucking hell, what a dumb mistake.
* Override a few HostObject functions
* Expose isReady, width, height, bytesPerRow and planesCount
* use hook again
* Expose property names
* FrameProcessor -> Frame
* Update CameraView+RecordVideo.swift
* Add Swift support for Frame Processors Plugins
* Add macros for plugin installation
* Add ObjC frame processor plugin
* Correctly install frame processor plugins
* Don't require custom name for macro
* Check if plugin already exists
* Implement QR Code Frame Processor Plugin in Swift
* Adjust ObjC style frame processor macro
* optimize
* Add `frameProcessorFrameDropRate`
* Fix types
* Only log once
* Log if it executes slowly
* Implement `frameProcessorFps`
* Implement manual encoded video recordings
* Use recommended video settings
* Add fileType types
* Ignore if input is not ready for media data
* Add completion handler
* Add audio buffer sampling
* Init only for video frame
* use AVAssetWriterInputPixelBufferAdaptor
* Remove AVAssetWriterInputPixelBufferAdaptor
* Rotate VideoWriter
* Always assume portrait orientation
* Update RecordingSession.swift
* Use a separate Queue for Audio
* Format Swift
* Update CameraView+RecordVideo.swift
* Use `videoQueue` instead of `cameraQueue`
* Move example plugins to example app
* Fix hardcoded name in plugin macro
* QRFrame... -> QRCodeFrame...
* Update FrameProcessorPlugin.h
* Add example frame processors to JS base
* Update QRCodeFrameProcessorPluginSwift.m
* Add docs to create FP Plugins
* Update FRAME_PROCESSORS_CREATE.mdx
* Update FRAME_PROCESSORS_CREATE.mdx
* Use `AVAssetWriterInputPixelBufferAdaptor` for efficient pixel buffer recycling
* Add customizable `pixelFormat`
* Use native format if available
* Update project.pbxproj
* Set video width and height as source-pixel-buffer attributes
* Catch
* Update App.tsx
* Don't explicitly set video dimensions, let CVPixelBufferPool handle it
* Add a few logs
* Cleanup
* Update CameraView+RecordVideo.swift
* Eagerly initialize asset writer to fix stutter at first frame
* Use `cameraQueue` DispatchQueue to not block CaptureDataOutputDelegate
* Fix duration calculation
* cleanup
* Cleanup
* Swiftformat
* Return available video codecs
* Only show frame drop notification for video output
* Remove photo and video codec functionality
It was too much complexity and probably never used anyways.
* Revert all android related changes for now
* Cleanup
* Remove unused header
* Update AVAssetWriter.Status+descriptor.swift
* Only call Frame Processor for Video Frames
* Fix `if`
* Add support for Frame Processor plugin parameters/arguments
* Fix arg support
* Move to JSIUtils.mm
* Update JSIUtils.h
* Update FRAME_PROCESSORS_CREATE.mdx
* Update FRAME_PROCESSORS_CREATE.mdx
* Upgrade packages for docs/
* fix docs
* Rename
* highlight lines
* docs
* community plugins
* Update FRAME_PROCESSOR_CREATE_FINAL.mdx
* Update FRAME_PROCESSOR_PLUGIN_LIST.mdx
* Update FRAME_PROCESSOR_PLUGIN_LIST.mdx
* Update dependencies (1/2)
* Update dependencies (2/2)
* Update Gemfile.lock
* add FP docs
* Update README.md
* Make `lastFrameProcessor` private
* add `frameProcessor` docs
* fix docs
* adjust docs
* Update DEVICES.mdx
* fix
* s
* Add logs demo
* add metro restart note
* Update FRAME_PROCESSOR_CREATE_PLUGIN_IOS.mdx
* Mirror video device
* Update AVCaptureVideoDataOutput+mirror.swift
* Create .swift-version
* Enable whole module optimization
* Fix recording mirrored video
* Swift format
* Clean dictionary on `markInvalid`
* Fix cleanup
* Add docs for disabling frame processors
* Update project.pbxproj
* Revert "Update project.pbxproj"
This reverts commit e67861e51b88b4888a6940e2d20388f3044211d0.
* Log frame drop reason
* Format
* add more samples
* Add clang-format
* also check .mm
* Revert "also check .mm"
This reverts commit 8b9d5e2c29866b05909530d104f6633d6c49eadd.
* Revert "Add clang-format"
This reverts commit 7643ac808e0fc34567ea1f814e73d84955381636.
* Use `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange` as default
* Read matching video attributes from videoSettings
* Add TODO
* Swiftformat
* Conditionally disable frame processors
* Assert if trying to use frame processors when disabled
* Add frame-processors demo gif
* Allow disabling frame processors via `VISION_CAMERA_DISABLE_FRAME_PROCESSORS`
* Update FrameProcessorRuntimeManager.mm
* Update FRAME_PROCESSORS.mdx
* Update project.pbxproj
* Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
2021-05-06 14:11:55 +02:00
