Marc Rousavy
86dd703c2b
feat: Rewrite Android C++ part (VisionCameraProxy + JFrame) ( #1661 )
...
* First Android rewrite
* Rewrite Android C++ backend
* Pass `ReadableNativeMap`, fix build error
* fix: Fix FrameProcessor init
* Make a bunch of stuff const reference to avoid copies
* Indents
* Cleanup
* indents
* docs: Update Android docs
* Update CameraView.kt
* fix: Format C++ code
2023-07-22 00:15:11 +02:00
Marc Rousavy
375e894038
feat: Complete iOS Codebase rewrite ( #1647 )
...
* Make Frame Processors an extra subspec
* Update VisionCamera.podspec
* Make optional
* Make VisionCamera compile without Skia
* Fix
* Add skia again
* Update VisionCamera.podspec
* Make VisionCamera build without Frame Processors
* Rename error to `system/frame-processors-unavailable`
* Fix Frame Processor returning early
* Remove `preset`, FP partial rewrite
* Only warn on frame drop
* Fix wrong queue
* fix: Run on CameraQueue again
* Update CameraView.swift
* fix: Activate audio session asynchronously on audio queue
* Update CameraView+RecordVideo.swift
* Update PreviewView.h
* Cleanups
* Cleanup
* fix cast
* feat: Add LiDAR Depth Camera support
* Upgrade Ruby
* Add vector icons type
* Update Gemfile.lock
* fix: Stop queues on deinit
* Also load `builtInTrueDepthCamera`
* Update CameraViewManager.swift
* Update SkImageHelpers.mm
* Extract FrameProcessorCallback to FrameProcessor
Holds more context now :)
* Rename to .m
* fix: Add `RCTLog` import
* Create SkiaFrameProcessor
* Update CameraBridge.h
* Call Frame Processor
* Fix defines
* fix: Allow deleting callback funcs
* fix Skia build
* batch
* Just call `setSkiaFrameProcessor`
* Rewrite in Swift
* Pass `SkiaRenderer`
* Fix Import
* Move `PreviewView` to Swift
* Fix Layer
* Set Skia Canvas to Frame Host Object
* Make `DrawableFrameHostObject` subclass
* Fix TS types
* Use same MTLDevice and apply scale
* Make getter
* Extract `setTorch` and `Preview`
* fix: Fix nil metal device
* Don't wait for session stop in deinit
* Use main pixel ratio
* Use unique_ptr for Render Contexts
* fix: Fix SkiaPreviewDisplayLink broken after deinit
* inline `getTextureCache`
* Update CameraPage.tsx
* chore: Format iOS
* perf: Allow MTLLayer to be optimized for only frame buffers
* Add RN Video types
* fix: Fix Frame Processors if guard
* Find nodeModules recursively
* Create `Frame.isDrawable`
* Add `cocoapods-check` dependency
2023-07-20 15:30:04 +02:00
Marc Rousavy
12f850c8e1
feat: Draw onto Frame as if it was a Skia Canvas ( #1479 )
...
* Create Shaders.ts
* Add `previewType` and `enableFpsGraph`
* Add RN Skia native dependency
* Add Skia Preview View on iOS
* Pass 1
* Update FrameHostObject.mm
* Wrap Canvas
* Lockfiles
* fix: Fix stuff
* chore: Upgrade RNWorklets
* Add `previewType` to set the Preview
* feat: Add Example
* Update project.pbxproj
* `enableFpsGraph`
* Cache the `std::shared_ptr<FrameHostObject>`
* Update CameraView+RecordVideo.swift
* Update SkiaMetalCanvasProvider.mm
* Android: Integrate Skia Dependency
* fix: Use new Prefix
* Add example for rendering shader
* chore: Upgrade CameraX
* Remove KTX
* Enable `viewBinding`
* Revert "Enable `viewBinding`"
This reverts commit f2a603f53b33ea4311a296422ffd1a910ce03f9e.
* Revert "chore: Upgrade CameraX"
This reverts commit 8dc832cf8754490d31a6192e6c1a1f11cdcd94fe.
* Remove unneeded `ProcessCameraProvider.getInstance()` call
* fix: Add REA hotfix patch
* fix: Fix FrameHostObject dead in runAsync
* fix: Make `runAsync` run truly async by dropping new Frames while executing
* chore: Upgrade RN Worklets to latest
* chore: Upgrade RN Skia
* Revert "Remove KTX"
This reverts commit 253f586633f7af2da992d2279fc206dc62597129.
* Make Skia optional in CMake
* Fix import
* Update CMakeLists.txt
* Update build.gradle
* Update CameraView.kt
* Update CameraView.kt
* Update CameraView.kt
* Update Shaders.ts
* Center Blur
* chore: Upgrade RN Worklets
* feat: Add `toByteArray()`, `orientation`, `isMirrored` and `timestamp` to `Frame` (#1487 )
* feat: Implement `orientation` and `isMirrored` on Frame
* feat: Add `toArrayBuffer()` func
* perf: Do faster buffer copy
* feat: Implement `toArrayBuffer()` on Android
* feat: Add `orientation` and `isMirrored` to Android
* feat: Add `timestamp` to Frame
* Update Frame.ts
* Update JImageProxy.h
* Update FrameHostObject.cpp
* Update FrameHostObject.cpp
* Update CameraPage.tsx
* fix: Format Swift
2023-02-21 15:00:48 +01:00
Marc Rousavy
30b56153db
feat: Sync Frame Processors (plus runAsync and runAtTargetFps) ( #1472 )
...
Before, Frame Processors ran on a separate Thread.
After, Frame Processors run fully synchronous and always at the same FPS as the Camera.
Two new functions have been introduced:
* `runAtTargetFps(fps: number, func: () => void)`: Runs the given code as often as the given `fps`, effectively throttling it's calls.
* `runAsync(frame: Frame, func: () => void)`: Runs the given function on a separate Thread for Frame Processing. A strong reference to the Frame is held as long as the function takes to execute.
You can use `runAtTargetFps` to throttle calls to a specific API (e.g. if your Camera is running at 60 FPS, but you only want to run face detection at ~25 FPS, use `runAtTargetFps(25, ...)`.)
You can use `runAsync` to run a heavy algorithm asynchronous, so that the Camera is not blocked while your algorithm runs. This is useful if your main sync processor draws something, and your async processor is doing some image analysis on the side.
You can also combine both functions.
Examples:
```js
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
console.log("I'm running at 60 FPS!")
}, [])
```
```js
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
console.log("I'm running at 60 FPS!")
runAtTargetFps(10, () => {
'worklet'
console.log("I'm running at 10 FPS!")
})
}, [])
```
```js
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
console.log("I'm running at 60 FPS!")
runAsync(frame, () => {
'worklet'
console.log("I'm running on another Thread, I can block for longer!")
})
}, [])
```
```js
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
console.log("I'm running at 60 FPS!")
runAtTargetFps(10, () => {
'worklet'
runAsync(frame, () => {
'worklet'
console.log("I'm running on another Thread at 10 FPS, I can block for longer!")
})
})
}, [])
```
2023-02-15 16:47:09 +01:00
Marc Rousavy
48da1819fc
feat: Custom Orientation ( #715 )
...
* feat: Custom Orientation
* Update CameraView.swift
* Update CameraView.swift
* Try outputRotation approach
* whoops
* fix: Refactor `VideoCapture` instance
* Update orientation in didSetProps
* Update Orientation in iOS
* expose to objc
* Fix Orientation values
* format
2022-01-04 16:57:40 +01:00
Marc Rousavy
916278d3ea
feat: Add React Native 0.66 support ( #490 )
...
* feat: Add React Native 0.66 support
* Generate lockfiles
* Update Podfile.lock
* chore: Migrate from react-native-navigation to react-navigation (#491 )
* Migrate RNN -> RN
* Migrate all screens
* Fix get permission status
* fix app name
* Update AppDelegate.m
* Fix Info.plist
* Set `UIViewControllerBasedStatusBarAppearance` to `YES`
* Only enable `audio` if user granted microphone permission
* Update App.tsx
* Fix RNGH for Android
* Use `navigate` instead of `push`
* Fix animation
* Upgrade @types/react-native
* "Splash" -> "PermissionsPage"
2021-10-05 12:22:14 +02:00
Marc Rousavy
ad5e131f6a
feat: frameProcessorFps="auto" and automatic performance suggestions (throttle or increase FPS) ( #393 )
...
* Add `onFrameProcessorPerformanceSuggestionAvailable` and make `frameProcessorFps` support `auto`
* Implement performance suggestion and auto-adjusting
* Fix FPS setting, evaluate correctly
* Floor suggested FPS
* Remove `console.log` for frame drop warnings.
* Swift format
* Use `30` magic number
* only call if FPS is different
* Update CameraView.swift
* Implement Android 1/2
* Cleanup
* Update `frameProcessorFps` if available
* Optimize `FrameProcessorPerformanceDataCollector` initialization
* Cache call
* Set frameProcessorFps directly (Kotlin setter)
* Don't suggest if same value
* Call suggestion every second
* reset time on set
* Always store 15 last samples
* reset counter too
* Update FrameProcessorPerformanceDataCollector.swift
* Update CameraView+RecordVideo.swift
* Update CameraView.kt
* iOS: Redesign evaluation
* Update CameraView+RecordVideo.swift
* Android: Redesign evaluation
* Update CameraView.kt
* Update REA to latest alpha and install RNScreens
* Fix frameProcessorFps updating
2021-09-06 16:27:16 +02:00
Marc Rousavy
0f7ee51333
feat: Add console logging support for Frame Processors ( #297 )
...
* Try to log to console via runOnJS
* Call `console.log`
* Create custom `VisionCameraScheduler`
* Fix scheduler call
* Call with this
* Fix console setting
* Move J---- to `java-bindings`
* c++ style
* Android: 1/2 Create custom Scheduler
* Android: 2/2 Use custom Scheduler
* Don't use `runOnJS`, use `__callAsync` directly
2021-07-30 10:27:45 +02:00
Marc Rousavy
445af943c3
feat: BREAKING CHANGE: Express zoom factor always in actual factor value (1, 2, 128, ...) instead of 0.0-1.0 scale ( #306 )
...
* Make `zoom` go on "factor" scale
* Clean up `zoom` code
* fix float conversion
* fix zoom interpretation
* Update docs for new zoom scale
* fix float conversion
2021-07-29 11:44:22 +02:00
Marc Rousavy
87e6bb710e
feat: Frame Processors for Android ( #196 )
...
* Create android gradle build setup
* Fix `prefab` config
* Add `pickFirst **/*.so` to example build.gradle
* fix REA path
* cache gradle builds
* Update validate-android.yml
* Create Native Proxy
* Copy REA header
* implement ctor
* Rename CameraViewModule -> FrameProcessorRuntimeManager
* init FrameProcessorRuntimeManager
* fix name
* Update FrameProcessorRuntimeManager.h
* format
* Create AndroidErrorHandler.h
* Initialize runtime and install JSI funcs
* Update FrameProcessorRuntimeManager.cpp
* Update CameraViewModule.kt
* Make CameraView hybrid C++ class to find view & set frame processor
* Update FrameProcessorRuntimeManager.cpp
* pass function by rvalue
* pass by const &&
* extract hermes and JSC REA
* pass `FOR_HERMES`
* correctly prepare JSC and Hermes
* Update CMakeLists.txt
* add missing hermes include
* clean up imports
* Create JImageProxy.h
* pass ImageProxy to JNI as `jobject`
* try use `JImageProxy` C++ wrapper type
* Use `local_ref<JImageProxy>`
* Create `JImageProxyHostObject` for JSI interop
* debug call to frame processor
* Unset frame processor
* Fix CameraView native part not being registered
* close image
* use `jobject` instead of `JImageProxy` for now :(
* fix hermes build error
* Set enable FP callback
* fix JNI call
* Update CameraView.cpp
* Get Format
* Create plugin abstract
* Make `FrameProcessorPlugin` a hybrid object
* Register plugin CXX
* Call `registerPlugin`
* Catch
* remove JSI
* Create sample QR code plugin
* register plugins
* Fix missing JNI binding
* Add `mHybridData`
* prefix name with two underscores (`__`)
* Update CameraPage.tsx
* wrap `ImageProxy` in host object
* Use `jobject` for HO box
* Update JImageProxy.h
* reinterpret jobject
* Try using `JImageProxy` instead of `jobject`
* Update JImageProxy.h
* get bytes per row and plane count
* Update CameraView.cpp
* Return base
* add some docs and JNI JSI conversion
* indent
* Convert JSI value to JNI jobject
* using namespace facebook
* Try using class
* Use plain old Object[]
* Try convert JNI -> JSI
* fix decl
* fix bool init
* Correctly link folly
* Update CMakeLists.txt
* Convert Map to Object
* Use folly for Map and Array
* Return `alias_ref<jobject>` instead of raw `jobject`
* fix JNI <-> JSI conversion
* Update JSIJNIConversion.cpp
* Log parameters
* fix params index offset
* add more test cases
* Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
* fix types
* Rename to example plugin
* remove support for hashmap
* Try use HashMap iterable fbjni binding
* try using JReadableArray/JReadableMap
* Fix list return values
* Update JSIJNIConversion.cpp
* Update JSIJNIConversion.cpp
* (iOS) Rename ObjC QR Code Plugin to Example Plugin
* Rename Swift plugin QR -> Example
* Update ExamplePluginSwift.swift
* Fix Map/Dictionary logging format
* Update ExampleFrameProcessorPlugin.m
* Reconfigure session if frame processor changed
* Handle use-cases via `maxUseCasesCount`
* Don't crash app on `configureSession` error
* Document "use-cases"
* Update DEVICES.mdx
* fix merge
* Make `const &`
* iOS: Automatically enable `video` if a `frameProcessor` is set
* Update CameraView.cpp
* fix docs
* Automatically fallback to snapshot capture if `supportsParallelVideoProcessing` is false.
* Fix lookup
* Update CameraView.kt
* Implement `frameProcessorFps`
* Finalize Frame Processor Plugin Hybrid
* Update CameraViewModule.kt
* Support `flash` on `takeSnapshot()`
* Update docs
* Add docs
* Update CameraPage.tsx
* Attribute NonNull
* remove unused imports
* Add Android docs for Frame Processors
* Make JNI HashMap <-> JSI Object conversion faster
directly access `toHashMap` instead of going through java
* add todo
* Always run `prepareJSC` and `prepareHermes`
* switch jsc and hermes
* Specify ndkVersion `21.4.7075529`
* Update gradle.properties
* Update gradle.properties
* Create .aar
* Correctly prepare android package
* Update package.json
* Update package.json
* remove `prefab` build feature
* split
* Add docs for registering the FP plugin
* Add step for dep
* Update CaptureButton.tsx
* Move to `reanimated-headers/`
* Exclude reanimated-headers from cpplint
* disable `build/include_order` rule
* cpplint fixes
* perf: Make `JSIJNIConversion` a `namespace` instead of `class`
* Ignore runtime/references for `convert` funcs
* Build Android .aar in CI
* Run android build script only on `prepack`
* Update package.json
* Update package.json
* Update build-android-npm-package.sh
* Move to `yarn build`
* Also install node_modules in example step
* Update validate-android.yml
* sort imports
* fix torch
* Run ImageAnalysis on `FrameProcessorThread`
* Update Errors.kt
* Add clean android script
* Upgrade reanimated to 2.3.0-alpha.1
* Revert "Upgrade reanimated to 2.3.0-alpha.1"
This reverts commit c1d3bed5e03728d0b5e335a359524ff4f56f5035.
* ⚠️ TEMP FIX: hotfix reanimated build.gradle
* Update CameraView+TakeSnapshot.kt
* ⚠️ TEMP FIX: Disable ktlint action for now
* Update clean.sh
* Set max heap size to 4g
* rebuild lockfiles
* Update Podfile.lock
* rename
* Build lib .aar before example/
2021-06-27 12:37:54 +02:00
Marc Rousavy
9c579c65aa
try: Improvements from WWDC 2021 1:1 workshop ( #197 )
...
* perf: Automatically determine Pixel Format depending on active format. (More efficient video recording 🚀 )
* perf: Skip `AVAssetWriter` transform by directly correctly orienting the Video Output connection
* feat: Support camera flipping while recording
* feat: Run frame processor on separate queue, avoids stutters in video recordigns
* feat: Automatically drop late frame processor frames
2021-06-11 21:06:19 +02:00
Marc Rousavy
72a1fad78e
feat: Separate usecases (decouple microphone, video, photo) ( #168 )
...
* Add props
* add props (iOS)
* Add use-cases conditionally
* Update CameraView+RecordVideo.swift
* Update RecordingSession.swift
* reconfigure on change
* Throw correct errors
* Check for audio permission
* Move `#if` outward
* Throw appropriate errors
* Update CameraView+RecordVideo.swift
* fix Splashscreen
* Dynamic filePath
* Fix video extension
* add `avci` and `m4v` file types
* Fix RecordVideo errors
* Fix audio setup
* Enable `photo`, `video` and `audio`
* Check for `video={true}` in frameProcessor
* format
* Remove unused DispatchQueue
* Update docs
* Add `supportsPhotoAndVideoCapture`
* Fix view manager
* Fix error not being propagated
* Catch normal errors too
* Update DEVICES.mdx
* Update CAPTURING.mdx
* Update classdocs
2021-06-07 13:08:40 +02:00
Marc Rousavy
555474be7d
fix: Represent neutralZoom in factor instead of percentage ( #179 )
...
* Use factor instead of percent for `neutralZoom`
* fix zoom calculation
* Update CameraPage.tsx
2021-06-07 10:46:53 +02:00
Marc Rousavy
310ad5fc4c
chore: Simplify format sorting/filtering ( #140 )
...
* Simplify format sorting/filtering
* Update useCameraFormat.ts
* Also check photo HDR
* Simplify double tap
* Remove snapshot
* Remove custom `useCameraDevice` hook
* Update Podfile.lock
2021-05-14 11:52:28 +02:00
Marc Rousavy
f839bc23ac
chore: Cleanup codebase ( #137 )
...
* Remove `useCachedState`
* Add pressable opacity
* Update Media.tsx
* f
* Update FormatFilter.ts
* update
* App -> CameraPage, Media -> MediaPage
* Update CameraPage.tsx
* Create 60 FPS switch
* Update CameraPage.tsx
2021-05-11 12:59:05 +02:00
