Commit Graph

123 Commits

Author SHA1 Message Date
Marc Rousavy
622d3830f1
feat: Make Frame Processor Plugins object-oriented on iOS as well (#1496)
* feat: Make Frame Processor Plugins object-oriented on iOS as well

* Add Plugin in AppDelegate
2023-02-27 11:18:03 +01:00
Marc Rousavy
0c3cd66016
fix: Improve C++ safety by attaching Cache Invalidator to jsi::Runtime's lifecycle (#1488)
* fix: fix C++ lint

* fix: attach `InvalidateCacheOnDestroy` to `jsi::Runtime`
2023-02-21 15:44:43 +01:00
Marc Rousavy
12f850c8e1
feat: Draw onto Frame as if it was a Skia Canvas (#1479)
* Create Shaders.ts

* Add `previewType` and `enableFpsGraph`

* Add RN Skia native dependency

* Add Skia Preview View on iOS

* Pass 1

* Update FrameHostObject.mm

* Wrap Canvas

* Lockfiles

* fix: Fix stuff

* chore: Upgrade RNWorklets

* Add `previewType` to set the Preview

* feat: Add Example

* Update project.pbxproj

* `enableFpsGraph`

* Cache the `std::shared_ptr<FrameHostObject>`

* Update CameraView+RecordVideo.swift

* Update SkiaMetalCanvasProvider.mm

* Android: Integrate Skia Dependency

* fix: Use new Prefix

* Add example for rendering shader

* chore: Upgrade CameraX

* Remove KTX

* Enable `viewBinding`

* Revert "Enable `viewBinding`"

This reverts commit f2a603f53b33ea4311a296422ffd1a910ce03f9e.

* Revert "chore: Upgrade CameraX"

This reverts commit 8dc832cf8754490d31a6192e6c1a1f11cdcd94fe.

* Remove unneeded `ProcessCameraProvider.getInstance()` call

* fix: Add REA hotfix patch

* fix: Fix FrameHostObject dead in runAsync

* fix: Make `runAsync` run truly async by dropping new Frames while executing

* chore: Upgrade RN Worklets to latest

* chore: Upgrade RN Skia

* Revert "Remove KTX"

This reverts commit 253f586633f7af2da992d2279fc206dc62597129.

* Make Skia optional in CMake

* Fix import

* Update CMakeLists.txt

* Update build.gradle

* Update CameraView.kt

* Update CameraView.kt

* Update CameraView.kt

* Update Shaders.ts

* Center Blur

* chore: Upgrade RN Worklets

* feat: Add `toByteArray()`, `orientation`, `isMirrored` and `timestamp` to `Frame` (#1487)

* feat: Implement `orientation` and `isMirrored` on Frame

* feat: Add `toArrayBuffer()` func

* perf: Do faster buffer copy

* feat: Implement `toArrayBuffer()` on Android

* feat: Add `orientation` and `isMirrored` to Android

* feat: Add `timestamp` to Frame

* Update Frame.ts

* Update JImageProxy.h

* Update FrameHostObject.cpp

* Update FrameHostObject.cpp

* Update CameraPage.tsx

* fix: Format Swift
2023-02-21 15:00:48 +01:00
Marc Rousavy
222ded4728 fix: Prevent possible crash in log error handler 2023-02-15 18:15:13 +01:00
Marc Rousavy
f0ea18115e
fix: Fix CI for V3 (#1475)
* fix: Fix CI for "Build Android"

* update versions

* Update Gemfile.lock

* format swift

* fix: Fix swift lint

* Update .swiftlint.yml

* Use C++17 for lint

* fix: Fix C++ lints
2023-02-15 17:24:33 +01:00
Marc Rousavy
30b56153db
feat: Sync Frame Processors (plus runAsync and runAtTargetFps) (#1472)
Before, Frame Processors ran on a separate Thread.

After, Frame Processors run fully synchronous and always at the same FPS as the Camera.

Two new functions have been introduced:

* `runAtTargetFps(fps: number, func: () => void)`: Runs the given code as often as the given `fps`, effectively throttling it's calls.
* `runAsync(frame: Frame, func: () => void)`: Runs the given function on a separate Thread for Frame Processing. A strong reference to the Frame is held as long as the function takes to execute.

You can use `runAtTargetFps` to throttle calls to a specific API (e.g. if your Camera is running at 60 FPS, but you only want to run face detection at ~25 FPS, use `runAtTargetFps(25, ...)`.)

You can use `runAsync` to run a heavy algorithm asynchronous, so that the Camera is not blocked while your algorithm runs. This is useful if your main sync processor draws something, and your async processor is doing some image analysis on the side. 

You can also combine both functions.

Examples:

```js
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")
}, [])
```

```js
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")

  runAtTargetFps(10, () => {
    'worklet'
    console.log("I'm running at 10 FPS!")
  })
}, [])
```



```js
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")

  runAsync(frame, () => {
    'worklet'
    console.log("I'm running on another Thread, I can block for longer!")
  })
}, [])
```

```js
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")

  runAtTargetFps(10, () => {
    'worklet'
    runAsync(frame, () => {
      'worklet'
      console.log("I'm running on another Thread at 10 FPS, I can block for longer!")
    })
  })
}, [])
```
2023-02-15 16:47:09 +01:00
Marc Rousavy
a0590dccb5
feat: Replace Reanimated with RN Worklets (#1468)
* Setup RN Worklets

* Use RN Worklets on iOS

* Fix console

* Add `installFrameProcessorBindings()` function

* Add `FrameProcessorPlugins` proxy (BREAKING CHANGE)

* Clean up docs

* Update FRAME_PROCESSORS.mdx

* Use RN Worklets 0.2.5

* feat: Android build setup

* Rewrite Android Frame Processor Part

* Update CMakeLists.txt

* fix: Add react-native-worklets Gradle dependency

* Update Podfile.lock

* fix build

* gradle:7.4.1

* Init JSI Bindings in method on Android

* Fix Folly flags

* fix: Init `FrameProcessorRuntimeManager` later

* fix: Wrap in `<GestureHandlerRootView>`

* Refactor plugins

* fix: Remove enableFrameProcessors

* Install RN Worklets from current GH master

* Update babel.config.js

* Update CameraViewModule.kt

* Update ImageProxyUtils.java

* feat: Upgrade to Reanimated v3

* fix: Fix crash on Worklet init

* Update RN Worklets to latest master

* fix: Simplify FP Plugins Proxy
2023-02-13 15:22:45 +01:00
Marc Rousavy
938cb2e42a Update CameraView+Orientation.swift 2022-11-25 15:40:07 +01:00
Marc Rousavy
bf71901968 fix: Fix orientation glitching on first frame 2022-11-25 15:39:40 +01:00
LazyAfternoons
3416e94b18
fix: Use +load for registering Frame Processors (#1308)
Fixes crash with Xcode 14
2022-10-28 13:12:44 +02:00
Thomas Coldwell
52a1d50d91
fix: Frame Processor FPS (#1288)
* fix: Build using XCode 14

* fix: Throttle FP by start time rather than end time
2022-10-20 12:49:22 +02:00
Marc Rousavy
4781ad9835
feat: Continue to record audio when receiving a phone call (try to prevent interruptions) (#1278)
* feat: Continue to record audio when receiving a phone call (try to prevent interruptions)

Uses `setPrefersNoInterruptionsFromSystemAlerts` to prevent system alerts (like phone calls) from interrupting the audio session which records audio from the microphone.

* fix: Add `try`
2022-10-14 12:30:22 +02:00
Rupesh Chaudhari
724af31807
fix: Use facebook::jsi instead of jsi (#1109)
This PR fixes the Build error in React Native 0.69 for iOS
2022-07-05 10:51:24 +02:00
Thomas Coldwell
54703f386c
fix: Set initial timestamp on first frame (#1114) 2022-07-04 14:09:39 +02:00
Marc Rousavy
04a8794246
fix: Fix outputOrientation Main Thread API checker (#1094) 2022-06-14 11:00:28 +02:00
Thomas Coldwell
fb2156ec39
fix: Asset Writer Video-Audio Sync (#1075)
* fix: Start asset writer session on first frame

* fix: Remove debug log

* fix: Reset dev team
2022-06-11 11:15:24 +02:00
Hirbod
83943da112
fix: AVWriter status 1 crash (#996)
* Fix AVWriter status 1 crash followup

* Update RecordingSession.swift

removing trailing semicolon to make linter happy

* Update RecordingSession.swift

Set hasRunningWritingAttempt to true

* Update RecordingSession.swift

* Update RecordingSession.swift

* Update RecordingSession.swift

* Update RecordingSession.swift

Make the linter hate me again

* Update RecordingSession.swift
2022-05-23 14:27:25 +02:00
Marc Rousavy
971b824914
fix: Fix RecordingSession nil crash by keeping it local (#938)
* fix: Fix RecordingSession nil crash by keeping it local

* fix: Fix error init

* Update CameraView+RecordVideo.swift
2022-04-15 09:48:32 +02:00
Hirbod
2ccce3587d
fix: AVAssetWriter status is 1 crash (#995)
fixes #930

markAsFinished has to be called within the finishWritingWithCompletionHandler.
Moving that 2 lines allowed me to spam start / stop for hundreds of time without a single crash.
2022-04-15 09:45:31 +02:00
Marc Rousavy
4b9bcb37e0
feat: Add pauseRecording and resumeRecording 🔥 (#911)
* feat: Add `pauseRecording` and `resumeRecording` (iOS)

* feat: Add `pauseRecording` and `resumeRecording` (Android)

* feat: Add `pauseRecording` and `resumeRecording` (JS)

* fix: Simplify Swift code for Recording
2022-03-22 10:44:58 +01:00
Marc Rousavy
d9932f4b7a
fix: Prevent NaN/+Inf crash for auto frameProcessorFps 2022-02-09 18:05:32 +01:00
Marc Rousavy
0904767cf2
fix: Log Stacktrace on Frame Processor Error (#731)
* fix: Log JS Stack on Error

* Android

* Format Stacktrace better

* Update FrameProcessorUtils.mm

* Allow unapproved C++11 headers

* Use `.c_str()`
2022-01-10 16:37:47 +01:00
Marc Rousavy
48da1819fc
feat: Custom Orientation (#715)
* feat: Custom Orientation

* Update CameraView.swift

* Update CameraView.swift

* Try outputRotation approach

* whoops

* fix: Refactor `VideoCapture` instance

* Update orientation in didSetProps

* Update Orientation in iOS

* expose to objc

* Fix Orientation values

* format
2022-01-04 16:57:40 +01:00
Marc Rousavy
be5ec69b02
feat: Make Reanimated optional (disable Frame Processors if REA v2 is not installed) (#412)
* Fix building iOS without Reanimated

* Conditionally compile Frame Processors (gradle)

* Conditionally use externalNativeBuild

* Remove Reanimated import

* fix: Conditionally load REA/VisionCamera libraries

* fix: Add disable FP to docs

* fix: Fix dummy placeholder for Scheduler.mm

* fix: Fix dummy `Scheduler` declaration

* fix: Only init `CameraView` C++ side if frame processors are enabled

* fix: Install JSI Bindings on Frame Processor Manager ctor

* fix: Wrong conditional

* whoops
2022-01-02 17:35:26 +01:00
Marc Rousavy
e8a8e71a76
fix: Fix Frame.bytesPerRow showing wrong values on iOS (#688) 2021-12-31 16:59:05 +01:00
Marc Rousavy
b72176fae9
fix: Re-install FP plugins on bundle reload (#684) 2021-12-30 11:38:54 +01:00
Paul Rostorp
d96c5863c9
feat: Video Codec Option for recording video (#645)
* add video codec value

* add types

* use `recommendedVideoSettings` method instead

* lint

* refactor for better readability

* add a method to get available codecs (ios)

* imrove tsDoc description of the videoCodec option

Co-authored-by: Marc Rousavy <marcrousavy@hotmail.com>

* ios format

Co-authored-by: Marc Rousavy <marcrousavy@hotmail.com>
2021-12-30 10:47:23 +01:00
Marc Rousavy
a3cfcc2908 Remove conditional downcast 2021-12-30 10:34:46 +01:00
Marc Rousavy
6319e5cc49
fix: Fix torch staying on after recording a video (#584) 2021-12-10 09:57:05 +01:00
Marc Rousavy
934de142ee docs: Update Format.pixelFormat documentation 2021-12-10 09:52:40 +01:00
Marc Rousavy
9301430165 chore: Lint 2021-12-10 09:44:54 +01:00
Thomas Coldwell
c3039c94c6
feat: Add pixelFormat field (#559)
* Add `mediaSubType` field to device formats

* Create FourCharCode `toString` extension

* Move pixelFormat -> `AVCaptureDevice.Format+toDictionary`

* `mediaSubType` -> `pixelFormat` + non-optional

* ts pixelFormat `string` -> specific formats

* Add default pixelFormat value of `420v` on Android

Co-authored-by: tcoldwell72 <31568400+tcoldwell72@users.noreply.github.com>
2021-12-10 09:44:30 +01:00
Marc Rousavy
4a73cb96c1
fix: Fix view-not-found race condition in C++ code (#511)
* Add custom `onViewReady` event to get layout

`componentDidMount` is async, so the native view _might_ not exist yet causing a race condition in the `setFrameProcessor` code.

This PR fixes this by calling `setFrameProcessor` only after the native view has actually mounted, and to ensure that I created a custom event that fires at that point.

* Update CameraView.swift
2021-10-11 18:27:23 +02:00
Marc Rousavy
5dc8ded62a
Update JSIUtils.mm 2021-09-29 12:54:51 +02:00
Marc Rousavy
de0549de44
Add facebook copyright notice 2021-09-29 12:52:45 +02:00
Marc Rousavy
f9dbb6921c
fix: Fix divison by zero in Performance Sample collector (#416)
* fix: Fix divison by zero in Performance Sample collector

* use `isEmpty`
2021-09-08 17:18:12 +02:00
Marc Rousavy
ad5e131f6a
feat: frameProcessorFps="auto" and automatic performance suggestions (throttle or increase FPS) (#393)
* Add `onFrameProcessorPerformanceSuggestionAvailable` and make `frameProcessorFps` support `auto`

* Implement performance suggestion and auto-adjusting

* Fix FPS setting, evaluate correctly

* Floor suggested FPS

* Remove `console.log` for frame drop warnings.

* Swift format

* Use `30` magic number

* only call if FPS is different

* Update CameraView.swift

* Implement Android 1/2

* Cleanup

* Update `frameProcessorFps` if available

* Optimize `FrameProcessorPerformanceDataCollector` initialization

* Cache call

* Set frameProcessorFps directly (Kotlin setter)

* Don't suggest if same value

* Call suggestion every second

* reset time on set

* Always store 15 last samples

* reset counter too

* Update FrameProcessorPerformanceDataCollector.swift

* Update CameraView+RecordVideo.swift

* Update CameraView.kt

* iOS: Redesign evaluation

* Update CameraView+RecordVideo.swift

* Android: Redesign evaluation

* Update CameraView.kt

* Update REA to latest alpha and install RNScreens

* Fix frameProcessorFps updating
2021-09-06 16:27:16 +02:00
Marc Rousavy
3c845ed4b0
fix: Fix app hard-crashing when FPS value is not supported (#391) 2021-08-28 14:14:16 +02:00
Marc Rousavy
90f2a1ef7d
fix: Fix close being called on a collected reference (enforce alias_ref) (#379)
* fix: Fix `close` being called on a collected reference (enforce `alias_ref`)

* Remove `~FrameHostObject` on iOS too
2021-08-20 16:05:02 +02:00
Marc Rousavy
c5979688c8 chore: Lint Swift 2021-08-03 12:01:39 +02:00
Marc Rousavy
c078cdf933 feat: Correctly get videoDimensions on devices below iOS 13 2021-08-03 10:37:48 +02:00
Marc Rousavy
0f7ee51333
feat: Add console logging support for Frame Processors (#297)
* Try to log to console via runOnJS

* Call `console.log`

* Create custom `VisionCameraScheduler`

* Fix scheduler call

* Call with this

* Fix console setting

* Move J---- to `java-bindings`

* c++ style

* Android: 1/2 Create custom Scheduler

* Android: 2/2 Use custom Scheduler

* Don't use `runOnJS`, use `__callAsync` directly
2021-07-30 10:27:45 +02:00
Marc Rousavy
445af943c3
feat: BREAKING CHANGE: Express zoom factor always in actual factor value (1, 2, 128, ...) instead of 0.0-1.0 scale (#306)
* Make `zoom` go on "factor" scale

* Clean up `zoom` code

* fix float conversion

* fix zoom interpretation

* Update docs for new zoom scale

* fix float conversion
2021-07-29 11:44:22 +02:00
Marc Rousavy
ef455df865
feat: Support rotation (#301)
* feat: Android: Listen to rotation changes

* Only change rotation on configuration change

* feat: iOS: Support Rotation

* Swift lint
2021-07-26 11:32:58 +02:00
Marc Rousavy
d9233208a8 fix: Remove RCTTurboModuleBlockCopyEnabled() 2021-07-26 10:12:26 +02:00
Marc Rousavy
4b4ea0ff33
fix: Fix UI Thread race condition in setFrameProcessor(...) (#265)
* fix: Fix UI Thread race condition in `setFrameProcessor(...)`

* Revert "fix: Fix UI Thread race condition in `setFrameProcessor(...)`"

This reverts commit 9c524e123cff6843d7d11db602a5027d1bb06b4b.

* Use `setImmediate` to call `setFrameProcessor(...)`

* Fix frame processor order of applying

* Add `enableFrameProcessor` prop that defines if a FP is added

* rename constant

* Implement `enableFrameProcessor` prop for Android and make `frameProcessorFps` faster

* link to troubleshooting guide

* Update TROUBLESHOOTING.mdx

* Add logs for use-cases

* fix log

* set initial frame processor in `onLayout` instead of `componentDidMount`
2021-07-12 15:16:03 +02:00
Marc Rousavy
ba08b4dfce chore: Format Swift 2021-07-08 11:01:02 +02:00
Marc Rousavy
5126dd63b4 feat: Make videoWidth and videoHeight available on all devices 2021-07-08 10:59:27 +02:00
Marc Rousavy
2fa0f8fd46
chore: Clean up root directory (#236)
* Clean up root

* remove unused REA patches

* Remove `cpp/` and create headers per platform

* fix `#import`

* fix `VISION_CAMERA_DISABLE_FRAME_PROCESSORS`
2021-07-06 16:42:58 +02:00
Marc Rousavy
fa5f5c0cab
feat: Implement Frame.close() (#229)
* Implement `Frame.close()`

* close frame in dtor

* Update JImageProxyHostObject.cpp

* fix close

* Check if closed

* remove a few logs

* r

* fix `isValid` and `isReady`

* Add JImage

* Release JNI frame ref on destroy

* fix pod setup

* Fix isValid call

* Fix `close` not returning a function

* throw error if closed twice

* iOS: Schedule `console.error` call on JS thread

* Android: Log Frame Processor Error to JS

* fix syntax

* Check if valid `toString()`

* Update Frame.ts

* Remove `isReady`

* Fix JImage accessors

* remove `JImage` C++ sources

* Throw error if accessing props on closed Frame

* Delete `JImage.h`
2021-07-06 10:08:44 +02:00