Commit Graph

272 Commits

Author SHA1 Message Date
Marc Rousavy
37a3548a81
feat: Full Android rewrite (CameraX -> Camera2) (#1674)
* Nuke CameraX

* fix: Run View Finder on UI Thread

* Open Camera, set up Threads

* fix init

* Mirror if needed

* Try PreviewView

* Use max resolution

* Add `hardwareLevel` property

* Check if output type is supported

* Replace `frameRateRanges` with `minFps` and `maxFps`

* Remove `isHighestPhotoQualitySupported`

* Remove `colorSpace`

The native platforms will use the best / most accurate colorSpace by default anyways.

* HDR

* Check from format

* fix

* Remove `supportsParallelVideoProcessing`

* Correctly return video/photo sizes on Android now. Finally

* Log all Device props

* Log if optimized usecase is used

* Cleanup

* Configure Camera Input only once

* Revert "Configure Camera Input only once"

This reverts commit 0fd6c03f54c7566cb5592053720c4a8743aba92e.

* Extract Camera configuration

* Try to reconfigure all

* Hook based

* Properly set up `CameraSession`

* Delete unused

* fix: Fix recreate when outputs change

* Update NativePreviewView.kt

* Use callback for closing

* Catch CameraAccessException

* Finally got it stable

* Remove isMirrored

* Implement `takePhoto()`

* Add ExifInterface library

* Run findViewById on UI Thread

* Add Photo Output Surface to takePhoto

* Fix Video Stabilization Modes

* Optimize Imports

* More logs

* Update CameraSession.kt

* Close Image

* Use separate Executor in CameraQueue

* Delete hooks

* Use same Thread again

* If opened, call error

* Update CameraSession.kt

* Log HW level

* fix: Don't enable Stream Use Case if it's not 100% supported

* Move some stuff

* Cleanup PhotoOutputSynchronizer

* Try just open in suspend fun

* Some synchronization fixes

* fix logs

* Update CameraDevice+createCaptureSession.kt

* Update CameraDevice+createCaptureSession.kt

* fixes

* fix: Use Snapshot Template for speed capture prio

* Use PREVIEW template for repeating request

* Use `TEMPLATE_RECORD` if video use-case is attached

* Use `isRunning` flag

* Recreate session everytime on active/inactive

* Lazily get values in capture session

* Stability

* Rebuild session if outputs change

* Set `didOutputsChange` back to false

* Capture first in lock

* Try

* kinda fix it? idk

* fix: Keep Outputs

* Refactor into single method

* Update CameraView.kt

* Use Enums for type safety

* Implement Orientation (I think)

* Move RefCount management to Java (Frame)

* Don't crash when dropping a Frame

* Prefer Devices with higher max resolution

* Prefer multi-cams

* Use FastImage for Media Page

* Return orientation in takePhoto()

* Load orientation from EXIF Data

* Add `isMirrored` props and documentation for PhotoFile

* fix: Return `not-determined` on Android

* Update CameraViewModule.kt

* chore: Upgrade packages

* fix: Fix Metro Config

* Cleanup config

* Properly mirror Images on save

* Prepare MediaRecorder

* Start/Stop MediaRecorder

* Remove `takeSnapshot()`

It no longer works on Android and never worked on iOS. Users could use useFrameProcessor to take a Snapshot

* Use `MediaCodec`

* Move to `VideoRecording` class

* Cleanup Snapshot

* Create `SkiaPreviewView` hybrid class

* Create OpenGL context

* Create `SkiaPreviewView`

* Fix texture creation missing context

* Draw red frame

* Somehow get it working

* Add Skia CMake setup

* Start looping

* Init OpenGL

* Refactor into `SkiaRenderer`

* Cleanup PreviewSize

* Set up

* Only re-render UI if there is a new Frame

* Preview

* Fix init

* Try rendering Preview

* Update SkiaPreviewView.kt

* Log version

* Try using Skia (fail)

* Drawwwww!!!!!!!!!! 🎉

* Use Preview Size

* Clear first

* Refactor into SkiaRenderer

* Add `previewType: "none"` on iOS

* Simplify a lot

* Draw Camera? For some reason? I have no idea anymore

* Fix OpenGL errors

* Got it kinda working again?

* Actually draw Frame woah

* Clean up code

* Cleanup

* Update on main

* Synchronize render calls

* holy shit

* Update SkiaRenderer.cpp

* Update SkiaRenderer.cpp

* Refactor

* Update SkiaRenderer.cpp

* Check for `NO_INPUT_TEXTURE`^

* Post & Wait

* Set input size

* Add Video back again

* Allow session without preview

* Convert JPEG to byte[]

* feat: Use `ImageReader` and use YUV Image Buffers in Skia Context (#1689)

* Try to pass YUV Buffers as Pixmaps

* Create pixmap!

* Clean up

* Render to preview

* Only render if we have an output surface

* Update SkiaRenderer.cpp

* Fix Y+U+V sampling code

* Cleanup

* Fix Semaphore 0

* Use 4:2:0 YUV again idk

* Update SkiaRenderer.h

* Set minSdk to 26

* Set surface

* Revert "Set minSdk to 26"

This reverts commit c4085b7c16c628532e5c2d68cf7ed11c751d0b48.

* Set previewType

* feat: Video Recording with Camera2 (#1691)

* Rename

* Update CameraSession.kt

* Use `SurfaceHolder` instead of `SurfaceView` for output

* Update CameraOutputs.kt

* Update CameraSession.kt

* fix: Fix crash when Preview is null

* Check if snapshot capture is supported

* Update RecordingSession.kt

* S

* Use `MediaRecorder`

* Make audio optional

* Add Torch

* Output duration

* Update RecordingSession.kt

* Start RecordingSession

* logs

* More log

* Base for preparing pass-through Recording

* Use `ImageWriter` to append Images to the Recording Surface

* Stream PRIVATE GPU_SAMPLED_IMAGE Images

* Add flags

* Close session on stop

* Allow customizing `videoCodec` and `fileType`

* Enable Torch

* Fix Torch Mode

* Fix comparing outputs with hashCode

* Update CameraSession.kt

* Correctly pass along Frame Processor

* fix: Use AUDIO_BIT_RATE of 16 * 44,1Khz

* Use CAMCORDER instead of MIC microphone

* Use 1 channel

* fix: Use `Orientation`

* Add `native` PixelFormat

* Update iOS to latest Skia integration

* feat: Add `pixelFormat` property to Camera

* Catch error in configureSession

* Fix JPEG format

* Clean up best match finder

* Update CameraDeviceDetails.kt

* Clamp sizes by maximum CamcorderProfile size

* Remove `getAvailableVideoCodecs`

* chore: release 3.0.0-rc.5

* Use maximum video size of RECORD as default

* Update CameraDeviceDetails.kt

* Add a todo

* Add JSON device to issue report

* Prefer `full` devices and flash

* Lock to 30 FPS on Samsung

* Implement Zoom

* Refactor

* Format -> PixelFormat

* fix: Feat `pixelFormat` -> `pixelFormats`

* Update TROUBLESHOOTING.mdx

* Format

* fix: Implement `zoom` for Photo Capture

* fix: Don't run if `isActive` is `false`

* fix: Call `examplePlugin(frame)`

* fix: Fix Flash

* fix: Use `react-native-worklets-core`!

* fix: Fix import
2023-08-21 12:50:14 +02:00
Marc Rousavy
61fd4e0474
Merge branch 'main' into v3 2023-07-31 18:27:11 +02:00
Marc Rousavy
86dd703c2b
feat: Rewrite Android C++ part (VisionCameraProxy + JFrame) (#1661)
* First Android rewrite

* Rewrite Android C++ backend

* Pass `ReadableNativeMap`, fix build error

* fix: Fix FrameProcessor init

* Make a bunch of stuff const reference to avoid copies

* Indents

* Cleanup

* indents

* docs: Update Android docs

* Update CameraView.kt

* fix: Format C++ code
2023-07-22 00:15:11 +02:00
Marc Rousavy
44ed42d5d6
feat: Expose unified VisionCameraProxy object, make FrameProcessorPlugins object-oriented (#1660)
* feat: Replace `FrameProcessorRuntimeManager` with `VisionCameraProxy` (iOS)

* Make `FrameProcessorPlugin` a constructable HostObject

* fix: Fix `name` override

* Simplify `useFrameProcessor

* fix: Fix lint errors

* Remove FrameProcessorPlugin::name

* JSIUtils -> JSINSObjectConversion
2023-07-21 17:52:30 +02:00
Marc Rousavy
375e894038
feat: Complete iOS Codebase rewrite (#1647)
* Make Frame Processors an extra subspec

* Update VisionCamera.podspec

* Make optional

* Make VisionCamera compile without Skia

* Fix

* Add skia again

* Update VisionCamera.podspec

* Make VisionCamera build without Frame Processors

* Rename error to `system/frame-processors-unavailable`

* Fix Frame Processor returning early

* Remove `preset`, FP partial rewrite

* Only warn on frame drop

* Fix wrong queue

* fix: Run on CameraQueue again

* Update CameraView.swift

* fix: Activate audio session asynchronously on audio queue

* Update CameraView+RecordVideo.swift

* Update PreviewView.h

* Cleanups

* Cleanup

* fix cast

* feat: Add LiDAR Depth Camera support

* Upgrade Ruby

* Add vector icons type

* Update Gemfile.lock

* fix: Stop queues on deinit

* Also load `builtInTrueDepthCamera`

* Update CameraViewManager.swift

* Update SkImageHelpers.mm

* Extract FrameProcessorCallback to FrameProcessor

Holds more context now :)

* Rename to .m

* fix: Add `RCTLog` import

* Create SkiaFrameProcessor

* Update CameraBridge.h

* Call Frame Processor

* Fix defines

* fix: Allow deleting callback funcs

* fix Skia build

* batch

* Just call `setSkiaFrameProcessor`

* Rewrite in Swift

* Pass `SkiaRenderer`

* Fix Import

* Move `PreviewView` to Swift

* Fix Layer

* Set Skia Canvas to Frame Host Object

* Make `DrawableFrameHostObject` subclass

* Fix TS types

* Use same MTLDevice and apply scale

* Make getter

* Extract `setTorch` and `Preview`

* fix: Fix nil metal device

* Don't wait for session stop in deinit

* Use main pixel ratio

* Use unique_ptr for Render Contexts

* fix: Fix SkiaPreviewDisplayLink broken after deinit

* inline `getTextureCache`

* Update CameraPage.tsx

* chore: Format iOS

* perf: Allow MTLLayer to be optimized for only frame buffers

* Add RN Video types

* fix: Fix Frame Processors if guard

* Find nodeModules recursively

* Create `Frame.isDrawable`

* Add `cocoapods-check` dependency
2023-07-20 15:30:04 +02:00
hmaltr
3ed019d018
docs: Fix "siginificant" typo (#1626) 2023-06-19 16:12:01 +02:00
Marc Rousavy
8753af3633 docs: Use other Algolia App ID 2023-05-23 17:41:29 +02:00
Marc Rousavy
ba099e4044 docs: Add Multi-Camera zooming GIF 2023-03-16 13:40:05 -04:00
Marc Rousavy
3978406755 docs: Set up Google gtag 2023-03-16 13:30:59 -04:00
Marc Rousavy
1bd21a8d2b docs: Fix og:image 2023-03-16 13:24:15 -04:00
Marc Rousavy
49322f278b docs: Fix favicon 2023-03-16 13:22:59 -04:00
Marc Rousavy
06cbb742fb docs: Add V3 Announcement Banner 2023-03-08 12:11:03 +01:00
Marc Rousavy
c2fb5bf156 docs: fix baseUrl 2023-03-08 09:26:24 +01:00
Marc Rousavy
6fa8b7f26e docs: Add robots.txt 2023-03-08 09:25:12 +01:00
Marc Rousavy
35c779b1ad docs: Add @vercel/analytics 2023-03-07 11:53:32 +01:00
Marc Rousavy
73c5d675f2 docs: fix algolia app ID 2023-03-06 12:19:30 +01:00
Marc Rousavy
815e47b249 Add og:image to theme config 2023-03-06 11:43:39 +01:00
Marc Rousavy
22c0c029c8 docs: Add og:image 2023-03-06 11:42:01 +01:00
Marc Rousavy
77dd181da9 docs: fix meta 2023-03-06 11:40:46 +01:00
Marc Rousavy
ba20e76d38 remove build docs 2023-03-06 11:36:00 +01:00
Marc Rousavy
ab757125b5
docs: Upgrade Docusaurus (#1513)
* Add sitemaps plugin

* Update docusaurus.config.js

* Update libs

* Add OpenGraph metadata

* use same url
2023-03-06 11:25:44 +01:00
Marc Rousavy
757ecb0640 docs: Deploy to https://react-native-vision-camera.com 2023-03-06 11:03:40 +01:00
Marc Rousavy
622d3830f1
feat: Make Frame Processor Plugins object-oriented on iOS as well (#1496)
* feat: Make Frame Processor Plugins object-oriented on iOS as well

* Add Plugin in AppDelegate
2023-02-27 11:18:03 +01:00
Marc Rousavy
566eda2395 docs: Run yarn install for root as well 2023-02-25 12:00:49 +01:00
Marc Rousavy
30b56153db
feat: Sync Frame Processors (plus runAsync and runAtTargetFps) (#1472)
Before, Frame Processors ran on a separate Thread.

After, Frame Processors run fully synchronous and always at the same FPS as the Camera.

Two new functions have been introduced:

* `runAtTargetFps(fps: number, func: () => void)`: Runs the given code as often as the given `fps`, effectively throttling it's calls.
* `runAsync(frame: Frame, func: () => void)`: Runs the given function on a separate Thread for Frame Processing. A strong reference to the Frame is held as long as the function takes to execute.

You can use `runAtTargetFps` to throttle calls to a specific API (e.g. if your Camera is running at 60 FPS, but you only want to run face detection at ~25 FPS, use `runAtTargetFps(25, ...)`.)

You can use `runAsync` to run a heavy algorithm asynchronous, so that the Camera is not blocked while your algorithm runs. This is useful if your main sync processor draws something, and your async processor is doing some image analysis on the side. 

You can also combine both functions.

Examples:

```js
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")
}, [])
```

```js
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")

  runAtTargetFps(10, () => {
    'worklet'
    console.log("I'm running at 10 FPS!")
  })
}, [])
```



```js
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")

  runAsync(frame, () => {
    'worklet'
    console.log("I'm running on another Thread, I can block for longer!")
  })
}, [])
```

```js
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")

  runAtTargetFps(10, () => {
    'worklet'
    runAsync(frame, () => {
      'worklet'
      console.log("I'm running on another Thread at 10 FPS, I can block for longer!")
    })
  })
}, [])
```
2023-02-15 16:47:09 +01:00
Marc Rousavy
a0590dccb5
feat: Replace Reanimated with RN Worklets (#1468)
* Setup RN Worklets

* Use RN Worklets on iOS

* Fix console

* Add `installFrameProcessorBindings()` function

* Add `FrameProcessorPlugins` proxy (BREAKING CHANGE)

* Clean up docs

* Update FRAME_PROCESSORS.mdx

* Use RN Worklets 0.2.5

* feat: Android build setup

* Rewrite Android Frame Processor Part

* Update CMakeLists.txt

* fix: Add react-native-worklets Gradle dependency

* Update Podfile.lock

* fix build

* gradle:7.4.1

* Init JSI Bindings in method on Android

* Fix Folly flags

* fix: Init `FrameProcessorRuntimeManager` later

* fix: Wrap in `<GestureHandlerRootView>`

* Refactor plugins

* fix: Remove enableFrameProcessors

* Install RN Worklets from current GH master

* Update babel.config.js

* Update CameraViewModule.kt

* Update ImageProxyUtils.java

* feat: Upgrade to Reanimated v3

* fix: Fix crash on Worklet init

* Update RN Worklets to latest master

* fix: Simplify FP Plugins Proxy
2023-02-13 15:22:45 +01:00
Marc Rousavy
11d1e7178d
chore: Upgrade to RN 71 (#1465)
* chore: Upgrade Example to RN 0.71

* chore: Upgrade all libs

* fix: Fix CameraRoll installation

* Update Gradle Tools

* fix: Fix buildscripts

* Clean out build.gradle

* fix: Fix Kotlin setup

* fix: Move kotlin-android dependency to lib

* Move `_setGlobalConsole`

* Update gradle-wrapper.properties

* Rebuild lockfiles

* chore: Update build:gradle

* Update StatusBarBlurBackground.tsx

* Use Java 11 in Workflows

* Update MediaPage.tsx

* Add `google` repository to build.gradle

* Double Java Heap size

* Increase heap size

* Alternative args

* Update build.gradle
2023-02-09 11:52:41 +01:00
Rohan Barman
6b32cd9cd7
docs: Replace outdated react-native-reanimated doc links (#1304) 2022-10-28 12:18:36 +02:00
Lihang Xu
312b82b9f6
docs: Add Dynamsoft Label Recognizer Frame Processor Plugin (#1183) 2022-08-05 09:52:35 +02:00
Philipp Parzer
f289f91773
docs: Fix typo in ERRORS (#1160)
fix typo "promt" to "prompt"
2022-07-25 13:26:18 +02:00
Mateusz Mędrek
2a73c09be5
docs: add vision-camera-plugin-builder sections in Creating Frame Processor Plugins (#1123)
- add automatic setup section to "Creating Frame Processor Plugins (iOS|Android)" with
vision-camera-plugin-builder CLI
2022-07-07 14:06:31 +02:00
Joaquín Ossandon
de41f8be83
docs: 📄 Add mocking section to docs (#1061)
* docs: add mocking mdx guide

* docs: show mocking section on sidebar
2022-05-27 11:34:54 +02:00
Marc Rousavy
485f33eb1c docs: Add Margelo Banner (https://margelo.io) 2022-04-11 15:55:03 +02:00
Aaron Grider
e529591880
docs: Add OCR frame processor plugin (#842) 2022-02-22 13:27:27 +01:00
Dominique Rau
c2de683d67
Update TROUBLESHOOTING.mdx (#832) 2022-02-17 11:10:14 +01:00
Lihang Xu
5e7b6a7050
docs: Add Dynamsoft Barcode Reader plugin (#772) 2022-01-27 08:45:07 +01:00
Marc Rousavy
13516c706b Update FRAME_PROCESSORS.mdx 2022-01-11 12:41:41 +01:00
Marc Rousavy
a0b21775e3 Update FRAME_PROCESSORS.mdx 2022-01-11 12:41:19 +01:00
Marc Rousavy
d44ed2ea39 Update FRAME_PROCESSORS.mdx 2022-01-11 12:39:05 +01:00
Marc Rousavy
c2518bc40f docs: Add documentation for disabling Frame Processors on Expo 2022-01-11 12:38:32 +01:00
Marc Rousavy
9d3665d7b8
chore(deps): Upgrade docs deps (#697)
* chore(deps): Upgrade deps (docs)

* Update package.json

* Update yarn.lock
2022-01-02 18:06:19 +01:00
Marc Rousavy
be5ec69b02
feat: Make Reanimated optional (disable Frame Processors if REA v2 is not installed) (#412)
* Fix building iOS without Reanimated

* Conditionally compile Frame Processors (gradle)

* Conditionally use externalNativeBuild

* Remove Reanimated import

* fix: Conditionally load REA/VisionCamera libraries

* fix: Add disable FP to docs

* fix: Fix dummy placeholder for Scheduler.mm

* fix: Fix dummy `Scheduler` declaration

* fix: Only init `CameraView` C++ side if frame processors are enabled

* fix: Install JSI Bindings on Frame Processor Manager ctor

* fix: Wrong conditional

* whoops
2022-01-02 17:35:26 +01:00
Daniel
5bbe2bfbab
docs: Update compileSdkVersion and targetSdkVersion (#585)
SDK version 31 required after 8d24e344c9
2021-12-30 11:39:38 +01:00
Hirbod
c2ba4fb1af
docs: Update Zooming code link (#661)
Change example link from fixed commit to master (this prevents copying old/non-working code)
2021-12-30 10:10:15 +01:00
dependabot[bot]
b10ca27daa
chore(deps-dev): bump docusaurus-plugin-typedoc in /docs (#649)
Bumps [docusaurus-plugin-typedoc](https://github.com/tgreyuk/typedoc-plugin-markdown/tree/HEAD/packages/docusaurus-plugin-typedoc) from 0.16.3 to 0.16.6.
- [Release notes](https://github.com/tgreyuk/typedoc-plugin-markdown/releases)
- [Changelog](https://github.com/tgreyuk/typedoc-plugin-markdown/blob/master/packages/docusaurus-plugin-typedoc/CHANGELOG.md)
- [Commits](https://github.com/tgreyuk/typedoc-plugin-markdown/commits/docusaurus-plugin-typedoc@0.16.6/packages/docusaurus-plugin-typedoc)

---
updated-dependencies:
- dependency-name: docusaurus-plugin-typedoc
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-13 17:01:21 +01:00
Marc Rousavy
934de142ee docs: Update Format.pixelFormat documentation 2021-12-10 09:52:40 +01:00
dependabot[bot]
007cf8f72a
chore(deps-dev): bump typedoc-plugin-markdown in /docs (#509)
Bumps [typedoc-plugin-markdown](https://github.com/tgreyuk/typedoc-plugin-markdown/tree/HEAD/packages/typedoc-plugin-markdown) from 3.11.2 to 3.11.3.
- [Release notes](https://github.com/tgreyuk/typedoc-plugin-markdown/releases)
- [Changelog](https://github.com/tgreyuk/typedoc-plugin-markdown/blob/master/packages/typedoc-plugin-markdown/CHANGELOG.md)
- [Commits](https://github.com/tgreyuk/typedoc-plugin-markdown/commits/typedoc-plugin-markdown@3.11.3/packages/typedoc-plugin-markdown)

---
updated-dependencies:
- dependency-name: typedoc-plugin-markdown
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-11 18:29:02 +02:00
dependabot[bot]
eff2e17c97
chore(deps-dev): bump typedoc from 0.22.4 to 0.22.5 in /docs (#475)
Bumps [typedoc](https://github.com/TypeStrong/TypeDoc) from 0.22.4 to 0.22.5.
- [Release notes](https://github.com/TypeStrong/TypeDoc/releases)
- [Changelog](https://github.com/TypeStrong/typedoc/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TypeStrong/TypeDoc/compare/v0.22.4...v0.22.5)

---
updated-dependencies:
- dependency-name: typedoc
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-05 10:38:37 +02:00
Ben McDonald
c627febde5
docs: Fix small typo (#473) 2021-10-02 15:07:32 +02:00
dependabot[bot]
d2cb52fe92
chore(deps-dev): bump typedoc-plugin-markdown in /docs (#462)
Bumps [typedoc-plugin-markdown](https://github.com/tgreyuk/typedoc-plugin-markdown/tree/HEAD/packages/typedoc-plugin-markdown) from 3.11.0 to 3.11.2.
- [Release notes](https://github.com/tgreyuk/typedoc-plugin-markdown/releases)
- [Changelog](https://github.com/tgreyuk/typedoc-plugin-markdown/blob/master/packages/typedoc-plugin-markdown/CHANGELOG.md)
- [Commits](https://github.com/tgreyuk/typedoc-plugin-markdown/commits/typedoc-plugin-markdown@3.11.2/packages/typedoc-plugin-markdown)

---
updated-dependencies:
- dependency-name: typedoc-plugin-markdown
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-09-29 12:31:47 +02:00