Commit Graph

1590 Commits

Author SHA1 Message Date
Marc Rousavy
a7c137da07
feat: Skia for Android (#1731)
* feat: Call Skia Renderer

* Use default NativePreviewView for Skia

* Render to separate FBO

* It appears once

* Refactor a lot lol

* Pass width/height

* Read width/heights

* Update SkiaRenderer.cpp

* Read stencil/samples

* Use switch for target

* Clear full red

* Update VideoPipeline.cpp

* fix: Use `BorrowTextureFrom` instead of `AdoptTextureFrom`

* Get it to work

* Draw Camera Frame again (only works for first frame)

* glDisable(GL_BLEND)

* Use Frame Buffer again

* Simplify Skia offscreen surface creation

* fix: Get it to kinda work?

* fix: Remove `sampler2D` shader

Only the EXTERNAL_OES one kinda works

* Revert "fix: Remove `sampler2D` shader"

This reverts commit bf241a82f440f5a442f23a2b10329b813e7cdb3e.

* Revert "fix: Get it to kinda work?"

This reverts commit ea6a8784ad8dc7d05e8076591874f021b51dd84a.

* fix: Use Skia for rendering

* Simplify drawing code a lot

* Clean up drawing loop a bit more

* Some docs

* Update SkiaRenderer.cpp

* Surface

* try to use Matrix

* Use BottomLeft as a surface origin again

* Get actual surface dimensions

* Use 1x1 pbuffer instead

* Update SkiaRenderer.cpp

* Update SkiaRenderer.cpp

* feat: Implement Skia Frame Processor (#1735)

* feat: Implement JS Skia Frame Processor

* Update SkiaRenderer.cpp

* push

* Create Frame from C++

* compile

* Compile

* Update VideoPipeline.cpp

* Fix JNI local ref

* Use `HardwareBuffer` for implementation

* feat: Custom `Frame` implementation that uses CPU `ByteBuffer` (#1736)

* feat: Implement JS Skia Frame Processor

* Update SkiaRenderer.cpp

* push

* Create Frame from C++

* compile

* Compile

* Update VideoPipeline.cpp

* Fix JNI local ref

* Use `HardwareBuffer` for implementation

* try: Try to just create a CPU based ByteBuffer

* fix: Fix Java Type

* fix remaining errors

* try fixing FrameFactory

* Use `free`

* fix: Fix scene mode crash on some emulators

* fix: Fix scene mode crash on some emulators

* Fix getting pixels

* fix: Fix buffer not being freed

* Add some docs to `Frame`

* Test Skia again

* Use `getCurrentPresentationTime()`

* Remove `FrameFactory.cpp`

* Update VideoPipeline.h

* Update VideoPipeline.cpp
2023-09-01 10:43:19 +02:00
Marc Rousavy
6bbb44d541 fix: Fix scene mode crash on some emulators 2023-08-31 21:32:59 +02:00
Rocket13011
3b0475752d
fix: Make Skia and Worklets optional in package.json (#1728)
Co-authored-by: Rocket13011 <1051575+rocket13011@users.noreply.github.com>
2023-08-29 17:53:00 +02:00
Marc Rousavy
ea3686cb9a
feat: Create C++/OpenGL-based Video Pipeline for more efficient Recording and Frame Processing (#1721)
* Create `VideoPipeline` c++

* Remove folly C++ dependency

* Create `VideoPipeline` HybridClass

* Set up OpenGL

* Add outputs

* Update VideoPipeline.kt

* Bum `minSdkVersion` to `26`

* Create `VideoPipelineOutput`

* Create output funcs

* Set output pipelines

* Add FP/Recording on Output change

* Update VideoPipeline.cpp

* Create `PassThroughShader`

* Try to draw? I have honestly no idea

* fix: Fix `setFrameProcessor` nameclash

* fix: Fix `high-res-sizes` being null

* Add preview output

* Create `OpenGLContext.cpp`

* Make screen red

* This _should_ work (MESSY)

* FINALLY RENDER TEXTURE

* Rotate

* Mirror

* Clean up a bit

* Add `getWidth()`/`getHeight()`

* Cleanup

* fix: Use uniforms instead of attributes

* Draw with passed rotation/mirror mode

* feat: Use SurfaceTexture's transformMatrix in OpenGL pipeline (#1727)

* feat: Use Transform Matrix from SurfaceTexture

* Renam

* feat: Fix OpenGL Shader

* Update VideoPipeline.kt

* Measure elapsed time

* fix: Fix low resolution

* Render to offscreen

* Render to every context

* Release `SurfaceTexture` on close

* Use one OpenGL context to render to multiple EGLSurfaces

* Clean up a bit

* fix: Fix recording pipeline not triggering

* fix: Synchronize close to prevent nulls

* Update OpenGLRenderer.cpp

* fix: Hardcode Android recorder size
2023-08-29 17:52:03 +02:00
Marc Rousavy
dfb86e174b
feat: Replace *NativeMap and *NativeArray with Map<K,V> and List<T> for faster JSI -> JNI calls (#1720)
Replaces `ReadableNativeMap`/`WritableNativeMap` with `Map<String, Object>` and `ReadableNativeArray`/`WritableNativeArray` with `List<Object>`, making the JSI -> JNI conversion a bit faster and more logical.
Also, we could now convert Array Buffers or HostObjects if we wanted to.
2023-08-25 12:22:44 +02:00
Marc Rousavy
f87bc74de1 chore: release 3.0.0-rc.8 2023-08-25 11:28:43 +02:00
Marc Rousavy
828e1baed9 chore: Bump version to RC.7 2023-08-25 11:26:59 +02:00
Marc Rousavy
dfee3b16b7 fix: Export pixelFormat property on iOS 2023-08-24 17:14:24 +02:00
Marc Rousavy
e036b3162a fix: Lock PixelBuffer for read in toArrayBuffer() 2023-08-24 16:37:20 +02:00
Marc Rousavy
d27581243d fix: Error on disabled Frame Processors 2023-08-24 16:28:16 +02:00
Marc Rousavy
66c012f3b2 fix: Fix node_modules/ directory detection 2023-08-24 16:16:45 +02:00
Marc Rousavy
4833c3e2b8 chore: Bump version 2023-08-24 15:54:17 +02:00
Marc Rousavy
57592d4746 chore: release 3.0.0-rc.6 2023-08-24 15:48:38 +02:00
Marc Rousavy
23af74aaf1
feat: Support focus() on Android (#1713)
* feat: Support `focus()` on Android

* perf: Cache RequestBuilder

* Use 30 FPS as default

* feat: Implement focus(x, y)

* fix: Fix ByteBuffer resizing
2023-08-24 11:49:27 +02:00
Marc Rousavy
c88605e230 fix: Fix circular dependency in FrameProcessorPlugins.ts
shoutout @bglgwyng
2023-08-24 10:55:57 +02:00
Marc Rousavy
4b009a0053 fix: Improve performance of zoom, fps, hdr and format selection 2023-08-23 15:50:34 +02:00
Marc Rousavy
efe6556fc2 feat: Implement enableZoomGesture on Android 2023-08-23 15:39:24 +02:00
Marc Rousavy
86468e3e1f fix: Use direct ByteBuffer 2023-08-23 14:47:42 +02:00
Marc Rousavy
3a0d7b3c91 fix: Fix JByteBuffer 2023-08-23 14:45:10 +02:00
Marc Rousavy
521d7c8ccf feat: Use ByteBuffer for much faster toArrayBuffer() 2023-08-23 14:23:31 +02:00
Marc Rousavy
862e05b64f
feat: Allow build without Skia or Frame Processors (#1710)
* feat: Make Frame Processors optional in JS

* Allow Android build without Frame Processors

* fix: Fix `EncoderProfiles.width` null-error

* Update gradle.properties

* Update gradle.properties

* fix: Use `#ifdef` instead of `#if`

* Update JVisionCameraProxy.cpp

* fix: Fix definitions

* Revert "fix: Use `#ifdef` instead of `#if`"

This reverts commit b19f32e5ce7df558cadcc8c4b5006c9cdf2cbe66.

* fix: Fix build

* chore: Codestyle

* Update JFrameProcessor.cpp
2023-08-23 12:42:38 +02:00
Marc Rousavy
617c5607d4 fix: Remove unused imports 2023-08-23 10:34:13 +02:00
Marc Rousavy
a46839a57b
feat: Add enableShutterSound prop to takePhoto() 🔊 (#1702)
* feat: Add `enableShutterSound` prop to `takePhoto()` 🔊

* Swift lint
2023-08-21 15:27:42 +02:00
Marc Rousavy
0a51980f26 fix: Use default HDR if not applicable 2023-08-21 14:45:48 +02:00
Marc Rousavy
94d9b69903 fix: Remove unused constructor 2023-08-21 14:43:12 +02:00
Marc Rousavy
4d77bc3ea9 Try to select HDR profile 2023-08-21 14:42:34 +02:00
Marc Rousavy
e1b04088c6 fix: Clean up codebase 2023-08-21 14:24:06 +02:00
Marc Rousavy
07ba0e1a41 fix: Improve build.gradle 2023-08-21 14:08:10 +02:00
Marc Rousavy
1d12cd91cb Optimize imports 2023-08-21 14:02:20 +02:00
Marc Rousavy
1f12192baf fix: Don't error if session closes 2023-08-21 14:01:22 +02:00
Marc Rousavy
390f48d6a4 fix: Add GrMTLHandle import 2023-08-21 13:13:57 +02:00
Marc Rousavy
54240b08a9 chore: Update lockfiles 2023-08-21 12:56:50 +02:00
Marc Rousavy
37a3548a81
feat: Full Android rewrite (CameraX -> Camera2) (#1674)
* Nuke CameraX

* fix: Run View Finder on UI Thread

* Open Camera, set up Threads

* fix init

* Mirror if needed

* Try PreviewView

* Use max resolution

* Add `hardwareLevel` property

* Check if output type is supported

* Replace `frameRateRanges` with `minFps` and `maxFps`

* Remove `isHighestPhotoQualitySupported`

* Remove `colorSpace`

The native platforms will use the best / most accurate colorSpace by default anyways.

* HDR

* Check from format

* fix

* Remove `supportsParallelVideoProcessing`

* Correctly return video/photo sizes on Android now. Finally

* Log all Device props

* Log if optimized usecase is used

* Cleanup

* Configure Camera Input only once

* Revert "Configure Camera Input only once"

This reverts commit 0fd6c03f54c7566cb5592053720c4a8743aba92e.

* Extract Camera configuration

* Try to reconfigure all

* Hook based

* Properly set up `CameraSession`

* Delete unused

* fix: Fix recreate when outputs change

* Update NativePreviewView.kt

* Use callback for closing

* Catch CameraAccessException

* Finally got it stable

* Remove isMirrored

* Implement `takePhoto()`

* Add ExifInterface library

* Run findViewById on UI Thread

* Add Photo Output Surface to takePhoto

* Fix Video Stabilization Modes

* Optimize Imports

* More logs

* Update CameraSession.kt

* Close Image

* Use separate Executor in CameraQueue

* Delete hooks

* Use same Thread again

* If opened, call error

* Update CameraSession.kt

* Log HW level

* fix: Don't enable Stream Use Case if it's not 100% supported

* Move some stuff

* Cleanup PhotoOutputSynchronizer

* Try just open in suspend fun

* Some synchronization fixes

* fix logs

* Update CameraDevice+createCaptureSession.kt

* Update CameraDevice+createCaptureSession.kt

* fixes

* fix: Use Snapshot Template for speed capture prio

* Use PREVIEW template for repeating request

* Use `TEMPLATE_RECORD` if video use-case is attached

* Use `isRunning` flag

* Recreate session everytime on active/inactive

* Lazily get values in capture session

* Stability

* Rebuild session if outputs change

* Set `didOutputsChange` back to false

* Capture first in lock

* Try

* kinda fix it? idk

* fix: Keep Outputs

* Refactor into single method

* Update CameraView.kt

* Use Enums for type safety

* Implement Orientation (I think)

* Move RefCount management to Java (Frame)

* Don't crash when dropping a Frame

* Prefer Devices with higher max resolution

* Prefer multi-cams

* Use FastImage for Media Page

* Return orientation in takePhoto()

* Load orientation from EXIF Data

* Add `isMirrored` props and documentation for PhotoFile

* fix: Return `not-determined` on Android

* Update CameraViewModule.kt

* chore: Upgrade packages

* fix: Fix Metro Config

* Cleanup config

* Properly mirror Images on save

* Prepare MediaRecorder

* Start/Stop MediaRecorder

* Remove `takeSnapshot()`

It no longer works on Android and never worked on iOS. Users could use useFrameProcessor to take a Snapshot

* Use `MediaCodec`

* Move to `VideoRecording` class

* Cleanup Snapshot

* Create `SkiaPreviewView` hybrid class

* Create OpenGL context

* Create `SkiaPreviewView`

* Fix texture creation missing context

* Draw red frame

* Somehow get it working

* Add Skia CMake setup

* Start looping

* Init OpenGL

* Refactor into `SkiaRenderer`

* Cleanup PreviewSize

* Set up

* Only re-render UI if there is a new Frame

* Preview

* Fix init

* Try rendering Preview

* Update SkiaPreviewView.kt

* Log version

* Try using Skia (fail)

* Drawwwww!!!!!!!!!! 🎉

* Use Preview Size

* Clear first

* Refactor into SkiaRenderer

* Add `previewType: "none"` on iOS

* Simplify a lot

* Draw Camera? For some reason? I have no idea anymore

* Fix OpenGL errors

* Got it kinda working again?

* Actually draw Frame woah

* Clean up code

* Cleanup

* Update on main

* Synchronize render calls

* holy shit

* Update SkiaRenderer.cpp

* Update SkiaRenderer.cpp

* Refactor

* Update SkiaRenderer.cpp

* Check for `NO_INPUT_TEXTURE`^

* Post & Wait

* Set input size

* Add Video back again

* Allow session without preview

* Convert JPEG to byte[]

* feat: Use `ImageReader` and use YUV Image Buffers in Skia Context (#1689)

* Try to pass YUV Buffers as Pixmaps

* Create pixmap!

* Clean up

* Render to preview

* Only render if we have an output surface

* Update SkiaRenderer.cpp

* Fix Y+U+V sampling code

* Cleanup

* Fix Semaphore 0

* Use 4:2:0 YUV again idk

* Update SkiaRenderer.h

* Set minSdk to 26

* Set surface

* Revert "Set minSdk to 26"

This reverts commit c4085b7c16c628532e5c2d68cf7ed11c751d0b48.

* Set previewType

* feat: Video Recording with Camera2 (#1691)

* Rename

* Update CameraSession.kt

* Use `SurfaceHolder` instead of `SurfaceView` for output

* Update CameraOutputs.kt

* Update CameraSession.kt

* fix: Fix crash when Preview is null

* Check if snapshot capture is supported

* Update RecordingSession.kt

* S

* Use `MediaRecorder`

* Make audio optional

* Add Torch

* Output duration

* Update RecordingSession.kt

* Start RecordingSession

* logs

* More log

* Base for preparing pass-through Recording

* Use `ImageWriter` to append Images to the Recording Surface

* Stream PRIVATE GPU_SAMPLED_IMAGE Images

* Add flags

* Close session on stop

* Allow customizing `videoCodec` and `fileType`

* Enable Torch

* Fix Torch Mode

* Fix comparing outputs with hashCode

* Update CameraSession.kt

* Correctly pass along Frame Processor

* fix: Use AUDIO_BIT_RATE of 16 * 44,1Khz

* Use CAMCORDER instead of MIC microphone

* Use 1 channel

* fix: Use `Orientation`

* Add `native` PixelFormat

* Update iOS to latest Skia integration

* feat: Add `pixelFormat` property to Camera

* Catch error in configureSession

* Fix JPEG format

* Clean up best match finder

* Update CameraDeviceDetails.kt

* Clamp sizes by maximum CamcorderProfile size

* Remove `getAvailableVideoCodecs`

* chore: release 3.0.0-rc.5

* Use maximum video size of RECORD as default

* Update CameraDeviceDetails.kt

* Add a todo

* Add JSON device to issue report

* Prefer `full` devices and flash

* Lock to 30 FPS on Samsung

* Implement Zoom

* Refactor

* Format -> PixelFormat

* fix: Feat `pixelFormat` -> `pixelFormats`

* Update TROUBLESHOOTING.mdx

* Format

* fix: Implement `zoom` for Photo Capture

* fix: Don't run if `isActive` is `false`

* fix: Call `examplePlugin(frame)`

* fix: Fix Flash

* fix: Use `react-native-worklets-core`!

* fix: Fix import
2023-08-21 12:50:14 +02:00
Marc Rousavy
c62b5debe7 chore: release 2.15.6 2023-08-14 17:07:48 +02:00
Marc Rousavy
61fd4e0474
Merge branch 'main' into v3 2023-07-31 18:27:11 +02:00
Marc Rousavy
d62a2cd816
chore: Upgrade root dependencies (#1663)
* chore: Upgrade root dependencies

* chore: Update `example/` dependencies

* Upgrade npm

* fix versions

* fix prettier plugin
2023-07-31 18:21:55 +02:00
Marc Rousavy
1197df7094 fix: Fix make_shared not working on FrameHostObject 2023-07-31 18:11:43 +02:00
Marc Rousavy
1e57e5b95c ci: Add CI tests to build Android without Skia or FP 2023-07-31 17:38:58 +02:00
Marc Rousavy
0949dab086 chore: release 3.0.0-rc.4 2023-07-31 11:52:58 +02:00
Marc Rousavy
bc9c157294 fix: Move implementation for FrameProcessorPlugin to .m 2023-07-31 11:52:10 +02:00
Marc Rousavy
127656b44c fix: pragma once 2023-07-31 11:41:47 +02:00
Marc Rousavy
7c721ccb81 fix: Remove iOS 13 guard 2023-07-31 11:34:37 +02:00
Marc Rousavy
c2096a4c71 fix: Fix FP Plugin init call 2023-07-31 11:34:27 +02:00
Marc Rousavy
deeb8db59f fix: Fix namespace 2023-07-31 11:34:21 +02:00
Marc Rousavy
78a64abb8c chore: Bump version 2023-07-31 11:31:55 +02:00
Rutvik Nabhoya
3b3515c9d4
fix: Fix various Android crashes by using Stable CameraX version (#1665) 2023-07-31 11:28:07 +02:00
Marc Rousavy
b681fc59a6 chore: release 2.15.5 2023-07-31 11:19:47 +02:00
Marc Rousavy
8b9b702edc fix: Pin fbjni to version 0.4.0 2023-07-31 11:18:00 +02:00
Marc Rousavy
86dd703c2b
feat: Rewrite Android C++ part (VisionCameraProxy + JFrame) (#1661)
* First Android rewrite

* Rewrite Android C++ backend

* Pass `ReadableNativeMap`, fix build error

* fix: Fix FrameProcessor init

* Make a bunch of stuff const reference to avoid copies

* Indents

* Cleanup

* indents

* docs: Update Android docs

* Update CameraView.kt

* fix: Format C++ code
2023-07-22 00:15:11 +02:00
Marc Rousavy
44ed42d5d6
feat: Expose unified VisionCameraProxy object, make FrameProcessorPlugins object-oriented (#1660)
* feat: Replace `FrameProcessorRuntimeManager` with `VisionCameraProxy` (iOS)

* Make `FrameProcessorPlugin` a constructable HostObject

* fix: Fix `name` override

* Simplify `useFrameProcessor

* fix: Fix lint errors

* Remove FrameProcessorPlugin::name

* JSIUtils -> JSINSObjectConversion
2023-07-21 17:52:30 +02:00