feat: Full Android rewrite (CameraX -> Camera2) (#1674)

* Nuke CameraX

* fix: Run View Finder on UI Thread

* Open Camera, set up Threads

* fix init

* Mirror if needed

* Try PreviewView

* Use max resolution

* Add `hardwareLevel` property

* Check if output type is supported

* Replace `frameRateRanges` with `minFps` and `maxFps`

* Remove `isHighestPhotoQualitySupported`

* Remove `colorSpace`

The native platforms will use the best / most accurate colorSpace by default anyways.

* HDR

* Check from format

* fix

* Remove `supportsParallelVideoProcessing`

* Correctly return video/photo sizes on Android now. Finally

* Log all Device props

* Log if optimized usecase is used

* Cleanup

* Configure Camera Input only once

* Revert "Configure Camera Input only once"

This reverts commit 0fd6c03f54c7566cb5592053720c4a8743aba92e.

* Extract Camera configuration

* Try to reconfigure all

* Hook based

* Properly set up `CameraSession`

* Delete unused

* fix: Fix recreate when outputs change

* Update NativePreviewView.kt

* Use callback for closing

* Catch CameraAccessException

* Finally got it stable

* Remove isMirrored

* Implement `takePhoto()`

* Add ExifInterface library

* Run findViewById on UI Thread

* Add Photo Output Surface to takePhoto

* Fix Video Stabilization Modes

* Optimize Imports

* More logs

* Update CameraSession.kt

* Close Image

* Use separate Executor in CameraQueue

* Delete hooks

* Use same Thread again

* If opened, call error

* Update CameraSession.kt

* Log HW level

* fix: Don't enable Stream Use Case if it's not 100% supported

* Move some stuff

* Cleanup PhotoOutputSynchronizer

* Try just open in suspend fun

* Some synchronization fixes

* fix logs

* Update CameraDevice+createCaptureSession.kt

* Update CameraDevice+createCaptureSession.kt

* fixes

* fix: Use Snapshot Template for speed capture prio

* Use PREVIEW template for repeating request

* Use `TEMPLATE_RECORD` if video use-case is attached

* Use `isRunning` flag

* Recreate session everytime on active/inactive

* Lazily get values in capture session

* Stability

* Rebuild session if outputs change

* Set `didOutputsChange` back to false

* Capture first in lock

* Try

* kinda fix it? idk

* fix: Keep Outputs

* Refactor into single method

* Update CameraView.kt

* Use Enums for type safety

* Implement Orientation (I think)

* Move RefCount management to Java (Frame)

* Don't crash when dropping a Frame

* Prefer Devices with higher max resolution

* Prefer multi-cams

* Use FastImage for Media Page

* Return orientation in takePhoto()

* Load orientation from EXIF Data

* Add `isMirrored` props and documentation for PhotoFile

* fix: Return `not-determined` on Android

* Update CameraViewModule.kt

* chore: Upgrade packages

* fix: Fix Metro Config

* Cleanup config

* Properly mirror Images on save

* Prepare MediaRecorder

* Start/Stop MediaRecorder

* Remove `takeSnapshot()`

It no longer works on Android and never worked on iOS. Users could use useFrameProcessor to take a Snapshot

* Use `MediaCodec`

* Move to `VideoRecording` class

* Cleanup Snapshot

* Create `SkiaPreviewView` hybrid class

* Create OpenGL context

* Create `SkiaPreviewView`

* Fix texture creation missing context

* Draw red frame

* Somehow get it working

* Add Skia CMake setup

* Start looping

* Init OpenGL

* Refactor into `SkiaRenderer`

* Cleanup PreviewSize

* Set up

* Only re-render UI if there is a new Frame

* Preview

* Fix init

* Try rendering Preview

* Update SkiaPreviewView.kt

* Log version

* Try using Skia (fail)

* Drawwwww!!!!!!!!!! 🎉

* Use Preview Size

* Clear first

* Refactor into SkiaRenderer

* Add `previewType: "none"` on iOS

* Simplify a lot

* Draw Camera? For some reason? I have no idea anymore

* Fix OpenGL errors

* Got it kinda working again?

* Actually draw Frame woah

* Clean up code

* Cleanup

* Update on main

* Synchronize render calls

* holy shit

* Update SkiaRenderer.cpp

* Update SkiaRenderer.cpp

* Refactor

* Update SkiaRenderer.cpp

* Check for `NO_INPUT_TEXTURE`^

* Post & Wait

* Set input size

* Add Video back again

* Allow session without preview

* Convert JPEG to byte[]

* feat: Use `ImageReader` and use YUV Image Buffers in Skia Context (#1689)

* Try to pass YUV Buffers as Pixmaps

* Create pixmap!

* Clean up

* Render to preview

* Only render if we have an output surface

* Update SkiaRenderer.cpp

* Fix Y+U+V sampling code

* Cleanup

* Fix Semaphore 0

* Use 4:2:0 YUV again idk

* Update SkiaRenderer.h

* Set minSdk to 26

* Set surface

* Revert "Set minSdk to 26"

This reverts commit c4085b7c16c628532e5c2d68cf7ed11c751d0b48.

* Set previewType

* feat: Video Recording with Camera2 (#1691)

* Rename

* Update CameraSession.kt

* Use `SurfaceHolder` instead of `SurfaceView` for output

* Update CameraOutputs.kt

* Update CameraSession.kt

* fix: Fix crash when Preview is null

* Check if snapshot capture is supported

* Update RecordingSession.kt

* S

* Use `MediaRecorder`

* Make audio optional

* Add Torch

* Output duration

* Update RecordingSession.kt

* Start RecordingSession

* logs

* More log

* Base for preparing pass-through Recording

* Use `ImageWriter` to append Images to the Recording Surface

* Stream PRIVATE GPU_SAMPLED_IMAGE Images

* Add flags

* Close session on stop

* Allow customizing `videoCodec` and `fileType`

* Enable Torch

* Fix Torch Mode

* Fix comparing outputs with hashCode

* Update CameraSession.kt

* Correctly pass along Frame Processor

* fix: Use AUDIO_BIT_RATE of 16 * 44,1Khz

* Use CAMCORDER instead of MIC microphone

* Use 1 channel

* fix: Use `Orientation`

* Add `native` PixelFormat

* Update iOS to latest Skia integration

* feat: Add `pixelFormat` property to Camera

* Catch error in configureSession

* Fix JPEG format

* Clean up best match finder

* Update CameraDeviceDetails.kt

* Clamp sizes by maximum CamcorderProfile size

* Remove `getAvailableVideoCodecs`

* chore: release 3.0.0-rc.5

* Use maximum video size of RECORD as default

* Update CameraDeviceDetails.kt

* Add a todo

* Add JSON device to issue report

* Prefer `full` devices and flash

* Lock to 30 FPS on Samsung

* Implement Zoom

* Refactor

* Format -> PixelFormat

* fix: Feat `pixelFormat` -> `pixelFormats`

* Update TROUBLESHOOTING.mdx

* Format

* fix: Implement `zoom` for Photo Capture

* fix: Don't run if `isActive` is `false`

* fix: Call `examplePlugin(frame)`

* fix: Fix Flash

* fix: Use `react-native-worklets-core`!

* fix: Fix import

example/src/App.tsx

@@ -26,7 +26,7 @@ export function App(): React.ReactElement | null {
     return null;
   }
 
-  const showPermissionsPage = cameraPermission !== 'authorized' || microphonePermission === 'not-determined';
+  const showPermissionsPage = cameraPermission !== 'granted' || microphonePermission === 'not-determined';
   return (
     <NavigationContainer>
       <GestureHandlerRootView style={{ flex: 1 }}>

example/src/CameraPage.tsx

@@ -11,7 +11,7 @@ import {
   useFrameProcessor,
   VideoFile,
 } from 'react-native-vision-camera';
-import { Camera, frameRateIncluded } from 'react-native-vision-camera';
+import { Camera } from 'react-native-vision-camera';
 import { CONTENT_SPACING, MAX_ZOOM_FACTOR, SAFE_AREA_PADDING } from './Constants';
 import Reanimated, { Extrapolate, interpolate, useAnimatedGestureHandler, useAnimatedProps, useSharedValue } from 'react-native-reanimated';
 import { useEffect } from 'react';
@@ -72,13 +72,13 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
       return 30;
     }
 
-    const supportsHdrAt60Fps = formats.some((f) => f.supportsVideoHDR && f.frameRateRanges.some((r) => frameRateIncluded(r, 60)));
+    const supportsHdrAt60Fps = formats.some((f) => f.supportsVideoHDR && f.maxFps >= 60);
     if (enableHdr && !supportsHdrAt60Fps) {
       // User has enabled HDR, but HDR is not supported at 60 FPS.
       return 30;
     }
 
-    const supports60Fps = formats.some((f) => f.frameRateRanges.some((r) => frameRateIncluded(r, 60)));
+    const supports60Fps = formats.some((f) => f.maxFps >= 60);
     if (!supports60Fps) {
       // 60 FPS is not supported by any format.
       return 30;
@@ -90,7 +90,7 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
   const supportsCameraFlipping = useMemo(() => devices.back != null && devices.front != null, [devices.back, devices.front]);
   const supportsFlash = device?.hasFlash ?? false;
   const supportsHdr = useMemo(() => formats.some((f) => f.supportsVideoHDR || f.supportsPhotoHDR), [formats]);
-  const supports60Fps = useMemo(() => formats.some((f) => f.frameRateRanges.some((rate) => frameRateIncluded(rate, 60))), [formats]);
+  const supports60Fps = useMemo(() => formats.some((f) => f.maxFps >= 60), [formats]);
   const canToggleNightMode = enableNightMode
     ? true // it's enabled so you have to be able to turn it off again
     : (device?.supportsLowLightBoost ?? false) || fps > 30; // either we have native support, or we can lower the FPS
@@ -105,7 +105,7 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
     }
 
     // find the first format that includes the given FPS
-    return result.find((f) => f.frameRateRanges.some((r) => frameRateIncluded(r, fps)));
+    return result.find((f) => f.maxFps >= fps);
   }, [formats, fps, enableHdr]);
 
   //#region Animated Zoom
@@ -169,7 +169,7 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
   }, [neutralZoom, zoom]);
 
   useEffect(() => {
-    Camera.getMicrophonePermissionStatus().then((status) => setHasMicrophonePermission(status === 'authorized'));
+    Camera.getMicrophonePermissionStatus().then((status) => setHasMicrophonePermission(status === 'granted'));
   }, []);
   //#endregion
 
@@ -192,7 +192,7 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
   if (device != null && format != null) {
     console.log(
       `Re-rendering camera page with ${isActive ? 'active' : 'inactive'} camera. ` +
-        `Device: "${device.name}" (${format.photoWidth}x${format.photoHeight} @ ${fps}fps)`,
+        `Device: "${device.name}" (${format.photoWidth}x${format.photoHeight} photo / ${format.videoWidth}x${format.videoHeight} video @ ${fps}fps)`,
     );
   } else {
     console.log('re-rendering camera page without active camera');
@@ -221,9 +221,8 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
   const frameProcessor = useFrameProcessor((frame) => {
     'worklet';
 
-    console.log(`Width: ${frame.width}`);
-    const result = examplePlugin(frame);
-    console.log('Example Plugin: ', result);
+    console.log(frame.timestamp, frame.toString(), frame.pixelFormat);
+    examplePlugin(frame);
   }, []);
 
   return (
@@ -245,9 +244,11 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
                 onError={onError}
                 enableZoomGesture={false}
                 animatedProps={cameraAnimatedProps}
-                audio={hasMicrophonePermission}
                 enableFpsGraph={true}
                 orientation="portrait"
+                photo={true}
+                video={true}
+                audio={hasMicrophonePermission}
                 frameProcessor={frameProcessor}
               />
             </TapGestureHandler>

example/src/MediaPage.tsx

@@ -1,5 +1,5 @@
 import React, { useCallback, useMemo, useState } from 'react';
-import { StyleSheet, View, Image, ActivityIndicator, PermissionsAndroid, Platform } from 'react-native';
+import { StyleSheet, View, ActivityIndicator, PermissionsAndroid, Platform } from 'react-native';
 import Video, { LoadError, OnLoadData } from 'react-native-video';
 import { SAFE_AREA_PADDING } from './Constants';
 import { useIsForeground } from './hooks/useIsForeground';
@@ -8,11 +8,10 @@ import IonIcon from 'react-native-vector-icons/Ionicons';
 import { Alert } from 'react-native';
 import { CameraRoll } from '@react-native-camera-roll/camera-roll';
 import { StatusBarBlurBackground } from './views/StatusBarBlurBackground';
-import type { NativeSyntheticEvent } from 'react-native';
-import type { ImageLoadEventData } from 'react-native';
 import type { NativeStackScreenProps } from '@react-navigation/native-stack';
 import type { Routes } from './Routes';
 import { useIsFocused } from '@react-navigation/core';
+import FastImage, { OnLoadEvent } from 'react-native-fast-image';
 
 const requestSavePermission = async (): Promise<boolean> => {
   if (Platform.OS !== 'android') return true;
@@ -27,8 +26,7 @@ const requestSavePermission = async (): Promise<boolean> => {
   return hasPermission;
 };
 
-const isVideoOnLoadEvent = (event: OnLoadData | NativeSyntheticEvent<ImageLoadEventData>): event is OnLoadData =>
-  'duration' in event && 'naturalSize' in event;
+const isVideoOnLoadEvent = (event: OnLoadData | OnLoadEvent): event is OnLoadData => 'duration' in event && 'naturalSize' in event;
 
 type Props = NativeStackScreenProps<Routes, 'MediaPage'>;
 export function MediaPage({ navigation, route }: Props): React.ReactElement {
@@ -39,13 +37,13 @@ export function MediaPage({ navigation, route }: Props): React.ReactElement {
   const isVideoPaused = !isForeground || !isScreenFocused;
   const [savingState, setSavingState] = useState<'none' | 'saving' | 'saved'>('none');
 
-  const onMediaLoad = useCallback((event: OnLoadData | NativeSyntheticEvent<ImageLoadEventData>) => {
+  const onMediaLoad = useCallback((event: OnLoadData | OnLoadEvent) => {
     if (isVideoOnLoadEvent(event)) {
       console.log(
         `Video loaded. Size: ${event.naturalSize.width}x${event.naturalSize.height} (${event.naturalSize.orientation}, ${event.duration} seconds)`,
       );
     } else {
-      console.log(`Image loaded. Size: ${event.nativeEvent.source.width}x${event.nativeEvent.source.height}`);
+      console.log(`Image loaded. Size: ${event.nativeEvent.width}x${event.nativeEvent.height}`);
     }
   }, []);
   const onMediaLoadEnd = useCallback(() => {
@@ -83,7 +81,7 @@ export function MediaPage({ navigation, route }: Props): React.ReactElement {
   return (
     <View style={[styles.container, screenStyle]}>
       {type === 'photo' && (
-        <Image source={source} style={StyleSheet.absoluteFill} resizeMode="cover" onLoadEnd={onMediaLoadEnd} onLoad={onMediaLoad} />
+        <FastImage source={source} style={StyleSheet.absoluteFill} resizeMode="cover" onLoadEnd={onMediaLoadEnd} onLoad={onMediaLoad} />
       )}
       {type === 'video' && (
         <Video

example/src/PermissionsPage.tsx

@@ -34,7 +34,7 @@ export function PermissionsPage({ navigation }: Props): React.ReactElement {
   }, []);
 
   useEffect(() => {
-    if (cameraPermissionStatus === 'authorized' && microphonePermissionStatus === 'authorized') navigation.replace('CameraPage');
+    if (cameraPermissionStatus === 'granted' && microphonePermissionStatus === 'granted') navigation.replace('CameraPage');
   }, [cameraPermissionStatus, microphonePermissionStatus, navigation]);
 
   return (
@@ -42,7 +42,7 @@ export function PermissionsPage({ navigation }: Props): React.ReactElement {
       <Image source={BANNER_IMAGE} style={styles.banner} />
       <Text style={styles.welcome}>Welcome to{'\n'}Vision Camera.</Text>
       <View style={styles.permissionsContainer}>
-        {cameraPermissionStatus !== 'authorized' && (
+        {cameraPermissionStatus !== 'granted' && (
           <Text style={styles.permissionText}>
             Vision Camera needs <Text style={styles.bold}>Camera permission</Text>.{' '}
             <Text style={styles.hyperlink} onPress={requestCameraPermission}>
@@ -50,7 +50,7 @@ export function PermissionsPage({ navigation }: Props): React.ReactElement {
             </Text>
           </Text>
         )}
-        {microphonePermissionStatus !== 'authorized' && (
+        {microphonePermissionStatus !== 'granted' && (
           <Text style={styles.permissionText}>
             Vision Camera needs <Text style={styles.bold}>Microphone permission</Text>.{' '}
             <Text style={styles.hyperlink} onPress={requestMicrophonePermission}>

example/src/views/CaptureButton.tsx

@@ -19,7 +19,7 @@ import Reanimated, {
   useSharedValue,
   withRepeat,
 } from 'react-native-reanimated';
-import type { Camera, PhotoFile, TakePhotoOptions, TakeSnapshotOptions, VideoFile } from 'react-native-vision-camera';
+import type { Camera, PhotoFile, TakePhotoOptions, VideoFile } from 'react-native-vision-camera';
 import { CAPTURE_BUTTON_SIZE, SCREEN_HEIGHT, SCREEN_WIDTH } from './../Constants';
 
 const PAN_GESTURE_HANDLER_FAIL_X = [-SCREEN_WIDTH, SCREEN_WIDTH];
@@ -58,7 +58,7 @@ const _CaptureButton: React.FC<Props> = ({
   const pressDownDate = useRef<Date | undefined>(undefined);
   const isRecording = useRef(false);
   const recordingProgress = useSharedValue(0);
-  const takePhotoOptions = useMemo<TakePhotoOptions & TakeSnapshotOptions>(
+  const takePhotoOptions = useMemo<TakePhotoOptions>(
     () => ({
       photoCodec: 'jpeg',
       qualityPrioritization: 'speed',