react-native-vision-camera/package/ios/CameraView+AVCaptureSession.swift

370 lines
13 KiB
Swift
Raw Normal View History

//
// CameraView+AVCaptureSession.swift
// VisionCamera
//
// Created by Marc Rousavy on 26.03.21.
// Copyright © 2021 mrousavy. All rights reserved.
//
import AVFoundation
2021-03-26 09:28:08 -06:00
import Foundation
/**
Extension for CameraView that sets up the AVCaptureSession, Device and Format.
*/
extension CameraView {
fix: Move Audio Input initialization shortly before `startRecording` (#159) * rename * Update AVAudioSession+updateCategory.swift * fix bootstrap script * Update CameraView+AVAudioSession.swift * move audio input adding lower * Activate AudioSession only when starting recording * format * Deactivate Audio Session * remove audio input before deactivating audio session * Update CameraView+AVAudioSession.swift * log time * Update CameraView+AVAudioSession.swift * measure time with `measureElapsedTime` * Update project.pbxproj * only log in debug builds * bootstrap with bridge (RNN new API) * Mark two funcs as `@inlinable` * format * Update ReactLogger.swift * Make audioWriter optional (allow videos without sound) * only log frame drop reason in DEBUG * Make audio writing entirely optional * format * Use function name as label for measureElapsedTime * Update MeasureElapsedTime.swift * Update MeasureElapsedTime.swift * Mark AudioWriter as finished * set `automaticallyConfiguresApplicationAudioSession` once * Add JS console logging * log to JS console for a few logs * Update AVAudioSession+updateCategory.swift * format * Update JSConsoleHelper.mm * catch log errors * Update ReactLogger.swift * fix docs * Update RecordingSession.swift * Immediatelly add audio input * Update CameraView+AVCaptureSession.swift * Update CameraView+AVCaptureSession.swift * Update ReactLogger.swift * immediatelly set audio session * extract * format * Update TROUBLESHOOTING.mdx * hmm * Update AVAudioSession+updateCategory.swift * Create secondary `AVCaptureSession` for audio * Configure once, start stop on demand * format * fix audio notification interruptions * docs
2021-06-03 06:16:02 -06:00
// pragma MARK: Configure Capture Session
/**
Configures the Capture Session.
*/
2021-03-26 09:28:08 -06:00
final func configureCaptureSession() {
ReactLogger.log(level: .info, message: "Configuring Session...")
isReady = false
2021-03-26 09:28:08 -06:00
#if targetEnvironment(simulator)
invokeOnError(.device(.notAvailableOnSimulator))
return
#endif
2021-03-26 09:28:08 -06:00
guard cameraId != nil else {
invokeOnError(.device(.noDevice))
return
}
let cameraId = self.cameraId! as String
2021-03-26 09:28:08 -06:00
ReactLogger.log(level: .info, message: "Initializing Camera with device \(cameraId)...")
captureSession.beginConfiguration()
defer {
captureSession.commitConfiguration()
}
2021-03-26 09:28:08 -06:00
fix: Move Audio Input initialization shortly before `startRecording` (#159) * rename * Update AVAudioSession+updateCategory.swift * fix bootstrap script * Update CameraView+AVAudioSession.swift * move audio input adding lower * Activate AudioSession only when starting recording * format * Deactivate Audio Session * remove audio input before deactivating audio session * Update CameraView+AVAudioSession.swift * log time * Update CameraView+AVAudioSession.swift * measure time with `measureElapsedTime` * Update project.pbxproj * only log in debug builds * bootstrap with bridge (RNN new API) * Mark two funcs as `@inlinable` * format * Update ReactLogger.swift * Make audioWriter optional (allow videos without sound) * only log frame drop reason in DEBUG * Make audio writing entirely optional * format * Use function name as label for measureElapsedTime * Update MeasureElapsedTime.swift * Update MeasureElapsedTime.swift * Mark AudioWriter as finished * set `automaticallyConfiguresApplicationAudioSession` once * Add JS console logging * log to JS console for a few logs * Update AVAudioSession+updateCategory.swift * format * Update JSConsoleHelper.mm * catch log errors * Update ReactLogger.swift * fix docs * Update RecordingSession.swift * Immediatelly add audio input * Update CameraView+AVCaptureSession.swift * Update CameraView+AVCaptureSession.swift * Update ReactLogger.swift * immediatelly set audio session * extract * format * Update TROUBLESHOOTING.mdx * hmm * Update AVAudioSession+updateCategory.swift * Create secondary `AVCaptureSession` for audio * Configure once, start stop on demand * format * fix audio notification interruptions * docs
2021-06-03 06:16:02 -06:00
// pragma MARK: Capture Session Inputs
// Video Input
do {
2021-12-10 01:44:54 -07:00
if let videoDeviceInput = videoDeviceInput {
captureSession.removeInput(videoDeviceInput)
fix: Move Audio Input initialization shortly before `startRecording` (#159) * rename * Update AVAudioSession+updateCategory.swift * fix bootstrap script * Update CameraView+AVAudioSession.swift * move audio input adding lower * Activate AudioSession only when starting recording * format * Deactivate Audio Session * remove audio input before deactivating audio session * Update CameraView+AVAudioSession.swift * log time * Update CameraView+AVAudioSession.swift * measure time with `measureElapsedTime` * Update project.pbxproj * only log in debug builds * bootstrap with bridge (RNN new API) * Mark two funcs as `@inlinable` * format * Update ReactLogger.swift * Make audioWriter optional (allow videos without sound) * only log frame drop reason in DEBUG * Make audio writing entirely optional * format * Use function name as label for measureElapsedTime * Update MeasureElapsedTime.swift * Update MeasureElapsedTime.swift * Mark AudioWriter as finished * set `automaticallyConfiguresApplicationAudioSession` once * Add JS console logging * log to JS console for a few logs * Update AVAudioSession+updateCategory.swift * format * Update JSConsoleHelper.mm * catch log errors * Update ReactLogger.swift * fix docs * Update RecordingSession.swift * Immediatelly add audio input * Update CameraView+AVCaptureSession.swift * Update CameraView+AVCaptureSession.swift * Update ReactLogger.swift * immediatelly set audio session * extract * format * Update TROUBLESHOOTING.mdx * hmm * Update AVAudioSession+updateCategory.swift * Create secondary `AVCaptureSession` for audio * Configure once, start stop on demand * format * fix audio notification interruptions * docs
2021-06-03 06:16:02 -06:00
self.videoDeviceInput = nil
}
fix: Move Audio Input initialization shortly before `startRecording` (#159) * rename * Update AVAudioSession+updateCategory.swift * fix bootstrap script * Update CameraView+AVAudioSession.swift * move audio input adding lower * Activate AudioSession only when starting recording * format * Deactivate Audio Session * remove audio input before deactivating audio session * Update CameraView+AVAudioSession.swift * log time * Update CameraView+AVAudioSession.swift * measure time with `measureElapsedTime` * Update project.pbxproj * only log in debug builds * bootstrap with bridge (RNN new API) * Mark two funcs as `@inlinable` * format * Update ReactLogger.swift * Make audioWriter optional (allow videos without sound) * only log frame drop reason in DEBUG * Make audio writing entirely optional * format * Use function name as label for measureElapsedTime * Update MeasureElapsedTime.swift * Update MeasureElapsedTime.swift * Mark AudioWriter as finished * set `automaticallyConfiguresApplicationAudioSession` once * Add JS console logging * log to JS console for a few logs * Update AVAudioSession+updateCategory.swift * format * Update JSConsoleHelper.mm * catch log errors * Update ReactLogger.swift * fix docs * Update RecordingSession.swift * Immediatelly add audio input * Update CameraView+AVCaptureSession.swift * Update CameraView+AVCaptureSession.swift * Update ReactLogger.swift * immediatelly set audio session * extract * format * Update TROUBLESHOOTING.mdx * hmm * Update AVAudioSession+updateCategory.swift * Create secondary `AVCaptureSession` for audio * Configure once, start stop on demand * format * fix audio notification interruptions * docs
2021-06-03 06:16:02 -06:00
ReactLogger.log(level: .info, message: "Adding Video input...")
guard let videoDevice = AVCaptureDevice(uniqueID: cameraId) else {
invokeOnError(.device(.invalid))
return
}
videoDeviceInput = try AVCaptureDeviceInput(device: videoDevice)
guard captureSession.canAddInput(videoDeviceInput!) else {
invokeOnError(.parameter(.unsupportedInput(inputDescriptor: "video-input")))
return
}
captureSession.addInput(videoDeviceInput!)
} catch {
invokeOnError(.device(.invalid))
return
}
2021-03-26 09:28:08 -06:00
fix: Move Audio Input initialization shortly before `startRecording` (#159) * rename * Update AVAudioSession+updateCategory.swift * fix bootstrap script * Update CameraView+AVAudioSession.swift * move audio input adding lower * Activate AudioSession only when starting recording * format * Deactivate Audio Session * remove audio input before deactivating audio session * Update CameraView+AVAudioSession.swift * log time * Update CameraView+AVAudioSession.swift * measure time with `measureElapsedTime` * Update project.pbxproj * only log in debug builds * bootstrap with bridge (RNN new API) * Mark two funcs as `@inlinable` * format * Update ReactLogger.swift * Make audioWriter optional (allow videos without sound) * only log frame drop reason in DEBUG * Make audio writing entirely optional * format * Use function name as label for measureElapsedTime * Update MeasureElapsedTime.swift * Update MeasureElapsedTime.swift * Mark AudioWriter as finished * set `automaticallyConfiguresApplicationAudioSession` once * Add JS console logging * log to JS console for a few logs * Update AVAudioSession+updateCategory.swift * format * Update JSConsoleHelper.mm * catch log errors * Update ReactLogger.swift * fix docs * Update RecordingSession.swift * Immediatelly add audio input * Update CameraView+AVCaptureSession.swift * Update CameraView+AVCaptureSession.swift * Update ReactLogger.swift * immediatelly set audio session * extract * format * Update TROUBLESHOOTING.mdx * hmm * Update AVAudioSession+updateCategory.swift * Create secondary `AVCaptureSession` for audio * Configure once, start stop on demand * format * fix audio notification interruptions * docs
2021-06-03 06:16:02 -06:00
// pragma MARK: Capture Session Outputs
// Photo Output
2021-12-10 01:44:54 -07:00
if let photoOutput = photoOutput {
captureSession.removeOutput(photoOutput)
fix: Move Audio Input initialization shortly before `startRecording` (#159) * rename * Update AVAudioSession+updateCategory.swift * fix bootstrap script * Update CameraView+AVAudioSession.swift * move audio input adding lower * Activate AudioSession only when starting recording * format * Deactivate Audio Session * remove audio input before deactivating audio session * Update CameraView+AVAudioSession.swift * log time * Update CameraView+AVAudioSession.swift * measure time with `measureElapsedTime` * Update project.pbxproj * only log in debug builds * bootstrap with bridge (RNN new API) * Mark two funcs as `@inlinable` * format * Update ReactLogger.swift * Make audioWriter optional (allow videos without sound) * only log frame drop reason in DEBUG * Make audio writing entirely optional * format * Use function name as label for measureElapsedTime * Update MeasureElapsedTime.swift * Update MeasureElapsedTime.swift * Mark AudioWriter as finished * set `automaticallyConfiguresApplicationAudioSession` once * Add JS console logging * log to JS console for a few logs * Update AVAudioSession+updateCategory.swift * format * Update JSConsoleHelper.mm * catch log errors * Update ReactLogger.swift * fix docs * Update RecordingSession.swift * Immediatelly add audio input * Update CameraView+AVCaptureSession.swift * Update CameraView+AVCaptureSession.swift * Update ReactLogger.swift * immediatelly set audio session * extract * format * Update TROUBLESHOOTING.mdx * hmm * Update AVAudioSession+updateCategory.swift * Create secondary `AVCaptureSession` for audio * Configure once, start stop on demand * format * fix audio notification interruptions * docs
2021-06-03 06:16:02 -06:00
self.photoOutput = nil
}
if photo?.boolValue == true {
ReactLogger.log(level: .info, message: "Adding Photo output...")
photoOutput = AVCapturePhotoOutput()
if enableHighQualityPhotos?.boolValue == true {
feat: Full Android rewrite (CameraX -> Camera2) (#1674) * Nuke CameraX * fix: Run View Finder on UI Thread * Open Camera, set up Threads * fix init * Mirror if needed * Try PreviewView * Use max resolution * Add `hardwareLevel` property * Check if output type is supported * Replace `frameRateRanges` with `minFps` and `maxFps` * Remove `isHighestPhotoQualitySupported` * Remove `colorSpace` The native platforms will use the best / most accurate colorSpace by default anyways. * HDR * Check from format * fix * Remove `supportsParallelVideoProcessing` * Correctly return video/photo sizes on Android now. Finally * Log all Device props * Log if optimized usecase is used * Cleanup * Configure Camera Input only once * Revert "Configure Camera Input only once" This reverts commit 0fd6c03f54c7566cb5592053720c4a8743aba92e. * Extract Camera configuration * Try to reconfigure all * Hook based * Properly set up `CameraSession` * Delete unused * fix: Fix recreate when outputs change * Update NativePreviewView.kt * Use callback for closing * Catch CameraAccessException * Finally got it stable * Remove isMirrored * Implement `takePhoto()` * Add ExifInterface library * Run findViewById on UI Thread * Add Photo Output Surface to takePhoto * Fix Video Stabilization Modes * Optimize Imports * More logs * Update CameraSession.kt * Close Image * Use separate Executor in CameraQueue * Delete hooks * Use same Thread again * If opened, call error * Update CameraSession.kt * Log HW level * fix: Don't enable Stream Use Case if it's not 100% supported * Move some stuff * Cleanup PhotoOutputSynchronizer * Try just open in suspend fun * Some synchronization fixes * fix logs * Update CameraDevice+createCaptureSession.kt * Update CameraDevice+createCaptureSession.kt * fixes * fix: Use Snapshot Template for speed capture prio * Use PREVIEW template for repeating request * Use `TEMPLATE_RECORD` if video use-case is attached * Use `isRunning` flag * Recreate session everytime on active/inactive * Lazily get values in capture session * Stability * Rebuild session if outputs change * Set `didOutputsChange` back to false * Capture first in lock * Try * kinda fix it? idk * fix: Keep Outputs * Refactor into single method * Update CameraView.kt * Use Enums for type safety * Implement Orientation (I think) * Move RefCount management to Java (Frame) * Don't crash when dropping a Frame * Prefer Devices with higher max resolution * Prefer multi-cams * Use FastImage for Media Page * Return orientation in takePhoto() * Load orientation from EXIF Data * Add `isMirrored` props and documentation for PhotoFile * fix: Return `not-determined` on Android * Update CameraViewModule.kt * chore: Upgrade packages * fix: Fix Metro Config * Cleanup config * Properly mirror Images on save * Prepare MediaRecorder * Start/Stop MediaRecorder * Remove `takeSnapshot()` It no longer works on Android and never worked on iOS. Users could use useFrameProcessor to take a Snapshot * Use `MediaCodec` * Move to `VideoRecording` class * Cleanup Snapshot * Create `SkiaPreviewView` hybrid class * Create OpenGL context * Create `SkiaPreviewView` * Fix texture creation missing context * Draw red frame * Somehow get it working * Add Skia CMake setup * Start looping * Init OpenGL * Refactor into `SkiaRenderer` * Cleanup PreviewSize * Set up * Only re-render UI if there is a new Frame * Preview * Fix init * Try rendering Preview * Update SkiaPreviewView.kt * Log version * Try using Skia (fail) * Drawwwww!!!!!!!!!! :tada: * Use Preview Size * Clear first * Refactor into SkiaRenderer * Add `previewType: "none"` on iOS * Simplify a lot * Draw Camera? For some reason? I have no idea anymore * Fix OpenGL errors * Got it kinda working again? * Actually draw Frame woah * Clean up code * Cleanup * Update on main * Synchronize render calls * holy shit * Update SkiaRenderer.cpp * Update SkiaRenderer.cpp * Refactor * Update SkiaRenderer.cpp * Check for `NO_INPUT_TEXTURE`^ * Post & Wait * Set input size * Add Video back again * Allow session without preview * Convert JPEG to byte[] * feat: Use `ImageReader` and use YUV Image Buffers in Skia Context (#1689) * Try to pass YUV Buffers as Pixmaps * Create pixmap! * Clean up * Render to preview * Only render if we have an output surface * Update SkiaRenderer.cpp * Fix Y+U+V sampling code * Cleanup * Fix Semaphore 0 * Use 4:2:0 YUV again idk * Update SkiaRenderer.h * Set minSdk to 26 * Set surface * Revert "Set minSdk to 26" This reverts commit c4085b7c16c628532e5c2d68cf7ed11c751d0b48. * Set previewType * feat: Video Recording with Camera2 (#1691) * Rename * Update CameraSession.kt * Use `SurfaceHolder` instead of `SurfaceView` for output * Update CameraOutputs.kt * Update CameraSession.kt * fix: Fix crash when Preview is null * Check if snapshot capture is supported * Update RecordingSession.kt * S * Use `MediaRecorder` * Make audio optional * Add Torch * Output duration * Update RecordingSession.kt * Start RecordingSession * logs * More log * Base for preparing pass-through Recording * Use `ImageWriter` to append Images to the Recording Surface * Stream PRIVATE GPU_SAMPLED_IMAGE Images * Add flags * Close session on stop * Allow customizing `videoCodec` and `fileType` * Enable Torch * Fix Torch Mode * Fix comparing outputs with hashCode * Update CameraSession.kt * Correctly pass along Frame Processor * fix: Use AUDIO_BIT_RATE of 16 * 44,1Khz * Use CAMCORDER instead of MIC microphone * Use 1 channel * fix: Use `Orientation` * Add `native` PixelFormat * Update iOS to latest Skia integration * feat: Add `pixelFormat` property to Camera * Catch error in configureSession * Fix JPEG format * Clean up best match finder * Update CameraDeviceDetails.kt * Clamp sizes by maximum CamcorderProfile size * Remove `getAvailableVideoCodecs` * chore: release 3.0.0-rc.5 * Use maximum video size of RECORD as default * Update CameraDeviceDetails.kt * Add a todo * Add JSON device to issue report * Prefer `full` devices and flash * Lock to 30 FPS on Samsung * Implement Zoom * Refactor * Format -> PixelFormat * fix: Feat `pixelFormat` -> `pixelFormats` * Update TROUBLESHOOTING.mdx * Format * fix: Implement `zoom` for Photo Capture * fix: Don't run if `isActive` is `false` * fix: Call `examplePlugin(frame)` * fix: Fix Flash * fix: Use `react-native-worklets-core`! * fix: Fix import
2023-08-21 04:50:14 -06:00
// TODO: In iOS 16 this will be removed in favor of maxPhotoDimensions.
photoOutput!.isHighResolutionCaptureEnabled = true
if #available(iOS 13.0, *) {
feat: Full Android rewrite (CameraX -> Camera2) (#1674) * Nuke CameraX * fix: Run View Finder on UI Thread * Open Camera, set up Threads * fix init * Mirror if needed * Try PreviewView * Use max resolution * Add `hardwareLevel` property * Check if output type is supported * Replace `frameRateRanges` with `minFps` and `maxFps` * Remove `isHighestPhotoQualitySupported` * Remove `colorSpace` The native platforms will use the best / most accurate colorSpace by default anyways. * HDR * Check from format * fix * Remove `supportsParallelVideoProcessing` * Correctly return video/photo sizes on Android now. Finally * Log all Device props * Log if optimized usecase is used * Cleanup * Configure Camera Input only once * Revert "Configure Camera Input only once" This reverts commit 0fd6c03f54c7566cb5592053720c4a8743aba92e. * Extract Camera configuration * Try to reconfigure all * Hook based * Properly set up `CameraSession` * Delete unused * fix: Fix recreate when outputs change * Update NativePreviewView.kt * Use callback for closing * Catch CameraAccessException * Finally got it stable * Remove isMirrored * Implement `takePhoto()` * Add ExifInterface library * Run findViewById on UI Thread * Add Photo Output Surface to takePhoto * Fix Video Stabilization Modes * Optimize Imports * More logs * Update CameraSession.kt * Close Image * Use separate Executor in CameraQueue * Delete hooks * Use same Thread again * If opened, call error * Update CameraSession.kt * Log HW level * fix: Don't enable Stream Use Case if it's not 100% supported * Move some stuff * Cleanup PhotoOutputSynchronizer * Try just open in suspend fun * Some synchronization fixes * fix logs * Update CameraDevice+createCaptureSession.kt * Update CameraDevice+createCaptureSession.kt * fixes * fix: Use Snapshot Template for speed capture prio * Use PREVIEW template for repeating request * Use `TEMPLATE_RECORD` if video use-case is attached * Use `isRunning` flag * Recreate session everytime on active/inactive * Lazily get values in capture session * Stability * Rebuild session if outputs change * Set `didOutputsChange` back to false * Capture first in lock * Try * kinda fix it? idk * fix: Keep Outputs * Refactor into single method * Update CameraView.kt * Use Enums for type safety * Implement Orientation (I think) * Move RefCount management to Java (Frame) * Don't crash when dropping a Frame * Prefer Devices with higher max resolution * Prefer multi-cams * Use FastImage for Media Page * Return orientation in takePhoto() * Load orientation from EXIF Data * Add `isMirrored` props and documentation for PhotoFile * fix: Return `not-determined` on Android * Update CameraViewModule.kt * chore: Upgrade packages * fix: Fix Metro Config * Cleanup config * Properly mirror Images on save * Prepare MediaRecorder * Start/Stop MediaRecorder * Remove `takeSnapshot()` It no longer works on Android and never worked on iOS. Users could use useFrameProcessor to take a Snapshot * Use `MediaCodec` * Move to `VideoRecording` class * Cleanup Snapshot * Create `SkiaPreviewView` hybrid class * Create OpenGL context * Create `SkiaPreviewView` * Fix texture creation missing context * Draw red frame * Somehow get it working * Add Skia CMake setup * Start looping * Init OpenGL * Refactor into `SkiaRenderer` * Cleanup PreviewSize * Set up * Only re-render UI if there is a new Frame * Preview * Fix init * Try rendering Preview * Update SkiaPreviewView.kt * Log version * Try using Skia (fail) * Drawwwww!!!!!!!!!! :tada: * Use Preview Size * Clear first * Refactor into SkiaRenderer * Add `previewType: "none"` on iOS * Simplify a lot * Draw Camera? For some reason? I have no idea anymore * Fix OpenGL errors * Got it kinda working again? * Actually draw Frame woah * Clean up code * Cleanup * Update on main * Synchronize render calls * holy shit * Update SkiaRenderer.cpp * Update SkiaRenderer.cpp * Refactor * Update SkiaRenderer.cpp * Check for `NO_INPUT_TEXTURE`^ * Post & Wait * Set input size * Add Video back again * Allow session without preview * Convert JPEG to byte[] * feat: Use `ImageReader` and use YUV Image Buffers in Skia Context (#1689) * Try to pass YUV Buffers as Pixmaps * Create pixmap! * Clean up * Render to preview * Only render if we have an output surface * Update SkiaRenderer.cpp * Fix Y+U+V sampling code * Cleanup * Fix Semaphore 0 * Use 4:2:0 YUV again idk * Update SkiaRenderer.h * Set minSdk to 26 * Set surface * Revert "Set minSdk to 26" This reverts commit c4085b7c16c628532e5c2d68cf7ed11c751d0b48. * Set previewType * feat: Video Recording with Camera2 (#1691) * Rename * Update CameraSession.kt * Use `SurfaceHolder` instead of `SurfaceView` for output * Update CameraOutputs.kt * Update CameraSession.kt * fix: Fix crash when Preview is null * Check if snapshot capture is supported * Update RecordingSession.kt * S * Use `MediaRecorder` * Make audio optional * Add Torch * Output duration * Update RecordingSession.kt * Start RecordingSession * logs * More log * Base for preparing pass-through Recording * Use `ImageWriter` to append Images to the Recording Surface * Stream PRIVATE GPU_SAMPLED_IMAGE Images * Add flags * Close session on stop * Allow customizing `videoCodec` and `fileType` * Enable Torch * Fix Torch Mode * Fix comparing outputs with hashCode * Update CameraSession.kt * Correctly pass along Frame Processor * fix: Use AUDIO_BIT_RATE of 16 * 44,1Khz * Use CAMCORDER instead of MIC microphone * Use 1 channel * fix: Use `Orientation` * Add `native` PixelFormat * Update iOS to latest Skia integration * feat: Add `pixelFormat` property to Camera * Catch error in configureSession * Fix JPEG format * Clean up best match finder * Update CameraDeviceDetails.kt * Clamp sizes by maximum CamcorderProfile size * Remove `getAvailableVideoCodecs` * chore: release 3.0.0-rc.5 * Use maximum video size of RECORD as default * Update CameraDeviceDetails.kt * Add a todo * Add JSON device to issue report * Prefer `full` devices and flash * Lock to 30 FPS on Samsung * Implement Zoom * Refactor * Format -> PixelFormat * fix: Feat `pixelFormat` -> `pixelFormats` * Update TROUBLESHOOTING.mdx * Format * fix: Implement `zoom` for Photo Capture * fix: Don't run if `isActive` is `false` * fix: Call `examplePlugin(frame)` * fix: Fix Flash * fix: Use `react-native-worklets-core`! * fix: Fix import
2023-08-21 04:50:14 -06:00
// TODO: Test if this actually does any fusion or if this just calls the captureOutput twice. If the latter, remove it.
photoOutput!.isVirtualDeviceConstituentPhotoDeliveryEnabled = photoOutput!.isVirtualDeviceConstituentPhotoDeliverySupported
photoOutput!.maxPhotoQualityPrioritization = .quality
} else {
photoOutput!.isDualCameraDualPhotoDeliveryEnabled = photoOutput!.isDualCameraDualPhotoDeliverySupported
}
}
// TODO: Enable isResponsiveCaptureEnabled? (iOS 17+)
// TODO: Enable isFastCapturePrioritizationEnabled? (iOS 17+)
if enableDepthData {
photoOutput!.isDepthDataDeliveryEnabled = photoOutput!.isDepthDataDeliverySupported
}
if #available(iOS 12.0, *), enablePortraitEffectsMatteDelivery {
photoOutput!.isPortraitEffectsMatteDeliveryEnabled = photoOutput!.isPortraitEffectsMatteDeliverySupported
}
guard captureSession.canAddOutput(photoOutput!) else {
invokeOnError(.parameter(.unsupportedOutput(outputDescriptor: "photo-output")))
return
}
captureSession.addOutput(photoOutput!)
if videoDeviceInput!.device.position == .front {
photoOutput!.mirror()
}
}
2021-03-26 09:28:08 -06:00
feature: Frame Processors (iOS) (#2) * Clean up Frame Processor * Create FrameProcessorHolder * Create FrameProcessorDelegate in ObjC++ * Move frame processor to FrameProcessorDelegate * Decorate runtime, check for null * Update FrameProcessorDelegate.mm * Cleanup FrameProcessorBindings.mm * Fix RuntimeDecorator.h import * Update FrameProcessorDelegate.mm * "React" -> "React Helper" to avoid confusion * Rename folders again * Fix podspec flattening a lot of headers, causing REA nameclash * Fix header imports to avoid REA naming collision * Lazily initialize jsi::Runtime on DispatchQueue * Install frame processor bindings from Swift * First try to call jsi::Function (frame processor) :eyes: * Call viewForReactTag on RCT main thread * Fix bridge accessing * Add more logs * Update CameraViewManager.swift * Add more TODOs * Re-indent .cpp files * Fix RCTTurboModule import podspec * Remove unnecessary include check for swift umbrella header * Merge branch 'main' into frame-processors * Docs: use static width for images (283) * Create validate-cpp.yml * Update a lot of packages to latest * Set SWIFT_VERSION to 5.2 in podspec * Create clean.sh * Delete unused C++ files * podspec: Remove CLANG_CXX_LANGUAGE_STANDARD and OTHER_CFLAGS * Update pod lockfiles * Regenerate lockfiles * Remove IOSLogger * Use NSLog * Create FrameProcessorManager (inherits from REA RuntimeManager) * Create reanimated::RuntimeManager shared_ptr * Re-integrate pods * Add react-native-reanimated >=2 peerDependency * Add metro-config * blacklist -> exclusionList * Try to call worklet * Fix jsi::Value* initializer * Call ShareableValue::adapt (makeShareable) with React/JS Runtime * Add null-checks * Lift runtime manager creation out of delegate, into bindings * Remove debug statement * Make RuntimeManager unique_ptr * Set _FRAME_PROCESSOR * Extract convertJSIFunctionToFrameProcessorCallback * Print frame * Merge branch 'main' into frame-processors * Reformat Swift code * Install reanimated from npm again * Re-integrate Pods * Dependabot: Also scan example/ and docs/ * Update validate-cpp.yml * Create FrameProcessorUtils * Create Frame.h * Abstract HostObject creation away * Fix types * Fix frame processor call * Add todo * Update lockfiles * Add C++ contributing instructions * Update CONTRIBUTING.md * Add android/src/main/cpp to cpplint * Update cpplint.sh * Fix a few cpplint errors * Fix globals * Fix a few more cpplint errors * Update App.tsx * Update AndroidLogger.cpp * Format * Fix cpplint script (check-cpp) * Try to simplify frame processor * y * Update FrameProcessorUtils.mm * Update FrameProcessorBindings.mm * Update CameraView.swift * Update CameraViewManager.m * Restructure everything * fix * Fix `@objc` export (make public) * Refactor installFrameProcessorBindings into FrameProcessorRuntimeManager * Add swift RCTBridge.runOnJS helper * Fix run(onJS) * Add pragma once * Add `&self` to lambda * Update FrameProcessorRuntimeManager.mm * reorder imports * Fix imports * forward declare * Rename extension * Destroy buffer after execution * Add FrameProcessorPluginRegistry base * Merge branch 'main' into frame-processors * Add frameProcessor to types * Update Camera.tsx * Fix rebase merge * Remove movieOutput * Use `useFrameProcessor` * Fix bad merge * Add additional ESLint rules * Update lockfiles * Update CameraViewManager.m * Add support for V8 runtime * Add frame processor plugins API * Print plugin invoke * Fix React Utils in podspec * Fix runOnJS swift name * Remove invalid redecl of `captureSession` * Use REA 2.1.0 which includes all my big PRs :tada: * Update validate-cpp.yml * Update Podfile.lock * Remove Flipper * Fix dereferencing * Capture `self` by value. Fucking hell, what a dumb mistake. * Override a few HostObject functions * Expose isReady, width, height, bytesPerRow and planesCount * use hook again * Expose property names * FrameProcessor -> Frame * Update CameraView+RecordVideo.swift * Add Swift support for Frame Processors Plugins * Add macros for plugin installation * Add ObjC frame processor plugin * Correctly install frame processor plugins * Don't require custom name for macro * Check if plugin already exists * Implement QR Code Frame Processor Plugin in Swift * Adjust ObjC style frame processor macro * optimize * Add `frameProcessorFrameDropRate` * Fix types * Only log once * Log if it executes slowly * Implement `frameProcessorFps` * Implement manual encoded video recordings * Use recommended video settings * Add fileType types * Ignore if input is not ready for media data * Add completion handler * Add audio buffer sampling * Init only for video frame * use AVAssetWriterInputPixelBufferAdaptor * Remove AVAssetWriterInputPixelBufferAdaptor * Rotate VideoWriter * Always assume portrait orientation * Update RecordingSession.swift * Use a separate Queue for Audio * Format Swift * Update CameraView+RecordVideo.swift * Use `videoQueue` instead of `cameraQueue` * Move example plugins to example app * Fix hardcoded name in plugin macro * QRFrame... -> QRCodeFrame... * Update FrameProcessorPlugin.h * Add example frame processors to JS base * Update QRCodeFrameProcessorPluginSwift.m * Add docs to create FP Plugins * Update FRAME_PROCESSORS_CREATE.mdx * Update FRAME_PROCESSORS_CREATE.mdx * Use `AVAssetWriterInputPixelBufferAdaptor` for efficient pixel buffer recycling * Add customizable `pixelFormat` * Use native format if available * Update project.pbxproj * Set video width and height as source-pixel-buffer attributes * Catch * Update App.tsx * Don't explicitly set video dimensions, let CVPixelBufferPool handle it * Add a few logs * Cleanup * Update CameraView+RecordVideo.swift * Eagerly initialize asset writer to fix stutter at first frame * Use `cameraQueue` DispatchQueue to not block CaptureDataOutputDelegate * Fix duration calculation * cleanup * Cleanup * Swiftformat * Return available video codecs * Only show frame drop notification for video output * Remove photo and video codec functionality It was too much complexity and probably never used anyways. * Revert all android related changes for now * Cleanup * Remove unused header * Update AVAssetWriter.Status+descriptor.swift * Only call Frame Processor for Video Frames * Fix `if` * Add support for Frame Processor plugin parameters/arguments * Fix arg support * Move to JSIUtils.mm * Update JSIUtils.h * Update FRAME_PROCESSORS_CREATE.mdx * Update FRAME_PROCESSORS_CREATE.mdx * Upgrade packages for docs/ * fix docs * Rename * highlight lines * docs * community plugins * Update FRAME_PROCESSOR_CREATE_FINAL.mdx * Update FRAME_PROCESSOR_PLUGIN_LIST.mdx * Update FRAME_PROCESSOR_PLUGIN_LIST.mdx * Update dependencies (1/2) * Update dependencies (2/2) * Update Gemfile.lock * add FP docs * Update README.md * Make `lastFrameProcessor` private * add `frameProcessor` docs * fix docs * adjust docs * Update DEVICES.mdx * fix * s * Add logs demo * add metro restart note * Update FRAME_PROCESSOR_CREATE_PLUGIN_IOS.mdx * Mirror video device * Update AVCaptureVideoDataOutput+mirror.swift * Create .swift-version * Enable whole module optimization * Fix recording mirrored video * Swift format * Clean dictionary on `markInvalid` * Fix cleanup * Add docs for disabling frame processors * Update project.pbxproj * Revert "Update project.pbxproj" This reverts commit e67861e51b88b4888a6940e2d20388f3044211d0. * Log frame drop reason * Format * add more samples * Add clang-format * also check .mm * Revert "also check .mm" This reverts commit 8b9d5e2c29866b05909530d104f6633d6c49eadd. * Revert "Add clang-format" This reverts commit 7643ac808e0fc34567ea1f814e73d84955381636. * Use `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange` as default * Read matching video attributes from videoSettings * Add TODO * Swiftformat * Conditionally disable frame processors * Assert if trying to use frame processors when disabled * Add frame-processors demo gif * Allow disabling frame processors via `VISION_CAMERA_DISABLE_FRAME_PROCESSORS` * Update FrameProcessorRuntimeManager.mm * Update FRAME_PROCESSORS.mdx * Update project.pbxproj * Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
2021-05-06 06:11:55 -06:00
// Video Output + Frame Processor
2021-12-10 01:44:54 -07:00
if let videoOutput = videoOutput {
feature: Frame Processors (iOS) (#2) * Clean up Frame Processor * Create FrameProcessorHolder * Create FrameProcessorDelegate in ObjC++ * Move frame processor to FrameProcessorDelegate * Decorate runtime, check for null * Update FrameProcessorDelegate.mm * Cleanup FrameProcessorBindings.mm * Fix RuntimeDecorator.h import * Update FrameProcessorDelegate.mm * "React" -> "React Helper" to avoid confusion * Rename folders again * Fix podspec flattening a lot of headers, causing REA nameclash * Fix header imports to avoid REA naming collision * Lazily initialize jsi::Runtime on DispatchQueue * Install frame processor bindings from Swift * First try to call jsi::Function (frame processor) :eyes: * Call viewForReactTag on RCT main thread * Fix bridge accessing * Add more logs * Update CameraViewManager.swift * Add more TODOs * Re-indent .cpp files * Fix RCTTurboModule import podspec * Remove unnecessary include check for swift umbrella header * Merge branch 'main' into frame-processors * Docs: use static width for images (283) * Create validate-cpp.yml * Update a lot of packages to latest * Set SWIFT_VERSION to 5.2 in podspec * Create clean.sh * Delete unused C++ files * podspec: Remove CLANG_CXX_LANGUAGE_STANDARD and OTHER_CFLAGS * Update pod lockfiles * Regenerate lockfiles * Remove IOSLogger * Use NSLog * Create FrameProcessorManager (inherits from REA RuntimeManager) * Create reanimated::RuntimeManager shared_ptr * Re-integrate pods * Add react-native-reanimated >=2 peerDependency * Add metro-config * blacklist -> exclusionList * Try to call worklet * Fix jsi::Value* initializer * Call ShareableValue::adapt (makeShareable) with React/JS Runtime * Add null-checks * Lift runtime manager creation out of delegate, into bindings * Remove debug statement * Make RuntimeManager unique_ptr * Set _FRAME_PROCESSOR * Extract convertJSIFunctionToFrameProcessorCallback * Print frame * Merge branch 'main' into frame-processors * Reformat Swift code * Install reanimated from npm again * Re-integrate Pods * Dependabot: Also scan example/ and docs/ * Update validate-cpp.yml * Create FrameProcessorUtils * Create Frame.h * Abstract HostObject creation away * Fix types * Fix frame processor call * Add todo * Update lockfiles * Add C++ contributing instructions * Update CONTRIBUTING.md * Add android/src/main/cpp to cpplint * Update cpplint.sh * Fix a few cpplint errors * Fix globals * Fix a few more cpplint errors * Update App.tsx * Update AndroidLogger.cpp * Format * Fix cpplint script (check-cpp) * Try to simplify frame processor * y * Update FrameProcessorUtils.mm * Update FrameProcessorBindings.mm * Update CameraView.swift * Update CameraViewManager.m * Restructure everything * fix * Fix `@objc` export (make public) * Refactor installFrameProcessorBindings into FrameProcessorRuntimeManager * Add swift RCTBridge.runOnJS helper * Fix run(onJS) * Add pragma once * Add `&self` to lambda * Update FrameProcessorRuntimeManager.mm * reorder imports * Fix imports * forward declare * Rename extension * Destroy buffer after execution * Add FrameProcessorPluginRegistry base * Merge branch 'main' into frame-processors * Add frameProcessor to types * Update Camera.tsx * Fix rebase merge * Remove movieOutput * Use `useFrameProcessor` * Fix bad merge * Add additional ESLint rules * Update lockfiles * Update CameraViewManager.m * Add support for V8 runtime * Add frame processor plugins API * Print plugin invoke * Fix React Utils in podspec * Fix runOnJS swift name * Remove invalid redecl of `captureSession` * Use REA 2.1.0 which includes all my big PRs :tada: * Update validate-cpp.yml * Update Podfile.lock * Remove Flipper * Fix dereferencing * Capture `self` by value. Fucking hell, what a dumb mistake. * Override a few HostObject functions * Expose isReady, width, height, bytesPerRow and planesCount * use hook again * Expose property names * FrameProcessor -> Frame * Update CameraView+RecordVideo.swift * Add Swift support for Frame Processors Plugins * Add macros for plugin installation * Add ObjC frame processor plugin * Correctly install frame processor plugins * Don't require custom name for macro * Check if plugin already exists * Implement QR Code Frame Processor Plugin in Swift * Adjust ObjC style frame processor macro * optimize * Add `frameProcessorFrameDropRate` * Fix types * Only log once * Log if it executes slowly * Implement `frameProcessorFps` * Implement manual encoded video recordings * Use recommended video settings * Add fileType types * Ignore if input is not ready for media data * Add completion handler * Add audio buffer sampling * Init only for video frame * use AVAssetWriterInputPixelBufferAdaptor * Remove AVAssetWriterInputPixelBufferAdaptor * Rotate VideoWriter * Always assume portrait orientation * Update RecordingSession.swift * Use a separate Queue for Audio * Format Swift * Update CameraView+RecordVideo.swift * Use `videoQueue` instead of `cameraQueue` * Move example plugins to example app * Fix hardcoded name in plugin macro * QRFrame... -> QRCodeFrame... * Update FrameProcessorPlugin.h * Add example frame processors to JS base * Update QRCodeFrameProcessorPluginSwift.m * Add docs to create FP Plugins * Update FRAME_PROCESSORS_CREATE.mdx * Update FRAME_PROCESSORS_CREATE.mdx * Use `AVAssetWriterInputPixelBufferAdaptor` for efficient pixel buffer recycling * Add customizable `pixelFormat` * Use native format if available * Update project.pbxproj * Set video width and height as source-pixel-buffer attributes * Catch * Update App.tsx * Don't explicitly set video dimensions, let CVPixelBufferPool handle it * Add a few logs * Cleanup * Update CameraView+RecordVideo.swift * Eagerly initialize asset writer to fix stutter at first frame * Use `cameraQueue` DispatchQueue to not block CaptureDataOutputDelegate * Fix duration calculation * cleanup * Cleanup * Swiftformat * Return available video codecs * Only show frame drop notification for video output * Remove photo and video codec functionality It was too much complexity and probably never used anyways. * Revert all android related changes for now * Cleanup * Remove unused header * Update AVAssetWriter.Status+descriptor.swift * Only call Frame Processor for Video Frames * Fix `if` * Add support for Frame Processor plugin parameters/arguments * Fix arg support * Move to JSIUtils.mm * Update JSIUtils.h * Update FRAME_PROCESSORS_CREATE.mdx * Update FRAME_PROCESSORS_CREATE.mdx * Upgrade packages for docs/ * fix docs * Rename * highlight lines * docs * community plugins * Update FRAME_PROCESSOR_CREATE_FINAL.mdx * Update FRAME_PROCESSOR_PLUGIN_LIST.mdx * Update FRAME_PROCESSOR_PLUGIN_LIST.mdx * Update dependencies (1/2) * Update dependencies (2/2) * Update Gemfile.lock * add FP docs * Update README.md * Make `lastFrameProcessor` private * add `frameProcessor` docs * fix docs * adjust docs * Update DEVICES.mdx * fix * s * Add logs demo * add metro restart note * Update FRAME_PROCESSOR_CREATE_PLUGIN_IOS.mdx * Mirror video device * Update AVCaptureVideoDataOutput+mirror.swift * Create .swift-version * Enable whole module optimization * Fix recording mirrored video * Swift format * Clean dictionary on `markInvalid` * Fix cleanup * Add docs for disabling frame processors * Update project.pbxproj * Revert "Update project.pbxproj" This reverts commit e67861e51b88b4888a6940e2d20388f3044211d0. * Log frame drop reason * Format * add more samples * Add clang-format * also check .mm * Revert "also check .mm" This reverts commit 8b9d5e2c29866b05909530d104f6633d6c49eadd. * Revert "Add clang-format" This reverts commit 7643ac808e0fc34567ea1f814e73d84955381636. * Use `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange` as default * Read matching video attributes from videoSettings * Add TODO * Swiftformat * Conditionally disable frame processors * Assert if trying to use frame processors when disabled * Add frame-processors demo gif * Allow disabling frame processors via `VISION_CAMERA_DISABLE_FRAME_PROCESSORS` * Update FrameProcessorRuntimeManager.mm * Update FRAME_PROCESSORS.mdx * Update project.pbxproj * Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
2021-05-06 06:11:55 -06:00
captureSession.removeOutput(videoOutput)
self.videoOutput = nil
}
if video?.boolValue == true || enableFrameProcessor {
ReactLogger.log(level: .info, message: "Adding Video Data output...")
videoOutput = AVCaptureVideoDataOutput()
guard captureSession.canAddOutput(videoOutput!) else {
invokeOnError(.parameter(.unsupportedOutput(outputDescriptor: "video-output")))
return
}
feat: Complete iOS Codebase rewrite (#1647) * Make Frame Processors an extra subspec * Update VisionCamera.podspec * Make optional * Make VisionCamera compile without Skia * Fix * Add skia again * Update VisionCamera.podspec * Make VisionCamera build without Frame Processors * Rename error to `system/frame-processors-unavailable` * Fix Frame Processor returning early * Remove `preset`, FP partial rewrite * Only warn on frame drop * Fix wrong queue * fix: Run on CameraQueue again * Update CameraView.swift * fix: Activate audio session asynchronously on audio queue * Update CameraView+RecordVideo.swift * Update PreviewView.h * Cleanups * Cleanup * fix cast * feat: Add LiDAR Depth Camera support * Upgrade Ruby * Add vector icons type * Update Gemfile.lock * fix: Stop queues on deinit * Also load `builtInTrueDepthCamera` * Update CameraViewManager.swift * Update SkImageHelpers.mm * Extract FrameProcessorCallback to FrameProcessor Holds more context now :) * Rename to .m * fix: Add `RCTLog` import * Create SkiaFrameProcessor * Update CameraBridge.h * Call Frame Processor * Fix defines * fix: Allow deleting callback funcs * fix Skia build * batch * Just call `setSkiaFrameProcessor` * Rewrite in Swift * Pass `SkiaRenderer` * Fix Import * Move `PreviewView` to Swift * Fix Layer * Set Skia Canvas to Frame Host Object * Make `DrawableFrameHostObject` subclass * Fix TS types * Use same MTLDevice and apply scale * Make getter * Extract `setTorch` and `Preview` * fix: Fix nil metal device * Don't wait for session stop in deinit * Use main pixel ratio * Use unique_ptr for Render Contexts * fix: Fix SkiaPreviewDisplayLink broken after deinit * inline `getTextureCache` * Update CameraPage.tsx * chore: Format iOS * perf: Allow MTLLayer to be optimized for only frame buffers * Add RN Video types * fix: Fix Frame Processors if guard * Find nodeModules recursively * Create `Frame.isDrawable` * Add `cocoapods-check` dependency
2023-07-20 07:30:04 -06:00
videoOutput!.setSampleBufferDelegate(self, queue: CameraQueues.videoQueue)
videoOutput!.alwaysDiscardsLateVideoFrames = false
feat: Draw onto `Frame` as if it was a Skia Canvas (#1479) * Create Shaders.ts * Add `previewType` and `enableFpsGraph` * Add RN Skia native dependency * Add Skia Preview View on iOS * Pass 1 * Update FrameHostObject.mm * Wrap Canvas * Lockfiles * fix: Fix stuff * chore: Upgrade RNWorklets * Add `previewType` to set the Preview * feat: Add Example * Update project.pbxproj * `enableFpsGraph` * Cache the `std::shared_ptr<FrameHostObject>` * Update CameraView+RecordVideo.swift * Update SkiaMetalCanvasProvider.mm * Android: Integrate Skia Dependency * fix: Use new Prefix * Add example for rendering shader * chore: Upgrade CameraX * Remove KTX * Enable `viewBinding` * Revert "Enable `viewBinding`" This reverts commit f2a603f53b33ea4311a296422ffd1a910ce03f9e. * Revert "chore: Upgrade CameraX" This reverts commit 8dc832cf8754490d31a6192e6c1a1f11cdcd94fe. * Remove unneeded `ProcessCameraProvider.getInstance()` call * fix: Add REA hotfix patch * fix: Fix FrameHostObject dead in runAsync * fix: Make `runAsync` run truly async by dropping new Frames while executing * chore: Upgrade RN Worklets to latest * chore: Upgrade RN Skia * Revert "Remove KTX" This reverts commit 253f586633f7af2da992d2279fc206dc62597129. * Make Skia optional in CMake * Fix import * Update CMakeLists.txt * Update build.gradle * Update CameraView.kt * Update CameraView.kt * Update CameraView.kt * Update Shaders.ts * Center Blur * chore: Upgrade RN Worklets * feat: Add `toByteArray()`, `orientation`, `isMirrored` and `timestamp` to `Frame` (#1487) * feat: Implement `orientation` and `isMirrored` on Frame * feat: Add `toArrayBuffer()` func * perf: Do faster buffer copy * feat: Implement `toArrayBuffer()` on Android * feat: Add `orientation` and `isMirrored` to Android * feat: Add `timestamp` to Frame * Update Frame.ts * Update JImageProxy.h * Update FrameHostObject.cpp * Update FrameHostObject.cpp * Update CameraPage.tsx * fix: Format Swift
2023-02-21 07:00:48 -07:00
let pixelFormatType = getPixelFormat(videoOutput: videoOutput!)
videoOutput!.videoSettings = [
String(kCVPixelBufferPixelFormatTypeKey): pixelFormatType,
]
captureSession.addOutput(videoOutput!)
}
2021-03-26 09:28:08 -06:00
// Code Scanner
if let codeScannerOptions = codeScannerOptions {
guard let codeScanner = try? CodeScanner(fromJsValue: codeScannerOptions) else {
invokeOnError(.parameter(.invalid(unionName: "codeScanner", receivedValue: codeScannerOptions.description)))
return
}
let metadataOutput = AVCaptureMetadataOutput()
guard captureSession.canAddOutput(metadataOutput) else {
invokeOnError(.codeScanner(.notCompatibleWithOutputs))
return
}
captureSession.addOutput(metadataOutput)
for codeType in codeScanner.codeTypes {
// swiftlint:disable:next for_where
if !metadataOutput.availableMetadataObjectTypes.contains(codeType) {
invokeOnError(.codeScanner(.codeTypeNotSupported(codeType: codeType.descriptor)))
return
}
}
metadataOutput.setMetadataObjectsDelegate(self, queue: CameraQueues.codeScannerQueue)
metadataOutput.metadataObjectTypes = codeScanner.codeTypes
if let rectOfInterest = codeScanner.regionOfInterest {
metadataOutput.rectOfInterest = rectOfInterest
}
}
if outputOrientation != .portrait {
updateOrientation()
}
invokeOnInitialized()
isReady = true
ReactLogger.log(level: .info, message: "Session successfully configured!")
}
2021-03-26 09:28:08 -06:00
/**
Returns the pixel format that should be used for the AVCaptureVideoDataOutput.
If HDR is enabled, this will return YUV 4:2:0 10-bit.
If HDR is disabled, this will return whatever the user specified as a pixelFormat, or the most efficient format as a fallback.
*/
private func getPixelFormat(videoOutput: AVCaptureVideoDataOutput) -> OSType {
// as per documentation, the first value is always the most efficient format
var defaultFormat = videoOutput.availableVideoPixelFormatTypes.first!
if enableBufferCompression {
// use compressed format instead if we enabled buffer compression
if defaultFormat == kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange &&
videoOutput.availableVideoPixelFormatTypes.contains(kCVPixelFormatType_Lossless_420YpCbCr8BiPlanarVideoRange) {
// YUV 4:2:0 8-bit (limited video colors; compressed)
defaultFormat = kCVPixelFormatType_Lossless_420YpCbCr8BiPlanarVideoRange
}
if defaultFormat == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange &&
videoOutput.availableVideoPixelFormatTypes.contains(kCVPixelFormatType_Lossless_420YpCbCr8BiPlanarFullRange) {
// YUV 4:2:0 8-bit (full video colors; compressed)
defaultFormat = kCVPixelFormatType_Lossless_420YpCbCr8BiPlanarFullRange
}
}
// If the user enabled HDR, we can only use the YUV 4:2:0 10-bit pixel format.
if hdr == true {
guard pixelFormat == nil || pixelFormat == "yuv" else {
invokeOnError(.format(.incompatiblePixelFormatWithHDR))
return defaultFormat
}
var targetFormats = [kCVPixelFormatType_420YpCbCr10BiPlanarFullRange,
kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange]
if enableBufferCompression {
// If we enable buffer compression, try to use a lossless compressed YUV format first, otherwise fall back to the others.
targetFormats.insert(kCVPixelFormatType_Lossless_420YpCbCr10PackedBiPlanarVideoRange, at: 0)
}
// Find the best matching format
guard let format = videoOutput.findPixelFormat(firstOf: targetFormats) else {
invokeOnError(.format(.invalidHdr))
return defaultFormat
}
// YUV 4:2:0 10-bit (compressed/uncompressed)
return format
}
// If the user didn't specify a custom pixelFormat, just return the default one.
guard let pixelFormat = pixelFormat else {
return defaultFormat
}
// If we don't use HDR, we can use any other custom pixel format.
switch pixelFormat {
case "yuv":
// YUV 4:2:0 8-bit (full/limited video colors; uncompressed)
var targetFormats = [kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange]
if enableBufferCompression {
// YUV 4:2:0 8-bit (full/limited video colors; compressed)
targetFormats.insert(kCVPixelFormatType_Lossless_420YpCbCr8BiPlanarVideoRange, at: 0)
targetFormats.insert(kCVPixelFormatType_Lossless_420YpCbCr8BiPlanarFullRange, at: 0)
}
guard let format = videoOutput.findPixelFormat(firstOf: targetFormats) else {
invokeOnError(.device(.pixelFormatNotSupported))
return defaultFormat
}
return format
case "rgb":
// RGBA 8-bit (uncompressed)
var targetFormats = [kCVPixelFormatType_32BGRA]
if enableBufferCompression {
// RGBA 8-bit (compressed)
targetFormats.insert(kCVPixelFormatType_Lossless_32BGRA, at: 0)
}
guard let format = videoOutput.findPixelFormat(firstOf: targetFormats) else {
invokeOnError(.device(.pixelFormatNotSupported))
return defaultFormat
}
return format
case "native":
return defaultFormat
default:
invokeOnError(.parameter(.invalid(unionName: "pixelFormat", receivedValue: pixelFormat as String)))
return defaultFormat
}
}
fix: Move Audio Input initialization shortly before `startRecording` (#159) * rename * Update AVAudioSession+updateCategory.swift * fix bootstrap script * Update CameraView+AVAudioSession.swift * move audio input adding lower * Activate AudioSession only when starting recording * format * Deactivate Audio Session * remove audio input before deactivating audio session * Update CameraView+AVAudioSession.swift * log time * Update CameraView+AVAudioSession.swift * measure time with `measureElapsedTime` * Update project.pbxproj * only log in debug builds * bootstrap with bridge (RNN new API) * Mark two funcs as `@inlinable` * format * Update ReactLogger.swift * Make audioWriter optional (allow videos without sound) * only log frame drop reason in DEBUG * Make audio writing entirely optional * format * Use function name as label for measureElapsedTime * Update MeasureElapsedTime.swift * Update MeasureElapsedTime.swift * Mark AudioWriter as finished * set `automaticallyConfiguresApplicationAudioSession` once * Add JS console logging * log to JS console for a few logs * Update AVAudioSession+updateCategory.swift * format * Update JSConsoleHelper.mm * catch log errors * Update ReactLogger.swift * fix docs * Update RecordingSession.swift * Immediatelly add audio input * Update CameraView+AVCaptureSession.swift * Update CameraView+AVCaptureSession.swift * Update ReactLogger.swift * immediatelly set audio session * extract * format * Update TROUBLESHOOTING.mdx * hmm * Update AVAudioSession+updateCategory.swift * Create secondary `AVCaptureSession` for audio * Configure once, start stop on demand * format * fix audio notification interruptions * docs
2021-06-03 06:16:02 -06:00
// pragma MARK: Configure Device
/**
feat: Full Android rewrite (CameraX -> Camera2) (#1674) * Nuke CameraX * fix: Run View Finder on UI Thread * Open Camera, set up Threads * fix init * Mirror if needed * Try PreviewView * Use max resolution * Add `hardwareLevel` property * Check if output type is supported * Replace `frameRateRanges` with `minFps` and `maxFps` * Remove `isHighestPhotoQualitySupported` * Remove `colorSpace` The native platforms will use the best / most accurate colorSpace by default anyways. * HDR * Check from format * fix * Remove `supportsParallelVideoProcessing` * Correctly return video/photo sizes on Android now. Finally * Log all Device props * Log if optimized usecase is used * Cleanup * Configure Camera Input only once * Revert "Configure Camera Input only once" This reverts commit 0fd6c03f54c7566cb5592053720c4a8743aba92e. * Extract Camera configuration * Try to reconfigure all * Hook based * Properly set up `CameraSession` * Delete unused * fix: Fix recreate when outputs change * Update NativePreviewView.kt * Use callback for closing * Catch CameraAccessException * Finally got it stable * Remove isMirrored * Implement `takePhoto()` * Add ExifInterface library * Run findViewById on UI Thread * Add Photo Output Surface to takePhoto * Fix Video Stabilization Modes * Optimize Imports * More logs * Update CameraSession.kt * Close Image * Use separate Executor in CameraQueue * Delete hooks * Use same Thread again * If opened, call error * Update CameraSession.kt * Log HW level * fix: Don't enable Stream Use Case if it's not 100% supported * Move some stuff * Cleanup PhotoOutputSynchronizer * Try just open in suspend fun * Some synchronization fixes * fix logs * Update CameraDevice+createCaptureSession.kt * Update CameraDevice+createCaptureSession.kt * fixes * fix: Use Snapshot Template for speed capture prio * Use PREVIEW template for repeating request * Use `TEMPLATE_RECORD` if video use-case is attached * Use `isRunning` flag * Recreate session everytime on active/inactive * Lazily get values in capture session * Stability * Rebuild session if outputs change * Set `didOutputsChange` back to false * Capture first in lock * Try * kinda fix it? idk * fix: Keep Outputs * Refactor into single method * Update CameraView.kt * Use Enums for type safety * Implement Orientation (I think) * Move RefCount management to Java (Frame) * Don't crash when dropping a Frame * Prefer Devices with higher max resolution * Prefer multi-cams * Use FastImage for Media Page * Return orientation in takePhoto() * Load orientation from EXIF Data * Add `isMirrored` props and documentation for PhotoFile * fix: Return `not-determined` on Android * Update CameraViewModule.kt * chore: Upgrade packages * fix: Fix Metro Config * Cleanup config * Properly mirror Images on save * Prepare MediaRecorder * Start/Stop MediaRecorder * Remove `takeSnapshot()` It no longer works on Android and never worked on iOS. Users could use useFrameProcessor to take a Snapshot * Use `MediaCodec` * Move to `VideoRecording` class * Cleanup Snapshot * Create `SkiaPreviewView` hybrid class * Create OpenGL context * Create `SkiaPreviewView` * Fix texture creation missing context * Draw red frame * Somehow get it working * Add Skia CMake setup * Start looping * Init OpenGL * Refactor into `SkiaRenderer` * Cleanup PreviewSize * Set up * Only re-render UI if there is a new Frame * Preview * Fix init * Try rendering Preview * Update SkiaPreviewView.kt * Log version * Try using Skia (fail) * Drawwwww!!!!!!!!!! :tada: * Use Preview Size * Clear first * Refactor into SkiaRenderer * Add `previewType: "none"` on iOS * Simplify a lot * Draw Camera? For some reason? I have no idea anymore * Fix OpenGL errors * Got it kinda working again? * Actually draw Frame woah * Clean up code * Cleanup * Update on main * Synchronize render calls * holy shit * Update SkiaRenderer.cpp * Update SkiaRenderer.cpp * Refactor * Update SkiaRenderer.cpp * Check for `NO_INPUT_TEXTURE`^ * Post & Wait * Set input size * Add Video back again * Allow session without preview * Convert JPEG to byte[] * feat: Use `ImageReader` and use YUV Image Buffers in Skia Context (#1689) * Try to pass YUV Buffers as Pixmaps * Create pixmap! * Clean up * Render to preview * Only render if we have an output surface * Update SkiaRenderer.cpp * Fix Y+U+V sampling code * Cleanup * Fix Semaphore 0 * Use 4:2:0 YUV again idk * Update SkiaRenderer.h * Set minSdk to 26 * Set surface * Revert "Set minSdk to 26" This reverts commit c4085b7c16c628532e5c2d68cf7ed11c751d0b48. * Set previewType * feat: Video Recording with Camera2 (#1691) * Rename * Update CameraSession.kt * Use `SurfaceHolder` instead of `SurfaceView` for output * Update CameraOutputs.kt * Update CameraSession.kt * fix: Fix crash when Preview is null * Check if snapshot capture is supported * Update RecordingSession.kt * S * Use `MediaRecorder` * Make audio optional * Add Torch * Output duration * Update RecordingSession.kt * Start RecordingSession * logs * More log * Base for preparing pass-through Recording * Use `ImageWriter` to append Images to the Recording Surface * Stream PRIVATE GPU_SAMPLED_IMAGE Images * Add flags * Close session on stop * Allow customizing `videoCodec` and `fileType` * Enable Torch * Fix Torch Mode * Fix comparing outputs with hashCode * Update CameraSession.kt * Correctly pass along Frame Processor * fix: Use AUDIO_BIT_RATE of 16 * 44,1Khz * Use CAMCORDER instead of MIC microphone * Use 1 channel * fix: Use `Orientation` * Add `native` PixelFormat * Update iOS to latest Skia integration * feat: Add `pixelFormat` property to Camera * Catch error in configureSession * Fix JPEG format * Clean up best match finder * Update CameraDeviceDetails.kt * Clamp sizes by maximum CamcorderProfile size * Remove `getAvailableVideoCodecs` * chore: release 3.0.0-rc.5 * Use maximum video size of RECORD as default * Update CameraDeviceDetails.kt * Add a todo * Add JSON device to issue report * Prefer `full` devices and flash * Lock to 30 FPS on Samsung * Implement Zoom * Refactor * Format -> PixelFormat * fix: Feat `pixelFormat` -> `pixelFormats` * Update TROUBLESHOOTING.mdx * Format * fix: Implement `zoom` for Photo Capture * fix: Don't run if `isActive` is `false` * fix: Call `examplePlugin(frame)` * fix: Fix Flash * fix: Use `react-native-worklets-core`! * fix: Fix import
2023-08-21 04:50:14 -06:00
Configures the Video Device with the given FPS and HDR modes.
*/
2021-03-26 09:28:08 -06:00
final func configureDevice() {
ReactLogger.log(level: .info, message: "Configuring Device...")
guard let device = videoDeviceInput?.device else {
invokeOnError(.session(.cameraNotReady))
return
}
2021-03-26 09:28:08 -06:00
do {
try device.lockForConfiguration()
2021-03-26 09:28:08 -06:00
// Configure FPS
2021-12-10 01:44:54 -07:00
if let fps = fps?.int32Value {
let supportsGivenFps = device.activeFormat.videoSupportedFrameRateRanges.contains { range in
return range.includes(fps: Double(fps))
}
if !supportsGivenFps {
invokeOnError(.format(.invalidFps(fps: Int(fps))))
return
}
let duration = CMTimeMake(value: 1, timescale: fps)
device.activeVideoMinFrameDuration = duration
device.activeVideoMaxFrameDuration = duration
} else {
device.activeVideoMinFrameDuration = CMTime.invalid
device.activeVideoMaxFrameDuration = CMTime.invalid
}
// Configure Low-Light-Boost
if lowLightBoost != nil {
if lowLightBoost == true && !device.isLowLightBoostSupported {
invokeOnError(.device(.lowLightBoostNotSupported))
return
}
device.automaticallyEnablesLowLightBoostWhenAvailable = lowLightBoost!.boolValue
}
2021-03-26 09:28:08 -06:00
device.unlockForConfiguration()
ReactLogger.log(level: .info, message: "Device successfully configured!")
} catch let error as NSError {
invokeOnError(.device(.configureError), cause: error)
return
}
}
2021-03-26 09:28:08 -06:00
fix: Move Audio Input initialization shortly before `startRecording` (#159) * rename * Update AVAudioSession+updateCategory.swift * fix bootstrap script * Update CameraView+AVAudioSession.swift * move audio input adding lower * Activate AudioSession only when starting recording * format * Deactivate Audio Session * remove audio input before deactivating audio session * Update CameraView+AVAudioSession.swift * log time * Update CameraView+AVAudioSession.swift * measure time with `measureElapsedTime` * Update project.pbxproj * only log in debug builds * bootstrap with bridge (RNN new API) * Mark two funcs as `@inlinable` * format * Update ReactLogger.swift * Make audioWriter optional (allow videos without sound) * only log frame drop reason in DEBUG * Make audio writing entirely optional * format * Use function name as label for measureElapsedTime * Update MeasureElapsedTime.swift * Update MeasureElapsedTime.swift * Mark AudioWriter as finished * set `automaticallyConfiguresApplicationAudioSession` once * Add JS console logging * log to JS console for a few logs * Update AVAudioSession+updateCategory.swift * format * Update JSConsoleHelper.mm * catch log errors * Update ReactLogger.swift * fix docs * Update RecordingSession.swift * Immediatelly add audio input * Update CameraView+AVCaptureSession.swift * Update CameraView+AVCaptureSession.swift * Update ReactLogger.swift * immediatelly set audio session * extract * format * Update TROUBLESHOOTING.mdx * hmm * Update AVAudioSession+updateCategory.swift * Create secondary `AVCaptureSession` for audio * Configure once, start stop on demand * format * fix audio notification interruptions * docs
2021-06-03 06:16:02 -06:00
// pragma MARK: Configure Format
2021-03-26 09:28:08 -06:00
/**
Configures the Video Device to find the best matching Format.
*/
final func configureFormat() {
ReactLogger.log(level: .info, message: "Configuring Format...")
guard let jsFormat = format else {
// JS Format was null. Ignore it, use default.
2021-03-26 09:28:08 -06:00
return
}
guard let device = videoDeviceInput?.device else {
invokeOnError(.session(.cameraNotReady))
return
2021-03-26 09:28:08 -06:00
}
if device.activeFormat.isEqualTo(jsFormat: jsFormat) {
ReactLogger.log(level: .info, message: "Already selected active format.")
2021-03-26 09:28:08 -06:00
return
}
// get matching format
let format = device.formats.first { $0.isEqualTo(jsFormat: jsFormat) }
guard let format else {
invokeOnError(.format(.invalidFormat))
return
2021-03-26 09:28:08 -06:00
}
do {
try device.lockForConfiguration()
defer {
device.unlockForConfiguration()
}
let shouldReconfigurePhotoOutput = device.activeFormat.photoDimensions.toCGSize() != format.photoDimensions.toCGSize()
2021-03-26 09:28:08 -06:00
device.activeFormat = format
// The Photo Output uses the smallest available Dimension by default. We need to configure it for the maximum here
if shouldReconfigurePhotoOutput, #available(iOS 16.0, *) {
if let photoOutput = photoOutput {
photoOutput.maxPhotoDimensions = format.photoDimensions
}
}
ReactLogger.log(level: .info, message: "Format successfully configured!")
2021-03-26 09:28:08 -06:00
} catch let error as NSError {
invokeOnError(.device(.configureError), cause: error)
return
2021-03-26 09:28:08 -06:00
}
}
fix: Move Audio Input initialization shortly before `startRecording` (#159) * rename * Update AVAudioSession+updateCategory.swift * fix bootstrap script * Update CameraView+AVAudioSession.swift * move audio input adding lower * Activate AudioSession only when starting recording * format * Deactivate Audio Session * remove audio input before deactivating audio session * Update CameraView+AVAudioSession.swift * log time * Update CameraView+AVAudioSession.swift * measure time with `measureElapsedTime` * Update project.pbxproj * only log in debug builds * bootstrap with bridge (RNN new API) * Mark two funcs as `@inlinable` * format * Update ReactLogger.swift * Make audioWriter optional (allow videos without sound) * only log frame drop reason in DEBUG * Make audio writing entirely optional * format * Use function name as label for measureElapsedTime * Update MeasureElapsedTime.swift * Update MeasureElapsedTime.swift * Mark AudioWriter as finished * set `automaticallyConfiguresApplicationAudioSession` once * Add JS console logging * log to JS console for a few logs * Update AVAudioSession+updateCategory.swift * format * Update JSConsoleHelper.mm * catch log errors * Update ReactLogger.swift * fix docs * Update RecordingSession.swift * Immediatelly add audio input * Update CameraView+AVCaptureSession.swift * Update CameraView+AVCaptureSession.swift * Update ReactLogger.swift * immediatelly set audio session * extract * format * Update TROUBLESHOOTING.mdx * hmm * Update AVAudioSession+updateCategory.swift * Create secondary `AVCaptureSession` for audio * Configure once, start stop on demand * format * fix audio notification interruptions * docs
2021-06-03 06:16:02 -06:00
// pragma MARK: Notifications/Interruptions
2021-03-26 09:28:08 -06:00
@objc
func sessionRuntimeError(notification: Notification) {
ReactLogger.log(level: .error, message: "Unexpected Camera Runtime Error occured!")
2021-03-26 09:28:08 -06:00
guard let error = notification.userInfo?[AVCaptureSessionErrorKey] as? AVError else {
return
}
invokeOnError(.unknown(message: error._nsError.description), cause: error._nsError)
if isActive {
// restart capture session after an error occured
feat: Complete iOS Codebase rewrite (#1647) * Make Frame Processors an extra subspec * Update VisionCamera.podspec * Make optional * Make VisionCamera compile without Skia * Fix * Add skia again * Update VisionCamera.podspec * Make VisionCamera build without Frame Processors * Rename error to `system/frame-processors-unavailable` * Fix Frame Processor returning early * Remove `preset`, FP partial rewrite * Only warn on frame drop * Fix wrong queue * fix: Run on CameraQueue again * Update CameraView.swift * fix: Activate audio session asynchronously on audio queue * Update CameraView+RecordVideo.swift * Update PreviewView.h * Cleanups * Cleanup * fix cast * feat: Add LiDAR Depth Camera support * Upgrade Ruby * Add vector icons type * Update Gemfile.lock * fix: Stop queues on deinit * Also load `builtInTrueDepthCamera` * Update CameraViewManager.swift * Update SkImageHelpers.mm * Extract FrameProcessorCallback to FrameProcessor Holds more context now :) * Rename to .m * fix: Add `RCTLog` import * Create SkiaFrameProcessor * Update CameraBridge.h * Call Frame Processor * Fix defines * fix: Allow deleting callback funcs * fix Skia build * batch * Just call `setSkiaFrameProcessor` * Rewrite in Swift * Pass `SkiaRenderer` * Fix Import * Move `PreviewView` to Swift * Fix Layer * Set Skia Canvas to Frame Host Object * Make `DrawableFrameHostObject` subclass * Fix TS types * Use same MTLDevice and apply scale * Make getter * Extract `setTorch` and `Preview` * fix: Fix nil metal device * Don't wait for session stop in deinit * Use main pixel ratio * Use unique_ptr for Render Contexts * fix: Fix SkiaPreviewDisplayLink broken after deinit * inline `getTextureCache` * Update CameraPage.tsx * chore: Format iOS * perf: Allow MTLLayer to be optimized for only frame buffers * Add RN Video types * fix: Fix Frame Processors if guard * Find nodeModules recursively * Create `Frame.isDrawable` * Add `cocoapods-check` dependency
2023-07-20 07:30:04 -06:00
CameraQueues.cameraQueue.async {
2021-03-26 09:28:08 -06:00
self.captureSession.startRunning()
}
}
}
}