feat: Split videoHdr and photoHdr into two settings (#2161)

* feat: Split `videoHdr` and `photoHdr` into two settings

* fix: Rename all `hdr`

* fix: Fix HDR on Android

* Update CameraDeviceDetails.kt

* Update CameraDeviceDetails.kt

* fix: Correctly configure `pixelFormat` AFTER `format`

* Update CameraSession+Configuration.swift

* fix: Also after format changed
This commit is contained in:
Marc Rousavy 2023-11-15 18:33:12 +01:00 committed by GitHub
parent 75fd924899
commit c5dfb6c247
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
26 changed files with 129 additions and 88 deletions

View File

@ -42,8 +42,8 @@ To get all available formats, simply use the `CameraDevice`'s [`formats` propert
- [`minFps`](/docs/api/interfaces/CameraDeviceFormat#minfps)/[`maxFps`](/docs/api/interfaces/CameraDeviceFormat#maxfps): A range of possible values for the `fps` property. For example, if your format has `minFps: 1` and `maxFps: 60`, you can either use `fps={30}`, `fps={60}` or any other value in between for recording videos. - [`minFps`](/docs/api/interfaces/CameraDeviceFormat#minfps)/[`maxFps`](/docs/api/interfaces/CameraDeviceFormat#maxfps): A range of possible values for the `fps` property. For example, if your format has `minFps: 1` and `maxFps: 60`, you can either use `fps={30}`, `fps={60}` or any other value in between for recording videos.
- [`videoStabilizationModes`](/docs/api/interfaces/CameraDeviceFormat#videostabilizationmodes): All supported Video Stabilization Modes, digital and optical. If this specific format contains your desired [`VideoStabilizationMode`](/docs/api/#videostabilizationmode), you can pass it to your `<Camera>` via the [`videoStabilizationMode` property](/docs/api/interfaces/CameraProps#videoStabilizationMode). - [`videoStabilizationModes`](/docs/api/interfaces/CameraDeviceFormat#videostabilizationmodes): All supported Video Stabilization Modes, digital and optical. If this specific format contains your desired [`VideoStabilizationMode`](/docs/api/#videostabilizationmode), you can pass it to your `<Camera>` via the [`videoStabilizationMode` property](/docs/api/interfaces/CameraProps#videoStabilizationMode).
- [`pixelFormats`](/docs/api/interfaces/CameraDeviceFormat#pixelformats): All supported Pixel Formats. If this specific format contains your desired [`PixelFormat`](/docs/api/#PixelFormat), you can pass it to your `<Camera>` via the [`pixelFormat` property](/docs/api/interfaces/CameraProps#pixelFormat). - [`pixelFormats`](/docs/api/interfaces/CameraDeviceFormat#pixelformats): All supported Pixel Formats. If this specific format contains your desired [`PixelFormat`](/docs/api/#PixelFormat), you can pass it to your `<Camera>` via the [`pixelFormat` property](/docs/api/interfaces/CameraProps#pixelFormat).
- [`supportsVideoHDR`](/docs/api/interfaces/CameraDeviceFormat#supportsvideohdr): Whether this specific format supports true 10-bit HDR for video capture. If this is `true`, you can enable `hdr` on your `<Camera>`. - [`supportsVideoHdr`](/docs/api/interfaces/CameraDeviceFormat#supportsvideohdr): Whether this specific format supports true 10-bit HDR for video capture. If this is `true`, you can enable `videoHdr` on your `<Camera>`.
- [`supportsPhotoHDR`](/docs/api/interfaces/CameraDeviceFormat#supportsphotohdr): Whether this specific format supports HDR for photo capture. It will use multiple captures to fuse over-exposed and under-exposed Images together to form one HDR photo. If this is `true`, you can enable `hdr` on your `<Camera>`. - [`supportsPhotoHdr`](/docs/api/interfaces/CameraDeviceFormat#supportsphotohdr): Whether this specific format supports HDR for photo capture. It will use multiple captures to fuse over-exposed and under-exposed Images together to form one HDR photo. If this is `true`, you can enable `photoHdr` on your `<Camera>`.
- [`supportsDepthCapture`](/docs/api/interfaces/CameraDeviceFormat#supportsdepthcapture): Whether this specific format supports depth data capture. For devices like the TrueDepth/LiDAR cameras, this will always be true. - [`supportsDepthCapture`](/docs/api/interfaces/CameraDeviceFormat#supportsdepthcapture): Whether this specific format supports depth data capture. For devices like the TrueDepth/LiDAR cameras, this will always be true.
- ...and more. See the [`CameraDeviceFormat` type](/docs/api/interfaces/CameraDeviceFormat) for all supported properties. - ...and more. See the [`CameraDeviceFormat` type](/docs/api/interfaces/CameraDeviceFormat) for all supported properties.
@ -201,7 +201,8 @@ function App() {
Other props that depend on the `format`: Other props that depend on the `format`:
* `fps`: Specifies the frame rate to use * `fps`: Specifies the frame rate to use
* `hdr`: Enables HDR photo or video capture and preview * `videoHdr`: Enables HDR video capture and preview
* `photoHdr`: Enables HDR photo capture
* `lowLightBoost`: Enables a night-mode/low-light-boost for photo or video capture and preview * `lowLightBoost`: Enables a night-mode/low-light-boost for photo or video capture and preview
* `videoStabilizationMode`: Specifies the video stabilization mode to use for the video pipeline * `videoStabilizationMode`: Specifies the video stabilization mode to use for the video pipeline
* `pixelFormat`: Specifies the pixel format to use for the video pipeline * `pixelFormat`: Specifies the pixel format to use for the video pipeline

View File

@ -239,7 +239,7 @@ When running frame processors, it is often important to choose an appropriate [f
* If you are running heavy AI/ML calculations in your frame processor, make sure to [select a format](/docs/guides/formats) that has a lower resolution to optimize it's performance. You can also resize the Frame on-demand. * If you are running heavy AI/ML calculations in your frame processor, make sure to [select a format](/docs/guides/formats) that has a lower resolution to optimize it's performance. You can also resize the Frame on-demand.
* Sometimes a frame processor plugin only works with specific [pixel formats](/docs/api/interfaces/CameraDeviceFormat#pixelformats). Some plugins (like Tensorflow Lite Models) don't work with `yuv`, so use a [`pixelFormat`](/docs/api/interfaces/CameraProps#pixelformat) of `rgb` instead. * Sometimes a frame processor plugin only works with specific [pixel formats](/docs/api/interfaces/CameraDeviceFormat#pixelformats). Some plugins (like Tensorflow Lite Models) don't work with `yuv`, so use a [`pixelFormat`](/docs/api/interfaces/CameraProps#pixelformat) of `rgb` instead.
* Some Frame Processor plugins don't work with HDR formats. In this case you need to disable [`hdr`](/docs/api/interfaces/CameraProps#hdr). * Some Frame Processor plugins don't work with HDR formats. In this case you need to disable [`videoHdr`](/docs/api/interfaces/CameraProps#videohdr).
## Benchmarks ## Benchmarks

View File

@ -49,8 +49,8 @@ To enable HDR capture, you need to select a format (see ["Camera Formats"](forma
```ts ```ts
const format = useCameraFormat(device, [ const format = useCameraFormat(device, [
{ photoHDR: true }, { photoHdr: true },
{ videoHDR: true }, { videoHdr: true },
]) ])
``` ```
@ -59,21 +59,27 @@ const format = useCameraFormat(device, [
```ts ```ts
const format = getCameraFormat(device, [ const format = getCameraFormat(device, [
{ photoHDR: true }, { photoHdr: true },
{ videoHDR: true }, { videoHdr: true },
]) ])
``` ```
</TabItem> </TabItem>
</Tabs> </Tabs>
Then, pass the `format` to the Camera and enable the `hdr` prop if it is supported: Then, pass the `format` to the Camera and enable the `videoHdr`/`photoHdr` props if it is supported:
```tsx ```tsx
const format = ... const format = ...
const supportsHdr = format.supportsPhotoHDR && format.supportsVideoHDR
return <Camera {...props} format={format} hdr={supportsHdr} /> return (
<Camera
{...props}
format={format}
videoHdr={format.supportsVideoHdr}
photoHdr={format.supportsPhotoHdr}
/>
)
``` ```
Now, all captures (`takePhoto(..)` and `startRecording(..)`) will be configured to use HDR. Now, all captures (`takePhoto(..)` and `startRecording(..)`) will be configured to use HDR.

View File

@ -60,9 +60,9 @@ See ["Camera Devices"](devices) for more information.
Note: By default (when not passing the options object), a simpler device is already chosen. Note: By default (when not passing the options object), a simpler device is already chosen.
### No HDR ### No Video HDR
HDR uses 10-bit formats and/or additional processing steps that come with additional computation overhead. Disable HDR (don't pass `hdr` to the `<Camera>`) for higher efficiency. Video HDR uses 10-bit formats and/or additional processing steps that come with additional computation overhead. Disable Video HDR (don't pass `videoHdr` to the `<Camera>`) for higher efficiency.
### Buffer Compression ### Buffer Compression

View File

@ -145,7 +145,7 @@ To calculate your target bit-rate, you can use this formula:
```ts ```ts
let bitRate = baseBitRate let bitRate = baseBitRate
bitRate = bitRate / 30 * fps // FPS bitRate = bitRate / 30 * fps // FPS
if (hdr === true) bitRate *= 1.2 // HDR if (videoHdr === true) bitRate *= 1.2 // 10-Bit Video HDR
if (codec === 'h265') bitRate *= 0.8 // H.265 if (codec === 'h265') bitRate *= 0.8 // H.265
bitRate *= yourCustomFactor // e.g. 0.5x for half the bit-rate bitRate *= yourCustomFactor // e.g. 0.5x for half the bit-rate
``` ```

View File

@ -57,7 +57,7 @@ If you're experiencing build issues or runtime issues in VisionCamera, make sure
* For errors without messages, there's often an error code attached. Look up the error code on [osstatus.com](https://www.osstatus.com) to get more information about a specific error. * For errors without messages, there's often an error code attached. Look up the error code on [osstatus.com](https://www.osstatus.com) to get more information about a specific error.
2. If your Frame Processor is not running, make sure you check the native Xcode logs. There is useful information about the Frame Processor Runtime that will tell you if something goes wrong. 2. If your Frame Processor is not running, make sure you check the native Xcode logs. There is useful information about the Frame Processor Runtime that will tell you if something goes wrong.
3. If your Frame Processor is not running, make sure you are not using a remote JS debugger such as Google Chrome, since those don't work with JSI. 3. If your Frame Processor is not running, make sure you are not using a remote JS debugger such as Google Chrome, since those don't work with JSI.
4. If you are experiencing black-screens, try removing all properties such as `fps`, `hdr` or `format` on the `<Camera>` component except for the required ones: 4. If you are experiencing black-screens, try removing all properties such as `fps`, `videoHdr` or `format` on the `<Camera>` component except for the required ones:
```tsx ```tsx
<Camera device={device} isActive={true} style={{ width: 500, height: 500 }} /> <Camera device={device} isActive={true} style={{ width: 500, height: 500 }} />
``` ```
@ -112,7 +112,7 @@ If you're experiencing build issues or runtime issues in VisionCamera, make sure
2. If a camera device is not being returned by [`Camera.getAvailableCameraDevices()`](/docs/api/classes/Camera#getavailablecameradevices), make sure it is a Camera2 compatible device. See [this section in the Android docs](https://developer.android.com/reference/android/hardware/camera2/CameraDevice#reprocessing) for more information. 2. If a camera device is not being returned by [`Camera.getAvailableCameraDevices()`](/docs/api/classes/Camera#getavailablecameradevices), make sure it is a Camera2 compatible device. See [this section in the Android docs](https://developer.android.com/reference/android/hardware/camera2/CameraDevice#reprocessing) for more information.
3. If your Frame Processor is not running, make sure you check the native Android Studio/Logcat logs. There is useful information about the Frame Processor Runtime that will tell you if something goes wrong. 3. If your Frame Processor is not running, make sure you check the native Android Studio/Logcat logs. There is useful information about the Frame Processor Runtime that will tell you if something goes wrong.
4. If your Frame Processor is not running, make sure you are not using a remote JS debugger such as Google Chrome, since those don't work with JSI. 4. If your Frame Processor is not running, make sure you are not using a remote JS debugger such as Google Chrome, since those don't work with JSI.
5. If you are experiencing black-screens, try removing all properties such as `fps`, `hdr` or `format` on the `<Camera>` component except for the required ones: 5. If you are experiencing black-screens, try removing all properties such as `fps`, `videoHdr` or `format` on the `<Camera>` component except for the required ones:
```tsx ```tsx
<Camera device={device} isActive={true} style={{ width: 500, height: 500 }} /> <Camera device={device} isActive={true} style={{ width: 500, height: 500 }} />
``` ```

View File

@ -70,7 +70,8 @@ class CameraView(context: Context) :
var format: ReadableMap? = null var format: ReadableMap? = null
var fps: Int? = null var fps: Int? = null
var videoStabilizationMode: VideoStabilizationMode? = null var videoStabilizationMode: VideoStabilizationMode? = null
var hdr: Boolean? = null // nullable bool var videoHdr = false
var photoHdr = false
var lowLightBoost: Boolean? = null // nullable bool var lowLightBoost: Boolean? = null // nullable bool
// other props // other props
@ -177,6 +178,10 @@ class CameraView(context: Context) :
// Orientation // Orientation
config.orientation = orientation config.orientation = orientation
// HDR
config.videoHdr = videoHdr
config.photoHdr = photoHdr
// Format // Format
val format = format val format = format
if (format != null) { if (format != null) {
@ -188,7 +193,6 @@ class CameraView(context: Context) :
// Side-Props // Side-Props
config.fps = fps config.fps = fps
config.enableLowLightBoost = lowLightBoost ?: false config.enableLowLightBoost = lowLightBoost ?: false
config.enableHdr = hdr ?: false
config.torch = torch config.torch = torch
// Zoom // Zoom

View File

@ -107,9 +107,14 @@ class CameraViewManager : ViewGroupManager<CameraView>() {
view.fps = if (fps > 0) fps else null view.fps = if (fps > 0) fps else null
} }
@ReactProp(name = "hdr") @ReactProp(name = "photoHdr", defaultBoolean = false)
fun setHdr(view: CameraView, hdr: Boolean?) { fun setPhotoHdr(view: CameraView, photoHdr: Boolean) {
view.hdr = hdr view.photoHdr = photoHdr
}
@ReactProp(name = "videoHdr", defaultBoolean = false)
fun setVideoHdr(view: CameraView, videoHdr: Boolean) {
view.videoHdr = videoHdr
} }
@ReactProp(name = "lowLightBoost") @ReactProp(name = "lowLightBoost")
@ -117,7 +122,7 @@ class CameraViewManager : ViewGroupManager<CameraView>() {
view.lowLightBoost = lowLightBoost view.lowLightBoost = lowLightBoost
} }
@ReactProp(name = "isActive") @ReactProp(name = "isActive", defaultBoolean = false)
fun setIsActive(view: CameraView, isActive: Boolean) { fun setIsActive(view: CameraView, isActive: Boolean) {
view.isActive = isActive view.isActive = isActive
} }

View File

@ -17,7 +17,10 @@ data class CameraConfiguration(
var photo: Output<Photo> = Output.Disabled.create(), var photo: Output<Photo> = Output.Disabled.create(),
var video: Output<Video> = Output.Disabled.create(), var video: Output<Video> = Output.Disabled.create(),
var codeScanner: Output<CodeScanner> = Output.Disabled.create(), var codeScanner: Output<CodeScanner> = Output.Disabled.create(),
var enableHdr: Boolean = false,
// HDR
var videoHdr: Boolean = false,
var photoHdr: Boolean = false,
// Orientation // Orientation
var orientation: Orientation = Orientation.PORTRAIT, var orientation: Orientation = Orientation.PORTRAIT,
@ -87,7 +90,7 @@ data class CameraConfiguration(
val outputsChanged = deviceChanged || // input device val outputsChanged = deviceChanged || // input device
left?.photo != right.photo || left.video != right.video || left.codeScanner != right.codeScanner || left?.photo != right.photo || left.video != right.video || left.codeScanner != right.codeScanner ||
left.preview != right.preview || // outputs left.preview != right.preview || // outputs
left.enableHdr != right.enableHdr || left.format != right.format // props that affect the outputs (hdr, format, ..) left.videoHdr != right.videoHdr || left.photoHdr != right.photoHdr || left.format != right.format // props that affect the outputs
val sidePropsChanged = outputsChanged || // depend on outputs val sidePropsChanged = outputsChanged || // depend on outputs
left?.torch != right.torch || left.enableLowLightBoost != right.enableLowLightBoost || left.fps != right.fps || left?.torch != right.torch || left.enableLowLightBoost != right.enableLowLightBoost || left.fps != right.fps ||

View File

@ -4,7 +4,6 @@ import android.graphics.ImageFormat
import android.hardware.camera2.CameraCharacteristics import android.hardware.camera2.CameraCharacteristics
import android.hardware.camera2.CameraManager import android.hardware.camera2.CameraManager
import android.hardware.camera2.CameraMetadata import android.hardware.camera2.CameraMetadata
import android.hardware.camera2.params.DynamicRangeProfiles
import android.os.Build import android.os.Build
import android.util.Range import android.util.Range
import android.util.Size import android.util.Size
@ -90,10 +89,8 @@ class CameraDeviceDetails(private val cameraManager: CameraManager, private val
private fun getHasVideoHdr(): Boolean { private fun getHasVideoHdr(): Boolean {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) { if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) {
if (capabilities.contains(CameraMetadata.REQUEST_AVAILABLE_CAPABILITIES_DYNAMIC_RANGE_TEN_BIT)) { if (capabilities.contains(CameraMetadata.REQUEST_AVAILABLE_CAPABILITIES_DYNAMIC_RANGE_TEN_BIT)) {
val availableProfiles = characteristics.get(CameraCharacteristics.REQUEST_AVAILABLE_DYNAMIC_RANGE_PROFILES) val recommendedHdrProfile = characteristics.get(CameraCharacteristics.REQUEST_RECOMMENDED_TEN_BIT_DYNAMIC_RANGE_PROFILE)
?: DynamicRangeProfiles(LongArray(0)) return recommendedHdrProfile != null
return availableProfiles.supportedProfiles.contains(DynamicRangeProfiles.HLG10) ||
availableProfiles.supportedProfiles.contains(DynamicRangeProfiles.HDR10)
} }
} }
return false return false
@ -181,8 +178,8 @@ class CameraDeviceDetails(private val cameraManager: CameraManager, private val
map.putInt("maxFps", fpsRange.upper) map.putInt("maxFps", fpsRange.upper)
map.putDouble("maxZoom", maxZoom) map.putDouble("maxZoom", maxZoom)
map.putDouble("fieldOfView", getMaxFieldOfView()) map.putDouble("fieldOfView", getMaxFieldOfView())
map.putBoolean("supportsVideoHDR", supportsVideoHdr) map.putBoolean("supportsVideoHdr", supportsVideoHdr)
map.putBoolean("supportsPhotoHDR", supportsPhotoHdr) map.putBoolean("supportsPhotoHdr", supportsPhotoHdr)
map.putBoolean("supportsDepthCapture", supportsDepthCapture) map.putBoolean("supportsDepthCapture", supportsDepthCapture)
map.putString("autoFocusSystem", AutoFocusSystem.CONTRAST_DETECTION.unionValue) map.putString("autoFocusSystem", AutoFocusSystem.CONTRAST_DETECTION.unionValue)
map.putArray("videoStabilizationModes", createStabilizationModes()) map.putArray("videoStabilizationModes", createStabilizationModes())

View File

@ -285,7 +285,7 @@ class CameraSession(private val context: Context, private val cameraManager: Cam
val image = reader.acquireLatestImage() val image = reader.acquireLatestImage()
onPhotoCaptured(image) onPhotoCaptured(image)
}, CameraQueues.cameraQueue.handler) }, CameraQueues.cameraQueue.handler)
val output = PhotoOutput(imageReader, configuration.enableHdr) val output = PhotoOutput(imageReader, configuration.photoHdr)
outputs.add(output.toOutputConfiguration(characteristics)) outputs.add(output.toOutputConfiguration(characteristics))
photoOutput = output photoOutput = output
} }
@ -305,7 +305,7 @@ class CameraSession(private val context: Context, private val cameraManager: Cam
isSelfie, isSelfie,
video.config.enableFrameProcessor video.config.enableFrameProcessor
) )
val output = VideoPipelineOutput(videoPipeline, configuration.enableHdr) val output = VideoPipelineOutput(videoPipeline, configuration.videoHdr)
outputs.add(output.toOutputConfiguration(characteristics)) outputs.add(output.toOutputConfiguration(characteristics))
videoOutput = output videoOutput = output
} }
@ -327,7 +327,7 @@ class CameraSession(private val context: Context, private val cameraManager: Cam
preview.config.surface, preview.config.surface,
size, size,
SurfaceOutput.OutputType.PREVIEW, SurfaceOutput.OutputType.PREVIEW,
configuration.enableHdr configuration.videoHdr
) )
outputs.add(output.toOutputConfiguration(characteristics)) outputs.add(output.toOutputConfiguration(characteristics))
previewOutput = output previewOutput = output
@ -398,7 +398,7 @@ class CameraSession(private val context: Context, private val cameraManager: Cam
if (fps != null) { if (fps != null) {
if (!CAN_DO_60_FPS) { if (!CAN_DO_60_FPS) {
// If we can't do 60 FPS, we clamp it to 30 FPS - that's always supported. // If we can't do 60 FPS, we clamp it to 30 FPS - that's always supported.
fps = Math.min(30, fps) fps = 30.coerceAtMost(fps)
} }
captureRequest.set(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range(fps, fps)) captureRequest.set(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range(fps, fps))
} }
@ -427,7 +427,7 @@ class CameraSession(private val context: Context, private val cameraManager: Cam
// Set HDR // Set HDR
// TODO: Check if that value is even supported // TODO: Check if that value is even supported
if (config.enableHdr) { if (config.videoHdr) {
captureRequest.set(CaptureRequest.CONTROL_SCENE_MODE, CaptureRequest.CONTROL_SCENE_MODE_HDR) captureRequest.set(CaptureRequest.CONTROL_SCENE_MODE, CaptureRequest.CONTROL_SCENE_MODE_HDR)
} else if (config.enableLowLightBoost) { } else if (config.enableLowLightBoost) {
captureRequest.set(CaptureRequest.CONTROL_SCENE_MODE, CaptureRequest.CONTROL_SCENE_MODE_NIGHT) captureRequest.set(CaptureRequest.CONTROL_SCENE_MODE, CaptureRequest.CONTROL_SCENE_MODE_NIGHT)

View File

@ -17,8 +17,8 @@ data class CameraDeviceFormat(
val maxZoom: Double, val maxZoom: Double,
val videoStabilizationModes: List<VideoStabilizationMode>, val videoStabilizationModes: List<VideoStabilizationMode>,
val autoFocusSystem: AutoFocusSystem, val autoFocusSystem: AutoFocusSystem,
val supportsVideoHDR: Boolean, val supportsVideoHdr: Boolean,
val supportsPhotoHDR: Boolean, val supportsPhotoHdr: Boolean,
val pixelFormats: List<PixelFormat>, val pixelFormats: List<PixelFormat>,
val supportsDepthCapture: Boolean val supportsDepthCapture: Boolean
) { ) {
@ -50,8 +50,8 @@ data class CameraDeviceFormat(
value.getDouble("maxZoom"), value.getDouble("maxZoom"),
videoStabilizationModes, videoStabilizationModes,
autoFocusSystem, autoFocusSystem,
value.getBoolean("supportsVideoHDR"), value.getBoolean("supportsVideoHdr"),
value.getBoolean("supportsPhotoHDR"), value.getBoolean("supportsPhotoHdr"),
pixelFormats, pixelFormats,
value.getBoolean("supportsDepthCapture") value.getBoolean("supportsDepthCapture")
) )

View File

@ -68,7 +68,7 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
const fps = Math.min(format?.maxFps ?? 1, targetFps) const fps = Math.min(format?.maxFps ?? 1, targetFps)
const supportsFlash = device?.hasFlash ?? false const supportsFlash = device?.hasFlash ?? false
const supportsHdr = format?.supportsPhotoHDR const supportsHdr = format?.supportsPhotoHdr
const supports60Fps = useMemo(() => device?.formats.some((f) => f.maxFps >= 60), [device?.formats]) const supports60Fps = useMemo(() => device?.formats.some((f) => f.maxFps >= 60), [device?.formats])
const canToggleNightMode = device?.supportsLowLightBoost ?? false const canToggleNightMode = device?.supportsLowLightBoost ?? false
@ -181,7 +181,8 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
device={device} device={device}
format={format} format={format}
fps={fps} fps={fps}
hdr={enableHdr} photoHdr={enableHdr}
videoHdr={enableHdr}
lowLightBoost={device.supportsLowLightBoost && enableNightMode} lowLightBoost={device.supportsLowLightBoost && enableNightMode}
isActive={isActive} isActive={isActive}
onInitialized={onInitialized} onInitialized={onInitialized}

View File

@ -38,7 +38,8 @@ public final class CameraView: UIView, CameraSessionDelegate {
// props that require format reconfiguring // props that require format reconfiguring
@objc var format: NSDictionary? @objc var format: NSDictionary?
@objc var fps: NSNumber? @objc var fps: NSNumber?
@objc var hdr = false @objc var videoHdr = false
@objc var photoHdr = false
@objc var lowLightBoost = false @objc var lowLightBoost = false
@objc var orientation: NSString? @objc var orientation: NSString?
// other props // other props
@ -164,7 +165,7 @@ public final class CameraView: UIView, CameraSessionDelegate {
if video || enableFrameProcessor { if video || enableFrameProcessor {
config.video = .enabled(config: CameraConfiguration.Video(pixelFormat: getPixelFormat(), config.video = .enabled(config: CameraConfiguration.Video(pixelFormat: getPixelFormat(),
enableBufferCompression: enableBufferCompression, enableBufferCompression: enableBufferCompression,
enableHdr: hdr, enableHdr: videoHdr,
enableFrameProcessor: enableFrameProcessor)) enableFrameProcessor: enableFrameProcessor))
} else { } else {
config.video = .disabled config.video = .disabled

View File

@ -36,7 +36,8 @@ RCT_EXPORT_VIEW_PROPERTY(enableFrameProcessor, BOOL);
// device format // device format
RCT_EXPORT_VIEW_PROPERTY(format, NSDictionary); RCT_EXPORT_VIEW_PROPERTY(format, NSDictionary);
RCT_EXPORT_VIEW_PROPERTY(fps, NSNumber); RCT_EXPORT_VIEW_PROPERTY(fps, NSNumber);
RCT_EXPORT_VIEW_PROPERTY(hdr, BOOL); RCT_EXPORT_VIEW_PROPERTY(videoHdr, BOOL);
RCT_EXPORT_VIEW_PROPERTY(photoHdr, BOOL);
RCT_EXPORT_VIEW_PROPERTY(lowLightBoost, BOOL); RCT_EXPORT_VIEW_PROPERTY(lowLightBoost, BOOL);
RCT_EXPORT_VIEW_PROPERTY(videoStabilizationMode, NSString); RCT_EXPORT_VIEW_PROPERTY(videoStabilizationMode, NSString);
RCT_EXPORT_VIEW_PROPERTY(pixelFormat, NSString); RCT_EXPORT_VIEW_PROPERTY(pixelFormat, NSString);

View File

@ -207,7 +207,7 @@ extension CameraConfiguration.Video {
// Find the best matching format // Find the best matching format
guard let format = videoOutput.findPixelFormat(firstOf: targetFormats) else { guard let format = videoOutput.findPixelFormat(firstOf: targetFormats) else {
throw CameraError.format(.invalidHdr) throw CameraError.format(.invalidVideoHdr)
} }
// YUV 4:2:0 10-bit (compressed/uncompressed) // YUV 4:2:0 10-bit (compressed/uncompressed)
return format return format

View File

@ -108,7 +108,7 @@ enum DeviceError: String {
enum FormatError { enum FormatError {
case invalidFps(fps: Int) case invalidFps(fps: Int)
case invalidHdr case invalidVideoHdr
case invalidFormat case invalidFormat
case incompatiblePixelFormatWithHDR case incompatiblePixelFormatWithHDR
@ -118,8 +118,8 @@ enum FormatError {
return "invalid-format" return "invalid-format"
case .invalidFps: case .invalidFps:
return "invalid-fps" return "invalid-fps"
case .invalidHdr: case .invalidVideoHdr:
return "invalid-hdr" return "invalid-video-hdr"
case .incompatiblePixelFormatWithHDR: case .incompatiblePixelFormatWithHDR:
return "incompatible-pixel-format-with-hdr-setting" return "incompatible-pixel-format-with-hdr-setting"
} }
@ -131,8 +131,8 @@ enum FormatError {
return "The given format was invalid. Did you check if the current device supports the given format in `device.formats`?" return "The given format was invalid. Did you check if the current device supports the given format in `device.formats`?"
case let .invalidFps(fps): case let .invalidFps(fps):
return "The given format cannot run at \(fps) FPS! Make sure your FPS is lower than `format.maxFps` but higher than `format.minFps`." return "The given format cannot run at \(fps) FPS! Make sure your FPS is lower than `format.maxFps` but higher than `format.minFps`."
case .invalidHdr: case .invalidVideoHdr:
return "The currently selected format does not support HDR capture! Make sure you select a format which includes `supportsPhotoHDR`/`supportsVideoHDR`!" return "The currently selected format does not support 10-bit Video HDR streaming! Make sure you select a format which includes `supportsVideoHdr`!"
case .incompatiblePixelFormatWithHDR: case .incompatiblePixelFormatWithHDR:
return "The currently selected pixelFormat is not compatible with HDR! HDR only works with the `yuv` pixelFormat." return "The currently selected pixelFormat is not compatible with HDR! HDR only works with the `yuv` pixelFormat."
} }

View File

@ -100,7 +100,7 @@ extension CameraSession {
} }
// Video Output + Frame Processor // Video Output + Frame Processor
if case let .enabled(video) = configuration.video { if case .enabled = configuration.video {
ReactLogger.log(level: .info, message: "Adding Video Data output...") ReactLogger.log(level: .info, message: "Adding Video Data output...")
// 1. Add // 1. Add
@ -113,11 +113,6 @@ extension CameraSession {
// 2. Configure // 2. Configure
videoOutput.setSampleBufferDelegate(self, queue: CameraQueues.videoQueue) videoOutput.setSampleBufferDelegate(self, queue: CameraQueues.videoQueue)
videoOutput.alwaysDiscardsLateVideoFrames = true videoOutput.alwaysDiscardsLateVideoFrames = true
let pixelFormatType = try video.getPixelFormat(for: videoOutput)
videoOutput.videoSettings = [
String(kCVPixelBufferPixelFormatTypeKey): pixelFormatType,
]
self.videoOutput = videoOutput self.videoOutput = videoOutput
} }
@ -210,6 +205,21 @@ extension CameraSession {
ReactLogger.log(level: .info, message: "Successfully configured Format!") ReactLogger.log(level: .info, message: "Successfully configured Format!")
} }
func configurePixelFormat(configuration: CameraConfiguration) throws {
guard case let .enabled(video) = configuration.video,
let videoOutput else {
// Video is not enabled
return
}
// Configure the VideoOutput Settings to use the given Pixel Format.
// We need to run this after device.activeFormat has been set, otherwise the VideoOutput can't stream the given Pixel Format.
let pixelFormatType = try video.getPixelFormat(for: videoOutput)
videoOutput.videoSettings = [
String(kCVPixelBufferPixelFormatTypeKey): pixelFormatType,
]
}
// pragma MARK: Side-Props // pragma MARK: Side-Props
/** /**

View File

@ -141,21 +141,26 @@ class CameraSession: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate, AVC
if difference.formatChanged { if difference.formatChanged {
try self.configureFormat(configuration: config, device: device) try self.configureFormat(configuration: config, device: device)
} }
// 5. Configure side-props (fps, lowLightBoost) // 5. After step 2. and 4., we also need to configure the PixelFormat.
// This needs to be done AFTER we updated the `format`, as this controls the supported PixelFormats.
if difference.outputsChanged || difference.formatChanged {
try self.configurePixelFormat(configuration: config)
}
// 6. Configure side-props (fps, lowLightBoost)
if difference.sidePropsChanged { if difference.sidePropsChanged {
try self.configureSideProps(configuration: config, device: device) try self.configureSideProps(configuration: config, device: device)
} }
// 6. Configure zoom // 7. Configure zoom
if difference.zoomChanged { if difference.zoomChanged {
self.configureZoom(configuration: config, device: device) self.configureZoom(configuration: config, device: device)
} }
} }
} }
// 7. Start or stop the session if needed // 8. Start or stop the session if needed
self.checkIsActive(configuration: config) self.checkIsActive(configuration: config)
// 8. Enable or disable the Torch if needed (requires session to be running) // 9. Enable or disable the Torch if needed (requires session to be running)
if difference.torchChanged { if difference.torchChanged {
try self.withDeviceLock { device in try self.withDeviceLock { device in
try self.configureTorch(configuration: config, device: device) try self.configureTorch(configuration: config, device: device)

View File

@ -28,7 +28,7 @@ extension AVCaptureDevice.Format {
return maxRange?.maxFrameRate ?? 0 return maxRange?.maxFrameRate ?? 0
} }
var supportsVideoHDR: Bool { var supportsVideoHdr: Bool {
let pixelFormat = CMFormatDescriptionGetMediaSubType(formatDescription) let pixelFormat = CMFormatDescriptionGetMediaSubType(formatDescription)
let hdrFormats = [ let hdrFormats = [
kCVPixelFormatType_420YpCbCr10BiPlanarFullRange, kCVPixelFormatType_420YpCbCr10BiPlanarFullRange,
@ -38,7 +38,7 @@ extension AVCaptureDevice.Format {
return hdrFormats.contains(pixelFormat) return hdrFormats.contains(pixelFormat)
} }
var supportsPhotoHDR: Bool { var supportsPhotoHdr: Bool {
// TODO: Supports Photo HDR on iOS? // TODO: Supports Photo HDR on iOS?
return false return false
} }

View File

@ -31,8 +31,8 @@ struct CameraDeviceFormat: Equatable, CustomStringConvertible {
let videoStabilizationModes: [VideoStabilizationMode] let videoStabilizationModes: [VideoStabilizationMode]
let autoFocusSystem: AutoFocusSystem let autoFocusSystem: AutoFocusSystem
let supportsVideoHDR: Bool let supportsVideoHdr: Bool
let supportsPhotoHDR: Bool let supportsPhotoHdr: Bool
let pixelFormats: [PixelFormat] let pixelFormats: [PixelFormat]
@ -51,8 +51,8 @@ struct CameraDeviceFormat: Equatable, CustomStringConvertible {
maxZoom = format.videoMaxZoomFactor maxZoom = format.videoMaxZoomFactor
videoStabilizationModes = format.videoStabilizationModes.map { VideoStabilizationMode(from: $0) } videoStabilizationModes = format.videoStabilizationModes.map { VideoStabilizationMode(from: $0) }
autoFocusSystem = AutoFocusSystem(fromFocusSystem: format.autoFocusSystem) autoFocusSystem = AutoFocusSystem(fromFocusSystem: format.autoFocusSystem)
supportsVideoHDR = format.supportsVideoHDR supportsVideoHdr = format.supportsVideoHdr
supportsPhotoHDR = format.supportsPhotoHDR supportsPhotoHdr = format.supportsPhotoHdr
pixelFormats = CameraDeviceFormat.getAllPixelFormats() pixelFormats = CameraDeviceFormat.getAllPixelFormats()
supportsDepthCapture = format.supportsDepthCapture supportsDepthCapture = format.supportsDepthCapture
} }
@ -73,8 +73,8 @@ struct CameraDeviceFormat: Equatable, CustomStringConvertible {
videoStabilizationModes = try jsVideoStabilizationModes.map { try VideoStabilizationMode(jsValue: $0) } videoStabilizationModes = try jsVideoStabilizationModes.map { try VideoStabilizationMode(jsValue: $0) }
let jsAutoFocusSystem = jsValue["autoFocusSystem"] as! String let jsAutoFocusSystem = jsValue["autoFocusSystem"] as! String
autoFocusSystem = try AutoFocusSystem(jsValue: jsAutoFocusSystem) autoFocusSystem = try AutoFocusSystem(jsValue: jsAutoFocusSystem)
supportsVideoHDR = jsValue["supportsVideoHDR"] as! Bool supportsVideoHdr = jsValue["supportsVideoHdr"] as! Bool
supportsPhotoHDR = jsValue["supportsPhotoHDR"] as! Bool supportsPhotoHdr = jsValue["supportsPhotoHdr"] as! Bool
let jsPixelFormats = jsValue["pixelFormats"] as! [String] let jsPixelFormats = jsValue["pixelFormats"] as! [String]
pixelFormats = try jsPixelFormats.map { try PixelFormat(jsValue: $0) } pixelFormats = try jsPixelFormats.map { try PixelFormat(jsValue: $0) }
supportsDepthCapture = jsValue["supportsDepthCapture"] as! Bool supportsDepthCapture = jsValue["supportsDepthCapture"] as! Bool
@ -98,8 +98,8 @@ struct CameraDeviceFormat: Equatable, CustomStringConvertible {
"minISO": minISO, "minISO": minISO,
"fieldOfView": fieldOfView, "fieldOfView": fieldOfView,
"maxZoom": maxZoom, "maxZoom": maxZoom,
"supportsVideoHDR": supportsVideoHDR, "supportsVideoHdr": supportsVideoHdr,
"supportsPhotoHDR": supportsPhotoHDR, "supportsPhotoHdr": supportsPhotoHdr,
"minFps": minFps, "minFps": minFps,
"maxFps": maxFps, "maxFps": maxFps,
"pixelFormats": pixelFormats.map(\.jsValue), "pixelFormats": pixelFormats.map(\.jsValue),

View File

@ -142,8 +142,8 @@ export class Camera extends React.PureComponent<CameraProps> {
result = (result / 30) * fps result = (result / 30) * fps
// H.265 (HEVC) codec is 20% more efficient // H.265 (HEVC) codec is 20% more efficient
if (codec === 'h265') result = result * 0.8 if (codec === 'h265') result = result * 0.8
// HDR (10-bit) instead of SDR (8-bit) takes up 20% more pixels // 10-Bit Video HDR takes up 20% more pixels than standard range (8-bit SDR)
if (this.props.hdr) result = result * 1.2 if (this.props.videoHdr) result = result * 1.2
// Return overall result // Return overall result
return result * factor return result * factor
} }

View File

@ -91,11 +91,11 @@ export interface CameraDeviceFormat {
/** /**
* Specifies whether this format supports HDR mode for video capture * Specifies whether this format supports HDR mode for video capture
*/ */
supportsVideoHDR: boolean supportsVideoHdr: boolean
/** /**
* Specifies whether this format supports HDR mode for photo capture * Specifies whether this format supports HDR mode for photo capture
*/ */
supportsPhotoHDR: boolean supportsPhotoHdr: boolean
/** /**
* Specifies whether this format supports delivering depth data for photo or video capture. * Specifies whether this format supports delivering depth data for photo or video capture.
*/ */

View File

@ -15,7 +15,7 @@ export type DeviceError =
| 'device/camera-not-available-on-simulator' | 'device/camera-not-available-on-simulator'
export type FormatError = export type FormatError =
| 'format/invalid-fps' | 'format/invalid-fps'
| 'format/invalid-hdr' | 'format/invalid-video-hdr'
| 'format/incompatible-pixel-format-with-hdr-setting' | 'format/incompatible-pixel-format-with-hdr-setting'
| 'format/invalid-format' | 'format/invalid-format'
export type SessionError = export type SessionError =

View File

@ -116,7 +116,8 @@ export interface CameraProps extends ViewProps {
* *
* The format defines the possible values for properties like: * The format defines the possible values for properties like:
* - {@linkcode fps}: `format.minFps`...`format.maxFps` * - {@linkcode fps}: `format.minFps`...`format.maxFps`
* - {@linkcode hdr}: `format.supportsVideoHDR` * - {@linkcode videoHdr}: `format.supportsVideoHdr`
* - {@linkcode photoHdr}: `format.supportsPhotoHdr`
* - {@linkcode pixelFormat}: `format.pixelFormats`` * - {@linkcode pixelFormat}: `format.pixelFormats``
* - {@linkcode enableDepthData}: `format.supportsDepthCapture`` * - {@linkcode enableDepthData}: `format.supportsDepthCapture``
* - {@linkcode videoStabilizationMode}: `format.videoStabilizationModes`` * - {@linkcode videoStabilizationMode}: `format.videoStabilizationModes``
@ -139,11 +140,17 @@ export interface CameraProps extends ViewProps {
*/ */
fps?: number fps?: number
/** /**
* Enables or disables HDR streaming. * Enables or disables HDR Video Streaming for Preview, Video and Frame Processor via a 10-bit wide-color pixel format.
* *
* Make sure the given {@linkcode format} supports HDR (see {@linkcode CameraDeviceFormat.supportsVideoHDR format.supportsVideoHDR}). * Make sure the given {@linkcode format} supports HDR (see {@linkcode CameraDeviceFormat.supportsVideoHdr format.supportsVideoHdr}).
*/ */
hdr?: boolean videoHdr?: boolean
/**
* Enables or disables HDR Photo Capture via a double capture routine that combines low- and high exposure photos.
*
* Make sure the given {@linkcode format} supports HDR (see {@linkcode CameraDeviceFormat.supportsPhotoHdr format.supportsPhotoHdr}).
*/
photoHdr?: boolean
/** /**
* Enables or disables lossless buffer compression for the video stream. * Enables or disables lossless buffer compression for the video stream.
* If you only use {@linkcode video} or a {@linkcode frameProcessor}, this * If you only use {@linkcode video} or a {@linkcode frameProcessor}, this

View File

@ -63,11 +63,11 @@ export interface FormatFilter {
/** /**
* Whether you want to find a format that supports Photo HDR. * Whether you want to find a format that supports Photo HDR.
*/ */
photoHDR?: boolean photoHdr?: boolean
/** /**
* Whether you want to find a format that supports Photo HDR. * Whether you want to find a format that supports Photo HDR.
*/ */
videoHDR?: boolean videoHdr?: boolean
} }
type FilterWithPriority<T> = { type FilterWithPriority<T> = {
@ -198,15 +198,15 @@ export function getCameraFormat(device: CameraDevice, filters: FormatFilter[]):
} }
// Find Photo HDR formats // Find Photo HDR formats
if (filter.photoHDR != null) { if (filter.photoHdr != null) {
if (bestFormat.supportsPhotoHDR === filter.photoHDR.target) leftPoints++ if (bestFormat.supportsPhotoHdr === filter.photoHdr.target) leftPoints++
if (format.supportsPhotoHDR === filter.photoHDR.target) rightPoints++ if (format.supportsPhotoHdr === filter.photoHdr.target) rightPoints++
} }
// Find Video HDR formats // Find Video HDR formats
if (filter.videoHDR != null) { if (filter.videoHdr != null) {
if (bestFormat.supportsVideoHDR === filter.videoHDR.target) leftPoints++ if (bestFormat.supportsVideoHdr === filter.videoHdr.target) leftPoints++
if (format.supportsVideoHDR === filter.videoHDR.target) rightPoints++ if (format.supportsVideoHdr === filter.videoHdr.target) rightPoints++
} }
if (rightPoints > leftPoints) bestFormat = format if (rightPoints > leftPoints) bestFormat = format