`](https://github.com/react-native-video/react-native-video) component, uploaded to a backend, or saved to the Camera Roll using [react-native-cameraroll](https://github.com/react-native-cameraroll/react-native-cameraroll).
To pause/resume the recordings, you can use `pauseRecording()` and `resumeRecording()`:
diff --git a/docs/docs/guides/DEVICES.mdx b/docs/docs/guides/DEVICES.mdx
index a6d29d1..7b1032b 100644
--- a/docs/docs/guides/DEVICES.mdx
+++ b/docs/docs/guides/DEVICES.mdx
@@ -4,6 +4,8 @@ title: Camera Devices
sidebar_label: Camera Devices
---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
import useBaseUrl from '@docusaurus/useBaseUrl';
@@ -13,94 +15,221 @@ import useBaseUrl from '@docusaurus/useBaseUrl';
-### What are camera devices?
+## What are Camera Devices?
-Camera devices are the physical (or "virtual") devices that can be used to record videos or capture photos.
+Camera Devices are the physical (or "virtual") devices that can be used to record videos or capture photos.
-* **Physical**: A physical camera device is a **camera lens on your phone**. Different physical camera devices have different specifications, such as different capture formats, field of views, frame rates, focal lengths, and more. Some phones have multiple physical camera devices.
+* **Physical**: A physical Camera Device is a **camera lens on your phone**. Different physical Camera Devices have different specifications, such as different capture formats, resolutions, zoom levels, and more. Some phones have multiple physical Camera Devices.
- > Examples: _"Backside Wide-Angle Camera"_, _"Frontside Wide-Angle Camera (FaceTime HD)"_, _"Ultra-Wide-Angle back camera"_.
-
-* **Virtual**: A virtual camera device is a **combination of one or more physical camera devices**, and provides features such as _virtual-device-switchover_ while zooming or _combined photo delivery_ from all physical cameras to produce higher quality images.
+ > Examples: _"Backside Wide-Angle Camera"_, _"Frontside Wide-Angle Camera (FaceTime HD)"_, _"Ultra-Wide-Angle back camera"_
+* **Virtual**: A virtual camera device is a **combination of one or more physical camera devices**, and provides features such as _virtual-device-switchover_ while zooming (see video on the right) or _combined photo delivery_ from all physical cameras to produce higher quality images.
> Examples: _"Triple-Camera"_, _"Dual-Wide-Angle Camera"_
-### Get available camera devices
+## Select the default Camera
-To get a list of all available camera devices, use [the `getAvailableCameraDevices` function](/docs/api/classes/Camera#getavailablecameradevices):
+If you simply want to use the default [`CameraDevice`](/docs/api/interfaces/CameraDevice), you can just use whatever is available:
+
+
+
```ts
-const devices = await Camera.getAvailableCameraDevices()
+const device = useCameraDevice('back')
```
-Each camera device provides properties describing the features of this device. For example, a camera device provides the `hasFlash` property which is `true` if the device supports activating the flash when taking photos or recording videos.
+
+
-The most important properties are:
-
-* `devices`: A list of physical device types this camera device consists of. For a **single physical camera device**, this property is always an array of one element. **For virtual multi-cameras** this property contains all the physical camera devices that are combined to create this virtual multi-camera device
-* `position`: The position of the camera device relative to the phone (`front`, `back`)
-* `hasFlash`: Whether this camera device supports using the flash to take photos or record videos
-* `hasTorch`: Whether this camera device supports enabling/disabling the torch at any time ([`Camera.torch` prop](/docs/api/interfaces/CameraProps#torch))
-* `isMultiCam`: Determines whether the camera device is a virtual multi-camera device which contains multiple combined physical camera devices.
-* `minZoom`: The minimum available zoom factor. This value is often `1`. When you pass `zoom={0}` to the Camera, the `minZoom` factor will be applied.
-* `neutralZoom`: The zoom factor where the camera is "neutral". For any wide-angle cameras this property might be the same as `minZoom`, where as for ultra-wide-angle cameras ("fish-eye") this might be a value higher than `minZoom` (e.g. `2`). It is recommended that you always start at `neutralZoom` and let the user manually zoom out to `minZoom` on demand.
-* `maxZoom`: The maximum available zoom factor. When you pass `zoom={1}` to the Camera, the `maxZoom` factor will be applied.
-* `formats`: A list of all available formats (See [Camera Formats](formats))
-* `supportsFocus`: Determines whether this camera device supports focusing (See [Focusing](focusing))
-
-:::note
-See the [`CameraDevice` type](/docs/api/interfaces/CameraDevice) for full API reference
-:::
-
-For debugging purposes you can use the `id` or `name` properties to log and compare devices. You can also use the `devices` properties to determine the physical camera devices this camera device consists of, for example:
-
-* For a single Wide-Angle camera, this would be `["wide-angle-camera"]`
-* For a Triple-Camera, this would be `["wide-angle-camera", "ultra-wide-angle-camera", "telephoto-camera"]`
-
-Always choose a camera device that is best fitted for your use-case; so you might filter out any cameras that do not support flash, have low zoom values, are not on the back side of the phone, do not contain a format with high resolution or fps, and more.
-
-:::caution
-Make sure to be careful when filtering out unneeded camera devices, since not every phone supports all camera device types. Some phones don't even have front-cameras. You always want to have a camera device, even when it's not the one that has the best features.
-:::
-
-### The `useCameraDevices` hook
-
-VisionCamera provides a hook to make camera device selection a lot easier. You can specify a device type to only find devices with the given type:
-
-```tsx
-function App() {
- const devices = useCameraDevices('wide-angle-camera')
- const device = devices.back
-
- if (device == null) return
+
-> Example: `triple-camera` > `dual-wide-camera` > `dual-camera` > `wide-angle-camera` > `ultra-wide-angle-camera` > `telephoto-camera` > ...
+And VisionCamera will automatically find the best matching [`CameraDevice`](/docs/api/interfaces/CameraDevice) for you!
-```tsx
-function App() {
- const devices = useCameraDevices()
- const device = devices.back
+**🚀 Continue with: [Camera Lifecycle](lifecycle)**
- if (device == null) return
+
+
+```ts
+const device = useCameraDevice('back', {
+ physicalDevices: [
+ 'ultra-wide-angle-camera',
+ 'wide-angle-camera',
+ 'telephoto-camera'
+ ]
+})
```
+
+
+
+```ts
+const devices = Camera.getAvailableCameraDevices()
+const device = getCameraDevice(devices, 'back', {
+ physicalDevices: [
+ 'ultra-wide-angle-camera',
+ 'wide-angle-camera',
+ 'telephoto-camera'
+ ]
+})
+```
+
+
+
+
+This will try to find a [`CameraDevice`](/docs/api/interfaces/CameraDevice) that consists of all three physical Camera Devices, or the next best match (e.g. "Dual-Camera", or just a single wide-angle-camera) if not found. With the "Triple-Camera", we can now zoom out to a wider field of view:
+
+
+
+
+
+
+```ts
+const devices = useCameraDevices()
+const device = useMemo(() => findBestDevice(devices), [devices])
+```
+
+
+
+
+```ts
+const devices = Camera.getAvailableCameraDevices()
+const device = findBestDevice(devices)
+```
+
+
+
+
+### Selecting external Cameras
+
+VisionCamera supports using `external` Camera Devices, such as:
+- USB Camera Devices (if they support the [USB Video Class (UVC) Specification](https://en.wikipedia.org/wiki/List_of_USB_video_class_devices))
+- [Continuity Camera Devices](https://support.apple.com/en-us/HT213244) (e.g. your iPhone's or Mac's Camera connected through WiFi/Continuity)
+- Bluetooth/WiFi Camera Devices (if they are supported in the platform-native Camera APIs)
+
+Since `external` Camera Devices can be plugged in/out at any point, you need to make sure to listen for changes in the Camera Devices list when using `external` Cameras:
+
+
+
+
+The hooks (`useCameraDevice(..)` and `useCameraDevices()`) already automatically listen for Camera Device changes!
+
+```ts
+const usbCamera = useCameraDevice('external')
+```
+
+
+
+
+Add a listener by using the [`addCameraDevicesChangedListener(..)`](/docs/api/classes/Camera#addcameradeviceschangedlistener) API:
+
+```ts
+const listener = Camera.addCameraDevicesChangedListener((devices) => {
+ console.log(`Devices changed: ${devices}`)
+ this.usbCamera = devices.find((d) => d.position === "external")
+})
+// ...
+listener.remove()
+```
+
+
+
+
-### What are camera formats?
+## What are camera formats?
-Each camera device (see [Camera Devices](devices)) provides a number of capture formats that have different specifications. There are formats specifically designed for high-resolution photo capture, which have very high photo output quality but in return only support frame-rates of up to 30 FPS. On the other side, there might be formats that are designed for slow-motion video capture which have frame-rates up to 240 FPS.
+Each camera device (see ["Camera Devices"](devices)) provides a number of formats that have different specifications.
+There are formats specifically designed for high-resolution photo capture (but lower FPS), or formats that are designed for slow-motion video capture which have frame-rates of up to 240 FPS (but lower resolution).
-### What if I don't want to choose a format?
+## What if I don't want to choose a format?
-If you don't want to specify the best format for your camera device, you don't have to. The Camera _automatically chooses the best matching format_ for the current camera device. This is why the Camera's `format` property is _optional_.
+If you don't want to specify a Camera Format, you don't have to. The Camera automatically chooses the best matching format for the current camera device. This is why the Camera's `format` property is _optional_.
-### What you need to know about cameras
+**🚀 Continue with: [Taking Photos/Recording Videos](./capturing)**
+
+## Choosing custom formats
To understand a bit more about camera formats, you first need to understand a few "general camera basics":
-* Each camera device is built differently, e.g. _Telephoto devices_ often don't provide frame-rates as high as _Wide-Angle devices_.
-* Formats are designed for specific use-cases, so formats with high resolution photo output don't support frame-rates as high as formats with lower resolution.
-* Different formats provide different field-of-views (FOV), maximum zoom factors, color spaces (iOS only), resolutions, frame rate ranges, and systems to assist with capture (auto-focus systems, video stabilization systems, ...)
+* Each camera device is built differently, e.g. front-facing Cameras often don't have resolutions as high as the Cameras on the back. (see ["Camera Devices"](devices))
+* Formats are designed for specific use-cases, here are some examples for formats on a Camera Device:
+ * 4k Photos, 4k Videos, 30 FPS (high quality)
+ * 4k Photos, 1080p Videos, 60 FPS (high FPS)
+ * 4k Photos, 1080p Videos, 240 FPS (ultra high FPS/slow motion)
+ * 720p Photos, 720p Videos, 30 FPS (smaller buffers/e.g. faster face detection)
+* Each app has different requirements, so the format filtering is up to you.
-### Get started
+To get all available formats, simply use the `CameraDevice`'s [`formats` property](/docs/api/interfaces/CameraDevice#formats). These are a [CameraFormat's](/docs/api/interfaces/CameraDeviceFormat) props:
-Each application has different needs, so the format filtering is up to you.
+- [`photoHeight`](/docs/api/interfaces/CameraDeviceFormat#photoHeight)/[`photoWidth`](/docs/api/interfaces/CameraDeviceFormat#photoWidth): The resolution that will be used for capturing photos. Choose a format with your desired resolution.
+- [`videoHeight`](/docs/api/interfaces/CameraDeviceFormat#videoHeight)/[`videoWidth`](/docs/api/interfaces/CameraDeviceFormat#videoWidth): The resolution that will be used for recording videos. Choose a format with your desired resolution.
+- [`minFps`](/docs/api/interfaces/CameraDeviceFormat#minFps)/[`maxFps`](/docs/api/interfaces/CameraDeviceFormat#maxFps): A range of possible values for the `fps` property. For example, if your format has `minFps: 1` and `maxFps: 60`, you can either use `fps={30}`, `fps={60}` or any other value in between for recording videos.
+- [`videoStabilizationModes`](/docs/api/interfaces/CameraDeviceFormat#videoStabilizationModes): All supported Video Stabilization Modes, digital and optical. If this specific format contains your desired [`VideoStabilizationMode`](/docs/api/#videostabilizationmode), you can pass it to your `` via the [`videoStabilizationMode` property](/docs/api/interfaces/CameraProps#videoStabilizationMode).
+- [`pixelFormats`](/docs/api/interfaces/CameraDeviceFormat#pixelFormats): All supported Pixel Formats. If this specific format contains your desired [`PixelFormat`](/docs/api/#PixelFormat), you can pass it to your `` via the [`pixelFormat` property](/docs/api/interfaces/CameraProps#pixelFormat).
+- [`supportsVideoHDR`](/docs/api/interfaces/CameraDeviceFormat#supportsVideoHDR): Whether this specific format supports true 10-bit HDR for video capture. If this is `true`, you can enable `hdr` on your ``.
+- [`supportsPhotoHDR`](/docs/api/interfaces/CameraDeviceFormat#supportsPhotoHDR): Whether this specific format supports HDR for photo capture. It will use multiple captures to fuse over-exposed and under-exposed Images together to form one HDR photo. If this is `true`, you can enable `hdr` on your ``.
+- [`supportsDepthCapture`](/docs/api/interfaces/CameraDeviceFormat#supportsDepthCapture): Whether this specific format supports depth data capture. For devices like the TrueDepth/LiDAR cameras, this will always be true.
+- ...and more. See the [`CameraDeviceFormat` type](/docs/api/interfaces/CameraDeviceFormat) for all supported properties.
-To get all available formats, simply use the `CameraDevice`'s `.formats` property. See how to get a camera device in the [Camera Devices guide](devices).
+You can either find a matching format manually by looping through your `CameraDevice`'s [`formats` property](/docs/api/interfaces/CameraDevice#formats), or by using the helper functions from VisionCamera:
-:::note
-You can also manually get all camera devices and decide which device to use based on the available `formats`.
-:::
-
-This example shows how you would pick the format with the _highest frame rate_:
-
-```tsx
-function App() {
- const devices = useCameraDevices('wide-angle-camera')
- const device = devices.back
-
- const format = useMemo(() => {
- return device?.formats.reduce((prev, curr) => {
- if (prev == null) return curr
- if (curr.maxFps > prev.maxFps) return curr
- else return prev
- }, undefined)
- }, [device?.formats])
-
- if (device == null) return
+
```ts
-export const sortFormatsByResolution = (left: CameraDeviceFormat, right: CameraDeviceFormat): number => {
- // in this case, points aren't "normalized" (e.g. higher resolution = 1 point, lower resolution = -1 points)
- let leftPoints = left.photoHeight * left.photoWidth
- let rightPoints = right.photoHeight * right.photoWidth
-
- // we also care about video dimensions, not only photo.
- leftPoints += left.videoWidth * left.videoHeight
- rightPoints += right.videoWidth * right.videoHeight
-
- // you can also add points for FPS, etc
-
- return rightPoints - leftPoints
-}
-
-// and then call it:
-const formats = useMemo(() => device?.formats.sort(sortFormatsByResolution), [device?.formats])
+const device = ...
+const format = useCameraFormat(device, [
+ { videoResolution: { width: 3048, height: 2160 } },
+ { fps: 60 }
+])
```
-:::caution
-Be careful that you don't `filter` out a lot of formats since you might end up having no format to use at all. (_Remember; not all devices support e.g. 240 FPS._) Always carefully sort instead of filter, and pick the best available format - that way you are guaranteed to have a format available, even if your desired specifications aren't fully met.
-:::
+
+
-### Props
+```ts
+const device = ...
+const format = getCameraFormat(device, [
+ { videoResolution: { width: 3048, height: 2160 } },
+ { fps: 60 }
+])
+```
+
+
+
+
+The **filter is ordered by priority (descending)**, so if there is no format that supports both 4k and 60 FPS, the function will prefer 4k@30FPS formats over 1080p@60FPS formats, because 4k is a more important requirement than 60 FPS.
+
+If you want to record slow-motion videos, you want a format with a really high FPS setting, for example:
+
+
+
+
+```ts
+const device = ...
+const format = useCameraFormat(device, [
+ { fps: 240 }
+])
+```
+
+
+
+
+```ts
+const device = ...
+const format = getCameraFormat(device, [
+ { fps: 240 }
+])
+```
+
+
+
+
+If there is no format that has exactly 240 FPS, the closest thing to it will be used.
+
+You can also use the `'max'` flag to just use the maximum available resolution:
+
+
+
+
+```ts
+const device = ...
+const format = useCameraFormat(device, [
+ { videoResolution: 'max' },
+ { photoResolution: 'max' }
+])
+```
+
+
+
+
+```ts
+const device = ...
+const format = getCameraFormat(device, [
+ { videoResolution: 'max' },
+ { photoResolution: 'max' }
+])
+```
+
+
+
+
+## Camera Props
The `Camera` View provides a few props that depend on the specified `format`. For example, you can only set the `fps` prop to a value that is supported by the current `format`. So if you have a format that supports 240 FPS, you can set the `fps` to `240`:
```tsx
function App() {
// ...
+ const format = ...
+ const fps = format.maxFps >= 240 ? 240 : format.maxFps
+
return (
-
-)
+// Draw bananas in a Skia Canvas
+const onDraw = useDrawCallback((canvas) => {
+ for (const banana of bananas.value) {
+ const rect = Skia.XYWHRect(banana.x,
+ banana.y,
+ banana.width,
+ banana.height)
+ const paint = Skia.Paint()
+ paint.setColor(Skia.Color('red'))
+ frame.drawRect(rect, paint)
+ }
+})
```
+### Call functions
+
And you can also call back to the React-JS thread by using `createRunInJsFn(...)`:
-```tsx {1}
+```tsx
const onQRCodeDetected = Worklets.createRunInJsFn((qrCode: string) => {
navigation.push("ProductPage", { productId: qrCode })
})
@@ -120,14 +147,20 @@ const frameProcessor = useFrameProcessor((frame) => {
}, [onQRCodeDetected])
```
+## Threading
+
+By default, Frame Processors run synchronously with the Camera pipeline. Anything that takes longer than one Frame interval might block the Camera from streaming new Frames.
+For example, if your Camera is running at **30 FPS**, your Frame Processor has **33ms** to finish executing before the next Frame is dropped. At **60 FPS**, you only have **16ms**.
+
### Running asynchronously
-Since Frame Processors run synchronously with the Camera Pipeline, anything taking longer than one Frame interval might block the Camera from streaming new Frames. To avoid this, you can use `runAsync` to run code asynchronously on a different Thread:
+For longer running processing, you can use [`runAsync(..)`](/docs/api/#runasync) to run code asynchronously on a different Thread:
```ts
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
console.log("I'm running synchronously at 60 FPS!")
+
runAsync(() => {
'worklet'
console.log("I'm running asynchronously, possibly at a lower FPS rate!")
@@ -137,12 +170,13 @@ const frameProcessor = useFrameProcessor((frame) => {
### Running at a throttled FPS rate
-Some Frame Processor Plugins don't need to run on every Frame, for example a Frame Processor that detects the brightness in a Frame only needs to run twice per second:
+Some Frame Processor Plugins don't need to run on every Frame, for example a Frame Processor that detects the brightness in a Frame only needs to run twice per second. You can achieve this by using [`runAtTargetFps(..)`](/docs/api/#runAtTargetFps):
```ts
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
console.log("I'm running synchronously at 60 FPS!")
+
runAtTargetFps(2, () => {
'worklet'
console.log("I'm running synchronously at 2 FPS!")
@@ -150,9 +184,22 @@ const frameProcessor = useFrameProcessor((frame) => {
}, [])
```
-### Using Frame Processor Plugins
+## Native Frame Processor Plugins
-Frame Processor Plugins are distributed through npm. To install the [**vision-camera-image-labeler**](https://github.com/mrousavy/vision-camera-image-labeler) plugin, run:
+Since JavaScript is slower than native languages, it is recommended to use native Frame Processor Plugins for heavy processing.
+Such native plugins benefit of faster languages (Objective-C/Swift, Java/Kotlin, or C++), and can make use of CPU-Vector- or GPU-acceleration.
+
+### Creating native Frame Processor Plugins
+
+VisionCamera provides an easy-to-use API for creating native Frame Processor Plugins, which are used to either wrap existing algorithms (example: ["MLKit Face Detection"](https://developers.google.com/ml-kit/vision/face-detection)), or build your own custom algorithms.
+It's binding point is a simple callback function that gets called with the native frame type (`CMSampleBuffer` or `Image`), that you can use for any kind of processing.
+The native plugin can accept parameters (e.g. for configuration) and return any kind of values for result, which are bridged through JSI.
+
+See: ["Creating Frame Processor Plugins"](/docs/guides/frame-processors-plugins-overview).
+
+### Using Community Plugins
+
+Community Frame Processor Plugins are distributed through npm. To install the [**vision-camera-image-labeler**](https://github.com/mrousavy/vision-camera-image-labeler) plugin, run:
```bash
npm i vision-camera-image-labeler
@@ -169,53 +216,26 @@ const frameProcessor = useFrameProcessor((frame) => {
}, [])
```
-Check out [**Frame Processor community plugins**](/docs/guides/frame-processor-plugin-list) to discover plugins, or [**start creating a plugin yourself**](/docs/guides/frame-processors-plugins-overview)!
+Check out [Frame Processor community plugins](/docs/guides/frame-processor-plugin-list) to discover available community plugins.
-### Selecting a Format for a Frame Processor
+## Selecting a Format for a Frame Processor
When running frame processors, it is often important to choose an appropriate [format](/docs/guides/formats). Here are some general tips to consider:
-* If you are running heavy AI/ML calculations in your frame processor, make sure to [select a format](/docs/guides/formats) that has a lower resolution to optimize it's performance.
-* Sometimes a frame processor plugin only works with specific [pixel formats](/docs/api/interfaces/CameraDeviceFormat#pixelformat). Some plugins (like MLKit) don't work with `x420`.
+* If you are running heavy AI/ML calculations in your frame processor, make sure to [select a format](/docs/guides/formats) that has a lower resolution to optimize it's performance. You can also resize the Frame on-demand.
+* Sometimes a frame processor plugin only works with specific [pixel formats](/docs/api/interfaces/CameraDeviceFormat#pixelformats). Some plugins (like Tensorflow Lite Models) don't work with `yuv`, so use a [`pixelFormat`](/docs/api/interfaces/CameraProps#pixelFormat) of `rgb` instead.
+* Some Frame Processor plugins don't work with HDR formats. In this case you need to disable [`hdr`](/docs/api/interfaces/CameraProps#hdr).
-### Benchmarks
+## Benchmarks
Frame Processors are _really_ fast. I have used [MLKit Vision Image Labeling](https://firebase.google.com/docs/ml-kit/ios/label-images) to label 4k Camera frames in realtime, and measured the following results:
* Fully natively (written in pure Objective-C, no React interaction at all), I have measured an average of **68ms** per call.
* As a Frame Processor Plugin (written in Objective-C, called through a JS Frame Processor function), I have measured an average of **69ms** per call.
-This means that **the Frame Processor API only takes ~1ms longer than a fully native implementation**, making it **the fastest and easiest way to run any sort of Frame Processing in React Native**.
+This means that the Frame Processor API only takes ~1ms longer than a fully native implementation, making it **the fastest and easiest way to run any sort of Frame Processing in React Native**.
-> All measurements are recorded on an iPhone 11 Pro, benchmarked total execution time of the [`captureOutput`](https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutputsamplebufferdelegate/1385775-captureoutput) function by using [`CFAbsoluteTimeGetCurrent`](https://developer.apple.com/documentation/corefoundation/1543542-cfabsolutetimegetcurrent). Running smaller images (lower than 4k resolution) is much quicker and many algorithms can even run at 60 FPS.
-
-### Avoiding Frame-drops
-
-Frame Processors will be **synchronously** called for each frame the Camera sees and have to finish executing before the next frame arrives, otherwise the next frame(s) will be dropped. For a frame rate of **30 FPS**, you have about **33ms** to finish processing frames. For a QR Code Scanner, **5 FPS** (200ms) might suffice, while a object tracking AI might run at the same frame rate as the Camera itself (e.g. **60 FPS** (16ms)).
-
-### ESLint react-hooks plugin
-
-If you are using the [react-hooks ESLint plugin](https://www.npmjs.com/package/eslint-plugin-react-hooks), make sure to add `useFrameProcessor` to `additionalHooks` inside your ESLint config. (See ["advanced configuration"](https://www.npmjs.com/package/eslint-plugin-react-hooks#advanced-configuration))
-
-### Technical
-
-#### Frame Processors
-
-**Frame Processors** are JS functions that will be **workletized** using [react-native-worklets-core](https://github.com/margelo/react-native-worklets-core). They are created on a **parallel camera thread** using a separate JavaScript Runtime (_"VisionCamera JS-Runtime"_) and are **invoked synchronously** (using JSI) without ever going over the bridge. In a **Frame Processor** you can write normal JS code, call back to the React-JS Thread (e.g. `setState`), use [Shared Values](https://docs.swmansion.com/react-native-reanimated/docs/fundamentals/shared-values/) and call **Frame Processor Plugins**.
-
-#### Frame Processor Plugins
-
-**Frame Processor Plugins** are native functions (written in Objective-C, Swift, C++, Java or Kotlin) that are injected into the VisionCamera JS-Runtime. They can be **synchronously called** from your JS Frame Processors (using JSI) without ever going over the bridge. Because VisionCamera provides an easy-to-use plugin API, you can easily create a **Frame Processor Plugin** yourself. Some examples include [Barcode Scanning](https://developers.google.com/ml-kit/vision/barcode-scanning), [Face Detection](https://developers.google.com/ml-kit/vision/face-detection), [Image Labeling](https://developers.google.com/ml-kit/vision/image-labeling), [Text Recognition](https://developers.google.com/ml-kit/vision/text-recognition) and more.
-
-> Learn how to [**create Frame Processor Plugins**](frame-processors-plugins-overview), or check out the [**example Frame Processor Plugin for iOS**](https://github.com/mrousavy/react-native-vision-camera/blob/main/package/example/ios/Frame%20Processor%20Plugins/Example%20Plugin%20(Swift)/ExamplePluginSwift.swift) or [**Android**](https://github.com/mrousavy/react-native-vision-camera/blob/main/package/example/android/app/src/main/java/com/mrousavy/camera/example/ExampleFrameProcessorPlugin.java).
-
-#### The `Frame` object
-
-The Frame Processor gets called with a `Frame` object, which is a **JSI HostObject**. It holds a reference to the native (C++) Frame Image Buffer (~10 MB in size) and exposes properties such as `width`, `height`, `bytesPerRow` and more to JavaScript so you can synchronously access them. The `Frame` object can be passed around in JS, as well as returned from- and passed to a native **Frame Processor Plugin**.
-
-> See [**this tweet**](https://twitter.com/mrousavy/status/1412300883149393921) for more information.
-
-### Disabling Frame Processors
+## Disabling Frame Processors
The Frame Processor API spawns a secondary JavaScript Runtime which consumes a small amount of extra CPU and RAM. Additionally, compile time increases since Frame Processors are written in native C++. If you're not using Frame Processors at all, you can disable them:
@@ -228,7 +248,7 @@ The Frame Processor API spawns a secondary JavaScript Runtime which consumes a s
]}>
-#### Android
+### Android
Inside your `gradle.properties` file, add the `disableFrameProcessors` flag:
@@ -238,7 +258,7 @@ VisionCamera_disableFrameProcessors=true
Then, clean and rebuild your project.
-#### iOS
+### iOS
Inside your `Podfile`, add the `VCDisableFrameProcessors` flag:
diff --git a/docs/docs/guides/FRAME_PROCESSORS_CREATE_OVERVIEW.mdx b/docs/docs/guides/FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
index 9660fc0..f1b5aea 100644
--- a/docs/docs/guides/FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
+++ b/docs/docs/guides/FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
@@ -12,7 +12,7 @@ import TabItem from '@theme/TabItem';
Frame Processor Plugins are **native functions** which can be directly called from a JS Frame Processor. (See ["Frame Processors"](frame-processors))
-They **receive a frame from the Camera** as an input and can return any kind of output. For example, a `detectFaces` function returns an array of detected faces in the frame:
+They receive a frame from the Camera as an input and can return any kind of output. For example, a `detectFaces` function returns an array of detected faces in the frame:
```tsx {4-5}
function App() {
@@ -28,7 +28,7 @@ function App() {
}
```
-To achieve **maximum performance**, the `detectFaces` function is written in a native language (e.g. Objective-C), but it will be directly called from the VisionCamera Frame Processor JavaScript-Runtime.
+For maximum performance, the `detectFaces` function is written in a native language (e.g. Objective-C), but it will be directly called from the VisionCamera Frame Processor JavaScript-Runtime through JSI.
### Types
diff --git a/docs/docs/guides/FRAME_PROCESSORS_SKIA.mdx b/docs/docs/guides/FRAME_PROCESSORS_SKIA.mdx
index 1cea05f..bb948ae 100644
--- a/docs/docs/guides/FRAME_PROCESSORS_SKIA.mdx
+++ b/docs/docs/guides/FRAME_PROCESSORS_SKIA.mdx
@@ -1,7 +1,7 @@
---
-id: frame-processors-skia
-title: Skia Frame Processors
-sidebar_label: Skia Frame Processors
+id: skia-frame-processors
+title: Drawing to a Frame (Skia)
+sidebar_label: Drawing to a Frame (Skia)
---
import Tabs from '@theme/Tabs';
@@ -10,110 +10,62 @@ import useBaseUrl from '@docusaurus/useBaseUrl';
-
-### What are Skia Frame Processors?
+## Skia Frame Processors
-Skia Frame Processors are [Frame Processors](frame-processors) that allow you to draw onto the Frame using [react-native-skia](https://github.com/Shopify/react-native-skia).
+It is technically pretty difficult to draw onto a Camera Frame and have it rendered into the resulting photo or video in realtime, but I have built a working proof of concept of that using Metal/OpenGL and [Skia](https://skia.org) straight in VisionCamera.
-Skia Frame Processors were introduced in VisionCamera V3 RC.0, but were removed again after VisionCamera V3 RC.9 due to the significantly increased complexity of the video pipeline in the codebase.
+This allows you to draw onto a Frame using [react-native-skia](https://shopify.github.io/react-native-skia/)'s easy to use JavaScript APIs:
-```
-yarn add react-native-vision-camera@rc.9
+```ts
+const frameProcessor = useSkiaFrameProcessor((frame) => {
+ 'worklet'
+
+ // Detect objects using GPU-accelerated ML API
+ const bananas = detectBananas()
+
+ // Draw banana outline using GPU-accelerated Skia drawing API
+ for (const banana of bananas) {
+ const rect = Skia.XYWHRect(banana.x,
+ banana.y,
+ banana.width,
+ banana.height)
+ const paint = Skia.Paint()
+ paint.setColor(Skia.Color('red'))
+ frame.drawRect(rect, paint)
+ }
+}, [])
```
-They worked perfectly fine for those RCs with some minor inconsistencies (landscape orientation didn't work on Android), which proves the concept. If you want to learn more about Skia Frame Processors, we at [Margelo](https://margelo.io) can build a custom solution for your company to implement drawable Frame Processors (e.g. filters, blurring, masks, colors, etc). See [PR #1740](https://github.com/mrousavy/react-native-vision-camera/pull/1740) for more details.
+..or even apply color-filters in realtime:
-### Documentation
+```ts
+const invertColorsFilter = Skia.RuntimeEffect.Make(`
+ uniform shader image;
+ half4 main(vec2 pos) {
+ vec4 color = image.eval(pos);
+ return vec4((1.0 - color).rgb, 1.0);
+ }
+`)
+const paint = Skia.Paint(invertColorsFilter)
-For example, you might want to draw a rectangle around a user's face **without writing any native code**, while still **achieving native performance**:
+const frameProcessor = useSkiaFrameProcessor((frame) => {
+ 'worklet'
-```jsx
-function App() {
- const frameProcessor = useSkiaFrameProcessor((frame) => {
- 'worklet'
- const faces = detectFaces(frame)
- faces.forEach((face) => {
- frame.drawRect(face.rectangle, redPaint)
- })
- }, [])
-
- return (
-
+
+
+
+
+
+
+
+```ts
+const format = useCameraFormat(device, [
+ { photoHDR: true },
+ { videoHDR: true },
+])
+```
+
+
+
+
+```ts
+const format = getCameraFormat(device, [
+ { photoHDR: true },
+ { videoHDR: true },
+])
+```
+
+
+
+
+Then, pass the `format` to the Camera and enable the `hdr` prop if it is supported:
+
+```tsx
+const format = ...
+const supportsHdr = format.supportsPhotoHDR && format.supportsVideoHDR
+
+return
+
+
+```ts
+const fasterDevice = useCameraDevice('back', {
+ physicalDevices: ['wide-angle-camera']
+})
+const slowerDevice = useCameraDevice('back', {
+ physicalDevices: ['ultra-wide-angle-camera', 'wide-angle-camera', 'telephoto-camera']
+})
+```
+
+
+
+
+```ts
+const devices = Camera.getAvailableCameraDevices()
+const fasterDevice = getCameraDevice(devices, 'back', {
+ physicalDevices: ['wide-angle-camera']
+})
+const slowerDevice = getCameraDevice(devices, 'back', {
+ physicalDevices: ['ultra-wide-angle-camera', 'wide-angle-camera', 'telephoto-camera']
+})
+```
+
+
+
+
+See ["Camera Devices"](devices) for more information.
+
+Note: By default (when not passing the options object), a simpler device is already chosen.
+
+### No HDR
+
+HDR uses 10-bit formats and/or additional processing steps that come with additional computation overhead. Disable HDR (don't pass `hdr` to the ``) for higher efficiency.
+
+### Buffer Compression
+
+Enable Buffer Compression ([`enableBufferCompression`](/docs/api/interfaces/CameraProps#enablebuffercompression)) to use lossless-compressed buffers for the Camera's video pipeline. These buffers can use less memory and are more efficient.
+
+Note: When not using a `frameProcessor`, buffer compression is automatically enabled.
+
+### Video Stabilization
+
+Video Stabilization requires additional overhead to start the algorithm, so disabling [`videoStabilizationMode`](/docs/api/interfaces/CameraProps#videoStabilizationMode) can significantly speed up the Camera initialization time.
+
+### Pixel Format
+
+By default, the `native` [`PixelFormat`](/docs/api#PixelFormat) is used, which is much more efficient than `rgb`.
+
+### Disable unneeded pipelines
+
+Only enable [`photo`](/docs/api/interfaces/CameraProps#photo) and [`video`](/docs/api/interfaces/CameraProps#video) if needed.
+
+### Fast Photos
+
+If you need to take photos as fast as possible, use a [`qualityPrioritization`](/docs/api/interfaces/TakePhotoOptions#qualityprioritization) of `'speed'` to speed up the photo pipeline:
+
+```ts
+camera.current.takePhoto({
+ qualityPrioritization: 'speed'
+})
+```
+
+### Appropriate Format resolution
+
+Choose formats efficiently. If your backend can only handle 1080p videos, don't select a 4k format if you have to downsize it later anyways - instead use 1080p already for the Camera:
+
+
+
+
+```ts
+const format = useCameraFormat(device, [
+ { videoResolution: { width: 1920, height: 1080 } }
+])
+```
+
+
+
+
+```ts
+const format = getCameraFormat(device, [
+ { videoResolution: { width: 1920, height: 1080 } }
+])
+```
+
+
+
+
+### Appropriate Format FPS
+
+Same as with format resolutions, also record at the frame rate you expect. Setting your frame rate higher can use more memory and heat up the battery.
+If your backend can only handle 30 FPS, there is no need to record at 60 FPS, instead set the Camera' [`fps`](/docs/api/interfaces/CameraProps#fps) to 30:
+
+```jsx
+return
+
+
+Simply use the `useCameraPermission` or `useMicrophonePermission` hook:
+
+```ts
+const { hasPermission, requestPermission } = useCameraPermission()
+```
+
+And if you want to use the microphone:
+
+```ts
+const { hasPermission, requestPermission } = useMicrophonePermission()
+```
+
+There could be three states to this:
+
+1. First time opening the app, `hasPermission` is false. Call `requestPermission()` now.
+2. User already granted permission, `hasPermission` is true. Continue with [**using the `` view**](#use-the-camera-view).
+3. User explicitly denied permission, `hasPermission` is false and `requestPermission()` will return false. Tell the user that he needs to grant Camera Permission, potentially by using the [`Linking` API](https://reactnative.dev/docs/linking#opensettings) to open the App Settings.
+
+
+
+
### Getting Permissions
@@ -141,7 +173,7 @@ A permission status can have the following values:
* `granted`: Your app is authorized to use said permission. Continue with [**using the `` view**](#use-the-camera-view).
* `not-determined`: Your app has not yet requested permission from the user. [Continue by calling the **request** functions.](#requesting-permissions)
* `denied`: Your app has already requested permissions from the user, but was explicitly denied. You cannot use the **request** functions again, but you can use the [`Linking` API](https://reactnative.dev/docs/linking#opensettings) to redirect the user to the Settings App where he can manually grant the permission.
-* `restricted`: (iOS only) Your app cannot use the Camera or Microphone because that functionality has been restricted, possibly due to active restrictions such as parental controls being in place.
+* `restricted`: Your app cannot use the Camera or Microphone because that functionality has been restricted, possibly due to active restrictions such as parental controls being in place.
### Requesting Permissions
@@ -160,18 +192,20 @@ The permission request status can have the following values:
* `granted`: Your app is authorized to use said permission. Continue with [**using the `` view**](#use-the-camera-view).
* `denied`: The user explicitly denied the permission request alert. You cannot use the **request** functions again, but you can use the [`Linking` API](https://reactnative.dev/docs/linking#opensettings) to redirect the user to the Settings App where he can manually grant the permission.
-* `restricted`: (iOS only) Your app cannot use the Camera or Microphone because that functionality has been restricted, possibly due to active restrictions such as parental controls being in place.
+* `restricted`: Your app cannot use the Camera or Microphone because that functionality has been restricted, possibly due to active restrictions such as parental controls being in place.
+
+
+
## Use the `` view
-If your app has permission to use the Camera and Microphone, simply use the [`useCameraDevices(...)`](/docs/api#usecameradevices) hook to get a Camera device (see [Camera Devices](/docs/guides/devices)) and mount the `` view:
+If your app has permission to use the Camera and Microphone, simply use the [`useCameraDevice(...)`](/docs/api#usecameradevice) hook to get a Camera device (see [Camera Devices](/docs/guides/devices)) and mount the `` view:
```tsx
function App() {
- const devices = useCameraDevices()
- const device = devices.back
+ const device = useCameraDevice('back')
- if (device == null) return
-
-
-
-
-### Drawing onto Frame Processors
-
-It is technically pretty difficult to draw onto a Camera Frame and have it rendered into the resulting photo or video in realtime, but I have built a working proof of concept of that using Metal/OpenGL and Skia straight in VisionCamera.
-
-This allows you to draw onto a Frame using simple JavaScript APIs:
-
-```ts
-const frameProcessor = useSkiaFrameProcessor((frame) => {
- 'worklet'
-
- // Detect faces using GPU-accelerated ML API
- const faces = detectFaces()
-
- // Draw faces using GPU-accelerated Skia drawing API
- for (const face of faces) {
- const rect = Skia.XYWHRect(face.x, face.y, face.width, face.height)
- const paint = Skia.Paint()
- paint.setColor(Skia.Color('red'))
- frame.drawRect(rect, paint)
- }
-}, [])
-```
-
-..or even apply color-filters in realtime:
-
-```ts
-const invertColorsFilter = Skia.RuntimeEffect.Make(`
- uniform shader image;
- half4 main(vec2 pos) {
- vec4 color = image.eval(pos);
- return vec4((1.0 - color).rgb, 1.0);
- }
-`)
-const paint = Skia.Paint(invertColorsFilter)
-
-const frameProcessor = useSkiaFrameProcessor((frame) => {
- 'worklet'
-
- // Draw frame using Skia Shader to invert colors
- frame.render(paint)
-}, [])
-```
-
-While this API was originally planned for V3, I decided to remove it again because VisionCamera got insanely complex with that code integrated (custom Metal/OpenGL rendering pipeline with intermediate steps and tons of build setup) and I wanted to keep VisionCamera lean so other users aren't affected by build issues caused by Skia or those GPU APIs. See [this PR for more information](https://github.com/mrousavy/react-native-vision-camera/pull/1740).
-
-In my agency, [Margelo](https://margelo.io), we have worked a lot with 2D and 3D graphics and Camera realtime processing (see the Snapchat-style mask filter on our website for example - that is running in VisionCamera/React Native!), if you need this feature [reach out to us](https://margelo.io#contact) and we'll build a customized/tailored solution for your company! :)
-
-
+
+
+
+
+```ts
+const format = useCameraFormat(device, [
+ { videoStabilizationMode: 'cinematic-extended' }
+])
+```
+
+
+
+
+```ts
+const format = getCameraFormat(device, [
+ { videoStabilizationMode: 'cinematic-extended' }
+])
+```
+
+
+
+
+Then, pass the `format` to the Camera and enable the `videoStabilizationMode` prop if it is supported:
+
+```tsx
+const format = ...
+const supportsVideoStabilization = format.videoStabilizationModes.includes('cinematic-extended')
+
+return (
+
-Before opening an issue, make sure you try the following:
+## Steps to try
-## iOS
+If you're experiencing build issues or runtime issues in VisionCamera, make sure you try the following before opening an issue:
+
+
+
### Build Issues
@@ -52,10 +63,12 @@ Before opening an issue, make sure you try the following:
```
5. Investigate the camera devices this phone has and make sure you're using a valid one. Look for properties such as `pixelFormats`, `id`, and `hardwareLevel`.
```tsx
- Camera.getAvailableCameraDevices().then((d) => console.log(JSON.stringify(d, null, 2)))
+ const devices = Camera.getAvailableCameraDevices()
+ console.log(JSON.stringify(d, null, 2))
```
-## Android
+
+
### Build Issues
@@ -88,7 +101,14 @@ Before opening an issue, make sure you try the following:
### Runtime Issues
-1. Check the logs in Android Studio/Logcat to find out more. In Android Studio, go to **View** > **Tool Windows** > **Logcat** (⌘ +6 ) or run `adb logcat` in Terminal.
+1. Check the logs in Android Studio/Logcat to find out more. In Android Studio, go to **View** > **Tool Windows** > **Logcat** (⌘ +6 ) or run `adb logcat` in Terminal. Android Logcat logs look like this:
+ ```logcat
+ 09:03:46 I ReactNativeJS: Running "App" with {"rootTag":11}
+ 09:03:47 I VisionCamera: Loading native C++ library...
+ 09:03:47 I VisionCamera: Installing JSI bindings...
+ 09:03:47 I VisionCamera: Finished installing JSI bindings!
+ ...
+ ```
2. If a camera device is not being returned by [`Camera.getAvailableCameraDevices()`](/docs/api/classes/Camera#getavailablecameradevices), make sure it is a Camera2 compatible device. See [this section in the Android docs](https://developer.android.com/reference/android/hardware/camera2/CameraDevice#reprocessing) for more information.
3. If your Frame Processor is not running, make sure you check the native Android Studio/Logcat logs. There is useful information about the Frame Processor Runtime that will tell you if something goes wrong.
4. If your Frame Processor is not running, make sure you are not using a remote JS debugger such as Google Chrome, since those don't work with JSI.
@@ -98,9 +118,13 @@ Before opening an issue, make sure you try the following:
```
6. Investigate the camera devices this phone has and make sure you're using a valid one. Look for properties such as `pixelFormats`, `id`, and `hardwareLevel`.
```tsx
- Camera.getAvailableCameraDevices().then((d) => console.log(JSON.stringify(d, null, 2)))
+ const devices = Camera.getAvailableCameraDevices()
+ console.log(JSON.stringify(d, null, 2))
```
+
+
+
## Issues
If nothing has helped so far, try browsing the [GitHub issues](https://github.com/mrousavy/react-native-vision-camera/issues?q=is%3Aissue). If your issue doesn't exist, [create a new one](https://github.com/mrousavy/react-native-vision-camera/issues/new/choose). Make sure to fill out the template and include as many details as possible. Also try to reproduce the issue in the [example app](https://github.com/mrousavy/react-native-vision-camera/blob/main/package/example).
diff --git a/docs/docs/guides/ZOOMING.mdx b/docs/docs/guides/ZOOMING.mdx
index 099565e..9bf6a96 100644
--- a/docs/docs/guides/ZOOMING.mdx
+++ b/docs/docs/guides/ZOOMING.mdx
@@ -13,15 +13,42 @@ import useBaseUrl from '@docusaurus/useBaseUrl';
-The `` component already provides a natively implemented zoom gesture which you can enable with the [`enableZoomGesture`](/docs/api/interfaces/CameraProps#enablezoomgesture) prop. This does not require any additional work, but if you want to setup a custom gesture, such as the one in Snapchat or Instagram where you move up your finger while recording, continue reading.
+## Native Zoom Gesture
-### Animation libraries
+The `` component already provides a natively implemented zoom gesture which you can enable with the [`enableZoomGesture`](/docs/api/interfaces/CameraProps#enablezoomgesture) prop. If you don't need any additional logic in your zoom gesture, you can skip to the next section.
-While you can use any animation library to animate the `zoom` property (or use no animation library at all) it is recommended to use [react-native-reanimated](https://github.com/software-mansion/react-native-reanimated) (v2) to achieve best performance. Head over to their [Installation guide](https://docs.swmansion.com/react-native-reanimated/docs/installation) to install Reanimated if you haven't already.
+**🚀 Next section: [Focusing](focusing)**
-### Implementation
+If you want to setup a custom gesture, such as the one in Snapchat or Instagram where you move up your finger while recording, first understand how zoom is expressed.
-#### Overview
+## Min, Max and Neutral Zoom
+
+A Camera device has different minimum, maximum and neutral zoom values. Those values are expressed through the `CameraDevice`'s [`minZoom`](/docs/api/interfaces/CameraDevice#minzoom), [`maxZoom`](/docs/api/interfaces/CameraDevice#maxzoom) and [`neutralZoom`](/docs/api/interfaces/CameraDevice#neutralzoom) props, and are represented in "scale". So if the `maxZoom` property of a device is `2`, that means the view can be enlarged by twice it's zoom, aka the viewport halves.
+
+* The `minZoom` value is always `1`.
+* The `maxZoom` value can have very high values (such as `128`), but often you want to clamp this value to something realistic like `16`.
+* The `neutralZoom` value is often `1`, but can be larger than `1` for devices with "fish-eye" (ultra-wide-angle) cameras. In those cases, the user expects to be at whatever zoom value `neutralZoom` is (e.g. `2`) per default, and if he tries to zoom out even more, he goes to `minZoom` (`1`), which switches over to the "fish-eye" (ultra-wide-angle) camera as seen in this GIF:
+
+
+
+
`, ``), try to flatten the `zoom` property to a **linear scale** by raising it **exponentially**. (`zoom.value ** 2`)
+
+## Pinch-to-zoom
+
+The above example only demonstrates how to animate the `zoom` property. To actually implement pinch-to-zoom or pan-to-zoom, take a look at the [VisionCamera example app](https://github.com/mrousavy/react-native-vision-camera/tree/main/package/example), the pinch-to-zoom gesture can be found [here](https://github.com/mrousavy/react-native-vision-camera/blob/main/package/example/src/views/CaptureButton.tsx#L189-L208), and the pan-to-zoom gesture can be found [here](https://github.com/mrousavy/react-native-vision-camera/blob/d8551792e97eaa6fa768f54059ffce054bf748d9/example/src/views/CaptureButton.tsx#L185-L205). They implement a real world use-case, where the maximum zoom value is clamped to a realistic value, and the zoom responds very gracefully by using a logarithmic scale.
+
+
+## Implementation (Reanimated)
+
+While you can use any animation library to animate the `zoom` property (or use no animation library at all) it is recommended to use [react-native-reanimated](https://github.com/software-mansion/react-native-reanimated) to achieve best performance. Head over to their [Installation guide](https://docs.swmansion.com/react-native-reanimated/docs/installation) to install Reanimated if you haven't already.
+
+### Overview
1. Make the Camera View animatable using `createAnimatedComponent`
2. Make the Camera's `zoom` property animatable using `addWhitelistedNativeProps`
@@ -29,7 +56,7 @@ While you can use any animation library to animate the `zoom` property (or use n
4. Use [`useAnimatedProps`](https://docs.swmansion.com/react-native-reanimated/docs/api/useAnimatedProps) to map the zoom SharedValue to the zoom property.
5. We apply the animated props to the `ReanimatedCamera` component's `animatedProps` property.
-#### Code
+### Code
The following example implements a button which smoothly zooms to a random value using [react-native-reanimated](https://github.com/software-mansion/react-native-reanimated):
@@ -38,28 +65,28 @@ import Reanimated, {
useAnimatedProps,
useSharedValue,
withSpring,
+ addWhitelistedNativeProps
} from "react-native-reanimated"
const ReanimatedCamera = Reanimated.createAnimatedComponent(Camera)
-Reanimated.addWhitelistedNativeProps({
+addWhitelistedNativeProps({
zoom: true,
})
export function App() {
- const devices = useCameraDevices()
- const device = devices.back
+ const device = useCameraDevice('back')
const zoom = useSharedValue(0)
const onRandomZoomPress = useCallback(() => {
zoom.value = withSpring(Math.random())
- }, [])
+ }, [zoom])
const animatedProps = useAnimatedProps>(
() => ({ zoom: zoom.value }),
[zoom]
)
- if (device == null) return
- `, ``), try to flatten the `zoom` property to a **linear scale** by raising it **exponentially**. (`zoom.value ** 2`)
-
-### Pinch-to-zoom
-
-The above example only demonstrates how to animate the `zoom` property. To actually implement pinch-to-zoom or pan-to-zoom, take a look at the [VisionCamera example app](https://github.com/mrousavy/react-native-vision-camera/tree/main/package/example), the pinch-to-zoom gesture can be found [here](https://github.com/mrousavy/react-native-vision-camera/blob/main/package/example/src/views/CaptureButton.tsx#L189-L208), and the pan-to-zoom gesture can be found [here](https://github.com/mrousavy/react-native-vision-camera/blob/d8551792e97eaa6fa768f54059ffce054bf748d9/example/src/views/CaptureButton.tsx#L185-L205). They implement a real world use-case, where the maximum zoom value is clamped to a realistic value, and the zoom responds very gracefully by using a logarithmic scale.
-
& Readonly;
*
* Read the [VisionCamera documentation](https://react-native-vision-camera.com/) for more information.
*
- * The `` component's most important (and therefore _required_) properties are:
+ * The `` component's most important properties are:
*
- * * {@linkcode CameraProps.device | device}: Specifies the {@linkcode CameraDevice} to use. Get a {@linkcode CameraDevice} by using the {@linkcode useCameraDevice | useCameraDevice()} hook, or manually by using the {@linkcode CameraDevices.getAvailableCameraDevices CameraDevices.getAvailableCameraDevices()} function.
+ * * {@linkcode CameraProps.device | device}: Specifies the {@linkcode CameraDevice} to use. Get a {@linkcode CameraDevice} by using the {@linkcode useCameraDevice | useCameraDevice(..)} hook, or manually by using the {@linkcode CameraDevices.getAvailableCameraDevices CameraDevices.getAvailableCameraDevices()} function.
* * {@linkcode CameraProps.isActive | isActive}: A boolean value that specifies whether the Camera should actively stream video frames or not. This can be compared to a Video component, where `isActive` specifies whether the video is paused or not. If you fully unmount the `` component instead of using `isActive={false}`, the Camera will take a bit longer to start again.
*
* @example
* ```tsx
* function App() {
- * const devices = useCameraDevices('wide-angle-camera')
- * const device = devices.back
+ * const device = useCameraDevice('back')
*
- * if (device == null) return {
/**
* Get a list of all available camera devices on the current phone.
*
- * If you use Hooks, use the `useCameraDevices()` hook instead.
+ * If you use Hooks, use the `useCameraDevices(..)` hook instead.
*
* * For Camera Devices attached to the phone, it is safe to assume that this will never change.
* * For external Camera Devices (USB cameras, Mac continuity cameras, etc.) the available Camera Devices could change over time when the external Camera device gets plugged in or plugged out, so use {@link addCameraDevicesChangedListener | addCameraDevicesChangedListener(...)} to listen for such changes.
diff --git a/package/src/CameraDevice.ts b/package/src/CameraDevice.ts
index fb1f1ec..c423ac9 100644
--- a/package/src/CameraDevice.ts
+++ b/package/src/CameraDevice.ts
@@ -4,9 +4,9 @@ import type { PixelFormat } from './PixelFormat';
/**
* Represents the camera device position.
*
- * * `"back"`: Indicates that the device is physically located on the back of the system hardware
- * * `"front"`: Indicates that the device is physically located on the front of the system hardware
- * * `"external"`: The camera device is an external camera, and has no fixed facing relative to the device's screen.
+ * * `"back"`: Indicates that the device is physically located on the back of the phone
+ * * `"front"`: Indicates that the device is physically located on the front of the phone
+ * * `"external"`: The camera device is an external camera, and has no fixed facing relative to the phone. (e.g. USB or Continuity Cameras)
*/
export type CameraPosition = 'front' | 'back' | 'external';
@@ -46,7 +46,14 @@ export type AutoFocusSystem = 'contrast-detection' | 'phase-detection' | 'none';
export type VideoStabilizationMode = 'off' | 'standard' | 'cinematic' | 'cinematic-extended' | 'auto';
/**
- * A Camera Device's video format. Do not create instances of this type yourself, only use {@linkcode Camera.getAvailableCameraDevices | Camera.getAvailableCameraDevices()}.
+ * A Camera Device's stream-configuration format.
+ *
+ * A format specifies:
+ * - Video Resolution (`videoWidth`/`videoHeight`)
+ * - Photo Resolution (`photoWidth`/`photoHeight`)
+ * - Possible FPS ranges (`fps`)
+ * - Video Stabilization Modes (`videoStabilizationModes`)
+ * - Pixel Formats (`pixelFormats`)
*/
export interface CameraDeviceFormat {
/**
@@ -134,7 +141,13 @@ export interface CameraDevice {
*/
physicalDevices: PhysicalCameraDeviceType[];
/**
- * Specifies the physical position of this camera. (back or front)
+ * Specifies the physical position of this camera.
+ * - `back`: The Camera Device is located on the back of the phone. These devices can be used for capturing what's in front of the user.
+ * - `front`: The Camera Device is located on the front of the phone. These devices can be used for selfies or FaceTime.
+ * - `external`: The Camera Device is an external device. These devices can be either:
+ * - USB Camera Devices (if they support the [USB Video Class (UVC) Specification](https://en.wikipedia.org/wiki/List_of_USB_video_class_devices))
+ * - [Continuity Camera Devices](https://support.apple.com/en-us/HT213244) (e.g. your iPhone's or Mac's Camera connected through WiFi/Continuity)
+ * - Bluetooth/WiFi Camera Devices (if they are supported in the platform-native Camera APIs; Camera2 and AVFoundation)
*/
position: CameraPosition;
/**
diff --git a/package/src/CameraProps.ts b/package/src/CameraProps.ts
index 1df0005..65543dd 100644
--- a/package/src/CameraProps.ts
+++ b/package/src/CameraProps.ts
@@ -22,9 +22,9 @@ export interface CameraProps extends ViewProps {
*
* @example
* ```tsx
- * const devices = useCameraDevices('wide-angle-camera')
- * const device = devices.back
+ * const device = useCameraDevice('back')
*
+ * if (device == null) return {
+ * 'worklet'
+ * console.log(`Frame: ${frame.width}x${frame.height} (${frame.pixelFormat})`)
+ * }, [])
+ * ```
*/
export interface Frame {
/**
- * Whether the underlying buffer is still valid or not. The buffer will be released after the frame processor returns, or `close()` is called.
+ * Whether the underlying buffer is still valid or not.
+ * A Frame is valid as long as your Frame Processor (or a `runAsync(..)` operation) is still running
*/
isValid: boolean;
/**
@@ -37,7 +47,7 @@ export interface Frame {
* Represents the orientation of the Frame.
*
* Some ML Models are trained for specific orientations, so they need to be taken into
- * consideration when running a frame processor. See also: `isMirrored`
+ * consideration when running a frame processor. See also: {@linkcode isMirrored}
*/
orientation: Orientation;
/**
@@ -47,8 +57,21 @@ export interface Frame {
/**
* Get the underlying data of the Frame as a uint8 array buffer.
+ * The format of the buffer depends on the Frame's {@linkcode pixelFormat}.
*
* Note that Frames are allocated on the GPU, so calling `toArrayBuffer()` will copy from the GPU to the CPU.
+ *
+ * @example
+ * ```ts
+ * const frameProcessor = useFrameProcessor((frame) => {
+ * 'worklet'
+ *
+ * if (frame.pixelFormat === 'rgb') {
+ * const data = frame.toArrayBuffer()
+ * console.log(`Pixel at 0,0: RGB(${data[0]}, ${data[1]}, ${data[2]})`)
+ * }
+ * }, [])
+ * ```
*/
toArrayBuffer(): Uint8Array;
/**
@@ -61,17 +84,20 @@ export interface Frame {
toString(): string;
}
+/** @internal */
export interface FrameInternal extends Frame {
/**
* Increment the Frame Buffer ref-count by one.
*
* This is a private API, do not use this.
+ * @internal
*/
incrementRefCount(): void;
/**
* Increment the Frame Buffer ref-count by one.
*
* This is a private API, do not use this.
+ * @internal
*/
decrementRefCount(): void;
}
diff --git a/package/src/PhotoFile.ts b/package/src/PhotoFile.ts
index 9c87a77..419d303 100644
--- a/package/src/PhotoFile.ts
+++ b/package/src/PhotoFile.ts
@@ -25,7 +25,7 @@ export interface TakePhotoOptions {
*/
enableAutoRedEyeReduction?: boolean;
/**
- * Indicates whether still image stabilization will be employed when capturing the photo
+ * Indicates whether still image stabilization will be enabled when capturing the photo
*
* @default false
*/
diff --git a/package/src/devices/getCameraDevice.ts b/package/src/devices/getCameraDevice.ts
index 61c2f21..b752d66 100644
--- a/package/src/devices/getCameraDevice.ts
+++ b/package/src/devices/getCameraDevice.ts
@@ -24,39 +24,57 @@ export interface DeviceFilter {
}
/**
- * Get the best matching Camera device that satisfies your requirements using a sorting filter.
- * @param devices All available Camera Devices this function will use for filtering. To get devices, use `Camera.getAvailableCameraDevices()`.
- * @param filter The filter you want to use. The device that matches your filter the closest will be returned.
- * @returns The device that matches your filter the closest.
+ * Get the best matching Camera device that best satisfies your requirements using a sorting filter.
+ * @param position The position of the Camera device relative to the phone.
+ * @param filter The filter you want to use. The Camera device that matches your filter the closest will be returned
+ * @returns The Camera device that matches your filter the closest, or `undefined` if no such Camera Device exists on the given {@linkcode position}.
+ * @example
+ * ```ts
+ * const devices = Camera.getAvailableCameraDevices()
+ * const device = getCameraDevice(devices, 'back', {
+ * physicalDevices: ['wide-angle-camera']
+ * })
+ * ```
*/
export function getCameraDevice(devices: CameraDevice[], position: CameraPosition, filter: DeviceFilter = {}): CameraDevice {
+ const explicitlyWantsNonWideAngle = filter.physicalDevices != null && !filter.physicalDevices.includes('wide-angle-camera');
+
const filtered = devices.filter((d) => d.position === position);
- const sortedDevices = filtered.sort((left, right) => {
+
+ let bestDevice = filtered[0];
+ if (bestDevice == null) throw new CameraRuntimeError('device/invalid-device', 'No Camera Device could be found!');
+
+ // Compare each device using a point scoring system
+ for (const device of devices) {
let leftPoints = 0;
let rightPoints = 0;
// prefer higher hardware-level
- if (left.hardwareLevel === 'full') leftPoints += 4;
- if (right.hardwareLevel === 'full') rightPoints += 4;
+ if (bestDevice.hardwareLevel === 'full') leftPoints += 4;
+ if (device.hardwareLevel === 'full') rightPoints += 4;
+
+ if (!explicitlyWantsNonWideAngle) {
+ // prefer wide-angle-camera as a default
+ if (bestDevice.physicalDevices.includes('wide-angle-camera')) leftPoints += 1;
+ if (device.physicalDevices.includes('wide-angle-camera')) rightPoints += 1;
+ }
// compare devices. two possible scenarios:
// 1. user wants all cameras ([ultra-wide, wide, tele]) to zoom. prefer those devices that have all 3 cameras.
// 2. user wants only one ([wide]) for faster performance. prefer those devices that only have one camera, if they have more, we rank them lower.
if (filter.physicalDevices != null) {
- for (const device of left.physicalDevices) {
- if (filter.physicalDevices.includes(device)) leftPoints += 1;
+ for (const d of bestDevice.physicalDevices) {
+ if (filter.physicalDevices.includes(d)) leftPoints += 1;
else leftPoints -= 1;
}
- for (const device of right.physicalDevices) {
- if (filter.physicalDevices.includes(device)) rightPoints += 1;
+ for (const d of device.physicalDevices) {
+ if (filter.physicalDevices.includes(d)) rightPoints += 1;
else rightPoints -= 1;
}
}
- return leftPoints - rightPoints;
- });
+ if (rightPoints > leftPoints) bestDevice = device;
+ }
- const device = sortedDevices[0];
- if (device == null) throw new CameraRuntimeError('device/invalid-device', 'No Camera Device could be found!');
- return device;
+ return bestDevice;
}
diff --git a/package/src/devices/getCameraFormat.ts b/package/src/devices/getCameraFormat.ts
index fbface2..ec3744c 100644
--- a/package/src/devices/getCameraFormat.ts
+++ b/package/src/devices/getCameraFormat.ts
@@ -12,12 +12,12 @@ export interface FormatFilter {
* The target resolution of the video (and frame processor) output pipeline.
* If no format supports the given resolution, the format closest to this value will be used.
*/
- videoResolution?: Size;
+ videoResolution?: Size | 'max';
/**
* The target resolution of the photo output pipeline.
* If no format supports the given resolution, the format closest to this value will be used.
*/
- photoResolution?: Size;
+ photoResolution?: Size | 'max';
/**
* The target aspect ratio of the video (and preview) output, expressed as a factor: `width / height`.
*
@@ -58,6 +58,14 @@ export interface FormatFilter {
* If no format supports the target pixel format, the best other matching format will be used.
*/
pixelFormat?: PixelFormat;
+ /**
+ * Whether you want to find a format that supports Photo HDR.
+ */
+ photoHDR?: boolean;
+ /**
+ * Whether you want to find a format that supports Photo HDR.
+ */
+ videoHDR?: boolean;
}
type FilterWithPriority = {
@@ -84,96 +92,121 @@ function filtersToFilterMap(filters: FormatFilter[]): FilterMap {
/**
* Get the best matching Camera format for the given device that satisfies your requirements using a sorting filter. By default, formats are sorted by highest to lowest resolution.
+ *
+ * The {@linkcode filters | filters} are ranked by priority, from highest to lowest.
+ * This means the first item you pass will have a higher priority than the second, and so on.
+ *
* @param device The Camera Device you're currently using
- * @param filters The filter you want to use. The format that matches your filter the closest will be returned. The filter is ranked by priority, descending.
+ * @param filter The filter you want to use. The format that matches your filter the closest will be returned
* @returns The format that matches your filter the closest.
+ *
+ * @example
+ * ```ts
+ * const format = getCameraFormat(device, [
+ * { videoResolution: { width: 3048, height: 2160 } },
+ * { fps: 60 }
+ * ])
+ * ```
*/
export function getCameraFormat(device: CameraDevice, filters: FormatFilter[]): CameraDeviceFormat {
// Combine filters into a single filter map for constant-time lookup
const filter = filtersToFilterMap(filters);
- // Sort list because we will pick first element
- // TODO: Use reduce instead of sort?
- const copy = [...device.formats];
- const sortedFormats = copy.sort((left, right) => {
+ let bestFormat = device.formats[0];
+ if (bestFormat == null)
+ throw new CameraRuntimeError('device/invalid-device', `The given Camera Device (${device.id}) does not have any formats!`);
+
+ // Compare each format using a point scoring system
+ for (const format of device.formats) {
let leftPoints = 0;
let rightPoints = 0;
- const leftVideoResolution = left.videoWidth * left.videoHeight;
- const rightVideoResolution = right.videoWidth * right.videoHeight;
+ const leftVideoResolution = bestFormat.videoWidth * bestFormat.videoHeight;
+ const rightVideoResolution = format.videoWidth * format.videoHeight;
if (filter.videoResolution != null) {
- // Find video resolution closest to the filter (ignoring orientation)
- const targetResolution = filter.videoResolution.target.width * filter.videoResolution.target.height;
- const leftDiff = Math.abs(leftVideoResolution - targetResolution);
- const rightDiff = Math.abs(rightVideoResolution - targetResolution);
- if (leftDiff < rightDiff) leftPoints += filter.videoResolution.priority;
- else if (rightDiff < leftDiff) rightPoints += filter.videoResolution.priority;
- } else {
- // No filter is set, so just prefer higher resolutions
- if (leftVideoResolution > rightVideoResolution) leftPoints++;
- else if (rightVideoResolution > leftVideoResolution) rightPoints++;
+ if (filter.videoResolution.target === 'max') {
+ // We just want the maximum resolution
+ if (leftVideoResolution > rightVideoResolution) leftPoints += filter.videoResolution.priority;
+ if (rightVideoResolution > leftVideoResolution) rightPoints += filter.videoResolution.priority;
+ } else {
+ // Find video resolution closest to the filter (ignoring orientation)
+ const targetResolution = filter.videoResolution.target.width * filter.videoResolution.target.height;
+ const leftDiff = Math.abs(leftVideoResolution - targetResolution);
+ const rightDiff = Math.abs(rightVideoResolution - targetResolution);
+ if (leftDiff < rightDiff) leftPoints += filter.videoResolution.priority;
+ if (rightDiff < leftDiff) rightPoints += filter.videoResolution.priority;
+ }
}
- const leftPhotoResolution = left.photoWidth * left.photoHeight;
- const rightPhotoResolution = right.photoWidth * right.photoHeight;
+ const leftPhotoResolution = bestFormat.photoWidth * bestFormat.photoHeight;
+ const rightPhotoResolution = format.photoWidth * format.photoHeight;
if (filter.photoResolution != null) {
- // Find closest photo resolution to the filter (ignoring orientation)
- const targetResolution = filter.photoResolution.target.width * filter.photoResolution.target.height;
- const leftDiff = Math.abs(leftPhotoResolution - targetResolution);
- const rightDiff = Math.abs(rightPhotoResolution - targetResolution);
- if (leftDiff < rightDiff) leftPoints += filter.photoResolution.priority;
- else if (rightDiff < leftDiff) rightPoints += filter.photoResolution.priority;
- } else {
- // No filter is set, so just prefer higher resolutions
- if (leftPhotoResolution > rightPhotoResolution) leftPoints++;
- else if (rightPhotoResolution > leftPhotoResolution) rightPoints++;
+ if (filter.photoResolution.target === 'max') {
+ // We just want the maximum resolution
+ if (leftPhotoResolution > rightPhotoResolution) leftPoints += filter.photoResolution.priority;
+ if (rightPhotoResolution > leftPhotoResolution) rightPoints += filter.photoResolution.priority;
+ } else {
+ // Find closest photo resolution to the filter (ignoring orientation)
+ const targetResolution = filter.photoResolution.target.width * filter.photoResolution.target.height;
+ const leftDiff = Math.abs(leftPhotoResolution - targetResolution);
+ const rightDiff = Math.abs(rightPhotoResolution - targetResolution);
+ if (leftDiff < rightDiff) leftPoints += filter.photoResolution.priority;
+ if (rightDiff < leftDiff) rightPoints += filter.photoResolution.priority;
+ }
}
// Find closest aspect ratio (video)
if (filter.videoAspectRatio != null) {
- const leftAspect = left.videoWidth / right.videoHeight;
- const rightAspect = right.videoWidth / right.videoHeight;
+ const leftAspect = bestFormat.videoWidth / bestFormat.videoHeight;
+ const rightAspect = format.videoWidth / format.videoHeight;
const leftDiff = Math.abs(leftAspect - filter.videoAspectRatio.target);
const rightDiff = Math.abs(rightAspect - filter.videoAspectRatio.target);
if (leftDiff < rightDiff) leftPoints += filter.videoAspectRatio.priority;
- else if (rightDiff < leftDiff) rightPoints += filter.videoAspectRatio.priority;
+ if (rightDiff < leftDiff) rightPoints += filter.videoAspectRatio.priority;
}
// Find closest aspect ratio (photo)
if (filter.photoAspectRatio != null) {
- const leftAspect = left.photoWidth / right.photoHeight;
- const rightAspect = right.photoWidth / right.photoHeight;
+ const leftAspect = bestFormat.photoWidth / bestFormat.photoHeight;
+ const rightAspect = format.photoWidth / format.photoHeight;
const leftDiff = Math.abs(leftAspect - filter.photoAspectRatio.target);
const rightDiff = Math.abs(rightAspect - filter.photoAspectRatio.target);
if (leftDiff < rightDiff) leftPoints += filter.photoAspectRatio.priority;
- else if (rightDiff < leftDiff) rightPoints += filter.photoAspectRatio.priority;
+ if (rightDiff < leftDiff) rightPoints += filter.photoAspectRatio.priority;
}
// Find closest max FPS
if (filter.fps != null) {
- const leftDiff = Math.abs(left.maxFps - filter.fps.target);
- const rightDiff = Math.abs(right.maxFps - filter.fps.target);
- if (leftDiff < rightDiff) leftPoints += filter.fps.priority;
- else if (rightDiff < leftDiff) rightPoints += filter.fps.priority;
+ if (bestFormat.maxFps >= filter.fps.target) leftPoints += filter.fps.priority;
+ if (format.maxFps >= filter.fps.target) rightPoints += filter.fps.priority;
}
// Find video stabilization mode
if (filter.videoStabilizationMode != null) {
- if (left.videoStabilizationModes.includes(filter.videoStabilizationMode.target)) leftPoints++;
- if (right.videoStabilizationModes.includes(filter.videoStabilizationMode.target)) rightPoints++;
+ if (bestFormat.videoStabilizationModes.includes(filter.videoStabilizationMode.target)) leftPoints++;
+ if (format.videoStabilizationModes.includes(filter.videoStabilizationMode.target)) rightPoints++;
}
// Find pixel format
if (filter.pixelFormat != null) {
- if (left.pixelFormats.includes(filter.pixelFormat.target)) leftPoints++;
- if (right.pixelFormats.includes(filter.pixelFormat.target)) rightPoints++;
+ if (bestFormat.pixelFormats.includes(filter.pixelFormat.target)) leftPoints++;
+ if (format.pixelFormats.includes(filter.pixelFormat.target)) rightPoints++;
}
- return rightPoints - leftPoints;
- });
+ // Find Photo HDR formats
+ if (filter.photoHDR != null) {
+ if (bestFormat.supportsPhotoHDR === filter.photoHDR.target) leftPoints++;
+ if (format.supportsPhotoHDR === filter.photoHDR.target) rightPoints++;
+ }
- const format = sortedFormats[0];
- if (format == null)
- throw new CameraRuntimeError('device/invalid-device', `The given Camera Device (${device.id}) does not have any formats!`);
- return format;
+ // Find Video HDR formats
+ if (filter.videoHDR != null) {
+ if (bestFormat.supportsVideoHDR === filter.videoHDR.target) leftPoints++;
+ if (format.supportsVideoHDR === filter.videoHDR.target) rightPoints++;
+ }
+
+ if (rightPoints > leftPoints) bestFormat = format;
+ }
+
+ return bestFormat;
}
diff --git a/package/src/hooks/useCameraDevice.ts b/package/src/hooks/useCameraDevice.ts
index 22056c5..d2bc174 100644
--- a/package/src/hooks/useCameraDevice.ts
+++ b/package/src/hooks/useCameraDevice.ts
@@ -7,11 +7,10 @@ import { useCameraDevices } from './useCameraDevices';
* Get the best matching Camera device that best satisfies your requirements using a sorting filter.
* @param position The position of the Camera device relative to the phone.
* @param filter The filter you want to use. The Camera device that matches your filter the closest will be returned
- * @returns The Camera device that matches your filter the closest.
+ * @returns The Camera device that matches your filter the closest, or `undefined` if no such Camera Device exists on the given {@linkcode position}.
* @example
* ```ts
- * const [position, setPosition] = useState('back')
- * const device = useCameraDevice(position, {
+ * const device = useCameraDevice('back', {
* physicalDevices: ['wide-angle-camera']
* })
* ```
diff --git a/package/src/index.ts b/package/src/index.ts
index 6548119..b1ead91 100644
--- a/package/src/index.ts
+++ b/package/src/index.ts
@@ -2,7 +2,7 @@ export * from './Camera';
export * from './CameraDevice';
export * from './CameraError';
export * from './CameraProps';
-export { Frame } from './Frame';
+export * from './Frame';
export * from './FrameProcessorPlugins';
export * from './Orientation';
export * from './PhotoFile';