railbird/react-native-vision-camera
10
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: Frame Processors: Allow returning Frames (support for resize and other frame manipulations) (#185)

Browse Source 
* batch

* Init Frame as box

* Use ObjC syntax

* Fix access

* Revert "Fix access"

This reverts commit 7de09e52739d4c2b53f485d5ed696f1665fa5737.

* Revert "Use ObjC syntax"

This reverts commit e33f05ae8451cc4ee24af41d14dc76a57c157554.

* Revert "Init Frame as box"

This reverts commit 5adafb6109bfbf7fddb8ddc4af7d306b7b76b476.

* use holder

* convert buffer <-> jsi object

* add docs

* add more docs

* Update JSIUtils.mm

* Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx

* Update CameraView+RecordVideo.swift
This commit is contained in:
Marc Rousavy
2021-06-08 14:20:07 +02:00
committed by GitHub
parent 1b08c0cbae
commit 4038db2e28
10 changed files with 112 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
docs/docs/guides/FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
View File

@@ -32,15 +32,16 @@ To achieve **maximum performance**, the `scanQRCodes` function is written in a n
The Frame Processor Plugin Registry API automatically manages type conversion from JS <-> native. They are converted into the most efficient data-structures, as seen here:
| JS Type | Objective-C Type | Java Type |
|----------------------|--------------------------|----------------------------|
| `number` | `NSNumber` (double) | `double` |
| `boolean` | `NSNumber` (boolean) | `boolean` |
| `string` | `NSString` | `String` |
| `[]` | `NSArray` | `Array<Object>` |
| `{}` | `NSDictionary` | `HashMap<Object>` |
| `undefined` | `nil` / `NSNull` | `null` |
| `(any, any) => void` | `RCTResponseSenderBlock` | `(Object, Object) -> void` |
| JS Type | Objective-C Type | Java Type |
|----------------------|---------------------------|----------------------------|
| `number` | `NSNumber*` (double) | `double` |
| `boolean` | `NSNumber*` (boolean) | `boolean` |
| `string` | `NSString*` | `String` |
| `[]` | `NSArray*` | `Array<Object>` |
| `{}` | `NSDictionary*` | `HashMap<Object>` |
| `undefined` / `null` | `nil` / `NSNull` | `null` |
| `(any, any) => void` | `RCTResponseSenderBlock` | `(Object, Object) -> void` |
| `Frame` | `CMSampleBufferRefHolder` | `ImageProxy` |
### Return values
@@ -56,12 +57,36 @@ Returns a `string` in JS:
```js
export function detectObject(frame: Frame): string {
'worklet';
const result = __detectObject(frame);
'worklet'
const result = __detectObject(frame)
_log(result) // <-- "cat"
}
```
You can also manipulate the buffer and return it (or a copy) by using the `CMSampleBufferRefHolder` class:
```objc
static inline id resize(CMSampleBufferRef buffer, NSArray args) {
NSNumber* width = [arguments objectAtIndex:0];
NSNumber* height = [arguments objectAtIndex:1];
CMSampleBufferRef resizedBuffer = CMSampleBufferCopyAndResize(buffer, width, height);
return [[CMSampleBufferRefHolder alloc] initWithBuffer:resizedBuffer];
}
```
Which returns a `Frame` in JS:
```js
const frameProcessor = useFrameProcessor((frame) => {
'worklet';
// by downscaling the frame, the `detectObjects` function runs faster.
const resizedFrame = resize(frame, 720, 480)
const objects = detectObjects(resizedFrame)
_log(objects)
}, [])
```
### Parameters
Frame Processors can also accept parameters, following the same type convention as [return values](#return-values):