feat: Full Android rewrite (CameraX -> Camera2) (#1674)
* Nuke CameraX * fix: Run View Finder on UI Thread * Open Camera, set up Threads * fix init * Mirror if needed * Try PreviewView * Use max resolution * Add `hardwareLevel` property * Check if output type is supported * Replace `frameRateRanges` with `minFps` and `maxFps` * Remove `isHighestPhotoQualitySupported` * Remove `colorSpace` The native platforms will use the best / most accurate colorSpace by default anyways. * HDR * Check from format * fix * Remove `supportsParallelVideoProcessing` * Correctly return video/photo sizes on Android now. Finally * Log all Device props * Log if optimized usecase is used * Cleanup * Configure Camera Input only once * Revert "Configure Camera Input only once" This reverts commit 0fd6c03f54c7566cb5592053720c4a8743aba92e. * Extract Camera configuration * Try to reconfigure all * Hook based * Properly set up `CameraSession` * Delete unused * fix: Fix recreate when outputs change * Update NativePreviewView.kt * Use callback for closing * Catch CameraAccessException * Finally got it stable * Remove isMirrored * Implement `takePhoto()` * Add ExifInterface library * Run findViewById on UI Thread * Add Photo Output Surface to takePhoto * Fix Video Stabilization Modes * Optimize Imports * More logs * Update CameraSession.kt * Close Image * Use separate Executor in CameraQueue * Delete hooks * Use same Thread again * If opened, call error * Update CameraSession.kt * Log HW level * fix: Don't enable Stream Use Case if it's not 100% supported * Move some stuff * Cleanup PhotoOutputSynchronizer * Try just open in suspend fun * Some synchronization fixes * fix logs * Update CameraDevice+createCaptureSession.kt * Update CameraDevice+createCaptureSession.kt * fixes * fix: Use Snapshot Template for speed capture prio * Use PREVIEW template for repeating request * Use `TEMPLATE_RECORD` if video use-case is attached * Use `isRunning` flag * Recreate session everytime on active/inactive * Lazily get values in capture session * Stability * Rebuild session if outputs change * Set `didOutputsChange` back to false * Capture first in lock * Try * kinda fix it? idk * fix: Keep Outputs * Refactor into single method * Update CameraView.kt * Use Enums for type safety * Implement Orientation (I think) * Move RefCount management to Java (Frame) * Don't crash when dropping a Frame * Prefer Devices with higher max resolution * Prefer multi-cams * Use FastImage for Media Page * Return orientation in takePhoto() * Load orientation from EXIF Data * Add `isMirrored` props and documentation for PhotoFile * fix: Return `not-determined` on Android * Update CameraViewModule.kt * chore: Upgrade packages * fix: Fix Metro Config * Cleanup config * Properly mirror Images on save * Prepare MediaRecorder * Start/Stop MediaRecorder * Remove `takeSnapshot()` It no longer works on Android and never worked on iOS. Users could use useFrameProcessor to take a Snapshot * Use `MediaCodec` * Move to `VideoRecording` class * Cleanup Snapshot * Create `SkiaPreviewView` hybrid class * Create OpenGL context * Create `SkiaPreviewView` * Fix texture creation missing context * Draw red frame * Somehow get it working * Add Skia CMake setup * Start looping * Init OpenGL * Refactor into `SkiaRenderer` * Cleanup PreviewSize * Set up * Only re-render UI if there is a new Frame * Preview * Fix init * Try rendering Preview * Update SkiaPreviewView.kt * Log version * Try using Skia (fail) * Drawwwww!!!!!!!!!! 🎉 * Use Preview Size * Clear first * Refactor into SkiaRenderer * Add `previewType: "none"` on iOS * Simplify a lot * Draw Camera? For some reason? I have no idea anymore * Fix OpenGL errors * Got it kinda working again? * Actually draw Frame woah * Clean up code * Cleanup * Update on main * Synchronize render calls * holy shit * Update SkiaRenderer.cpp * Update SkiaRenderer.cpp * Refactor * Update SkiaRenderer.cpp * Check for `NO_INPUT_TEXTURE`^ * Post & Wait * Set input size * Add Video back again * Allow session without preview * Convert JPEG to byte[] * feat: Use `ImageReader` and use YUV Image Buffers in Skia Context (#1689) * Try to pass YUV Buffers as Pixmaps * Create pixmap! * Clean up * Render to preview * Only render if we have an output surface * Update SkiaRenderer.cpp * Fix Y+U+V sampling code * Cleanup * Fix Semaphore 0 * Use 4:2:0 YUV again idk * Update SkiaRenderer.h * Set minSdk to 26 * Set surface * Revert "Set minSdk to 26" This reverts commit c4085b7c16c628532e5c2d68cf7ed11c751d0b48. * Set previewType * feat: Video Recording with Camera2 (#1691) * Rename * Update CameraSession.kt * Use `SurfaceHolder` instead of `SurfaceView` for output * Update CameraOutputs.kt * Update CameraSession.kt * fix: Fix crash when Preview is null * Check if snapshot capture is supported * Update RecordingSession.kt * S * Use `MediaRecorder` * Make audio optional * Add Torch * Output duration * Update RecordingSession.kt * Start RecordingSession * logs * More log * Base for preparing pass-through Recording * Use `ImageWriter` to append Images to the Recording Surface * Stream PRIVATE GPU_SAMPLED_IMAGE Images * Add flags * Close session on stop * Allow customizing `videoCodec` and `fileType` * Enable Torch * Fix Torch Mode * Fix comparing outputs with hashCode * Update CameraSession.kt * Correctly pass along Frame Processor * fix: Use AUDIO_BIT_RATE of 16 * 44,1Khz * Use CAMCORDER instead of MIC microphone * Use 1 channel * fix: Use `Orientation` * Add `native` PixelFormat * Update iOS to latest Skia integration * feat: Add `pixelFormat` property to Camera * Catch error in configureSession * Fix JPEG format * Clean up best match finder * Update CameraDeviceDetails.kt * Clamp sizes by maximum CamcorderProfile size * Remove `getAvailableVideoCodecs` * chore: release 3.0.0-rc.5 * Use maximum video size of RECORD as default * Update CameraDeviceDetails.kt * Add a todo * Add JSON device to issue report * Prefer `full` devices and flash * Lock to 30 FPS on Samsung * Implement Zoom * Refactor * Format -> PixelFormat * fix: Feat `pixelFormat` -> `pixelFormats` * Update TROUBLESHOOTING.mdx * Format * fix: Implement `zoom` for Photo Capture * fix: Don't run if `isActive` is `false` * fix: Call `examplePlugin(frame)` * fix: Fix Flash * fix: Use `react-native-worklets-core`! * fix: Fix import
This commit is contained in:
Marc Rousavy
GitHub
@@ -8,7 +8,7 @@
|
||||
#include <fbjni/fbjni.h>
|
||||
#include <jni.h>
|
||||
|
||||
#include <react-native-worklets/WKTJsiHostObject.h>
|
||||
#include <react-native-worklets-core/WKTJsiHostObject.h>
|
||||
#include "JSITypedArray.h"
|
||||
|
||||
#include <vector>
|
||||
@@ -18,7 +18,7 @@ namespace vision {
|
||||
|
||||
using namespace facebook;
|
||||
|
||||
FrameHostObject::FrameHostObject(const jni::alias_ref<JFrame::javaobject>& frame): frame(make_global(frame)), _refCount(0) { }
|
||||
FrameHostObject::FrameHostObject(const jni::alias_ref<JFrame::javaobject>& frame): frame(make_global(frame)) { }
|
||||
|
||||
FrameHostObject::~FrameHostObject() {
|
||||
// Hermes' Garbage Collector (Hades GC) calls destructors on a separate Thread
|
||||
@@ -37,6 +37,7 @@ std::vector<jsi::PropNameID> FrameHostObject::getPropertyNames(jsi::Runtime& rt)
|
||||
result.push_back(jsi::PropNameID::forUtf8(rt, std::string("orientation")));
|
||||
result.push_back(jsi::PropNameID::forUtf8(rt, std::string("isMirrored")));
|
||||
result.push_back(jsi::PropNameID::forUtf8(rt, std::string("timestamp")));
|
||||
result.push_back(jsi::PropNameID::forUtf8(rt, std::string("pixelFormat")));
|
||||
// Conversion
|
||||
result.push_back(jsi::PropNameID::forUtf8(rt, std::string("toString")));
|
||||
result.push_back(jsi::PropNameID::forUtf8(rt, std::string("toArrayBuffer")));
|
||||
@@ -94,8 +95,7 @@ jsi::Value FrameHostObject::get(jsi::Runtime& runtime, const jsi::PropNameID& pr
|
||||
if (name == "incrementRefCount") {
|
||||
auto incrementRefCount = JSI_HOST_FUNCTION_LAMBDA {
|
||||
// Increment retain count by one.
|
||||
std::lock_guard lock(this->_refCountMutex);
|
||||
this->_refCount++;
|
||||
this->frame->incrementRefCount();
|
||||
return jsi::Value::undefined();
|
||||
};
|
||||
return jsi::Function::createFromHostFunction(runtime,
|
||||
@@ -106,12 +106,8 @@ jsi::Value FrameHostObject::get(jsi::Runtime& runtime, const jsi::PropNameID& pr
|
||||
|
||||
if (name == "decrementRefCount") {
|
||||
auto decrementRefCount = JSI_HOST_FUNCTION_LAMBDA {
|
||||
// Decrement retain count by one. If the retain count is zero, we close the Frame.
|
||||
std::lock_guard lock(this->_refCountMutex);
|
||||
this->_refCount--;
|
||||
if (_refCount < 1) {
|
||||
this->frame->close();
|
||||
}
|
||||
// Decrement retain count by one. If the retain count is zero, the Frame gets closed.
|
||||
this->frame->decrementRefCount();
|
||||
return jsi::Value::undefined();
|
||||
};
|
||||
return jsi::Function::createFromHostFunction(runtime,
|
||||
@@ -136,6 +132,10 @@ jsi::Value FrameHostObject::get(jsi::Runtime& runtime, const jsi::PropNameID& pr
|
||||
auto string = this->frame->getOrientation();
|
||||
return jsi::String::createFromUtf8(runtime, string->toStdString());
|
||||
}
|
||||
if (name == "pixelFormat") {
|
||||
auto string = this->frame->getPixelFormat();
|
||||
return jsi::String::createFromUtf8(runtime, string->toStdString());
|
||||
}
|
||||
if (name == "timestamp") {
|
||||
return jsi::Value(static_cast<double>(this->frame->getTimestamp()));
|
||||
}
|
||||
|
||||
@@ -9,7 +9,6 @@
|
||||
#include <fbjni/fbjni.h>
|
||||
#include <vector>
|
||||
#include <string>
|
||||
#include <mutex>
|
||||
|
||||
#include "java-bindings/JFrame.h"
|
||||
|
||||
@@ -31,9 +30,6 @@ class JSI_EXPORT FrameHostObject : public jsi::HostObject {
|
||||
|
||||
private:
|
||||
static auto constexpr TAG = "VisionCamera";
|
||||
|
||||
size_t _refCount;
|
||||
std::mutex _refCountMutex;
|
||||
};
|
||||
|
||||
} // namespace vision
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
#include "java-bindings/JFrameProcessor.h"
|
||||
#include "java-bindings/JVisionCameraProxy.h"
|
||||
#include "VisionCameraProxy.h"
|
||||
#include "skia/SkiaRenderer.h"
|
||||
|
||||
JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *vm, void *) {
|
||||
return facebook::jni::initialize(vm, [] {
|
||||
@@ -11,5 +12,6 @@ JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *vm, void *) {
|
||||
vision::JFrameProcessor::registerNatives();
|
||||
vision::JVisionCameraProxy::registerNatives();
|
||||
vision::JVisionCameraScheduler::registerNatives();
|
||||
vision::SkiaRenderer::registerNatives();
|
||||
});
|
||||
}
|
||||
|
||||
@@ -42,6 +42,11 @@ local_ref<JString> JFrame::getOrientation() const {
|
||||
return getOrientationMethod(self());
|
||||
}
|
||||
|
||||
local_ref<JString> JFrame::getPixelFormat() const {
|
||||
static const auto getPixelFormatMethod = getClass()->getMethod<JString()>("getPixelFormat");
|
||||
return getPixelFormatMethod(self());
|
||||
}
|
||||
|
||||
int JFrame::getPlanesCount() const {
|
||||
static const auto getPlanesCountMethod = getClass()->getMethod<jint()>("getPlanesCount");
|
||||
return getPlanesCountMethod(self());
|
||||
@@ -57,6 +62,16 @@ local_ref<JArrayByte> JFrame::toByteArray() const {
|
||||
return toByteArrayMethod(self());
|
||||
}
|
||||
|
||||
void JFrame::incrementRefCount() {
|
||||
static const auto incrementRefCountMethod = getClass()->getMethod<void()>("incrementRefCount");
|
||||
incrementRefCountMethod(self());
|
||||
}
|
||||
|
||||
void JFrame::decrementRefCount() {
|
||||
static const auto decrementRefCountMethod = getClass()->getMethod<void()>("decrementRefCount");
|
||||
decrementRefCountMethod(self());
|
||||
}
|
||||
|
||||
void JFrame::close() {
|
||||
static const auto closeMethod = getClass()->getMethod<void()>("close");
|
||||
closeMethod(self());
|
||||
|
||||
@@ -24,7 +24,10 @@ struct JFrame : public JavaClass<JFrame> {
|
||||
int getBytesPerRow() const;
|
||||
jlong getTimestamp() const;
|
||||
local_ref<JString> getOrientation() const;
|
||||
local_ref<JString> getPixelFormat() const;
|
||||
local_ref<JArrayByte> toByteArray() const;
|
||||
void incrementRefCount();
|
||||
void decrementRefCount();
|
||||
void close();
|
||||
};
|
||||
|
||||
|
||||
@@ -9,8 +9,8 @@
|
||||
#include <jni.h>
|
||||
#include <fbjni/fbjni.h>
|
||||
|
||||
#include <react-native-worklets/WKTJsiWorklet.h>
|
||||
#include <react-native-worklets/WKTJsiHostObject.h>
|
||||
#include <react-native-worklets-core/WKTJsiWorklet.h>
|
||||
#include <react-native-worklets-core/WKTJsiHostObject.h>
|
||||
|
||||
#include "JFrame.h"
|
||||
#include "FrameHostObject.h"
|
||||
|
||||
@@ -11,8 +11,8 @@
|
||||
#include <jsi/jsi.h>
|
||||
#include <react/jni/ReadableNativeMap.h>
|
||||
|
||||
#include <react-native-worklets/WKTJsiWorklet.h>
|
||||
#include <react-native-worklets/WKTJsiWorkletContext.h>
|
||||
#include <react-native-worklets-core/WKTJsiWorklet.h>
|
||||
#include <react-native-worklets-core/WKTJsiWorkletContext.h>
|
||||
|
||||
#include "FrameProcessorPluginHostObject.h"
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
#include <fbjni/fbjni.h>
|
||||
#include <jsi/jsi.h>
|
||||
#include <react-native-worklets/WKTJsiWorkletContext.h>
|
||||
#include <react-native-worklets-core/WKTJsiWorkletContext.h>
|
||||
#include <react/jni/ReadableNativeMap.h>
|
||||
|
||||
#include "JFrameProcessorPlugin.h"
|
||||
|
||||
26
android/src/main/cpp/skia/OpenGLError.h
Normal file
26
android/src/main/cpp/skia/OpenGLError.h
Normal file
@@ -0,0 +1,26 @@
|
||||
//
|
||||
// Created by Marc Rousavy on 09.08.23.
|
||||
//
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <string>
|
||||
#include <stdexcept>
|
||||
#include <GLES2/gl2.h>
|
||||
|
||||
namespace vision {
|
||||
|
||||
inline std::string getEglErrorIfAny() {
|
||||
EGLint error = glGetError();
|
||||
if (error != GL_NO_ERROR) return " Error: " + std::to_string(error);
|
||||
error = eglGetError();
|
||||
if (error != EGL_SUCCESS) return " Error: " + std::to_string(error);
|
||||
return "";
|
||||
}
|
||||
|
||||
class OpenGLError: public std::runtime_error {
|
||||
public:
|
||||
explicit OpenGLError(const std::string&& message): std::runtime_error(message + getEglErrorIfAny()) {}
|
||||
};
|
||||
|
||||
} // namespace vision
|
||||
327
android/src/main/cpp/skia/SkiaRenderer.cpp
Normal file
327
android/src/main/cpp/skia/SkiaRenderer.cpp
Normal file
@@ -0,0 +1,327 @@
|
||||
//
|
||||
// Created by Marc Rousavy on 10.08.23.
|
||||
//
|
||||
|
||||
#include "SkiaRenderer.h"
|
||||
#include <android/log.h>
|
||||
#include "OpenGLError.h"
|
||||
|
||||
#include <core/SkColorSpace.h>
|
||||
#include <core/SkCanvas.h>
|
||||
#include <core/SkYUVAPixmaps.h>
|
||||
|
||||
#include <gpu/gl/GrGLInterface.h>
|
||||
#include <gpu/GrDirectContext.h>
|
||||
#include <gpu/GrBackendSurface.h>
|
||||
#include <gpu/ganesh/SkSurfaceGanesh.h>
|
||||
#include <gpu/ganesh/SkImageGanesh.h>
|
||||
|
||||
#include <android/native_window_jni.h>
|
||||
#include <android/surface_texture_jni.h>
|
||||
|
||||
// from <gpu/ganesh/gl/GrGLDefines.h>
|
||||
#define GR_GL_TEXTURE_EXTERNAL 0x8D65
|
||||
#define GR_GL_RGBA8 0x8058
|
||||
#define ACTIVE_SURFACE_ID 0
|
||||
|
||||
namespace vision {
|
||||
|
||||
|
||||
jni::local_ref<SkiaRenderer::jhybriddata> SkiaRenderer::initHybrid(jni::alias_ref<jhybridobject> javaPart) {
|
||||
return makeCxxInstance(javaPart);
|
||||
}
|
||||
|
||||
SkiaRenderer::SkiaRenderer(const jni::alias_ref<jhybridobject>& javaPart) {
|
||||
_javaPart = jni::make_global(javaPart);
|
||||
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Initializing SkiaRenderer...");
|
||||
|
||||
_previewSurface = nullptr;
|
||||
_previewWidth = 0;
|
||||
_previewHeight = 0;
|
||||
_inputSurfaceTextureId = NO_INPUT_TEXTURE;
|
||||
}
|
||||
|
||||
SkiaRenderer::~SkiaRenderer() {
|
||||
if (_glDisplay != EGL_NO_DISPLAY) {
|
||||
eglMakeCurrent(_glDisplay, EGL_NO_SURFACE, EGL_NO_SURFACE, EGL_NO_CONTEXT);
|
||||
if (_glSurface != EGL_NO_SURFACE) {
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Destroying OpenGL Surface...");
|
||||
eglDestroySurface(_glDisplay, _glSurface);
|
||||
_glSurface = EGL_NO_SURFACE;
|
||||
}
|
||||
if (_glContext != EGL_NO_CONTEXT) {
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Destroying OpenGL Context...");
|
||||
eglDestroyContext(_glDisplay, _glContext);
|
||||
_glContext = EGL_NO_CONTEXT;
|
||||
}
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Destroying OpenGL Display...");
|
||||
eglTerminate(_glDisplay);
|
||||
_glDisplay = EGL_NO_DISPLAY;
|
||||
}
|
||||
if (_skiaContext != nullptr) {
|
||||
_skiaContext->abandonContext();
|
||||
_skiaContext = nullptr;
|
||||
}
|
||||
destroyOutputSurface();
|
||||
}
|
||||
|
||||
void SkiaRenderer::ensureOpenGL(ANativeWindow* surface) {
|
||||
bool successful;
|
||||
// EGLDisplay
|
||||
if (_glDisplay == EGL_NO_DISPLAY) {
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Initializing EGLDisplay..");
|
||||
_glDisplay = eglGetDisplay(EGL_DEFAULT_DISPLAY);
|
||||
if (_glDisplay == EGL_NO_DISPLAY) throw OpenGLError("Failed to get default OpenGL Display!");
|
||||
|
||||
EGLint major;
|
||||
EGLint minor;
|
||||
successful = eglInitialize(_glDisplay, &major, &minor);
|
||||
if (!successful) throw OpenGLError("Failed to initialize OpenGL!");
|
||||
}
|
||||
|
||||
// EGLConfig
|
||||
if (_glConfig == nullptr) {
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Initializing EGLConfig..");
|
||||
EGLint attributes[] = {EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT,
|
||||
EGL_SURFACE_TYPE, EGL_WINDOW_BIT,
|
||||
EGL_ALPHA_SIZE, 8,
|
||||
EGL_BLUE_SIZE, 8,
|
||||
EGL_GREEN_SIZE, 8,
|
||||
EGL_RED_SIZE, 8,
|
||||
EGL_DEPTH_SIZE, 0,
|
||||
EGL_STENCIL_SIZE, 0,
|
||||
EGL_NONE};
|
||||
EGLint numConfigs;
|
||||
successful = eglChooseConfig(_glDisplay, attributes, &_glConfig, 1, &numConfigs);
|
||||
if (!successful || numConfigs == 0) throw OpenGLError("Failed to choose OpenGL config!");
|
||||
}
|
||||
|
||||
// EGLContext
|
||||
if (_glContext == EGL_NO_CONTEXT) {
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Initializing EGLContext..");
|
||||
EGLint contextAttributes[] = {EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE};
|
||||
_glContext = eglCreateContext(_glDisplay, _glConfig, nullptr, contextAttributes);
|
||||
if (_glContext == EGL_NO_CONTEXT) throw OpenGLError("Failed to create OpenGL context!");
|
||||
}
|
||||
|
||||
// EGLSurface
|
||||
if (_glSurface == EGL_NO_SURFACE) {
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Initializing EGLSurface..");
|
||||
_glSurface = eglCreateWindowSurface(_glDisplay, _glConfig, surface, nullptr);
|
||||
_skiaContext = GrDirectContext::MakeGL();
|
||||
}
|
||||
|
||||
successful = eglMakeCurrent(_glDisplay, _glSurface, _glSurface, _glContext);
|
||||
if (!successful || eglGetError() != EGL_SUCCESS) throw OpenGLError("Failed to use current OpenGL context!");
|
||||
}
|
||||
|
||||
void SkiaRenderer::setOutputSurface(jobject previewSurface) {
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Setting Output Surface..");
|
||||
destroyOutputSurface();
|
||||
|
||||
_previewSurface = ANativeWindow_fromSurface(jni::Environment::current(), previewSurface);
|
||||
_glSurface = EGL_NO_SURFACE;
|
||||
}
|
||||
|
||||
void SkiaRenderer::destroyOutputSurface() {
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Destroying Output Surface..");
|
||||
if (_glSurface != EGL_NO_SURFACE) {
|
||||
eglDestroySurface(_glDisplay, _glSurface);
|
||||
_glSurface = EGL_NO_SURFACE;
|
||||
if (_skiaContext != nullptr) {
|
||||
_skiaContext->abandonContext();
|
||||
_skiaContext = nullptr;
|
||||
}
|
||||
}
|
||||
if (_previewSurface != nullptr) {
|
||||
ANativeWindow_release(_previewSurface);
|
||||
_previewSurface = nullptr;
|
||||
}
|
||||
}
|
||||
|
||||
void SkiaRenderer::setOutputSurfaceSize(int width, int height) {
|
||||
_previewWidth = width;
|
||||
_previewHeight = height;
|
||||
}
|
||||
|
||||
void SkiaRenderer::setInputTextureSize(int width, int height) {
|
||||
_inputWidth = width;
|
||||
_inputHeight = height;
|
||||
}
|
||||
|
||||
void SkiaRenderer::renderLatestFrameToPreview() {
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "renderLatestFrameToPreview()");
|
||||
if (_previewSurface == nullptr) {
|
||||
throw std::runtime_error("Cannot render latest frame to preview without a preview surface! "
|
||||
"renderLatestFrameToPreview() needs to be called after setPreviewSurface().");
|
||||
}
|
||||
return;
|
||||
if (_inputSurfaceTextureId == NO_INPUT_TEXTURE) {
|
||||
throw std::runtime_error("Cannot render latest frame to preview without an input texture! "
|
||||
"renderLatestFrameToPreview() needs to be called after prepareInputTexture().");
|
||||
}
|
||||
ensureOpenGL(_previewSurface);
|
||||
|
||||
if (_skiaContext == nullptr) {
|
||||
_skiaContext = GrDirectContext::MakeGL();
|
||||
}
|
||||
_skiaContext->resetContext();
|
||||
|
||||
GrGLTextureInfo textureInfo {
|
||||
// OpenGL will automatically convert YUV -> RGB because it's an EXTERNAL texture
|
||||
.fTarget = GR_GL_TEXTURE_EXTERNAL,
|
||||
.fID = _inputSurfaceTextureId,
|
||||
.fFormat = GR_GL_RGBA8,
|
||||
.fProtected = skgpu::Protected::kNo,
|
||||
};
|
||||
GrBackendTexture texture(_inputWidth,
|
||||
_inputHeight,
|
||||
GrMipMapped::kNo,
|
||||
textureInfo);
|
||||
sk_sp<SkImage> frame = SkImages::AdoptTextureFrom(_skiaContext.get(),
|
||||
texture,
|
||||
kTopLeft_GrSurfaceOrigin,
|
||||
kN32_SkColorType,
|
||||
kOpaque_SkAlphaType);
|
||||
|
||||
GrGLFramebufferInfo fboInfo {
|
||||
// FBO #0 is the currently active OpenGL Surface (eglMakeCurrent)
|
||||
.fFBOID = ACTIVE_SURFACE_ID,
|
||||
.fFormat = GR_GL_RGBA8,
|
||||
.fProtected = skgpu::Protected::kNo,
|
||||
};;
|
||||
GrBackendRenderTarget renderTarget(_previewWidth,
|
||||
_previewHeight,
|
||||
0,
|
||||
8,
|
||||
fboInfo);
|
||||
SkSurfaceProps props(0, kUnknown_SkPixelGeometry);
|
||||
sk_sp<SkSurface> surface = SkSurfaces::WrapBackendRenderTarget(_skiaContext.get(),
|
||||
renderTarget,
|
||||
kTopLeft_GrSurfaceOrigin,
|
||||
kN32_SkColorType,
|
||||
nullptr,
|
||||
&props);
|
||||
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Rendering %ix%i Frame to %ix%i Preview..", frame->width(), frame->height(), surface->width(), surface->height());
|
||||
|
||||
auto canvas = surface->getCanvas();
|
||||
|
||||
canvas->clear(SkColors::kBlack);
|
||||
|
||||
auto duration = std::chrono::system_clock::now().time_since_epoch();
|
||||
auto millis = std::chrono::duration_cast<std::chrono::milliseconds>(duration).count();
|
||||
|
||||
canvas->drawImage(frame, 0, 0);
|
||||
|
||||
// TODO: Run Skia Frame Processor
|
||||
auto rect = SkRect::MakeXYWH(150, 250, millis % 3000 / 10, millis % 3000 / 10);
|
||||
auto paint = SkPaint();
|
||||
paint.setColor(SkColors::kRed);
|
||||
canvas->drawRect(rect, paint);
|
||||
|
||||
// Flush
|
||||
canvas->flush();
|
||||
|
||||
bool successful = eglSwapBuffers(_glDisplay, _glSurface);
|
||||
if (!successful || eglGetError() != EGL_SUCCESS) throw OpenGLError("Failed to swap OpenGL buffers!");
|
||||
}
|
||||
|
||||
|
||||
void SkiaRenderer::renderCameraFrameToOffscreenCanvas(jni::JByteBuffer yBuffer,
|
||||
jni::JByteBuffer uBuffer,
|
||||
jni::JByteBuffer vBuffer) {
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Begin render...");
|
||||
ensureOpenGL(_previewSurface);
|
||||
if (_skiaContext == nullptr) {
|
||||
_skiaContext = GrDirectContext::MakeGL();
|
||||
}
|
||||
_skiaContext->resetContext();
|
||||
|
||||
// See https://en.wikipedia.org/wiki/Chroma_subsampling - we're in 4:2:0
|
||||
size_t bytesPerRow = sizeof(uint8_t) * _inputWidth;
|
||||
|
||||
SkImageInfo yInfo = SkImageInfo::MakeA8(_inputWidth, _inputHeight);
|
||||
SkPixmap yPixmap(yInfo, yBuffer.getDirectAddress(), bytesPerRow);
|
||||
|
||||
SkImageInfo uInfo = SkImageInfo::MakeA8(_inputWidth / 2, _inputHeight / 2);
|
||||
SkPixmap uPixmap(uInfo, uBuffer.getDirectAddress(), bytesPerRow / 2);
|
||||
|
||||
SkImageInfo vInfo = SkImageInfo::MakeA8(_inputWidth / 2, _inputHeight / 2);
|
||||
SkPixmap vPixmap(vInfo, vBuffer.getDirectAddress(), bytesPerRow / 2);
|
||||
|
||||
SkYUVAInfo info(SkISize::Make(_inputWidth, _inputHeight),
|
||||
SkYUVAInfo::PlaneConfig::kY_U_V,
|
||||
SkYUVAInfo::Subsampling::k420,
|
||||
SkYUVColorSpace::kRec709_Limited_SkYUVColorSpace);
|
||||
SkPixmap externalPixmaps[3] = { yPixmap, uPixmap, vPixmap };
|
||||
SkYUVAPixmaps pixmaps = SkYUVAPixmaps::FromExternalPixmaps(info, externalPixmaps);
|
||||
|
||||
sk_sp<SkImage> image = SkImages::TextureFromYUVAPixmaps(_skiaContext.get(), pixmaps);
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
GrGLFramebufferInfo fboInfo {
|
||||
// FBO #0 is the currently active OpenGL Surface (eglMakeCurrent)
|
||||
.fFBOID = ACTIVE_SURFACE_ID,
|
||||
.fFormat = GR_GL_RGBA8,
|
||||
.fProtected = skgpu::Protected::kNo,
|
||||
};;
|
||||
GrBackendRenderTarget renderTarget(_previewWidth,
|
||||
_previewHeight,
|
||||
0,
|
||||
8,
|
||||
fboInfo);
|
||||
SkSurfaceProps props(0, kUnknown_SkPixelGeometry);
|
||||
sk_sp<SkSurface> surface = SkSurfaces::WrapBackendRenderTarget(_skiaContext.get(),
|
||||
renderTarget,
|
||||
kTopLeft_GrSurfaceOrigin,
|
||||
kN32_SkColorType,
|
||||
nullptr,
|
||||
&props);
|
||||
|
||||
auto canvas = surface->getCanvas();
|
||||
|
||||
canvas->clear(SkColors::kBlack);
|
||||
|
||||
auto duration = std::chrono::system_clock::now().time_since_epoch();
|
||||
auto millis = std::chrono::duration_cast<std::chrono::milliseconds>(duration).count();
|
||||
|
||||
canvas->drawImage(image, 0, 0);
|
||||
|
||||
// TODO: Run Skia Frame Processor
|
||||
auto rect = SkRect::MakeXYWH(150, 250, millis % 3000 / 10, millis % 3000 / 10);
|
||||
auto paint = SkPaint();
|
||||
paint.setColor(SkColors::kRed);
|
||||
canvas->drawRect(rect, paint);
|
||||
|
||||
// Flush
|
||||
canvas->flush();
|
||||
|
||||
bool successful = eglSwapBuffers(_glDisplay, _glSurface);
|
||||
if (!successful || eglGetError() != EGL_SUCCESS) throw OpenGLError("Failed to swap OpenGL buffers!");
|
||||
|
||||
|
||||
__android_log_print(ANDROID_LOG_INFO, TAG, "Rendered!");
|
||||
}
|
||||
|
||||
|
||||
void SkiaRenderer::registerNatives() {
|
||||
registerHybrid({
|
||||
makeNativeMethod("initHybrid", SkiaRenderer::initHybrid),
|
||||
makeNativeMethod("setInputTextureSize", SkiaRenderer::setInputTextureSize),
|
||||
makeNativeMethod("setOutputSurface", SkiaRenderer::setOutputSurface),
|
||||
makeNativeMethod("destroyOutputSurface", SkiaRenderer::destroyOutputSurface),
|
||||
makeNativeMethod("setOutputSurfaceSize", SkiaRenderer::setOutputSurfaceSize),
|
||||
makeNativeMethod("renderLatestFrameToPreview", SkiaRenderer::renderLatestFrameToPreview),
|
||||
makeNativeMethod("renderCameraFrameToOffscreenCanvas", SkiaRenderer::renderCameraFrameToOffscreenCanvas),
|
||||
});
|
||||
}
|
||||
|
||||
} // namespace vision
|
||||
77
android/src/main/cpp/skia/SkiaRenderer.h
Normal file
77
android/src/main/cpp/skia/SkiaRenderer.h
Normal file
@@ -0,0 +1,77 @@
|
||||
//
|
||||
// Created by Marc Rousavy on 10.08.23.
|
||||
//
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <jni.h>
|
||||
#include <fbjni/fbjni.h>
|
||||
#include <fbjni/ByteBuffer.h>
|
||||
|
||||
#include <GLES2/gl2.h>
|
||||
#include <EGL/egl.h>
|
||||
#include <include/core/SkSurface.h>
|
||||
#include <android/native_window.h>
|
||||
|
||||
namespace vision {
|
||||
|
||||
using namespace facebook;
|
||||
|
||||
#define NO_INPUT_TEXTURE 7654321
|
||||
|
||||
class SkiaRenderer: public jni::HybridClass<SkiaRenderer> {
|
||||
// JNI Stuff
|
||||
public:
|
||||
static auto constexpr kJavaDescriptor = "Lcom/mrousavy/camera/skia/SkiaRenderer;";
|
||||
static void registerNatives();
|
||||
|
||||
private:
|
||||
friend HybridBase;
|
||||
jni::global_ref<SkiaRenderer::javaobject> _javaPart;
|
||||
explicit SkiaRenderer(const jni::alias_ref<jhybridobject>& javaPart);
|
||||
|
||||
public:
|
||||
static jni::local_ref<jhybriddata> initHybrid(jni::alias_ref<jhybridobject> javaPart);
|
||||
~SkiaRenderer();
|
||||
|
||||
private:
|
||||
// Input Texture (Camera)
|
||||
void setInputTextureSize(int width, int height);
|
||||
// Output Surface (Preview)
|
||||
void setOutputSurface(jobject previewSurface);
|
||||
void destroyOutputSurface();
|
||||
void setOutputSurfaceSize(int width, int height);
|
||||
|
||||
/**
|
||||
* Renders the latest Camera Frame from the Input Texture onto the Preview Surface. (60 FPS)
|
||||
*/
|
||||
void renderLatestFrameToPreview();
|
||||
/**
|
||||
* Renders the latest Camera Frame into it's Input Texture and run the Skia Frame Processor (1..240 FPS)
|
||||
*/
|
||||
void renderCameraFrameToOffscreenCanvas(jni::JByteBuffer yBuffer,
|
||||
jni::JByteBuffer uBuffer,
|
||||
jni::JByteBuffer vBuffer);
|
||||
|
||||
private:
|
||||
// OpenGL Context
|
||||
EGLContext _glContext = EGL_NO_CONTEXT;
|
||||
EGLDisplay _glDisplay = EGL_NO_DISPLAY;
|
||||
EGLSurface _glSurface = EGL_NO_SURFACE;
|
||||
EGLConfig _glConfig = nullptr;
|
||||
// Skia Context
|
||||
sk_sp<GrDirectContext> _skiaContext;
|
||||
|
||||
// Input Texture (Camera/Offscreen)
|
||||
GLuint _inputSurfaceTextureId = NO_INPUT_TEXTURE;
|
||||
int _inputWidth, _inputHeight;
|
||||
// Output Texture (Surface/Preview)
|
||||
ANativeWindow* _previewSurface;
|
||||
int _previewWidth, _previewHeight;
|
||||
|
||||
void ensureOpenGL(ANativeWindow* surface);
|
||||
|
||||
static auto constexpr TAG = "SkiaRenderer";
|
||||
};
|
||||
|
||||
} // namespace vision
|
||||
Reference in New Issue
Block a user