最近協(xié)助一個攝像頭相關(guān)的項目遇到了一個有意思的問題，這里記錄一下。

原問題大概是使用TextureView預(yù)覽攝像頭,關(guān)閉攝像頭之后畫面會殘留最后一幀,需要把他清除。我一開始使用的方式是獲取Surface的Canvas去將整個畫布畫上黑色來實現(xiàn)清除畫面:

Canvas canvas = mPreviewSurface.lockCanvas(null);
canvas.drawColor(Color.BLACK);
mPreviewSurface.unlockCanvasAndPost(canvas);

但是遇到了下面的問題:

1. CameraDevice.close之前l(fā)ockCanvas會拋出IllegalArgumentException
2. 在CameraDevice.close之后lockCanvas雖然可以清除畫面,但是再次打開調(diào)用CameraDevice.createCaptureSession會失敗,回調(diào)onConfigureFailed

在網(wǎng)上搜索了下stackoverflow上fadden大神是這么解釋的:

You can't do this, due to a limitation of the Android app framework (as of Android 4.4 at least).

The SurfaceTexture that underlies the TextureView is a buffer consumer. The MediaPlayer is one example of a buffer producer, Canvas is another. Once you attach a producer, you have to detach it before you can attach a second producer.

The trouble is that there is no way to detach a software-based (Canvas) buffer producer. There could be, but isn't. So once you draw with Canvas, you're stuck. (There's a note to that effect here.)

You can detach a GLES producer. For example, in one of Grafika's video player classes you can find a clearSurface() method that clears the surface to black using GLES. Note the EGL context and window are created and explicitly released within the scope of the method. You could expand the method to show an image instead.

大概意思就是TextureView作為一個畫面的消費者,可以綁定到不同的畫面生產(chǎn)者(Canvas是其中一種,另外像MediaPlayer、Camera這些也可以作為畫面生產(chǎn)者)。一旦連接上一個生產(chǎn)者之后就不能再次連接其他的生產(chǎn)者了,而Canvas這個生產(chǎn)者比較野蠻,并沒有提供解除綁定的方法。所以一旦TextureView綁定到Canvas之后，MediaPlayer、Camera就不能再使用這個Surface區(qū)顯示畫面了。

然后他提供的解決方法是參考Grafika使用OpenGL去做清除。

消費者生產(chǎn)者模型

消費者生產(chǎn)者模型在安卓的圖像系統(tǒng)里面還是比較重要的一個東西,從官方文檔的介紹里面我們可以大概看出整個工作流程:

bufferqueue.png

• Producer 如Camera、視頻解碼器、OpenGL ES、Canvas等調(diào)用dequeue從BufferQueue里面獲取一個空白Buffer,然后使用Buffer做繪制,繪制完成之后調(diào)用queue把Buffer交還給BufferQueue。
• Consumer 如SurfaceFlinger調(diào)用acquire從BufferQueue里面獲取一個繪制好的Buffer,然后進行畫面的渲染,渲染完成之后調(diào)用release把Buffer交還給BufferQueue作為空白Buffer。

Canvas canvas = mPreviewSurface.lockCanvas(null);
canvas.drawColor(Color.BLACK);
mPreviewSurface.unlockCanvasAndPost(canvas);

用上面的lockCanvas來舉例。在代碼中Producer具體為IGraphicBufferProducer接口,在Surface構(gòu)造的時候傳入,在connect的的時候去連接:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/Surface.cpp
Surface::Surface(const sp<IGraphicBufferProducer>& bufferProducer, bool controlledByApp,
                 const sp<IBinder>& surfaceControlHandle)
      : mGraphicBufferProducer(bufferProducer),
      ...

int Surface::connect(
        int api, const sp<IProducerListener>& listener, bool reportBufferRemoval) {
    ...
    int err = mGraphicBufferProducer->connect(listener, api, mProducerControlledByApp, &output);
    ...
}

然后Surface.lockCanvas調(diào)用到native層的nativeLockCanvas去用Surface::lock來dequeueBuffer獲取Buffer提供給Canvas繪制:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/base/core/jni/android_view_Surface.cpp
static jlong nativeLockCanvas(JNIEnv* env, jclass clazz,
        jlong nativeObject, jobject canvasObj, jobject dirtyRectObj) {
    ANativeWindow_Buffer buffer;
    status_t err = surface->lock(&buffer, dirtyRectPtr);
    ...
    graphics::Canvas canvas(env, canvasObj);
    canvas.setBuffer(&buffer, static_cast<int32_t>(surface->getBuffersDataSpace()));
    ...
}

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/Surface.cpp
status_t Surface::lock(
        ANativeWindow_Buffer* outBuffer, ARect* inOutDirtyBounds)
{
    ...
    status_t err = dequeueBuffer(&out, &fenceFd);
    ...
}

int Surface::dequeueBuffer(android_native_buffer_t** buffer, int* fenceFd) {
    ...
    status_t result = mGraphicBufferProducer->dequeueBuffer(&buf, &fence, dqInput.width,
                                                            dqInput.height, dqInput.format,
                                                            dqInput.usage, &mBufferAge,
                                                            dqInput.getTimestamps ?
                                                            &frameTimestamps : nullptr);
    ...
}

而Surface.unlockCanvasAndPost會調(diào)用native層的nativeUnlockCanvasAndPost去調(diào)用Surface::unlockAndPost去queueBuffer:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/base/core/jni/android_view_Surface.cpp
static void nativeUnlockCanvasAndPost(JNIEnv* env, jclass clazz,
        jlong nativeObject, jobject canvasObj) {
    ...
    // detach the canvas from the surface
    graphics::Canvas canvas(env, canvasObj);
    canvas.setBuffer(nullptr, ADATASPACE_UNKNOWN);

    // unlock surface
    status_t err = surface->unlockAndPost();
    ...
}

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/Surface.cpp
status_t Surface::unlockAndPost()
{
    ...
    err = queueBuffer(mLockedBuffer.get(), fd);
    ...
}

int Surface::queueBuffer(android_native_buffer_t* buffer, int fenceFd) {
    ...
    status_t err = mGraphicBufferProducer->queueBuffer(i, input, &output);
    ...
}

這樣不斷循環(huán)lockCanvas、繪制Canvas、unlockCanvasAndPost就能往SurfaceFlinger這個Consumer不斷提供畫面去渲染了。

雖然大概的原因和解決方法都講清楚了,但是我還是有三點疑問:

1. 需要在CameraDevice.close之后才能lockCanvas是不是意味著CameraDevice.close里面會做解綁
2. 調(diào)用unlockCanvasAndPost為什么沒有解除Canvas這個內(nèi)容生產(chǎn)者的綁定?
3. GLES可以解除綁定,那它又是怎么解除的呢?

CameraDevice.close之后才能lockCanvas

網(wǎng)上搜索沒有找到答案,那就只能自己分析源碼了,首先我們從unlockCanvasAndPost之后再次createCaptureSession會失敗的日志入手看看能不能找到什么有用的信息:

06-06 18:55:13.130 28137 25285 E BufferQueueProducer: [SurfaceTexture-0-28137-0](id:6de900000001,api:2,p:28137,c:28137) connect: already connected (cur=2 req=4)
06-06 18:55:13.130  1905  8873 E Camera3-OutputStream: configureConsumerQueueLocked: Unable to connect to native window for stream 0
06-06 18:55:13.130  1905  8873 E Camera3-Stream: finishConfiguration: Unable to configure stream 0 queue: Invalid argument (-22)
06-06 18:55:13.130  1905  8873 E Camera3-Device: Camera 0: configureStreamsLocked: Can't finish configuring output stream 0: Invalid argument (
-22)
06-06 18:55:13.130  1047  1365 E minksocket: MinkIPC_QRTR_Service: client with node 1 port 6838 went down
06-06 18:55:13.130  1905  8873 D CameraService: CameraPerf: setpriority success, tid is 8873, priority is 0
06-06 18:55:13.130  1905  8873 E CameraDeviceClient: endConfigure: Camera 0: Unsupported set of inputs/outputs provided

從日志里面可以看到在Camera3OutputStream::configureConsumerQueueLocked里面會去調(diào)用Surface::connect:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/av/services/camera/libcameraservice/device3/Camera3OutputStream.h
sp<Surface> mConsumer;

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/av/services/camera/libcameraservice/device3/Camera3OutputStream.cpp
status_t Camera3OutputStream::configureConsumerQueueLocked(bool allowPreviewRespace) {
    ...
    // Configure consumer-side ANativeWindow interface. The listener may be used
    // to notify buffer manager (if it is used) of the returned buffers.
    res = mConsumer->connect(NATIVE_WINDOW_API_CAMERA,
            /*reportBufferRemoval*/true,
            /*listener*/mBufferProducerListener);
    if (res != OK) {
        ALOGE("%s: Unable to connect to native window for stream %d",
                __FUNCTION__, mId);
        return res;
    }
    ...
}

而在Surface::connect里面會調(diào)用BufferQueueProducer::connect:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/Surface.cpp
int Surface::connect(int api) {
    static sp<IProducerListener> listener = new StubProducerListener();
    return connect(api, listener);
}

int Surface::connect(int api, const sp<IProducerListener>& listener) {
    return connect(api, listener, false);
}

int Surface::connect(
        int api, const sp<IProducerListener>& listener, bool reportBufferRemoval) {
    ...
    int err = mGraphicBufferProducer->connect(listener, api, mProducerControlledByApp, &output);
    ...
}

在BufferQueueProducer::connect里面會判斷如果mCore->mConnectedApi不為BufferQueueCore::NO_CONNECTED_API(即已經(jīng)connect過了)就不能再connect:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/include/gui/BufferQueueProducer.h
sp<BufferQueueCore> mCore;

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/BufferQueueProducer.cpp
status_t BufferQueueProducer::connect(const sp<IProducerListener>& listener,
        int api, bool producerControlledByApp, QueueBufferOutput *output) {
    ...
    if (mCore->mConnectedApi != BufferQueueCore::NO_CONNECTED_API) {
        BQ_LOGE("connect: already connected (cur=%d req=%d)",
                mCore->mConnectedApi, api);
        return BAD_VALUE;
    }
    ...
    mCore->mConnectedApi = api;
    ...
}

所以我們看到的already connected日志就是從這里打印的。

06-06 18:55:13.130 28137 25285 E BufferQueueProducer: [SurfaceTexture-0-28137-0](id:6de900000001,api:2,p:28137,c:28137) connect: already connected (cur=2 req=4)

connect api的類型有下面幾種,所以從日志上我們可以分析出,SurfaceTexture已經(jīng)connect到NATIVE_WINDOW_API_CPU了,不能再connect到NATIVE_WINDOW_API_CAMERA:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/nativewindow/include/system/window.h
/* parameter for NATIVE_WINDOW_[API_][DIS]CONNECT */
enum {
    /* Buffers will be queued by EGL via eglSwapBuffers after being filled using
     * OpenGL ES.
     */
    NATIVE_WINDOW_API_EGL = 1,

    /* Buffers will be queued after being filled using the CPU
     */
    NATIVE_WINDOW_API_CPU = 2,

    /* Buffers will be queued by Stagefright after being filled by a video
     * decoder.  The video decoder can either be a software or hardware decoder.
     */
    NATIVE_WINDOW_API_MEDIA = 3,

    /* Buffers will be queued by the the camera HAL.
     */
    NATIVE_WINDOW_API_CAMERA = 4,
};

而在CameraDevice.close里面會調(diào)用Camera3OutputStream::disconnectLocked最終會調(diào)用到BufferQueueProducer::disconnect將mCore->mConnectedApi賦值回BufferQueueCore::NO_CONNECTED_API:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/av/services/camera/libcameraservice/device3/Camera3OutputStream.cpp
status_t Camera3OutputStream::disconnectLocked() {
    ...
    ALOGV("%s: disconnecting stream %d from native window", __FUNCTION__, getId());

    res = native_window_api_disconnect(mConsumer.get(),
                                       NATIVE_WINDOW_API_CAMERA);
    ...
}

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/nativewindow/include/system/window.h
static inline int native_window_api_disconnect(
        struct ANativeWindow* window, int api)
{
    return window->perform(window, NATIVE_WINDOW_API_DISCONNECT, api);
}

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/Surface.cpp
int Surface::perform(int operation, va_list args)
{
    ...
    case NATIVE_WINDOW_API_DISCONNECT:
        res = dispatchDisconnect(args);
        break;
    ...
}

int Surface::dispatchDisconnect(va_list args) {
    int api = va_arg(args, int);
    return disconnect(api);
}

int Surface::disconnect(int api, IGraphicBufferProducer::DisconnectMode mode) {
    ...
    int err = mGraphicBufferProducer->disconnect(api, mode);
    ...
}

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/BufferQueueProducer.cpp
status_t BufferQueueProducer::disconnect(int api, DisconnectMode mode) {
    ...
    mCore->mConnectedApi = BufferQueueCore::NO_CONNECTED_API;
    ...
}

所以在CameraDevice.close之后mCore->mConnectedApi被賦值成了BufferQueueCore::NO_CONNECTED_API,lockCanvas再去BufferQueueProducer::connect就不會失敗。

lockCanvas & unlockCanvasAndPost

Surface.lockCanvas最終會去到Surface::lock里調(diào)用Surface::connect(NATIVE_WINDOW_API_CPU):

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/base/core/jni/android_view_Surface.cpp
static jlong nativeLockCanvas(JNIEnv* env, jclass clazz,
        jlong nativeObject, jobject canvasObj, jobject dirtyRectObj) {
    sp<Surface> surface(reinterpret_cast<Surface *>(nativeObject));
    ...
    status_t err = surface->lock(&buffer, dirtyRectPtr);
    ...
}

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/Surface.cpp
status_t Surface::lock(
        ANativeWindow_Buffer* outBuffer, ARect* inOutDirtyBounds)
{
    ...
    if (!mConnectedToCpu) {
        int err = Surface::connect(NATIVE_WINDOW_API_CPU);
        if (err) {
            return err;
        }
        // we're intending to do software rendering from this point
        setUsage(GRALLOC_USAGE_SW_READ_OFTEN | GRALLOC_USAGE_SW_WRITE_OFTEN);
    }
    ...
}

后面的流程就和Camera3OutputStream::configureConsumerQueueLocked里面調(diào)用Surface::connect類似了,最終會調(diào)用BufferQueueProducer::connect把mCore->mConnectedApi賦值成NATIVE_WINDOW_API_CPU。但是稍有不同的是在Surface::connect里面會判斷這個connect api,將mConnectedToCpu賦值為true:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/Surface.cpp
int Surface::connect(
        int api, const sp<IProducerListener>& listener, bool reportBufferRemoval) {
    int err = mGraphicBufferProducer->connect(listener, api, mProducerControlledByApp, &output);
    ...
    if (!err && api == NATIVE_WINDOW_API_CPU) {
        mConnectedToCpu = true;
        // Clear the dirty region in case we're switching from a non-CPU API
        mDirtyRegion.clear();
    }
    ...
}

所以之后unlockCanvasAndPost沒有disconnect BufferQueueProducer也不會在再次調(diào)用Surface.lockCanvas的時候造成重復(fù)Surface::connect(NATIVE_WINDOW_API_CPU)的問題:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/base/core/jni/android_view_Surface.cpp
static void nativeUnlockCanvasAndPost(JNIEnv* env, jclass clazz,
        jlong nativeObject, jobject canvasObj) {
    sp<Surface> surface(reinterpret_cast<Surface *>(nativeObject));
    if (!isSurfaceValid(surface)) {
        return;
    }

    // detach the canvas from the surface
    graphics::Canvas canvas(env, canvasObj);
    canvas.setBuffer(nullptr, ADATASPACE_UNKNOWN);

    // unlock surface
    status_t err = surface->unlockAndPost();
    if (err < 0) {
        jniThrowException(env, IllegalArgumentException, NULL);
    }
}

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/Surface.cpp
status_t Surface::unlockAndPost()
{
    if (mLockedBuffer == nullptr) {
        ALOGE("Surface::unlockAndPost failed, no locked buffer");
        return INVALID_OPERATION;
    }

    int fd = -1;
    status_t err = mLockedBuffer->unlockAsync(&fd);
    ALOGE_IF(err, "failed unlocking buffer (%p)", mLockedBuffer->handle);

    err = queueBuffer(mLockedBuffer.get(), fd);
    ALOGE_IF(err, "queueBuffer (handle=%p) failed (%s)",
            mLockedBuffer->handle, strerror(-err));

    mPostedBuffer = mLockedBuffer;
    mLockedBuffer = nullptr;
    return err;
}

從上面的代碼也可以看出來Surface.unlockCanvasAndPost只是將Canvas從Surface上分離,但是BufferQueueProducer沒有disconnect,它的mCore->mConnectedApi還是NATIVE_WINDOW_API_CPU。于是再次連接Camera的時候去connect NATIVE_WINDOW_API_CAMERA就會失敗。

NATIVE_WINDOW_API_CPU的類型只有在Surface析構(gòu)的時候才會去disconnect:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/Surface.cpp
Surface::~Surface() {
    if (mConnectedToCpu) {
        Surface::disconnect(NATIVE_WINDOW_API_CPU);
    }
}

GLES disconnect

實際上GLES是靠EGL14.eglDestroySurface去調(diào)用BufferQueueProducer::disconnect的,如果沒有調(diào)用,再次去連接攝像頭也會失敗:

06-06 20:13:59.940 29586 25849 E BufferQueueProducer: [SurfaceTexture-0-29586-0](id:739200000001,api:1,p:29586,c:29586) connect: already connected (cur=1 req=4)

這次就是NATIVE_WINDOW_API_EGL已連接,請求NATIVE_WINDOW_API_CAMERA連接失敗了。

區(qū)分connect api

為什么需要區(qū)分connect api呢? 這是由于不同api的connect類型可能會有些不一樣的處理邏輯,例如BufferQueueProducer::queueBuffer里就對NATIVE_WINDOW_API_EGL類型做了判斷:

// https://cs.android.com/android/platform/superproject/+/android-13.0.0_r8:frameworks/native/libs/gui/BufferQueueProducer.cpp
status_t BufferQueueProducer::queueBuffer(int slot,
        const QueueBufferInput &input, QueueBufferOutput *output) {
    ...
    // Wait without lock held
    if (connectedApi == NATIVE_WINDOW_API_EGL) {
        // Waiting here allows for two full buffers to be queued but not a
        // third. In the event that frames take varying time, this makes a
        // small trade-off in favor of latency rather than throughput.
        lastQueuedFence->waitForever("Throttling EGL Production");
    }
    ...
}