本文檔描述使用AVAssetWriterInputPixelBufferAdaptor簡化Metal渲染結果從BGRA到yuv420p的轉換,適用于寫成MP4或MOV容器。以前有人用類似的辦法暴力實現iOS 8以下的H.264硬解,代價是消耗用戶的閃存寫次數。
文檔結構:
- 初始化AVAssetWriterInput
- 初始化AVAssetWriterInputPixelBufferAdaptor
- CVPixelBuffer寫入AVAssetWriterInputPixelBufferAdaptor
- 討論:指定BT. 709 YUV轉RGB矩陣
Metal Camera開發4:渲染到CVPixelBuffer實現了CVPixelBuffer存儲渲染結果,對于攝像頭開發,通常希望將渲染結果編碼成H.264或H.265(iOS 11支持),iOS平臺的H.264編碼器原生支持yuv420sp VideoRange及FullRange,而Metal渲染到CVPixelBuffer時像素格式為BGRA,顯然需要進行像素格式轉換,比如使用libyuv、Metal 著色器實現RGBA轉yuv或GLSL實現。對于將數據寫到本地的需求,AVFoundation提供了AVAssetWriterInputPixelBufferAdaptor簡化RGB轉yuv的步驟。GPUImage也使用了這種方式實現OpenGL ES的渲染結果編碼成H.264。下面描述它的編程過程。
AVAssetWriterInputPixelBufferAdaptor必須配合AVAssetWriterInput使用,不然沒法調用構造函數(手動斜眼.jpg)。AVAssetWriterInputPixelBufferAdaptor類的頭文件描述了它的功能:
Pixel buffers not in a natively supported format will be converted internally prior to encoding when possible.
對于編碼Metal的渲染結果,初始化AVAssetWriterInputPixelBufferAdaptor指定sourcePixelBufferAttributes使用kCVPixelBufferPixelFormatTypeKey為kCVPixelFormatType_32BGRA,之后的像素格式轉換都由AVFoundation內部實現,省去了手動實現BGRA轉yuv。當編碼CoreImage的渲染結果,kCVPixelBufferPixelFormatTypeKey對應的值為kCVPixelFormatType_32ARGB。總之,需要和數據源對應起來,否則視頻會表現出異常顏色。接下來逐一描述編程步驟。
1. 初始化AVAssetWriterInput
let videoCompressionProperties = [
AVVideoAverageBitRateKey: 1080 * 1920 * 15.15
]
let videoSettings: [String : Any] = [
AVVideoCodecKey: AVVideoCodecH264,
AVVideoWidthKey: 1080,
AVVideoHeightKey: 1920,
AVVideoCompressionPropertiesKey: videoCompressionProperties
]
self.videoWriterInput = AVAssetWriterInput(mediaType: AVMediaTypeVideo, outputSettings: videoSettings)
AVAssetWriterInput還需添加到AVAssetWriter,網上參考代碼非常豐富,在此不一一描述。
2. 初始化AVAssetWriterInputPixelBufferAdaptor
var videoWriterInputPixelBufferAdaptor: AVAssetWriterInputPixelBufferAdaptor?
//-------
let sourcePixelBufferAttributes = [
kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA,
kCVPixelBufferWidthKey as String: 1080,
kCVPixelBufferHeightKey as String: 1920
]
self.videoWriterInputPixelBufferAdaptor = AVAssetWriterInputPixelBufferAdaptor(
assetWriterInput: self.videoWriterInput!,
sourcePixelBufferAttributes: sourcePixelBufferAttributes
)
AVAssetWriterInputPixelBufferAdaptor是否應該在AVAssetWriterInput添加到AVAssetWriter前初始化?經實驗,這一操作的順序不影響程序的正常運行,在AVAssetWriter.startWriting()前初始化即可。
3. CVPixelBuffer寫入AVAssetWriterInputPixelBufferAdaptor
self.videoWriterInputPixelBufferAdaptor!.assetWriterInput.isReadyForMoreMediaData {
let whetherPixelBufferAppendedtoAdaptor = self.videoWriterInputPixelBufferAdaptor!.append(renderPixelBuffer!, withPresentationTime: presentationTime)
if whetherPixelBufferAppendedtoAdaptor {
print("adaptor appended CVPixelBuffer successfully")
} else {
print("adaptor appended CVPixelBuffer failed")
}
}
renderPixelBuffer為Metal渲染的目標CVPixelBuffer,即它存儲了最終的渲染結果,細節可參考Metal Camera開發4:渲染到CVPixelBuffer。
4. 討論:指定BT. 709 YUV轉RGB矩陣
GPUImageMovieWriter類的createDataFBO方法有這么一段注釋:
/* AVAssetWriter will use BT.601 conversion matrix for RGB to YCbCr conversion
* regardless of the kCVImageBufferYCbCrMatrixKey value.
* Tagging the resulting video file as BT.601, is the best option right now.
* Creating a proper BT.709 video is not possible at the moment.
*/
在iOS 10的測試過程中,發現已支持kCVImageBufferYCbCrMatrixKey設置為kCVImageBufferTransferFunction_ITU_R_709_2。參考代碼如下。
CVBufferSetAttachment(pixelBuffer!, kCVImageBufferColorPrimariesKey, kCVImageBufferColorPrimaries_ITU_R_709_2, CVAttachmentMode.shouldPropagate)
CVBufferSetAttachment(pixelBuffer!, kCVImageBufferYCbCrMatrixKey, kCVImageBufferTransferFunction_ITU_R_709_2, CVAttachmentMode.shouldPropagate)
CVBufferSetAttachment(pixelBuffer!, kCVImageBufferTransferFunctionKey, kCVImageBufferTransferFunction_ITU_R_709_2, CVAttachmentMode.shouldPropagate)
FFmpeg讀取生成的視頻信息如下所示。