In this article, we are going to show how to add and retrieve closed-caption data to the AVC and MPEG2 streams. In the first part, we will illustrate using code examples how to add CC messages to the AVC and MPEG2 streams. In the second part, we will show how to retrieve these messages using the decoder.
To test the closed-captioning in AVC/MPEG2 using the following code snippets, we recommend using the tutorials instead of the samples. (You can download the tutorials from the TUTORIALS tab here). The tutorials are much easier to understand and have more comments to help you. For encoding, simple_6_encode_vmem_lowlatency tutorial, and for decoding, simple_2_decode tutorial. In the code snippets below, we point you to the location where this code should be added for encode and decode.
Let's get started!
Adding Closed-caption Messages to the AVC and MPEG2 Encode Stream
In this section, we will see how to add closed-caption data to AVC and MPEG2 streams during the encoding stage. Below are the high-level how-to steps, and we will follow that up with a code snippet.
Step 1: Create the SEI/user_data payload with the appropriate header and message fields. For details on the AVC and MPEG2 payload structures, refer to this document. You can also refer to the following - for AVC stream (Section 6.4.2 in the document), and for MPEG2 stream (Section 6.2.3 in the document).
Step 2a:Populate the mfxPayload structure with this payload and header information.
Step 2b: Pass the mfxPayload structure to mfxEncodeCtrl structure
Step 3:Pass this structure to the EncodeFrameAsync function (first parameter), and you're done!
Code example to add CC messages to AVC, in the main encoding loop:
#define SEI_USER_DATA_REGISTERED_ITU_T_T35 4
#define MESSAGE_SIZE 20
typedef struct
{
unsigned char countryCode;
unsigned char countryCodeExtension[2];
unsigned char user_identifier[4];
unsigned char type_code;
unsigned char payloadBytes[MESSAGE_SIZE];
unsigned char marker_bits;
} userdata_reg_t35;
/*** STEP 1: START: SEI payload: refer to <link to AVC> for the format */
/* Populating the header for the SEI payload. In most cases, these assignments will not change */
userdata_reg_t35 m_userSEIData;
m_userSEIData.countryCode = 0xB5;
m_userSEIData.countryCodeExtension[0] = 0x31;
m_userSEIData.countryCodeExtension[1] = 0x00;
m_userSEIData.user_identifier[0] = 0x34;
m_userSEIData.user_identifier[1] = 0x39;
m_userSEIData.user_identifier[2] = 0x41;
m_userSEIData.user_identifier[3] = 0x47;
m_userSEIData.type_code = 0x03;
m_userSEIData.marker_bits = 0xFF;
/* Populate the actual message. In this example it is "Frame: <frameNum>" */
memset(m_userSEIData.payloadBytes,0,MESSAGE_SIZE);
sprintf((char*)m_userSEIData.payloadBytes, "%s%d", "Frame: ", nFrame);
/*** STEP 1: END: SEI payload: refer to <link to AVC> for the format */
/*** STEP 2a: START: Fill mfxPayload structure with SEI Payload */
mfxU8 m_seiData[100]; // Arbitrary size
mfxPayload m_mySEIPayload;
memset(&m_mySEIPayload, 0, sizeof(m_mySEIPayload));
m_mySEIPayload.Type = SEI_USER_DATA_REGISTERED_ITU_T_T35;
m_mySEIPayload.BufSize = sizeof(userdata_reg_t35) + 2; // 2 bytes for header
m_mySEIPayload.NumBit = m_mySEIPayload.BufSize * 8;
m_mySEIPayload.Data = m_seiData;
// Insert SEI header and SEI msg into data buffer
m_seiData[0] = (mfxU8)m_mySEIPayload.Type; // SEI type
m_seiData[1] = (mfxU8)(m_mySEIPayload.BufSize-2); // Size of following msg
memcpy(m_seiData+2, &m_userSEIData, sizeof(userdata_reg_t35));
mfxPayload* m_payloads[1];
m_payloads[0] = &m_mySEIPayload;
/*** STEP 2a: END: Fill mfxPayload structure with SEI Payload */
/*** STEP 2b: START: Encode control structure initialization */
mfxEncodeCtrl m_encodeCtrl;
memset(&m_encodeCtrl, 0, sizeof(m_encodeCtrl));
m_encodeCtrl.Payload = (mfxPayload**)&m_payloads[0];
m_encodeCtrl.NumPayload = 1;
/*** STEP 2b: END: Encode control structure initialization */
....
nEncSurfIdx = Get Free Surface;
Sucface Lock;
pmfxSurfaces[nEncSurfIdx] = Load Raw Frame;
Surface Unlock;
....
/*** STEP 3: Encode frame: Pass mfxEncodeCtrl pointer */
sts = mfxENC.EncodeFrameAsync(&m_encodeCtrl, pmfxSurfaces[nEncSurfIdx], &mfxBS, &syncp);
Code example to add CC messages to MPEG2, in the main encoding loop:
Adding payloads to MPEG2 is similar to the AVC example above. The difference comes from the format used for the MPEG2 user_start_code as compared to SEI message. Below, we illustrate how to populate an MPEG2 payload. This is in accordance to the ATSC standard for MPEG2 Video.
#define USER_START_CODE 0x1B2
#define MESSAGE_SIZE 20
typedef struct{
mfxU8 atsc_identifier[4];
mfxU8 type_code;
/* For type 0x03, some additional bits before the data field starts - refer to <link to cc_data()> */
mfxU8 additional_bits[2];
unsigned char cc_data[MESSAGE_SIZE];
}ATSC_user_data;
/** STEP 1 */
ATSC_user_data atsc;
atsc.atsc_identifier[0] = (mfxU8)0x34;
atsc.atsc_identifier[1] = (mfxU8)0x39;
atsc.atsc_identifier[2] = (mfxU8)0x41;
atsc.atsc_identifier[3] = (mfxU8)0x47;
atsc.type_code = 0x03;
atsc.additional_bits[0] = (mfxU8)0x12; //00010010; //cc_count, addnl data, cc data, em data processed
atsc.additional_bits[1] = (mfxU8)0xff; //reserved
memset(atsc.cc_data,0,MESSAGE_SIZE);
sprintf((char*)atsc.cc_data, "%s%d", "Frame: ", nFrame);
/** STEP 2a */
mfxU8 m_seiData[100]; // Arbitrary size
mfxPayload m_Payload;
memset(&m_Payload, 0, sizeof(m_Payload));
m_Payload.Type = USER_START_CODE;
m_Payload.BufSize = MESSAGE_SIZE + 7;
m_Payload.NumBit = m_Payload.BufSize * 8;
m_Payload.Data = m_seiData;
memcpy(m_seiData, &atsc, 7);
memcpy(m_seiData+7, &atsc.cc_data, MESSAGE_SIZE);
mfxPayload* m_payloads[1];
m_payloads[0] = &m_Payload;
/** STEP 2b */
mfxEncodeCtrl m_encodeCtrl;
memset(&m_encodeCtrl, 0, sizeof(m_encodeCtrl));
m_encodeCtrl.Payload = (mfxPayload**)&m_payloads[0];
m_encodeCtrl.NumPayload = 1;
....
nEncSurfIdx = Get Free Surface;
Sucface Lock;
pmfxSurfaces[nEncSurfIdx] = Load Raw Frame;
Surface Unlock;
....
/*** STEP 3 */
sts = mfxENC.EncodeFrameAsync(&m_encodeCtrl, pmfxSurfaces[nEncSurfIdx], &mfxBS, &syncp);In this section, we have seen how to add payloads to the AVC and MPEG2 encode stream. You can verify the payloads are added by opening the output bitstream in an editor and viewing in HEX mode (for instance, GVim). You should see each frame carrying a payload that has the "Frame: <framenum>" along with the payload meta information. Below is an encoded out.h264 file that uses the above-mentioned code snippet in simple_6_encode_vmem_lowlatency tutorial to add CC captions. You can see the SEI message "Frame: 6" highlighted.
Retrieving Messages from the Decoded Stream
We just saw how to add CC messages to the encode stream. In this section, we will see how to retrieve the encoded SEI/userData messages from the AVC or MPEG2 streams respectively. The SDK provides an API GetPayload() for this purpose, and we will illustrate how to use this for AVC and MPEG cases. The GetPayload() function follows the DecodeFrameAsync() function call, and returns the mfxPayload structure populated with the message, number of bytes and timestamp. Please note that you are required to initialize the BufSize and Data members of the structure before you call the GetPayload() function.
sts = mfxDEC.DecodeFrameAsync(&mfxBS, pmfxSurfaces[nIndex], &pmfxOutSurface, &syncp);
mfxPayload dec_payload;
mfxU64 ts;
dec_payload.Data = new mfxU8[100];
dec_payload.BufSize = MESSAGE_SIZE;
dec_payload.NumBit = 1;
/* Since the decode function is asynchronous, we will loop over the GetPayload() function until we drain all the messages ready */
while(dec_payload.NumBit > 0)
{
mfxStatus st = mfxDEC.GetPayload(&ts, &dec_payload);
#ifdef AVC
if((st==MFX_ERR_NONE)&&(dec_payload.Type==SEI_USER_DATA_REGISTERED_ITU_T_T35))
#endif
#ifdef MPEG2
if((st==MFX_ERR_NONE)&&(dec_payload.Type==USER_START_CODE))
#endif
{
fwrite(dec_payload.Data, sizeof(unsigned char), dec_payload.BufSize, stdout); //For debug purpose - Prints out the payload on to the screen.
}
}Below is the output of the fwrite for decode of out.h264 file we encoded above. On the console is printed the output of the GetPayload() call for each frame.
This concludes our howto add and retrieve CC messages in AVC and MPEG2 stream. You can also refer to our documentation section 4.14 for the pseudo-code, and for information on the APIs, you can refer to the mediasdk-man.pdf in the doc folder of the Media SDK install folder.
Изображение значка:
