The process of live video can be divided into the following steps:
Capture -> Process -> Coding and Encapsulation -> Push to Server -> Server Stream Distribution -> Player Stream Play
Audio data can be combined with the image combination of video data, but also to pure audio capture and playback, which in many mature applications such as online radio and voice stations play a very important role. Audio acquisition process mainly through the equipment in the environment of the analog signal collected into PCM encoded raw data, and then encoded into MP3 and other formats of data distributed out. Common audio compression formats are: MP3, AAC, HE-AAC, Opus, FLAC, Vorbis (Ogg), Speex and AMR.
The main challenges of audio capture and coding are latency sensitive, Caton sensitive, Denoise, AEC, MAD, and various mixing algorithms.
The image collection of image results into a group of continuous playback of animation, which constitutes the video can be naked eye to watch the content. The image acquisition process is mainly taken by the camera and other equipment into YUV encoded raw data, and then encoded into H.264 and other formats of data distributed out. Common video packaging formats are: MP4, 3GP, AVI, MKV, WMV, MPG, VOB, FLV, SWF, MOV, RMVB and WebM.
Image because of its intuitive feel the strongest and the volume is relatively large, constitute a video content of the main part. The main challenges of image acquisition and coding are poor compatibility, latency sensitive, Caton sensitive, and various image manipulation operations such as beauty and watermarks.
The main source of video acquisition is camera acquisition, screen recording and streaming from video files.
Video or audio to complete the acquisition of the original data, in order to enhance some of the field effects or add some additional effects, we will generally be encoded before the processing, such as marking the timestamp or the company’s logo watermark, freckle and sound Confused and so on. In the anchor and audience Lianmei scene, the anchor needs to talk with one or more viewers, and the dialogue results in real time to share to all other viewers, even wheat processing also part of the work in the push end.
Video encoder has undergone decades of development, has been from the beginning only to support the evolution of intra-coding to today’s H.265 and VP9 as the representative of the new generation of encoders, the following are some common video encoder:
1) H.264 / AVC
2) HEVC / H.265
Note: audio encoder with Mp3, AAC and so on.
In front of the analogy, the package can be understood as the use of which truck to transport, that is, the media container.
The so-called container, is the encoder generated multimedia content (video, audio, subtitles, chapter information, etc.) mixed together standard. The container makes it easy to play with different multimedia content, and the other function of the container is to index the multimedia content, that is, if there is no container, then a video you can only see from the beginning, can not drag the progress And if you do not go to the manual to load the audio, there is no sound. Here are a few common package formats:
1) AVI format (suffix .avi)
2) DV-AVI format (suffix .avi)
3) QuickTime File Format format (suffix .mov)
4) MPEG format (file suffix can be. Mpg .mpeg .mpe .dat .vob .asf .3gp .mp4, etc.)
5) WMV format (suffix .wmv .asf)
6) Real Video format (suffix .rm .rmvb)
7) Flash Video format (suffix .flv)
8) Matroska format (suffix .mkv)
9) MPEG2-TS format (suffix .ts)
At present, we are in streaming media transmission, especially in the live broadcast is mainly used in FLV and MPEG2-TS format, respectively, for RTMP / HTTP-FLV and HLS protocol.
5. Push to the server
The flow is the first mile of the live broadcast, and the live stream has a great impact on the live link. If the network is unstable, the audience experience will be bad no matter how we do it. So it is the first step in our investigation of the problem, how to systematically solve such problems need our understanding of the relevant theory.
There are three main push agreements:
RTSP (Real Time Streaming Protocol): real-time streaming protocol, is used to control the sound or image of the multimedia streaming protocol, by Real Networks and Netscape jointly proposed;
RTMP (Real Time Messaging Protocol): Real-time messaging protocol, Adobe is Flash player and server audio, video and data transmission between the open protocol;
HLS (HTTP Live Streaming): Apple (Apple Inc.) to achieve the HTTP-based streaming media transmission protocol;
RTMP protocol based on TCP, is designed for real-time data communication network protocol, mainly used in the flash / AIR platform and support RTMP protocol streaming media / interactive server audio and video and data communications between. Software that supports this protocol includes Adobe Media Server / Ultrant Media Server / red5 and so on.
It has three variants:
RTMP works on TCP over the plaintext protocol, using port 1935;
RTMPT encapsulated in HTTP requests, through the firewall;
RTMPS is similar to RTMPT, but uses an HTTPS connection;
RTMP is the mainstream streaming media transmission protocol, widely used in the field of live, it can be said that the vast majority of live products are used in this agreement.
The RTMP protocol is like a container for loading packets, which can be AMF-formatted data or video / audio data in FLV. A single connection can transmit multiple network streams through different channels. The packets in these channels are transmitted in a fixed size packet.
6. Server flow distribution
The role of streaming media server is responsible for the distribution and broadcast distribution of streaming functions.
Streaming media servers have many options, such as the commercial version of Wowza. But I chose Nginx, it is a good free Web server, I will detail how to build Nginx server.
7. Player stream playback
Mainly to achieve live broadcast on the terminal show. Because I use the transfer protocol here is RTMP, so as long as the support RTMP stream protocol player can be used, for example:
Computer side: VLC and so on
Mobile side: Vitamio and ijkplayer and so on
In general, we put the top four steps of the above process as the first part, that is, the operation of the video host. Video capture and then flow to the streaming media server, the first part of the function is completed. The second part is the streaming media server, which is responsible for processing and distributing the streams received from the first part to the audience. The third part is the audience, only need to support the flow of the agreement to the player can be.