[Tutorial] MadSonic / Enigma2 Transcoding Live TV

[ebeneezer]

Hi Community,

i have integrated my DM8000 Enigma2 Sat HD Settop Box into MadSonic.

I am able to stream HD/SD TV with live transcoding.

My setup does not need any external programs, but uses UNIX commands like tail,curl and xargs.

Anyone interested in a howto?

Regards,
EG

[Madevil]

hi there,

Well great ebeneezer!!!

Looks very interesting, using myself 4 Dreambox receivers at home.

Maybe a portation to windows is also possible.

best regards,
Madevil

[ebeneezer]

Oops - the boss himself replies

Hi Madevil,

ok then I will start the howto. Portation to Windows should be possible as one only needs Windows-replacements for the mentioned UNIX shell commands - nothing else.

CU
Ebeneezer

[ebeneezer]

Howto to make Enigma2 Set-Top Boxes integrate with Madsonic:

1. Whats about

We have a Madsonic server and a Enigma2 Set-Top Box with enabled Web UI. What we want is all the streams from the Engima2 Box to be transcoded by Madsonic and sent out towards a client on the LAN or Inet. TV channels should be selectable from Madsonics Web UI or a mobile client app (iOS/Android).

2. What we need

On the Madsonic Server we need the following commands: ffmpeg (obvious), lame (obvious), curl, tail, xargs.
I used Homebrew for Mavericks to build a 64 Bit version of ffmpeg and lame. Other frameworks also apply: Macports as an example. Linux has its repos and Windows users have to search the Inet. All those commands must reside in the Madsonic transcoding folder (see explanation inside Madsonic UI).

3. How it works

3.1. We will generate a new sort of video file using the extension ".tv"

Screenshot 1

Those are simple text files containing the HTTP URL of the TV Channels we want to receive from the Enigma2 box. Those streams are uncoded und far to big to be sent over the internet (typically mpeg2ts).

3.1.1. How do we get/make those .tv files?
Simple: Open the web UI of your Enigma2 Box, right click on the streaming button

Right click and save the .m3u stream info file

and save that file under the new ".tv" extension. Use the filename to describe the channel - this is displayed in Madsonics media database.

You should use a folder to hold all those files which we can tell Madsonic to generate a own folder/section for TV channels from.

saving stream info file under new name and extension into our new folder of TV stream info files

to be continued ...

[ebeneezer]

Part 2

Inside the stream descriptor files you will find:

inside an Enigma2 stream descriptor

We are interested in the last line of text, which is the URL of the input mpeg2ts stream sent from the Enigma2 box towards our Madsonic transcoder. Madsonic will have to extract that last line of text from the stream descriptor open that URL and send it through ffmpeg towards the client.

Under 3.1 we already told Madsonic, that .tv files are video media files. Thus if Madsonic scans a directory containing those files all of them will be added to the video files library inside Madsonic. To keep them in place we tell Madsonic that there is a new section containing our self generated video files.

new section for TV channels containing our self generated file descriptors

3.2. We will tell Madsonic that there is a new transcoder for .tv media files called "TV"

a new born transcoder inside Madsonic

That transcoder is called "tv" (see screenshot) and takes ".tv" files as an input (see screenshot second field). It generates video output (obviously) which is configured as flv (see field 3). To do all that our transcoder does the following magic in three simple steps:

to be continued ...

[ebeneezer]

1. cut of the last line of text out of your .tv file (containing the stream URL as seen above)

Code: Select all

tail -n1 %s


does this for us.

2. use curl to open that URL and send the binary stream data towards step 3

step 2 of your tv transcoder

3. ffmpeg takes curls output from stdin and generates a h.264 compatible output stream that is encapsulated and shaped as requested by the client via %b and all the other parameters:

Code: Select all

ffmpeg -ss %o -i - -async 1 -b %bk -s %wx%h -ar 44100 -ac 2 -v 0 -f flv -vcodec libx264 -preset superfast -threads 0 -


Note that "-i -" parameter: ffmpeg is told to simply use stdin as input file. This is important, because ffmpeg does not have any knowledge of what it is transcoding. ffmpeg does not access the file system at any time.

Note the last parameter "-": ffmpeg is told to sent its encoded stream towards stdout. And at that end of the pipe the data is displayed by the Madsonic UI embedded player or a mobile Madsonic client.

Note that this ffmpeg transcoding is a simple copy of the basic Madsonic transcoding for video files except "-i %s" is changed into "-i -", because we have to generate our input file "on the fly".

3.3 Last step
We must tell the Madsonic UI player that it should also accept "TV" transcoded streams.

embedded player is also responsible for .tv files

Thats it. After a Rescan of your media folders Madsonic shows a new folder (see upper circle) and the stream descriptors on the lower left side (see rectangle). Clicking on play will start our transcoder and after a few seconds the Live TV starts. You can change the bitrate - our transcoder will follow easily.

If you like what I have done please leave a comment.

Regards, Ebeneezer.

Magic done.

[Madevil]

Many, many thanks for your epic detailed guide!!

Best Regards,
Madevil

[ebeneezer]

Script to auto-sync channel bouquets from set top-box:

Code: Select all

#!/bin/bash
# inurl: services.m3u input URL
# outpath: output directory

inurl=http://dm8000.private/web/services.m3u?bRef=1%3A7%3A1%3A0%3A0%3A0%3A0%3A0%3A0%3A0%3AFROM%20BOUQUET%20%22userbouquet.dbe00.tv%22%20ORDER%20BY%20bouquet
outpath="/Volumes/LaCie/iDocs TV Bouquets/iDocs TV"

cd "$outpath"
rm -f *.tv

curl "$inurl" -s | while read line           
do
    token=$(echo $line | cut -f1 -d,)
    sender=$(echo $line | cut -f2 -d, | tr '/' ' ')
    if [ "$token" == "#EXTINF:-1" ]; then
       read line
       printf "Madsonic Stream URL for $sender generated $(date)\n$line\n" > "$outpath/$sender.tv"
    fi
done

After compiling a newer x64 binary of ffmpeg from Macports i fiddled around with the ffmpeg parameters and found this code to optimize CPU usage and picture quality (my Mac Mini has 4 cores - adjust your thread parameter):

Code: Select all

ffmpeg -i - -async 1 -b:v %bk -analyzeduration 4000 -probesize 10000 -ar 44100 -ac 2 -vcodec libx264 -preset ultrafast -acodec libmp3lame -loglevel info -threads 4 -filter:v yadif -f flv -

I also tested avconv but it seems to consume more CPU than ffmpeg. Be sure to include all CPU capabilities when compiling libx264. Please consider

Code: Select all

sudo port upgrade --enforce-variants x264 +asm


to get

[2014-01-07 15:57:10,044] DEBUG InputStreamReaderThread - (/Library/Application Support/madsonic/transcode/ffmpeg) [libx264 @ 0x7f7fb4874600] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX

Code: Select all

ffmpeg version 2.1.git-a044a18 Copyright (c) 2000-2013 the FFmpeg developers
  built on Jan  7 2014 14:10:02 with Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-libtheora --enable-libschroedinger --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libspeex --enable-libass --enable-libbluray --enable-gnutls --enable-libfreetype --disable-indev=jack --disable-outdev=xv --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --arch=x86_64 --enable-yasm --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid --enable-nonfree --enable-libfaac
  libavutil      52. 56.100 / 52. 56.100
  libavcodec     55. 45.100 / 55. 45.100
  libavformat    55. 22.100 / 55. 22.100
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     3. 92.100 /  3. 92.100
  libavresample   1.  1.  0 /  1.  1.  0
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...


Regards,
ebeneezer_goode

[O2L]

hello ebeneezer,


great news, thx for you tuto.

Do you think the transcoding live tv can be use on freenas?

What is your cpu consume and your configuration for the SD and HD channel?

best regards

[O2L]

Hello,


I reinstall madsonic in my Freenas (9.2.1) because Offset and bitrate doesn't work.

Now all work perfectly for music and video.

I try your tutorial ebeneezer, but nothing worked with the enigma1 stream worked.

When i try "reading" .tv file, i have this error

Code: Select all

Video not found or access denied: /stream?id=...

I rename a .mp4 file in .tv file and i can read without perfectly, just i cannot change offset and bitrate in this case, it's not the case with the same file with .mp4 extension. Probably the ffmpeg parameters are different.

I test the steps (tail xarg and curl) in a ssh and it's work, and i can "see" the streaming with the ffmpeg line in ssh.

2 questions:
Most important, do you can change bitrate for the tv stream? (i wan't the tv stream outside of my network)
Where can i activate a log in madsonic or ffmpeg more verbose ? to understand what's occur.

[ebeneezer]

Hi,
can you post the complete content of one of your .tv files?

And a system log of madsonic with debug setting exactly at the point where you start the tv streaming.

What kind of client do you use?

And last: Yes, you can change the bitrate on-the-fly within the flash player while streaming.

Regards,

EG

[O2L]

Thx ebeneezer for the reply,

my .tv contain:

Code: Select all

#EXTM3U
#EXTVLCOPT--http-reconnect=true
http://192.168.0.5:31344

When i apply tail or xargs on the .tv in ssh it's works

the log appear strange, the .tv files don't generate any ffmepg event, it's not the case for a .mp4

Code: Select all

[2014-02-21 16:03:31,824] INFO PlayQueueInputStream - admin listening to "Streams/stream-31344.tv"
[2014-02-21 16:03:34,738] INFO PlayQueueInputStream - admin listening to "Streams/stream-31344.tv"
[2014-02-21 16:05:19,481] INFO PlayQueueInputStream - admin listening to "Streams/Family.Guy.S12E09.mp4"
[2014-02-21 16:05:19,509] DEBUG TranscodeInputStream - Starting transcoder: [/home/ZikUser/madsonic/transcode/ffmpeg] [-ss] [0] [-i] [/mnt/5HD-Downloads/mp3/Streams/Family.Guy.S12E09.mp4] [-async] [1] [-b] [500k] [-s] [480x272] [-ar] [44100] [-ac] [2] [-v] [0] [-f] [flv] [-vcodec] [libx264] [-preset] [superfast] [-threads] [0] [-]
[2014-02-21 16:05:19,585] DEBUG InputStreamReaderThread - (/home/ZikUser/madsonic/transcode/ffmpeg) ffmpeg version 0.7.15, Copyright (c) 2000-2013 the FFmpeg developers
[2014-02-21 16:05:19,585] DEBUG InputStreamReaderThread - (/home/ZikUser/madsonic/transcode/ffmpeg)   built on Aug 12 2013 11:01:01 with gcc 4.2.1 20070831 patched [FreeBSD]
[2014-02-21 16:05:19,585] DEBUG InputStreamReaderThread - (/home/ZikUser/madsonic/transcode/ffmpeg)   configuration: --

More strange, the same .mp4 with the .tv extension, i can read the file, but changing the bitrate have no effect (that's why my first question )
and in the log no event for ffmpeg

Code: Select all

[2014-02-21 16:05:32,994] INFO PlayQueueInputStream - admin listening to "Streams/Family.Guy.S12E09.HDTV.x264-LOL.tv"
[2014-02-21 16:05:47,389] INFO PlayQueueInputStream - admin listening to "Streams/Family.Guy.S12E09.HDTV.x264-LOL.tv"


I will dive in the commands (tail, curl xargs) of 2 first steps, maybe a problem of permission.

Can you compare the .tv file event with your log ?


I try on 2 pc with win7 on chrome, FF and IE, and samsung explorer in android phone.

[ebeneezer]

Hi O2L,

seems your stream descriptors are wrong. Following is one of my descriptors - note the correct stream URL on the last line of the file. You have to get a .ts stream from your Enigma2 box. The correct URL for that can be retrieved out of the Enigma Web UI as described in the tutorial.

Code: Select all

Madsonic Stream URL for 3sat HD generated Tue Jan 28 00:01:25 CET 2014
http://dm8000.private:8001/1:0:19:2B8E:3F2:1:C00000:0:0:0:

Log

size.
[2014-02-21 20:37:54,455] DEBUG TranscodeInputStream - Starting transcoder: [/Library/Application Support/mads
onic/transcode/tail] [-n1] [/Volumes/LaCie/iDocs TV Bouquets/iDocs TV/3sat HD.tv]
[2014-02-21 20:37:54,468] DEBUG TranscodeInputStream - Starting transcoder: [/Library/Application Support/mads
onic/transcode/xargs] [curl] [-s]
[2014-02-21 20:37:54,481] DEBUG TranscodeInputStream - Starting transcoder: [/Library/Application Support/mads
onic/transcode/ffmpeg] [-i] [-] [-async] [1] [-b:v] [1000k] [-analyzeduration] [4000] [-probesize] [10000] [-a
r] [44100] [-ac] [2] [-vcodec] [libx264] [-preset] [ultrafast] [-acodec] [libmp3lame] [-loglevel] [info] [-thr
eads] [4] [-filter:v] [yadif] [-f] [flv] [-]
[2014-02-21 20:37:54,611] DEBUG InputStreamReaderThread - (/Library/Application Support/madsonic/transcode/ffm
peg) ffmpeg version 2.1.git-a044a18 Copyright (c) 2000-2013 the FFmpeg developers
[2014-02-21 20:37:54,611] DEBUG InputStreamReaderThread - (/Library/Application Support/madsonic/transcode/ffm
peg) built on Jan 7 2014 14:10:02 with Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
[2014-02-21 20:37:54,611] DEBUG InputStreamReaderThread - (/Library/Application Support/madsonic/transcode/ffm
peg) configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libm
p3lame --enable-libvorbis --enable-libopus --enable-libtheora --enable-libschroedinger --enable-libopenjpeg --
enable-libmodplug --enable-libvpx --enable-libspeex --enable-libass --enable-libbluray --enable-gnutls --enabl
e-libfreetype --disable-indev=jack --disable-outdev=xv --mandir=/opt/local/share/man --enable-shared --enable-
pthreads --cc=/usr/bin/clang --arch=x86_64 --enable-yasm --enable-gpl --enable-postproc --enable-libx264 --ena
ble-libxvid --enable-nonfree --enable-libfaac
[2014-02-21 20:37:54,615] DEBUG InputStreamReaderThread - (/Library/Application Support/madsonic/transcode/ffm
peg) libavutil 52. 56.100 / 52. 56.100
[2014-02-21 20:37:54,615] DEBUG InputStreamReaderThread - (/Library/Application Support/madsonic/transcode/ffm
peg) libavcodec 55. 45.100 / 55. 45.100
[2014-02-21 20:37:54,615] DEBUG InputStreamReaderThread - (/Library/Application Support/madsonic/transcode/ffm
peg) libavformat 55. 22.100 / 55. 22.100
[2014-02-21 20:37:54,615] DEBUG InputStreamReaderThread - (/Library/Application Support/madsonic/transcode/ffm
peg) libavdevice 55. 5.102 / 55. 5.102
[2014-02-21 20:37:54,615] DEBUG InputStreamReaderThread - (/Library/Application Support/madsonic/transcode/ffmpeg) libavfilter 3. 92.100 / 3. 92.100
[2014-02-21 20:37:54,615] DEBUG InputStreamReaderThread - (/Library/Application Support/madsonic/transcode/ffmpeg) libavresample 1. 1. 0 / 1. 1. 0
[2014-02-21 20:37:54,615] DEBUG InputStreamRea

Bitrate

Dynamic bitrate selection

Regards,
EG

[O2L]

Hello ebeneezer


Thx a lot to taking time to reply.

Your log help me to understand that i have to copy tail xargs and curl to the transcode folder too.

Now all works perfect.

FYI: in enigma1 the stream is send in a different port than enigma2/DM800 and with the Zapstream function the path of the stream is always the same for all channels.
I change channel direct in the webinterface (webIf) of Dreambox.

Maybe in your box the path of each stream channel activate the channel without you need to zap to the channel by remote or webIf?

Any solution to Watch the channels in apple products (no flash)?.

Thx again for your time.

[ebeneezer]

Glad it worked - but may i cite from the tutorial:

All those commands must reside in the Madsonic transcoding folder (see explanation inside Madsonic UI).

Regards,
Ebeneezer

[muiz]

I tried this, but it didnt work for me.
im running madsonic on a Synology NAS.
So i had to find out another way to do this.

What i do is install VLC player with plugins.
Then in Madsonic: Go to settings Internet TV/radio , and add the stream urls there.



When i click them VLC player will play them for me.

[ebeneezer]

Hi muiz,

nicely done - but is there really transcoding with your solution?

It seems that the mpegts stream isn't processed in any matter.

regards,
EG

[muiz]

No need for that.
It uses the VLC plugin.

And when you forward port 8001, you can use your public ip and watch tv from anywhere.

[ebeneezer]

In that case you don't need madsonic whatsoever.

[Papillion2012]

Hello Ebeneezer,
I Madsonic installed and set everything up according to your instructions.
Then I bought the Madsonic client on Google and tried everything.
Music plays on the server and on the client.
Video, TV talk shows on the server and works well.
On the client I can see the transmitter, but not play.
What should I do to play the TV transmitter on the client
Please kindly to your or your help in this case.
I work with Madsonic 5.1.4800.beta2
My client Version is 5.5 from 2014.02.05
Kind Regards Jurgen