Tuesday, April 22, 2008

Grab Radio Shows Instantly

Whoohoo! I just found out the radio show I listen to is automatically posted every day on the newsgroup alt.binaries.sounds.radio.misc! This makes the program from my last post much simpler. Instead of capturing and encoding and tagging all I need to do is download the show using nget.

First I downloaded the nget binaries for Windows. I extracted this into "C:\nget" using 7-Zip. Then I setup the nget configuration with this .bat file.

@echo off
rem Create the _nget5 directory.
mkdir "%USERPROFILE%\nget5"
rem Copy a fresh _ngetrc
copy C:\nget\_ngetrc "%USERPROFILE%\nget5"
rem Open notepad to edit the file
notepad "%USERPROFILE%\nget5\_ngetrc"

The _ngetrc configuration file is fairly easy to setup. When notepad opens all you need is to look for the (halias line and change <yourhostalias> to MyHost, <yourhostaddress> to "news.whateveryourhostis.com", remove the "#" from the "user" and "password" lines, then change them to match your user name and password. In my case it was for GigaNews. So I changed it to news.giganews.com and then added my user name and password. Incredibly easy.

I then wrote this PureBasic script to fetch the file, extract it and clean up.

;; Change These Variables ;;

home.s = GetEnvironmentVariable("USERPROFILE")
dir.s = home + "/My Documents/My Music/Show"
tmpdir.s = home + "/Local Settings/Temp"
e7z.s = "C:/Program Files/7-Zip/7z.exe"
nget.s = "C:/nget/nget.exe"
group.s = "alt.binaries.sounds.radio.misc"

;; Do The Download ;;

started = Date()
showExp.s = FormatDate("Show.%dd-%mm-%yyyy.", started)
SetEnvironmentVariable("HOME", home)
SetEnvironmentVariable("TMPDIR", tmpdir)

; If it's Saturday or Sunday then skip out.
If DayOfWeek(started) = 0 Or DayOfWeek(started) = 6
PrintN("Show is weekdays only! Exiting..")

; Grab the show.
If RunProgram(nget, "-g "+Chr(34)+group+Chr(34)+" -r "+Chr(34)+showExp+"*"+Chr(34), dir, #PB_Program_Wait) = 0
PrintN("NGet failed!")

; Check to see if we got it.
If FileSize(showExp+"part1.rar") <= 0
PrintN("Failed to get show: "+showExp)

; UnRar the files.
If RunProgram(e7z, "e "+showExp+"part1.rar", dir, #PB_Program_Wait) = 0
PrintN("Could not start 7-Zip for decompression!")

; Check if that succeeded.
; Note: I could use ProgramExitCode() to see
; if it failed or not. I'm just being lazy.
If FileSize(showExp+"mp3") <= 0
PrintN("Decompression failed!")

; Now unlink the .rar and .nzb files.
#dir = 0

; Note: ExamineDirectory() supports wildcards, that
; is what the +"*" is. It's almost as cool as
; Perl filename globbing.
If ExamineDirectory(#dir, dir, showExp+"*")
While NextDirectoryEntry(#dir)
; Skip "." and ".."
If DirectoryEntryType(#dir) = #PB_DirectoryEntry_File
name.s = DirectoryEntryName(#dir)
; Skip deleting the mp3.
If Right(LCase(name), 3) = "mp3"
; But delete everything else.
If DeleteFile(dir+"/"+name) = 0
PrintN("Failed to remove file: "+dir+"/"+name+" : "+GetErrorDescription())

You know what the amazing thing is? When this is compiled I get a 25kB .exe file! That's all that is required to run it. I don't need any kind of interpreter, because there is no interpreter. It's a compiled language baby! This is the stuff magic is made of, the worlds first ultra high level compiled language.

Ruby Programming

Ruby sucks. It's that simple.

I'll admit I'm a long time Perl programmer, I'm biased. I started programming life on a Commodore Amiga 2000 back in 1988. My first "Hello World" was in Lattice C. Followed swiftly by a change to Easy AMOS Basic. After the Amiga died the death of Beyond 2000, I switched to the wild and woolly world of PC programming. It was a bloodbath back then. The Amiga programmers were feeling their wild oats all over the faces of PC programmers. Nibbles was ASCII, a beep speaker made beeps. The Amiga programmers could play music and add sound effects with thousands of colors and countless animated naked guys with swords. Eventually I found Linux in '97 and never looked back. We had a website, I wrote a shopping cart for it in Perl. This language made computers what they always should have been. It's a lever on the heart of the machine. You need the answer to a question? You can fire off a quick one liner and bring those precious bogoMIPS to bear in a explosive shower of scrolling ASCII and mysterious regular expressions.

Well, after years of consistent daily Perl, C and Javascript programming I thought I had found all the tools my box might ever need. So naturally I decided to uninstall Perl from my box and install Ruby. Dim witted people are fascinated with Ruby on Rails. I have never been. I've written (or been involved with the writing of) three different complete (still in production) commercial web application frameworks in two different languages. RoR has a Reality Distortion Field equaled only by Steve Jobs. So it's a nice framework, so what? It's dog slow, written in a language that (still) lacks a decent backing library (I'd coin the acronym RINC, RAA Is Not CPAN) and it really offers nothing that isn't available in Perl, Python, PHP or Java. Yet, this is almost single handedly the main thing that brings people to ruby. It's amazing, with a nice name and a good propaganda line you can sell people anything. But I digress, ruby is actually a fairly cool language (I'll get to that in a minute.)

At the very least it whips Python like a tied up diaper sissy. Why you ask? Python is a toy. If programming languages were vehicles, then Python would be a tricycle. It's what you use before you learn how to ride a bicycle, motorcycle or a car. Python protects you, it tells you what you can and cannot do. Python is a crossing guard at the programming kindergarten. It's easy to use, it saves you from what it thinks are bad programming practices. And it single handedly taught a generation of programmers to be whitespace Nazis. Nevertheless, when you grow up, you will decide that touching other object's private variables can be an exciting, naughty and only slightly guilt laden activity. Perl and Ruby make this secret delight a worshipful experience, Python is a bitch nun about it. I can do whatever I want dammit! No matter what Guido says! Fine! I'll leave! Become a Perl programmer. It's nature's natural progression. Big boys can handle name space management.

This was James Gosling's fatal flaw when he created Java. He had no understanding of human nature. His obsessive compulsive control freakdom has tarnished the programming landscape forever. Have you ever met a Java programmer? They're poor shadows of men, rife with cognitive dissonance. They function much like any religion, in that they claim one thing and then do another. The first great act of a Java programmer is finding out ways to beat the system. You see the guilty publications and "brown bag" software out among the Java programming landscape. It hides under names like "introspection" and "byte code decompilation". But admittedly, Gosling's baby grew up into an irate old maid. Only the deep magicians can brew up enough GHB to get her to put out. But there is always, always, always those who spend hours and hours trying.

Take a look in C++ to see why this philosophy is a miserable failure. How many times have you seen (or used) '#define private public' in commercial code? I've seen it several times. Thankfully c++ has this feature, when your ass in on the line it comes in handy.

But this post is about my experiences in the last few weeks using Ruby as my system administration scripting language. I will say that Ruby is pretty cool. It's like a light infusion of Smalltalk, Perl and BASIC. Ruby is just as pragmatic as Perl about providing useful syntactic sugar. The difference is, Ruby did not have to support legacy code. So it's a very sweet language. I mean, Ruby has most of the syntactic sugar of Perl and it doesn't have confusing dollar variables (for local variables anyway). This single handedly fixed the primary objection dimwitted Python programmers levied against Perl. It opens up a whole new world! Look ma, no dollar sign!

The first general system administration Ruby script I wrote was a command line tool for editing ID3 tags. It used the id3lib and Ruby's excellent optparse command line argument processor. It took me maybe ten minutes to finish with no prior Ruby programming experience. It's as simple as:

include "rubygems"
include "id3lib"

mp3tag = ID3Lib::Tag->new("mymp3.mp3")
mp3tag.artist = "Bob Dylan"

It can hardly be any easier and there does not appear to be a CPAN module that is nearly as elegant. I couldn't find anything for Python either.

The next project was for my company, so I was on the clock. It was a fairly ambitious script called from one of our video conversion daemon processes. It's job was to make an encrypted, basic authorize protected HTTP connection to an XML-RPC perl script on one of our web servers. It would then ask if users have uploaded any new videos to convert. If they have then it would download them via FTP and INSERT a record into a SQLite3 database indicating this to our conversion software. Simple right? With ruby, yes it was. I won't reproduce the script here, but I can assure you that it's less than a few hundred lines of highly readable code. I don't think I could've written it as succinctly in Perl and I know I couldn't write it as clearly.

Which brings me to my final project and unexpected twist.

Yesterday I wanted to do what seemed like a two minute scripting task. Use mplayer to dump an audio stream of a radio talk show, then convert the file to an MP3 and tag it. It would run daily from crond or svchost and do its thing so I wouldn't miss my daily dose of right wing bantering. It had to be cross platform (so I could use it on my Windows laptop as well as my Linux desktop.) It sounds simple right? If I didn't want a single solution then I could just use a few BASH scripts called from the crontab; one to start the capture, another one to kill the first at the end of the show and a third one to do the conversion and tagging. Simple. But with Ruby on my side, I could whip something out easily, right?

Wrong. It turns out that Ruby isn't very good at Windows scripting tasks (it doesn't know how to fork() or kill() properly.) My script started out looking like this:


# gem install win32-process
require 'win32/process' # Comment this out when we're on Linux.
require 'timeout'

mplayer = "C:\\mplayer\\mplayer.exe"
stream = "http://someradiostation.co.uk/stream.asf"
dir =
"C:\\Documents and Settings\\u\\My Documents\\My Music\\Show"
title = Time.now.strftime("Radio Show - %Y-%m-%d")

t = Time.now

pid = fork()
if pid != nil
puts "one: "+ pid.to_s
t = Time.now
while 1
n = Time.now
if (n - t) > 15.0
puts "watcher: Killing: "+pid.to_s
Process.kill "SIGINT", pid
puts "watcher: Sleeping..."
sleep 5
Dir.chdir dir
system mplayer, '-streamdump', stream

Doesn't that look beautiful?? It's so simple! It doesn't work, and it's not possible for it to work, but it's beautiful!

Why doesn't it work you ask? Well, kill "SIGINT" was obliterating the mplayer process. It was causing the process to do odd things, sometimes it would die, other times it would appear to be dead but would still show in the process list (and it would happily keep grabbing the stream.) This seems to be irreparable, it's a problem with win32-process.

The solution?? Is it Perl? No (but that would work since fork() and kill() work.) Is it Python? No (but that would work too.) The answer came in a whole different language that you would never suspect as a scripting language; PureBasic. I had bought a copy of it last year and I've used it for small systems programming tasks around the company. The thingg is, it's a truly compiled language. It generates native machine code in the same way a C, C++ or Pascal compiler does. The thing is, the syntax is nearly as flexible as Ruby or Perl. Granted, there isn't as much syntactic sugar. But as you can see, it's makes for a great maintenance/administration language. This is the final script in PureBasic.

;; User Serviceable Variables ;;

ffmpeg.s = "C:/mplayer/ffmpeg.exe"
mplayer.s = "C:/mplayer/mplayer.exe"
stream.s = "http://someradiostation.com/stream.asf"
dir.s = "C:/Documents and Settings/u/My Documents/My Music/Show"
title.s = FormatDate("%yyyy-%mm-%dd Show", Date())
author.s = "Show Name"
genre.s = "Speech"
album.s = "Show"
logFile.s = dir+"/grabShow.log"
checkEvery.l = 60
showLength.l = (3 * 60 * 60) + (5 * 60) ; 3 hours plus five minutes of leeway.

;; "Private" Program Variables ;;

success.l = #False
tries.l = 0
started.l = Date()
when.s = FormatDate("%yyyy-%mm-%ddT%hh-%ii-%ss", started)
dump.s = dir+"/stream.dump"
wma.s = dir+"/stream.wma"
checkEvery = checkEvery * 1000

; Delete yesterday's scratch files.


; Open the log file.
If OpenFile(#Log, logFile) = 0
MessageRequester("Error: grab.exe", "Failed to open log file: "+Chr(10)+logFile)

; Position the file pointer to the end of the file.
FileSeek(#Log, Lof(#Log))
WriteStringN(#Log, "Starting process: "+when)

; If it's Sunday or Saturday then exit (since I can't find
; the Weekdays Only option in Scheduled Tasks on Windows.)
If DayOfWeek(started) = 0 Or DayOfWeek(started) = 6
WriteStringN(#Log, "Show is weekdays only! Exiting..")
Goto CleanUp

If tries <= 1 WriteStringN(#Log, "Starting mplayer: "+mplayer+" "+"-dumpstream "+Chr(34)+stream+Chr(34)) pid = RunProgram(mplayer, "-dumpstream "+Chr(34)+stream+Chr(34), dir, #PB_Program_Open) If pid <> 0
Delay(2000) ; Give mplayer time to start up.
; Check if it's still running (i.e. capturing stream.)
If IsProgram(pid) And ProgramRunning(pid)
While 1
now = Date()
; If the show is over.
If now - started > showLength
; Check if mplayer is still running.
If IsProgram(pid)
If ProgramRunning(pid)
; Kill it if it is.
WriteStringN(#Log, "Killing mplayer...")
; And bellow a battle cry of success.
success = #True
; If it's not over then check if it's still grabbing.
If IsProgram(pid) And ProgramRunning(pid)
; Write a note in the log saying we checked on it.
WriteStringN(#Log, "Recorded "+Str(now - started)+FormatDate(" seconds, as of %hh:%ii:%ss...", now))
; Otherwise write a note saying it died unexpectedly.
WriteStringN(#Log, "MPlayer died after "+Str(now - started)+FormatDate(" seconds, as of %hh:%ii:%ss...", now))
; And open a dialog notifying the user so they can fix it.
MessageRequester("Error: grab.exe", "Mplayer died unexpectedly durning stream capture!")
Goto CleanUp

; All went well, go to sleep.
; Mplayer died before we even got started.
; Give it another try, for some reason on Windows
; it does this sometimes and the second try
; always works.
WriteStringN(#Log, "mplayer died!")
tries = tries + 1
Goto TryAgain

; Did the conversion succeed?
If success = #True
WriteStringN(#Log, "Renaming '"+dump+"' -> '"+wma+"'")
dest.s = dir+"/"+title+".mp3"

; Rename the file to a .wma for aesthetic reasons.
If FileSize(dump) <= 0 WriteStringN(#Log, "Did not get a file!") Goto CleanUp Else If RenameFile(dump, wma) = 0 WriteStringN(#Log, "Move failed: "+GetErrorDescription()) Goto CleanUp EndIf EndIf ; At first I was using MPlayer alone to do the capture ; and the conversion to mp3. Then I would use my id3 script ; (written in Ruby) to add an id3 tag. Installing FFMpeg made ; things easier since it supports ID3 and conversion all in ; one elegant command. If RunProgram(ffmpeg, "-i "+Chr(34)+wma+Chr(34)+" -title "+Chr(34)+title+Chr(34)+" -author "+Chr(34)+author+Chr(34)+" -genre "+Chr(34)+genre+Chr(34)+" -album "+Chr(34)+album+Chr(34)+" -y -v -1 "+Chr(34)+dest+Chr(34), dir, #PB_Program_Wait) = 0 WriteStringN(#Log, "Encoding with ffmpeg failed to start!") Goto CleanUp EndIf If FileSize(dest) <= 0 WriteStringN(#Log, "Ran ffmpeg but it did not produce a file!") Goto CleanUp Else WriteStringN(#Log, "File finished in: "+dest) EndIf EndIf CleanUp: WriteStringN(#Log, "Ending process: "+when) CloseFile(#Log)

How about that huh? Simple, cross platform (Windows,Linux and MacOSX), reasonably easy to read and it only take up 35k of memory when it's run. The only disadvantage is the need to recompile whenever you change something, but honestly, how often do you change crontabs like this? It's mostly write and find out about again two years later when it failed in the middle of the night and filled your Inbox with warning emails (or emails from your boss, if you didn't write a Email Me On Failure system.)

The point is, which is more elegant and easy to understand; item a "Time.now.strftime("%Y-%m-%d Show")" or item B "FormatDate("%yyyy-%mm-%ddT%hh-%ii-%ss", Date())? I think PureBasic is actually a good deal more cleaner than Ruby. They both share the fact that you must typecast variables. If PureBasic added some regular expression syntactic sugar (possibly implemented through PCRE) then you might not need a scripting language like Python, Perl or Ruby at all. PureBasic has the added advantage that you have the option to choose closed source distribution of your application. It's also blazing fast and has cross platform native GUI support (without all the bloat of wxWidgets or even FLTK.) Furthermore, it has the full power of the C API's coated in easy to use goodness. If there isn't a cross platform built in function - it's easy to write one using native APIs. What's not to like?

Well, it doesn't have Objects. That's a fly up the nose of OOP evangelicals. I like OOP, I always use it when it's available don't get me wrong, the thing is that all the really massive systems around today are all procedural. All of them. Windows? Procedural (with OOP grafted on.) Linux kernel, written in C? Isn't that right? OS/360? MySQL? Apache? DB2? Perl? Ruby? All written in procedural languages.

The point is not that Objects aren't nice, it's that procedural programming can accomplish quite a lot. It's not an obstacle, in fact, OOP looks more like LISP when it comes to actual applied use versus academic evangelism.

So over the next week I'm going to play with PureBasic as a scripting language, just for the fun of it. And actually ruby is kind of cool but every tool has it's place. I'll use Ruby when I need a clean, easy to maintain, reasonably complex scripting task done that does not rely on external libraries or POSIX functionality. I'll use Perl when I need to fire off a write-once one liner or a high performance mod_perl website. I'll use PureBasic when I feel like it. And I'll never use Python, for the same reason I'll never know what it's like to give birth.