baküzen

Sphinx4 has changed a lot since my original posts about it. The conversion to mp3 is the same, but the original post has changed a little bit. It’s easier than ever to simply download and use sphinx4 without too much effort.

The new and improved website: http://cmusphinx.sourceforge.net/

You can either checkout the svn, or just get the tar. You will also need some acoustic models.

If you download the tar, untar it (tar -xvf sphinx4.tar). You will need to go to the lib folder and extract the jsapi jar from either the .exe file or the .sh file (just run ./jsapi.sh and follow the instructions). You should also download the WSJ acoustic model and copy it into your lib folder if it isn’t already there. After that, you can navigate to the sphinx4 root folder and run the following command:

java -mx312m -jar bin/HelloWorld.jar

This will invoke the HelloWorld jar which uses a simple grammar and records your microphone input.

If you want to get it working in eclipse, you can checkout subversion or create your own project by including the sphinx4.jar and jsapi.jar. As before, you will need a config.xml file. If you check it out from svn, you will see several apps that you can build from src/apps (edu.cmu.sphinx.demo). Each has its own class and config file which you can use as an example.

For example, HelloWorld uses a grammar to determine which words to use, wheras HelloNGram uses a language model. One nice thing about this version of sphinx4 is that it separates the dictionary and language model from the acoustic model. For example, you can use the WSJ acoustic model, but create your own grammar or language model tailored to your specific needs. You can use any ARPA formatted language model file (you can specify the order that you want to use in the config file). You can always use the online tool which makes simple trigram language model files: http://www.speech.cs.cmu.edu/tools/lmtool-adv.html

Feel free to ask questions, but the documentation and demos make it easy to understand and begin to use.

It’s easy to get boost in Ubuntu Maverick:

sudo apt-get install libboost-dbg libboost-dev libboost-all-dev libboost-doc libboost-date-time-dev libboost-filesystem-dev libboost-graph-dev libboost-iostreams-dev libboost-math-dev libboost-program-options-dev libboost-python-dev libboost-regex-dev libboost-serialization-dev libboost-signals-dev libboost-system-dev libboost-test-dev libboost-thread-dev libboost-wave-dev

Which installs boost 1.42. If you need an earlier version, then you might have to get the tarball and compile it yourself, which, if you can get past bjam, is pretty straight forward.

Moses is, as stated in the title, a machine translation system. That is, it’s an open-source system that one can download and use to translate, potentially, from any language to any language. That’s saying a lot, but there’s also a lot to do in order to get it working.The moses website is actually quite good, so before you try my instructions, follow the instructions there: http://www.statmt.org/moses/But if you still don’t know what to do, I might be able to help. I’m going to step through getting the source code from an SVN repository via eclipse. You can get the source code via the website in a tar file, or you can just download the binary and run it without compiling, assuming your computer can handle the binary.<br>

The Environment: One could potentially use any means of an editor and svn, but, like I said, I will use eclipse. Eclipse was written for Java, but I find it, though not perfect, pretty functional with other languages, including C/C++ in which moses was written. So, you will need a version of eclipse that has the plugin for C/C++. If you don’t know how to install it, just download a fresh eclipse with it already included here:http://www.eclipse.org/cdt/downloads.phpThe other thing eclipse will need is a way to access svn. If you know of one, add it and make it work. Or, you can just install subclipse. Once you have your eclipse up and running with C/C++ capabilities, you can then go to help->install new software->add (type Subclipse for the name and the URL is: http://subclipse.tigris.org/svn/subclipse/tags/subclipse/1.6.5) then just continue on until it downloads and installs it. You’ll need to restart eclipse to have it take effect.Now, with C/C++ and SVN capabilities in eclipse, you are ready to get your hands dirty with moses.

Getting Moses: In eclipse, go to File->new->other->svn->Checkout Projects from SVN. Create a new location and click next. Add the following:https://mosesdecoder.svn.sourceforge.new/svnroot/mosesdecoderandhit next. It will take a second to download the information. There are many branches that people are working on (I currently use the config-switching branch for several reasons) but if you just want to get in and play, you can just click on “trunk” and then click next. Lucky for you, the people who maintain moses include eclipse project files, so it is quite easy to get set up with eclipse. Anyway, you may want to change the project name, or you can leave it as trunk, it doesn’t really matter. Make sure it sets it up as a C/C++ project, or things won’t work out right later. It should be an empty C++ project. Then click finish. It’ll take a few minutes to download. Once it’s done, all is not quite ready.

To Build: I went to Project and de-selected “Build Automatically” so I could tell it when to build (compile).There are a few things done be done before we can build. The makefiles that tell eclipse what to compile aren’t even there yet, we have to generate them. It’s quite easy, however. Open up a console and navigate to the directory where your moses code is. Then run:./regenerate-makefiles.shThis will take a few seconds and will generate your makefiles based on how your computer is configured. The only problem I ran into was this: possibly undefined macro: AC_PROG_LIBTOOL and I was able to fix it easily by installing libtool (in ubuntu: sudo apt-get install libtool -thanks to this site for the info) and then tried again and it worked fine. Now, this next step will separate those who just want to get their homework done from those who really want to use moses. You have your makefiles, but you also need to tell moses where to find a few things. If you already have the phrase tables, that is the data required to train the moses (statistical) machine translation engine, then you can simply type./configureand let it run. If you want to make your own phrase tables, you’ll need to install either srilm or irstlm, or both. These are separate pieces of software that do a lot of the data building necessary to make moses work (why reinvent the wheel?). Moses is nice to be compatible with different kinds, so pick the one you want. Installation for both can be tricky (I found irstlm much easier), but doable. Perhaps in a later post I’ll explain how to install them. Until then, be happy with the little bit that moses comes with.Now, go back to eclipse, right-click on your project and hit refresh. Then, click on Project-Properties->C/C++ build. I deselected “Generate Makefiles automatically” and then click on “Workspace” and just then clicked on the root workspace folder and clicked okay (something like ${workspace_loc:/moses} showed in the Build directory field, where moses was the name of my project). This tells it to look in the moses folder for a Makefile, which we generated earlier.Now, press Ctrl+B and it will take a few minutes to build. It’s compiling C and C++ code using make, so eclipse really isn’t doing much but calling it for you. You can click on the Console tab in the lower part of eclipse to see what it’s doing at any given time

To Run: When it’s done compiling, you can give it a test. Have fun. Just kidding, this is how you try it out: notice that you have a new list called “Binaries” in your project explorer. Expand that and you’ll see everything that was just compiled. Right-click on moses and run as local C/C++ Application. Then it will run, but not really. It just spits out the help information because you provided no command line arguments. The problem is, we don’t have any phrase tables for it to use to actually do translation. The moses website provides a sample one for testing to see if your compile worked. You can download it here:http://www.statmt.org/moses/download/sample-models.tgz

Now, by no means is this going to be what you use to actually do some translating. This is just a tiny sample that utilizes the moses MT system, but with a very small amount of training data and only select phrases to translate. Sorry, if you want data, you’ll have to make your own using some parallel corpora (something I hope to discuss later).Once you downloaded the sample_models.tgz file, you can open a command window and navigate to where it is and then run:tar -xzf sample-models.tgzand then go into the new sample-models directory that it just made. Then go into the phrase-tables directory. This is what you need. Open up the moses.ini file with an editor and change the line under [ttable-file] to the path where it currently is (in my case it was on my desktop under Desktop/sample-models/phrase-table/phrase-table) and then save it and close.We’re getting close. Now, you need to note where the moses.ini file is on your computer. Now, go to eclipse and then click on the green “Run” button (looks like a “Play” button”) make sure you hit the down-arrow part, and then click on Run Configurations. Under C/C++ Application, you’ll see moses (it’s there because you tried to run it before). Select it and go to the Arguments tab on the right. Then type in the following:-f {path to the moses.ini file}In my case it was something like

:-f /home/something/Desktop/phrase-table/phrase-model/moses.ini

That’s all you need. Now, click “Apply” and then “Run” and then you’ll notice that it gets into motion. After a few seconds of running, it stops. At this point it is waiting for input. As this is a German-English sample phrase table, you can type:das ist ein kleines hauspress enter, and you should see the translation:this is a small houseThat’s it. You’ve successfully used moses to translate something. Congratulations. Now, to actually use moses in a big way is up to you. You can look into the boost library and work with multi-threading, or you can get the srilm and create your own phrase tables to feed to moses to do your own translations. There is a lot you can do, so check the website and see what’s available.

Appendix:You may need some other things, so I included them here without descriptive steps:If you need tcl (srilm uses tcl):.sudo apt-get install tcl tcllib tcl-devTCL_INCLUDE, TCL_LIBRARY: to whatever is needed to find the Tcl header files and library. If Tcl is not available, set NO_TCL=X and leave the above variables empty.

I had to copy the /usr/include/tcl8.5/ files to the srilm/misc/src dir

Also, exclude LanguageModelRandLM from compile!

I also put the srilm directory in the same as moses

To get boost:

http://cl.aist-nara.ac.jp/~eric-n/ubuntu-nlp/dists/jaunty/all//etc/ld.so.confboost:sudo apt-get install libboost-date-time-dev libboost-date-time1.34.1 libboost-dev libboost-doc libboost-filesystem-dev libboost-filesystem1.34.1 libboost-graph-dev libboost-graph1.34.1 libboost-iostreams-dev libboost-iostreams1.34.1 libboost-program-options-dev libboost-program-options1.34.1 libboost-python-dev libboost-python1.34.1 libboost-regex-dev libboost-regex1.34.1 libboost-signals-dev libboost-signals1.34.1 libboost-test-dev libboost-test1.34.1 libboost-thread-dev libboost-thread1.34.1tUse flag for compiler:-std=c++0xhttp://www.52nlp.com/moses-support-digest-moses-compilation-problem-on-fedora-11/in sphinx, i removed the randlm related flags to the compilehttp://www.statmt.org/moses_steps.htmlbinarize: (make sure you use the .gz compressed version)processPhraseTableprocessLexicalTable

I’ve been getting a lot of comments, questions, and emails lately which is great, but this week is finals week for me in my master’s program. Therefore, I’m quite busy. I’ll have time to look at some of your comments in a week or two.

The Python Natural Language Tool Kit has a lot of stuff to offer the DIY NLPer. It has a parser, POS tagger, lambda calculus, a chunker, a classifier, a tokenizer, even a WordNet interface, and much, much…..much more. It’s loaded and it’s not terribly difficult to use granted you know some python and at least a little bit about NLP.

First, you need python, the programming language in which that the tool kit is developed. Most distros of Linux will have python installed, but if you don’t have it you can go to www.python.org to get it and download it. If you don’t know how to do that, you’re hard-pressed to know how to use the tool kit anyway. So, spend some time learning python before you go crazy with the tool kit.

If you’re beyond that and you’re ready for the tool kit, you can go to www.nltk.org and download it. I tried a few different things and ended up just getting the zip file and downloading that, extracting it, then go into the directory with your console and type (as root or sudo):

python setup.py install

and you’re almost done. Run python by just typing:

python

and you’ll see the python command-line interpreter interface. Type:

include nltk
nltk.probability.demo()

and you should see some output with some frequency distributions. There are more tutorials on how to use the tools individually on the www.nltk.org website.

One more thing. You might want to include some of the other optional packages, like numpy. Go back to the same download site as the nltk and grab what you want.  Open a console and get to where you downloaded the file,then run:

tar -xvf numpy[ver]

Then go into your numpy directory and run (again, as root/sudo):

python setup.py install

It will take some time because it is also compiling a lot of c code. Best of luck. I’ll post more as I learn more about it.