1.15.2011

MathWorks MATLAB vs. GNU Octave Speed/Performance Comparison

When I went back to school in 2009 after a year hiatus, I was forced by many classes to start using MATLAB. I of course did not want to use such closed-source software and immediately started using GNU Octave instead. Octave has been great and with the exception of a missing Simulink counterpart, has provided me with every capability needed. I however have been wondering how fast MATLAB is in comparison. I didn't think I'd get a chance...until I needed Simulink and had to install the student version at home.

Using an older laptop with a 32-bit 1.6 GHz Intel Core Duo processor, I installed and compared Octave 3.2.4 and MATLAB R2009b (on Windows). I have been learning some advanced numerical methods from the mechanical engineering department's CFD expert and selected five different homework problems for testing purposes. They all solve the same problem and all use the same initial conditions but use different algorithms obviously. I ran them all with both programs, recorded the times, and calculated the percent differences.

Conclusion: MATLAB will take non-vectorized loops and still give them almost full compiled performance. All of the partially vectorized scripts showed at least a 99.4% reduction in time when run in MATLAB. This translates to MATLAB being about 200-300 times faster than Octave for non-vectorized code. Even for code that was fully vectorized, MATLAB still showed a 2x improvement over Octave. In any case, MATLAB is definitely faster.

This is very disappointing...pretty much a slap in the face to open-source software. I am obviously a big proponent of open-source software, but I can't ignore these numbers. Anyway, while Octave is a nice piece of software, it's obviously too slow for use with anything more complicated than simpler homework assignments.

Edit: Since Filedropper.com sucks and lost my only copy of the data table, I have rerun the scripts on a different machine. This machine has the same clock speed, but is a single-core Intel Atom processor. The software versions and platforms are the same (32-bit Windows).

Edit 1/29/2012: I have re-uploaded the scripts so that anyone may try them. Download them here.

FileMATLAB*Octave*Speedup
HW3_16.1210.171.7
HW3_24.361092.4250.6
HW4_15.55777.6140.1
HW4_20.6884.1123.7
HW51.8366.2203.4
* = time in seconds

The improvement on a single-core processor is of course slightly less than that of the Core Duo; I'm sure that MATLAB automatically parallelizes a lot of the assignments and parts of the for loops. In any case, MATLAB is much, much faster than Octave.

13 comments:

  1. Were you using built in functions/solver or hand writting everything yourself?

    ReplyDelete
  2. Well the scripts were written by me, but they utilized built-in functions for speed when able.

    ReplyDelete
  3. I agree. I wanted to see if I could get my Matlab code to work in Octave.

    I needed to make some changes to my code as follows:

    1. I replaced uigetfile with uigetfile_java where uigetfile_java was taken from the bottom of this page:
    http://octave.1599824.n4.nabble.com/uigetfile-td1899460.html

    I then had to drop all my GUIs and hardcode write these input in a separate file.

    The good news was that I got it running. The bad news is that it was at least 30-40 times slower in Octave than in Matlab. Oh well, worth a try.

    ReplyDelete
  4. Hi, what was the code that you ran? I'm interested in seeing how the vectorised code was slower, before you slap us in the face again. Please send me a copy to jordigh@octave.org

    Thanks

    ReplyDelete
    Replies
    1. So I looked at your code...

      You're always tic-tocking loops, sometimes deeply nested loops, and the more you nest, the more dramatic the difference becomes between Octave and Matlab. Yes, Octave doesn't have JIT compiling (although this may change soon), so this explains the difference.

      For the least nested loop (HW3_1), what was the exact clock speed you're using? I got tic-tocks of an average of 2.06 seconds, which is faster than what you got with Matlab. I'm using an Intel Core 2 Duo with a clock speed of 2.2 GHz. Are you using a clock speed less than 1/3 of that with Matlab?

      I think the old Windows build you're using is also slow... I don't know why, but mingw seems to produce slower code. Do you have access to a GNU/Linux machine with both Octave and Matlab? I'm curious to see if the compiler is generating bad code for Windows.

      Delete
    2. Hello Jordi,

      This article was written quite a while ago, and used the Octave-Forge-made 3.2.4 installer. The 2nd table (lost the 1st as stated) was run using a 1.6 GHz Intel Atom processor.

      I would not doubt that MinGW is the culprit here. I know it links to the old MSVCRT.dll if you have any C code amongst your C++. This is why a lot of people compile C programs with MSVC on Windows instead of MinGW.

      I wish that Octave wasn't so long-winded to compile; it's disappointing to have to wait until Octave-Forge recompiles it for Windows. It was a very long wait to get the 3.4.3 release, and even then it didn't have an installer like the previous release (3.2.4). I know that Octave-Forge is actually a separate project, so that's not necessarily directed at the core Octave team.

      Delete
    3. Though I am speaking in favor of MATLAB in this article, I still really appreciate and like Octave. Octave is very versatile and actually has a much better parser than MATLAB. My main gripe with it is the choice of GNUplot as the default plotting backend. GNUplot's look and speed are just terrible. It would be nice if you guys could use something like matplotlib or something else that is much cleaner looking.

      That aside, you mentioned that Octave might have JIT capability soon. If you guys haven't already checked it out, check out MAJIC:
      http://polaris.cs.uiuc.edu/majic/
      The licensing may be too restrictive to be compatible with the GPL, but there is at least an example of JIT with MATLAB-style syntax.

      Delete
    4. Octave already has its native plot backend for some time. Just use graphics_toolkit fltk

      Delete
    5. Even though Octave has many graphics backends, the only one polished enough for reliable use is the Gnuplot backend.

      Delete
  5. Have you tried SAGE?

    It's OSS but pretty fast

    ReplyDelete
    Replies
    1. I haven't extensively, but have it in my personal stash of bookmarks for future use

      Delete
  6. Try to manually compile Octave with optimized blas, lapack, fftw etc, you will se that Matlab is not the best perfomance.

    ReplyDelete
    Replies
    1. Jose, this may be true for individual statements, but in the case of for loops, especially nested for loops, MATLAB employs JIT compilation to beat out Octave by many orders of magnitude.

      Delete