A few hours ago, I wanted to show to my friends how speedy LuaJIT is. Naturally I went to the Computer Language Benchmarks Game website, but to my great surprise, LuaJIT is gone. I then posted a brief question in this blog asking why. Sindisil has very kindly pointed me to this informative discussion. In short, LuaJIT, along with a few other great language implementations, is gone and probably will never be back.
I have been watching this wonderful benchmark suite for several years and been greatly benefit from it. I really appreciate all that Issac, the owner of this website, has done. I know it must have been taking a lot of his own time.
However, with the withdrawal of various languages implementations, I have to say the benchmark is not of much use now. The great virtue of the old benchmark suite is we can compare different languages AND different language implementations. Take Javascript as an example. There are multiple Javascript engines and we see various benchmarks on how they are compared to each other. Nonetheless, we are also interested in knowing if the best Javascript engine is comparable to the best implementation of other scripting languages. For a long time, the best place to know this is the Computer Language Benchmark Game. Unfortunately, not any more. Now we see that Javascript V8 is the fastest scripting language, 5X as fast as Lua as LuaJIT is not there any more. Ah, the new benchmark is not only of little use but also misleading.
This reddit discussion and a post from a core PyPy developer also point out the weakness of the benchmark, such as over-optimization for one implementation, the use of foreign libraries (especially C libraries), different coding qualities for different languages and the use of unusual/unportable optimization.
Perhaps it is time for someone to come up with a more proper, more open and less biased benchmark.
1) “Probably Issac will argue again that my opinion is biased…”
Please quote my words from that discussion that make you think I suggested your opinion is biased.
I have not suggested that your opinion is biased.
2) “a post from a core PyPy developer”
Alex Gaynor put stuff in a blog post – putting stuff in a blog post doesn’t make it into The Truth.
Sometimes there are facts that you the reader can easily check for yourself –
Alex Gaynor wrote – “It’s also not possible to send any messages once your ticket has been marked as closed, meaning to dispute a decision you basically need to pray the maintainer reopens it for some reason.”
a) Not true. You can check for yourself that there’s a public discussion forum and people dispute decisions –
http://alioth.debian.org/forum/forum.php?forum_id=999
b) Not true. (But hard for you to check.) Followup comments can be added to a ticket that is marked Closed in exactly the same way they can be added to a ticket that is marked Open – and adding a followup comment triggers an email message to me whether the ticket is marked Open, Closed, Deleted, …
(Actually Alex Gaynor did not even add a followup comment to his Closed ticket.)
c) Not true. You can send an email to anyone who has an Alioth login simply by clicking on the email address in their profile – you don’t even need an Alioth login yourself.
etc
3) “the use of foreign libraries (especially C libraries)”
Do you understand that Alex Gaynor wanted to use libc’s write() function in his Python program, and that’s why his program was rejected?
@Issac
As you do not think my opinion is strongly biased, I have removed the relevant sentence. As to Alex Gaynor’s blog post, I think some of his arguments are also biased. It is great that you have these comments to clarify.
Nonetheless, I think it is valuable to add luajit and pypy back, perhaps (if you have time) along with a few other minor languages such as D, scheme and ML.
1) Thank you for the correction to your blog post.
2) The problem is not that some of Alex Gaynor’s “arguments are also biased”.
The problem is that some of Alex Gaynor’s statements are simply not true, and after being told why they are not true Alex Gaynor has done nothing to correct his false statements.
3) If you have time perhaps you should add Scheme and ML and … to your PLB