Thou Shalt Not Create Independent Tests

And software companies wonder why we users have trouble taking software end-user license agreements seriously. Roberto Togni actually read the license that accompanies On2’s VP7 Decoder. It contains this clause:

You may NOT:
4.publish or provide any results of tests, including without limitation benchmark tests, run on the Software to any third party without On2’s prior written consent

In fairness to On2, this is not an uncommon clause in EULAs. However, it presents some very curious scenarios. Am I allowed to publish something like this?

I looked at a VP7 sample and it did not look as good as WMV9, H.264, or even On2’s own VP6.

Maybe I should write and ask permission. A colleague brought up another point: Since this “no benchmark tests” is such a common EULA clause, it should stand to reason that Microsoft’s Windows Media Encoder carries the same license. It is extremely unlikely that Microsoft would have granted written permission for this whitepaper exercise.

Update: Again, to be fair, the decoder license (On2’s Truecast Player) does not appear to mention anything about publishing benchmark tests. That is on the limited trial codec license.