I was away on a brief Christmas vacation for the last week or so. I was completely absent from ffmpeg-devel list traffic during that time, as well as all email. Occasionally, I would sneak a look at FATE’s main page when I was in the vicinity of an unguarded web browser and clench my fists when I saw that various FFmpeg items were broken, powerless to alert anyone to these facts via email. My tensions eased, however, and my spirits lifted when I would check back a little while later and see that the problems were gone and that the SVN log messages indicated that code was being submitted specifically to address the breakages.
Go team.
However, being away from the action and having FATE as my only window into FFmpeg development reminded me of the various ways the web interface desperately needs to improve. The main fate.multimedia.cx page needs to have a brutally concise summary of the health of the FFmpeg codebase as tested by FATE. I envision that the spirit would be, “Everything is okay… except for the following problems: …”, followed by a list of problems, sorted by newest to oldest.
How to do this? For starters, I have had it on my TODO list for several months now to experiment with some popular PHP web frameworks. I have done enough homework to get over my distaste for the notion of such frameworks and I’m thinking (hoping, really) that one can be of substantial benefit.
Of course, another oft-requested feature is for email notifications when something breaks. Believe me, I really want this too. As you can imagine, this is something I really want to get correct from the start and verify that all the bugs are stamped out. Specifically, I’m paranoid about accidentally spamming a mailing list due to a stupid bug. Also, I want to make sure that such a notification service reports the useful information, where I’m currently defining “useful” as state transitions: report when a build or a test goes from working to broken or vice versa. But I also want to be able to aggregate information about breakages. Example: There are presently 20 FATE configurations. If an SVN commit breaks a test for all the configurations, it would be better for an alert email to report concisely that a recent commit broke a particular test for ALL supported FATE configurations, rather than 20 individual emails representing each configuration, or even one email listing each individual configuration.
It’s a real concern. If I don’t get this right, it’s only going to irritate people right out of the gate and defeat the whole purpose when it goes largely filtered and ignored. I know from whence I speak due to working with similar notification systems on other large codebases.
Indeed, consolidating notifications to per commit rather than per platform is a good idea for traffic reduction, as well as limiting notifications to state changes. I think these alone should probably limit the traffic to a reasonable amount.
Maybe the automated notification could include links to the error output of the test(s) that failed, as well as information about which commit caused the issue and a link to the diff for the commit in web-based source code browser. I think all this would be useful information to which one would like quick access and links aren’t too verbose.
I think it might be good to test this out on a separate mailing list for a while until we’re happy with the traffic it produces. That way you can spam it as much or as little as you like, people can see what’s being sent and can make suggestions or give feedback. Maybe it can even be maintained on a separate mailing list anyway to allow modularity in the e-mail traffic people interested in FFmpeg receive.
Good work! :)
Thanks for the feedback, Robert. I had the same idea about possibly confining the emails to a separate mailing list. But another school of thought is that all developers should be required to see the notifications on the main list. Again, if we design this with the right trade-offs in mind, the emails should not be an annoyance, even during times of high breakage. The important part will be the state transition thing, i.e., not sending emails constantly reporting whether a build worked or did not work (as I have seen with similar systems).
One thing is for certain: My personal email address will be the guinea pig, and I already have a list of situations that the system has to handle appropriately.
Good idea on the links; make it as efficient as possible for developers to see the precise problems.